Skip to contentSkip to conversation

Local kptrs, BPF linked lists

RFC PATCH In review Kumar Kartikeya Dwivedi 5 participants 82 messages Sep 4, 2022 – Oct 18, 2022

Subsystems (7)
lore
Share & tools
lore

Kumar Kartikeya Dwivedi

author maintainer
#0 lore raw

cover Local kptrs, BPF linked lists

WARNING: This is an RFC. WARN_ON_ONCE is sprinkled around the code liberally (useful while working on this stuff). I'll be doing a thorough pass and clean all that up before sending out non-RFC v1.
TODO before non-RFC v1.
- A lot more corner case tests, failure tests, more tests for the new local kptr support. I did test the basic stuff (which the verifier complained about when writing linked_list.c).
- More tests for kptr support in new map types.
- More self review.

--

This series introduces user defined BPF objects, by introducing the idea of local kptrs. These are kptrs (strongly typed pointers) that refer to objects of a user defined type, hence called "local" kptrs. This allows BPF programs to allocate their own objects, build their own object hierarchies, and use the basic building blocks provided by BPF runtime to build their own data structures flexibly.

Then, we introduce the support for single ownership BPF linked lists, which can be put inside BPF maps, or local kptrs, and hold such allocated local kptrs as elements. It works as an instrusive collection, which is done to allow making local kptrs part of multiple data structures at the same time in the future.

The eventual goal of this and future patches is to allow one to do some limited form of kernel style programming in BPF C, and allow programmers to build their own complex data structures flexibly out of basic building blocks.

The key difference will be that such programs are verified to be safe, preserve runtime integrity of the system, and are proven to be bug free as far as the invariants of BPF specific APIs are concerned.

One immediate use case that will be using the entire infrastructure this series is introducing will be managing percpu NMI safe linked lists inside BPF programs.

The other use case this will serve in the near future will be linking kernel structures like XDP frame and sk_buff directly into user data structures (rbtree, pifomap, etc.) for packet queueing. This will follow single ownership concept included in this series.

The user has complete control of the internal locking, and hence also the batching of operations for each critical section.

Eventually, with some more support in future patches, users will be able to write fully concurrent RCU protected hash table using BPF_MAP_TYPE_ARRAY for buckets and embed BPF linked lists in these buckets. All of this will be possible in safe BPF C, which will be proven for runtime safety by the BPF verifier.

The features, core infrastructure, and other improvements in this set are:
- Allow storing kptrs in local storage and percpu maps.
- Local kptrs - User defined kernel objects.
- bpf_kptr_alloc, bpf_kptr_free to allocate and free them.
- BPF memory object model, similar to what C and C++ abstract machines have, now verifier reasons about an object's lifetime, i.e. the concept of object lifetime, visibility, construction, destruction is reified.
The separation of storage and object lifetime is understood by the verifier.
- Single ownership BPF linked lists.
- Support for them in BPF maps.
- Support for them in local kptrs.
- Global spin locks.
- Spin locks inside local kptrs.
- Allow storing local kptrs in all BPF maps with support for kernel kptrs.

Some other notable things:
- Completely static verification of locking.
- Kfunc argument handling has been completely reworked.
- Argument rewriting support for kfuncs.
Now we can also support inlining block of BPF insns for certain kfuncs.
- Iteration over all registers in verifier state has a new lambda based iterator (and can be nifty or crazy - depending on your love for GNU C).
- Search pruning now understands non-size precise registers.
- A new bpf_experimental.h header as a dumping ground for these APIs.

Any functionality exposed in this series is **NOT** part of UAPI. It is only available through use of kfuncs, and structs that can be added to map value may also change their size or name in the future. Hence, every feature in this series must be considered **EXPERIMENTAL**.

Next steps:
-----------
* NMI safe percpu single ownership linked lists (using local_t protection).
- This enables open coded freelist use case
* Lockless linked lists.
* Allow RCU protected local kptrs. This then allows RCU protected list lookups, since spinlock protection for readers does not scale.
* Introduce explicit RCU read sections (using kfuncs).
* Introduce bpf_refcount for local kptrs, shared ownership.
* Introduce shared ownership linked lists.
* Documentation.

Notes:
------
* Delyan's work to expose Alexei's BPF memory allocator as global allocator is still needed before this can be merged. For now, direct kmalloc and kfree is used.

Links:
------
* Dave's BPF RB-Tree RFC series
v1 (Discussion thread)

     https://lore.kernel.org/bpf/[email protected]
   v2 (With support for static locks)
     https://lore.kernel.org/bpf/[email protected]

* BPF Linked Lists Discussion
https://lore.kernel.org/bpf/CAP01T74U30+yeBHEgmgzTJ-XYxZ0zj71kqCDJtTH9YQNfTK+Xw@mail.gmail.com
* BPF Memory Allocator from Alexei
https://lore.kernel.org/bpf/[email protected]
* BPF Memory Allocator UAPI Discussion
https://lore.kernel.org/bpf/[email protected]

Daniel Xu (1):
bpf: Remove duplicate PTR_TO_BTF_ID RO check

Dave Marchevsky (1):
libbpf: Add support for private BSS map section

Kumar Kartikeya Dwivedi (30):
bpf: Add copy_map_value_long to copy to remote percpu memory
bpf: Support kptrs in percpu arraymap
bpf: Add zero_map_value to zero map value with special fields
bpf: Support kptrs in percpu hashmap and percpu LRU hashmap
bpf: Support kptrs in local storage maps
bpf: Annotate data races in bpf_local_storage
bpf: Allow specifying volatile type modifier for kptrs
bpf: Add comment about kptr's PTR_TO_MAP_VALUE handling
bpf: Rewrite kfunc argument handling
bpf: Drop kfunc support from btf_check_func_arg_match
bpf: Support constant scalar arguments for kfuncs
bpf: Teach verifier about non-size constant arguments
bpf: Introduce bpf_list_head support for BPF maps
bpf: Introduce bpf_kptr_alloc helper
bpf: Add helper macro bpf_expr_for_each_reg_in_vstate
bpf: Introduce BPF memory object model
bpf: Support bpf_list_node in local kptrs
bpf: Support bpf_spin_lock in local kptrs
bpf: Support bpf_list_head in local kptrs
bpf: Introduce bpf_kptr_free helper
bpf: Allow locking bpf_spin_lock global variables
bpf: Bump BTF_KFUNC_SET_MAX_CNT
bpf: Add single ownership BPF linked list API
bpf: Permit NULL checking pointer with non-zero fixed offset
bpf: Allow storing local kptrs in BPF maps
bpf: Wire up freeing of bpf_list_heads in maps
bpf: Add destructor for bpf_list_head in local kptr selftests/bpf: Add BTF tag macros for local kptrs, BPF linked lists selftests/bpf: Add BPF linked list API tests selftests/bpf: Add referenced local kptr tests

 Documentation/bpf/kfuncs.rst                  |   30 +
 include/linux/bpf.h                           |  177 +-
 include/linux/bpf_local_storage.h             |    2 +-
 include/linux/bpf_verifier.h                  |   77 +-
 include/linux/btf.h                           |   76 +-
 include/linux/poison.h                        |    3 +
 kernel/bpf/arraymap.c                         |   43 +-
 kernel/bpf/bpf_local_storage.c                |   53 +-
 kernel/bpf/btf.c                              |  727 +++---
 kernel/bpf/hashtab.c                          |   91 +-
 kernel/bpf/helpers.c                          |  137 +-
 kernel/bpf/map_in_map.c                       |    5 +-
 kernel/bpf/syscall.c                          |  231 +-
 kernel/bpf/verifier.c                         | 2084 ++++++++++++++---
 net/bpf/bpf_dummy_struct_ops.c                |    5 +-
 net/ipv4/bpf_tcp_ca.c                         |    5 +-
 tools/lib/bpf/libbpf.c                        |   65 +-
 .../testing/selftests/bpf/bpf_experimental.h  |  120 +
 .../selftests/bpf/prog_tests/linked_list.c    |   88 +
 .../selftests/bpf/prog_tests/map_kptr.c       |    2 +-
 .../testing/selftests/bpf/progs/linked_list.c |  347 +++
 tools/testing/selftests/bpf/progs/map_kptr.c  |   38 +
 tools/testing/selftests/bpf/verifier/calls.c  |    2 +-
 23 files changed, 3626 insertions(+), 782 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/bpf_experimental.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/linked_list.c
 create mode 100644 tools/testing/selftests/bpf/progs/linked_list.c
Signature · 1 line
2.34.1
author maintainer
#1 lore raw

RFC PATCH 1/32 bpf: Add copy_map_value_long to copy to remote percpu memory

bpf_long_memcpy is used while copying to remote percpu regions from BPF syscall and helpers, so that the copy is atomic at word size granularity.

This might not be possible when you copy from map value hosting kptrs from or to percpu maps, as the alignment or size in disjoint regions may not be multiple of word size.

Hence, to avoid complicating the copy loop, we only use bpf_long_memcpy when special fields are not present, otherwise use normal memcpy to copy the disjoint regions.

---
 include/linux/bpf.h | 52 ++++++++++++++++++++++++++++-----------------
 1 file changed, 33 insertions(+), 19 deletions(-)
include/linux/bpf.h +33 -19
index 9c1674973e03..a6a0c0025b46 100644
@@ -280,14 +280,33 @@ static inline void check_and_init_map_value(struct bpf_map *map, void *dst)
 	}
 }
 
-/* copy everything but bpf_spin_lock and bpf_timer. There could be one of each. */
-static inline void copy_map_value(struct bpf_map *map, void *dst, void *src)
+/* memcpy that is used with 8-byte aligned pointers, power-of-8 size and
+ * forced to use 'long' read/writes to try to atomically copy long counters.
+ * Best-effort only.  No barriers here, since it _will_ race with concurrent
+ * updates from BPF programs. Called from bpf syscall and mostly used with
+ * size 8 or 16 bytes, so ask compiler to inline it.
+ */
+static inline void bpf_long_memcpy(void *dst, const void *src, u32 size)
+{
+	const long *lsrc = src;
+	long *ldst = dst;
+
+	size /= sizeof(long);
+	while (size--)
+		*ldst++ = *lsrc++;
+}
+
+/* copy everything but bpf_spin_lock, bpf_timer, and kptrs. There could be one of each. */
+static inline void __copy_map_value(struct bpf_map *map, void *dst, void *src, bool long_memcpy)
 {
 	u32 curr_off = 0;
 	int i;
 
 	if (likely(!map->off_arr)) {
-		memcpy(dst, src, map->value_size);
+		if (long_memcpy)
+			bpf_long_memcpy(dst, src, round_up(map->value_size, 8));
+		else
+			memcpy(dst, src, map->value_size);
 		return;
 	}
 
@@ -299,6 +318,17 @@ static inline void copy_map_value(struct bpf_map *map, void *dst, void *src)
 	}
 	memcpy(dst + curr_off, src + curr_off, map->value_size - curr_off);
 }
+
+static inline void copy_map_value(struct bpf_map *map, void *dst, void *src)
+{
+	__copy_map_value(map, dst, src, false);
+}
+
+static inline void copy_map_value_long(struct bpf_map *map, void *dst, void *src)
+{
+	__copy_map_value(map, dst, src, true);
+}
+
 void copy_map_value_locked(struct bpf_map *map, void *dst, void *src,
 			   bool lock_src);
 void bpf_timer_cancel_and_free(void *timer);
@@ -1823,22 +1853,6 @@ int bpf_get_file_flag(int flags);
 int bpf_check_uarg_tail_zero(bpfptr_t uaddr, size_t expected_size,
 			     size_t actual_size);
 
-/* memcpy that is used with 8-byte aligned pointers, power-of-8 size and
- * forced to use 'long' read/writes to try to atomically copy long counters.
- * Best-effort only.  No barriers here, since it _will_ race with concurrent
- * updates from BPF programs. Called from bpf syscall and mostly used with
- * size 8 or 16 bytes, so ask compiler to inline it.
- */
-static inline void bpf_long_memcpy(void *dst, const void *src, u32 size)
-{
-	const long *lsrc = src;
-	long *ldst = dst;
-
-	size /= sizeof(long);
-	while (size--)
-		*ldst++ = *lsrc++;
-}
-
 /* verify correctness of eBPF program */
 int bpf_check(struct bpf_prog **fp, union bpf_attr *attr, bpfptr_t uattr);
 
Signature · 1 line
2.34.1
author maintainer
#2 lore raw

RFC PATCH 2/32 bpf: Support kptrs in percpu arraymap

In reply to Kumar Kartikeya Dwivedi

Enable support for kptrs in percpu BPF arraymap by wiring up the freeing of these kptrs from percpu map elements.

---
 kernel/bpf/arraymap.c | 33 ++++++++++++++++++++++++---------
 kernel/bpf/syscall.c  |  3 ++-
 2 files changed, 26 insertions(+), 10 deletions(-)
kernel/bpf/arraymap.c +24 -9
index 624527401d4d..832b2659e96e 100644
@@ -279,7 +279,8 @@ int bpf_percpu_array_copy(struct bpf_map *map, void *key, void *value)
 	rcu_read_lock();
 	pptr = array->pptrs[index & array->index_mask];
 	for_each_possible_cpu(cpu) {
-		bpf_long_memcpy(value + off, per_cpu_ptr(pptr, cpu), size);
+		copy_map_value_long(map, value + off, per_cpu_ptr(pptr, cpu));
+		check_and_init_map_value(map, value + off);
 		off += size;
 	}
 	rcu_read_unlock();
@@ -338,8 +339,9 @@ static int array_map_update_elem(struct bpf_map *map, void *key, void *value,
 		return -EINVAL;
 
 	if (array->map.map_type == BPF_MAP_TYPE_PERCPU_ARRAY) {
-		memcpy(this_cpu_ptr(array->pptrs[index & array->index_mask]),
-		       value, map->value_size);
+		val = this_cpu_ptr(array->pptrs[index & array->index_mask]);
+		copy_map_value(map, val, value);
+		check_and_free_fields(array, val);
 	} else {
 		val = array->value +
 			(u64)array->elem_size * (index & array->index_mask);
@@ -383,7 +385,8 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value,
 	rcu_read_lock();
 	pptr = array->pptrs[index & array->index_mask];
 	for_each_possible_cpu(cpu) {
-		bpf_long_memcpy(per_cpu_ptr(pptr, cpu), value + off, size);
+		copy_map_value_long(map, per_cpu_ptr(pptr, cpu), value + off);
+		check_and_free_fields(array, per_cpu_ptr(pptr, cpu));
 		off += size;
 	}
 	rcu_read_unlock();
@@ -421,8 +424,20 @@ static void array_map_free(struct bpf_map *map)
 	int i;
 
 	if (map_value_has_kptrs(map)) {
-		for (i = 0; i < array->map.max_entries; i++)
-			bpf_map_free_kptrs(map, array_map_elem_ptr(array, i));
+		if (array->map.map_type == BPF_MAP_TYPE_PERCPU_ARRAY) {
+			for (i = 0; i < array->map.max_entries; i++) {
+				void __percpu *pptr = array->pptrs[i & array->index_mask];
+				int cpu;
+
+				for_each_possible_cpu(cpu) {
+					bpf_map_free_kptrs(map, per_cpu_ptr(pptr, cpu));
+					cond_resched();
+				}
+			}
+		} else {
+			for (i = 0; i < array->map.max_entries; i++)
+				bpf_map_free_kptrs(map, array_map_elem_ptr(array, i));
+		}
 		bpf_map_free_kptr_off_tab(map);
 	}
 
@@ -608,9 +623,9 @@ static int __bpf_array_map_seq_show(struct seq_file *seq, void *v)
 			pptr = v;
 			size = array->elem_size;
 			for_each_possible_cpu(cpu) {
-				bpf_long_memcpy(info->percpu_value_buf + off,
-						per_cpu_ptr(pptr, cpu),
-						size);
+				copy_map_value_long(map, info->percpu_value_buf + off,
+						    per_cpu_ptr(pptr, cpu));
+				check_and_init_map_value(map, info->percpu_value_buf + off);
 				off += size;
 			}
 			ctx.value = info->percpu_value_buf;
kernel/bpf/syscall.c +2 -1
index 4e9d4622aef7..723699263a62 100644
@@ -1046,7 +1046,8 @@ static int map_check_btf(struct bpf_map *map, const struct btf *btf,
 		}
 		if (map->map_type != BPF_MAP_TYPE_HASH &&
 		    map->map_type != BPF_MAP_TYPE_LRU_HASH &&
-		    map->map_type != BPF_MAP_TYPE_ARRAY) {
+		    map->map_type != BPF_MAP_TYPE_ARRAY &&
+		    map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) {
 			ret = -EOPNOTSUPP;
 			goto free_map_tab;
 		}
Signature · 1 line
2.34.1
author maintainer
#3 lore raw

RFC PATCH 3/32 bpf: Add zero_map_value to zero map value with special fields

In reply to Kumar Kartikeya Dwivedi

We need this helper to skip over special fields (bpf_spin_lock, bpf_timer, kptrs) while zeroing a map value. Use the same logic as copy_map_value but memset instead of memcpy.

Currently, the code zeroing map value memory does not have to deal with special fields, hence this is a prerequisite for introducing such support.

---
 include/linux/bpf.h | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
include/linux/bpf.h +19 -0
index a6a0c0025b46..cdc0a8c1b1d1 100644
@@ -329,6 +329,25 @@ static inline void copy_map_value_long(struct bpf_map *map, void *dst, void *src
 	__copy_map_value(map, dst, src, true);
 }
 
+static inline void zero_map_value(struct bpf_map *map, void *dst)
+{
+	u32 curr_off = 0;
+	int i;
+
+	if (likely(!map->off_arr)) {
+		memset(dst, 0, map->value_size);
+		return;
+	}
+
+	for (i = 0; i < map->off_arr->cnt; i++) {
+		u32 next_off = map->off_arr->field_off[i];
+
+		memset(dst + curr_off, 0, next_off - curr_off);
+		curr_off += map->off_arr->field_sz[i];
+	}
+	memset(dst + curr_off, 0, map->value_size - curr_off);
+}
+
 void copy_map_value_locked(struct bpf_map *map, void *dst, void *src,
 			   bool lock_src);
 void bpf_timer_cancel_and_free(void *timer);
Signature · 1 line
2.34.1
author maintainer
#4 lore raw

RFC PATCH 4/32 bpf: Support kptrs in percpu hashmap and percpu LRU hashmap

In reply to Kumar Kartikeya Dwivedi

Enable support for kptrs in percpu BPF hashmap and percpu BPF LRU hashmap by wiring up the freeing of these kptrs from percpu map elements.

---
 kernel/bpf/hashtab.c | 70 ++++++++++++++++++++++++++++----------------
 kernel/bpf/syscall.c |  2 ++
 2 files changed, 46 insertions(+), 26 deletions(-)
kernel/bpf/hashtab.c +44 -26
index eb1263f03e9b..bb3f8a63c221 100644
@@ -282,8 +282,18 @@ static void htab_free_prealloced_kptrs(struct bpf_htab *htab)
 		struct htab_elem *elem;
 
 		elem = get_htab_elem(htab, i);
-		bpf_map_free_kptrs(&htab->map, elem->key + round_up(htab->map.key_size, 8));
-		cond_resched();
+		if (htab_is_percpu(htab)) {
+			void __percpu *pptr = htab_elem_get_ptr(elem, htab->map.key_size);
+			int cpu;
+
+			for_each_possible_cpu(cpu) {
+				bpf_map_free_kptrs(&htab->map, per_cpu_ptr(pptr, cpu));
+				cond_resched();
+			}
+		} else {
+			bpf_map_free_kptrs(&htab->map, elem->key + round_up(htab->map.key_size, 8));
+			cond_resched();
+		}
 	}
 }
 
@@ -761,8 +771,17 @@ static void check_and_free_fields(struct bpf_htab *htab,
 
 	if (map_value_has_timer(&htab->map))
 		bpf_timer_cancel_and_free(map_value + htab->map.timer_off);
-	if (map_value_has_kptrs(&htab->map))
-		bpf_map_free_kptrs(&htab->map, map_value);
+	if (map_value_has_kptrs(&htab->map)) {
+		if (htab_is_percpu(htab)) {
+			void __percpu *pptr = htab_elem_get_ptr(elem, htab->map.key_size);
+			int cpu;
+
+			for_each_possible_cpu(cpu)
+				bpf_map_free_kptrs(&htab->map, per_cpu_ptr(pptr, cpu));
+		} else {
+			bpf_map_free_kptrs(&htab->map, map_value);
+		}
+	}
 }
 
 /* It is called from the bpf_lru_list when the LRU needs to delete
@@ -859,9 +878,9 @@ static int htab_map_get_next_key(struct bpf_map *map, void *key, void *next_key)
 
 static void htab_elem_free(struct bpf_htab *htab, struct htab_elem *l)
 {
+	check_and_free_fields(htab, l);
 	if (htab->map.map_type == BPF_MAP_TYPE_PERCPU_HASH)
 		free_percpu(htab_elem_get_ptr(l, htab->map.key_size));
-	check_and_free_fields(htab, l);
 	kfree(l);
 }
 
@@ -903,14 +922,13 @@ static void pcpu_copy_value(struct bpf_htab *htab, void __percpu *pptr,
 {
 	if (!onallcpus) {
 		/* copy true value_size bytes */
-		memcpy(this_cpu_ptr(pptr), value, htab->map.value_size);
+		copy_map_value(&htab->map, this_cpu_ptr(pptr), value);
 	} else {
 		u32 size = round_up(htab->map.value_size, 8);
 		int off = 0, cpu;
 
 		for_each_possible_cpu(cpu) {
-			bpf_long_memcpy(per_cpu_ptr(pptr, cpu),
-					value + off, size);
+			copy_map_value_long(&htab->map, per_cpu_ptr(pptr, cpu), value + off);
 			off += size;
 		}
 	}
@@ -926,16 +944,16 @@ static void pcpu_init_value(struct bpf_htab *htab, void __percpu *pptr,
 	 * (onallcpus=false always when coming from bpf prog).
 	 */
 	if (htab_is_prealloc(htab) && !onallcpus) {
-		u32 size = round_up(htab->map.value_size, 8);
 		int current_cpu = raw_smp_processor_id();
 		int cpu;
 
 		for_each_possible_cpu(cpu) {
-			if (cpu == current_cpu)
-				bpf_long_memcpy(per_cpu_ptr(pptr, cpu), value,
-						size);
-			else
-				memset(per_cpu_ptr(pptr, cpu), 0, size);
+			if (cpu == current_cpu) {
+				copy_map_value_long(&htab->map, per_cpu_ptr(pptr, cpu), value);
+			} else {
+				/* Since elem is preallocated, we cannot touch special fields */
+				zero_map_value(&htab->map, per_cpu_ptr(pptr, cpu));
+			}
 		}
 	} else {
 		pcpu_copy_value(htab, pptr, value, onallcpus);
@@ -993,8 +1011,9 @@ static struct htab_elem *alloc_htab_elem(struct bpf_htab *htab, void *key,
 			l_new = ERR_PTR(-ENOMEM);
 			goto dec_count;
 		}
-		check_and_init_map_value(&htab->map,
-					 l_new->key + round_up(key_size, 8));
+
+		if (!percpu)
+			check_and_init_map_value(&htab->map, l_new->key + round_up(key_size, 8));
 	}
 
 	memcpy(l_new->key, key, key_size);
@@ -1562,9 +1581,8 @@ static int __htab_map_lookup_and_delete_elem(struct bpf_map *map, void *key,
 
 			pptr = htab_elem_get_ptr(l, key_size);
 			for_each_possible_cpu(cpu) {
-				bpf_long_memcpy(value + off,
-						per_cpu_ptr(pptr, cpu),
-						roundup_value_size);
+				copy_map_value_long(&htab->map, value + off, per_cpu_ptr(pptr, cpu));
+				check_and_init_map_value(&htab->map, value + off);
 				off += roundup_value_size;
 			}
 		} else {
@@ -1758,8 +1776,8 @@ __htab_map_lookup_and_delete_batch(struct bpf_map *map,
 
 			pptr = htab_elem_get_ptr(l, map->key_size);
 			for_each_possible_cpu(cpu) {
-				bpf_long_memcpy(dst_val + off,
-						per_cpu_ptr(pptr, cpu), size);
+				copy_map_value_long(&htab->map, dst_val + off, per_cpu_ptr(pptr, cpu));
+				check_and_init_map_value(&htab->map, dst_val + off);
 				off += size;
 			}
 		} else {
@@ -2031,9 +2049,9 @@ static int __bpf_hash_map_seq_show(struct seq_file *seq, struct htab_elem *elem)
 				roundup_value_size = round_up(map->value_size, 8);
 				pptr = htab_elem_get_ptr(elem, map->key_size);
 				for_each_possible_cpu(cpu) {
-					bpf_long_memcpy(info->percpu_value_buf + off,
-							per_cpu_ptr(pptr, cpu),
-							roundup_value_size);
+					copy_map_value_long(map, info->percpu_value_buf + off,
+							    per_cpu_ptr(pptr, cpu));
+					check_and_init_map_value(map, info->percpu_value_buf + off);
 					off += roundup_value_size;
 				}
 				ctx.value = info->percpu_value_buf;
@@ -2277,8 +2295,8 @@ int bpf_percpu_hash_copy(struct bpf_map *map, void *key, void *value)
 	 */
 	pptr = htab_elem_get_ptr(l, map->key_size);
 	for_each_possible_cpu(cpu) {
-		bpf_long_memcpy(value + off,
-				per_cpu_ptr(pptr, cpu), size);
+		copy_map_value_long(map, value + off, per_cpu_ptr(pptr, cpu));
+		check_and_init_map_value(map, value + off);
 		off += size;
 	}
 	ret = 0;
kernel/bpf/syscall.c +2 -0
index 723699263a62..3214bab5b462 100644
@@ -1045,7 +1045,9 @@ static int map_check_btf(struct bpf_map *map, const struct btf *btf,
 			goto free_map_tab;
 		}
 		if (map->map_type != BPF_MAP_TYPE_HASH &&
+		    map->map_type != BPF_MAP_TYPE_PERCPU_HASH &&
 		    map->map_type != BPF_MAP_TYPE_LRU_HASH &&
+		    map->map_type != BPF_MAP_TYPE_LRU_PERCPU_HASH &&
 		    map->map_type != BPF_MAP_TYPE_ARRAY &&
 		    map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) {
 			ret = -EOPNOTSUPP;
Signature · 1 line
2.34.1
author maintainer
#5 lore raw

RFC PATCH 5/32 bpf: Support kptrs in local storage maps

In reply to Kumar Kartikeya Dwivedi

Enable support for kptrs in local storage maps by wiring up the freeing of these kptrs from map value.

---
 include/linux/bpf_local_storage.h |  2 +-
 kernel/bpf/bpf_local_storage.c    | 33 +++++++++++++++++++++++++++----
 kernel/bpf/syscall.c              |  5 ++++-
 kernel/bpf/verifier.c             |  9 ++++++---
 4 files changed, 40 insertions(+), 9 deletions(-)
include/linux/bpf_local_storage.h +1 -1
index 7ea18d4da84b..6786d00f004e 100644
@@ -74,7 +74,7 @@ struct bpf_local_storage_elem {
 	struct hlist_node snode;	/* Linked to bpf_local_storage */
 	struct bpf_local_storage __rcu *local_storage;
 	struct rcu_head rcu;
-	/* 8 bytes hole */
+	struct bpf_map *map;		/* Only set for bpf_selem_free_rcu */
 	/* The data is stored in another cacheline to minimize
 	 * the number of cachelines access during a cache hit.
 	 */
kernel/bpf/bpf_local_storage.c +29 -4
index 802fc15b0d73..4a725379d761 100644
@@ -74,7 +74,8 @@ bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner,
 				gfp_flags | __GFP_NOWARN);
 	if (selem) {
 		if (value)
-			memcpy(SDATA(selem)->data, value, smap->map.value_size);
+			copy_map_value(&smap->map, SDATA(selem)->data, value);
+		/* No call to check_and_init_map_value as memory is zero init */
 		return selem;
 	}
 
@@ -92,12 +93,27 @@ void bpf_local_storage_free_rcu(struct rcu_head *rcu)
 	kfree_rcu(local_storage, rcu);
 }
 
+static void check_and_free_fields(struct bpf_local_storage_elem *selem)
+{
+	if (map_value_has_kptrs(selem->map))
+		bpf_map_free_kptrs(selem->map, SDATA(selem));
+}
+
 static void bpf_selem_free_rcu(struct rcu_head *rcu)
 {
 	struct bpf_local_storage_elem *selem;
 
 	selem = container_of(rcu, struct bpf_local_storage_elem, rcu);
-	kfree_rcu(selem, rcu);
+	check_and_free_fields(selem);
+	kfree(selem);
+}
+
+static void bpf_selem_free_tasks_trace_rcu(struct rcu_head *rcu)
+{
+	struct bpf_local_storage_elem *selem;
+
+	selem = container_of(rcu, struct bpf_local_storage_elem, rcu);
+	call_rcu(&selem->rcu, bpf_selem_free_rcu);
 }
 
 /* local_storage->lock must be held and selem->local_storage == local_storage.
@@ -150,10 +166,11 @@ bool bpf_selem_unlink_storage_nolock(struct bpf_local_storage *local_storage,
 	    SDATA(selem))
 		RCU_INIT_POINTER(local_storage->cache[smap->cache_idx], NULL);
 
+	selem->map = &smap->map;
 	if (use_trace_rcu)
-		call_rcu_tasks_trace(&selem->rcu, bpf_selem_free_rcu);
+		call_rcu_tasks_trace(&selem->rcu, bpf_selem_free_tasks_trace_rcu);
 	else
-		kfree_rcu(selem, rcu);
+		call_rcu(&selem->rcu, bpf_selem_free_rcu);
 
 	return free_local_storage;
 }
@@ -581,6 +598,14 @@ void bpf_local_storage_map_free(struct bpf_local_storage_map *smap,
 	 */
 	synchronize_rcu();
 
+	/* When local storage map has kptrs, the call_rcu callback accesses
+	 * kptr_off_tab, hence we need the bpf_selem_free_rcu callbacks to
+	 * finish before we free it.
+	 */
+	if (map_value_has_kptrs(&smap->map)) {
+		rcu_barrier();
+		bpf_map_free_kptr_off_tab(&smap->map);
+	}
 	kvfree(smap->buckets);
 	bpf_map_area_free(smap);
 }
kernel/bpf/syscall.c +4 -1
index 3214bab5b462..0311acca19f6 100644
@@ -1049,7 +1049,10 @@ static int map_check_btf(struct bpf_map *map, const struct btf *btf,
 		    map->map_type != BPF_MAP_TYPE_LRU_HASH &&
 		    map->map_type != BPF_MAP_TYPE_LRU_PERCPU_HASH &&
 		    map->map_type != BPF_MAP_TYPE_ARRAY &&
-		    map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) {
+		    map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY &&
+		    map->map_type != BPF_MAP_TYPE_SK_STORAGE &&
+		    map->map_type != BPF_MAP_TYPE_INODE_STORAGE &&
+		    map->map_type != BPF_MAP_TYPE_TASK_STORAGE) {
 			ret = -EOPNOTSUPP;
 			goto free_map_tab;
 		}
kernel/bpf/verifier.c +6 -3
index 0194a36d0b36..b7bf68f3b2ec 100644
@@ -6276,17 +6276,20 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env,
 		break;
 	case BPF_MAP_TYPE_SK_STORAGE:
 		if (func_id != BPF_FUNC_sk_storage_get &&
-		    func_id != BPF_FUNC_sk_storage_delete)
+		    func_id != BPF_FUNC_sk_storage_delete &&
+		    func_id != BPF_FUNC_kptr_xchg)
 			goto error;
 		break;
 	case BPF_MAP_TYPE_INODE_STORAGE:
 		if (func_id != BPF_FUNC_inode_storage_get &&
-		    func_id != BPF_FUNC_inode_storage_delete)
+		    func_id != BPF_FUNC_inode_storage_delete &&
+		    func_id != BPF_FUNC_kptr_xchg)
 			goto error;
 		break;
 	case BPF_MAP_TYPE_TASK_STORAGE:
 		if (func_id != BPF_FUNC_task_storage_get &&
-		    func_id != BPF_FUNC_task_storage_delete)
+		    func_id != BPF_FUNC_task_storage_delete &&
+		    func_id != BPF_FUNC_kptr_xchg)
 			goto error;
 		break;
 	case BPF_MAP_TYPE_BLOOM_FILTER:
Signature · 1 line
2.34.1
#6 lore raw

Re: [PATCH RFC bpf-next v1 05/32] bpf: Support kptrs in local storage maps

Quoted 84 lines · Kumar Kartikeya Dwivedi

Enable support for kptrs in local storage maps by wiring up the freeing of these kptrs from map value.

---
 include/linux/bpf_local_storage.h |  2 +-
 kernel/bpf/bpf_local_storage.c    | 33 +++++++++++++++++++++++++++----
 kernel/bpf/syscall.c              |  5 ++++-
 kernel/bpf/verifier.c             |  9 ++++++---
 4 files changed, 40 insertions(+), 9 deletions(-)
include/linux/bpf_local_storage.h +1 -1
index 7ea18d4da84b..6786d00f004e 100644
@@ -74,7 +74,7 @@ struct bpf_local_storage_elem {
 	struct hlist_node snode;	/* Linked to bpf_local_storage */
 	struct bpf_local_storage __rcu *local_storage;
 	struct rcu_head rcu;
-	/* 8 bytes hole */
+	struct bpf_map *map;		/* Only set for bpf_selem_free_rcu */
 	/* The data is stored in another cacheline to minimize
 	 * the number of cachelines access during a cache hit.
 	 */
kernel/bpf/bpf_local_storage.c +21 -4
index 802fc15b0d73..4a725379d761 100644
@@ -74,7 +74,8 @@ bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner,
 				gfp_flags | __GFP_NOWARN);
 	if (selem) {
 		if (value)
-			memcpy(SDATA(selem)->data, value, smap->map.value_size);
+			copy_map_value(&smap->map, SDATA(selem)->data, value);
+		/* No call to check_and_init_map_value as memory is zero init */
 		return selem;
 	}
 
@@ -92,12 +93,27 @@ void bpf_local_storage_free_rcu(struct rcu_head *rcu)
 	kfree_rcu(local_storage, rcu);
 }
 
+static void check_and_free_fields(struct bpf_local_storage_elem *selem)
+{
+	if (map_value_has_kptrs(selem->map))
+		bpf_map_free_kptrs(selem->map, SDATA(selem));
+}
+
 static void bpf_selem_free_rcu(struct rcu_head *rcu)
 {
 	struct bpf_local_storage_elem *selem;
 
 	selem = container_of(rcu, struct bpf_local_storage_elem, rcu);
-	kfree_rcu(selem, rcu);
+	check_and_free_fields(selem);
+	kfree(selem);
+}
+
+static void bpf_selem_free_tasks_trace_rcu(struct rcu_head *rcu)
+{
+	struct bpf_local_storage_elem *selem;
+
+	selem = container_of(rcu, struct bpf_local_storage_elem, rcu);
+	call_rcu(&selem->rcu, bpf_selem_free_rcu);
 }
 
 /* local_storage->lock must be held and selem->local_storage == local_storage.
@@ -150,10 +166,11 @@ bool bpf_selem_unlink_storage_nolock(struct bpf_local_storage *local_storage,
 	    SDATA(selem))
 		RCU_INIT_POINTER(local_storage->cache[smap->cache_idx], NULL);
 
+	selem->map = &smap->map;
 	if (use_trace_rcu)
-		call_rcu_tasks_trace(&selem->rcu, bpf_selem_free_rcu);
+		call_rcu_tasks_trace(&selem->rcu, bpf_selem_free_tasks_trace_rcu);
 	else
-		kfree_rcu(selem, rcu);
+		call_rcu(&selem->rcu, bpf_selem_free_rcu);
 
 	return free_local_storage;
 }

kernel/bpf/bpf_local_storage.c void bpf_local_storage_map_free(struct bpf_local_storage_map *smap, · line 607

Quoted 10 lines
kernel/bpf/bpf_local_storage.c +7 -0
@@ -581,6 +598,14 @@ void bpf_local_storage_map_free(struct bpf_local_storage_map *smap,
 	 */
 	synchronize_rcu();
 
+	/* When local storage map has kptrs, the call_rcu callback accesses
+	 * kptr_off_tab, hence we need the bpf_selem_free_rcu callbacks to
+	 * finish before we free it.
+	 */
+	if (map_value_has_kptrs(&smap->map)) {
+		rcu_barrier();
+		bpf_map_free_kptr_off_tab(&smap->map);

probably needs conditional rcu_barrier_tasks_trace before rcu_barrier? With or without it will be a significant delay in map freeing. Maybe we should generalize the destroy_mem_alloc trick?

Patch 4 needs rebase. Applied patches 1-3.
The first 5 look great to me.
Pls follow up with kptr specific tests.

author maintainer
#7 lore raw

Re: [PATCH RFC bpf-next v1 05/32] bpf: Support kptrs in local storage maps

Quoted 89 lines · Alexei Starovoitov
Quoted 87 lines · Kumar Kartikeya Dwivedi

Enable support for kptrs in local storage maps by wiring up the freeing of these kptrs from map value.

---
 include/linux/bpf_local_storage.h |  2 +-
 kernel/bpf/bpf_local_storage.c    | 33 +++++++++++++++++++++++++++----
 kernel/bpf/syscall.c              |  5 ++++-
 kernel/bpf/verifier.c             |  9 ++++++---
 4 files changed, 40 insertions(+), 9 deletions(-)
include/linux/bpf_local_storage.h +1 -1
index 7ea18d4da84b..6786d00f004e 100644
@@ -74,7 +74,7 @@ struct bpf_local_storage_elem {
      struct hlist_node snode;        /* Linked to bpf_local_storage */
      struct bpf_local_storage __rcu *local_storage;
      struct rcu_head rcu;
-     /* 8 bytes hole */
+     struct bpf_map *map;            /* Only set for bpf_selem_free_rcu */
      /* The data is stored in another cacheline to minimize
       * the number of cachelines access during a cache hit.
       */
kernel/bpf/bpf_local_storage.c +21 -4
index 802fc15b0d73..4a725379d761 100644
@@ -74,7 +74,8 @@ bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner,
                              gfp_flags | __GFP_NOWARN);
      if (selem) {
              if (value)
-                     memcpy(SDATA(selem)->data, value, smap->map.value_size);
+                     copy_map_value(&smap->map, SDATA(selem)->data, value);
+             /* No call to check_and_init_map_value as memory is zero init */
              return selem;
      }
 
@@ -92,12 +93,27 @@ void bpf_local_storage_free_rcu(struct rcu_head *rcu)
      kfree_rcu(local_storage, rcu);
 }
 
+static void check_and_free_fields(struct bpf_local_storage_elem *selem)
+{
+     if (map_value_has_kptrs(selem->map))
+             bpf_map_free_kptrs(selem->map, SDATA(selem));
+}
+
 static void bpf_selem_free_rcu(struct rcu_head *rcu)
 {
      struct bpf_local_storage_elem *selem;
 
      selem = container_of(rcu, struct bpf_local_storage_elem, rcu);
-     kfree_rcu(selem, rcu);
+     check_and_free_fields(selem);
+     kfree(selem);
+}
+
+static void bpf_selem_free_tasks_trace_rcu(struct rcu_head *rcu)
+{
+     struct bpf_local_storage_elem *selem;
+
+     selem = container_of(rcu, struct bpf_local_storage_elem, rcu);
+     call_rcu(&selem->rcu, bpf_selem_free_rcu);
 }
 
 /* local_storage->lock must be held and selem->local_storage == local_storage.
@@ -150,10 +166,11 @@ bool bpf_selem_unlink_storage_nolock(struct bpf_local_storage *local_storage,
          SDATA(selem))
              RCU_INIT_POINTER(local_storage->cache[smap->cache_idx], NULL);
 
+     selem->map = &smap->map;
      if (use_trace_rcu)
-             call_rcu_tasks_trace(&selem->rcu, bpf_selem_free_rcu);
+             call_rcu_tasks_trace(&selem->rcu, bpf_selem_free_tasks_trace_rcu);
      else
-             kfree_rcu(selem, rcu);
+             call_rcu(&selem->rcu, bpf_selem_free_rcu);
 
      return free_local_storage;
 }
@@ -581,6 +598,14 @@ void bpf_local_storage_map_free(struct bpf_local_storage_map *smap,
       */
      synchronize_rcu();
 
Quoted 12 lines

kernel/bpf/bpf_local_storage.c void bpf_local_storage_map_free(struct bpf_local_storage_map *smap, · line 607

Quoted 7 lines
kernel/bpf/bpf_local_storage.c +7 -0
+     /* When local storage map has kptrs, the call_rcu callback accesses
+      * kptr_off_tab, hence we need the bpf_selem_free_rcu callbacks to
+      * finish before we free it.
+      */
+     if (map_value_has_kptrs(&smap->map)) {
+             rcu_barrier();
+             bpf_map_free_kptr_off_tab(&smap->map);

probably needs conditional rcu_barrier_tasks_trace before rcu_barrier? With or without it will be a significant delay in map freeing. Maybe we should generalize the destroy_mem_alloc trick?

Yes, let me take a closer look tomorrow and ask questions if any. Otherwise I will rework it. Thanks for catching this.

Quoted 3 lines

Patch 4 needs rebase. Applied patches 1-3.
The first 5 look great to me.
Pls follow up with kptr specific tests.

Thanks, I will split those out into another series with its own test.

#8 lore raw

Re: [PATCH RFC bpf-next v1 05/32] bpf: Support kptrs in local storage maps

In reply to Kumar Kartikeya Dwivedi

include/linux/bpf_local_storage.h struct bpf_local_storage_elem { · line 77

Quoted 10 lines · Kumar Kartikeya Dwivedi
include/linux/bpf_local_storage.h +1 -1
index 7ea18d4da84b..6786d00f004e 100644
@@ -74,7 +74,7 @@ struct bpf_local_storage_elem {
  	struct hlist_node snode;	/* Linked to bpf_local_storage */
  	struct bpf_local_storage __rcu *local_storage;
  	struct rcu_head rcu;
-	/* 8 bytes hole */
+	struct bpf_map *map;		/* Only set for bpf_selem_free_rcu */

Instead of adding another map ptr and using the last 8 bytes hole,

Quoted 18 lines
include/linux/bpf_local_storage.h +0 -0
  	/* The data is stored in another cacheline to minimize
  	 * the number of cachelines access during a cache hit.
  	 */
kernel/bpf/bpf_local_storage.c +2 -1
index 802fc15b0d73..4a725379d761 100644
@@ -74,7 +74,8 @@ bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner,
  				gfp_flags | __GFP_NOWARN);
  	if (selem) {
  		if (value)
-			memcpy(SDATA(selem)->data, value, smap->map.value_size);
+			copy_map_value(&smap->map, SDATA(selem)->data, value);
+		/* No call to check_and_init_map_value as memory is zero init */
  		return selem;
  	}
 

kernel/bpf/bpf_local_storage.c void bpf_local_storage_free_rcu(struct rcu_head *rcu) · line 98

Quoted 6 lines
kernel/bpf/bpf_local_storage.c +3 -0
@@ -92,12 +93,27 @@ void bpf_local_storage_free_rcu(struct rcu_head *rcu)
  	kfree_rcu(local_storage, rcu);
  }
 
+static void check_and_free_fields(struct bpf_local_storage_elem *selem)
+{
+	if (map_value_has_kptrs(selem->map))

could SDATA(selem)->smap->map be used here ?

Quoted 34 lines
kernel/bpf/bpf_local_storage.c +16 -3
+		bpf_map_free_kptrs(selem->map, SDATA(selem));
+}
+
  static void bpf_selem_free_rcu(struct rcu_head *rcu)
  {
  	struct bpf_local_storage_elem *selem;
 
  	selem = container_of(rcu, struct bpf_local_storage_elem, rcu);
-	kfree_rcu(selem, rcu);
+	check_and_free_fields(selem);
+	kfree(selem);
+}
+
+static void bpf_selem_free_tasks_trace_rcu(struct rcu_head *rcu)
+{
+	struct bpf_local_storage_elem *selem;
+
+	selem = container_of(rcu, struct bpf_local_storage_elem, rcu);
+	call_rcu(&selem->rcu, bpf_selem_free_rcu);
  }
 
  /* local_storage->lock must be held and selem->local_storage == local_storage.
@@ -150,10 +166,11 @@ bool bpf_selem_unlink_storage_nolock(struct bpf_local_storage *local_storage,
  	    SDATA(selem))
  		RCU_INIT_POINTER(local_storage->cache[smap->cache_idx], NULL);
 
+	selem->map = &smap->map;
  	if (use_trace_rcu)
-		call_rcu_tasks_trace(&selem->rcu, bpf_selem_free_rcu);
+		call_rcu_tasks_trace(&selem->rcu, bpf_selem_free_tasks_trace_rcu);
  	else
-		kfree_rcu(selem, rcu);
+		call_rcu(&selem->rcu, bpf_selem_free_rcu);
 
author maintainer
#9 lore raw

Re: [PATCH RFC bpf-next v1 05/32] bpf: Support kptrs in local storage maps

Quoted 36 lines · Martin KaFai Lau

include/linux/bpf_local_storage.h struct bpf_local_storage_elem { · line 77

Quoted 10 lines · Kumar Kartikeya Dwivedi
include/linux/bpf_local_storage.h +1 -1
index 7ea18d4da84b..6786d00f004e 100644
@@ -74,7 +74,7 @@ struct bpf_local_storage_elem {
      struct hlist_node snode;        /* Linked to bpf_local_storage */
      struct bpf_local_storage __rcu *local_storage;
      struct rcu_head rcu;
-     /* 8 bytes hole */
+     struct bpf_map *map;            /* Only set for bpf_selem_free_rcu */

Instead of adding another map ptr and using the last 8 bytes hole,

Quoted 21 lines
include/linux/bpf_local_storage.h +0 -0
      /* The data is stored in another cacheline to minimize
       * the number of cachelines access during a cache hit.
       */
kernel/bpf/bpf_local_storage.c +2 -1
index 802fc15b0d73..4a725379d761 100644
@@ -74,7 +74,8 @@ bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner,
                              gfp_flags | __GFP_NOWARN);
      if (selem) {
              if (value)
-                     memcpy(SDATA(selem)->data, value, smap->map.value_size);
+                     copy_map_value(&smap->map, SDATA(selem)->data, value);
+             /* No call to check_and_init_map_value as memory is zero init */
              return selem;
      }
 
@@ -92,12 +93,27 @@ void bpf_local_storage_free_rcu(struct rcu_head *rcu)
      kfree_rcu(local_storage, rcu);
  }
 
Quoted 6 lines

kernel/bpf/bpf_local_storage.c void bpf_local_storage_free_rcu(struct rcu_head *rcu) · line 98

Quoted 3 lines
kernel/bpf/bpf_local_storage.c +3 -0
+static void check_and_free_fields(struct bpf_local_storage_elem *selem)
+{
+     if (map_value_has_kptrs(selem->map))

could SDATA(selem)->smap->map be used here ?

Yeah, that should work. Thanks Martin.

Quoted 4 lines
Quoted 3 lines
kernel/bpf/bpf_local_storage.c +3 -0
+             bpf_map_free_kptrs(selem->map, SDATA(selem));
+}
+

[...]

author maintainer
#10 lore raw

RFC PATCH 6/32 bpf: Annotate data races in bpf_local_storage

In reply to Kumar Kartikeya Dwivedi

There are a few cases where hlist_node is checked to be unhashed without holding the lock protecting its modification. In this case, one must use hlist_unhashed_lockless to avoid load tearing and KCSAN reports. Fix this by using lockless variant in places not protected by the lock.

Since this is not prompted by any actual KCSAN reports but only from code review, I have not included a fixes tag.

---
 kernel/bpf/bpf_local_storage.c | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)
kernel/bpf/bpf_local_storage.c +13 -3
index 4a725379d761..58cb0c179097 100644
@@ -51,11 +51,21 @@ owner_storage(struct bpf_local_storage_map *smap, void *owner)
 	return map->ops->map_owner_storage_ptr(owner);
 }
 
+static bool selem_linked_to_storage_lockless(const struct bpf_local_storage_elem *selem)
+{
+	return !hlist_unhashed_lockless(&selem->snode);
+}
+
 static bool selem_linked_to_storage(const struct bpf_local_storage_elem *selem)
 {
 	return !hlist_unhashed(&selem->snode);
 }
 
+static bool selem_linked_to_map_lockless(const struct bpf_local_storage_elem *selem)
+{
+	return !hlist_unhashed_lockless(&selem->map_node);
+}
+
 static bool selem_linked_to_map(const struct bpf_local_storage_elem *selem)
 {
 	return !hlist_unhashed(&selem->map_node);
@@ -182,7 +192,7 @@ static void __bpf_selem_unlink_storage(struct bpf_local_storage_elem *selem,
 	bool free_local_storage = false;
 	unsigned long flags;
 
-	if (unlikely(!selem_linked_to_storage(selem)))
+	if (unlikely(!selem_linked_to_storage_lockless(selem)))
 		/* selem has already been unlinked from sk */
 		return;
 
@@ -216,7 +226,7 @@ void bpf_selem_unlink_map(struct bpf_local_storage_elem *selem)
 	struct bpf_local_storage_map_bucket *b;
 	unsigned long flags;
 
-	if (unlikely(!selem_linked_to_map(selem)))
+	if (unlikely(!selem_linked_to_map_lockless(selem)))
 		/* selem has already be unlinked from smap */
 		return;
 
@@ -427,7 +437,7 @@ bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap,
 		err = check_flags(old_sdata, map_flags);
 		if (err)
 			return ERR_PTR(err);
-		if (old_sdata && selem_linked_to_storage(SELEM(old_sdata))) {
+		if (old_sdata && selem_linked_to_storage_lockless(SELEM(old_sdata))) {
 			copy_map_value_locked(&smap->map, old_sdata->data,
 					      value, false);
 			return old_sdata;
Signature · 1 line
2.34.1
author maintainer
#11 lore raw

RFC PATCH 7/32 bpf: Allow specifying volatile type modifier for kptrs

In reply to Kumar Kartikeya Dwivedi

This is useful in particular to mark the pointer as volatile, so that compiler treats each load and store to the field as a volatile access. The alternative is having to define and use READ_ONCE and WRITE_ONCE in the BPF program.

---
 include/linux/btf.h | 5 +++++
 kernel/bpf/btf.c    | 3 +++
 2 files changed, 8 insertions(+)
include/linux/btf.h +5 -0
index ad93c2d9cc1c..b3d47e9b9d5c 100644
@@ -279,6 +279,11 @@ static inline bool btf_type_is_typedef(const struct btf_type *t)
 	return BTF_INFO_KIND(t->info) == BTF_KIND_TYPEDEF;
 }
 
+static inline bool btf_type_is_volatile(const struct btf_type *t)
+{
+	return BTF_INFO_KIND(t->info) == BTF_KIND_VOLATILE;
+}
+
 static inline bool btf_type_is_func(const struct btf_type *t)
 {
 	return BTF_INFO_KIND(t->info) == BTF_KIND_FUNC;
kernel/bpf/btf.c +3 -0
index 903719b89238..5e860f76595c 100644
@@ -3225,6 +3225,9 @@ static int btf_find_kptr(const struct btf *btf, const struct btf_type *t,
 	enum bpf_kptr_type type;
 	u32 res_id;
 
+	/* Permit modifiers on the pointer itself */
+	if (btf_type_is_volatile(t))
+		t = btf_type_by_id(btf, t->type);
 	/* For PTR, sz is always == 8 */
 	if (!btf_type_is_ptr(t))
 		return BTF_FIELD_IGNORE;
Signature · 1 line
2.34.1
author maintainer
#12 lore raw

RFC PATCH 8/32 bpf: Add comment about kptr's PTR_TO_MAP_VALUE handling

In reply to Kumar Kartikeya Dwivedi

In both process_ktpr_func and kptr_get handling for kfuncs, we expect PTR_TO_MAP_VALUE with a constant var_off and optionally fixed off, which in turn points to the kptr in the map value. We know that if we find such offset in the kptr_off_tab it will be < value_size.

Hence, we skip checking the memory region access. Once establishing that it is a kptr we also don't need to check whether the map value pointer touches any other special fields for [ptr, ptr+8) region we are about to access.

Finally, for check_map_access_type, we already ensure that neither BPF_F_RDONLY_PROG and BPF_F_WRONLY_PROG flags can be set for the map containing kptrs. Hence, checking that is also not required.

Encode all these implicit assumptions as comments where such checks are made, so that any future changes to these take the kptr related invariants into consideration, and avoid introducing bugs accidently.

All this information was also clarified in the commit adding kptr support, 61df10c7799e ("bpf: Allow storing unreferenced kptr in map").

---
 kernel/bpf/verifier.c | 5 +++++
 1 file changed, 5 insertions(+)
kernel/bpf/verifier.c +5 -0
index b7bf68f3b2ec..0c19a98c748d 100644
@@ -5196,6 +5196,11 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno,
 		return check_mem_region_access(env, regno, reg->off, access_size,
 					       reg->map_ptr->key_size, false);
 	case PTR_TO_MAP_VALUE:
+		/* process_kptr_func and kptr_get assume only map_access_type
+		 * and special field access is checked for PTR_TO_MAP_VALUE,
+		 * apart from verifying memory region access, hence they must be
+		 * revisited when that assumption changes here.
+		 */
 		if (check_map_access_type(env, regno, reg->off, access_size,
 					  meta && meta->raw_mode ? BPF_WRITE :
 					  BPF_READ))
Signature · 1 line
2.34.1
author maintainer
#13 lore raw

RFC PATCH 9/32 bpf: Rewrite kfunc argument handling

In reply to Kumar Kartikeya Dwivedi

As we continue to add more features, argument types, kfunc flags, and different extensions to kfuncs, the code to verify the correctness of the kfunc prototype wrt the passed in registers has become ad-hoc and ugly to read.

To make life easier, and make a very clear split between different stages of argument processing, move all the code into verifier.c and refactor into easier to read helpers and functions.

This also makes sharing code within the verifier easier with kfunc argument processing. This will be more and more useful in later patches as we are now moving to implement very core BPF helpers as kfuncs, to keep them experimental before baking into UAPI.

---
 include/linux/btf.h                          |  22 +
 kernel/bpf/btf.c                             |  12 +-
 kernel/bpf/verifier.c                        | 426 ++++++++++++++++++-
 tools/testing/selftests/bpf/verifier/calls.c |   2 +-
 4 files changed, 438 insertions(+), 24 deletions(-)
include/linux/btf.h +22 -0
index b3d47e9b9d5c..8062f9da7c40 100644
@@ -314,6 +314,16 @@ static inline bool btf_type_is_struct(const struct btf_type *t)
 	return kind == BTF_KIND_STRUCT || kind == BTF_KIND_UNION;
 }
 
+static inline bool __btf_type_is_struct(const struct btf_type *t)
+{
+	return BTF_INFO_KIND(t->info) == BTF_KIND_STRUCT;
+}
+
+static inline bool btf_type_is_array(const struct btf_type *t)
+{
+	return BTF_INFO_KIND(t->info) == BTF_KIND_ARRAY;
+}
+
 static inline u16 btf_type_vlen(const struct btf_type *t)
 {
 	return BTF_INFO_VLEN(t->info);
@@ -400,6 +410,7 @@ static inline struct btf_param *btf_params(const struct btf_type *t)
 
 #ifdef CONFIG_BPF_SYSCALL
 struct bpf_prog;
+struct bpf_verifier_log;
 
 const struct btf_type *btf_type_by_id(const struct btf *btf, u32 type_id);
 const char *btf_name_by_offset(const struct btf *btf, u32 offset);
@@ -413,6 +424,10 @@ int register_btf_kfunc_id_set(enum bpf_prog_type prog_type,
 s32 btf_find_dtor_kfunc(struct btf *btf, u32 btf_id);
 int register_btf_id_dtor_kfuncs(const struct btf_id_dtor_kfunc *dtors, u32 add_cnt,
 				struct module *owner);
+const struct btf_member *
+btf_get_prog_ctx_type(struct bpf_verifier_log *log, const struct btf *btf,
+		      const struct btf_type *t, enum bpf_prog_type prog_type,
+		      int arg);
 #else
 static inline const struct btf_type *btf_type_by_id(const struct btf *btf,
 						    u32 type_id)
@@ -444,6 +459,13 @@ static inline int register_btf_id_dtor_kfuncs(const struct btf_id_dtor_kfunc *dt
 {
 	return 0;
 }
+static inline const struct btf_member *
+btf_get_prog_ctx_type(struct bpf_verifier_log *log, const struct btf *btf,
+		      const struct btf_type *t, enum bpf_prog_type prog_type,
+		      int arg)
+{
+	return NULL;
+}
 #endif
 
 #endif
kernel/bpf/btf.c +1 -11
index 5e860f76595c..0ad809a3055d 100644
@@ -477,16 +477,6 @@ static bool btf_type_nosize_or_null(const struct btf_type *t)
 	return !t || btf_type_nosize(t);
 }
 
-static bool __btf_type_is_struct(const struct btf_type *t)
-{
-	return BTF_INFO_KIND(t->info) == BTF_KIND_STRUCT;
-}
-
-static bool btf_type_is_array(const struct btf_type *t)
-{
-	return BTF_INFO_KIND(t->info) == BTF_KIND_ARRAY;
-}
-
 static bool btf_type_is_datasec(const struct btf_type *t)
 {
 	return BTF_INFO_KIND(t->info) == BTF_KIND_DATASEC;
@@ -5089,7 +5079,7 @@ static u8 bpf_ctx_convert_map[] = {
 #undef BPF_MAP_TYPE
 #undef BPF_LINK_TYPE
 
-static const struct btf_member *
+const struct btf_member *
 btf_get_prog_ctx_type(struct bpf_verifier_log *log, const struct btf *btf,
 		      const struct btf_type *t, enum bpf_prog_type prog_type,
 		      int arg)
kernel/bpf/verifier.c +414 -12

472 lines not on this page · raw message

tools/testing/selftests/bpf/verifier/calls.c +1 -1
index 3fb4f69b1962..4604d0b1fb04 100644
@@ -109,7 +109,7 @@
 	},
 	.prog_type = BPF_PROG_TYPE_SCHED_CLS,
 	.result = REJECT,
-	.errstr = "arg#0 pointer type STRUCT prog_test_ref_kfunc must point",
+	.errstr = "arg#0 expected pointer to btf or socket",
 	.fixup_kfunc_btf_id = {
 		{ "bpf_kfunc_call_test_acquire", 3 },
 		{ "bpf_kfunc_call_test_release", 5 },
Signature · 1 line
2.34.1
author maintainer
#14 lore raw

RFC PATCH 10/32 bpf: Drop kfunc support from btf_check_func_arg_match

In reply to Kumar Kartikeya Dwivedi

Remove all kfunc related bits now from btf_check_func_arg_match, as users have been converted away to refactored kfunc argument handling.

This is split into a separate commit to aid review, in order to compare what has been preserved from the removed bits easily instead of mixing removed hunks with previous patch.

---
 include/linux/bpf.h          |   4 -
 include/linux/bpf_verifier.h |   2 -
 kernel/bpf/btf.c             | 262 ++---------------------------------
 kernel/bpf/verifier.c        |   4 +-
 4 files changed, 10 insertions(+), 262 deletions(-)
include/linux/bpf.h +0 -4
index cdc0a8c1b1d1..d4e6bf789c02 100644
@@ -1976,10 +1976,6 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,
 struct bpf_reg_state;
 int btf_check_subprog_arg_match(struct bpf_verifier_env *env, int subprog,
 				struct bpf_reg_state *regs);
-int btf_check_kfunc_arg_match(struct bpf_verifier_env *env,
-			      const struct btf *btf, u32 func_id,
-			      struct bpf_reg_state *regs,
-			      u32 kfunc_flags);
 int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog,
 			  struct bpf_reg_state *reg);
 int btf_check_type_match(struct bpf_verifier_log *log, const struct bpf_prog *prog,
include/linux/bpf_verifier.h +0 -2
index 1fdddbf3546b..b4a11ff56054 100644
@@ -567,8 +567,6 @@ int check_ptr_off_reg(struct bpf_verifier_env *env,
 int check_func_arg_reg_off(struct bpf_verifier_env *env,
 			   const struct bpf_reg_state *reg, int regno,
 			   enum bpf_arg_type arg_type);
-int check_kfunc_mem_size_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
-			     u32 regno);
 int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
 		   u32 regno, u32 mem_size);
 
kernel/bpf/btf.c +8 -254

328 lines not on this page · raw message

kernel/bpf/verifier.c +2 -2
index 663c91020f82..96fab14eb94e 100644
@@ -5338,8 +5338,8 @@ int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
 	return err;
 }
 
-int check_kfunc_mem_size_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
-			     u32 regno)
+static int check_kfunc_mem_size_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
+				    u32 regno)
 {
 	struct bpf_reg_state *mem_reg = &cur_regs(env)[regno - 1];
 	bool may_be_null = type_may_be_null(mem_reg->type);
Signature · 1 line
2.34.1
author maintainer
#15 lore raw

RFC PATCH 11/32 bpf: Support constant scalar arguments for kfuncs

In reply to Kumar Kartikeya Dwivedi

Allow passing known constant scalars as arguments to kfuncs that do not represent a size parameter. This makes the search pruning optimization of verifier more conservative for such kfunc calls, and each non-distinct argument is considered unequivalent.

We will use this support to then expose a global bpf_kptr_alloc function where it takes the local type ID in program BTF, and returns a PTR_TO_BTF_ID to the local type. These will be called local kptrs, and allows programs to allocate their own objects.

However, this is still not completely safe, as mark_chain_precision logic is buggy without more work when the constant argument is not a size, but still needs precise marker propagation for pruning checks. Next patch will fix this problem.

---
 Documentation/bpf/kfuncs.rst | 30 ++++++++++++++++
 kernel/bpf/verifier.c        | 67 +++++++++++++++++++++++++++---------
 2 files changed, 80 insertions(+), 17 deletions(-)
Documentation/bpf/kfuncs.rst +30 -0
index 781731749e55..31625393204a 100644
@@ -72,6 +72,36 @@ argument as its size. By default, without __sz annotation, the size of the type
 of the pointer is used. Without __sz annotation, a kfunc cannot accept a void
 pointer.
 
+2.2.1 __k Annotation
+--------------------
+
+This annotation is only understood for scalar arguments, where it indicates that
+the verifier must check the scalar argument to be a known constant, which does
+not indicate a size parameter. This distinction is important, as when the scalar
+argument does not represent a size parameter, verifier is more conservative in
+state search pruning and does not consider two arguments equivalent for safety
+purposes if the already verified value was within range of the new one.
+
+This assumption holds well for sizes (as memory accessed within smaller bounds
+in old verified state will also work for bigger bounds in current to be explored
+state), but not for other constant arguments where each carries a distinct
+semantic effect.
+
+An example is given below::
+
+        void *bpf_mem_alloc(u32 local_type_id__k)
+        {
+        ...
+        }
+
+Here, bpf_mem_alloc uses local_type_id argument to find out the size of that
+type ID in program's BTF and return a sized pointer to it. Each type ID will
+have a distinct size, hence it is crucial to treat each such call as distinct
+when values don't match.
+
+Hence, whenever a constant scalar argument is accepted by a kfunc which is not a
+size parameter, __k suffix must be used.
+
 .. _BPF_kfunc_nodef:
 
 2.3 Using an existing kernel function
kernel/bpf/verifier.c +50 -17

96 lines not on this page · raw message

Signature · 1 line
2.34.1
author maintainer
#16 lore raw

RFC PATCH 12/32 bpf: Teach verifier about non-size constant arguments

In reply to Kumar Kartikeya Dwivedi

Currently, the verifier has support for various arguments that either describe the size of the memory being passed in to a helper, or describe the size of the memory being returned. When a constant is passed in like this, it is assumed for the purposes of precision tracking that if the value in the already explored safe state is within the value in current state, it would fine to prune the search.

While this holds well for size arguments, arguments where each value may denote a distinct meaning and needs to be verified separately needs more work. Search can only be pruned if both are constant values and both are equal. In all other cases, it would be incorrect to treat those two precise registers as equivalent if the new value satisfies the old one (i.e. old <= cur).

Hence, make the register precision marker tri-state. There are now three values that reg->precise takes: NOT_PRECISE, PRECISE, PRECISE_ABSOLUTE.

Both PRECISE and PRECISE_ABSOLUTE are 'true' values. PRECISE_ABSOLUTE affects how regsafe decides whether both registers are equivalent for the purposes of verifier state equivalence. When it sees that one register has reg->precise == PRECISE_ABSOLUTE, unless both are absolute, it will return false. When both are, it returns true only when both are const and both have the same value. Otherwise, for PRECISE case it falls back to the default check that is present now (i.e. thinking that we're talking about sizes).

This is required as a future patch introduces a BPF memory allocator interface, where we take the program BTF's type ID as an argument. Each distinct type ID may result in the returned pointer obtaining a different size, hence precision tracking is needed, and pruning cannot just happen when the old value is within the current value. It must only happen when the type ID is equal. The type ID will always correspond to prog->aux->btf hence actual type match is not required.

Finally, change mark_chain_precision to mark_chain_precision_absolute for kfuncs constant non-size scalar arguments (tagged with __k suffix).

---
 include/linux/bpf_verifier.h |  8 +++-
 kernel/bpf/verifier.c        | 93 ++++++++++++++++++++++++++----------
 2 files changed, 76 insertions(+), 25 deletions(-)
include/linux/bpf_verifier.h +7 -1
index b4a11ff56054..c4d21568d192 100644
@@ -43,6 +43,12 @@ enum bpf_reg_liveness {
 	REG_LIVE_DONE = 0x8, /* liveness won't be updating this register anymore */
 };
 
+enum bpf_reg_precise {
+	NOT_PRECISE,
+	PRECISE,
+	PRECISE_ABSOLUTE,
+};
+
 struct bpf_reg_state {
 	/* Ordering of fields matters.  See states_equal() */
 	enum bpf_reg_type type;
@@ -180,7 +186,7 @@ struct bpf_reg_state {
 	s32 subreg_def;
 	enum bpf_reg_liveness live;
 	/* if (!precise && SCALAR_VALUE) min/max/tnum don't affect safety */
-	bool precise;
+	enum bpf_reg_precise precise;
 };
 
 enum bpf_stack_slot_type {
kernel/bpf/verifier.c +69 -24

211 lines not on this page · raw message

Signature · 1 line
2.34.1
#17 lore raw

Re: [PATCH RFC bpf-next v1 12/32] bpf: Teach verifier about non-size constant arguments

Quoted 48 lines · Kumar Kartikeya Dwivedi

Currently, the verifier has support for various arguments that either describe the size of the memory being passed in to a helper, or describe the size of the memory being returned. When a constant is passed in like this, it is assumed for the purposes of precision tracking that if the value in the already explored safe state is within the value in current state, it would fine to prune the search.

While this holds well for size arguments, arguments where each value may denote a distinct meaning and needs to be verified separately needs more work. Search can only be pruned if both are constant values and both are equal. In all other cases, it would be incorrect to treat those two precise registers as equivalent if the new value satisfies the old one (i.e. old <= cur).

Hence, make the register precision marker tri-state. There are now three values that reg->precise takes: NOT_PRECISE, PRECISE, PRECISE_ABSOLUTE.

Both PRECISE and PRECISE_ABSOLUTE are 'true' values. PRECISE_ABSOLUTE affects how regsafe decides whether both registers are equivalent for the purposes of verifier state equivalence. When it sees that one register has reg->precise == PRECISE_ABSOLUTE, unless both are absolute, it will return false. When both are, it returns true only when both are const and both have the same value. Otherwise, for PRECISE case it falls back to the default check that is present now (i.e. thinking that we're talking about sizes).

This is required as a future patch introduces a BPF memory allocator interface, where we take the program BTF's type ID as an argument. Each distinct type ID may result in the returned pointer obtaining a different size, hence precision tracking is needed, and pruning cannot just happen when the old value is within the current value. It must only happen when the type ID is equal. The type ID will always correspond to prog->aux->btf hence actual type match is not required.

Finally, change mark_chain_precision to mark_chain_precision_absolute for kfuncs constant non-size scalar arguments (tagged with __k suffix).

---
 include/linux/bpf_verifier.h |  8 +++-
 kernel/bpf/verifier.c        | 93 ++++++++++++++++++++++++++----------
 2 files changed, 76 insertions(+), 25 deletions(-)
include/linux/bpf_verifier.h +0 -0
index b4a11ff56054..c4d21568d192 100644

include/linux/bpf_verifier.h enum bpf_reg_liveness { · line 50

Quoted 8 lines
include/linux/bpf_verifier.h +5 -0
@@ -43,6 +43,12 @@ enum bpf_reg_liveness {
 	REG_LIVE_DONE = 0x8, /* liveness won't be updating this register anymore */
 };
 
+enum bpf_reg_precise {
+	NOT_PRECISE,
+	PRECISE,
+	PRECISE_ABSOLUTE,
+};

Can we make it less verbose ?

NOT_PRECISE,
PRECISE,
EXACT

include/linux/bpf_verifier.h struct bpf_reg_state { · line 189

Quoted 10 lines
include/linux/bpf_verifier.h +2 -1
+
 struct bpf_reg_state {
 	/* Ordering of fields matters.  See states_equal() */
 	enum bpf_reg_type type;
@@ -180,7 +186,7 @@ struct bpf_reg_state {
 	s32 subreg_def;
 	enum bpf_reg_liveness live;
 	/* if (!precise && SCALAR_VALUE) min/max/tnum don't affect safety */
-	bool precise;
+	enum bpf_reg_precise precise;

Have been thinking whether
bool precise;
bool exact;
would be better,
but doesn't look like it.

kernel/bpf/verifier.c static void print_verifier_state(struct bpf_verifier_env *env, · line 841

Quoted 13 lines
include/linux/bpf_verifier.h +0 -0
 };
 
 enum bpf_stack_slot_type {
kernel/bpf/verifier.c +1 -1
index b28e88d6fabd..571790ac58d4 100644
@@ -838,7 +838,7 @@ static void print_verifier_state(struct bpf_verifier_env *env,
 		print_liveness(env, reg->live);
 		verbose(env, "=");
 		if (t == SCALAR_VALUE && reg->precise)
-			verbose(env, "P");
+			verbose(env, reg->precise == PRECISE_ABSOLUTE ? "PA" : "P");

and here it would be just 'E'

Quoted 53 lines
kernel/bpf/verifier.c +15 -4
 		if ((t == SCALAR_VALUE || t == PTR_TO_STACK) &&
 		    tnum_is_const(reg->var_off)) {
 			/* reg->off should be 0 for SCALAR_VALUE */
@@ -935,7 +935,7 @@ static void print_verifier_state(struct bpf_verifier_env *env,
 			t = reg->type;
 			verbose(env, "=%s", t == SCALAR_VALUE ? "" : reg_type_str(env, t));
 			if (t == SCALAR_VALUE && reg->precise)
-				verbose(env, "P");
+				verbose(env, reg->precise == PRECISE_ABSOLUTE ? "PA" : "P");
 			if (t == SCALAR_VALUE && tnum_is_const(reg->var_off))
 				verbose(env, "%lld", reg->var_off.value + reg->off);
 		} else {
@@ -1668,7 +1668,17 @@ static void __mark_reg_unknown(const struct bpf_verifier_env *env,
 	reg->type = SCALAR_VALUE;
 	reg->var_off = tnum_unknown;
 	reg->frameno = 0;
-	reg->precise = env->subprog_cnt > 1 || !env->bpf_capable;
+	/* Helpers requiring PRECISE_ABSOLUTE for constant arguments cannot be
+	 * called from programs without CAP_BPF. This is because we don't
+	 * propagate precision markers for when CAP_BPF is missing. If we
+	 * allowed calling such heleprs in those programs, the default would
+	 * have to be PRECISE_ABSOLUTE for them, which would be too aggresive.
+	 *
+	 * We still propagate PRECISE_ABSOLUTE when subprog_cnt > 1, hence
+	 * those cases would still override the default PRECISE value when
+	 * we propagate the precision markers.
+	 */
+	reg->precise = (env->subprog_cnt > 1 || !env->bpf_capable) ? PRECISE : NOT_PRECISE;
 	__mark_reg_unbounded(reg);
 }
 
@@ -2717,7 +2727,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx,
  * For now backtracking falls back into conservative marking.
  */
 static void mark_all_scalars_precise(struct bpf_verifier_env *env,
-				     struct bpf_verifier_state *st)
+				     struct bpf_verifier_state *st,
+				     bool absolute)
 {
 	struct bpf_func_state *func;
 	struct bpf_reg_state *reg;
@@ -2733,7 +2744,7 @@ static void mark_all_scalars_precise(struct bpf_verifier_env *env,
 				reg = &func->regs[j];
 				if (reg->type != SCALAR_VALUE)
 					continue;
-				reg->precise = true;
+				reg->precise = absolute ? PRECISE_ABSOLUTE : PRECISE;
 			}
 			for (j = 0; j < func->allocated_stack / BPF_REG_SIZE; j++) {
 				if (!is_spilled_reg(&func->stack[j]))
@@ -2741,13 +2752,13 @@ static void mark_all_scalars_precise(struct bpf_verifier_env *env,
 				reg = &func->stack[j].spilled_ptr;
 				if (reg->type != SCALAR_VALUE)

kernel/bpf/verifier.c static void mark_all_scalars_precise(struct bpf_verifier_env *env, · line 2761

Quoted 10 lines
kernel/bpf/verifier.c +2 -2
 					continue;
-				reg->precise = true;
+				reg->precise = absolute ? PRECISE_ABSOLUTE : PRECISE;
 			}
 		}
 }
 
 static int __mark_chain_precision(struct bpf_verifier_env *env, int regno,
-				  int spi)
+				  int spi, bool absolute)

instead of bool pls pass enum bpf_reg_precise

Quoted 66 lines
kernel/bpf/verifier.c +10 -7
 {
 	struct bpf_verifier_state *st = env->cur_state;
 	int first_idx = st->first_insn_idx;
@@ -2774,7 +2785,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int regno,
 			new_marks = true;
 		else
 			reg_mask = 0;
-		reg->precise = true;
+		reg->precise = absolute ? PRECISE_ABSOLUTE : PRECISE;
 	}
 
 	while (spi >= 0) {
@@ -2791,7 +2802,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int regno,
 			new_marks = true;
 		else
 			stack_mask = 0;
-		reg->precise = true;
+		reg->precise = absolute ? PRECISE_ABSOLUTE : PRECISE;
 		break;
 	}
 
@@ -2813,7 +2824,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int regno,
 				err = backtrack_insn(env, i, &reg_mask, &stack_mask);
 			}
 			if (err == -ENOTSUPP) {
-				mark_all_scalars_precise(env, st);
+				mark_all_scalars_precise(env, st, absolute);
 				return 0;
 			} else if (err) {
 				return err;
@@ -2854,7 +2865,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int regno,
 			}
 			if (!reg->precise)
 				new_marks = true;
-			reg->precise = true;
+			reg->precise = absolute ? PRECISE_ABSOLUTE : PRECISE;
 		}
 
 		bitmap_from_u64(mask, stack_mask);
@@ -2873,7 +2884,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int regno,
 				 * fp-8 and it's "unallocated" stack space.
 				 * In such case fallback to conservative.
 				 */
-				mark_all_scalars_precise(env, st);
+				mark_all_scalars_precise(env, st, absolute);
 				return 0;
 			}
 
@@ -2888,7 +2899,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int regno,
 			}
 			if (!reg->precise)
 				new_marks = true;
-			reg->precise = true;
+			reg->precise = absolute ? PRECISE_ABSOLUTE : PRECISE;
 		}
 		if (env->log.level & BPF_LOG_LEVEL2) {
 			verbose(env, "parent %s regs=%x stack=%llx marks:",
@@ -2910,12 +2921,24 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int regno,
 
 static int mark_chain_precision(struct bpf_verifier_env *env, int regno)
 {
-	return __mark_chain_precision(env, regno, -1);
+	return __mark_chain_precision(env, regno, -1, false);
+}
+
+static int mark_chain_precision_absolute(struct bpf_verifier_env *env, int regno)

kernel/bpf/verifier.c static int __mark_chain_precision(struct bpf_verifier_env *env, int regno, · line 2936

Quoted 10 lines
kernel/bpf/verifier.c +5 -1
+{
+	WARN_ON_ONCE(!env->bpf_capable);
+	return __mark_chain_precision(env, regno, -1, true);
 }
 
 static int mark_chain_precision_stack(struct bpf_verifier_env *env, int spi)
 {
-	return __mark_chain_precision(env, -1, spi);
+	return __mark_chain_precision(env, -1, spi, false);
+}

No need to fork the functions so much.
Just add enum bpf_reg_precise to existing two functions.

Quoted 35 lines
kernel/bpf/verifier.c +9 -5
+
+static int mark_chain_precision_absolute_stack(struct bpf_verifier_env *env, int spi)
+{
+	WARN_ON_ONCE(!env->bpf_capable);
+	return __mark_chain_precision(env, -1, spi, true);
 }
 
 static bool is_spillable_regtype(enum bpf_reg_type type)
@@ -3253,7 +3276,7 @@ static void mark_reg_stack_read(struct bpf_verifier_env *env,
 		 * backtracking. Any register that contributed
 		 * to const 0 was marked precise before spill.
 		 */
-		state->regs[dst_regno].precise = true;
+		state->regs[dst_regno].precise = PRECISE;
 	} else {
 		/* have read misc data from the stack */
 		mark_reg_unknown(env, state->regs, dst_regno);
@@ -7903,7 +7926,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_arg_m
 					verbose(env, "R%d must be a known constant\n", regno);
 					return -EINVAL;
 				}
-				ret = mark_chain_precision(env, regno);
+				ret = mark_chain_precision_absolute(env, regno);
 				if (ret < 0)
 					return ret;
 				meta->arg_constant.found = true;
@@ -11899,9 +11922,23 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,
 		if (rcur->type == SCALAR_VALUE) {
 			if (!rold->precise && !rcur->precise)
 				return true;
-			/* new val must satisfy old val knowledge */
-			return range_within(rold, rcur) &&
-			       tnum_in(rold->var_off, rcur->var_off);
+			/* We can only determine safety when type of precision
+			 * needed is same. For absolute, we must compare actual

kernel/bpf/verifier.c static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold, · line 11936

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+			 * value, otherwise old being within the current value
+			 * suffices.
+			 */
+			if (rold->precise == PRECISE_ABSOLUTE || rcur->precise == PRECISE_ABSOLUTE) {
+				/* Both should be PRECISE_ABSOLUTE for a comparison */
+				if (rold->precise != rcur->precise)
+					return false;
+				if (!tnum_is_const(rold->var_off) || !tnum_is_const(rcur->var_off))
+					return false;
+				return rold->var_off.value == rcur->var_off.value;

Probably better to do
if (rold->precise == EXACT || rcu->precise == EXACT)
return false;

because
if (equal)

    return true;

should have already happened if they were exact match.

Quoted 42 lines
kernel/bpf/verifier.c +17 -4
+			} else {
+				/* new val must satisfy old val knowledge */
+				return range_within(rold, rcur) &&
+				       tnum_in(rold->var_off, rcur->var_off);
+			}
 		} else {
 			/* We're trying to use a pointer in place of a scalar.
 			 * Even if the scalar was unbounded, this could lead to
@@ -12229,8 +12266,12 @@ static int propagate_precision(struct bpf_verifier_env *env,
 		    !state_reg->precise)
 			continue;
 		if (env->log.level & BPF_LOG_LEVEL2)
-			verbose(env, "propagating r%d\n", i);
-		err = mark_chain_precision(env, i);
+			verbose(env, "propagating %sr%d\n",
+				state_reg->precise == PRECISE_ABSOLUTE ? "abs " : "", i);
+		if (state_reg->precise == PRECISE_ABSOLUTE)
+			err = mark_chain_precision_absolute(env, i);
+		else
+			err = mark_chain_precision(env, i);
 		if (err < 0)
 			return err;
 	}
@@ -12243,9 +12284,13 @@ static int propagate_precision(struct bpf_verifier_env *env,
 		    !state_reg->precise)
 			continue;
 		if (env->log.level & BPF_LOG_LEVEL2)
-			verbose(env, "propagating fp%d\n",
+			verbose(env, "propagating %sfp%d\n",
+				state_reg->precise == PRECISE_ABSOLUTE ? "abs " : "",
 				(-i - 1) * BPF_REG_SIZE);
-		err = mark_chain_precision_stack(env, i);
+		if (state_reg->precise == PRECISE_ABSOLUTE)
+			err = mark_chain_precision_absolute_stack(env, i);
+		else
+			err = mark_chain_precision_stack(env, i);
 		if (err < 0)
 			return err;
 	}
Signature · 1 line
2.34.1
author maintainer
#18 lore raw

Re: [PATCH RFC bpf-next v1 12/32] bpf: Teach verifier about non-size constant arguments

Quoted 61 lines · Alexei Starovoitov
Quoted 48 lines · Kumar Kartikeya Dwivedi

Currently, the verifier has support for various arguments that either describe the size of the memory being passed in to a helper, or describe the size of the memory being returned. When a constant is passed in like this, it is assumed for the purposes of precision tracking that if the value in the already explored safe state is within the value in current state, it would fine to prune the search.

While this holds well for size arguments, arguments where each value may denote a distinct meaning and needs to be verified separately needs more work. Search can only be pruned if both are constant values and both are equal. In all other cases, it would be incorrect to treat those two precise registers as equivalent if the new value satisfies the old one (i.e. old <= cur).

Hence, make the register precision marker tri-state. There are now three values that reg->precise takes: NOT_PRECISE, PRECISE, PRECISE_ABSOLUTE.

Both PRECISE and PRECISE_ABSOLUTE are 'true' values. PRECISE_ABSOLUTE affects how regsafe decides whether both registers are equivalent for the purposes of verifier state equivalence. When it sees that one register has reg->precise == PRECISE_ABSOLUTE, unless both are absolute, it will return false. When both are, it returns true only when both are const and both have the same value. Otherwise, for PRECISE case it falls back to the default check that is present now (i.e. thinking that we're talking about sizes).

This is required as a future patch introduces a BPF memory allocator interface, where we take the program BTF's type ID as an argument. Each distinct type ID may result in the returned pointer obtaining a different size, hence precision tracking is needed, and pruning cannot just happen when the old value is within the current value. It must only happen when the type ID is equal. The type ID will always correspond to prog->aux->btf hence actual type match is not required.

Finally, change mark_chain_precision to mark_chain_precision_absolute for kfuncs constant non-size scalar arguments (tagged with __k suffix).

---
 include/linux/bpf_verifier.h |  8 +++-
 kernel/bpf/verifier.c        | 93 ++++++++++++++++++++++++++----------
 2 files changed, 76 insertions(+), 25 deletions(-)
include/linux/bpf_verifier.h +0 -0
index b4a11ff56054..c4d21568d192 100644

include/linux/bpf_verifier.h enum bpf_reg_liveness { · line 50

Quoted 8 lines
include/linux/bpf_verifier.h +5 -0
@@ -43,6 +43,12 @@ enum bpf_reg_liveness {
      REG_LIVE_DONE = 0x8, /* liveness won't be updating this register anymore */
 };
 
+enum bpf_reg_precise {
+     NOT_PRECISE,
+     PRECISE,
+     PRECISE_ABSOLUTE,
+};

Can we make it less verbose ?

Quoted 4 lines

NOT_PRECISE,
PRECISE,
EXACT

Yes, looks better.

Quoted 229 lines

include/linux/bpf_verifier.h struct bpf_reg_state { · line 189

Quoted 10 lines
include/linux/bpf_verifier.h +2 -1
+
 struct bpf_reg_state {
      /* Ordering of fields matters.  See states_equal() */
      enum bpf_reg_type type;
@@ -180,7 +186,7 @@ struct bpf_reg_state {
      s32 subreg_def;
      enum bpf_reg_liveness live;
      /* if (!precise && SCALAR_VALUE) min/max/tnum don't affect safety */
-     bool precise;
+     enum bpf_reg_precise precise;

Have been thinking whether
bool precise;
bool exact;
would be better,
but doesn't look like it.

kernel/bpf/verifier.c static void print_verifier_state(struct bpf_verifier_env *env, · line 841

Quoted 13 lines
include/linux/bpf_verifier.h +0 -0
 };
 
 enum bpf_stack_slot_type {
kernel/bpf/verifier.c +1 -1
index b28e88d6fabd..571790ac58d4 100644
@@ -838,7 +838,7 @@ static void print_verifier_state(struct bpf_verifier_env *env,
              print_liveness(env, reg->live);
              verbose(env, "=");
              if (t == SCALAR_VALUE && reg->precise)
-                     verbose(env, "P");
+                     verbose(env, reg->precise == PRECISE_ABSOLUTE ? "PA" : "P");

and here it would be just 'E'

Quoted 53 lines
kernel/bpf/verifier.c +15 -4
              if ((t == SCALAR_VALUE || t == PTR_TO_STACK) &&
                  tnum_is_const(reg->var_off)) {
                      /* reg->off should be 0 for SCALAR_VALUE */
@@ -935,7 +935,7 @@ static void print_verifier_state(struct bpf_verifier_env *env,
                      t = reg->type;
                      verbose(env, "=%s", t == SCALAR_VALUE ? "" : reg_type_str(env, t));
                      if (t == SCALAR_VALUE && reg->precise)
-                             verbose(env, "P");
+                             verbose(env, reg->precise == PRECISE_ABSOLUTE ? "PA" : "P");
                      if (t == SCALAR_VALUE && tnum_is_const(reg->var_off))
                              verbose(env, "%lld", reg->var_off.value + reg->off);
              } else {
@@ -1668,7 +1668,17 @@ static void __mark_reg_unknown(const struct bpf_verifier_env *env,
      reg->type = SCALAR_VALUE;
      reg->var_off = tnum_unknown;
      reg->frameno = 0;
-     reg->precise = env->subprog_cnt > 1 || !env->bpf_capable;
+     /* Helpers requiring PRECISE_ABSOLUTE for constant arguments cannot be
+      * called from programs without CAP_BPF. This is because we don't
+      * propagate precision markers for when CAP_BPF is missing. If we
+      * allowed calling such heleprs in those programs, the default would
+      * have to be PRECISE_ABSOLUTE for them, which would be too aggresive.
+      *
+      * We still propagate PRECISE_ABSOLUTE when subprog_cnt > 1, hence
+      * those cases would still override the default PRECISE value when
+      * we propagate the precision markers.
+      */
+     reg->precise = (env->subprog_cnt > 1 || !env->bpf_capable) ? PRECISE : NOT_PRECISE;
      __mark_reg_unbounded(reg);
 }
 
@@ -2717,7 +2727,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx,
  * For now backtracking falls back into conservative marking.
  */
 static void mark_all_scalars_precise(struct bpf_verifier_env *env,
-                                  struct bpf_verifier_state *st)
+                                  struct bpf_verifier_state *st,
+                                  bool absolute)
 {
      struct bpf_func_state *func;
      struct bpf_reg_state *reg;
@@ -2733,7 +2744,7 @@ static void mark_all_scalars_precise(struct bpf_verifier_env *env,
                              reg = &func->regs[j];
                              if (reg->type != SCALAR_VALUE)
                                      continue;
-                             reg->precise = true;
+                             reg->precise = absolute ? PRECISE_ABSOLUTE : PRECISE;
                      }
                      for (j = 0; j < func->allocated_stack / BPF_REG_SIZE; j++) {
                              if (!is_spilled_reg(&func->stack[j]))
@@ -2741,13 +2752,13 @@ static void mark_all_scalars_precise(struct bpf_verifier_env *env,
                              reg = &func->stack[j].spilled_ptr;
                              if (reg->type != SCALAR_VALUE)

kernel/bpf/verifier.c static void mark_all_scalars_precise(struct bpf_verifier_env *env, · line 2761

Quoted 10 lines
kernel/bpf/verifier.c +2 -2
                                      continue;
-                             reg->precise = true;
+                             reg->precise = absolute ? PRECISE_ABSOLUTE : PRECISE;
                      }
              }
 }
 
 static int __mark_chain_precision(struct bpf_verifier_env *env, int regno,
-                               int spi)
+                               int spi, bool absolute)

instead of bool pls pass enum bpf_reg_precise

Quoted 66 lines
kernel/bpf/verifier.c +10 -7
 {
      struct bpf_verifier_state *st = env->cur_state;
      int first_idx = st->first_insn_idx;
@@ -2774,7 +2785,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int regno,
                      new_marks = true;
              else
                      reg_mask = 0;
-             reg->precise = true;
+             reg->precise = absolute ? PRECISE_ABSOLUTE : PRECISE;
      }
 
      while (spi >= 0) {
@@ -2791,7 +2802,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int regno,
                      new_marks = true;
              else
                      stack_mask = 0;
-             reg->precise = true;
+             reg->precise = absolute ? PRECISE_ABSOLUTE : PRECISE;
              break;
      }
 
@@ -2813,7 +2824,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int regno,
                              err = backtrack_insn(env, i, &reg_mask, &stack_mask);
                      }
                      if (err == -ENOTSUPP) {
-                             mark_all_scalars_precise(env, st);
+                             mark_all_scalars_precise(env, st, absolute);
                              return 0;
                      } else if (err) {
                              return err;
@@ -2854,7 +2865,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int regno,
                      }
                      if (!reg->precise)
                              new_marks = true;
-                     reg->precise = true;
+                     reg->precise = absolute ? PRECISE_ABSOLUTE : PRECISE;
              }
 
              bitmap_from_u64(mask, stack_mask);
@@ -2873,7 +2884,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int regno,
                               * fp-8 and it's "unallocated" stack space.
                               * In such case fallback to conservative.
                               */
-                             mark_all_scalars_precise(env, st);
+                             mark_all_scalars_precise(env, st, absolute);
                              return 0;
                      }
 
@@ -2888,7 +2899,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int regno,
                      }
                      if (!reg->precise)
                              new_marks = true;
-                     reg->precise = true;
+                     reg->precise = absolute ? PRECISE_ABSOLUTE : PRECISE;
              }
              if (env->log.level & BPF_LOG_LEVEL2) {
                      verbose(env, "parent %s regs=%x stack=%llx marks:",
@@ -2910,12 +2921,24 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int regno,
 
 static int mark_chain_precision(struct bpf_verifier_env *env, int regno)
 {
-     return __mark_chain_precision(env, regno, -1);
+     return __mark_chain_precision(env, regno, -1, false);
+}
+
+static int mark_chain_precision_absolute(struct bpf_verifier_env *env, int regno)

kernel/bpf/verifier.c static int __mark_chain_precision(struct bpf_verifier_env *env, int regno, · line 2936

Quoted 10 lines
kernel/bpf/verifier.c +5 -1
+{
+     WARN_ON_ONCE(!env->bpf_capable);
+     return __mark_chain_precision(env, regno, -1, true);
 }
 
 static int mark_chain_precision_stack(struct bpf_verifier_env *env, int spi)
 {
-     return __mark_chain_precision(env, -1, spi);
+     return __mark_chain_precision(env, -1, spi, false);
+}

No need to fork the functions so much.
Just add enum bpf_reg_precise to existing two functions.

Quoted 35 lines
kernel/bpf/verifier.c +9 -5
+
+static int mark_chain_precision_absolute_stack(struct bpf_verifier_env *env, int spi)
+{
+     WARN_ON_ONCE(!env->bpf_capable);
+     return __mark_chain_precision(env, -1, spi, true);
 }
 
 static bool is_spillable_regtype(enum bpf_reg_type type)
@@ -3253,7 +3276,7 @@ static void mark_reg_stack_read(struct bpf_verifier_env *env,
               * backtracking. Any register that contributed
               * to const 0 was marked precise before spill.
               */
-             state->regs[dst_regno].precise = true;
+             state->regs[dst_regno].precise = PRECISE;
      } else {
              /* have read misc data from the stack */
              mark_reg_unknown(env, state->regs, dst_regno);
@@ -7903,7 +7926,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_arg_m
                                      verbose(env, "R%d must be a known constant\n", regno);
                                      return -EINVAL;
                              }
-                             ret = mark_chain_precision(env, regno);
+                             ret = mark_chain_precision_absolute(env, regno);
                              if (ret < 0)
                                      return ret;
                              meta->arg_constant.found = true;
@@ -11899,9 +11922,23 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,
              if (rcur->type == SCALAR_VALUE) {
                      if (!rold->precise && !rcur->precise)
                              return true;
-                     /* new val must satisfy old val knowledge */
-                     return range_within(rold, rcur) &&
-                            tnum_in(rold->var_off, rcur->var_off);
+                     /* We can only determine safety when type of precision
+                      * needed is same. For absolute, we must compare actual

kernel/bpf/verifier.c static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold, · line 11936

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+                      * value, otherwise old being within the current value
+                      * suffices.
+                      */
+                     if (rold->precise == PRECISE_ABSOLUTE || rcur->precise == PRECISE_ABSOLUTE) {
+                             /* Both should be PRECISE_ABSOLUTE for a comparison */
+                             if (rold->precise != rcur->precise)
+                                     return false;
+                             if (!tnum_is_const(rold->var_off) || !tnum_is_const(rcur->var_off))
+                                     return false;
+                             return rold->var_off.value == rcur->var_off.value;

Probably better to do
if (rold->precise == EXACT || rcu->precise == EXACT)
return false;

Quoted 5 lines

because
if (equal)

    return true;

should have already happened if they were exact match.

I'll add a comment about that, just to make it clear. and I agree with all the suggestions.

author maintainer
#19 lore raw

RFC PATCH 13/32 bpf: Introduce bpf_list_head support for BPF maps

In reply to Kumar Kartikeya Dwivedi

Add the basic support on the map side to parse, recognize, verify, and build metadata table for a new special field of the type struct bpf_list_head. To parameterize the bpf_list_head for a certain value type and the list_node member it will accept in that value type, we use BTF declaration tags.

The definition of bpf_list_head in a map value will be done as follows:

struct foo {
	int data;
	struct bpf_list_node list;
};
struct map_value {
	struct bpf_list_head list __contains(struct, foo, node);
};

Then, the bpf_list_head only allows adding to the list using the bpf_list_node 'list' for the type struct foo.

The 'contains' annotation is a BTF declaration tag composed of four parts, "contains:kind:name:node" where the kind and name is then used to look up the type in the map BTF. The node defines name of the member in this type that has the type struct bpf_list_node, which is actually used for linking into the linked list.

This allows building intrusive linked lists in BPF, using container_of to obtain pointer to entry, while being completely type safe from the perspective of the verifier. The verifier knows exactly the type of the nodes, and knows that list helpers return that type at some fixed offset where the bpf_list_node member used for this list exists. The verifier also uses this information to disallow adding types that are not accepted by a certain list.

For now, no elements can be added to such lists. Support for that is coming in future patches, hence draining and freeing items is left out for now, and just freeing the list_head_off_tab is done, since it is still built and populated when bpf_list_head is specified in the map value.

---
 include/linux/bpf.h                           |  64 +++++--
 include/linux/btf.h                           |   2 +
 kernel/bpf/arraymap.c                         |   2 +
 kernel/bpf/bpf_local_storage.c                |   1 +
 kernel/bpf/btf.c                              | 173 +++++++++++++++++-
 kernel/bpf/hashtab.c                          |   1 +
 kernel/bpf/map_in_map.c                       |   5 +-
 kernel/bpf/syscall.c                          | 131 +++++++++++--
 kernel/bpf/verifier.c                         |  21 +++
 .../testing/selftests/bpf/bpf_experimental.h  |  21 +++
 10 files changed, 378 insertions(+), 43 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/bpf_experimental.h
include/linux/bpf.h +49 -15
index d4e6bf789c02..35c2e9caeb98 100644
@@ -28,6 +28,9 @@
 #include <linux/btf.h>
 #include <linux/rcupdate_trace.h>
 
+/* Experimental BPF APIs header for type definitions */
+#include "../../../tools/testing/selftests/bpf/bpf_experimental.h"
+
 struct bpf_verifier_env;
 struct bpf_verifier_log;
 struct perf_event;
@@ -164,27 +167,40 @@ struct bpf_map_ops {
 };
 
 enum {
-	/* Support at most 8 pointers in a BPF map value */
-	BPF_MAP_VALUE_OFF_MAX = 8,
-	BPF_MAP_OFF_ARR_MAX   = BPF_MAP_VALUE_OFF_MAX +
-				1 + /* for bpf_spin_lock */
-				1,  /* for bpf_timer */
-};
-
-enum bpf_kptr_type {
+	/* Support at most 8 offsets in a table */
+	BPF_MAP_VALUE_OFF_MAX		= 8,
+	/* Support at most 8 pointer in a BPF map value */
+	BPF_MAP_VALUE_KPTR_MAX		= BPF_MAP_VALUE_OFF_MAX,
+	/* Support at most 8 list_head in a BPF map value */
+	BPF_MAP_VALUE_LIST_HEAD_MAX	= BPF_MAP_VALUE_OFF_MAX,
+	BPF_MAP_OFF_ARR_MAX		= BPF_MAP_VALUE_KPTR_MAX +
+					  BPF_MAP_VALUE_LIST_HEAD_MAX +
+					  1 + /* for bpf_spin_lock */
+					  1,  /* for bpf_timer */
+};
+
+enum bpf_off_type {
 	BPF_KPTR_UNREF,
 	BPF_KPTR_REF,
+	BPF_LIST_HEAD,
 };
 
 struct bpf_map_value_off_desc {
 	u32 offset;
-	enum bpf_kptr_type type;
-	struct {
-		struct btf *btf;
-		struct module *module;
-		btf_dtor_kfunc_t dtor;
-		u32 btf_id;
-	} kptr;
+	enum bpf_off_type type;
+	union {
+		struct {
+			struct btf *btf;
+			struct module *module;
+			btf_dtor_kfunc_t dtor;
+			u32 btf_id;
+		} kptr; /* for BPF_KPTR_{UNREF,REF} */
+		struct {
+			struct btf *btf;
+			u32 value_type_id;
+			u32 list_node_off;
+		} list_head; /* for BPF_LIST_HEAD */
+	};
 };
 
 struct bpf_map_value_off {
@@ -215,6 +231,7 @@ struct bpf_map {
 	u32 map_flags;
 	int spin_lock_off; /* >=0 valid offset, <0 error */
 	struct bpf_map_value_off *kptr_off_tab;
+	struct bpf_map_value_off *list_head_off_tab;
 	int timer_off; /* >=0 valid offset, <0 error */
 	u32 id;
 	int numa_node;
@@ -265,6 +282,11 @@ static inline bool map_value_has_kptrs(const struct bpf_map *map)
 	return !IS_ERR_OR_NULL(map->kptr_off_tab);
 }
 
+static inline bool map_value_has_list_heads(const struct bpf_map *map)
+{
+	return !IS_ERR_OR_NULL(map->list_head_off_tab);
+}
+
 static inline void check_and_init_map_value(struct bpf_map *map, void *dst)
 {
 	if (unlikely(map_value_has_spin_lock(map)))
@@ -278,6 +300,13 @@ static inline void check_and_init_map_value(struct bpf_map *map, void *dst)
 		for (i = 0; i < tab->nr_off; i++)
 			*(u64 *)(dst + tab->off[i].offset) = 0;
 	}
+	if (unlikely(map_value_has_list_heads(map))) {
+		struct bpf_map_value_off *tab = map->list_head_off_tab;
+		int i;
+
+		for (i = 0; i < tab->nr_off; i++)
+			memset(dst + tab->off[i].offset, 0, sizeof(struct list_head));
+	}
 }
 
 /* memcpy that is used with 8-byte aligned pointers, power-of-8 size and
@@ -1676,6 +1705,11 @@ struct bpf_map_value_off *bpf_map_copy_kptr_off_tab(const struct bpf_map *map);
 bool bpf_map_equal_kptr_off_tab(const struct bpf_map *map_a, const struct bpf_map *map_b);
 void bpf_map_free_kptrs(struct bpf_map *map, void *map_value);
 
+struct bpf_map_value_off_desc *bpf_map_list_head_off_contains(struct bpf_map *map, u32 offset);
+void bpf_map_free_list_head_off_tab(struct bpf_map *map);
+struct bpf_map_value_off *bpf_map_copy_list_head_off_tab(const struct bpf_map *map);
+bool bpf_map_equal_list_head_off_tab(const struct bpf_map *map_a, const struct bpf_map *map_b);
+
 struct bpf_map *bpf_map_get(u32 ufd);
 struct bpf_map *bpf_map_get_with_uref(u32 ufd);
 struct bpf_map *__bpf_map_get(struct fd f);
include/linux/btf.h +2 -0
index 8062f9da7c40..9b62b8b2117e 100644
@@ -156,6 +156,8 @@ int btf_find_spin_lock(const struct btf *btf, const struct btf_type *t);
 int btf_find_timer(const struct btf *btf, const struct btf_type *t);
 struct bpf_map_value_off *btf_parse_kptrs(const struct btf *btf,
 					  const struct btf_type *t);
+struct bpf_map_value_off *btf_parse_list_heads(struct btf *btf,
+					       const struct btf_type *t);
 bool btf_type_is_void(const struct btf_type *t);
 s32 btf_find_by_name_kind(const struct btf *btf, const char *name, u8 kind);
 const struct btf_type *btf_type_skip_modifiers(const struct btf *btf,
kernel/bpf/arraymap.c +2 -0
index 832b2659e96e..c7263ee3a35f 100644
@@ -423,6 +423,8 @@ static void array_map_free(struct bpf_map *map)
 	struct bpf_array *array = container_of(map, struct bpf_array, map);
 	int i;
 
+	bpf_map_free_list_head_off_tab(map);
+
 	if (map_value_has_kptrs(map)) {
 		if (array->map.map_type == BPF_MAP_TYPE_PERCPU_ARRAY) {
 			for (i = 0; i < array->map.max_entries; i++) {
kernel/bpf/bpf_local_storage.c +1 -0
index 58cb0c179097..b5ccd76026b6 100644
@@ -616,6 +616,7 @@ void bpf_local_storage_map_free(struct bpf_local_storage_map *smap,
 		rcu_barrier();
 		bpf_map_free_kptr_off_tab(&smap->map);
 	}
+	bpf_map_free_list_head_off_tab(&smap->map);
 	kvfree(smap->buckets);
 	bpf_map_area_free(smap);
 }
kernel/bpf/btf.c +165 -8

241 lines not on this page · raw message

kernel/bpf/hashtab.c +1 -0
index bb3f8a63c221..270e0ecf4ba3 100644
@@ -1518,6 +1518,7 @@ static void htab_map_free(struct bpf_map *map)
 		prealloc_destroy(htab);
 	}
 
+	bpf_map_free_list_head_off_tab(map);
 	bpf_map_free_kptr_off_tab(map);
 	free_percpu(htab->extra_elems);
 	bpf_map_area_free(htab->buckets);
kernel/bpf/map_in_map.c +4 -1
index 135205d0d560..ced2559129ab 100644
@@ -53,6 +53,7 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd)
 	inner_map_meta->spin_lock_off = inner_map->spin_lock_off;
 	inner_map_meta->timer_off = inner_map->timer_off;
 	inner_map_meta->kptr_off_tab = bpf_map_copy_kptr_off_tab(inner_map);
+	inner_map_meta->list_head_off_tab = bpf_map_copy_list_head_off_tab(inner_map);
 	if (inner_map->btf) {
 		btf_get(inner_map->btf);
 		inner_map_meta->btf = inner_map->btf;
@@ -72,6 +73,7 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd)
 
 void bpf_map_meta_free(struct bpf_map *map_meta)
 {
+	bpf_map_free_list_head_off_tab(map_meta);
 	bpf_map_free_kptr_off_tab(map_meta);
 	btf_put(map_meta->btf);
 	kfree(map_meta);
@@ -86,7 +88,8 @@ bool bpf_map_meta_equal(const struct bpf_map *meta0,
 		meta0->value_size == meta1->value_size &&
 		meta0->timer_off == meta1->timer_off &&
 		meta0->map_flags == meta1->map_flags &&
-		bpf_map_equal_kptr_off_tab(meta0, meta1);
+		bpf_map_equal_kptr_off_tab(meta0, meta1) &&
+		bpf_map_equal_list_head_off_tab(meta0, meta1);
 }
 
 void *bpf_map_fd_get_ptr(struct bpf_map *map,
kernel/bpf/syscall.c +112 -19

226 lines not on this page · raw message

kernel/bpf/verifier.c +21 -0
index 571790ac58d4..ab91e5ca7e41 100644
@@ -3879,6 +3879,20 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno,
 			}
 		}
 	}
+	if (map_value_has_list_heads(map)) {
+		struct bpf_map_value_off *tab = map->list_head_off_tab;
+		int i;
+
+		for (i = 0; i < tab->nr_off; i++) {
+			u32 p = tab->off[i].offset;
+
+			if (reg->smin_value + off < p + sizeof(struct bpf_list_head) &&
+			    p < reg->umax_value + off + size) {
+				verbose(env, "bpf_list_head cannot be accessed directly by load/store\n");
+				return -EACCES;
+			}
+		}
+	}
 	return err;
 }
 
@@ -13165,6 +13179,13 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env,
 		}
 	}
 
+	if (map_value_has_list_heads(map)) {
+		if (is_tracing_prog_type(prog_type)) {
+			verbose(env, "tracing progs cannot use bpf_list_head yet\n");
+			return -EINVAL;
+		}
+	}
+
 	if ((bpf_prog_is_dev_bound(prog->aux) || bpf_map_is_dev_bound(map)) &&
 	    !bpf_offload_prog_map_match(prog, map)) {
 		verbose(env, "offload device mismatch between prog and map\n");
tools/testing/selftests/bpf/bpf_experimental.h new file +21 -0
new file mode 100644
index 000000000000..ea1b3b1839d1
@@ -0,0 +1,21 @@
+#ifndef __KERNEL__
+
+#include <bpf/bpf_tracing.h>
+#include <bpf/bpf_helpers.h>
+
+#else
+
+struct bpf_list_head {
+	__u64 __a;
+	__u64 __b;
+} __attribute__((aligned(8)));
+
+struct bpf_list_node {
+	__u64 __a;
+	__u64 __b;
+} __attribute__((aligned(8)));
+
+#endif
+
+#ifndef __KERNEL__
+#endif
Signature · 1 line
2.34.1
#20 lore raw

Re: [PATCH RFC bpf-next v1 13/32] bpf: Introduce bpf_list_head support for BPF maps

Quoted 8 lines · Kumar Kartikeya Dwivedi

Add the basic support on the map side to parse, recognize, verify, and build metadata table for a new special field of the type struct bpf_list_head. To parameterize the bpf_list_head for a certain value type and the list_node member it will accept in that value type, we use BTF declaration tags.

The definition of bpf_list_head in a map value will be done as follows:

Quoted 8 lines
struct foo {
	int data;
	struct bpf_list_node list;
};
struct map_value {
	struct bpf_list_head list __contains(struct, foo, node);
};

kptrs are only for structs.
So I would drop explicit 1st argument which is going to be 'struct' for foreseeable future and leave it as:
struct bpf_list_head list __contains(foo, node);

There is typo s/list;/node;/ in struct foo, right?

Quoted 109 lines

Then, the bpf_list_head only allows adding to the list using the bpf_list_node 'list' for the type struct foo.

The 'contains' annotation is a BTF declaration tag composed of four parts, "contains:kind:name:node" where the kind and name is then used to look up the type in the map BTF. The node defines name of the member in this type that has the type struct bpf_list_node, which is actually used for linking into the linked list.

This allows building intrusive linked lists in BPF, using container_of to obtain pointer to entry, while being completely type safe from the perspective of the verifier. The verifier knows exactly the type of the nodes, and knows that list helpers return that type at some fixed offset where the bpf_list_node member used for this list exists. The verifier also uses this information to disallow adding types that are not accepted by a certain list.

For now, no elements can be added to such lists. Support for that is coming in future patches, hence draining and freeing items is left out for now, and just freeing the list_head_off_tab is done, since it is still built and populated when bpf_list_head is specified in the map value.

---
 include/linux/bpf.h                           |  64 +++++--
 include/linux/btf.h                           |   2 +
 kernel/bpf/arraymap.c                         |   2 +
 kernel/bpf/bpf_local_storage.c                |   1 +
 kernel/bpf/btf.c                              | 173 +++++++++++++++++-
 kernel/bpf/hashtab.c                          |   1 +
 kernel/bpf/map_in_map.c                       |   5 +-
 kernel/bpf/syscall.c                          | 131 +++++++++++--
 kernel/bpf/verifier.c                         |  21 +++
 .../testing/selftests/bpf/bpf_experimental.h  |  21 +++
 10 files changed, 378 insertions(+), 43 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/bpf_experimental.h
include/linux/bpf.h +31 -15
index d4e6bf789c02..35c2e9caeb98 100644
@@ -28,6 +28,9 @@
 #include <linux/btf.h>
 #include <linux/rcupdate_trace.h>
 
+/* Experimental BPF APIs header for type definitions */
+#include "../../../tools/testing/selftests/bpf/bpf_experimental.h"
+
 struct bpf_verifier_env;
 struct bpf_verifier_log;
 struct perf_event;
@@ -164,27 +167,40 @@ struct bpf_map_ops {
 };
 
 enum {
-	/* Support at most 8 pointers in a BPF map value */
-	BPF_MAP_VALUE_OFF_MAX = 8,
-	BPF_MAP_OFF_ARR_MAX   = BPF_MAP_VALUE_OFF_MAX +
-				1 + /* for bpf_spin_lock */
-				1,  /* for bpf_timer */
-};
-
-enum bpf_kptr_type {
+	/* Support at most 8 offsets in a table */
+	BPF_MAP_VALUE_OFF_MAX		= 8,
+	/* Support at most 8 pointer in a BPF map value */
+	BPF_MAP_VALUE_KPTR_MAX		= BPF_MAP_VALUE_OFF_MAX,
+	/* Support at most 8 list_head in a BPF map value */
+	BPF_MAP_VALUE_LIST_HEAD_MAX	= BPF_MAP_VALUE_OFF_MAX,
+	BPF_MAP_OFF_ARR_MAX		= BPF_MAP_VALUE_KPTR_MAX +
+					  BPF_MAP_VALUE_LIST_HEAD_MAX +
+					  1 + /* for bpf_spin_lock */
+					  1,  /* for bpf_timer */
+};
+
+enum bpf_off_type {
 	BPF_KPTR_UNREF,
 	BPF_KPTR_REF,
+	BPF_LIST_HEAD,
 };
 
 struct bpf_map_value_off_desc {
 	u32 offset;
-	enum bpf_kptr_type type;
-	struct {
-		struct btf *btf;
-		struct module *module;
-		btf_dtor_kfunc_t dtor;
-		u32 btf_id;
-	} kptr;
+	enum bpf_off_type type;
+	union {
+		struct {
+			struct btf *btf;
+			struct module *module;
+			btf_dtor_kfunc_t dtor;
+			u32 btf_id;
+		} kptr; /* for BPF_KPTR_{UNREF,REF} */
+		struct {
+			struct btf *btf;
+			u32 value_type_id;
+			u32 list_node_off;
+		} list_head; /* for BPF_LIST_HEAD */
+	};
 };
 
 struct bpf_map_value_off {

include/linux/bpf.h struct bpf_map { · line 234

Quoted 4 lines
include/linux/bpf.h +1 -0
@@ -215,6 +231,7 @@ struct bpf_map {
 	u32 map_flags;
 	int spin_lock_off; /* >=0 valid offset, <0 error */
 	struct bpf_map_value_off *kptr_off_tab;
+	struct bpf_map_value_off *list_head_off_tab;

The union in bpf_map_value_off_desc prompts the question why separate array is needed.
Sorting gets uglier.

Quoted 16 lines
include/linux/bpf.h +5 -0
 	int timer_off; /* >=0 valid offset, <0 error */
 	u32 id;
 	int numa_node;
@@ -265,6 +282,11 @@ static inline bool map_value_has_kptrs(const struct bpf_map *map)
 	return !IS_ERR_OR_NULL(map->kptr_off_tab);
 }
 
+static inline bool map_value_has_list_heads(const struct bpf_map *map)
+{
+	return !IS_ERR_OR_NULL(map->list_head_off_tab);
+}
+
 static inline void check_and_init_map_value(struct bpf_map *map, void *dst)
 {
 	if (unlikely(map_value_has_spin_lock(map)))

include/linux/bpf.h static inline void check_and_init_map_value(struct bpf_map *map, void *dst) · line 309

Quoted 10 lines
include/linux/bpf.h +7 -0
@@ -278,6 +300,13 @@ static inline void check_and_init_map_value(struct bpf_map *map, void *dst)
 		for (i = 0; i < tab->nr_off; i++)
 			*(u64 *)(dst + tab->off[i].offset) = 0;
 	}
+	if (unlikely(map_value_has_list_heads(map))) {
+		struct bpf_map_value_off *tab = map->list_head_off_tab;
+		int i;
+
+		for (i = 0; i < tab->nr_off; i++)
+			memset(dst + tab->off[i].offset, 0, sizeof(struct list_head));
+	}

Do we really need to distinguish map_value_has_kptrs vs map_value_has_list_heads ? Can they be generalized?
rb_root will be next.
that would be yet another array and another 'if'-s everywhere?
And then another special pseudo-map type that will cause a bunch of copy-paste again? Maybe it's inevitable.
Trying to brainstorm.

Quoted 234 lines
include/linux/bpf.h +5 -0
 }
 
 /* memcpy that is used with 8-byte aligned pointers, power-of-8 size and
@@ -1676,6 +1705,11 @@ struct bpf_map_value_off *bpf_map_copy_kptr_off_tab(const struct bpf_map *map);
 bool bpf_map_equal_kptr_off_tab(const struct bpf_map *map_a, const struct bpf_map *map_b);
 void bpf_map_free_kptrs(struct bpf_map *map, void *map_value);
 
+struct bpf_map_value_off_desc *bpf_map_list_head_off_contains(struct bpf_map *map, u32 offset);
+void bpf_map_free_list_head_off_tab(struct bpf_map *map);
+struct bpf_map_value_off *bpf_map_copy_list_head_off_tab(const struct bpf_map *map);
+bool bpf_map_equal_list_head_off_tab(const struct bpf_map *map_a, const struct bpf_map *map_b);
+
 struct bpf_map *bpf_map_get(u32 ufd);
 struct bpf_map *bpf_map_get_with_uref(u32 ufd);
 struct bpf_map *__bpf_map_get(struct fd f);
include/linux/btf.h +2 -0
index 8062f9da7c40..9b62b8b2117e 100644
@@ -156,6 +156,8 @@ int btf_find_spin_lock(const struct btf *btf, const struct btf_type *t);
 int btf_find_timer(const struct btf *btf, const struct btf_type *t);
 struct bpf_map_value_off *btf_parse_kptrs(const struct btf *btf,
 					  const struct btf_type *t);
+struct bpf_map_value_off *btf_parse_list_heads(struct btf *btf,
+					       const struct btf_type *t);
 bool btf_type_is_void(const struct btf_type *t);
 s32 btf_find_by_name_kind(const struct btf *btf, const char *name, u8 kind);
 const struct btf_type *btf_type_skip_modifiers(const struct btf *btf,
kernel/bpf/arraymap.c +2 -0
index 832b2659e96e..c7263ee3a35f 100644
@@ -423,6 +423,8 @@ static void array_map_free(struct bpf_map *map)
 	struct bpf_array *array = container_of(map, struct bpf_array, map);
 	int i;
 
+	bpf_map_free_list_head_off_tab(map);
+
 	if (map_value_has_kptrs(map)) {
 		if (array->map.map_type == BPF_MAP_TYPE_PERCPU_ARRAY) {
 			for (i = 0; i < array->map.max_entries; i++) {
kernel/bpf/bpf_local_storage.c +1 -0
index 58cb0c179097..b5ccd76026b6 100644
@@ -616,6 +616,7 @@ void bpf_local_storage_map_free(struct bpf_local_storage_map *smap,
 		rcu_barrier();
 		bpf_map_free_kptr_off_tab(&smap->map);
 	}
+	bpf_map_free_list_head_off_tab(&smap->map);
 	kvfree(smap->buckets);
 	bpf_map_area_free(smap);
 }
kernel/bpf/btf.c +96 -8
index 6740c3ade8f1..0fb045be3837 100644
@@ -3185,6 +3185,7 @@ enum btf_field_type {
 	BTF_FIELD_SPIN_LOCK,
 	BTF_FIELD_TIMER,
 	BTF_FIELD_KPTR,
+	BTF_FIELD_LIST_HEAD,
 };
 
 enum {
@@ -3193,9 +3194,17 @@ enum {
 };
 
 struct btf_field_info {
-	u32 type_id;
 	u32 off;
-	enum bpf_kptr_type type;
+	union {
+		struct {
+			u32 type_id;
+			enum bpf_off_type type;
+		} kptr;
+		struct {
+			u32 value_type_id;
+			const char *node_name;
+		} list_head;
+	};
 };
 
 static int btf_find_struct(const struct btf *btf, const struct btf_type *t,
@@ -3212,7 +3221,7 @@ static int btf_find_struct(const struct btf *btf, const struct btf_type *t,
 static int btf_find_kptr(const struct btf *btf, const struct btf_type *t,
 			 u32 off, int sz, struct btf_field_info *info)
 {
-	enum bpf_kptr_type type;
+	enum bpf_off_type type;
 	u32 res_id;
 
 	/* Permit modifiers on the pointer itself */
@@ -3241,9 +3250,71 @@ static int btf_find_kptr(const struct btf *btf, const struct btf_type *t,
 	if (!__btf_type_is_struct(t))
 		return -EINVAL;
 
-	info->type_id = res_id;
 	info->off = off;
-	info->type = type;
+	info->kptr.type_id = res_id;
+	info->kptr.type = type;
+	return BTF_FIELD_FOUND;
+}
+
+static const char *btf_find_decl_tag_value(const struct btf *btf,
+					   const struct btf_type *pt,
+					   int comp_idx, const char *tag_key)
+{
+	int i;
+
+	for (i = 1; i < btf_nr_types(btf); i++) {
+		const struct btf_type *t = btf_type_by_id(btf, i);
+		int len = strlen(tag_key);
+
+		if (!btf_type_is_decl_tag(t))
+			continue;
+		/* TODO: Instead of btf_type pt, it would be much better if we had BTF
+		 * ID of the map value type. This would avoid btf_type_by_id call here.
+		 */
+		if (pt != btf_type_by_id(btf, t->type) ||
+		    btf_type_decl_tag(t)->component_idx != comp_idx)
+			continue;
+		if (strncmp(__btf_name_by_offset(btf, t->name_off), tag_key, len))
+			continue;
+		return __btf_name_by_offset(btf, t->name_off) + len;
+	}
+	return NULL;
+}
+
+static int btf_find_list_head(const struct btf *btf, const struct btf_type *pt,
+			      int comp_idx, const struct btf_type *t,
+			      u32 off, int sz, struct btf_field_info *info)
+{
+	const char *value_type;
+	const char *list_node;
+	s32 id;
+
+	if (!__btf_type_is_struct(t))
+		return BTF_FIELD_IGNORE;
+	if (t->size != sz)
+		return BTF_FIELD_IGNORE;
+	value_type = btf_find_decl_tag_value(btf, pt, comp_idx, "contains:");
+	if (!value_type)
+		return -EINVAL;
+	if (strncmp(value_type, "struct:", sizeof("struct:") - 1))
+		return -EINVAL;
+	value_type += sizeof("struct:") - 1;
+	list_node = strstr(value_type, ":");
+	if (!list_node)
+		return -EINVAL;
+	value_type = kstrndup(value_type, list_node - value_type, GFP_ATOMIC);
+	if (!value_type)
+		return -ENOMEM;
+	id = btf_find_by_name_kind(btf, value_type, BTF_KIND_STRUCT);
+	kfree(value_type);
+	if (id < 0)
+		return id;
+	list_node++;
+	if (str_is_empty(list_node))
+		return -EINVAL;
+	info->off = off;
+	info->list_head.value_type_id = id;
+	info->list_head.node_name = list_node;
 	return BTF_FIELD_FOUND;
 }
 
@@ -3286,6 +3357,12 @@ static int btf_find_struct_field(const struct btf *btf, const struct btf_type *t
 			if (ret < 0)
 				return ret;
 			break;
+		case BTF_FIELD_LIST_HEAD:
+			ret = btf_find_list_head(btf, t, i, member_type, off, sz,
+						 idx < info_cnt ? &info[idx] : &tmp);
+			if (ret < 0)
+				return ret;
+			break;
 		default:
 			return -EFAULT;
 		}
@@ -3336,6 +3413,12 @@ static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t,
 			if (ret < 0)
 				return ret;
 			break;
+		case BTF_FIELD_LIST_HEAD:
+			ret = btf_find_list_head(btf, var, -1, var_type, off, sz,
+						 idx < info_cnt ? &info[idx] : &tmp);
+			if (ret < 0)
+				return ret;
+			break;
 		default:
 			return -EFAULT;
 		}
@@ -3372,6 +3455,11 @@ static int btf_find_field(const struct btf *btf, const struct btf_type *t,
 		sz = sizeof(u64);
 		align = 8;
 		break;
+	case BTF_FIELD_LIST_HEAD:
+		name = "bpf_list_head";
+		sz = sizeof(struct bpf_list_head);
+		align = __alignof__(struct bpf_list_head);
+		break;
 	default:
 		return -EFAULT;
 	}
@@ -3440,7 +3528,7 @@ struct bpf_map_value_off *btf_parse_kptrs(const struct btf *btf,
 		/* Find type in map BTF, and use it to look up the matching type
 		 * in vmlinux or module BTFs, by name and kind.
 		 */
-		t = btf_type_by_id(btf, info_arr[i].type_id);
+		t = btf_type_by_id(btf, info_arr[i].kptr.type_id);
 		id = bpf_find_btf_id(__btf_name_by_offset(btf, t->name_off), BTF_INFO_KIND(t->info),
 				     &kernel_btf);
 		if (id < 0) {
@@ -3451,7 +3539,7 @@ struct bpf_map_value_off *btf_parse_kptrs(const struct btf *btf,
 		/* Find and stash the function pointer for the destruction function that
 		 * needs to be eventually invoked from the map free path.
 		 */
-		if (info_arr[i].type == BPF_KPTR_REF) {
+		if (info_arr[i].kptr.type == BPF_KPTR_REF) {
 			const struct btf_type *dtor_func;
 			const char *dtor_func_name;
 			unsigned long addr;
@@ -3494,7 +3582,7 @@ struct bpf_map_value_off *btf_parse_kptrs(const struct btf *btf,
 		}
 
 		tab->off[i].offset = info_arr[i].off;
-		tab->off[i].type = info_arr[i].type;
+		tab->off[i].type = info_arr[i].kptr.type;
 		tab->off[i].kptr.btf_id = id;
 		tab->off[i].kptr.btf = kernel_btf;
 		tab->off[i].kptr.module = mod;

kernel/bpf/btf.c struct bpf_map_value_off *btf_parse_kptrs(const struct btf *btf, · line 3612

Quoted 10 lines
kernel/bpf/btf.c +7 -0
@@ -3515,6 +3603,75 @@ struct bpf_map_value_off *btf_parse_kptrs(const struct btf *btf,
 	return ERR_PTR(ret);
 }
 
+struct bpf_map_value_off *btf_parse_list_heads(struct btf *btf, const struct btf_type *t)
+{
+	struct btf_field_info info_arr[BPF_MAP_VALUE_OFF_MAX];
+	struct bpf_map_value_off *tab;
+	int ret, i, nr_off;
+
+	ret = btf_find_field(btf, t, BTF_FIELD_LIST_HEAD, info_arr, ARRAY_SIZE(info_arr));

Like if search for both LIST_HEAD and KPTR here to know the size.

Quoted 5 lines
kernel/bpf/btf.c +5 -0
+	if (ret < 0)
+		return ERR_PTR(ret);
+	if (!ret)
+		return NULL;
+

kernel/bpf/btf.c struct bpf_map_value_off *btf_parse_kptrs(const struct btf *btf, · line 3627

Quoted 10 lines
kernel/bpf/btf.c +10 -0
+	nr_off = ret;
+	tab = kzalloc(offsetof(struct bpf_map_value_off, off[nr_off]), GFP_KERNEL | __GFP_NOWARN);
+	if (!tab)
+		return ERR_PTR(-ENOMEM);
+
+	for (i = 0; i < nr_off; i++) {
+		const struct btf_type *t, *n = NULL;
+		const struct btf_member *member;
+		u32 offset;
+		int j;

and here we can process both, since field_info has type.

Quoted 21 lines
kernel/bpf/btf.c +21 -0
+
+		t = btf_type_by_id(btf, info_arr[i].list_head.value_type_id);
+		/* We've already checked that value_type_id is a struct type. We
+		 * just need to figure out the offset of the list_node, and
+		 * verify its type.
+		 */
+		ret = -EINVAL;
+		for_each_member(j, t, member) {
+			if (strcmp(info_arr[i].list_head.node_name, __btf_name_by_offset(btf, member->name_off)))
+				continue;
+			/* Invalid BTF, two members with same name */
+			if (n) {
+				/* We also need to btf_put for the current iteration! */
+				i++;
+				goto end;
+			}
+			n = btf_type_by_id(btf, member->type);
+			if (!__btf_type_is_struct(n))
+				goto end;
+			if (strcmp("bpf_list_node", __btf_name_by_offset(btf, n->name_off)))
+				goto end;

kernel/bpf/btf.c struct bpf_map_value_off *btf_parse_kptrs(const struct btf *btf, · line 3658

Quoted 10 lines
kernel/bpf/btf.c +10 -0
+			offset = __btf_member_bit_offset(n, member);
+			if (offset % 8)
+				goto end;
+			offset /= 8;
+			if (offset % __alignof__(struct bpf_list_node))
+				goto end;
+
+			tab->off[i].offset = info_arr[i].off;
+			tab->off[i].type = BPF_LIST_HEAD;
+			btf_get(btf);

Do we need to btf_get? The btf should be pinned already and not going to be released until prog ends.

Quoted 371 lines
kernel/bpf/btf.c +16 -0
+			tab->off[i].list_head.btf = btf;
+			tab->off[i].list_head.value_type_id = info_arr[i].list_head.value_type_id;
+			tab->off[i].list_head.list_node_off = offset;
+		}
+		if (!n)
+			goto end;
+	}
+	tab->nr_off = nr_off;
+	return tab;
+end:
+	while (i--)
+		btf_put(tab->off[i].list_head.btf);
+	kfree(tab);
+	return ERR_PTR(ret);
+}
+
 static void __btf_struct_show(const struct btf *btf, const struct btf_type *t,
 			      u32 type_id, void *data, u8 bits_offset,
 			      struct btf_show *show)
kernel/bpf/hashtab.c +1 -0
index bb3f8a63c221..270e0ecf4ba3 100644
@@ -1518,6 +1518,7 @@ static void htab_map_free(struct bpf_map *map)
 		prealloc_destroy(htab);
 	}
 
+	bpf_map_free_list_head_off_tab(map);
 	bpf_map_free_kptr_off_tab(map);
 	free_percpu(htab->extra_elems);
 	bpf_map_area_free(htab->buckets);
kernel/bpf/map_in_map.c +4 -1
index 135205d0d560..ced2559129ab 100644
@@ -53,6 +53,7 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd)
 	inner_map_meta->spin_lock_off = inner_map->spin_lock_off;
 	inner_map_meta->timer_off = inner_map->timer_off;
 	inner_map_meta->kptr_off_tab = bpf_map_copy_kptr_off_tab(inner_map);
+	inner_map_meta->list_head_off_tab = bpf_map_copy_list_head_off_tab(inner_map);
 	if (inner_map->btf) {
 		btf_get(inner_map->btf);
 		inner_map_meta->btf = inner_map->btf;
@@ -72,6 +73,7 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd)
 
 void bpf_map_meta_free(struct bpf_map *map_meta)
 {
+	bpf_map_free_list_head_off_tab(map_meta);
 	bpf_map_free_kptr_off_tab(map_meta);
 	btf_put(map_meta->btf);
 	kfree(map_meta);
@@ -86,7 +88,8 @@ bool bpf_map_meta_equal(const struct bpf_map *meta0,
 		meta0->value_size == meta1->value_size &&
 		meta0->timer_off == meta1->timer_off &&
 		meta0->map_flags == meta1->map_flags &&
-		bpf_map_equal_kptr_off_tab(meta0, meta1);
+		bpf_map_equal_kptr_off_tab(meta0, meta1) &&
+		bpf_map_equal_list_head_off_tab(meta0, meta1);
 }
 
 void *bpf_map_fd_get_ptr(struct bpf_map *map,
kernel/bpf/syscall.c +112 -19
index 0311acca19f6..e1749e0d2143 100644
@@ -495,7 +495,7 @@ static void bpf_map_release_memcg(struct bpf_map *map)
 }
 #endif
 
-static int bpf_map_kptr_off_cmp(const void *a, const void *b)
+static int bpf_map_off_cmp(const void *a, const void *b)
 {
 	const struct bpf_map_value_off_desc *off_desc1 = a, *off_desc2 = b;
 
@@ -506,18 +506,22 @@ static int bpf_map_kptr_off_cmp(const void *a, const void *b)
 	return 0;
 }
 
-struct bpf_map_value_off_desc *bpf_map_kptr_off_contains(struct bpf_map *map, u32 offset)
+static struct bpf_map_value_off_desc *
+__bpf_map_off_contains(struct bpf_map_value_off *off_tab, u32 offset)
 {
 	/* Since members are iterated in btf_find_field in increasing order,
-	 * offsets appended to kptr_off_tab are in increasing order, so we can
+	 * offsets appended to an off_tab are in increasing order, so we can
 	 * do bsearch to find exact match.
 	 */
-	struct bpf_map_value_off *tab;
+	return bsearch(&offset, off_tab->off, off_tab->nr_off, sizeof(off_tab->off[0]),
+		       bpf_map_off_cmp);
+}
 
+struct bpf_map_value_off_desc *bpf_map_kptr_off_contains(struct bpf_map *map, u32 offset)
+{
 	if (!map_value_has_kptrs(map))
 		return NULL;
-	tab = map->kptr_off_tab;
-	return bsearch(&offset, tab->off, tab->nr_off, sizeof(tab->off[0]), bpf_map_kptr_off_cmp);
+	return __bpf_map_off_contains(map->kptr_off_tab, offset);
 }
 
 void bpf_map_free_kptr_off_tab(struct bpf_map *map)
@@ -563,15 +567,15 @@ struct bpf_map_value_off *bpf_map_copy_kptr_off_tab(const struct bpf_map *map)
 	return new_tab;
 }
 
-bool bpf_map_equal_kptr_off_tab(const struct bpf_map *map_a, const struct bpf_map *map_b)
+static bool __bpf_map_equal_off_tab(const struct bpf_map_value_off *tab_a,
+				    const struct bpf_map_value_off *tab_b,
+				    bool has_a, bool has_b)
 {
-	struct bpf_map_value_off *tab_a = map_a->kptr_off_tab, *tab_b = map_b->kptr_off_tab;
-	bool a_has_kptr = map_value_has_kptrs(map_a), b_has_kptr = map_value_has_kptrs(map_b);
 	int size;
 
-	if (!a_has_kptr && !b_has_kptr)
+	if (!has_a && !has_b)
 		return true;
-	if (a_has_kptr != b_has_kptr)
+	if (has_a != has_b)
 		return false;
 	if (tab_a->nr_off != tab_b->nr_off)
 		return false;
@@ -579,6 +583,13 @@ bool bpf_map_equal_kptr_off_tab(const struct bpf_map *map_a, const struct bpf_ma
 	return !memcmp(tab_a, tab_b, size);
 }
 
+bool bpf_map_equal_kptr_off_tab(const struct bpf_map *map_a, const struct bpf_map *map_b)
+{
+	return __bpf_map_equal_off_tab(map_a->kptr_off_tab, map_b->kptr_off_tab,
+				       map_value_has_kptrs(map_a),
+				       map_value_has_kptrs(map_b));
+}
+
 /* Caller must ensure map_value_has_kptrs is true. Note that this function can
  * be called on a map value while the map_value is visible to BPF programs, as
  * it ensures the correct synchronization, and we already enforce the same using
@@ -606,6 +617,50 @@ void bpf_map_free_kptrs(struct bpf_map *map, void *map_value)
 	}
 }
 
+struct bpf_map_value_off_desc *bpf_map_list_head_off_contains(struct bpf_map *map, u32 offset)
+{
+	if (!map_value_has_list_heads(map))
+		return NULL;
+	return __bpf_map_off_contains(map->list_head_off_tab, offset);
+}
+
+void bpf_map_free_list_head_off_tab(struct bpf_map *map)
+{
+	struct bpf_map_value_off *tab = map->list_head_off_tab;
+	int i;
+
+	if (!map_value_has_list_heads(map))
+		return;
+	for (i = 0; i < tab->nr_off; i++)
+		btf_put(tab->off[i].list_head.btf);
+	kfree(tab);
+	map->list_head_off_tab = NULL;
+}
+
+struct bpf_map_value_off *bpf_map_copy_list_head_off_tab(const struct bpf_map *map)
+{
+	struct bpf_map_value_off *tab = map->list_head_off_tab, *new_tab;
+	int size, i;
+
+	if (!map_value_has_list_heads(map))
+		return ERR_PTR(-ENOENT);
+	size = offsetof(struct bpf_map_value_off, off[tab->nr_off]);
+	new_tab = kmemdup(tab, size, GFP_KERNEL | __GFP_NOWARN);
+	if (!new_tab)
+		return ERR_PTR(-ENOMEM);
+	/* Do a deep copy of the list_head_off_tab */
+	for (i = 0; i < tab->nr_off; i++)
+		btf_get(tab->off[i].list_head.btf);
+	return new_tab;
+}
+
+bool bpf_map_equal_list_head_off_tab(const struct bpf_map *map_a, const struct bpf_map *map_b)
+{
+	return __bpf_map_equal_off_tab(map_a->list_head_off_tab, map_b->list_head_off_tab,
+				       map_value_has_list_heads(map_a),
+				       map_value_has_list_heads(map_b));
+}
+
 /* called from workqueue */
 static void bpf_map_free_deferred(struct work_struct *work)
 {
@@ -776,7 +831,8 @@ static int bpf_map_mmap(struct file *filp, struct vm_area_struct *vma)
 	int err;
 
 	if (!map->ops->map_mmap || map_value_has_spin_lock(map) ||
-	    map_value_has_timer(map) || map_value_has_kptrs(map))
+	    map_value_has_timer(map) || map_value_has_kptrs(map) ||
+	    map_value_has_list_heads(map))
 		return -ENOTSUPP;
 
 	if (!(vma->vm_flags & VM_SHARED))
@@ -931,13 +987,14 @@ static void map_off_arr_swap(void *_a, void *_b, int size, const void *priv)
 
 static int bpf_map_alloc_off_arr(struct bpf_map *map)
 {
+	bool has_list_heads = map_value_has_list_heads(map);
 	bool has_spin_lock = map_value_has_spin_lock(map);
 	bool has_timer = map_value_has_timer(map);
 	bool has_kptrs = map_value_has_kptrs(map);
 	struct bpf_map_off_arr *off_arr;
 	u32 i;
 
-	if (!has_spin_lock && !has_timer && !has_kptrs) {
+	if (!has_spin_lock && !has_timer && !has_kptrs && !has_list_heads) {
 		map->off_arr = NULL;
 		return 0;
 	}
@@ -973,6 +1030,17 @@ static int bpf_map_alloc_off_arr(struct bpf_map *map)
 		}
 		off_arr->cnt += tab->nr_off;
 	}
+	if (has_list_heads) {
+		struct bpf_map_value_off *tab = map->list_head_off_tab;
+		u32 *off = &off_arr->field_off[off_arr->cnt];
+		u8 *sz = &off_arr->field_sz[off_arr->cnt];
+
+		for (i = 0; i < tab->nr_off; i++) {
+			*off++ = tab->off[i].offset;
+			*sz++ = sizeof(struct bpf_list_head);
+		}
+		off_arr->cnt += tab->nr_off;
+	}
 
 	if (off_arr->cnt == 1)
 		return 0;
@@ -1038,11 +1106,11 @@ static int map_check_btf(struct bpf_map *map, const struct btf *btf,
 	if (map_value_has_kptrs(map)) {
 		if (!bpf_capable()) {
 			ret = -EPERM;
-			goto free_map_tab;
+			goto free_map_kptr_tab;
 		}
 		if (map->map_flags & (BPF_F_RDONLY_PROG | BPF_F_WRONLY_PROG)) {
 			ret = -EACCES;
-			goto free_map_tab;
+			goto free_map_kptr_tab;
 		}
 		if (map->map_type != BPF_MAP_TYPE_HASH &&
 		    map->map_type != BPF_MAP_TYPE_PERCPU_HASH &&
@@ -1054,18 +1122,42 @@ static int map_check_btf(struct bpf_map *map, const struct btf *btf,
 		    map->map_type != BPF_MAP_TYPE_INODE_STORAGE &&
 		    map->map_type != BPF_MAP_TYPE_TASK_STORAGE) {
 			ret = -EOPNOTSUPP;
-			goto free_map_tab;
+			goto free_map_kptr_tab;
+		}
+	}
+
+	/* We need to take ref on the BTF, so pass it as non-const */
+	map->list_head_off_tab = btf_parse_list_heads((struct btf *)btf, value_type);
+	if (map_value_has_list_heads(map)) {
+		if (!bpf_capable()) {
+			ret = -EACCES;
+			goto free_map_list_head_tab;
+		}
+		if (map->map_flags & (BPF_F_RDONLY_PROG | BPF_F_WRONLY_PROG)) {
+			ret = -EACCES;
+			goto free_map_list_head_tab;
+		}
+		if (map->map_type != BPF_MAP_TYPE_HASH &&
+		    map->map_type != BPF_MAP_TYPE_LRU_HASH &&
+		    map->map_type != BPF_MAP_TYPE_ARRAY &&
+		    map->map_type != BPF_MAP_TYPE_SK_STORAGE &&
+		    map->map_type != BPF_MAP_TYPE_INODE_STORAGE &&
+		    map->map_type != BPF_MAP_TYPE_TASK_STORAGE) {
+			ret = -EOPNOTSUPP;
+			goto free_map_list_head_tab;
 		}
 	}
 
 	if (map->ops->map_check_btf) {
 		ret = map->ops->map_check_btf(map, btf, key_type, value_type);
 		if (ret < 0)
-			goto free_map_tab;
+			goto free_map_list_head_tab;
 	}
 
 	return ret;
-free_map_tab:
+free_map_list_head_tab:
+	bpf_map_free_list_head_off_tab(map);
+free_map_kptr_tab:
 	bpf_map_free_kptr_off_tab(map);
 	return ret;
 }
@@ -1889,7 +1981,8 @@ static int map_freeze(const union bpf_attr *attr)
 		return PTR_ERR(map);
 
 	if (map->map_type == BPF_MAP_TYPE_STRUCT_OPS ||
-	    map_value_has_timer(map) || map_value_has_kptrs(map)) {
+	    map_value_has_timer(map) || map_value_has_kptrs(map) ||
+	    map_value_has_list_heads(map)) {
 		fdput(f);
 		return -ENOTSUPP;
 	}
kernel/bpf/verifier.c +21 -0
index 571790ac58d4..ab91e5ca7e41 100644
@@ -3879,6 +3879,20 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno,
 			}
 		}
 	}
+	if (map_value_has_list_heads(map)) {
+		struct bpf_map_value_off *tab = map->list_head_off_tab;
+		int i;
+
+		for (i = 0; i < tab->nr_off; i++) {
+			u32 p = tab->off[i].offset;
+
+			if (reg->smin_value + off < p + sizeof(struct bpf_list_head) &&
+			    p < reg->umax_value + off + size) {
+				verbose(env, "bpf_list_head cannot be accessed directly by load/store\n");
+				return -EACCES;
+			}
+		}
+	}
 	return err;
 }
 
@@ -13165,6 +13179,13 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env,
 		}
 	}
 
+	if (map_value_has_list_heads(map)) {
+		if (is_tracing_prog_type(prog_type)) {
+			verbose(env, "tracing progs cannot use bpf_list_head yet\n");
+			return -EINVAL;
+		}
+	}
+
 	if ((bpf_prog_is_dev_bound(prog->aux) || bpf_map_is_dev_bound(map)) &&
 	    !bpf_offload_prog_map_match(prog, map)) {
 		verbose(env, "offload device mismatch between prog and map\n");
tools/testing/selftests/bpf/bpf_experimental.h new file +21 -0
new file mode 100644
index 000000000000..ea1b3b1839d1
@@ -0,0 +1,21 @@
+#ifndef __KERNEL__
+
+#include <bpf/bpf_tracing.h>
+#include <bpf/bpf_helpers.h>
+
+#else
+
+struct bpf_list_head {
+	__u64 __a;
+	__u64 __b;
+} __attribute__((aligned(8)));
+
+struct bpf_list_node {
+	__u64 __a;
+	__u64 __b;
+} __attribute__((aligned(8)));
+
+#endif
+
+#ifndef __KERNEL__
+#endif
Signature · 1 line
2.34.1
author maintainer
#21 lore raw

Re: [PATCH RFC bpf-next v1 13/32] bpf: Introduce bpf_list_head support for BPF maps

Quoted 15 lines · Alexei Starovoitov
Quoted 13 lines · Kumar Kartikeya Dwivedi

Add the basic support on the map side to parse, recognize, verify, and build metadata table for a new special field of the type struct bpf_list_head. To parameterize the bpf_list_head for a certain value type and the list_node member it will accept in that value type, we use BTF declaration tags.

The definition of bpf_list_head in a map value will be done as follows:

struct foo {
      int data;
      struct bpf_list_node list;
};
Quoted 9 lines
Quoted 3 lines
struct map_value {
      struct bpf_list_head list __contains(struct, foo, node);
};

kptrs are only for structs.
So I would drop explicit 1st argument which is going to be 'struct' for foreseeable future and leave it as:
struct bpf_list_head list __contains(foo, node);

Ok.

Quoted 1 line

There is typo s/list;/node;/ in struct foo, right?

Yes.

Quoted 108 lines
Quoted 108 lines

Then, the bpf_list_head only allows adding to the list using the bpf_list_node 'list' for the type struct foo.

The 'contains' annotation is a BTF declaration tag composed of four parts, "contains:kind:name:node" where the kind and name is then used to look up the type in the map BTF. The node defines name of the member in this type that has the type struct bpf_list_node, which is actually used for linking into the linked list.

This allows building intrusive linked lists in BPF, using container_of to obtain pointer to entry, while being completely type safe from the perspective of the verifier. The verifier knows exactly the type of the nodes, and knows that list helpers return that type at some fixed offset where the bpf_list_node member used for this list exists. The verifier also uses this information to disallow adding types that are not accepted by a certain list.

For now, no elements can be added to such lists. Support for that is coming in future patches, hence draining and freeing items is left out for now, and just freeing the list_head_off_tab is done, since it is still built and populated when bpf_list_head is specified in the map value.

---
 include/linux/bpf.h                           |  64 +++++--
 include/linux/btf.h                           |   2 +
 kernel/bpf/arraymap.c                         |   2 +
 kernel/bpf/bpf_local_storage.c                |   1 +
 kernel/bpf/btf.c                              | 173 +++++++++++++++++-
 kernel/bpf/hashtab.c                          |   1 +
 kernel/bpf/map_in_map.c                       |   5 +-
 kernel/bpf/syscall.c                          | 131 +++++++++++--
 kernel/bpf/verifier.c                         |  21 +++
 .../testing/selftests/bpf/bpf_experimental.h  |  21 +++
 10 files changed, 378 insertions(+), 43 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/bpf_experimental.h
include/linux/bpf.h +31 -15
index d4e6bf789c02..35c2e9caeb98 100644
@@ -28,6 +28,9 @@
 #include <linux/btf.h>
 #include <linux/rcupdate_trace.h>
 
+/* Experimental BPF APIs header for type definitions */
+#include "../../../tools/testing/selftests/bpf/bpf_experimental.h"
+
 struct bpf_verifier_env;
 struct bpf_verifier_log;
 struct perf_event;
@@ -164,27 +167,40 @@ struct bpf_map_ops {
 };
 
 enum {
-     /* Support at most 8 pointers in a BPF map value */
-     BPF_MAP_VALUE_OFF_MAX = 8,
-     BPF_MAP_OFF_ARR_MAX   = BPF_MAP_VALUE_OFF_MAX +
-                             1 + /* for bpf_spin_lock */
-                             1,  /* for bpf_timer */
-};
-
-enum bpf_kptr_type {
+     /* Support at most 8 offsets in a table */
+     BPF_MAP_VALUE_OFF_MAX           = 8,
+     /* Support at most 8 pointer in a BPF map value */
+     BPF_MAP_VALUE_KPTR_MAX          = BPF_MAP_VALUE_OFF_MAX,
+     /* Support at most 8 list_head in a BPF map value */
+     BPF_MAP_VALUE_LIST_HEAD_MAX     = BPF_MAP_VALUE_OFF_MAX,
+     BPF_MAP_OFF_ARR_MAX             = BPF_MAP_VALUE_KPTR_MAX +
+                                       BPF_MAP_VALUE_LIST_HEAD_MAX +
+                                       1 + /* for bpf_spin_lock */
+                                       1,  /* for bpf_timer */
+};
+
+enum bpf_off_type {
      BPF_KPTR_UNREF,
      BPF_KPTR_REF,
+     BPF_LIST_HEAD,
 };
 
 struct bpf_map_value_off_desc {
      u32 offset;
-     enum bpf_kptr_type type;
-     struct {
-             struct btf *btf;
-             struct module *module;
-             btf_dtor_kfunc_t dtor;
-             u32 btf_id;
-     } kptr;
+     enum bpf_off_type type;
+     union {
+             struct {
+                     struct btf *btf;
+                     struct module *module;
+                     btf_dtor_kfunc_t dtor;
+                     u32 btf_id;
+             } kptr; /* for BPF_KPTR_{UNREF,REF} */
+             struct {
+                     struct btf *btf;
+                     u32 value_type_id;
+                     u32 list_node_off;
+             } list_head; /* for BPF_LIST_HEAD */
+     };
 };
 
Quoted 11 lines

include/linux/bpf.h struct bpf_map { · line 234

Quoted 6 lines
include/linux/bpf.h +1 -0
 struct bpf_map_value_off {
@@ -215,6 +231,7 @@ struct bpf_map {
      u32 map_flags;
      int spin_lock_off; /* >=0 valid offset, <0 error */
      struct bpf_map_value_off *kptr_off_tab;
+     struct bpf_map_value_off *list_head_off_tab;

The union in bpf_map_value_off_desc prompts the question why separate array is needed.
Sorting gets uglier.

I'll try to create a unified offset array, there aren't going to be any collisions anyway, and we can discern between field types using the type field.

Quoted 27 lines
Quoted 16 lines
include/linux/bpf.h +5 -0
      int timer_off; /* >=0 valid offset, <0 error */
      u32 id;
      int numa_node;
@@ -265,6 +282,11 @@ static inline bool map_value_has_kptrs(const struct bpf_map *map)
      return !IS_ERR_OR_NULL(map->kptr_off_tab);
 }
 
+static inline bool map_value_has_list_heads(const struct bpf_map *map)
+{
+     return !IS_ERR_OR_NULL(map->list_head_off_tab);
+}
+
 static inline void check_and_init_map_value(struct bpf_map *map, void *dst)
 {
      if (unlikely(map_value_has_spin_lock(map)))

include/linux/bpf.h static inline void check_and_init_map_value(struct bpf_map *map, void *dst) · line 309

Quoted 10 lines
include/linux/bpf.h +7 -0
@@ -278,6 +300,13 @@ static inline void check_and_init_map_value(struct bpf_map *map, void *dst)
              for (i = 0; i < tab->nr_off; i++)
                      *(u64 *)(dst + tab->off[i].offset) = 0;
      }
+     if (unlikely(map_value_has_list_heads(map))) {
+             struct bpf_map_value_off *tab = map->list_head_off_tab;
+             int i;
+
+             for (i = 0; i < tab->nr_off; i++)
+                     memset(dst + tab->off[i].offset, 0, sizeof(struct list_head));
+     }
Quoted 8 lines

Do we really need to distinguish map_value_has_kptrs vs map_value_has_list_heads ? Can they be generalized?
rb_root will be next.
that would be yet another array and another 'if'-s everywhere?
And then another special pseudo-map type that will cause a bunch of copy-paste again? Maybe it's inevitable.
Trying to brainstorm.

Yes, it's a bit unfortunate how this is turning out to be. If we use a unified array we might be able to do it in one go though, taking size from the type.

Quoted 237 lines
Quoted 237 lines
include/linux/bpf.h +5 -0
 }
 
 /* memcpy that is used with 8-byte aligned pointers, power-of-8 size and
@@ -1676,6 +1705,11 @@ struct bpf_map_value_off *bpf_map_copy_kptr_off_tab(const struct bpf_map *map);
 bool bpf_map_equal_kptr_off_tab(const struct bpf_map *map_a, const struct bpf_map *map_b);
 void bpf_map_free_kptrs(struct bpf_map *map, void *map_value);
 
+struct bpf_map_value_off_desc *bpf_map_list_head_off_contains(struct bpf_map *map, u32 offset);
+void bpf_map_free_list_head_off_tab(struct bpf_map *map);
+struct bpf_map_value_off *bpf_map_copy_list_head_off_tab(const struct bpf_map *map);
+bool bpf_map_equal_list_head_off_tab(const struct bpf_map *map_a, const struct bpf_map *map_b);
+
 struct bpf_map *bpf_map_get(u32 ufd);
 struct bpf_map *bpf_map_get_with_uref(u32 ufd);
 struct bpf_map *__bpf_map_get(struct fd f);
include/linux/btf.h +2 -0
index 8062f9da7c40..9b62b8b2117e 100644
@@ -156,6 +156,8 @@ int btf_find_spin_lock(const struct btf *btf, const struct btf_type *t);
 int btf_find_timer(const struct btf *btf, const struct btf_type *t);
 struct bpf_map_value_off *btf_parse_kptrs(const struct btf *btf,
                                        const struct btf_type *t);
+struct bpf_map_value_off *btf_parse_list_heads(struct btf *btf,
+                                            const struct btf_type *t);
 bool btf_type_is_void(const struct btf_type *t);
 s32 btf_find_by_name_kind(const struct btf *btf, const char *name, u8 kind);
 const struct btf_type *btf_type_skip_modifiers(const struct btf *btf,
kernel/bpf/arraymap.c +2 -0
index 832b2659e96e..c7263ee3a35f 100644
@@ -423,6 +423,8 @@ static void array_map_free(struct bpf_map *map)
      struct bpf_array *array = container_of(map, struct bpf_array, map);
      int i;
 
+     bpf_map_free_list_head_off_tab(map);
+
      if (map_value_has_kptrs(map)) {
              if (array->map.map_type == BPF_MAP_TYPE_PERCPU_ARRAY) {
                      for (i = 0; i < array->map.max_entries; i++) {
kernel/bpf/bpf_local_storage.c +1 -0
index 58cb0c179097..b5ccd76026b6 100644
@@ -616,6 +616,7 @@ void bpf_local_storage_map_free(struct bpf_local_storage_map *smap,
              rcu_barrier();
              bpf_map_free_kptr_off_tab(&smap->map);
      }
+     bpf_map_free_list_head_off_tab(&smap->map);
      kvfree(smap->buckets);
      bpf_map_area_free(smap);
 }
kernel/bpf/btf.c +96 -8
index 6740c3ade8f1..0fb045be3837 100644
@@ -3185,6 +3185,7 @@ enum btf_field_type {
      BTF_FIELD_SPIN_LOCK,
      BTF_FIELD_TIMER,
      BTF_FIELD_KPTR,
+     BTF_FIELD_LIST_HEAD,
 };
 
 enum {
@@ -3193,9 +3194,17 @@ enum {
 };
 
 struct btf_field_info {
-     u32 type_id;
      u32 off;
-     enum bpf_kptr_type type;
+     union {
+             struct {
+                     u32 type_id;
+                     enum bpf_off_type type;
+             } kptr;
+             struct {
+                     u32 value_type_id;
+                     const char *node_name;
+             } list_head;
+     };
 };
 
 static int btf_find_struct(const struct btf *btf, const struct btf_type *t,
@@ -3212,7 +3221,7 @@ static int btf_find_struct(const struct btf *btf, const struct btf_type *t,
 static int btf_find_kptr(const struct btf *btf, const struct btf_type *t,
                       u32 off, int sz, struct btf_field_info *info)
 {
-     enum bpf_kptr_type type;
+     enum bpf_off_type type;
      u32 res_id;
 
      /* Permit modifiers on the pointer itself */
@@ -3241,9 +3250,71 @@ static int btf_find_kptr(const struct btf *btf, const struct btf_type *t,
      if (!__btf_type_is_struct(t))
              return -EINVAL;
 
-     info->type_id = res_id;
      info->off = off;
-     info->type = type;
+     info->kptr.type_id = res_id;
+     info->kptr.type = type;
+     return BTF_FIELD_FOUND;
+}
+
+static const char *btf_find_decl_tag_value(const struct btf *btf,
+                                        const struct btf_type *pt,
+                                        int comp_idx, const char *tag_key)
+{
+     int i;
+
+     for (i = 1; i < btf_nr_types(btf); i++) {
+             const struct btf_type *t = btf_type_by_id(btf, i);
+             int len = strlen(tag_key);
+
+             if (!btf_type_is_decl_tag(t))
+                     continue;
+             /* TODO: Instead of btf_type pt, it would be much better if we had BTF
+              * ID of the map value type. This would avoid btf_type_by_id call here.
+              */
+             if (pt != btf_type_by_id(btf, t->type) ||
+                 btf_type_decl_tag(t)->component_idx != comp_idx)
+                     continue;
+             if (strncmp(__btf_name_by_offset(btf, t->name_off), tag_key, len))
+                     continue;
+             return __btf_name_by_offset(btf, t->name_off) + len;
+     }
+     return NULL;
+}
+
+static int btf_find_list_head(const struct btf *btf, const struct btf_type *pt,
+                           int comp_idx, const struct btf_type *t,
+                           u32 off, int sz, struct btf_field_info *info)
+{
+     const char *value_type;
+     const char *list_node;
+     s32 id;
+
+     if (!__btf_type_is_struct(t))
+             return BTF_FIELD_IGNORE;
+     if (t->size != sz)
+             return BTF_FIELD_IGNORE;
+     value_type = btf_find_decl_tag_value(btf, pt, comp_idx, "contains:");
+     if (!value_type)
+             return -EINVAL;
+     if (strncmp(value_type, "struct:", sizeof("struct:") - 1))
+             return -EINVAL;
+     value_type += sizeof("struct:") - 1;
+     list_node = strstr(value_type, ":");
+     if (!list_node)
+             return -EINVAL;
+     value_type = kstrndup(value_type, list_node - value_type, GFP_ATOMIC);
+     if (!value_type)
+             return -ENOMEM;
+     id = btf_find_by_name_kind(btf, value_type, BTF_KIND_STRUCT);
+     kfree(value_type);
+     if (id < 0)
+             return id;
+     list_node++;
+     if (str_is_empty(list_node))
+             return -EINVAL;
+     info->off = off;
+     info->list_head.value_type_id = id;
+     info->list_head.node_name = list_node;
      return BTF_FIELD_FOUND;
 }
 
@@ -3286,6 +3357,12 @@ static int btf_find_struct_field(const struct btf *btf, const struct btf_type *t
                      if (ret < 0)
                              return ret;
                      break;
+             case BTF_FIELD_LIST_HEAD:
+                     ret = btf_find_list_head(btf, t, i, member_type, off, sz,
+                                              idx < info_cnt ? &info[idx] : &tmp);
+                     if (ret < 0)
+                             return ret;
+                     break;
              default:
                      return -EFAULT;
              }
@@ -3336,6 +3413,12 @@ static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t,
                      if (ret < 0)
                              return ret;
                      break;
+             case BTF_FIELD_LIST_HEAD:
+                     ret = btf_find_list_head(btf, var, -1, var_type, off, sz,
+                                              idx < info_cnt ? &info[idx] : &tmp);
+                     if (ret < 0)
+                             return ret;
+                     break;
              default:
                      return -EFAULT;
              }
@@ -3372,6 +3455,11 @@ static int btf_find_field(const struct btf *btf, const struct btf_type *t,
              sz = sizeof(u64);
              align = 8;
              break;
+     case BTF_FIELD_LIST_HEAD:
+             name = "bpf_list_head";
+             sz = sizeof(struct bpf_list_head);
+             align = __alignof__(struct bpf_list_head);
+             break;
      default:
              return -EFAULT;
      }
@@ -3440,7 +3528,7 @@ struct bpf_map_value_off *btf_parse_kptrs(const struct btf *btf,
              /* Find type in map BTF, and use it to look up the matching type
               * in vmlinux or module BTFs, by name and kind.
               */
-             t = btf_type_by_id(btf, info_arr[i].type_id);
+             t = btf_type_by_id(btf, info_arr[i].kptr.type_id);
              id = bpf_find_btf_id(__btf_name_by_offset(btf, t->name_off), BTF_INFO_KIND(t->info),
                                   &kernel_btf);
              if (id < 0) {
@@ -3451,7 +3539,7 @@ struct bpf_map_value_off *btf_parse_kptrs(const struct btf *btf,
              /* Find and stash the function pointer for the destruction function that
               * needs to be eventually invoked from the map free path.
               */
-             if (info_arr[i].type == BPF_KPTR_REF) {
+             if (info_arr[i].kptr.type == BPF_KPTR_REF) {
                      const struct btf_type *dtor_func;
                      const char *dtor_func_name;
                      unsigned long addr;
@@ -3494,7 +3582,7 @@ struct bpf_map_value_off *btf_parse_kptrs(const struct btf *btf,
              }
 
              tab->off[i].offset = info_arr[i].off;
-             tab->off[i].type = info_arr[i].type;
+             tab->off[i].type = info_arr[i].kptr.type;
              tab->off[i].kptr.btf_id = id;
              tab->off[i].kptr.btf = kernel_btf;
              tab->off[i].kptr.module = mod;
@@ -3515,6 +3603,75 @@ struct bpf_map_value_off *btf_parse_kptrs(const struct btf *btf,
      return ERR_PTR(ret);
 }
 
Quoted 10 lines

kernel/bpf/btf.c struct bpf_map_value_off *btf_parse_kptrs(const struct btf *btf, · line 3612

Quoted 7 lines
kernel/bpf/btf.c +7 -0
+struct bpf_map_value_off *btf_parse_list_heads(struct btf *btf, const struct btf_type *t)
+{
+     struct btf_field_info info_arr[BPF_MAP_VALUE_OFF_MAX];
+     struct bpf_map_value_off *tab;
+     int ret, i, nr_off;
+
+     ret = btf_find_field(btf, t, BTF_FIELD_LIST_HEAD, info_arr, ARRAY_SIZE(info_arr));

Like if search for both LIST_HEAD and KPTR here to know the size.

Yes, it seems like a good idea to unify things, I'll do it in v1.

Quoted 44 lines
Quoted 5 lines
kernel/bpf/btf.c +5 -0
+     if (ret < 0)
+             return ERR_PTR(ret);
+     if (!ret)
+             return NULL;
+

kernel/bpf/btf.c struct bpf_map_value_off *btf_parse_kptrs(const struct btf *btf, · line 3627

Quoted 10 lines
kernel/bpf/btf.c +10 -0
+     nr_off = ret;
+     tab = kzalloc(offsetof(struct bpf_map_value_off, off[nr_off]), GFP_KERNEL | __GFP_NOWARN);
+     if (!tab)
+             return ERR_PTR(-ENOMEM);
+
+     for (i = 0; i < nr_off; i++) {
+             const struct btf_type *t, *n = NULL;
+             const struct btf_member *member;
+             u32 offset;
+             int j;

and here we can process both, since field_info has type.

Quoted 26 lines
kernel/bpf/btf.c +26 -0
+
+             t = btf_type_by_id(btf, info_arr[i].list_head.value_type_id);
+             /* We've already checked that value_type_id is a struct type. We
+              * just need to figure out the offset of the list_node, and
+              * verify its type.
+              */
+             ret = -EINVAL;
+             for_each_member(j, t, member) {
+                     if (strcmp(info_arr[i].list_head.node_name, __btf_name_by_offset(btf, member->name_off)))
+                             continue;
+                     /* Invalid BTF, two members with same name */
+                     if (n) {
+                             /* We also need to btf_put for the current iteration! */
+                             i++;
+                             goto end;
+                     }
+                     n = btf_type_by_id(btf, member->type);
+                     if (!__btf_type_is_struct(n))
+                             goto end;
+                     if (strcmp("bpf_list_node", __btf_name_by_offset(btf, n->name_off)))
+                             goto end;
+                     offset = __btf_member_bit_offset(n, member);
+                     if (offset % 8)
+                             goto end;
+                     offset /= 8;
+                     if (offset % __alignof__(struct bpf_list_node))
Quoted 9 lines

kernel/bpf/btf.c struct bpf_map_value_off *btf_parse_kptrs(const struct btf *btf, · line 3658

Quoted 5 lines
kernel/bpf/btf.c +5 -0
+                             goto end;
+
+                     tab->off[i].offset = info_arr[i].off;
+                     tab->off[i].type = BPF_LIST_HEAD;
+                     btf_get(btf);

Do we need to btf_get? The btf should be pinned already and not going to be released until prog ends.

Hm, I think that's true. This is also the map BTF, not prog BTF, which I guess map holds a ref to it anyway until __bpf_map_put, so it should be fine. I'll add a comment.

author maintainer
#22 lore raw

RFC PATCH 14/32 bpf: Introduce bpf_kptr_alloc helper

In reply to Kumar Kartikeya Dwivedi

To allocate local kptr of types pointing into program BTF instead of kernel BTF, bpf_kptr_alloc is a new helper that takes the local type's BTF ID and returns a pointer to it. The size is automatically inferred from the type ID by the BPF verifier, so user only passes the BTF ID and flags, if any. For now, no flags are supported.

First, we use the new constant argument type support for kfuncs that enforces argument is a constant. We need to know the local type's BTF ID statically to enforce safety properties for the allocation. Next, we remember this and dynamically assign the return type. During that phase, we also query the actual size of the structure being allocated, and whether it is a struct type. If so, we stash the actual size for do_misc_fixups phase where we rewrite the first argument to be size instead of local type's BTF ID, which we can then pass on to the kernel allocator.

This needs some additional support for kfuncs as we were not doing argument rewrites for them. The fixup has been moved inside fixup_kfunc_call itself to avoid polluting the huge do_misc_fixups, and delta, prog, and insn pointers are recalculated based on if any instructions were patched.

The returned pointer needs to be handled specially as well. While normally, only struct pointers may be returned, a new internal kfunc flag __KF_RET_DYN_BTF is used to indicate the BTF is ascertained from arguments dynamically, hence it is now forced to be void * instead. For now, bpf_kptr_alloc is the only user of this support.

Hence, allocations using bpf_kptr_alloc are type safe. Later patches will introduce constructor and destructor support to local kptrs allocated from this helper. This would allow embedding kernel objects like bpf_spin_lock, bpf_list_node, bpf_list_head inside a local kptr allocation, and ensuring they are correctly initialized before use.

A new type flag is associated with PTR_TO_BTF_ID returned from
bpf_kptr_alloc: MEM_TYPE_LOCAL. This indicates that the type of the memory is of a local type coming from program's BTF.

The btf_struct_access mechanism is tuned to allow BPF_WRITE access to these allocated objects, so that programs can store data as usual in them. On following a pointer type inside such PTR_TO_BTF_ID, WALK_PTR sets the destination register as scalar instead. It would not be safe to recognize pointer types in local types. This can be changed in the future if it is allowed to embed kptrs inside such local kptrs.

---
 include/linux/bpf.h                           |  12 +-
 include/linux/bpf_verifier.h                  |   1 +
 include/linux/btf.h                           |   3 +
 kernel/bpf/btf.c                              |   8 +-
 kernel/bpf/helpers.c                          |  17 ++
 kernel/bpf/verifier.c                         | 156 +++++++++++++++---
 net/bpf/bpf_dummy_struct_ops.c                |   5 +-
 net/ipv4/bpf_tcp_ca.c                         |   5 +-
 .../testing/selftests/bpf/bpf_experimental.h  |  14 ++
 9 files changed, 191 insertions(+), 30 deletions(-)
include/linux/bpf.h +10 -2
index 35c2e9caeb98..5c8bfb0eba17 100644
@@ -486,6 +486,12 @@ enum bpf_type_flag {
 	/* Size is known at compile time. */
 	MEM_FIXED_SIZE		= BIT(10 + BPF_BASE_TYPE_BITS),
 
+	/* MEM is of a type from program BTF, not kernel BTF. This is used to
+	 * tag PTR_TO_BTF_ID allocated using bpf_kptr_alloc, since they have
+	 * entirely different semantics.
+	 */
+	MEM_TYPE_LOCAL		= BIT(11 + BPF_BASE_TYPE_BITS),
+
 	__BPF_TYPE_FLAG_MAX,
 	__BPF_TYPE_LAST_FLAG	= __BPF_TYPE_FLAG_MAX - 1,
 };
@@ -757,7 +763,8 @@ struct bpf_verifier_ops {
 				 const struct btf *btf,
 				 const struct btf_type *t, int off, int size,
 				 enum bpf_access_type atype,
-				 u32 *next_btf_id, enum bpf_type_flag *flag);
+				 u32 *next_btf_id, enum bpf_type_flag *flag,
+				 bool local_type);
 };
 
 struct bpf_prog_offload_ops {
@@ -1995,7 +2002,8 @@ static inline bool bpf_tracing_btf_ctx_access(int off, int size,
 int btf_struct_access(struct bpf_verifier_log *log, const struct btf *btf,
 		      const struct btf_type *t, int off, int size,
 		      enum bpf_access_type atype,
-		      u32 *next_btf_id, enum bpf_type_flag *flag);
+		      u32 *next_btf_id, enum bpf_type_flag *flag,
+		      bool local_type);
 bool btf_struct_ids_match(struct bpf_verifier_log *log,
 			  const struct btf *btf, u32 id, int off,
 			  const struct btf *need_btf, u32 need_type_id,
include/linux/bpf_verifier.h +1 -0
index c4d21568d192..c6d550978d63 100644
@@ -403,6 +403,7 @@ struct bpf_insn_aux_data {
 		 */
 		struct bpf_loop_inline_state loop_inline_state;
 	};
+	u64 kptr_alloc_size; /* used to store size of local kptr allocation */
 	u64 map_key_state; /* constant (32 bit) key tracking for maps */
 	int ctx_field_size; /* the ctx field size for load insn, maybe 0 */
 	u32 seen; /* this insn was processed by the verifier at env->pass_cnt */
include/linux/btf.h +3 -0
index 9b62b8b2117e..fc35c932e89e 100644
@@ -52,6 +52,9 @@
 #define KF_SLEEPABLE    (1 << 5) /* kfunc may sleep */
 #define KF_DESTRUCTIVE  (1 << 6) /* kfunc performs destructive actions */
 
+/* Internal kfunc flags, not meant for general use */
+#define __KF_RET_DYN_BTF (1 << 7) /* kfunc returns dynamically ascertained PTR_TO_BTF_ID */
+
 struct btf;
 struct btf_member;
 struct btf_type;
kernel/bpf/btf.c +7 -1
index 0fb045be3837..17977e0f4e09 100644
@@ -5919,7 +5919,8 @@ static int btf_struct_walk(struct bpf_verifier_log *log, const struct btf *btf,
 int btf_struct_access(struct bpf_verifier_log *log, const struct btf *btf,
 		      const struct btf_type *t, int off, int size,
 		      enum bpf_access_type atype __maybe_unused,
-		      u32 *next_btf_id, enum bpf_type_flag *flag)
+		      u32 *next_btf_id, enum bpf_type_flag *flag,
+		      bool local_type)
 {
 	enum bpf_type_flag tmp_flag = 0;
 	int err;
@@ -5930,6 +5931,11 @@ int btf_struct_access(struct bpf_verifier_log *log, const struct btf *btf,
 
 		switch (err) {
 		case WALK_PTR:
+			/* For local types, the destination register cannot
+			 * become a pointer again.
+			 */
+			if (local_type)
+				return SCALAR_VALUE;
 			/* If we found the pointer or scalar on t+off,
 			 * we're done.
 			 */
kernel/bpf/helpers.c +17 -0
index fc08035f14ed..d417aa4f0b22 100644
@@ -1696,10 +1696,27 @@ bpf_base_func_proto(enum bpf_func_id func_id)
 	}
 }
 
+__diag_push();
+__diag_ignore_all("-Wmissing-prototypes",
+		  "Global functions as their definitions will be in vmlinux BTF");
+
+void *bpf_kptr_alloc(u64 local_type_id__k, u64 flags)
+{
+	/* Verifier patches local_type_id__k to size */
+	u64 size = local_type_id__k;
+
+	if (flags)
+		return NULL;
+	return kmalloc(size, GFP_ATOMIC);
+}
+
+__diag_pop();
+
 BTF_SET8_START(tracing_btf_ids)
 #ifdef CONFIG_KEXEC_CORE
 BTF_ID_FLAGS(func, crash_kexec, KF_DESTRUCTIVE)
 #endif
+BTF_ID_FLAGS(func, bpf_kptr_alloc, KF_ACQUIRE | KF_RET_NULL | __KF_RET_DYN_BTF)
 BTF_SET8_END(tracing_btf_ids)
 
 static const struct btf_kfunc_id_set tracing_kfunc_set = {
kernel/bpf/verifier.c +133 -23

221 lines not on this page · raw message

net/bpf/bpf_dummy_struct_ops.c +3 -2
index e78dadfc5829..fa572714c6f6 100644
@@ -160,7 +160,8 @@ static int bpf_dummy_ops_btf_struct_access(struct bpf_verifier_log *log,
 					   const struct btf_type *t, int off,
 					   int size, enum bpf_access_type atype,
 					   u32 *next_btf_id,
-					   enum bpf_type_flag *flag)
+					   enum bpf_type_flag *flag,
+					   bool local_type)
 {
 	const struct btf_type *state;
 	s32 type_id;
@@ -178,7 +179,7 @@ static int bpf_dummy_ops_btf_struct_access(struct bpf_verifier_log *log,
 	}
 
 	err = btf_struct_access(log, btf, t, off, size, atype, next_btf_id,
-				flag);
+				flag, false);
 	if (err < 0)
 		return err;
 
net/ipv4/bpf_tcp_ca.c +3 -2
index 85a9e500c42d..869b6266833c 100644
@@ -73,13 +73,14 @@ static int bpf_tcp_ca_btf_struct_access(struct bpf_verifier_log *log,
 					const struct btf_type *t, int off,
 					int size, enum bpf_access_type atype,
 					u32 *next_btf_id,
-					enum bpf_type_flag *flag)
+					enum bpf_type_flag *flag,
+					bool local_type)
 {
 	size_t end;
 
 	if (atype == BPF_READ)
 		return btf_struct_access(log, btf, t, off, size, atype, next_btf_id,
-					 flag);
+					 flag, false);
 
 	if (t != tcp_sock_type) {
 		bpf_log(log, "only read is supported\n");
tools/testing/selftests/bpf/bpf_experimental.h +14 -0
index ea1b3b1839d1..bddd77093d1e 100644
@@ -18,4 +18,18 @@ struct bpf_list_node {
 #endif
 
 #ifndef __KERNEL__
+
+/* Description
+ *	Allocates a local kptr of type represented by 'local_type_id' in program
+ *	BTF. User may use the bpf_core_type_id_local macro to pass the type ID
+ *	of a struct in program BTF.
+ *
+ *	The 'local_type_id' parameter must be a known constant.
+ *	The 'flags' parameter must be 0.
+ * Returns
+ *	A local kptr corresponding to passed in 'local_type_id', or NULL on
+ *	failure.
+ */
+void *bpf_kptr_alloc(__u64 local_type_id, __u64 flags) __ksym;
+
 #endif
Signature · 1 line
2.34.1
#23 lore raw

Re: [PATCH RFC bpf-next v1 14/32] bpf: Introduce bpf_kptr_alloc helper

Quoted 112 lines · Kumar Kartikeya Dwivedi

To allocate local kptr of types pointing into program BTF instead of kernel BTF, bpf_kptr_alloc is a new helper that takes the local type's BTF ID and returns a pointer to it. The size is automatically inferred from the type ID by the BPF verifier, so user only passes the BTF ID and flags, if any. For now, no flags are supported.

First, we use the new constant argument type support for kfuncs that enforces argument is a constant. We need to know the local type's BTF ID statically to enforce safety properties for the allocation. Next, we remember this and dynamically assign the return type. During that phase, we also query the actual size of the structure being allocated, and whether it is a struct type. If so, we stash the actual size for do_misc_fixups phase where we rewrite the first argument to be size instead of local type's BTF ID, which we can then pass on to the kernel allocator.

This needs some additional support for kfuncs as we were not doing argument rewrites for them. The fixup has been moved inside fixup_kfunc_call itself to avoid polluting the huge do_misc_fixups, and delta, prog, and insn pointers are recalculated based on if any instructions were patched.

The returned pointer needs to be handled specially as well. While normally, only struct pointers may be returned, a new internal kfunc flag __KF_RET_DYN_BTF is used to indicate the BTF is ascertained from arguments dynamically, hence it is now forced to be void * instead. For now, bpf_kptr_alloc is the only user of this support.

Hence, allocations using bpf_kptr_alloc are type safe. Later patches will introduce constructor and destructor support to local kptrs allocated from this helper. This would allow embedding kernel objects like bpf_spin_lock, bpf_list_node, bpf_list_head inside a local kptr allocation, and ensuring they are correctly initialized before use.

A new type flag is associated with PTR_TO_BTF_ID returned from
bpf_kptr_alloc: MEM_TYPE_LOCAL. This indicates that the type of the memory is of a local type coming from program's BTF.

The btf_struct_access mechanism is tuned to allow BPF_WRITE access to these allocated objects, so that programs can store data as usual in them. On following a pointer type inside such PTR_TO_BTF_ID, WALK_PTR sets the destination register as scalar instead. It would not be safe to recognize pointer types in local types. This can be changed in the future if it is allowed to embed kptrs inside such local kptrs.

---
 include/linux/bpf.h                           |  12 +-
 include/linux/bpf_verifier.h                  |   1 +
 include/linux/btf.h                           |   3 +
 kernel/bpf/btf.c                              |   8 +-
 kernel/bpf/helpers.c                          |  17 ++
 kernel/bpf/verifier.c                         | 156 +++++++++++++++---
 net/bpf/bpf_dummy_struct_ops.c                |   5 +-
 net/ipv4/bpf_tcp_ca.c                         |   5 +-
 .../testing/selftests/bpf/bpf_experimental.h  |  14 ++
 9 files changed, 191 insertions(+), 30 deletions(-)
include/linux/bpf.h +10 -2
index 35c2e9caeb98..5c8bfb0eba17 100644
@@ -486,6 +486,12 @@ enum bpf_type_flag {
 	/* Size is known at compile time. */
 	MEM_FIXED_SIZE		= BIT(10 + BPF_BASE_TYPE_BITS),
 
+	/* MEM is of a type from program BTF, not kernel BTF. This is used to
+	 * tag PTR_TO_BTF_ID allocated using bpf_kptr_alloc, since they have
+	 * entirely different semantics.
+	 */
+	MEM_TYPE_LOCAL		= BIT(11 + BPF_BASE_TYPE_BITS),
+
 	__BPF_TYPE_FLAG_MAX,
 	__BPF_TYPE_LAST_FLAG	= __BPF_TYPE_FLAG_MAX - 1,
 };
@@ -757,7 +763,8 @@ struct bpf_verifier_ops {
 				 const struct btf *btf,
 				 const struct btf_type *t, int off, int size,
 				 enum bpf_access_type atype,
-				 u32 *next_btf_id, enum bpf_type_flag *flag);
+				 u32 *next_btf_id, enum bpf_type_flag *flag,
+				 bool local_type);
 };
 
 struct bpf_prog_offload_ops {
@@ -1995,7 +2002,8 @@ static inline bool bpf_tracing_btf_ctx_access(int off, int size,
 int btf_struct_access(struct bpf_verifier_log *log, const struct btf *btf,
 		      const struct btf_type *t, int off, int size,
 		      enum bpf_access_type atype,
-		      u32 *next_btf_id, enum bpf_type_flag *flag);
+		      u32 *next_btf_id, enum bpf_type_flag *flag,
+		      bool local_type);
 bool btf_struct_ids_match(struct bpf_verifier_log *log,
 			  const struct btf *btf, u32 id, int off,
 			  const struct btf *need_btf, u32 need_type_id,
include/linux/bpf_verifier.h +1 -0
index c4d21568d192..c6d550978d63 100644
@@ -403,6 +403,7 @@ struct bpf_insn_aux_data {
 		 */
 		struct bpf_loop_inline_state loop_inline_state;
 	};
+	u64 kptr_alloc_size; /* used to store size of local kptr allocation */
 	u64 map_key_state; /* constant (32 bit) key tracking for maps */
 	int ctx_field_size; /* the ctx field size for load insn, maybe 0 */
 	u32 seen; /* this insn was processed by the verifier at env->pass_cnt */
include/linux/btf.h +0 -0
index 9b62b8b2117e..fc35c932e89e 100644

include/linux/btf.h line 56

Quoted 5 lines
include/linux/btf.h +2 -0
@@ -52,6 +52,9 @@
 #define KF_SLEEPABLE    (1 << 5) /* kfunc may sleep */
 #define KF_DESTRUCTIVE  (1 << 6) /* kfunc performs destructive actions */
 
+/* Internal kfunc flags, not meant for general use */
+#define __KF_RET_DYN_BTF (1 << 7) /* kfunc returns dynamically ascertained PTR_TO_BTF_ID */

Is there going to be another func that returns similar dynamic type? We have one such func already kptr_xhcg. I don't see why we need this flag. We can just compare func_id-s.
In this patch it will be just fund_id == kfunc_ids[KF_kptr_alloc]; When more kfuncs become alloc-like we will just add few ||.

Quoted 92 lines
include/linux/btf.h +1 -0
+
 struct btf;
 struct btf_member;
 struct btf_type;
kernel/bpf/btf.c +7 -1
index 0fb045be3837..17977e0f4e09 100644
@@ -5919,7 +5919,8 @@ static int btf_struct_walk(struct bpf_verifier_log *log, const struct btf *btf,
 int btf_struct_access(struct bpf_verifier_log *log, const struct btf *btf,
 		      const struct btf_type *t, int off, int size,
 		      enum bpf_access_type atype __maybe_unused,
-		      u32 *next_btf_id, enum bpf_type_flag *flag)
+		      u32 *next_btf_id, enum bpf_type_flag *flag,
+		      bool local_type)
 {
 	enum bpf_type_flag tmp_flag = 0;
 	int err;
@@ -5930,6 +5931,11 @@ int btf_struct_access(struct bpf_verifier_log *log, const struct btf *btf,
 
 		switch (err) {
 		case WALK_PTR:
+			/* For local types, the destination register cannot
+			 * become a pointer again.
+			 */
+			if (local_type)
+				return SCALAR_VALUE;
 			/* If we found the pointer or scalar on t+off,
 			 * we're done.
 			 */
kernel/bpf/helpers.c +17 -0
index fc08035f14ed..d417aa4f0b22 100644
@@ -1696,10 +1696,27 @@ bpf_base_func_proto(enum bpf_func_id func_id)
 	}
 }
 
+__diag_push();
+__diag_ignore_all("-Wmissing-prototypes",
+		  "Global functions as their definitions will be in vmlinux BTF");
+
+void *bpf_kptr_alloc(u64 local_type_id__k, u64 flags)
+{
+	/* Verifier patches local_type_id__k to size */
+	u64 size = local_type_id__k;
+
+	if (flags)
+		return NULL;
+	return kmalloc(size, GFP_ATOMIC);
+}
+
+__diag_pop();
+
 BTF_SET8_START(tracing_btf_ids)
 #ifdef CONFIG_KEXEC_CORE
 BTF_ID_FLAGS(func, crash_kexec, KF_DESTRUCTIVE)
 #endif
+BTF_ID_FLAGS(func, bpf_kptr_alloc, KF_ACQUIRE | KF_RET_NULL | __KF_RET_DYN_BTF)
 BTF_SET8_END(tracing_btf_ids)
 
 static const struct btf_kfunc_id_set tracing_kfunc_set = {
kernel/bpf/verifier.c +11 -2
index ab91e5ca7e41..8f28aa7f1e8d 100644
@@ -472,6 +472,11 @@ static bool type_may_be_null(u32 type)
 	return type & PTR_MAYBE_NULL;
 }
 
+static bool type_is_local(u32 type)
+{
+	return type & MEM_TYPE_LOCAL;
+}
+
 static bool is_acquire_function(enum bpf_func_id func_id,
 				const struct bpf_map *map)
 {
@@ -4556,17 +4561,22 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env,
 		return -EACCES;
 	}
 
-	if (env->ops->btf_struct_access) {
+	/* For allocated PTR_TO_BTF_ID pointing to a local type, we cannot do
+	 * btf_struct_access callback.
+	 */
+	if (env->ops->btf_struct_access && !type_is_local(reg->type)) {
 		ret = env->ops->btf_struct_access(&env->log, reg->btf, t,
-						  off, size, atype, &btf_id, &flag);
+						  off, size, atype, &btf_id, &flag,
+						  false);
 	} else {

kernel/bpf/verifier.c static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · line 4579

Quoted 10 lines
kernel/bpf/verifier.c +3 -2
-		if (atype != BPF_READ) {
+		/* It is allowed to write to pointer to a local type */
+		if (atype != BPF_READ && !type_is_local(reg->type)) {
 			verbose(env, "only read is supported\n");
 			return -EACCES;
 		}
 
 		ret = btf_struct_access(&env->log, reg->btf, t, off, size,
-					atype, &btf_id, &flag);
+					atype, &btf_id, &flag, type_is_local(reg->type));

imo it's cleaner to pass 'reg' instead of 'reg->btf', so we don't have to pass another boolean.
And check type_is_local(reg) inside btf_struct_access().

Quoted 35 lines
kernel/bpf/verifier.c +13 -1
 	}
 
 	if (ret < 0)
@@ -4630,7 +4640,7 @@ static int check_ptr_to_map_access(struct bpf_verifier_env *env,
 		return -EACCES;
 	}
 
-	ret = btf_struct_access(&env->log, btf_vmlinux, t, off, size, atype, &btf_id, &flag);
+	ret = btf_struct_access(&env->log, btf_vmlinux, t, off, size, atype, &btf_id, &flag, false);
 	if (ret < 0)
 		return ret;
 
@@ -7661,6 +7671,11 @@ static bool is_kfunc_destructive(struct bpf_kfunc_arg_meta *meta)
 	return meta->kfunc_flags & KF_DESTRUCTIVE;
 }
 
+static bool __is_kfunc_ret_dyn_btf(struct bpf_kfunc_arg_meta *meta)
+{
+	return meta->kfunc_flags & __KF_RET_DYN_BTF;
+}
+
 static bool is_kfunc_arg_kptr_get(struct bpf_kfunc_arg_meta *meta, int arg)
 {
 	return arg == 0 && (meta->kfunc_flags & KF_KPTR_GET);
@@ -7751,6 +7766,24 @@ static u32 *reg2btf_ids[__BPF_REG_TYPE_MAX] = {
 #endif
 };
 
+BTF_ID_LIST(special_kfuncs)
+BTF_ID(func, bpf_kptr_alloc)
+
+enum bpf_special_kfuncs {
+	KF_SPECIAL_bpf_kptr_alloc,
+	KF_SPECIAL_MAX,
+};

kernel/bpf/verifier.c static u32 *reg2btf_ids[__BPF_REG_TYPE_MAX] = { · line 7785

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+
+static bool __is_kfunc_special(const struct btf *btf, u32 func_id, unsigned int kf_sp)
+{
+	if (btf != btf_vmlinux || kf_sp >= KF_SPECIAL_MAX)
+		return false;
+	return func_id == special_kfuncs[kf_sp];
+}
+
+#define is_kfunc_special(btf, func_id, func_name) \
+	__is_kfunc_special(btf, func_id, KF_SPECIAL_##func_name)

This looks like reinventing the wheel.
I'd think similar to btf_tracing_ids[BTF_TRACING_TYPE_VMA] would work just as well. It's less magic. No need for above macro
and btf != btf_vmlinux should really be explicit in the code
and done early and once.

Quoted 18 lines
kernel/bpf/verifier.c +1 -10
+
 enum kfunc_ptr_arg_types {
 	KF_ARG_PTR_TO_CTX,
 	KF_ARG_PTR_TO_BTF_ID,	     /* Also covers reg2btf_ids conversions */
@@ -8120,20 +8153,55 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		mark_reg_unknown(env, regs, BPF_REG_0);
 		mark_btf_func_reg_size(env, BPF_REG_0, t->size);
 	} else if (btf_type_is_ptr(t)) {
-		ptr_type = btf_type_skip_modifiers(desc_btf, t->type,
-						   &ptr_type_id);
-		if (!btf_type_is_struct(ptr_type)) {
-			ptr_type_name = btf_name_by_offset(desc_btf,
-							   ptr_type->name_off);
-			verbose(env, "kernel function %s returns pointer type %s %s is not supported\n",
-				func_name, btf_type_str(ptr_type),
-				ptr_type_name);
-			return -EINVAL;
-		}

kernel/bpf/verifier.c static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 8163

Quoted 10 lines
kernel/bpf/verifier.c +6 -2
+		struct btf *ret_btf;
+		u32 ret_btf_id;
+
+		ptr_type = btf_type_skip_modifiers(desc_btf, t->type, &ptr_type_id);
 		mark_reg_known_zero(env, regs, BPF_REG_0);
-		regs[BPF_REG_0].btf = desc_btf;
 		regs[BPF_REG_0].type = PTR_TO_BTF_ID;
-		regs[BPF_REG_0].btf_id = ptr_type_id;
+
+		if (__is_kfunc_ret_dyn_btf(&meta)) {

just check meta.func_id == kfunc_ids[KF_kptr_alloc] instead?

kernel/bpf/verifier.c static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 8167

Quoted 4 lines
kernel/bpf/verifier.c +4 -0
+			const struct btf_type *ret_t;
+
+			/* Currently, only bpf_kptr_alloc needs special handling */
+			if (!is_kfunc_special(meta.btf, meta.func_id, bpf_kptr_alloc) ||

same thing.

Quoted 104 lines
kernel/bpf/verifier.c +76 -6
+			    !meta.arg_constant.found || !btf_type_is_void(ptr_type)) {
+				verbose(env, "verifier internal error: misconfigured kfunc\n");
+				return -EFAULT;
+			}
+
+			if (((u64)(u32)meta.arg_constant.value) != meta.arg_constant.value) {
+				verbose(env, "local type ID argument must be in range [0, U32_MAX]\n");
+				return -EINVAL;
+			}
+
+			ret_btf = env->prog->aux->btf;
+			ret_btf_id = meta.arg_constant.value;
+
+			ret_t = btf_type_by_id(ret_btf, ret_btf_id);
+			if (!ret_t || !__btf_type_is_struct(ret_t)) {
+				verbose(env, "local type ID %d passed to bpf_kptr_alloc does not refer to struct\n",
+					ret_btf_id);
+				return -EINVAL;
+			}
+			/* Remember this so that we can rewrite R1 as size in fixup_kfunc_call */
+			env->insn_aux_data[insn_idx].kptr_alloc_size = ret_t->size;
+			/* For now, since we hardcode prog->btf, also hardcode
+			 * setting of this flag.
+			 */
+			regs[BPF_REG_0].type |= MEM_TYPE_LOCAL;
+		} else {
+			if (!btf_type_is_struct(ptr_type)) {
+				ptr_type_name = btf_name_by_offset(desc_btf, ptr_type->name_off);
+				verbose(env, "kernel function %s returns pointer type %s %s is not supported\n",
+					func_name, btf_type_str(ptr_type), ptr_type_name);
+				return -EINVAL;
+			}
+			ret_btf = desc_btf;
+			ret_btf_id = ptr_type_id;
+		}
+		regs[BPF_REG_0].btf = ret_btf;
+		regs[BPF_REG_0].btf_id = ret_btf_id;
 		if (is_kfunc_ret_null(&meta)) {
 			regs[BPF_REG_0].type |= PTR_MAYBE_NULL;
 			/* For mark_ptr_or_null_reg, see 93c230e3f5bd6 */
@@ -14371,8 +14439,43 @@ static int fixup_call_args(struct bpf_verifier_env *env)
 	return err;
 }
 
+static int do_kfunc_fixups(struct bpf_verifier_env *env, struct bpf_insn *insn,
+			   s32 imm, int insn_idx, int delta)
+{
+	struct bpf_insn insn_buf[16];
+	struct bpf_prog *new_prog;
+	int cnt;
+
+	/* No need to lookup btf, only vmlinux kfuncs are supported for special
+	 * kfuncs handling. Hence when insn->off is zero, check if it is a
+	 * special kfunc by hardcoding btf as btf_vmlinux.
+	 */
+	if (!insn->off && is_kfunc_special(btf_vmlinux, insn->imm, bpf_kptr_alloc)) {
+		u64 local_type_size = env->insn_aux_data[insn_idx + delta].kptr_alloc_size;
+
+		insn_buf[0] = BPF_MOV64_IMM(BPF_REG_1, local_type_size);
+		insn_buf[1] = *insn;
+		cnt = 2;
+
+		new_prog = bpf_patch_insn_data(env, insn_idx + delta, insn_buf, cnt);
+		if (!new_prog)
+			return -ENOMEM;
+
+		delta += cnt - 1;
+		insn = new_prog->insnsi + insn_idx + delta;
+		goto patch_call_imm;
+	}
+
+	insn->imm = imm;
+	return 0;
+patch_call_imm:
+	insn->imm = imm;
+	return cnt - 1;
+}
+
 static int fixup_kfunc_call(struct bpf_verifier_env *env,
-			    struct bpf_insn *insn)
+			    struct bpf_insn *insn,
+			    int insn_idx, int delta)
 {
 	const struct bpf_kfunc_desc *desc;
 
@@ -14391,9 +14494,7 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env,
 		return -EFAULT;
 	}
 
-	insn->imm = desc->imm;
-
-	return 0;
+	return do_kfunc_fixups(env, insn, desc->imm, insn_idx, delta);
 }
 
 /* Do various post-verification rewrites in a single program pass.
@@ -14534,9 +14635,18 @@ static int do_misc_fixups(struct bpf_verifier_env *env)
 		if (insn->src_reg == BPF_PSEUDO_CALL)
 			continue;
 		if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL) {
-			ret = fixup_kfunc_call(env, insn);
-			if (ret)
+			ret = fixup_kfunc_call(env, insn, i, delta);
+			if (ret < 0)

kernel/bpf/verifier.c static int do_misc_fixups(struct bpf_verifier_env *env) · line 14649

Quoted 10 lines
kernel/bpf/verifier.c +9 -0
 				return ret;
+			/* If ret > 0, fixup_kfunc_call did some instruction
+			 * rewrites. Increment delta, reload prog and insn,
+			 * env->prog is already set by it to the new_prog.
+			 */
+			if (ret) {
+				delta += ret;
+				prog = env->prog;
+				insn = prog->insnsi + i + delta;
+			}

See how Yonghong did it:
https://lore.kernel.org/all/[email protected]/

It's cleaner to patch and adjust here instead of patch in one place and adjust in another.

Quoted 73 lines
kernel/bpf/verifier.c +0 -0
 			continue;
 		}
 
net/bpf/bpf_dummy_struct_ops.c +3 -2
index e78dadfc5829..fa572714c6f6 100644
@@ -160,7 +160,8 @@ static int bpf_dummy_ops_btf_struct_access(struct bpf_verifier_log *log,
 					   const struct btf_type *t, int off,
 					   int size, enum bpf_access_type atype,
 					   u32 *next_btf_id,
-					   enum bpf_type_flag *flag)
+					   enum bpf_type_flag *flag,
+					   bool local_type)
 {
 	const struct btf_type *state;
 	s32 type_id;
@@ -178,7 +179,7 @@ static int bpf_dummy_ops_btf_struct_access(struct bpf_verifier_log *log,
 	}
 
 	err = btf_struct_access(log, btf, t, off, size, atype, next_btf_id,
-				flag);
+				flag, false);
 	if (err < 0)
 		return err;
 
net/ipv4/bpf_tcp_ca.c +3 -2
index 85a9e500c42d..869b6266833c 100644
@@ -73,13 +73,14 @@ static int bpf_tcp_ca_btf_struct_access(struct bpf_verifier_log *log,
 					const struct btf_type *t, int off,
 					int size, enum bpf_access_type atype,
 					u32 *next_btf_id,
-					enum bpf_type_flag *flag)
+					enum bpf_type_flag *flag,
+					bool local_type)
 {
 	size_t end;
 
 	if (atype == BPF_READ)
 		return btf_struct_access(log, btf, t, off, size, atype, next_btf_id,
-					 flag);
+					 flag, false);
 
 	if (t != tcp_sock_type) {
 		bpf_log(log, "only read is supported\n");
tools/testing/selftests/bpf/bpf_experimental.h +14 -0
index ea1b3b1839d1..bddd77093d1e 100644
@@ -18,4 +18,18 @@ struct bpf_list_node {
 #endif
 
 #ifndef __KERNEL__
+
+/* Description
+ *	Allocates a local kptr of type represented by 'local_type_id' in program
+ *	BTF. User may use the bpf_core_type_id_local macro to pass the type ID
+ *	of a struct in program BTF.
+ *
+ *	The 'local_type_id' parameter must be a known constant.
+ *	The 'flags' parameter must be 0.
+ * Returns
+ *	A local kptr corresponding to passed in 'local_type_id', or NULL on
+ *	failure.
+ */
+void *bpf_kptr_alloc(__u64 local_type_id, __u64 flags) __ksym;
+
 #endif
Signature · 1 line
2.34.1
author maintainer
#24 lore raw

Re: [PATCH RFC bpf-next v1 14/32] bpf: Introduce bpf_kptr_alloc helper

Quoted 117 lines · Alexei Starovoitov
Quoted 115 lines · Kumar Kartikeya Dwivedi

To allocate local kptr of types pointing into program BTF instead of kernel BTF, bpf_kptr_alloc is a new helper that takes the local type's BTF ID and returns a pointer to it. The size is automatically inferred from the type ID by the BPF verifier, so user only passes the BTF ID and flags, if any. For now, no flags are supported.

First, we use the new constant argument type support for kfuncs that enforces argument is a constant. We need to know the local type's BTF ID statically to enforce safety properties for the allocation. Next, we remember this and dynamically assign the return type. During that phase, we also query the actual size of the structure being allocated, and whether it is a struct type. If so, we stash the actual size for do_misc_fixups phase where we rewrite the first argument to be size instead of local type's BTF ID, which we can then pass on to the kernel allocator.

This needs some additional support for kfuncs as we were not doing argument rewrites for them. The fixup has been moved inside fixup_kfunc_call itself to avoid polluting the huge do_misc_fixups, and delta, prog, and insn pointers are recalculated based on if any instructions were patched.

The returned pointer needs to be handled specially as well. While normally, only struct pointers may be returned, a new internal kfunc flag __KF_RET_DYN_BTF is used to indicate the BTF is ascertained from arguments dynamically, hence it is now forced to be void * instead. For now, bpf_kptr_alloc is the only user of this support.

Hence, allocations using bpf_kptr_alloc are type safe. Later patches will introduce constructor and destructor support to local kptrs allocated from this helper. This would allow embedding kernel objects like bpf_spin_lock, bpf_list_node, bpf_list_head inside a local kptr allocation, and ensuring they are correctly initialized before use.

A new type flag is associated with PTR_TO_BTF_ID returned from
bpf_kptr_alloc: MEM_TYPE_LOCAL. This indicates that the type of the memory is of a local type coming from program's BTF.

The btf_struct_access mechanism is tuned to allow BPF_WRITE access to these allocated objects, so that programs can store data as usual in them. On following a pointer type inside such PTR_TO_BTF_ID, WALK_PTR sets the destination register as scalar instead. It would not be safe to recognize pointer types in local types. This can be changed in the future if it is allowed to embed kptrs inside such local kptrs.

---
 include/linux/bpf.h                           |  12 +-
 include/linux/bpf_verifier.h                  |   1 +
 include/linux/btf.h                           |   3 +
 kernel/bpf/btf.c                              |   8 +-
 kernel/bpf/helpers.c                          |  17 ++
 kernel/bpf/verifier.c                         | 156 +++++++++++++++---
 net/bpf/bpf_dummy_struct_ops.c                |   5 +-
 net/ipv4/bpf_tcp_ca.c                         |   5 +-
 .../testing/selftests/bpf/bpf_experimental.h  |  14 ++
 9 files changed, 191 insertions(+), 30 deletions(-)
include/linux/bpf.h +10 -2
index 35c2e9caeb98..5c8bfb0eba17 100644
@@ -486,6 +486,12 @@ enum bpf_type_flag {
      /* Size is known at compile time. */
      MEM_FIXED_SIZE          = BIT(10 + BPF_BASE_TYPE_BITS),
 
+     /* MEM is of a type from program BTF, not kernel BTF. This is used to
+      * tag PTR_TO_BTF_ID allocated using bpf_kptr_alloc, since they have
+      * entirely different semantics.
+      */
+     MEM_TYPE_LOCAL          = BIT(11 + BPF_BASE_TYPE_BITS),
+
      __BPF_TYPE_FLAG_MAX,
      __BPF_TYPE_LAST_FLAG    = __BPF_TYPE_FLAG_MAX - 1,
 };
@@ -757,7 +763,8 @@ struct bpf_verifier_ops {
                               const struct btf *btf,
                               const struct btf_type *t, int off, int size,
                               enum bpf_access_type atype,
-                              u32 *next_btf_id, enum bpf_type_flag *flag);
+                              u32 *next_btf_id, enum bpf_type_flag *flag,
+                              bool local_type);
 };
 
 struct bpf_prog_offload_ops {
@@ -1995,7 +2002,8 @@ static inline bool bpf_tracing_btf_ctx_access(int off, int size,
 int btf_struct_access(struct bpf_verifier_log *log, const struct btf *btf,
                    const struct btf_type *t, int off, int size,
                    enum bpf_access_type atype,
-                   u32 *next_btf_id, enum bpf_type_flag *flag);
+                   u32 *next_btf_id, enum bpf_type_flag *flag,
+                   bool local_type);
 bool btf_struct_ids_match(struct bpf_verifier_log *log,
                        const struct btf *btf, u32 id, int off,
                        const struct btf *need_btf, u32 need_type_id,
include/linux/bpf_verifier.h +1 -0
index c4d21568d192..c6d550978d63 100644
@@ -403,6 +403,7 @@ struct bpf_insn_aux_data {
               */
              struct bpf_loop_inline_state loop_inline_state;
      };
+     u64 kptr_alloc_size; /* used to store size of local kptr allocation */
      u64 map_key_state; /* constant (32 bit) key tracking for maps */
      int ctx_field_size; /* the ctx field size for load insn, maybe 0 */
      u32 seen; /* this insn was processed by the verifier at env->pass_cnt */
include/linux/btf.h +0 -0
index 9b62b8b2117e..fc35c932e89e 100644
@@ -52,6 +52,9 @@
 #define KF_SLEEPABLE    (1 << 5) /* kfunc may sleep */
 #define KF_DESTRUCTIVE  (1 << 6) /* kfunc performs destructive actions */
 
Quoted 9 lines

include/linux/btf.h line 56

Quoted 2 lines
include/linux/btf.h +2 -0
+/* Internal kfunc flags, not meant for general use */
+#define __KF_RET_DYN_BTF (1 << 7) /* kfunc returns dynamically ascertained PTR_TO_BTF_ID */

Is there going to be another func that returns similar dynamic type? We have one such func already kptr_xhcg. I don't see why we need this flag. We can just compare func_id-s.
In this patch it will be just fund_id == kfunc_ids[KF_kptr_alloc]; When more kfuncs become alloc-like we will just add few ||.

There are, bpf_list_pop_{front,back}, even bpf_list_del, probably more as we add more variants of lists.
But I don't mind keeping a list, they all need to be handled a bit differently anyway to ascertain the type of PTR_TO_BTF_ID.

Quoted 99 lines
Quoted 99 lines
include/linux/btf.h +1 -0
+
 struct btf;
 struct btf_member;
 struct btf_type;
kernel/bpf/btf.c +7 -1
index 0fb045be3837..17977e0f4e09 100644
@@ -5919,7 +5919,8 @@ static int btf_struct_walk(struct bpf_verifier_log *log, const struct btf *btf,
 int btf_struct_access(struct bpf_verifier_log *log, const struct btf *btf,
                    const struct btf_type *t, int off, int size,
                    enum bpf_access_type atype __maybe_unused,
-                   u32 *next_btf_id, enum bpf_type_flag *flag)
+                   u32 *next_btf_id, enum bpf_type_flag *flag,
+                   bool local_type)
 {
      enum bpf_type_flag tmp_flag = 0;
      int err;
@@ -5930,6 +5931,11 @@ int btf_struct_access(struct bpf_verifier_log *log, const struct btf *btf,
 
              switch (err) {
              case WALK_PTR:
+                     /* For local types, the destination register cannot
+                      * become a pointer again.
+                      */
+                     if (local_type)
+                             return SCALAR_VALUE;
                      /* If we found the pointer or scalar on t+off,
                       * we're done.
                       */
kernel/bpf/helpers.c +17 -0
index fc08035f14ed..d417aa4f0b22 100644
@@ -1696,10 +1696,27 @@ bpf_base_func_proto(enum bpf_func_id func_id)
      }
 }
 
+__diag_push();
+__diag_ignore_all("-Wmissing-prototypes",
+               "Global functions as their definitions will be in vmlinux BTF");
+
+void *bpf_kptr_alloc(u64 local_type_id__k, u64 flags)
+{
+     /* Verifier patches local_type_id__k to size */
+     u64 size = local_type_id__k;
+
+     if (flags)
+             return NULL;
+     return kmalloc(size, GFP_ATOMIC);
+}
+
+__diag_pop();
+
 BTF_SET8_START(tracing_btf_ids)
 #ifdef CONFIG_KEXEC_CORE
 BTF_ID_FLAGS(func, crash_kexec, KF_DESTRUCTIVE)
 #endif
+BTF_ID_FLAGS(func, bpf_kptr_alloc, KF_ACQUIRE | KF_RET_NULL | __KF_RET_DYN_BTF)
 BTF_SET8_END(tracing_btf_ids)
 
 static const struct btf_kfunc_id_set tracing_kfunc_set = {
kernel/bpf/verifier.c +13 -3
index ab91e5ca7e41..8f28aa7f1e8d 100644
@@ -472,6 +472,11 @@ static bool type_may_be_null(u32 type)
      return type & PTR_MAYBE_NULL;
 }
 
+static bool type_is_local(u32 type)
+{
+     return type & MEM_TYPE_LOCAL;
+}
+
 static bool is_acquire_function(enum bpf_func_id func_id,
                              const struct bpf_map *map)
 {
@@ -4556,17 +4561,22 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env,
              return -EACCES;
      }
 
-     if (env->ops->btf_struct_access) {
+     /* For allocated PTR_TO_BTF_ID pointing to a local type, we cannot do
+      * btf_struct_access callback.
+      */
+     if (env->ops->btf_struct_access && !type_is_local(reg->type)) {
              ret = env->ops->btf_struct_access(&env->log, reg->btf, t,
-                                               off, size, atype, &btf_id, &flag);
+                                               off, size, atype, &btf_id, &flag,
+                                               false);
      } else {
-             if (atype != BPF_READ) {
+             /* It is allowed to write to pointer to a local type */
+             if (atype != BPF_READ && !type_is_local(reg->type)) {
                      verbose(env, "only read is supported\n");
                      return -EACCES;
              }
 
Quoted 8 lines

kernel/bpf/verifier.c static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · line 4579

Quoted 3 lines
kernel/bpf/verifier.c +1 -1
              ret = btf_struct_access(&env->log, reg->btf, t, off, size,
-                                     atype, &btf_id, &flag);
+                                     atype, &btf_id, &flag, type_is_local(reg->type));

imo it's cleaner to pass 'reg' instead of 'reg->btf', so we don't have to pass another boolean.
And check type_is_local(reg) inside btf_struct_access().

Yes, makes sense, will change in v1.

Quoted 43 lines
Quoted 43 lines
kernel/bpf/verifier.c +21 -1
      }
 
      if (ret < 0)
@@ -4630,7 +4640,7 @@ static int check_ptr_to_map_access(struct bpf_verifier_env *env,
              return -EACCES;
      }
 
-     ret = btf_struct_access(&env->log, btf_vmlinux, t, off, size, atype, &btf_id, &flag);
+     ret = btf_struct_access(&env->log, btf_vmlinux, t, off, size, atype, &btf_id, &flag, false);
      if (ret < 0)
              return ret;
 
@@ -7661,6 +7671,11 @@ static bool is_kfunc_destructive(struct bpf_kfunc_arg_meta *meta)
      return meta->kfunc_flags & KF_DESTRUCTIVE;
 }
 
+static bool __is_kfunc_ret_dyn_btf(struct bpf_kfunc_arg_meta *meta)
+{
+     return meta->kfunc_flags & __KF_RET_DYN_BTF;
+}
+
 static bool is_kfunc_arg_kptr_get(struct bpf_kfunc_arg_meta *meta, int arg)
 {
      return arg == 0 && (meta->kfunc_flags & KF_KPTR_GET);
@@ -7751,6 +7766,24 @@ static u32 *reg2btf_ids[__BPF_REG_TYPE_MAX] = {
 #endif
 };
 
+BTF_ID_LIST(special_kfuncs)
+BTF_ID(func, bpf_kptr_alloc)
+
+enum bpf_special_kfuncs {
+     KF_SPECIAL_bpf_kptr_alloc,
+     KF_SPECIAL_MAX,
+};
+
+static bool __is_kfunc_special(const struct btf *btf, u32 func_id, unsigned int kf_sp)
+{
+     if (btf != btf_vmlinux || kf_sp >= KF_SPECIAL_MAX)
+             return false;
+     return func_id == special_kfuncs[kf_sp];
+}
+
Quoted 9 lines

kernel/bpf/verifier.c static u32 *reg2btf_ids[__BPF_REG_TYPE_MAX] = { · line 7785

Quoted 2 lines
kernel/bpf/verifier.c +2 -0
+#define is_kfunc_special(btf, func_id, func_name) \
+     __is_kfunc_special(btf, func_id, KF_SPECIAL_##func_name)

This looks like reinventing the wheel.
I'd think similar to btf_tracing_ids[BTF_TRACING_TYPE_VMA] would work just as well. It's less magic. No need for above macro
and btf != btf_vmlinux should really be explicit in the code
and done early and once.

Ok.

Quoted 31 lines
Quoted 18 lines
kernel/bpf/verifier.c +1 -10
+
 enum kfunc_ptr_arg_types {
      KF_ARG_PTR_TO_CTX,
      KF_ARG_PTR_TO_BTF_ID,        /* Also covers reg2btf_ids conversions */
@@ -8120,20 +8153,55 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
              mark_reg_unknown(env, regs, BPF_REG_0);
              mark_btf_func_reg_size(env, BPF_REG_0, t->size);
      } else if (btf_type_is_ptr(t)) {
-             ptr_type = btf_type_skip_modifiers(desc_btf, t->type,
-                                                &ptr_type_id);
-             if (!btf_type_is_struct(ptr_type)) {
-                     ptr_type_name = btf_name_by_offset(desc_btf,
-                                                        ptr_type->name_off);
-                     verbose(env, "kernel function %s returns pointer type %s %s is not supported\n",
-                             func_name, btf_type_str(ptr_type),
-                             ptr_type_name);
-                     return -EINVAL;
-             }

kernel/bpf/verifier.c static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 8163

Quoted 10 lines
kernel/bpf/verifier.c +6 -2
+             struct btf *ret_btf;
+             u32 ret_btf_id;
+
+             ptr_type = btf_type_skip_modifiers(desc_btf, t->type, &ptr_type_id);
              mark_reg_known_zero(env, regs, BPF_REG_0);
-             regs[BPF_REG_0].btf = desc_btf;
              regs[BPF_REG_0].type = PTR_TO_BTF_ID;
-             regs[BPF_REG_0].btf_id = ptr_type_id;
+
+             if (__is_kfunc_ret_dyn_btf(&meta)) {

just check meta.func_id == kfunc_ids[KF_kptr_alloc] instead?

Quoted 7 lines

kernel/bpf/verifier.c static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 8167

Quoted 4 lines
kernel/bpf/verifier.c +4 -0
+                     const struct btf_type *ret_t;
+
+                     /* Currently, only bpf_kptr_alloc needs special handling */
+                     if (!is_kfunc_special(meta.btf, meta.func_id, bpf_kptr_alloc) ||

same thing.

Ack.

Quoted 118 lines
Quoted 104 lines
kernel/bpf/verifier.c +76 -6
+                         !meta.arg_constant.found || !btf_type_is_void(ptr_type)) {
+                             verbose(env, "verifier internal error: misconfigured kfunc\n");
+                             return -EFAULT;
+                     }
+
+                     if (((u64)(u32)meta.arg_constant.value) != meta.arg_constant.value) {
+                             verbose(env, "local type ID argument must be in range [0, U32_MAX]\n");
+                             return -EINVAL;
+                     }
+
+                     ret_btf = env->prog->aux->btf;
+                     ret_btf_id = meta.arg_constant.value;
+
+                     ret_t = btf_type_by_id(ret_btf, ret_btf_id);
+                     if (!ret_t || !__btf_type_is_struct(ret_t)) {
+                             verbose(env, "local type ID %d passed to bpf_kptr_alloc does not refer to struct\n",
+                                     ret_btf_id);
+                             return -EINVAL;
+                     }
+                     /* Remember this so that we can rewrite R1 as size in fixup_kfunc_call */
+                     env->insn_aux_data[insn_idx].kptr_alloc_size = ret_t->size;
+                     /* For now, since we hardcode prog->btf, also hardcode
+                      * setting of this flag.
+                      */
+                     regs[BPF_REG_0].type |= MEM_TYPE_LOCAL;
+             } else {
+                     if (!btf_type_is_struct(ptr_type)) {
+                             ptr_type_name = btf_name_by_offset(desc_btf, ptr_type->name_off);
+                             verbose(env, "kernel function %s returns pointer type %s %s is not supported\n",
+                                     func_name, btf_type_str(ptr_type), ptr_type_name);
+                             return -EINVAL;
+                     }
+                     ret_btf = desc_btf;
+                     ret_btf_id = ptr_type_id;
+             }
+             regs[BPF_REG_0].btf = ret_btf;
+             regs[BPF_REG_0].btf_id = ret_btf_id;
              if (is_kfunc_ret_null(&meta)) {
                      regs[BPF_REG_0].type |= PTR_MAYBE_NULL;
                      /* For mark_ptr_or_null_reg, see 93c230e3f5bd6 */
@@ -14371,8 +14439,43 @@ static int fixup_call_args(struct bpf_verifier_env *env)
      return err;
 }
 
+static int do_kfunc_fixups(struct bpf_verifier_env *env, struct bpf_insn *insn,
+                        s32 imm, int insn_idx, int delta)
+{
+     struct bpf_insn insn_buf[16];
+     struct bpf_prog *new_prog;
+     int cnt;
+
+     /* No need to lookup btf, only vmlinux kfuncs are supported for special
+      * kfuncs handling. Hence when insn->off is zero, check if it is a
+      * special kfunc by hardcoding btf as btf_vmlinux.
+      */
+     if (!insn->off && is_kfunc_special(btf_vmlinux, insn->imm, bpf_kptr_alloc)) {
+             u64 local_type_size = env->insn_aux_data[insn_idx + delta].kptr_alloc_size;
+
+             insn_buf[0] = BPF_MOV64_IMM(BPF_REG_1, local_type_size);
+             insn_buf[1] = *insn;
+             cnt = 2;
+
+             new_prog = bpf_patch_insn_data(env, insn_idx + delta, insn_buf, cnt);
+             if (!new_prog)
+                     return -ENOMEM;
+
+             delta += cnt - 1;
+             insn = new_prog->insnsi + insn_idx + delta;
+             goto patch_call_imm;
+     }
+
+     insn->imm = imm;
+     return 0;
+patch_call_imm:
+     insn->imm = imm;
+     return cnt - 1;
+}
+
 static int fixup_kfunc_call(struct bpf_verifier_env *env,
-                         struct bpf_insn *insn)
+                         struct bpf_insn *insn,
+                         int insn_idx, int delta)
 {
      const struct bpf_kfunc_desc *desc;
 
@@ -14391,9 +14494,7 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env,
              return -EFAULT;
      }
 
-     insn->imm = desc->imm;
-
-     return 0;
+     return do_kfunc_fixups(env, insn, desc->imm, insn_idx, delta);
 }
 
 /* Do various post-verification rewrites in a single program pass.
@@ -14534,9 +14635,18 @@ static int do_misc_fixups(struct bpf_verifier_env *env)
              if (insn->src_reg == BPF_PSEUDO_CALL)
                      continue;
              if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL) {
-                     ret = fixup_kfunc_call(env, insn);
-                     if (ret)
+                     ret = fixup_kfunc_call(env, insn, i, delta);
+                     if (ret < 0)

kernel/bpf/verifier.c static int do_misc_fixups(struct bpf_verifier_env *env) · line 14649

Quoted 10 lines
kernel/bpf/verifier.c +9 -0
                              return ret;
+                     /* If ret > 0, fixup_kfunc_call did some instruction
+                      * rewrites. Increment delta, reload prog and insn,
+                      * env->prog is already set by it to the new_prog.
+                      */
+                     if (ret) {
+                             delta += ret;
+                             prog = env->prog;
+                             insn = prog->insnsi + i + delta;
+                     }

See how Yonghong did it:
https://lore.kernel.org/all/[email protected]/

Quoted 3 lines

It's cleaner to patch and adjust here instead of patch in one place and adjust in another.

Agreed, will fix it in v1.

author maintainer
#25 lore raw

RFC PATCH 15/32 bpf: Add helper macro bpf_expr_for_each_reg_in_vstate

In reply to Kumar Kartikeya Dwivedi

For a lot of use cases in future patches, we will want to modify the state of registers part of some same 'group' (e.g. same ref_obj_id). It won't just be limited to releasing reference state, but setting a type flag dynamically based on certain actions, etc.

Hence, we need a way to easily pass a callback to the function that iterates over all registers in current bpf_verifier_state in all frames upto (and including) the curframe.

While in C++ we would be able to easily use a lambda to pass state and the callback together, sadly we aren't using C++ in the kernel. The next best thing to avoid defining a function for each case seems like statement expressions in GNU C. The kernel already uses them heavily, hence they can passed to the macro in the style of a lambda. The statement expression will then be substituted in the for loop bodies.

Variables __state and __reg are set to current bpf_func_state and reg for each invocation of the expression inside the passed in verifier state.

Then, convert mark_ptr_or_null_regs, clear_all_pkt_pointers, release_reference, find_good_pkt_pointers, find_equal_scalars to use bpf_expr_for_each_reg_in_vstate.

---
 include/linux/bpf_verifier.h |  21 ++++++
 kernel/bpf/verifier.c        | 135 ++++++++---------------------------
 2 files changed, 49 insertions(+), 107 deletions(-)
include/linux/bpf_verifier.h +21 -0
index c6d550978d63..73d9443d0074 100644
@@ -354,6 +354,27 @@ struct bpf_verifier_state {
 	     iter < frame->allocated_stack / BPF_REG_SIZE;		\
 	     iter++, reg = bpf_get_spilled_reg(iter, frame))
 
+/* Invoke __expr over regsiters in __vst, setting __state and __reg */
+#define bpf_expr_for_each_reg_in_vstate(__vst, __state, __reg, __expr)   \
+	({                                                               \
+		struct bpf_verifier_state *___vstate = __vst;            \
+		int ___i, ___j;                                          \
+		for (___i = 0; ___i <= ___vstate->curframe; ___i++) {    \
+			struct bpf_reg_state *___regs;                   \
+			__state = ___vstate->frame[___i];                \
+			___regs = __state->regs;                         \
+			for (___j = 0; ___j < MAX_BPF_REG; ___j++) {     \
+				__reg = &___regs[___j];                  \
+				(void)(__expr);                          \
+			}                                                \
+			bpf_for_each_spilled_reg(___j, __state, __reg) { \
+				if (!__reg)                              \
+					continue;                        \
+				(void)(__expr);                          \
+			}                                                \
+		}                                                        \
+	})
+
 /* linked list of verifier states used to prune search */
 struct bpf_verifier_state_list {
 	struct bpf_verifier_state state;
kernel/bpf/verifier.c +28 -107

213 lines not on this page · raw message

Signature · 1 line
2.34.1
#26 lore raw

Re: [PATCH RFC bpf-next v1 15/32] bpf: Add helper macro bpf_expr_for_each_reg_in_vstate

Quoted 35 lines · Kumar Kartikeya Dwivedi

For a lot of use cases in future patches, we will want to modify the state of registers part of some same 'group' (e.g. same ref_obj_id). It won't just be limited to releasing reference state, but setting a type flag dynamically based on certain actions, etc.

Hence, we need a way to easily pass a callback to the function that iterates over all registers in current bpf_verifier_state in all frames upto (and including) the curframe.

While in C++ we would be able to easily use a lambda to pass state and the callback together, sadly we aren't using C++ in the kernel. The next best thing to avoid defining a function for each case seems like statement expressions in GNU C. The kernel already uses them heavily, hence they can passed to the macro in the style of a lambda. The statement expression will then be substituted in the for loop bodies.

Variables __state and __reg are set to current bpf_func_state and reg for each invocation of the expression inside the passed in verifier state.

Then, convert mark_ptr_or_null_regs, clear_all_pkt_pointers, release_reference, find_good_pkt_pointers, find_equal_scalars to use bpf_expr_for_each_reg_in_vstate.

---
 include/linux/bpf_verifier.h |  21 ++++++
 kernel/bpf/verifier.c        | 135 ++++++++---------------------------
 2 files changed, 49 insertions(+), 107 deletions(-)
include/linux/bpf_verifier.h +0 -0
index c6d550978d63..73d9443d0074 100644

include/linux/bpf_verifier.h struct bpf_verifier_state { · line 358

Quoted 5 lines
include/linux/bpf_verifier.h +2 -0
@@ -354,6 +354,27 @@ struct bpf_verifier_state {
 	     iter < frame->allocated_stack / BPF_REG_SIZE;		\
 	     iter++, reg = bpf_get_spilled_reg(iter, frame))
 
+/* Invoke __expr over regsiters in __vst, setting __state and __reg */
+#define bpf_expr_for_each_reg_in_vstate(__vst, __state, __reg, __expr)   \

Very nice.
I renamed it to bpf_for_each_reg_in_vstate to make it less verbose and more consistent with existing bpf_for_each_spilled_reg. And applied to bpf-next.

author maintainer
#27 lore raw

RFC PATCH 16/32 bpf: Introduce BPF memory object model

In reply to Kumar Kartikeya Dwivedi

Add the concept of a memory object model to BPF verifier.

What this means is that there are now some types that are not just plain old data, but require explicit action when they are allocated on a storage, before their lifetime is considered as started and before it is allowed for them to escape the program. The verifier will track state of such fields during the various phases of the object lifetime, where it can be sure about certain invariants.

Some inspiration is taken from existing memory object and lifetime models in C and C++ which have stood the test of time. See [0], [1], [2] for more information, to find some similarities. In the future, the separation of storage and object lifetime may be made more stark by allowing to change effective type of storage allocated for a local kptr. For now, that has been left out. It is only possible when verifier understands when the program has exclusive access to storage, and when the object it is hosting is no longer accessible to other CPUs.

This can be useful to maintain size-class based freelists inside BPF programs and reuse storage of same size for different types. This would only be safe to allow if verifier can ensure that while storage lifetime has not ended, object lifetime for the current type has. This necessiates separating the two and accomodating a simple model to track object lifetime (composed recursively of more objects whose lifetime is individually tracked).

Everytime a BPF program allocates such non-trivial types, it must call a set of constructors on the object to fully begin its lifetime before it can make use of the pointer to this type. If the program does not do so, the verifier will complain and lead to failure in loading of the program.

Similarly, when ending the lifetime of such types, it is required to fully destruct the object using a series of destructors for each non-trivial member, before finally freeing the storage the object is making use of.

During both the construction and destruction phase, there can be only one program that can own and access such an object, hence their is no need of any explicit synchronization. The single ownership of such objects makes it easy for the verifier to enforce the safety around the beginning and end of the lifetime without resorting to dynamic checks.

When there are multiple fields needing construction or destruction, the program must call their constructors in ascending order of the offset of the field.

For example, consider the following type (support for such fields will be added in subsequent patches):

struct data {
	struct bpf_spin_lock lock;
	struct bpf_list_head list __contains(struct, foo, node);
	int data;
};

struct data *d = bpf_kptr_alloc(...);
if (!d) { ... }

Now, the type of d would be PTR_TO_BTF_ID | MEM_TYPE_LOCAL | OBJ_CONSTRUCTING, as it needs two constructor calls (for lock and head), before it can be considered fully initialized and alive.

Hence, we must do (in order of field offsets):

bpf_spin_lock_init(&d->lock);
bpf_list_head_init(&d->list);

Once the final constructor call that is required for the type is made, in this case bpf_list_head_init, the verifier will unmark the OBJ_CONSTRUCTING flag. Now, the type is PTR_TO_BTF_ID | MEM_TYPE_LOCAL, so the pointer can be used anywhere these local kptrs are allowed, and it can also escape the program.

The verifier ensures that the pointer can only be made visible once the construction of the object is complete.

Likewise, once the first call to destroy the non-trivial field with greatest offset is made, the verifier marks the pointer as OBJ_DESTRUCTING, ensuring that the destruction is taken to its conclusion and renders the object unusable except for destruction and consequent freeing of the storage it is occupying.

Construction is always done in ascending order of field offsets, and destruction is done in descending order of field offsets.

[0]: https://eel.is/c++draft/basic.life
"C++: Memory and Objects"
[1]: https://en.cppreference.com/w/cpp/language/lifetime
"C++: Object Lifetime"
[2]: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2318r1.pdf
"A Provenance-aware Memory Object Model for C"

---
 include/linux/bpf.h          |  15 ++
 include/linux/bpf_verifier.h |  37 ++++-
 kernel/bpf/verifier.c        | 286 +++++++++++++++++++++++++++++++++++
 3 files changed, 336 insertions(+), 2 deletions(-)
include/linux/bpf.h +15 -0
index 5c8bfb0eba17..910aa891b97a 100644
@@ -492,6 +492,21 @@ enum bpf_type_flag {
 	 */
 	MEM_TYPE_LOCAL		= BIT(11 + BPF_BASE_TYPE_BITS),
 
+	/* This is applied to PTR_TO_BTF_ID pointing to object of a local type
+	 * (also called local kptr) whose lifetime start needs explicit
+	 * constructor calls in the BPF program before it can be considered
+	 * fully intialized and ready for use, escape program, etc.
+	 */
+	OBJ_CONSTRUCTING	= BIT(12 + BPF_BASE_TYPE_BITS),
+
+	/* This is applied to PTR_TO_BTF_ID pointing to object of a local type
+	 * (also called local kptr) whose lifetime has ended officially and it
+	 * needs destructor calls to be invoked in the BPF program that has
+	 * final ownership of its storage before it can be released back to the
+	 * memory allocator.
+	 */
+	OBJ_DESTRUCTING		= BIT(13 + BPF_BASE_TYPE_BITS),
+
 	__BPF_TYPE_FLAG_MAX,
 	__BPF_TYPE_LAST_FLAG	= __BPF_TYPE_FLAG_MAX - 1,
 };
include/linux/bpf_verifier.h +35 -2
index 73d9443d0074..2a9dcefca3b6 100644
@@ -49,6 +49,15 @@ enum bpf_reg_precise {
 	PRECISE_ABSOLUTE,
 };
 
+enum {
+	FIELD_STATE_UNKNOWN	  = 0,
+	FIELD_STATE_CONSTRUCTED   = 1,
+	FIELD_STATE_DESTRUCTED	  = 2,
+	/* We only have room for one more state */
+	FIELD_STATE_MAX,
+};
+static_assert(FIELD_STATE_MAX <= (1 << 2));
+
 struct bpf_reg_state {
 	/* Ordering of fields matters.  See states_equal() */
 	enum bpf_reg_type type;
@@ -74,6 +83,17 @@ struct bpf_reg_state {
 		struct {
 			struct btf *btf;
 			u32 btf_id;
+			/* In case of PTR_TO_BTF_ID to a local type, sometimes
+			 * it may embed some special kernel types that we need
+			 * to track the state of.
+			 * To save space, we use 2 bits per field for state
+			 * tracking, and so have room for 16 fields. The special
+			 * field with lowest offset takes first two bits,
+			 * special field with second lowest offset takes next
+			 * two bits, and so on. The mapping can be determined
+			 * each time we encounter the type.
+			 */
+			u32 states;
 		};
 
 		u32 mem_size; /* for PTR_TO_MEM | PTR_TO_MEM_OR_NULL */
@@ -92,8 +112,8 @@ struct bpf_reg_state {
 
 		/* Max size from any of the above. */
 		struct {
-			unsigned long raw1;
-			unsigned long raw2;
+			u64 raw1;
+			u64 raw2;
 		} raw;
 
 		u32 subprogno; /* for PTR_TO_FUNC */
@@ -645,4 +665,17 @@ static inline enum bpf_prog_type resolve_prog_type(struct bpf_prog *prog)
 		prog->aux->dst_prog->type : prog->type;
 }
 
+static inline int local_kptr_get_state(struct bpf_reg_state *reg, u8 index)
+{
+	WARN_ON_ONCE(index >= 16);
+	return (reg->states >> (2 * index)) & 0x3;
+}
+
+static inline void local_kptr_set_state(struct bpf_reg_state *reg, u8 index, u32 state)
+{
+	WARN_ON_ONCE(state >= FIELD_STATE_MAX);
+	reg->states &= ~(0x3UL << (2 * index));
+	reg->states |= (state << (2 * index));
+}
+
 #endif /* _LINUX_BPF_VERIFIER_H */
kernel/bpf/verifier.c +286 -0

334 lines not on this page · raw message

Signature · 1 line
2.34.1
#28 lore raw

Re: [PATCH RFC bpf-next v1 16/32] bpf: Introduce BPF memory object model

Quoted 59 lines · Kumar Kartikeya Dwivedi

Add the concept of a memory object model to BPF verifier.

What this means is that there are now some types that are not just plain old data, but require explicit action when they are allocated on a storage, before their lifetime is considered as started and before it is allowed for them to escape the program. The verifier will track state of such fields during the various phases of the object lifetime, where it can be sure about certain invariants.

Some inspiration is taken from existing memory object and lifetime models in C and C++ which have stood the test of time. See [0], [1], [2] for more information, to find some similarities. In the future, the separation of storage and object lifetime may be made more stark by allowing to change effective type of storage allocated for a local kptr. For now, that has been left out. It is only possible when verifier understands when the program has exclusive access to storage, and when the object it is hosting is no longer accessible to other CPUs.

This can be useful to maintain size-class based freelists inside BPF programs and reuse storage of same size for different types. This would only be safe to allow if verifier can ensure that while storage lifetime has not ended, object lifetime for the current type has. This necessiates separating the two and accomodating a simple model to track object lifetime (composed recursively of more objects whose lifetime is individually tracked).

Everytime a BPF program allocates such non-trivial types, it must call a set of constructors on the object to fully begin its lifetime before it can make use of the pointer to this type. If the program does not do so, the verifier will complain and lead to failure in loading of the program.

Similarly, when ending the lifetime of such types, it is required to fully destruct the object using a series of destructors for each non-trivial member, before finally freeing the storage the object is making use of.

During both the construction and destruction phase, there can be only one program that can own and access such an object, hence their is no need of any explicit synchronization. The single ownership of such objects makes it easy for the verifier to enforce the safety around the beginning and end of the lifetime without resorting to dynamic checks.

When there are multiple fields needing construction or destruction, the program must call their constructors in ascending order of the offset of the field.

For example, consider the following type (support for such fields will be added in subsequent patches):

struct data {
	struct bpf_spin_lock lock;
	struct bpf_list_head list __contains(struct, foo, node);
	int data;
};

struct data *d = bpf_kptr_alloc(...);
if (!d) { ... }

Quoted 8 lines

Now, the type of d would be PTR_TO_BTF_ID | MEM_TYPE_LOCAL | OBJ_CONSTRUCTING, as it needs two constructor calls (for lock and head), before it can be considered fully initialized and alive.

Hence, we must do (in order of field offsets):

bpf_spin_lock_init(&d->lock);
bpf_list_head_init(&d->list);

All sounds great in theory, but I think it's unnecessary complex at this point. There is still a need to __bpf_list_head_init_zeroed as seen in later patches. So all this verifier enforced constructors we don't need _today_. Zero init of everything works.
It's the case for list_head, list_node, spin_lock, rb_root, rb_node. Pretty much all new data structures will work with zero init
and all of them need async dtors.
The verifier cannot help during destruction.
dtors have to be specified declaratively in a bpf prog for new types and as known kfuncs for list_head/node, rb_root/node.
There will be unfreed link lists in maps and the later patches handle that without OBJ_DESTRUCTING.
So let's postpone this patch.

author maintainer
#29 lore raw

Re: [PATCH RFC bpf-next v1 16/32] bpf: Introduce BPF memory object model

Quoted 65 lines · Alexei Starovoitov
Quoted 63 lines · Kumar Kartikeya Dwivedi

Add the concept of a memory object model to BPF verifier.

What this means is that there are now some types that are not just plain old data, but require explicit action when they are allocated on a storage, before their lifetime is considered as started and before it is allowed for them to escape the program. The verifier will track state of such fields during the various phases of the object lifetime, where it can be sure about certain invariants.

Some inspiration is taken from existing memory object and lifetime models in C and C++ which have stood the test of time. See [0], [1], [2] for more information, to find some similarities. In the future, the separation of storage and object lifetime may be made more stark by allowing to change effective type of storage allocated for a local kptr. For now, that has been left out. It is only possible when verifier understands when the program has exclusive access to storage, and when the object it is hosting is no longer accessible to other CPUs.

This can be useful to maintain size-class based freelists inside BPF programs and reuse storage of same size for different types. This would only be safe to allow if verifier can ensure that while storage lifetime has not ended, object lifetime for the current type has. This necessiates separating the two and accomodating a simple model to track object lifetime (composed recursively of more objects whose lifetime is individually tracked).

Everytime a BPF program allocates such non-trivial types, it must call a set of constructors on the object to fully begin its lifetime before it can make use of the pointer to this type. If the program does not do so, the verifier will complain and lead to failure in loading of the program.

Similarly, when ending the lifetime of such types, it is required to fully destruct the object using a series of destructors for each non-trivial member, before finally freeing the storage the object is making use of.

During both the construction and destruction phase, there can be only one program that can own and access such an object, hence their is no need of any explicit synchronization. The single ownership of such objects makes it easy for the verifier to enforce the safety around the beginning and end of the lifetime without resorting to dynamic checks.

When there are multiple fields needing construction or destruction, the program must call their constructors in ascending order of the offset of the field.

For example, consider the following type (support for such fields will be added in subsequent patches):

struct data {
      struct bpf_spin_lock lock;
      struct bpf_list_head list __contains(struct, foo, node);
      int data;
};

struct data *d = bpf_kptr_alloc(...);
if (!d) { ... }

Now, the type of d would be PTR_TO_BTF_ID | MEM_TYPE_LOCAL | OBJ_CONSTRUCTING, as it needs two constructor calls (for lock and head), before it can be considered fully initialized and alive.

Quoted 7 lines
Quoted 4 lines

Hence, we must do (in order of field offsets):

bpf_spin_lock_init(&d->lock);
bpf_list_head_init(&d->list);

All sounds great in theory, but I think it's unnecessary complex at this point. There is still a need to __bpf_list_head_init_zeroed as seen in later patches.

This particular call is only because of map values. INIT_LIST_HEAD for prealloc init or alloc_elem would be costly.
There won't be any concern to do it in check_and_init_map_value, we zero out the field there already. Nothing else needs this check.

List helpers I am planning to inline, it doesn't make sense to have two loads/stores inside kfuncs. And then for local kptrs there is no need to zero init. pop_front/pop_back are even uglier. There you need NULL check + zero init, _then_ check for list_empty. Same with future list_splice.

I don't believe list helpers are going to be so infrequent such that all this might not matter at all.

But fine, I still consider this a fair point. I thought a lot about this too.

It really boils down to: do we really want to always zero init?

What seems more desirable to me is forcing initialization like this, esp. since memory reuse is going to be the more common case, and then simply relaxing initialization when we know it comes from bpf_kptr_zalloc. needs_construction similar to needs_destruction. We aren't requiring bpf_list_node_fini, same idea there.

Zeroing the entire big struct vs zeroing/initing two fields makes a huge difference.

Quoted 7 lines

So all this verifier enforced constructors we don't need _today_. Zero init of everything works.
It's the case for list_head, list_node, spin_lock, rb_root, rb_node. Pretty much all new data structures will work with zero init and all of them need async dtors.
The verifier cannot help during destruction.
dtors have to be specified declaratively in a bpf prog for new types

I think about it the other way around.

There actually isn't a need to specify any dtor IMO for custom types. Just init and free your type inline. Much more familiar to people doing C already.
Custom types are always just data without special fields, and we know how to destroy BPF special fields.
Map already knows how to 'destruct' these types, just like it has to know how to destruct map value.

map value type and local kptr type are similar in that regard. They are both local types in prog BTF with special fields.
If it can do it for map value, it can do it for local kptr if it finds it in map (it has to).

To me taking prog reference and setting up per-type dtor is the uglier solution. It's unnecessary for the user. That then forces you to have similar semantics like bpf_timer. map_release_uref will be used to break the reference cycle between map and prog, which is undesirable. More effort then - to think about some way to alleviate that, or live with it and compromise.

Later, asynchronous destruction (RCU case where it won't be done immediately) is just setting reg->states for all fields as FIELD_STATE_CONSTRUCTED for reg in callback, but type as OBJ_DESTRUCTING, forcing you to do nothing but unwind and free in that context.

The real reason to give destruction control to users for local kptrs is the ability to manage what to do with drained resources. They might as well splice out their list when freeing a node to a local list_head, or move it to a map. Same with more cases in the future (kptr inside kptr).

It shouldn't be invoked on bpf_kptr_free automagically. That is the job of the language and best suited to that.
Verifier will see BPF ASM after translation from C/C++/Rust/etc., so for us the destruction at language level appears as the destructing phase of local kptr in verifier. For maps it's the last resort, where programs are already gone, so there is nothing left to do but free stuff.

Quoted 4 lines

and as known kfuncs for list_head/node, rb_root/node.
There will be unfreed link lists in maps and the later patches handle that without OBJ_DESTRUCTING.
So let's postpone this patch.

#30 lore raw

Re: [PATCH RFC bpf-next v1 16/32] bpf: Introduce BPF memory object model

Quoted 80 lines · Kumar Kartikeya Dwivedi
Quoted 65 lines · Alexei Starovoitov
Quoted 63 lines · Kumar Kartikeya Dwivedi

Add the concept of a memory object model to BPF verifier.

What this means is that there are now some types that are not just plain old data, but require explicit action when they are allocated on a storage, before their lifetime is considered as started and before it is allowed for them to escape the program. The verifier will track state of such fields during the various phases of the object lifetime, where it can be sure about certain invariants.

Some inspiration is taken from existing memory object and lifetime models in C and C++ which have stood the test of time. See [0], [1], [2] for more information, to find some similarities. In the future, the separation of storage and object lifetime may be made more stark by allowing to change effective type of storage allocated for a local kptr. For now, that has been left out. It is only possible when verifier understands when the program has exclusive access to storage, and when the object it is hosting is no longer accessible to other CPUs.

This can be useful to maintain size-class based freelists inside BPF programs and reuse storage of same size for different types. This would only be safe to allow if verifier can ensure that while storage lifetime has not ended, object lifetime for the current type has. This necessiates separating the two and accomodating a simple model to track object lifetime (composed recursively of more objects whose lifetime is individually tracked).

Everytime a BPF program allocates such non-trivial types, it must call a set of constructors on the object to fully begin its lifetime before it can make use of the pointer to this type. If the program does not do so, the verifier will complain and lead to failure in loading of the program.

Similarly, when ending the lifetime of such types, it is required to fully destruct the object using a series of destructors for each non-trivial member, before finally freeing the storage the object is making use of.

During both the construction and destruction phase, there can be only one program that can own and access such an object, hence their is no need of any explicit synchronization. The single ownership of such objects makes it easy for the verifier to enforce the safety around the beginning and end of the lifetime without resorting to dynamic checks.

When there are multiple fields needing construction or destruction, the program must call their constructors in ascending order of the offset of the field.

For example, consider the following type (support for such fields will be added in subsequent patches):

struct data {
      struct bpf_spin_lock lock;
      struct bpf_list_head list __contains(struct, foo, node);
      int data;
};

struct data *d = bpf_kptr_alloc(...);
if (!d) { ... }

Now, the type of d would be PTR_TO_BTF_ID | MEM_TYPE_LOCAL | OBJ_CONSTRUCTING, as it needs two constructor calls (for lock and head), before it can be considered fully initialized and alive.

Quoted 7 lines
Quoted 4 lines

Hence, we must do (in order of field offsets):

bpf_spin_lock_init(&d->lock);
bpf_list_head_init(&d->list);

All sounds great in theory, but I think it's unnecessary complex at this point. There is still a need to __bpf_list_head_init_zeroed as seen in later patches.

This particular call is only because of map values. INIT_LIST_HEAD for prealloc init or alloc_elem would be costly.
There won't be any concern to do it in check_and_init_map_value, we zero out the field there already. Nothing else needs this check.

Quoted 5 lines

List helpers I am planning to inline, it doesn't make sense to have two loads/stores inside kfuncs. And then for local kptrs there is no need to zero init. pop_front/pop_back are even uglier. There you need NULL check + zero init, _then_ check for list_empty. Same with future list_splice.

The inlining is an orthogonal topic.
It doesn't have to be done the way of map_gen_lookup().

Quoted 6 lines

I don't believe list helpers are going to be so infrequent such that all this might not matter at all.

But fine, I still consider this a fair point. I thought a lot about this too.

It really boils down to: do we really want to always zero init?

Special fields like locks, timers, lists, trees -> yes.

Quoted 9 lines

What seems more desirable to me is forcing initialization like this, esp. since memory reuse is going to be the more common case, and then simply relaxing initialization when we know it comes from bpf_kptr_zalloc. needs_construction similar to needs_destruction. We aren't requiring bpf_list_node_fini, same idea there.

Zeroing the entire big struct vs zeroing/initing two fields makes a huge difference.

Right, but too many assumptions in this reasoning.
I wasn't proposing to do bzero the whole sizeof(struct foo) in bpf_kptr_zalloc. I wasn't proposing to have zalloc flavor either.
We can do selective zeroing.
The prog will call bpf_kptr_alloc and bpf_kptr_free,
but it doesn't have to be the same kfunc-s for all btf types.
We can substitute kfuncs with custom implicit dtors and ctors based on type info. Sort of like C++ calls constructors in operator new.
But here we can go pretty far with _implicit_ ctors/dtors only.

Quoted 23 lines
Quoted 7 lines

So all this verifier enforced constructors we don't need _today_. Zero init of everything works.
It's the case for list_head, list_node, spin_lock, rb_root, rb_node. Pretty much all new data structures will work with zero init and all of them need async dtors.
The verifier cannot help during destruction.
dtors have to be specified declaratively in a bpf prog for new types

I think about it the other way around.

There actually isn't a need to specify any dtor IMO for custom types. Just init and free your type inline. Much more familiar to people doing C already.
Custom types are always just data without special fields, and we know how to destroy BPF special fields.
Map already knows how to 'destruct' these types, just like it has to know how to destruct map value.

map value type and local kptr type are similar in that regard. They are both local types in prog BTF with special fields.
If it can do it for map value, it can do it for local kptr if it finds it in map (it has to).

Quoted 6 lines

To me taking prog reference and setting up per-type dtor is the uglier solution. It's unnecessary for the user. That then forces you to have similar semantics like bpf_timer. map_release_uref will be used to break the reference cycle between map and prog, which is undesirable. More effort then - to think about some way to alleviate that, or live with it and compromise.

Completely agree. I think explicit (bpf prog provided) dtor is an extreme case. Hopefully we won't need to add support for it for long time.
The verifier should be able to do implicit ctor/dtor based on BTF only and that will allow us to build pretty complex data structures with rbtrees, link lists, etc.

The main point is dtor of bpf_list_head in map value has to be implicit anyway. The prog can do:
struct foo {
struct bpf_list_head head;
struct bpf_spin_lock lock;
};

bpf_list_lock
bpf_list_add(&val->head, ...);
bpf_list_unlock
exit

The map will have elements allocated and these elements will contain kptrs and populated link lists and rbtrees.
The bpf infra has to be able to free all these things automatically based on BTF and it obviously can do so.
Since it has to do it anyway we can allow:
foo_ptr = bpf_kptr_xchg(...);
bpf_kptr_free(foo_ptr);
and that free function will do the same implicit destruction of the link list which will include walking the list and deleting elements recursively.
Maybe it means that it would have to grab the locks automatically as well. Not sure. For single owner case locks won't be needed.

Quoted 7 lines

It shouldn't be invoked on bpf_kptr_free automagically. That is the job of the language and best suited to that.
Verifier will see BPF ASM after translation from C/C++/Rust/etc., so for us the destruction at language level appears as the destructing phase of local kptr in verifier. For maps it's the last resort, where programs are already gone, so there is nothing left to do but free stuff.

The C++ compiler generated ctors/dtors sequences instructs "dumb" cpu execute them. We have the verifier in-between and the whole run-time. The map destruction case is not "last resort". It's a feature provided by the bpf run-time. Just like golang garbage collector is not a "last resort".

Anyway back to original point which is:
we don't have to add support to the verifier to enforce explicit ctor/dtor sequences. We can solve practical use cases without that additional complexity. There are plenty of other complex things in this patch set.

author maintainer
#31 lore raw

Re: [PATCH RFC bpf-next v1 16/32] bpf: Introduce BPF memory object model

Quoted 168 lines · Alexei Starovoitov
Quoted 80 lines · Kumar Kartikeya Dwivedi
Quoted 65 lines · Alexei Starovoitov
Quoted 63 lines · Kumar Kartikeya Dwivedi

Add the concept of a memory object model to BPF verifier.

What this means is that there are now some types that are not just plain old data, but require explicit action when they are allocated on a storage, before their lifetime is considered as started and before it is allowed for them to escape the program. The verifier will track state of such fields during the various phases of the object lifetime, where it can be sure about certain invariants.

Some inspiration is taken from existing memory object and lifetime models in C and C++ which have stood the test of time. See [0], [1], [2] for more information, to find some similarities. In the future, the separation of storage and object lifetime may be made more stark by allowing to change effective type of storage allocated for a local kptr. For now, that has been left out. It is only possible when verifier understands when the program has exclusive access to storage, and when the object it is hosting is no longer accessible to other CPUs.

This can be useful to maintain size-class based freelists inside BPF programs and reuse storage of same size for different types. This would only be safe to allow if verifier can ensure that while storage lifetime has not ended, object lifetime for the current type has. This necessiates separating the two and accomodating a simple model to track object lifetime (composed recursively of more objects whose lifetime is individually tracked).

Everytime a BPF program allocates such non-trivial types, it must call a set of constructors on the object to fully begin its lifetime before it can make use of the pointer to this type. If the program does not do so, the verifier will complain and lead to failure in loading of the program.

Similarly, when ending the lifetime of such types, it is required to fully destruct the object using a series of destructors for each non-trivial member, before finally freeing the storage the object is making use of.

During both the construction and destruction phase, there can be only one program that can own and access such an object, hence their is no need of any explicit synchronization. The single ownership of such objects makes it easy for the verifier to enforce the safety around the beginning and end of the lifetime without resorting to dynamic checks.

When there are multiple fields needing construction or destruction, the program must call their constructors in ascending order of the offset of the field.

For example, consider the following type (support for such fields will be added in subsequent patches):

struct data {
      struct bpf_spin_lock lock;
      struct bpf_list_head list __contains(struct, foo, node);
      int data;
};

struct data *d = bpf_kptr_alloc(...);
if (!d) { ... }

Now, the type of d would be PTR_TO_BTF_ID | MEM_TYPE_LOCAL | OBJ_CONSTRUCTING, as it needs two constructor calls (for lock and head), before it can be considered fully initialized and alive.

Quoted 7 lines
Quoted 4 lines

Hence, we must do (in order of field offsets):

bpf_spin_lock_init(&d->lock);
bpf_list_head_init(&d->list);

All sounds great in theory, but I think it's unnecessary complex at this point. There is still a need to __bpf_list_head_init_zeroed as seen in later patches.

This particular call is only because of map values. INIT_LIST_HEAD for prealloc init or alloc_elem would be costly.
There won't be any concern to do it in check_and_init_map_value, we zero out the field there already. Nothing else needs this check.

Quoted 5 lines

List helpers I am planning to inline, it doesn't make sense to have two loads/stores inside kfuncs. And then for local kptrs there is no need to zero init. pop_front/pop_back are even uglier. There you need NULL check + zero init, _then_ check for list_empty. Same with future list_splice.

The inlining is an orthogonal topic.
It doesn't have to be done the way of map_gen_lookup().

Quoted 6 lines

I don't believe list helpers are going to be so infrequent such that all this might not matter at all.

But fine, I still consider this a fair point. I thought a lot about this too.

It really boils down to: do we really want to always zero init?

Special fields like locks, timers, lists, trees -> yes.

Quoted 9 lines

What seems more desirable to me is forcing initialization like this, esp. since memory reuse is going to be the more common case, and then simply relaxing initialization when we know it comes from bpf_kptr_zalloc. needs_construction similar to needs_destruction. We aren't requiring bpf_list_node_fini, same idea there.

Zeroing the entire big struct vs zeroing/initing two fields makes a huge difference.

Right, but too many assumptions in this reasoning.
I wasn't proposing to do bzero the whole sizeof(struct foo) in bpf_kptr_zalloc. I wasn't proposing to have zalloc flavor either.
We can do selective zeroing.
The prog will call bpf_kptr_alloc and bpf_kptr_free,
but it doesn't have to be the same kfunc-s for all btf types.
We can substitute kfuncs with custom implicit dtors and ctors based on type info. Sort of like C++ calls constructors in operator new.
But here we can go pretty far with _implicit_ ctors/dtors only.

Quoted 23 lines
Quoted 7 lines

So all this verifier enforced constructors we don't need _today_. Zero init of everything works.
It's the case for list_head, list_node, spin_lock, rb_root, rb_node. Pretty much all new data structures will work with zero init and all of them need async dtors.
The verifier cannot help during destruction.
dtors have to be specified declaratively in a bpf prog for new types

I think about it the other way around.

There actually isn't a need to specify any dtor IMO for custom types. Just init and free your type inline. Much more familiar to people doing C already.
Custom types are always just data without special fields, and we know how to destroy BPF special fields.
Map already knows how to 'destruct' these types, just like it has to know how to destruct map value.

map value type and local kptr type are similar in that regard. They are both local types in prog BTF with special fields.
If it can do it for map value, it can do it for local kptr if it finds it in map (it has to).

Quoted 6 lines

To me taking prog reference and setting up per-type dtor is the uglier solution. It's unnecessary for the user. That then forces you to have similar semantics like bpf_timer. map_release_uref will be used to break the reference cycle between map and prog, which is undesirable. More effort then - to think about some way to alleviate that, or live with it and compromise.

Completely agree. I think explicit (bpf prog provided) dtor is an extreme case. Hopefully we won't need to add support for it for long time.
The verifier should be able to do implicit ctor/dtor based on BTF only and that will allow us to build pretty complex data structures with rbtrees, link lists, etc.

The main point is dtor of bpf_list_head in map value has to be implicit anyway. The prog can do:
struct foo {
struct bpf_list_head head;
struct bpf_spin_lock lock;
};

bpf_list_lock
bpf_list_add(&val->head, ...);
bpf_list_unlock
exit

Quoted 13 lines

The map will have elements allocated and these elements will contain kptrs and populated link lists and rbtrees.
The bpf infra has to be able to free all these things automatically based on BTF and it obviously can do so.
Since it has to do it anyway we can allow:
foo_ptr = bpf_kptr_xchg(...);
bpf_kptr_free(foo_ptr);
and that free function will do the same implicit destruction of the link list which will include walking the list and deleting elements recursively.
Maybe it means that it would have to grab the locks automatically as well. Not sure. For single owner case locks won't be needed.

Nope, no locks will be held whenever bpf_kptr_free is called. At that point, concurrency wrt BPF special fields is always zero. Even with bpf_refcount you will call it in true branch of if (bpf_refcount_put(...)) { bpf_kptr_free(kptr); }

For RCU protected ones, lock may be held without refcount to e.g. manipulate data, but manipulation of BPF fields will require refcount, to protect against concurrent destruction.

Quoted 13 lines
Quoted 7 lines

It shouldn't be invoked on bpf_kptr_free automagically. That is the job of the language and best suited to that.
Verifier will see BPF ASM after translation from C/C++/Rust/etc., so for us the destruction at language level appears as the destructing phase of local kptr in verifier. For maps it's the last resort, where programs are already gone, so there is nothing left to do but free stuff.

The C++ compiler generated ctors/dtors sequences instructs "dumb" cpu execute them. We have the verifier in-between and the whole run-time. The map destruction case is not "last resort". It's a feature provided by the bpf run-time. Just like golang garbage collector is not a "last resort".

Quoted 4 lines

Anyway back to original point which is:
we don't have to add support to the verifier to enforce explicit ctor/dtor sequences. We can solve practical use cases without that additional complexity. There are plenty of other complex things in this patch set.

I slept over this. I think I can get behind this idea of implicit ctor/dtor. We might have open coded construction/destruction later if we want.

I am however thinking of naming these helpers:
bpf_kptr_new
bpf_kptr_delete
to make it clear it does a little more than just allocating the type. The open coded cases can later derive their allocation from the more bare bones bpf_kptr_alloc instead in the future.

The main reason to have open coded-ness was being able to 'manage' resources once visibility reduces to current CPU (bpf_refcount_put, single ownership after xchg, etc.). Even with RCU, we won't allow touching the BPF special fields without refcount. bpf_spin_lock is different, as it protects more than just bpf special fields.

But one can still splice or kptr_xchg before passing to bpf_kptr_free to do that. bpf_kptr_free is basically cleaning up whatever is left by then, forcefully. In the future, we might even be able to do elision of implicit dtors based on the seen data flow (splicing in single ownership implies list is empty, any other op will undo that, etc.) if there are big structs with too many fields. Can also support that in open coded cases.

What I want to think about more is whether we should still force calling bpf_refcount_set vs always setting it to 1.

I know we don't agree about whether list_add in shared mode should take ref vs transfer ref. I'm leaning towards transfer since that will be most intuitive. It then works the same way in both cases, single ownership only transfers the sole reference you have, so you lose access, but in shared you may have more than one. If you have just one you will still lose access.

It will be odd for list_add to consume it in one case and not the other. People should already be fully conscious of how they are managing the lifetime of their object.

It then seems better to require users to set the initial refcount themselves. When doing the initial linking it can be very cheap. Later get/put/inc are always available.

But forcing it to be called is going to be much simpler than this patch.

#32 lore raw

Re: [PATCH RFC bpf-next v1 16/32] bpf: Introduce BPF memory object model

Quoted 11 lines · Kumar Kartikeya Dwivedi

I slept over this. I think I can get behind this idea of implicit ctor/dtor. We might have open coded construction/destruction later if we want.

I am however thinking of naming these helpers:
bpf_kptr_new
bpf_kptr_delete
to make it clear it does a little more than just allocating the type. The open coded cases can later derive their allocation from the more bare bones bpf_kptr_alloc instead in the future.

New names make complete sense. Good idea.

Quoted 6 lines

The main reason to have open coded-ness was being able to 'manage' resources once visibility reduces to current CPU (bpf_refcount_put, single ownership after xchg, etc.). Even with RCU, we won't allow touching the BPF special fields without refcount. bpf_spin_lock is different, as it protects more than just bpf special fields.

Quoted 7 lines

But one can still splice or kptr_xchg before passing to bpf_kptr_free to do that. bpf_kptr_free is basically cleaning up whatever is left by then, forcefully. In the future, we might even be able to do elision of implicit dtors based on the seen data flow (splicing in single ownership implies list is empty, any other op will undo that, etc.) if there are big structs with too many fields. Can also support that in open coded cases.

Right.

Quoted 15 lines

What I want to think about more is whether we should still force calling bpf_refcount_set vs always setting it to 1.

I know we don't agree about whether list_add in shared mode should take ref vs transfer ref. I'm leaning towards transfer since that will be most intuitive. It then works the same way in both cases, single ownership only transfers the sole reference you have, so you lose access, but in shared you may have more than one. If you have just one you will still lose access.

It will be odd for list_add to consume it in one case and not the other. People should already be fully conscious of how they are managing the lifetime of their object.

Quoted 5 lines

It then seems better to require users to set the initial refcount themselves. When doing the initial linking it can be very cheap. Later get/put/inc are always available.

But forcing it to be called is going to be much simpler than this patch.

I'm not convinced yet :)
Pls hold on implementing one way or another.
Let's land the single ownership case for locks, lists, rbtrees, allocators. That's plenty of patches.
Then we can start a deeper discussion into the shared case. Whether it will be different in terms of 'lose access after list_add' is not critical to decide now. It can change in the future too.

The other reason to do implicit inits and ref count sets is to avoid fighting llvm.
obj = bpf_kptr_new();
obj->var1 = 1;
some_func(&obj->var2);
In many cases the compiler is allowed to sink stores.
If there are two calls that "init" two different fields the compiler is allowed to change the order as well
even if it doesn't see the body of the function and the function is marked as __pure. Technically initializers as pure functions. The verifier and llvm already "fight" a lot.
We gotta be very careful in the verifier and not assume that the code stays as written in C.

author maintainer
#33 lore raw

Re: [PATCH RFC bpf-next v1 16/32] bpf: Introduce BPF memory object model

Quoted 53 lines · Alexei Starovoitov
Quoted 11 lines · Kumar Kartikeya Dwivedi

I slept over this. I think I can get behind this idea of implicit ctor/dtor. We might have open coded construction/destruction later if we want.

I am however thinking of naming these helpers:
bpf_kptr_new
bpf_kptr_delete
to make it clear it does a little more than just allocating the type. The open coded cases can later derive their allocation from the more bare bones bpf_kptr_alloc instead in the future.

New names make complete sense. Good idea.

Quoted 6 lines

The main reason to have open coded-ness was being able to 'manage' resources once visibility reduces to current CPU (bpf_refcount_put, single ownership after xchg, etc.). Even with RCU, we won't allow touching the BPF special fields without refcount. bpf_spin_lock is different, as it protects more than just bpf special fields.

Quoted 7 lines

But one can still splice or kptr_xchg before passing to bpf_kptr_free to do that. bpf_kptr_free is basically cleaning up whatever is left by then, forcefully. In the future, we might even be able to do elision of implicit dtors based on the seen data flow (splicing in single ownership implies list is empty, any other op will undo that, etc.) if there are big structs with too many fields. Can also support that in open coded cases.

Right.

Quoted 15 lines

What I want to think about more is whether we should still force calling bpf_refcount_set vs always setting it to 1.

I know we don't agree about whether list_add in shared mode should take ref vs transfer ref. I'm leaning towards transfer since that will be most intuitive. It then works the same way in both cases, single ownership only transfers the sole reference you have, so you lose access, but in shared you may have more than one. If you have just one you will still lose access.

It will be odd for list_add to consume it in one case and not the other. People should already be fully conscious of how they are managing the lifetime of their object.

Quoted 5 lines

It then seems better to require users to set the initial refcount themselves. When doing the initial linking it can be very cheap. Later get/put/inc are always available.

But forcing it to be called is going to be much simpler than this patch.

Quoted 8 lines

I'm not convinced yet :)
Pls hold on implementing one way or another.
Let's land the single ownership case for locks, lists, rbtrees, allocators. That's plenty of patches.
Then we can start a deeper discussion into the shared case. Whether it will be different in terms of 'lose access after list_add' is not critical to decide now. It can change in the future too.

Right, I'm not implementing it yet. There's a lot of work left to even finish single ownership structures, then lots of testing. But it's helpful to keep thinking about future use cases while working on the current stuff, just to make sure we're not
digging ourselves into a design hole.

We have the option to undo damage here, since this is all experimental, but there's still an expectation that the API is not broken at whim. That wouldn't be very useful for users.

Quoted 1 line

The other reason to do implicit inits and ref count sets is to

I am not contesting implicit construction.
Other lists already work with zero initialization so list_head seems more of an exception.
But it's done for good reasons to avoid extra NULL checks unnecessarily, and make the implementation of list helpers more efficient and simple at the same time.

Quoted 9 lines

avoid fighting llvm.
obj = bpf_kptr_new();
obj->var1 = 1;
some_func(&obj->var2);
In many cases the compiler is allowed to sink stores.
If there are two calls that "init" two different fields the compiler is allowed to change the order as well
even if it doesn't see the body of the function and the function is marked as __pure. Technically initializers as pure functions.

But bpf_refcount_set won't be marked __pure, neither am I proposing to allow direct stores to 'set' it.
I'm not a compiler expert by any means, but AFAIK it should not be doing such reordering for functions otherwise.
What if the function inside has a memory barrier? That would completely screw up things.
It's going to have external linkage, so I don't think it can assume anything about side effects or not. So IMO this is not a good point.

Unless you're talking about some new way of inlining such helpers from the compiler side that doesn't exist yet.

Quoted 3 lines

The verifier and llvm already "fight" a lot. We gotta be very careful in the verifier and not assume that the code stays as written in C.

So will these implicit zero stores be done when we enter != NULL branch, or lazily on first access (helper arg, load, store)? This is the flip side: rewritings insns to add stores to local kptr can only happen after the NULL check, in the != NULL branch, at that point we cannot assume R1-R5 are free for use, so complicated field initialization will be uglier to do implicitly (e.g. if it involves calling functions etc.).
There are pros and cons for both.

#34 lore raw

Re: [PATCH RFC bpf-next v1 16/32] bpf: Introduce BPF memory object model

Quoted 71 lines · Kumar Kartikeya Dwivedi
Quoted 53 lines · Alexei Starovoitov
Quoted 11 lines · Kumar Kartikeya Dwivedi

I slept over this. I think I can get behind this idea of implicit ctor/dtor. We might have open coded construction/destruction later if we want.

I am however thinking of naming these helpers:
bpf_kptr_new
bpf_kptr_delete
to make it clear it does a little more than just allocating the type. The open coded cases can later derive their allocation from the more bare bones bpf_kptr_alloc instead in the future.

New names make complete sense. Good idea.

Quoted 6 lines

The main reason to have open coded-ness was being able to 'manage' resources once visibility reduces to current CPU (bpf_refcount_put, single ownership after xchg, etc.). Even with RCU, we won't allow touching the BPF special fields without refcount. bpf_spin_lock is different, as it protects more than just bpf special fields.

Quoted 7 lines

But one can still splice or kptr_xchg before passing to bpf_kptr_free to do that. bpf_kptr_free is basically cleaning up whatever is left by then, forcefully. In the future, we might even be able to do elision of implicit dtors based on the seen data flow (splicing in single ownership implies list is empty, any other op will undo that, etc.) if there are big structs with too many fields. Can also support that in open coded cases.

Right.

Quoted 15 lines

What I want to think about more is whether we should still force calling bpf_refcount_set vs always setting it to 1.

I know we don't agree about whether list_add in shared mode should take ref vs transfer ref. I'm leaning towards transfer since that will be most intuitive. It then works the same way in both cases, single ownership only transfers the sole reference you have, so you lose access, but in shared you may have more than one. If you have just one you will still lose access.

It will be odd for list_add to consume it in one case and not the other. People should already be fully conscious of how they are managing the lifetime of their object.

Quoted 5 lines

It then seems better to require users to set the initial refcount themselves. When doing the initial linking it can be very cheap. Later get/put/inc are always available.

But forcing it to be called is going to be much simpler than this patch.

Quoted 8 lines

I'm not convinced yet :)
Pls hold on implementing one way or another.
Let's land the single ownership case for locks, lists, rbtrees, allocators. That's plenty of patches.
Then we can start a deeper discussion into the shared case. Whether it will be different in terms of 'lose access after list_add' is not critical to decide now. It can change in the future too.

Right, I'm not implementing it yet. There's a lot of work left to even finish single ownership structures, then lots of testing. But it's helpful to keep thinking about future use cases while working on the current stuff, just to make sure we're not
digging ourselves into a design hole.

Quoted 3 lines

We have the option to undo damage here, since this is all experimental, but there's still an expectation that the API is not broken at whim. That wouldn't be very useful for users.

imo this part is minor.
The whole lock + list_or_rbtree in a single allocation restriction bothers me a lot more.
We will find out for sure only when wwe have a prototype of lock + list + rbtree and let folks who requested it actually code things.

Quoted 19 lines
Quoted 1 line

The other reason to do implicit inits and ref count sets is to

I am not contesting implicit construction.
Other lists already work with zero initialization so list_head seems more of an exception.
But it's done for good reasons to avoid extra NULL checks unnecessarily, and make the implementation of list helpers more efficient and simple at the same time.

Quoted 9 lines

avoid fighting llvm.
obj = bpf_kptr_new();
obj->var1 = 1;
some_func(&obj->var2);
In many cases the compiler is allowed to sink stores.
If there are two calls that "init" two different fields the compiler is allowed to change the order as well
even if it doesn't see the body of the function and the function is marked as __pure. Technically initializers as pure functions.

Quoted 8 lines

But bpf_refcount_set won't be marked __pure, neither am I proposing to allow direct stores to 'set' it.
I'm not a compiler expert by any means, but AFAIK it should not be doing such reordering for functions otherwise.
What if the function inside has a memory barrier? That would completely screw up things.
It's going to have external linkage, so I don't think it can assume anything about side effects or not. So IMO this is not a good point.

The pure attribute tells the compiler that the function doesn't have side effects. We even use it in the kernel code base. Sooner or later we'll start using it in bpf too. Things like memcmp is a primary example.
I have to correct myself though. refcount_set shouldn't be considered pure.

Quoted 9 lines

Unless you're talking about some new way of inlining such helpers from the compiler side that doesn't exist yet.

Quoted 3 lines

The verifier and llvm already "fight" a lot. We gotta be very careful in the verifier and not assume that the code stays as written in C.

So will these implicit zero stores be done when we enter != NULL branch, or lazily on first access (helper arg, load, store)?

Whichever way is faster and still safe.
I assumed that we'd have to zero them after successful alloc.

Quoted 6 lines

This is the flip side: rewritings insns to add stores to local kptr can only happen after the NULL check, in the != NULL branch, at that point we cannot assume R1-R5 are free for use, so complicated field initialization will be uglier to do implicitly (e.g. if it involves calling functions etc.).
There are pros and cons for both.

Are you expecting the verifier to insert zero inits as actual insns after 'call bpf_kptr_new' insn ? Hmm. I imagined bpf_kptr_new helper will do it. Just a simple loop that is inverse of zero_map_value(). Not the fastest thing, but good enough to start and can be optimized later.
The verifier can insert ST insn too in !null branch, since that insn only needs one register and it's known in that branch.
It's questionable that a bunch of ST insns will be faster than a zero_map_value-like loop.

author maintainer
#35 lore raw

Re: [PATCH RFC bpf-next v1 16/32] bpf: Introduce BPF memory object model

Quoted 77 lines · Alexei Starovoitov
Quoted 71 lines · Kumar Kartikeya Dwivedi
Quoted 53 lines · Alexei Starovoitov
Quoted 11 lines · Kumar Kartikeya Dwivedi

I slept over this. I think I can get behind this idea of implicit ctor/dtor. We might have open coded construction/destruction later if we want.

I am however thinking of naming these helpers:
bpf_kptr_new
bpf_kptr_delete
to make it clear it does a little more than just allocating the type. The open coded cases can later derive their allocation from the more bare bones bpf_kptr_alloc instead in the future.

New names make complete sense. Good idea.

Quoted 6 lines

The main reason to have open coded-ness was being able to 'manage' resources once visibility reduces to current CPU (bpf_refcount_put, single ownership after xchg, etc.). Even with RCU, we won't allow touching the BPF special fields without refcount. bpf_spin_lock is different, as it protects more than just bpf special fields.

Quoted 7 lines

But one can still splice or kptr_xchg before passing to bpf_kptr_free to do that. bpf_kptr_free is basically cleaning up whatever is left by then, forcefully. In the future, we might even be able to do elision of implicit dtors based on the seen data flow (splicing in single ownership implies list is empty, any other op will undo that, etc.) if there are big structs with too many fields. Can also support that in open coded cases.

Right.

Quoted 15 lines

What I want to think about more is whether we should still force calling bpf_refcount_set vs always setting it to 1.

I know we don't agree about whether list_add in shared mode should take ref vs transfer ref. I'm leaning towards transfer since that will be most intuitive. It then works the same way in both cases, single ownership only transfers the sole reference you have, so you lose access, but in shared you may have more than one. If you have just one you will still lose access.

It will be odd for list_add to consume it in one case and not the other. People should already be fully conscious of how they are managing the lifetime of their object.

Quoted 5 lines

It then seems better to require users to set the initial refcount themselves. When doing the initial linking it can be very cheap. Later get/put/inc are always available.

But forcing it to be called is going to be much simpler than this patch.

Quoted 8 lines

I'm not convinced yet :)
Pls hold on implementing one way or another.
Let's land the single ownership case for locks, lists, rbtrees, allocators. That's plenty of patches.
Then we can start a deeper discussion into the shared case. Whether it will be different in terms of 'lose access after list_add' is not critical to decide now. It can change in the future too.

Right, I'm not implementing it yet. There's a lot of work left to even finish single ownership structures, then lots of testing. But it's helpful to keep thinking about future use cases while working on the current stuff, just to make sure we're not
digging ourselves into a design hole.

Quoted 3 lines

We have the option to undo damage here, since this is all experimental, but there's still an expectation that the API is not broken at whim. That wouldn't be very useful for users.

Quoted 7 lines

imo this part is minor.
The whole lock + list_or_rbtree in a single allocation restriction bothers me a lot more.
We will find out for sure only when wwe have a prototype of lock + list + rbtree and let folks who requested it actually code things.

Sure.
But when I look in the kernel, I often see data and its lock often allocated together.
The lock is just there to serialize access to the data structure. It might as well not be if it's a bpf_llist_head, or only optionally. It might be an entirely different way of serializing access (local_t + percpu list).
But usually having both together is also great for locality. Different use cases have different needs, the simple and common cases are often well served by having both together.

Not every use case needs both list and/or rbtree. Some require access to only one at a time.
e.g. We might reserve struct rb_node in xdp_frame, allowing struct bpf_rb_tree __contains(xdp_frame, rb_node)
or struct bpf_rb_tree __contains(sk_buff, rb_node), unifying the queueing primitives for XDP and TC.
It makes sense to make simple cases faster and simpler to use.

This is why I eventually plan to add a RCU based hash table using these single ownership lists in selftests, at least as a showcase it can serve a 'real world' use case.

Dave's dynamic lock checks are conceptually not very different from the verifier's perspective. A bpf_spin_lock * vs bpf_spin_lock protecting the list. Indirection allows it to assume a dynamic value at runtime. Some checks can still be done statically, and some where the pointer's indeterminism hinders static analysis can be offloaded to runtime. Different tradeoffs.

Quoted 55 lines
Quoted 19 lines
Quoted 1 line

The other reason to do implicit inits and ref count sets is to

I am not contesting implicit construction.
Other lists already work with zero initialization so list_head seems more of an exception.
But it's done for good reasons to avoid extra NULL checks unnecessarily, and make the implementation of list helpers more efficient and simple at the same time.

Quoted 9 lines

avoid fighting llvm.
obj = bpf_kptr_new();
obj->var1 = 1;
some_func(&obj->var2);
In many cases the compiler is allowed to sink stores.
If there are two calls that "init" two different fields the compiler is allowed to change the order as well
even if it doesn't see the body of the function and the function is marked as __pure. Technically initializers as pure functions.

Quoted 8 lines

But bpf_refcount_set won't be marked __pure, neither am I proposing to allow direct stores to 'set' it.
I'm not a compiler expert by any means, but AFAIK it should not be doing such reordering for functions otherwise.
What if the function inside has a memory barrier? That would completely screw up things.
It's going to have external linkage, so I don't think it can assume anything about side effects or not. So IMO this is not a good point.

The pure attribute tells the compiler that the function doesn't have side effects. We even use it in the kernel code base. Sooner or later we'll start using it in bpf too. Things like memcmp is a primary example.
I have to correct myself though. refcount_set shouldn't be considered pure.

Quoted 9 lines

Unless you're talking about some new way of inlining such helpers from the compiler side that doesn't exist yet.

Quoted 3 lines

The verifier and llvm already "fight" a lot. We gotta be very careful in the verifier and not assume that the code stays as written in C.

So will these implicit zero stores be done when we enter != NULL branch, or lazily on first access (helper arg, load, store)?

Whichever way is faster and still safe.
I assumed that we'd have to zero them after successful alloc.

Quoted 6 lines

This is the flip side: rewritings insns to add stores to local kptr can only happen after the NULL check, in the != NULL branch, at that point we cannot assume R1-R5 are free for use, so complicated field initialization will be uglier to do implicitly (e.g. if it involves calling functions etc.).
There are pros and cons for both.

Quoted 11 lines

Are you expecting the verifier to insert zero inits as actual insns after 'call bpf_kptr_new' insn ? Hmm. I imagined bpf_kptr_new helper will do it. Just a simple loop that is inverse of zero_map_value(). Not the fastest thing, but good enough to start and can be optimized later.
The verifier can insert ST insn too in !null branch, since that insn only needs one register and it's known in that branch.
It's questionable that a bunch of ST insns will be faster than a zero_map_value-like loop.

I would definitely think a bunch of direct 0 stores would be faster. zero_map_value will access some kind of offset array in memory and then read from it and loop over to do the stores.
Does seem like more work to do, even if it's hot in cache those are unneeded extra accesses for information we know statically. So I'll most likely emit a bunch of ST insns zeroing it out in v1.

#36 lore raw

Re: [PATCH RFC bpf-next v1 16/32] bpf: Introduce BPF memory object model

Quoted 13 lines · Kumar Kartikeya Dwivedi
Quoted 12 lines

Are you expecting the verifier to insert zero inits as actual insns after 'call bpf_kptr_new' insn ? Hmm. I imagined bpf_kptr_new helper will do it. Just a simple loop that is inverse of zero_map_value(). Not the fastest thing, but good enough to start and can be optimized later.
The verifier can insert ST insn too in !null branch, since that insn only needs one register and it's known in that branch.
It's questionable that a bunch of ST insns will be faster than a zero_map_value-like loop.

Quoted 6 lines

I would definitely think a bunch of direct 0 stores would be faster. zero_map_value will access some kind of offset array in memory and then read from it and loop over to do the stores.
Does seem like more work to do, even if it's hot in cache those are unneeded extra accesses for information we know statically. So I'll most likely emit a bunch of ST insns zeroing it out in v1.

Premature optimization is ...

author maintainer
#37 lore raw

RFC PATCH 17/32 bpf: Support bpf_list_node in local kptrs

In reply to Kumar Kartikeya Dwivedi

To allow a user to link their kptr allocated node into a linked list, we must have a linked list node type that is recognized by the verifier fit for this purpose. Its name and offset will be matched with the specification on the bpf_list_head it is being added to. This would allow precise verification and type safety in BPF programs.

Since bpf_list_node does not correspond to local type, but it is embedded in a local type (i.e. a type present in program BTF, not kernel BTF), we need to specially tag such a field so that verifier knows that it is a special kernel object whose invariants must hold during use of the kptr allocation. For instance, reading and writing is allowed to all other offsets in the kptr allocation, but access to this special field would be rejected.

To do so, it needs to be tagged using a "kernel" BTF declaration tag, like so:

struct item {
	int data;
	struct bpf_list_node node __kernel;
};

In future commits, more objects (such as kptrs inside kptrs, spin_lock, even bpf_list_head) will be allowed in kptr allocation. But those need more plumbing before it can all be made safe.

---
 include/linux/btf.h                           | 15 ++++
 kernel/bpf/btf.c                              | 86 ++++++++++++++++---
 kernel/bpf/helpers.c                          |  8 ++
 kernel/bpf/verifier.c                         | 46 ++++++++--
 .../testing/selftests/bpf/bpf_experimental.h  |  9 ++
 5 files changed, 146 insertions(+), 18 deletions(-)
include/linux/btf.h +15 -0
index fc35c932e89e..062bc45e1cc9 100644
@@ -433,6 +433,10 @@ const struct btf_member *
 btf_get_prog_ctx_type(struct bpf_verifier_log *log, const struct btf *btf,
 		      const struct btf_type *t, enum bpf_prog_type prog_type,
 		      int arg);
+int btf_local_type_has_bpf_list_node(const struct btf *btf,
+				     const struct btf_type *t, u32 *offsetp);
+bool btf_local_type_has_special_fields(const struct btf *btf,
+				       const struct btf_type *t);
 #else
 static inline const struct btf_type *btf_type_by_id(const struct btf *btf,
 						    u32 type_id)
@@ -471,6 +475,17 @@ btf_get_prog_ctx_type(struct bpf_verifier_log *log, const struct btf *btf,
 {
 	return NULL;
 }
+static inline int btf_local_type_has_bpf_list_node(const struct btf *btf,
+						   const struct btf_type *t,
+						   u32 *offsetp)
+{
+	return -ENOENT;
+}
+static inline bool btf_local_type_has_special_fields(const struct btf *btf,
+						     const struct btf_type *t)
+{
+	return false;
+}
 #endif
 
 #endif
kernel/bpf/btf.c +75 -11

182 lines not on this page · raw message

kernel/bpf/helpers.c +8 -0
index d417aa4f0b22..0bb11d8bcaca 100644
@@ -1710,6 +1710,13 @@ void *bpf_kptr_alloc(u64 local_type_id__k, u64 flags)
 	return kmalloc(size, GFP_ATOMIC);
 }
 
+void bpf_list_node_init(struct bpf_list_node *node__clkptr)
+{
+	BUILD_BUG_ON(sizeof(struct bpf_list_node) != sizeof(struct list_head));
+	BUILD_BUG_ON(__alignof__(struct bpf_list_node) != __alignof__(struct list_head));
+	INIT_LIST_HEAD((struct list_head *)node__clkptr);
+}
+
 __diag_pop();
 
 BTF_SET8_START(tracing_btf_ids)
@@ -1717,6 +1724,7 @@ BTF_SET8_START(tracing_btf_ids)
 BTF_ID_FLAGS(func, crash_kexec, KF_DESTRUCTIVE)
 #endif
 BTF_ID_FLAGS(func, bpf_kptr_alloc, KF_ACQUIRE | KF_RET_NULL | __KF_RET_DYN_BTF)
+BTF_ID_FLAGS(func, bpf_list_node_init)
 BTF_SET8_END(tracing_btf_ids)
 
 static const struct btf_kfunc_id_set tracing_kfunc_set = {
kernel/bpf/verifier.c +39 -7

74 lines not on this page · raw message

tools/testing/selftests/bpf/bpf_experimental.h +9 -0
index bddd77093d1e..c3c5442742dc 100644
@@ -32,4 +32,13 @@ struct bpf_list_node {
  */
 void *bpf_kptr_alloc(__u64 local_type_id, __u64 flags) __ksym;
 
+/* Description
+ *	Initialize bpf_list_node field in a local kptr. This kfunc has
+ *	constructor semantics, and thus can only be called on a local kptr in
+ *	'constructing' phase.
+ * Returns
+ *	Void.
+ */
+void bpf_list_node_init(struct bpf_list_node *node) __ksym;
+
 #endif
Signature · 1 line
2.34.1
author maintainer
#38 lore raw

RFC PATCH 18/32 bpf: Support bpf_spin_lock in local kptrs

In reply to Kumar Kartikeya Dwivedi

To allow users to lock and protect data in their local kptr allocation, add support for embedding a bpf_spin_lock as a member. This is following how bpf_list_node is supported as a member already, by suitably tagging the field. We will later use bpf_spin_lock to allow implementing map-in-map style intrusive collection, while still associating the lock of linked list together in one single allocation. Only one bpf_spin_lock is allowed, and when a PTR_TO_BTF_ID | MEM_TYPE_LOCAL reg->type has it, it will preserve its reg->id during mark_ptr_or_null_reg marking, so that verifier can figure out the pairing of spin_lock and spin_unlock calls.

Existing process_spin_lock is refactored to support such spin_locks in allocated items. Still the restriction of holding single spin lock at one point does not go away, which is needed for deadlock safety.

The tagging works similar to bpf_list_node, it needs to be tagged using a "kernel" BTF declaration tag, like so:

struct item {
    int data;
    struct bpf_spin_lock lock __kernel;
};
---
 include/linux/btf.h                           |   8 ++
 include/linux/poison.h                        |   3 +
 kernel/bpf/btf.c                              |  10 +-
 kernel/bpf/helpers.c                          |  14 ++-
 kernel/bpf/verifier.c                         | 104 ++++++++++++++----
 .../testing/selftests/bpf/bpf_experimental.h  |   9 ++
 6 files changed, 121 insertions(+), 27 deletions(-)
include/linux/btf.h +8 -0
index 062bc45e1cc9..d99cad21e6d9 100644
@@ -435,6 +435,8 @@ btf_get_prog_ctx_type(struct bpf_verifier_log *log, const struct btf *btf,
 		      int arg);
 int btf_local_type_has_bpf_list_node(const struct btf *btf,
 				     const struct btf_type *t, u32 *offsetp);
+int btf_local_type_has_bpf_spin_lock(const struct btf *btf,
+				     const struct btf_type *t, u32 *offsetp);
 bool btf_local_type_has_special_fields(const struct btf *btf,
 				       const struct btf_type *t);
 #else
@@ -481,6 +483,12 @@ static inline int btf_local_type_has_bpf_list_node(const struct btf *btf,
 {
 	return -ENOENT;
 }
+static inline int btf_local_type_has_bpf_spin_lock(const struct btf *btf,
+					           const struct btf_type *t,
+					           u32 *offsetp)
+{
+	return -ENOENT;
+}
 static inline bool btf_local_type_has_special_fields(const struct btf *btf,
 						     const struct btf_type *t)
 {
include/linux/poison.h +3 -0
index d62ef5a6b4e9..753e00b81acf 100644
@@ -81,4 +81,7 @@
 /********** net/core/page_pool.c **********/
 #define PP_SIGNATURE		(0x40 + POISON_POINTER_DELTA)
 
+/********** kernel/bpf/helpers.c **********/
+#define BPF_PTR_POISON		((void *)((0xeB9FUL << 2) + POISON_POINTER_DELTA))
+
 #endif
kernel/bpf/btf.c +9 -1
index d8bc4752204c..63193c324898 100644
@@ -5954,9 +5954,16 @@ int btf_local_type_has_bpf_list_node(const struct btf *btf,
 	return btf_find_local_type_field(btf, t, BTF_FIELD_LIST_NODE, offsetp);
 }
 
+int btf_local_type_has_bpf_spin_lock(const struct btf *btf,
+				     const struct btf_type *t, u32 *offsetp)
+{
+	return btf_find_local_type_field(btf, t, BTF_FIELD_SPIN_LOCK, offsetp);
+}
+
 bool btf_local_type_has_special_fields(const struct btf *btf, const struct btf_type *t)
 {
-	return btf_local_type_has_bpf_list_node(btf, t, NULL) == 1;
+	return btf_local_type_has_bpf_list_node(btf, t, NULL) == 1 ||
+	       btf_local_type_has_bpf_spin_lock(btf, t, NULL) == 1;
 }
 
 int btf_struct_access(struct bpf_verifier_log *log, const struct btf *btf,
@@ -5985,6 +5992,7 @@ int btf_struct_access(struct bpf_verifier_log *log, const struct btf *btf,
 		}									\
 	}
 		PREVENT_DIRECT_WRITE(bpf_list_node);
+		PREVENT_DIRECT_WRITE(bpf_spin_lock);
 
 #undef PREVENT_DIRECT_WRITE
 		err = 0;
kernel/bpf/helpers.c +11 -3
index 0bb11d8bcaca..94a23a544aee 100644
@@ -335,6 +335,7 @@ const struct bpf_func_proto bpf_spin_lock_proto = {
 	.gpl_only	= false,
 	.ret_type	= RET_VOID,
 	.arg1_type	= ARG_PTR_TO_SPIN_LOCK,
+	.arg1_btf_id	= BPF_PTR_POISON,
 };
 
 static inline void __bpf_spin_unlock_irqrestore(struct bpf_spin_lock *lock)
@@ -357,6 +358,7 @@ const struct bpf_func_proto bpf_spin_unlock_proto = {
 	.gpl_only	= false,
 	.ret_type	= RET_VOID,
 	.arg1_type	= ARG_PTR_TO_SPIN_LOCK,
+	.arg1_btf_id	= BPF_PTR_POISON,
 };
 
 void copy_map_value_locked(struct bpf_map *map, void *dst, void *src,
@@ -1375,10 +1377,10 @@ BPF_CALL_2(bpf_kptr_xchg, void *, map_value, void *, ptr)
 	return xchg(kptr, (unsigned long)ptr);
 }
 
-/* Unlike other PTR_TO_BTF_ID helpers the btf_id in bpf_kptr_xchg()
- * helper is determined dynamically by the verifier.
+/* Unlike other PTR_TO_BTF_ID helpers the btf_id in bpf_kptr_xchg() helper is
+ * determined dynamically by the verifier. Hence, BPF_PTR_POISON is used as the
+ * placeholder pointer.
  */
-#define BPF_PTR_POISON ((void *)((0xeB9FUL << 2) + POISON_POINTER_DELTA))
 
 static const struct bpf_func_proto bpf_kptr_xchg_proto = {
 	.func         = bpf_kptr_xchg,
@@ -1717,6 +1719,11 @@ void bpf_list_node_init(struct bpf_list_node *node__clkptr)
 	INIT_LIST_HEAD((struct list_head *)node__clkptr);
 }
 
+void bpf_spin_lock_init(struct bpf_spin_lock *lock__clkptr)
+{
+	memset(lock__clkptr, 0, sizeof(*lock__clkptr));
+}
+
 __diag_pop();
 
 BTF_SET8_START(tracing_btf_ids)
@@ -1725,6 +1732,7 @@ BTF_ID_FLAGS(func, crash_kexec, KF_DESTRUCTIVE)
 #endif
 BTF_ID_FLAGS(func, bpf_kptr_alloc, KF_ACQUIRE | KF_RET_NULL | __KF_RET_DYN_BTF)
 BTF_ID_FLAGS(func, bpf_list_node_init)
+BTF_ID_FLAGS(func, bpf_spin_lock_init)
 BTF_SET8_END(tracing_btf_ids)
 
 static const struct btf_kfunc_id_set tracing_kfunc_set = {
kernel/bpf/verifier.c +81 -23

189 lines not on this page · raw message

tools/testing/selftests/bpf/bpf_experimental.h +9 -0
index c3c5442742dc..8b1cdfb2f6bc 100644
@@ -41,4 +41,13 @@ void *bpf_kptr_alloc(__u64 local_type_id, __u64 flags) __ksym;
  */
 void bpf_list_node_init(struct bpf_list_node *node) __ksym;
 
+/* Description
+ *	Initialize bpf_spin_lock field in a local kptr. This kfunc has
+ *	constructor semantics, and thus can only be called on a local kptr in
+ *	'constructing' phase.
+ * Returns
+ *	Void.
+ */
+void bpf_spin_lock_init(struct bpf_spin_lock *node) __ksym;
+
 #endif
Signature · 1 line
2.34.1
#39 lore raw

Re: [PATCH RFC bpf-next v1 18/32] bpf: Support bpf_spin_lock in local kptrs

include/linux/poison.h line 86

Quoted 11 lines · Kumar Kartikeya Dwivedi
include/linux/poison.h +3 -0
index d62ef5a6b4e9..753e00b81acf 100644
@@ -81,4 +81,7 @@
 /********** net/core/page_pool.c **********/
 #define PP_SIGNATURE		(0x40 + POISON_POINTER_DELTA)
 
+/********** kernel/bpf/helpers.c **********/
+#define BPF_PTR_POISON		((void *)((0xeB9FUL << 2) + POISON_POINTER_DELTA))
+

That was part of Dave's patch set as well.
Please keep his SOB and authorship and keep it as separate patch.

#40 lore raw

Re: [PATCH RFC bpf-next v1 18/32] bpf: Support bpf_spin_lock in local kptrs

Quoted 9 lines · Alexei Starovoitov
Quoted 8 lines · Kumar Kartikeya Dwivedi
include/linux/poison.h +0 -0
index d62ef5a6b4e9..753e00b81acf 100644
@@ -81,4 +81,7 @@
 /********** net/core/page_pool.c **********/
 #define PP_SIGNATURE		(0x40 + POISON_POINTER_DELTA)
 
Quoted 6 lines

include/linux/poison.h line 86

Quoted 3 lines
include/linux/poison.h +3 -0
+/********** kernel/bpf/helpers.c **********/
+#define BPF_PTR_POISON		((void *)((0xeB9FUL << 2) + POISON_POINTER_DELTA))
+

That was part of Dave's patch set as well.
Please keep his SOB and authorship and keep it as separate patch.

My patch picked a different constant :). But on that note, it also added some checking in verifier.c so that verification fails if any arg or retval type was BPF_PTR_POISON after it should've been replaced. Perhaps it's worth shipping that patch ("bpf: Add verifier check for BPF_PTR_POISON retval and arg") separately? Would allow both rbtree series and this lock-focused patch to drop BPF_PTR_POISON changes after rebase.

author maintainer
#41 lore raw

Re: [PATCH RFC bpf-next v1 18/32] bpf: Support bpf_spin_lock in local kptrs

Quoted 18 lines · Dave Marchevsky
Quoted 9 lines · Alexei Starovoitov
Quoted 8 lines · Kumar Kartikeya Dwivedi
include/linux/poison.h +0 -0
index d62ef5a6b4e9..753e00b81acf 100644
@@ -81,4 +81,7 @@
 /********** net/core/page_pool.c **********/
 #define PP_SIGNATURE                (0x40 + POISON_POINTER_DELTA)
 
Quoted 6 lines

include/linux/poison.h line 86

Quoted 3 lines
include/linux/poison.h +3 -0
+/********** kernel/bpf/helpers.c **********/
+#define BPF_PTR_POISON              ((void *)((0xeB9FUL << 2) + POISON_POINTER_DELTA))
+

That was part of Dave's patch set as well.
Please keep his SOB and authorship and keep it as separate patch.

Quoted 6 lines

My patch picked a different constant :). But on that note, it also added some checking in verifier.c so that verification fails if any arg or retval type was BPF_PTR_POISON after it should've been replaced. Perhaps it's worth shipping that patch ("bpf: Add verifier check for BPF_PTR_POISON retval and arg") separately? Would allow both rbtree series and this lock-focused patch to drop BPF_PTR_POISON changes after rebase.

Yeah, feel free to post it separately. I'm using the constant in this patch for a different purpose (I separate the BPF_PTR_POISON case into its own different argument type, and then check that it is always set for it in check_btf_id_ok, to ensure DYN_BTF_ID is not setting some static real BTF ID).

But why change the constant, eB9F looks very close to eBPF already :). UL is just for unsigned long.

#42 lore raw

Re: [PATCH RFC bpf-next v1 18/32] bpf: Support bpf_spin_lock in local kptrs

Quoted 27 lines · Kumar Kartikeya Dwivedi
Quoted 18 lines · Dave Marchevsky
Quoted 9 lines · Alexei Starovoitov
Quoted 8 lines · Kumar Kartikeya Dwivedi
include/linux/poison.h +0 -0
index d62ef5a6b4e9..753e00b81acf 100644
@@ -81,4 +81,7 @@
 /********** net/core/page_pool.c **********/
 #define PP_SIGNATURE                (0x40 + POISON_POINTER_DELTA)
 
Quoted 6 lines

include/linux/poison.h line 86

Quoted 3 lines
include/linux/poison.h +3 -0
+/********** kernel/bpf/helpers.c **********/
+#define BPF_PTR_POISON              ((void *)((0xeB9FUL << 2) + POISON_POINTER_DELTA))
+

That was part of Dave's patch set as well.
Please keep his SOB and authorship and keep it as separate patch.

Quoted 6 lines

My patch picked a different constant :). But on that note, it also added some checking in verifier.c so that verification fails if any arg or retval type was BPF_PTR_POISON after it should've been replaced. Perhaps it's worth shipping that patch ("bpf: Add verifier check for BPF_PTR_POISON retval and arg") separately? Would allow both rbtree series and this lock-focused patch to drop BPF_PTR_POISON changes after rebase.

Quoted 7 lines

Yeah, feel free to post it separately. I'm using the constant in this patch for a different purpose (I separate the BPF_PTR_POISON case into its own different argument type, and then check that it is always set for it in check_btf_id_ok, to ensure DYN_BTF_ID is not setting some static real BTF ID).

But why change the constant, eB9F looks very close to eBPF already :).

+1. It's already in many spots:
git grep eB9F

author maintainer
#43 lore raw

RFC PATCH 19/32 bpf: Support bpf_list_head in local kptrs

In reply to Kumar Kartikeya Dwivedi

To support map-in-map style use case, allow embedding a bpf_list_head inside a allocated kptr representing local type. Now, this is a field that will actually need explicit action while destructing the object, i.e. popping off all the nodes owned by the bpf_list_head and then freeing each one of them. Hence, the destruction state needs tracking when we are in the phase, and we also need to reject freeing when the embedded list_head is not destructed.

For now, needs_destruction is false. Future patch will flip it to true once adding items to such list_head is supported.

---
 include/linux/btf.h                           |  8 ++
 kernel/bpf/btf.c                              | 89 ++++++++++++++++---
 kernel/bpf/helpers.c                          |  8 ++
 kernel/bpf/verifier.c                         |  4 +
 .../testing/selftests/bpf/bpf_experimental.h  |  9 ++
 5 files changed, 106 insertions(+), 12 deletions(-)
include/linux/btf.h +8 -0
index d99cad21e6d9..42c7f0283887 100644
@@ -437,6 +437,8 @@ int btf_local_type_has_bpf_list_node(const struct btf *btf,
 				     const struct btf_type *t, u32 *offsetp);
 int btf_local_type_has_bpf_spin_lock(const struct btf *btf,
 				     const struct btf_type *t, u32 *offsetp);
+int btf_local_type_has_bpf_list_head(const struct btf *btf,
+				     const struct btf_type *t, u32 *offsetp);
 bool btf_local_type_has_special_fields(const struct btf *btf,
 				       const struct btf_type *t);
 #else
@@ -489,6 +491,12 @@ static inline int btf_local_type_has_bpf_spin_lock(const struct btf *btf,
 {
 	return -ENOENT;
 }
+static inline int btf_local_type_has_bpf_list_head(const struct btf *btf,
+					           const struct btf_type *t,
+					           u32 *offsetp)
+{
+	return -ENOENT;
+}
 static inline bool btf_local_type_has_special_fields(const struct btf *btf,
 						     const struct btf_type *t)
 {
kernel/bpf/btf.c +77 -12

175 lines not on this page · raw message

kernel/bpf/helpers.c +8 -0
index 94a23a544aee..8eee0793c7f1 100644
@@ -1724,6 +1724,13 @@ void bpf_spin_lock_init(struct bpf_spin_lock *lock__clkptr)
 	memset(lock__clkptr, 0, sizeof(*lock__clkptr));
 }
 
+void bpf_list_head_init(struct bpf_list_head *head__clkptr)
+{
+	BUILD_BUG_ON(sizeof(struct bpf_list_head) != sizeof(struct list_head));
+	BUILD_BUG_ON(__alignof__(struct bpf_list_head) != __alignof__(struct list_head));
+	INIT_LIST_HEAD((struct list_head *)head__clkptr);
+}
+
 __diag_pop();
 
 BTF_SET8_START(tracing_btf_ids)
@@ -1733,6 +1740,7 @@ BTF_ID_FLAGS(func, crash_kexec, KF_DESTRUCTIVE)
 BTF_ID_FLAGS(func, bpf_kptr_alloc, KF_ACQUIRE | KF_RET_NULL | __KF_RET_DYN_BTF)
 BTF_ID_FLAGS(func, bpf_list_node_init)
 BTF_ID_FLAGS(func, bpf_spin_lock_init)
+BTF_ID_FLAGS(func, bpf_list_head_init)
 BTF_SET8_END(tracing_btf_ids)
 
 static const struct btf_kfunc_id_set tracing_kfunc_set = {
kernel/bpf/verifier.c +4 -0
index 130a4f0550f5..a5aa5de4b246 100644
@@ -7811,12 +7811,14 @@ BTF_ID_LIST(special_kfuncs)
 BTF_ID(func, bpf_kptr_alloc)
 BTF_ID(func, bpf_list_node_init)
 BTF_ID(func, bpf_spin_lock_init)
+BTF_ID(func, bpf_list_head_init)
 BTF_ID(struct, btf) /* empty entry */
 
 enum bpf_special_kfuncs {
 	KF_SPECIAL_bpf_kptr_alloc,
 	KF_SPECIAL_bpf_list_node_init,
 	KF_SPECIAL_bpf_spin_lock_init,
+	KF_SPECIAL_bpf_list_head_init,
 	KF_SPECIAL_bpf_empty,
 	KF_SPECIAL_MAX = KF_SPECIAL_bpf_empty,
 };
@@ -7984,6 +7986,7 @@ struct local_type_field {
 	enum {
 		FIELD_bpf_list_node,
 		FIELD_bpf_spin_lock,
+		FIELD_bpf_list_head,
 		FIELD_MAX,
 	} type;
 	enum bpf_special_kfuncs ctor_kfunc;
@@ -8030,6 +8033,7 @@ static int find_local_type_fields(const struct btf *btf, u32 btf_id, struct loca
 
 	FILL_LOCAL_TYPE_FIELD(bpf_list_node, bpf_list_node_init, bpf_empty, false);
 	FILL_LOCAL_TYPE_FIELD(bpf_spin_lock, bpf_spin_lock_init, bpf_empty, false);
+	FILL_LOCAL_TYPE_FIELD(bpf_list_head, bpf_list_head_init, bpf_empty, false);
 
 #undef FILL_LOCAL_TYPE_FIELD
 
tools/testing/selftests/bpf/bpf_experimental.h +9 -0
index 8b1cdfb2f6bc..f0b6e92c6908 100644
@@ -50,4 +50,13 @@ void bpf_list_node_init(struct bpf_list_node *node) __ksym;
  */
 void bpf_spin_lock_init(struct bpf_spin_lock *node) __ksym;
 
+/* Description
+ *	Initialize bpf_list_head field in a local kptr. This kfunc has
+ *	constructor semantics, and thus can only be called on a local kptr in
+ *	'constructing' phase.
+ * Returns
+ *	Void.
+ */
+void bpf_list_head_init(struct bpf_list_head *node) __ksym;
+
 #endif
Signature · 1 line
2.34.1
author maintainer
#44 lore raw

RFC PATCH 20/32 bpf: Introduce bpf_kptr_free helper

In reply to Kumar Kartikeya Dwivedi

After ensuring that the verifier can recognize normal vs destructing objects, add bpf_kptr_free support, and then verify whether normal object can directly be freed, or whether it needs destruction. If already in destructing phase, ensure all fields have been destructed.

Having this state in the verifier simplifies how we release resources for kptrs to local types with arbitrary fields considerably. The verifier just needs to ensure that destruction happens while program has ownership of object, and then it can just release the storage using bpf_kptr_free.

---
 kernel/bpf/helpers.c                          |  6 ++++
 kernel/bpf/verifier.c                         | 29 +++++++++++++++++++
 .../testing/selftests/bpf/bpf_experimental.h  |  8 +++++
 3 files changed, 43 insertions(+)
kernel/bpf/helpers.c +6 -0
index 8eee0793c7f1..4a6fffe401ae 100644
@@ -1731,6 +1731,11 @@ void bpf_list_head_init(struct bpf_list_head *head__clkptr)
 	INIT_LIST_HEAD((struct list_head *)head__clkptr);
 }
 
+void bpf_kptr_free(void *p__dlkptr)
+{
+	kfree(p__dlkptr);
+}
+
 __diag_pop();
 
 BTF_SET8_START(tracing_btf_ids)
@@ -1741,6 +1746,7 @@ BTF_ID_FLAGS(func, bpf_kptr_alloc, KF_ACQUIRE | KF_RET_NULL | __KF_RET_DYN_BTF)
 BTF_ID_FLAGS(func, bpf_list_node_init)
 BTF_ID_FLAGS(func, bpf_spin_lock_init)
 BTF_ID_FLAGS(func, bpf_list_head_init)
+BTF_ID_FLAGS(func, bpf_kptr_free, KF_RELEASE)
 BTF_SET8_END(tracing_btf_ids)
 
 static const struct btf_kfunc_id_set tracing_kfunc_set = {
kernel/bpf/verifier.c +29 -0
index a5aa5de4b246..b1754fd69f7d 100644
@@ -7812,6 +7812,7 @@ BTF_ID(func, bpf_kptr_alloc)
 BTF_ID(func, bpf_list_node_init)
 BTF_ID(func, bpf_spin_lock_init)
 BTF_ID(func, bpf_list_head_init)
+BTF_ID(func, bpf_kptr_free)
 BTF_ID(struct, btf) /* empty entry */
 
 enum bpf_special_kfuncs {
@@ -7819,6 +7820,7 @@ enum bpf_special_kfuncs {
 	KF_SPECIAL_bpf_list_node_init,
 	KF_SPECIAL_bpf_spin_lock_init,
 	KF_SPECIAL_bpf_list_head_init,
+	KF_SPECIAL_bpf_kptr_free,
 	KF_SPECIAL_bpf_empty,
 	KF_SPECIAL_MAX = KF_SPECIAL_bpf_empty,
 };
@@ -8156,6 +8158,33 @@ process_kf_arg_destructing_local_kptr(struct bpf_verifier_env *env,
 		}));
 	}
 
+	/* Handle bpf_kptr_free */
+	if (is_kfunc_special(meta->btf, meta->func_id, bpf_kptr_free)) {
+		for (i = cnt - 1; i >= 0; i--) {
+			if (!fields[i].needs_destruction)
+				continue;
+			/* If a field needs destruction, it must be in
+			 * destructed state when calling bpf_kptr_free.
+			 */
+			switch (local_kptr_get_state(reg, i)) {
+			case FIELD_STATE_CONSTRUCTED:
+				verbose(env, "'%s' field needs to be destructed before bpf_kptr_free\n",
+					fields[i].name);
+				return -EINVAL;
+			case FIELD_STATE_DESTRUCTED:
+				break;
+			case FIELD_STATE_UNKNOWN:
+				if (reg->type & OBJ_CONSTRUCTING)
+					break;
+				fallthrough;
+			default:
+				verbose(env, "verifier internal error: unknown field state\n");
+				return -EFAULT;
+			}
+		}
+		return 0;
+	}
+
 	for (i = 0; i < cnt; i++) {
 		bool mark_dtor = false, unmark_ctor = false;
 		int j;
tools/testing/selftests/bpf/bpf_experimental.h +8 -0
index f0b6e92c6908..595e99d5cbc2 100644
@@ -59,4 +59,12 @@ void bpf_spin_lock_init(struct bpf_spin_lock *node) __ksym;
  */
 void bpf_list_head_init(struct bpf_list_head *node) __ksym;
 
+/* Description
+ *	Free a local kptr. All fields of local kptr that require destruction
+ *	need to be in destructed state before this call is made.
+ * Returns
+ *	Void.
+ */
+void bpf_kptr_free(void *kptr) __ksym;
+
 #endif
Signature · 1 line
2.34.1
author maintainer
#45 lore raw

RFC PATCH 21/32 bpf: Allow locking bpf_spin_lock global variables

In reply to Kumar Kartikeya Dwivedi

Global variables reside in maps accessible using direct_value_addr callbacks, so giving each load instruction's rewrite a unique reg->id disallows us from holding locks which are global.

This is not great, so refactor the active_spin_lock into two separate fields, active_spin_lock_ptr and active_spin_lock_id, which is generic enough to allow it for global variables, map lookups, and local kptr registers at the same time.

Held vs non-held is indicated by active_spin_lock_ptr, which stores the reg->map_ptr or reg->btf pointer of the register used for locking spin lock. But the active_spin_lock_id also needs to be compared to ensure whether bpf_spin_unlock is for the same register.

Next, pseudo load instructions are not given a unique reg->id, as they are doing lookup for the same map value (max_entries is never greater than 1).

Essentially, we consider that the tuple of (active_spin_lock_ptr, active_spin_lock_id) will always be unique for any kind of argument to bpf_spin_{lock,unlock}.

Note that this can be extended in the future to also remember offset used for locking, so that we can introduce multiple bpf_spin_lock fields in the same allocation.

---
 include/linux/bpf_verifier.h |  3 ++-
 kernel/bpf/verifier.c        | 39 +++++++++++++++++++++++++-----------
 2 files changed, 29 insertions(+), 13 deletions(-)
include/linux/bpf_verifier.h +2 -1
index 2a9dcefca3b6..00c21ad6f61c 100644
@@ -348,7 +348,8 @@ struct bpf_verifier_state {
 	u32 branches;
 	u32 insn_idx;
 	u32 curframe;
-	u32 active_spin_lock;
+	void *active_spin_lock_ptr;
+	u32 active_spin_lock_id;
 	bool speculative;
 
 	/* first and last insn idx of this verifier state */
kernel/bpf/verifier.c +27 -12
index b1754fd69f7d..ed19e4036b0a 100644
@@ -1202,7 +1202,8 @@ static int copy_verifier_state(struct bpf_verifier_state *dst_state,
 	}
 	dst_state->speculative = src->speculative;
 	dst_state->curframe = src->curframe;
-	dst_state->active_spin_lock = src->active_spin_lock;
+	dst_state->active_spin_lock_ptr = src->active_spin_lock_ptr;
+	dst_state->active_spin_lock_id = src->active_spin_lock_id;
 	dst_state->branches = src->branches;
 	dst_state->parent = src->parent;
 	dst_state->first_insn_idx = src->first_insn_idx;
@@ -5504,22 +5505,35 @@ static int process_spin_lock(struct bpf_verifier_env *env, int regno,
 		return -EINVAL;
 	}
 	if (is_lock) {
-		if (cur->active_spin_lock) {
+		if (cur->active_spin_lock_ptr) {
 			verbose(env,
 				"Locking two bpf_spin_locks are not allowed\n");
 			return -EINVAL;
 		}
-		cur->active_spin_lock = reg->id;
+		if (map)
+			cur->active_spin_lock_ptr = map;
+		else
+			cur->active_spin_lock_ptr = btf;
+		cur->active_spin_lock_id = reg->id;
 	} else {
-		if (!cur->active_spin_lock) {
+		void *ptr;
+
+		if (map)
+			ptr = map;
+		else
+			ptr = btf;
+
+		if (!cur->active_spin_lock_ptr) {
 			verbose(env, "bpf_spin_unlock without taking a lock\n");
 			return -EINVAL;
 		}
-		if (cur->active_spin_lock != reg->id) {
+		if (cur->active_spin_lock_ptr != ptr ||
+		    cur->active_spin_lock_id != reg->id) {
 			verbose(env, "bpf_spin_unlock of different lock\n");
 			return -EINVAL;
 		}
-		cur->active_spin_lock = 0;
+		cur->active_spin_lock_ptr = NULL;
+		cur->active_spin_lock_id = 0;
 	}
 	return 0;
 }
@@ -11207,8 +11221,8 @@ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn)
 	    insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE) {
 		dst_reg->type = PTR_TO_MAP_VALUE;
 		dst_reg->off = aux->map_off;
-		if (map_value_has_spin_lock(map))
-			dst_reg->id = ++env->id_gen;
+		WARN_ON_ONCE(map->max_entries != 1);
+		/* We want reg->id to be same (0) as map_value is not distinct */
 	} else if (insn->src_reg == BPF_PSEUDO_MAP_FD ||
 		   insn->src_reg == BPF_PSEUDO_MAP_IDX) {
 		dst_reg->type = CONST_PTR_TO_MAP;
@@ -11286,7 +11300,7 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn)
 		return err;
 	}
 
-	if (env->cur_state->active_spin_lock) {
+	if (env->cur_state->active_spin_lock_ptr) {
 		verbose(env, "BPF_LD_[ABS|IND] cannot be used inside bpf_spin_lock-ed region\n");
 		return -EINVAL;
 	}
@@ -12566,7 +12580,8 @@ static bool states_equal(struct bpf_verifier_env *env,
 	if (old->speculative && !cur->speculative)
 		return false;
 
-	if (old->active_spin_lock != cur->active_spin_lock)
+	if (old->active_spin_lock_ptr != cur->active_spin_lock_ptr ||
+	    old->active_spin_lock_id != cur->active_spin_lock_id)
 		return false;
 
 	/* for states to be equal callsites have to be the same
@@ -13213,7 +13228,7 @@ static int do_check(struct bpf_verifier_env *env)
 					return -EINVAL;
 				}
 
-				if (env->cur_state->active_spin_lock &&
+				if (env->cur_state->active_spin_lock_ptr &&
 				    (insn->src_reg == BPF_PSEUDO_CALL ||
 				     insn->imm != BPF_FUNC_spin_unlock)) {
 					verbose(env, "function calls are not allowed while holding a lock\n");
@@ -13250,7 +13265,7 @@ static int do_check(struct bpf_verifier_env *env)
 					return -EINVAL;
 				}
 
-				if (env->cur_state->active_spin_lock) {
+				if (env->cur_state->active_spin_lock_ptr) {
 					verbose(env, "bpf_spin_unlock is missing\n");
 					return -EINVAL;
 				}
Signature · 1 line
2.34.1
#46 lore raw

Re: [PATCH RFC bpf-next v1 21/32] bpf: Allow locking bpf_spin_lock global variables

Quoted 37 lines · Kumar Kartikeya Dwivedi

Global variables reside in maps accessible using direct_value_addr callbacks, so giving each load instruction's rewrite a unique reg->id disallows us from holding locks which are global.

This is not great, so refactor the active_spin_lock into two separate fields, active_spin_lock_ptr and active_spin_lock_id, which is generic enough to allow it for global variables, map lookups, and local kptr registers at the same time.

Held vs non-held is indicated by active_spin_lock_ptr, which stores the reg->map_ptr or reg->btf pointer of the register used for locking spin lock. But the active_spin_lock_id also needs to be compared to ensure whether bpf_spin_unlock is for the same register.

Next, pseudo load instructions are not given a unique reg->id, as they are doing lookup for the same map value (max_entries is never greater than 1).

Essentially, we consider that the tuple of (active_spin_lock_ptr, active_spin_lock_id) will always be unique for any kind of argument to bpf_spin_{lock,unlock}.

Note that this can be extended in the future to also remember offset used for locking, so that we can introduce multiple bpf_spin_lock fields in the same allocation.

---
 include/linux/bpf_verifier.h |  3 ++-
 kernel/bpf/verifier.c        | 39 +++++++++++++++++++++++++-----------
 2 files changed, 29 insertions(+), 13 deletions(-)
include/linux/bpf_verifier.h +0 -0
index 2a9dcefca3b6..00c21ad6f61c 100644

include/linux/bpf_verifier.h struct bpf_verifier_state { · line 352

Quoted 6 lines
include/linux/bpf_verifier.h +2 -1
@@ -348,7 +348,8 @@ struct bpf_verifier_state {
 	u32 branches;
 	u32 insn_idx;
 	u32 curframe;
-	u32 active_spin_lock;
+	void *active_spin_lock_ptr;
+	u32 active_spin_lock_id;

{map, id=0} is indeed enough to distinguish different global locks and {map, id} for locks in map values,
but what 'btf' is for?
When is the case when reg->map_ptr is not set?
locks in allocated objects?
Feels too early to add that in this patch.

Also this patch is heavily influenced by Dave's patch with a realization that max_entries==1 simplifies the logic. I think you gotta give him more credit.
Maybe as much as his SOB and authorship.

Quoted 109 lines
include/linux/bpf_verifier.h +0 -0
 	bool speculative;
 
 	/* first and last insn idx of this verifier state */
kernel/bpf/verifier.c +27 -12
index b1754fd69f7d..ed19e4036b0a 100644
@@ -1202,7 +1202,8 @@ static int copy_verifier_state(struct bpf_verifier_state *dst_state,
 	}
 	dst_state->speculative = src->speculative;
 	dst_state->curframe = src->curframe;
-	dst_state->active_spin_lock = src->active_spin_lock;
+	dst_state->active_spin_lock_ptr = src->active_spin_lock_ptr;
+	dst_state->active_spin_lock_id = src->active_spin_lock_id;
 	dst_state->branches = src->branches;
 	dst_state->parent = src->parent;
 	dst_state->first_insn_idx = src->first_insn_idx;
@@ -5504,22 +5505,35 @@ static int process_spin_lock(struct bpf_verifier_env *env, int regno,
 		return -EINVAL;
 	}
 	if (is_lock) {
-		if (cur->active_spin_lock) {
+		if (cur->active_spin_lock_ptr) {
 			verbose(env,
 				"Locking two bpf_spin_locks are not allowed\n");
 			return -EINVAL;
 		}
-		cur->active_spin_lock = reg->id;
+		if (map)
+			cur->active_spin_lock_ptr = map;
+		else
+			cur->active_spin_lock_ptr = btf;
+		cur->active_spin_lock_id = reg->id;
 	} else {
-		if (!cur->active_spin_lock) {
+		void *ptr;
+
+		if (map)
+			ptr = map;
+		else
+			ptr = btf;
+
+		if (!cur->active_spin_lock_ptr) {
 			verbose(env, "bpf_spin_unlock without taking a lock\n");
 			return -EINVAL;
 		}
-		if (cur->active_spin_lock != reg->id) {
+		if (cur->active_spin_lock_ptr != ptr ||
+		    cur->active_spin_lock_id != reg->id) {
 			verbose(env, "bpf_spin_unlock of different lock\n");
 			return -EINVAL;
 		}
-		cur->active_spin_lock = 0;
+		cur->active_spin_lock_ptr = NULL;
+		cur->active_spin_lock_id = 0;
 	}
 	return 0;
 }
@@ -11207,8 +11221,8 @@ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn)
 	    insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE) {
 		dst_reg->type = PTR_TO_MAP_VALUE;
 		dst_reg->off = aux->map_off;
-		if (map_value_has_spin_lock(map))
-			dst_reg->id = ++env->id_gen;
+		WARN_ON_ONCE(map->max_entries != 1);
+		/* We want reg->id to be same (0) as map_value is not distinct */
 	} else if (insn->src_reg == BPF_PSEUDO_MAP_FD ||
 		   insn->src_reg == BPF_PSEUDO_MAP_IDX) {
 		dst_reg->type = CONST_PTR_TO_MAP;
@@ -11286,7 +11300,7 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn)
 		return err;
 	}
 
-	if (env->cur_state->active_spin_lock) {
+	if (env->cur_state->active_spin_lock_ptr) {
 		verbose(env, "BPF_LD_[ABS|IND] cannot be used inside bpf_spin_lock-ed region\n");
 		return -EINVAL;
 	}
@@ -12566,7 +12580,8 @@ static bool states_equal(struct bpf_verifier_env *env,
 	if (old->speculative && !cur->speculative)
 		return false;
 
-	if (old->active_spin_lock != cur->active_spin_lock)
+	if (old->active_spin_lock_ptr != cur->active_spin_lock_ptr ||
+	    old->active_spin_lock_id != cur->active_spin_lock_id)
 		return false;
 
 	/* for states to be equal callsites have to be the same
@@ -13213,7 +13228,7 @@ static int do_check(struct bpf_verifier_env *env)
 					return -EINVAL;
 				}
 
-				if (env->cur_state->active_spin_lock &&
+				if (env->cur_state->active_spin_lock_ptr &&
 				    (insn->src_reg == BPF_PSEUDO_CALL ||
 				     insn->imm != BPF_FUNC_spin_unlock)) {
 					verbose(env, "function calls are not allowed while holding a lock\n");
@@ -13250,7 +13265,7 @@ static int do_check(struct bpf_verifier_env *env)
 					return -EINVAL;
 				}
 
-				if (env->cur_state->active_spin_lock) {
+				if (env->cur_state->active_spin_lock_ptr) {
 					verbose(env, "bpf_spin_unlock is missing\n");
 					return -EINVAL;
 				}
Signature · 1 line
2.34.1
author maintainer
#47 lore raw

Re: [PATCH RFC bpf-next v1 21/32] bpf: Allow locking bpf_spin_lock global variables

Quoted 53 lines · Alexei Starovoitov
Quoted 37 lines · Kumar Kartikeya Dwivedi

Global variables reside in maps accessible using direct_value_addr callbacks, so giving each load instruction's rewrite a unique reg->id disallows us from holding locks which are global.

This is not great, so refactor the active_spin_lock into two separate fields, active_spin_lock_ptr and active_spin_lock_id, which is generic enough to allow it for global variables, map lookups, and local kptr registers at the same time.

Held vs non-held is indicated by active_spin_lock_ptr, which stores the reg->map_ptr or reg->btf pointer of the register used for locking spin lock. But the active_spin_lock_id also needs to be compared to ensure whether bpf_spin_unlock is for the same register.

Next, pseudo load instructions are not given a unique reg->id, as they are doing lookup for the same map value (max_entries is never greater than 1).

Essentially, we consider that the tuple of (active_spin_lock_ptr, active_spin_lock_id) will always be unique for any kind of argument to bpf_spin_{lock,unlock}.

Note that this can be extended in the future to also remember offset used for locking, so that we can introduce multiple bpf_spin_lock fields in the same allocation.

---
 include/linux/bpf_verifier.h |  3 ++-
 kernel/bpf/verifier.c        | 39 +++++++++++++++++++++++++-----------
 2 files changed, 29 insertions(+), 13 deletions(-)
include/linux/bpf_verifier.h +0 -0
index 2a9dcefca3b6..00c21ad6f61c 100644

include/linux/bpf_verifier.h struct bpf_verifier_state { · line 352

Quoted 6 lines
include/linux/bpf_verifier.h +2 -1
@@ -348,7 +348,8 @@ struct bpf_verifier_state {
      u32 branches;
      u32 insn_idx;
      u32 curframe;
-     u32 active_spin_lock;
+     void *active_spin_lock_ptr;
+     u32 active_spin_lock_id;

{map, id=0} is indeed enough to distinguish different global locks and {map, id} for locks in map values,
but what 'btf' is for?
When is the case when reg->map_ptr is not set?
locks in allocated objects?
Feels too early to add that in this patch.

Quoted 2 lines

Also this patch is heavily influenced by Dave's patch with a realization that max_entries==1 simplifies the logic.

You mean this one?
https://lore.kernel.org/bpf/[email protected]

Quoted 3 lines

I think you gotta give him more credit.
Maybe as much as his SOB and authorship.

Don't mind sharing the credit where due, but for the record:

15/8: pushed my prototype:
https://github.com/kkdwivedi/linux/commits/bpf-list-15-08-22
15/8: patch with roughly the same logic as above, comitted 24 days ago
https://github.com/kkdwivedi/linux/commit/4a152df6a1f6e096616e02c9b4dd54c5d5c902a1
16/8: Our meeting, described the same idea to you.
17/8: Published notes,
https://lore.kernel.org/bpf/CAP01T74U30+yeBHEgmgzTJ-XYxZ0zj71kqCDJtTH9YQNfTK+Xw@mail.gmail.com
19/8: Described the same thing in detail again in response to Dave's question:

Quoted 4 lines

This ergonomics idea doesn't solve the map-in-map issue, I'm still unsure how to statically verify lock in that case. Have you had a chance to think about it further?

at https://lore.kernel.org/bpf/CAP01T77PBfQ8QvgU-ezxGgUh8WmSYL3wsMT7yo4tGuZRW0qLnQ@mail.gmail.com
30/8: Dave sends patch with this idea:
https://lore.kernel.org/bpf/[email protected]

What did I miss?

#48 lore raw

Re: [PATCH RFC bpf-next v1 21/32] bpf: Allow locking bpf_spin_lock global variables

Quoted 79 lines · Kumar Kartikeya Dwivedi
Quoted 53 lines · Alexei Starovoitov
Quoted 37 lines · Kumar Kartikeya Dwivedi

Global variables reside in maps accessible using direct_value_addr callbacks, so giving each load instruction's rewrite a unique reg->id disallows us from holding locks which are global.

This is not great, so refactor the active_spin_lock into two separate fields, active_spin_lock_ptr and active_spin_lock_id, which is generic enough to allow it for global variables, map lookups, and local kptr registers at the same time.

Held vs non-held is indicated by active_spin_lock_ptr, which stores the reg->map_ptr or reg->btf pointer of the register used for locking spin lock. But the active_spin_lock_id also needs to be compared to ensure whether bpf_spin_unlock is for the same register.

Next, pseudo load instructions are not given a unique reg->id, as they are doing lookup for the same map value (max_entries is never greater than 1).

Essentially, we consider that the tuple of (active_spin_lock_ptr, active_spin_lock_id) will always be unique for any kind of argument to bpf_spin_{lock,unlock}.

Note that this can be extended in the future to also remember offset used for locking, so that we can introduce multiple bpf_spin_lock fields in the same allocation.

---
 include/linux/bpf_verifier.h |  3 ++-
 kernel/bpf/verifier.c        | 39 +++++++++++++++++++++++++-----------
 2 files changed, 29 insertions(+), 13 deletions(-)
include/linux/bpf_verifier.h +0 -0
index 2a9dcefca3b6..00c21ad6f61c 100644

include/linux/bpf_verifier.h struct bpf_verifier_state { · line 352

Quoted 6 lines
include/linux/bpf_verifier.h +2 -1
@@ -348,7 +348,8 @@ struct bpf_verifier_state {
      u32 branches;
      u32 insn_idx;
      u32 curframe;
-     u32 active_spin_lock;
+     void *active_spin_lock_ptr;
+     u32 active_spin_lock_id;

{map, id=0} is indeed enough to distinguish different global locks and {map, id} for locks in map values,
but what 'btf' is for?
When is the case when reg->map_ptr is not set?
locks in allocated objects?
Feels too early to add that in this patch.

Quoted 2 lines

Also this patch is heavily influenced by Dave's patch with a realization that max_entries==1 simplifies the logic.

You mean this one?
https://lore.kernel.org/bpf/[email protected]

Quoted 3 lines

I think you gotta give him more credit.
Maybe as much as his SOB and authorship.

Don't mind sharing the credit where due, but for the record:

15/8: pushed my prototype:
https://github.com/kkdwivedi/linux/commits/bpf-list-15-08-22
15/8: patch with roughly the same logic as above, comitted 24 days ago
https://github.com/kkdwivedi/linux/commit/4a152df6a1f6e096616e02c9b4dd54c5d5c902a1
16/8: Our meeting, described the same idea to you.
17/8: Published notes,
https://lore.kernel.org/bpf/CAP01T74U30+yeBHEgmgzTJ-XYxZ0zj71kqCDJtTH9YQNfTK+Xw@mail.gmail.com
19/8: Described the same thing in detail again in response to Dave's question:

Quoted 4 lines

This ergonomics idea doesn't solve the map-in-map issue, I'm still unsure how to statically verify lock in that case. Have you had a chance to think about it further?

Quoted 5 lines

at https://lore.kernel.org/bpf/CAP01T77PBfQ8QvgU-ezxGgUh8WmSYL3wsMT7yo4tGuZRW0qLnQ@mail.gmail.com
30/8: Dave sends patch with this idea:
https://lore.kernel.org/bpf/[email protected]

What did I miss?

Just that I saw Dave's patch first. Yours 8-22 private branch I glanced over and simply missed that patch. github UI is not for everyone.
As far as that notes thread it's hard to connect words to patches. Re-reading it now I see what you mean.
Feel free to keep the authorship for this one.

Please answer btf question though.

author maintainer
#49 lore raw

Re: [PATCH RFC bpf-next v1 21/32] bpf: Allow locking bpf_spin_lock global variables

In reply to Alexei Starovoitov
Quoted 44 lines · Alexei Starovoitov
Quoted 42 lines · Kumar Kartikeya Dwivedi

Global variables reside in maps accessible using direct_value_addr callbacks, so giving each load instruction's rewrite a unique reg->id disallows us from holding locks which are global.

This is not great, so refactor the active_spin_lock into two separate fields, active_spin_lock_ptr and active_spin_lock_id, which is generic enough to allow it for global variables, map lookups, and local kptr registers at the same time.

Held vs non-held is indicated by active_spin_lock_ptr, which stores the reg->map_ptr or reg->btf pointer of the register used for locking spin lock. But the active_spin_lock_id also needs to be compared to ensure whether bpf_spin_unlock is for the same register.

Next, pseudo load instructions are not given a unique reg->id, as they are doing lookup for the same map value (max_entries is never greater than 1).

Essentially, we consider that the tuple of (active_spin_lock_ptr, active_spin_lock_id) will always be unique for any kind of argument to bpf_spin_{lock,unlock}.

Note that this can be extended in the future to also remember offset used for locking, so that we can introduce multiple bpf_spin_lock fields in the same allocation.

---
 include/linux/bpf_verifier.h |  3 ++-
 kernel/bpf/verifier.c        | 39 +++++++++++++++++++++++++-----------
 2 files changed, 29 insertions(+), 13 deletions(-)
include/linux/bpf_verifier.h +1 -1
index 2a9dcefca3b6..00c21ad6f61c 100644
@@ -348,7 +348,8 @@ struct bpf_verifier_state {
      u32 branches;
      u32 insn_idx;
      u32 curframe;
-     u32 active_spin_lock;
+     void *active_spin_lock_ptr;
Quoted 9 lines

include/linux/bpf_verifier.h struct bpf_verifier_state { · line 352

Quoted 1 line
include/linux/bpf_verifier.h +1 -0
+     u32 active_spin_lock_id;

{map, id=0} is indeed enough to distinguish different global locks and {map, id} for locks in map values,
but what 'btf' is for?
When is the case when reg->map_ptr is not set?
locks in allocated objects?
Feels too early to add that in this patch.

It makes active_spin_lock check simpler, just checking active_spin_lock_ptr that to be non-NULL indicates lock is held. Don't have to always check both ptr and id, only need to compare both when verifying that lock is in the same allocation as reg.

#50 lore raw

Re: [PATCH RFC bpf-next v1 21/32] bpf: Allow locking bpf_spin_lock global variables

Quoted 57 lines · Kumar Kartikeya Dwivedi
Quoted 44 lines · Alexei Starovoitov
Quoted 42 lines · Kumar Kartikeya Dwivedi

Global variables reside in maps accessible using direct_value_addr callbacks, so giving each load instruction's rewrite a unique reg->id disallows us from holding locks which are global.

This is not great, so refactor the active_spin_lock into two separate fields, active_spin_lock_ptr and active_spin_lock_id, which is generic enough to allow it for global variables, map lookups, and local kptr registers at the same time.

Held vs non-held is indicated by active_spin_lock_ptr, which stores the reg->map_ptr or reg->btf pointer of the register used for locking spin lock. But the active_spin_lock_id also needs to be compared to ensure whether bpf_spin_unlock is for the same register.

Next, pseudo load instructions are not given a unique reg->id, as they are doing lookup for the same map value (max_entries is never greater than 1).

Essentially, we consider that the tuple of (active_spin_lock_ptr, active_spin_lock_id) will always be unique for any kind of argument to bpf_spin_{lock,unlock}.

Note that this can be extended in the future to also remember offset used for locking, so that we can introduce multiple bpf_spin_lock fields in the same allocation.

---
 include/linux/bpf_verifier.h |  3 ++-
 kernel/bpf/verifier.c        | 39 +++++++++++++++++++++++++-----------
 2 files changed, 29 insertions(+), 13 deletions(-)
include/linux/bpf_verifier.h +1 -1
index 2a9dcefca3b6..00c21ad6f61c 100644
@@ -348,7 +348,8 @@ struct bpf_verifier_state {
      u32 branches;
      u32 insn_idx;
      u32 curframe;
-     u32 active_spin_lock;
+     void *active_spin_lock_ptr;
Quoted 9 lines

include/linux/bpf_verifier.h struct bpf_verifier_state { · line 352

Quoted 1 line
include/linux/bpf_verifier.h +1 -0
+     u32 active_spin_lock_id;

{map, id=0} is indeed enough to distinguish different global locks and {map, id} for locks in map values,
but what 'btf' is for?
When is the case when reg->map_ptr is not set?
locks in allocated objects?
Feels too early to add that in this patch.

Quoted 4 lines

It makes active_spin_lock check simpler, just checking active_spin_lock_ptr that to be non-NULL indicates lock is held. Don't have to always check both ptr and id, only need to compare both when verifying that lock is in the same allocation as reg.

Not following. There is always non-null reg->map_ptr when we come down this path.
At least in the current state of the verifier. So it never assigns that btf afacs.

author maintainer
#51 lore raw

Re: [PATCH RFC bpf-next v1 21/32] bpf: Allow locking bpf_spin_lock global variables

Quoted 64 lines · Alexei Starovoitov
Quoted 57 lines · Kumar Kartikeya Dwivedi
Quoted 44 lines · Alexei Starovoitov
Quoted 42 lines · Kumar Kartikeya Dwivedi

Global variables reside in maps accessible using direct_value_addr callbacks, so giving each load instruction's rewrite a unique reg->id disallows us from holding locks which are global.

This is not great, so refactor the active_spin_lock into two separate fields, active_spin_lock_ptr and active_spin_lock_id, which is generic enough to allow it for global variables, map lookups, and local kptr registers at the same time.

Held vs non-held is indicated by active_spin_lock_ptr, which stores the reg->map_ptr or reg->btf pointer of the register used for locking spin lock. But the active_spin_lock_id also needs to be compared to ensure whether bpf_spin_unlock is for the same register.

Next, pseudo load instructions are not given a unique reg->id, as they are doing lookup for the same map value (max_entries is never greater than 1).

Essentially, we consider that the tuple of (active_spin_lock_ptr, active_spin_lock_id) will always be unique for any kind of argument to bpf_spin_{lock,unlock}.

Note that this can be extended in the future to also remember offset used for locking, so that we can introduce multiple bpf_spin_lock fields in the same allocation.

---
 include/linux/bpf_verifier.h |  3 ++-
 kernel/bpf/verifier.c        | 39 +++++++++++++++++++++++++-----------
 2 files changed, 29 insertions(+), 13 deletions(-)
include/linux/bpf_verifier.h +1 -1
index 2a9dcefca3b6..00c21ad6f61c 100644
@@ -348,7 +348,8 @@ struct bpf_verifier_state {
      u32 branches;
      u32 insn_idx;
      u32 curframe;
-     u32 active_spin_lock;
+     void *active_spin_lock_ptr;
Quoted 9 lines

include/linux/bpf_verifier.h struct bpf_verifier_state { · line 352

Quoted 1 line
include/linux/bpf_verifier.h +1 -0
+     u32 active_spin_lock_id;

{map, id=0} is indeed enough to distinguish different global locks and {map, id} for locks in map values,
but what 'btf' is for?
When is the case when reg->map_ptr is not set?
locks in allocated objects?
Feels too early to add that in this patch.

Quoted 4 lines

It makes active_spin_lock check simpler, just checking active_spin_lock_ptr that to be non-NULL indicates lock is held. Don't have to always check both ptr and id, only need to compare both when verifying that lock is in the same allocation as reg.

Quoted 4 lines

Not following. There is always non-null reg->map_ptr when we come down this path.
At least in the current state of the verifier. So it never assigns that btf afacs.

map is only set when reg->type == PTR_TO_MAP_VALUE, otherwise btf = reg->btf for local kptrs (else branch). Then the map ptr is NULL.
See patch 18 which already added support to local kptrs.

#52 lore raw

Re: [PATCH RFC bpf-next v1 21/32] bpf: Allow locking bpf_spin_lock global variables

Quoted 72 lines · Kumar Kartikeya Dwivedi
Quoted 64 lines · Alexei Starovoitov
Quoted 57 lines · Kumar Kartikeya Dwivedi
Quoted 44 lines · Alexei Starovoitov
Quoted 42 lines · Kumar Kartikeya Dwivedi

Global variables reside in maps accessible using direct_value_addr callbacks, so giving each load instruction's rewrite a unique reg->id disallows us from holding locks which are global.

This is not great, so refactor the active_spin_lock into two separate fields, active_spin_lock_ptr and active_spin_lock_id, which is generic enough to allow it for global variables, map lookups, and local kptr registers at the same time.

Held vs non-held is indicated by active_spin_lock_ptr, which stores the reg->map_ptr or reg->btf pointer of the register used for locking spin lock. But the active_spin_lock_id also needs to be compared to ensure whether bpf_spin_unlock is for the same register.

Next, pseudo load instructions are not given a unique reg->id, as they are doing lookup for the same map value (max_entries is never greater than 1).

Essentially, we consider that the tuple of (active_spin_lock_ptr, active_spin_lock_id) will always be unique for any kind of argument to bpf_spin_{lock,unlock}.

Note that this can be extended in the future to also remember offset used for locking, so that we can introduce multiple bpf_spin_lock fields in the same allocation.

---
 include/linux/bpf_verifier.h |  3 ++-
 kernel/bpf/verifier.c        | 39 +++++++++++++++++++++++++-----------
 2 files changed, 29 insertions(+), 13 deletions(-)
include/linux/bpf_verifier.h +1 -1
index 2a9dcefca3b6..00c21ad6f61c 100644
@@ -348,7 +348,8 @@ struct bpf_verifier_state {
      u32 branches;
      u32 insn_idx;
      u32 curframe;
-     u32 active_spin_lock;
+     void *active_spin_lock_ptr;
Quoted 9 lines

include/linux/bpf_verifier.h struct bpf_verifier_state { · line 352

Quoted 1 line
include/linux/bpf_verifier.h +1 -0
+     u32 active_spin_lock_id;

{map, id=0} is indeed enough to distinguish different global locks and {map, id} for locks in map values,
but what 'btf' is for?
When is the case when reg->map_ptr is not set?
locks in allocated objects?
Feels too early to add that in this patch.

Quoted 4 lines

It makes active_spin_lock check simpler, just checking active_spin_lock_ptr that to be non-NULL indicates lock is held. Don't have to always check both ptr and id, only need to compare both when verifying that lock is in the same allocation as reg.

Quoted 4 lines

Not following. There is always non-null reg->map_ptr when we come down this path.
At least in the current state of the verifier. So it never assigns that btf afacs.

Quoted 4 lines

map is only set when reg->type == PTR_TO_MAP_VALUE, otherwise btf = reg->btf for local kptrs (else branch). Then the map ptr is NULL.
See patch 18 which already added support to local kptrs.

I see. That's what I was missing.

#53 lore raw

Re: [PATCH RFC bpf-next v1 21/32] bpf: Allow locking bpf_spin_lock global variables

In reply to Kumar Kartikeya Dwivedi
Quoted 9 lines · Kumar Kartikeya Dwivedi

Global variables reside in maps accessible using direct_value_addr callbacks, so giving each load instruction's rewrite a unique reg->id disallows us from holding locks which are global.

This is not great, so refactor the active_spin_lock into two separate fields, active_spin_lock_ptr and active_spin_lock_id, which is generic enough to allow it for global variables, map lookups, and local kptr registers at the same time.

Quoted 9 lines

Held vs non-held is indicated by active_spin_lock_ptr, which stores the reg->map_ptr or reg->btf pointer of the register used for locking spin lock. But the active_spin_lock_id also needs to be compared to ensure whether bpf_spin_unlock is for the same register.

Next, pseudo load instructions are not given a unique reg->id, as they are doing lookup for the same map value (max_entries is never greater than 1).

For libbpf-style "internal maps" - like .bss.private further in this series - all the SEC(".bss.private") vars are globbed together into one map_value. e.g.

  struct bpf_spin_lock lock1 SEC(".bss.private");
  struct bpf_spin_lock lock2 SEC(".bss.private");
  ...
  spin_lock(&lock1);
  ...
  spin_lock(&lock2);

will result in same map but different offsets for the direct read (and different aux->map_off set in resolve_pseudo_ldimm64 for use in check_ld_imm). Seems like this patch would assign both same (active_spin_lock_ptr, active_spin_lock_id).

Quoted 8 lines

Essentially, we consider that the tuple of (active_spin_lock_ptr, active_spin_lock_id) will always be unique for any kind of argument to bpf_spin_{lock,unlock}.

Note that this can be extended in the future to also remember offset used for locking, so that we can introduce multiple bpf_spin_lock fields in the same allocation.

In light of the above the "multiple spin locks in same map_value" is probably needed for the common case, probably similar enough to "same allocation" logic.

Quoted 11 lines
---
 include/linux/bpf_verifier.h |  3 ++-
 kernel/bpf/verifier.c        | 39 +++++++++++++++++++++++++-----------
 2 files changed, 29 insertions(+), 13 deletions(-)
include/linux/bpf_verifier.h +0 -0
index 2a9dcefca3b6..00c21ad6f61c 100644

include/linux/bpf_verifier.h struct bpf_verifier_state { · line 352

Quoted 6 lines
include/linux/bpf_verifier.h +2 -1
@@ -348,7 +348,8 @@ struct bpf_verifier_state {
 	u32 branches;
 	u32 insn_idx;
 	u32 curframe;
-	u32 active_spin_lock;
+	void *active_spin_lock_ptr;
+	u32 active_spin_lock_id;

It would be good to make this "(lock_ptr, lock_id) is identifier for lock" concept more concrete by grouping these fields in a struct w/ type enum + union, or something similar. Will make it more obvious that they should be used / set together.

But if you'd prefer to keep it as two fields, active_spin_lock_ptr is a confusing name. In the future with no context as to what that field is, I'd assume that it holds a pointer to a spin_lock instead of a "spin lock identity pointer".

Quoted 59 lines
include/linux/bpf_verifier.h +0 -0
 	bool speculative;
 
 	/* first and last insn idx of this verifier state */
kernel/bpf/verifier.c +20 -6
index b1754fd69f7d..ed19e4036b0a 100644
@@ -1202,7 +1202,8 @@ static int copy_verifier_state(struct bpf_verifier_state *dst_state,
 	}
 	dst_state->speculative = src->speculative;
 	dst_state->curframe = src->curframe;
-	dst_state->active_spin_lock = src->active_spin_lock;
+	dst_state->active_spin_lock_ptr = src->active_spin_lock_ptr;
+	dst_state->active_spin_lock_id = src->active_spin_lock_id;
 	dst_state->branches = src->branches;
 	dst_state->parent = src->parent;
 	dst_state->first_insn_idx = src->first_insn_idx;
@@ -5504,22 +5505,35 @@ static int process_spin_lock(struct bpf_verifier_env *env, int regno,
 		return -EINVAL;
 	}
 	if (is_lock) {
-		if (cur->active_spin_lock) {
+		if (cur->active_spin_lock_ptr) {
 			verbose(env,
 				"Locking two bpf_spin_locks are not allowed\n");
 			return -EINVAL;
 		}
-		cur->active_spin_lock = reg->id;
+		if (map)
+			cur->active_spin_lock_ptr = map;
+		else
+			cur->active_spin_lock_ptr = btf;
+		cur->active_spin_lock_id = reg->id;
 	} else {
-		if (!cur->active_spin_lock) {
+		void *ptr;
+
+		if (map)
+			ptr = map;
+		else
+			ptr = btf;
+
+		if (!cur->active_spin_lock_ptr) {
 			verbose(env, "bpf_spin_unlock without taking a lock\n");
 			return -EINVAL;
 		}
-		if (cur->active_spin_lock != reg->id) {
+		if (cur->active_spin_lock_ptr != ptr ||
+		    cur->active_spin_lock_id != reg->id) {
 			verbose(env, "bpf_spin_unlock of different lock\n");
 			return -EINVAL;
 		}
-		cur->active_spin_lock = 0;
+		cur->active_spin_lock_ptr = NULL;
+		cur->active_spin_lock_id = 0;
 	}
 	return 0;
 }

kernel/bpf/verifier.c static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn) · line 11223

Quoted 3 lines
kernel/bpf/verifier.c +0 -0
@@ -11207,8 +11221,8 @@ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn)
 	    insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE) {
 		dst_reg->type = PTR_TO_MAP_VALUE;
 		dst_reg->off = aux->map_off;

Here's where check_ld_imm uses aux->map_off.

Quoted 44 lines
kernel/bpf/verifier.c +7 -6
-		if (map_value_has_spin_lock(map))
-			dst_reg->id = ++env->id_gen;
+		WARN_ON_ONCE(map->max_entries != 1);
+		/* We want reg->id to be same (0) as map_value is not distinct */
 	} else if (insn->src_reg == BPF_PSEUDO_MAP_FD ||
 		   insn->src_reg == BPF_PSEUDO_MAP_IDX) {
 		dst_reg->type = CONST_PTR_TO_MAP;
@@ -11286,7 +11300,7 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn)
 		return err;
 	}
 
-	if (env->cur_state->active_spin_lock) {
+	if (env->cur_state->active_spin_lock_ptr) {
 		verbose(env, "BPF_LD_[ABS|IND] cannot be used inside bpf_spin_lock-ed region\n");
 		return -EINVAL;
 	}
@@ -12566,7 +12580,8 @@ static bool states_equal(struct bpf_verifier_env *env,
 	if (old->speculative && !cur->speculative)
 		return false;
 
-	if (old->active_spin_lock != cur->active_spin_lock)
+	if (old->active_spin_lock_ptr != cur->active_spin_lock_ptr ||
+	    old->active_spin_lock_id != cur->active_spin_lock_id)
 		return false;
 
 	/* for states to be equal callsites have to be the same
@@ -13213,7 +13228,7 @@ static int do_check(struct bpf_verifier_env *env)
 					return -EINVAL;
 				}
 
-				if (env->cur_state->active_spin_lock &&
+				if (env->cur_state->active_spin_lock_ptr &&
 				    (insn->src_reg == BPF_PSEUDO_CALL ||
 				     insn->imm != BPF_FUNC_spin_unlock)) {
 					verbose(env, "function calls are not allowed while holding a lock\n");
@@ -13250,7 +13265,7 @@ static int do_check(struct bpf_verifier_env *env)
 					return -EINVAL;
 				}
 
-				if (env->cur_state->active_spin_lock) {
+				if (env->cur_state->active_spin_lock_ptr) {
 					verbose(env, "bpf_spin_unlock is missing\n");
 					return -EINVAL;
 				}
author maintainer
#54 lore raw

Re: [PATCH RFC bpf-next v1 21/32] bpf: Allow locking bpf_spin_lock global variables

Quoted 31 lines · Dave Marchevsky
Quoted 9 lines · Kumar Kartikeya Dwivedi

Global variables reside in maps accessible using direct_value_addr callbacks, so giving each load instruction's rewrite a unique reg->id disallows us from holding locks which are global.

This is not great, so refactor the active_spin_lock into two separate fields, active_spin_lock_ptr and active_spin_lock_id, which is generic enough to allow it for global variables, map lookups, and local kptr registers at the same time.

Quoted 9 lines

Held vs non-held is indicated by active_spin_lock_ptr, which stores the reg->map_ptr or reg->btf pointer of the register used for locking spin lock. But the active_spin_lock_id also needs to be compared to ensure whether bpf_spin_unlock is for the same register.

Next, pseudo load instructions are not given a unique reg->id, as they are doing lookup for the same map value (max_entries is never greater than 1).

For libbpf-style "internal maps" - like .bss.private further in this series - all the SEC(".bss.private") vars are globbed together into one map_value. e.g.

  struct bpf_spin_lock lock1 SEC(".bss.private");
  struct bpf_spin_lock lock2 SEC(".bss.private");
  ...
  spin_lock(&lock1);
  ...
  spin_lock(&lock2);
Quoted 4 lines

will result in same map but different offsets for the direct read (and different aux->map_off set in resolve_pseudo_ldimm64 for use in check_ld_imm). Seems like this patch would assign both same (active_spin_lock_ptr, active_spin_lock_id).

That won't be a problem. Two spin locks in a map value or datasec are already rejected on BPF_MAP_CREATE,
so there is no bug. See idx >= info_cnt check in btf_find_struct_field, btf_find_datasec_var.

I can include offset as the third part of the tuple. The problem then is figuring out which lock protects which bpf_list_head. We need another __guarded_by annotation and force users to use that to eliminate the ambiguity. So for now I just put it in the commit log and left it for the future.

But it does seem like it's going to be needed at least for the global case, which should probably do it from the get go.
How does the above idea sound to you?

Quoted 4 lines
Quoted 4 lines

Essentially, we consider that the tuple of (active_spin_lock_ptr, active_spin_lock_id) will always be unique for any kind of argument to bpf_spin_{lock,unlock}.

Quoted 9 lines
Quoted 4 lines

Note that this can be extended in the future to also remember offset used for locking, so that we can introduce multiple bpf_spin_lock fields in the same allocation.

In light of the above the "multiple spin locks in same map_value" is probably needed for the common case, probably similar enough to "same allocation" logic.

Yes, it would be the same idea. Only need to remember an additional field, the 'offset'.
{ptr, id} already distinguishes between allocations. Offset allows to distinguish locks within the same allocation or {ptr, id}.

Quoted 23 lines
Quoted 11 lines
---
 include/linux/bpf_verifier.h |  3 ++-
 kernel/bpf/verifier.c        | 39 +++++++++++++++++++++++++-----------
 2 files changed, 29 insertions(+), 13 deletions(-)
include/linux/bpf_verifier.h +0 -0
index 2a9dcefca3b6..00c21ad6f61c 100644

include/linux/bpf_verifier.h struct bpf_verifier_state { · line 352

Quoted 6 lines
include/linux/bpf_verifier.h +2 -1
@@ -348,7 +348,8 @@ struct bpf_verifier_state {
      u32 branches;
      u32 insn_idx;
      u32 curframe;
-     u32 active_spin_lock;
+     void *active_spin_lock_ptr;
+     u32 active_spin_lock_id;

It would be good to make this "(lock_ptr, lock_id) is identifier for lock" concept more concrete by grouping these fields in a struct w/ type enum + union, or something similar. Will make it more obvious that they should be used / set together.

Quoted 5 lines

But if you'd prefer to keep it as two fields, active_spin_lock_ptr is a confusing name. In the future with no context as to what that field is, I'd assume that it holds a pointer to a spin_lock instead of a "spin lock identity pointer".

That's a good point.

I'm thinking
struct active_lock {
void *id_ptr;
u32 offset;
u32 reg_id;
};
How does that look?

Quoted 109 lines
Quoted 59 lines
include/linux/bpf_verifier.h +0 -0
      bool speculative;
 
      /* first and last insn idx of this verifier state */
kernel/bpf/verifier.c +20 -6
index b1754fd69f7d..ed19e4036b0a 100644
@@ -1202,7 +1202,8 @@ static int copy_verifier_state(struct bpf_verifier_state *dst_state,
      }
      dst_state->speculative = src->speculative;
      dst_state->curframe = src->curframe;
-     dst_state->active_spin_lock = src->active_spin_lock;
+     dst_state->active_spin_lock_ptr = src->active_spin_lock_ptr;
+     dst_state->active_spin_lock_id = src->active_spin_lock_id;
      dst_state->branches = src->branches;
      dst_state->parent = src->parent;
      dst_state->first_insn_idx = src->first_insn_idx;
@@ -5504,22 +5505,35 @@ static int process_spin_lock(struct bpf_verifier_env *env, int regno,
              return -EINVAL;
      }
      if (is_lock) {
-             if (cur->active_spin_lock) {
+             if (cur->active_spin_lock_ptr) {
                      verbose(env,
                              "Locking two bpf_spin_locks are not allowed\n");
                      return -EINVAL;
              }
-             cur->active_spin_lock = reg->id;
+             if (map)
+                     cur->active_spin_lock_ptr = map;
+             else
+                     cur->active_spin_lock_ptr = btf;
+             cur->active_spin_lock_id = reg->id;
      } else {
-             if (!cur->active_spin_lock) {
+             void *ptr;
+
+             if (map)
+                     ptr = map;
+             else
+                     ptr = btf;
+
+             if (!cur->active_spin_lock_ptr) {
                      verbose(env, "bpf_spin_unlock without taking a lock\n");
                      return -EINVAL;
              }
-             if (cur->active_spin_lock != reg->id) {
+             if (cur->active_spin_lock_ptr != ptr ||
+                 cur->active_spin_lock_id != reg->id) {
                      verbose(env, "bpf_spin_unlock of different lock\n");
                      return -EINVAL;
              }
-             cur->active_spin_lock = 0;
+             cur->active_spin_lock_ptr = NULL;
+             cur->active_spin_lock_id = 0;
      }
      return 0;
 }

kernel/bpf/verifier.c static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn) · line 11223

Quoted 3 lines
kernel/bpf/verifier.c +0 -0
@@ -11207,8 +11221,8 @@ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn)
          insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE) {
              dst_reg->type = PTR_TO_MAP_VALUE;
              dst_reg->off = aux->map_off;

Here's where check_ld_imm uses aux->map_off.

Quoted 44 lines
kernel/bpf/verifier.c +7 -6
-             if (map_value_has_spin_lock(map))
-                     dst_reg->id = ++env->id_gen;
+             WARN_ON_ONCE(map->max_entries != 1);
+             /* We want reg->id to be same (0) as map_value is not distinct */
      } else if (insn->src_reg == BPF_PSEUDO_MAP_FD ||
                 insn->src_reg == BPF_PSEUDO_MAP_IDX) {
              dst_reg->type = CONST_PTR_TO_MAP;
@@ -11286,7 +11300,7 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn)
              return err;
      }
 
-     if (env->cur_state->active_spin_lock) {
+     if (env->cur_state->active_spin_lock_ptr) {
              verbose(env, "BPF_LD_[ABS|IND] cannot be used inside bpf_spin_lock-ed region\n");
              return -EINVAL;
      }
@@ -12566,7 +12580,8 @@ static bool states_equal(struct bpf_verifier_env *env,
      if (old->speculative && !cur->speculative)
              return false;
 
-     if (old->active_spin_lock != cur->active_spin_lock)
+     if (old->active_spin_lock_ptr != cur->active_spin_lock_ptr ||
+         old->active_spin_lock_id != cur->active_spin_lock_id)
              return false;
 
      /* for states to be equal callsites have to be the same
@@ -13213,7 +13228,7 @@ static int do_check(struct bpf_verifier_env *env)
                                      return -EINVAL;
                              }
 
-                             if (env->cur_state->active_spin_lock &&
+                             if (env->cur_state->active_spin_lock_ptr &&
                                  (insn->src_reg == BPF_PSEUDO_CALL ||
                                   insn->imm != BPF_FUNC_spin_unlock)) {
                                      verbose(env, "function calls are not allowed while holding a lock\n");
@@ -13250,7 +13265,7 @@ static int do_check(struct bpf_verifier_env *env)
                                      return -EINVAL;
                              }
 
-                             if (env->cur_state->active_spin_lock) {
+                             if (env->cur_state->active_spin_lock_ptr) {
                                      verbose(env, "bpf_spin_unlock is missing\n");
                                      return -EINVAL;
                              }
#55 lore raw

Re: [PATCH RFC bpf-next v1 21/32] bpf: Allow locking bpf_spin_lock global variables

Quoted 43 lines · Kumar Kartikeya Dwivedi
Quoted 31 lines · Dave Marchevsky
Quoted 9 lines · Kumar Kartikeya Dwivedi

Global variables reside in maps accessible using direct_value_addr callbacks, so giving each load instruction's rewrite a unique reg->id disallows us from holding locks which are global.

This is not great, so refactor the active_spin_lock into two separate fields, active_spin_lock_ptr and active_spin_lock_id, which is generic enough to allow it for global variables, map lookups, and local kptr registers at the same time.

Quoted 9 lines

Held vs non-held is indicated by active_spin_lock_ptr, which stores the reg->map_ptr or reg->btf pointer of the register used for locking spin lock. But the active_spin_lock_id also needs to be compared to ensure whether bpf_spin_unlock is for the same register.

Next, pseudo load instructions are not given a unique reg->id, as they are doing lookup for the same map value (max_entries is never greater than 1).

For libbpf-style "internal maps" - like .bss.private further in this series - all the SEC(".bss.private") vars are globbed together into one map_value. e.g.

  struct bpf_spin_lock lock1 SEC(".bss.private");
  struct bpf_spin_lock lock2 SEC(".bss.private");
  ...
  spin_lock(&lock1);
  ...
  spin_lock(&lock2);
Quoted 4 lines

will result in same map but different offsets for the direct read (and different aux->map_off set in resolve_pseudo_ldimm64 for use in check_ld_imm). Seems like this patch would assign both same (active_spin_lock_ptr, active_spin_lock_id).

That won't be a problem. Two spin locks in a map value or datasec are already rejected on BPF_MAP_CREATE,
so there is no bug. See idx >= info_cnt check in btf_find_struct_field, btf_find_datasec_var.

Quoted 5 lines

I can include offset as the third part of the tuple. The problem then is figuring out which lock protects which bpf_list_head. We need another __guarded_by annotation and force users to use that to eliminate the ambiguity. So for now I just put it in the commit log and left it for the future.

Let's not go that far yet.
Extra annotations are just as confusing and non-obvious as putting locks in different sections.
Let's keep one lock per map value limitation for now. libbpf side needs to allow many non-mappable sections though. Single bss.private name is too limiting.

author maintainer
#56 lore raw

Re: [PATCH RFC bpf-next v1 21/32] bpf: Allow locking bpf_spin_lock global variables

Quoted 51 lines · Alexei Starovoitov
Quoted 43 lines · Kumar Kartikeya Dwivedi
Quoted 31 lines · Dave Marchevsky
Quoted 9 lines · Kumar Kartikeya Dwivedi

Global variables reside in maps accessible using direct_value_addr callbacks, so giving each load instruction's rewrite a unique reg->id disallows us from holding locks which are global.

This is not great, so refactor the active_spin_lock into two separate fields, active_spin_lock_ptr and active_spin_lock_id, which is generic enough to allow it for global variables, map lookups, and local kptr registers at the same time.

Quoted 9 lines

Held vs non-held is indicated by active_spin_lock_ptr, which stores the reg->map_ptr or reg->btf pointer of the register used for locking spin lock. But the active_spin_lock_id also needs to be compared to ensure whether bpf_spin_unlock is for the same register.

Next, pseudo load instructions are not given a unique reg->id, as they are doing lookup for the same map value (max_entries is never greater than 1).

For libbpf-style "internal maps" - like .bss.private further in this series - all the SEC(".bss.private") vars are globbed together into one map_value. e.g.

  struct bpf_spin_lock lock1 SEC(".bss.private");
  struct bpf_spin_lock lock2 SEC(".bss.private");
  ...
  spin_lock(&lock1);
  ...
  spin_lock(&lock2);
Quoted 4 lines

will result in same map but different offsets for the direct read (and different aux->map_off set in resolve_pseudo_ldimm64 for use in check_ld_imm). Seems like this patch would assign both same (active_spin_lock_ptr, active_spin_lock_id).

That won't be a problem. Two spin locks in a map value or datasec are already rejected on BPF_MAP_CREATE,
so there is no bug. See idx >= info_cnt check in btf_find_struct_field, btf_find_datasec_var.

Quoted 5 lines

I can include offset as the third part of the tuple. The problem then is figuring out which lock protects which bpf_list_head. We need another __guarded_by annotation and force users to use that to eliminate the ambiguity. So for now I just put it in the commit log and left it for the future.

Quoted 6 lines

Let's not go that far yet.
Extra annotations are just as confusing and non-obvious as putting locks in different sections.
Let's keep one lock per map value limitation for now. libbpf side needs to allow many non-mappable sections though. Single bss.private name is too limiting.

In that case,
Dave, since the libbpf patch is yours, would you be fine with reworking it to support multiple private maps?
Maybe it can just ignore the .XXX part in .bss.private.XXX? Also I think Andrii mentioned once that he wants to eventually merge data and bss, so it might be a good idea to call it .data.private from the start?

#57 lore raw

Re: [PATCH RFC bpf-next v1 21/32] bpf: Allow locking bpf_spin_lock global variables

Quoted 61 lines · Kumar Kartikeya Dwivedi
Quoted 51 lines · Alexei Starovoitov
Quoted 43 lines · Kumar Kartikeya Dwivedi
Quoted 31 lines · Dave Marchevsky
Quoted 9 lines · Kumar Kartikeya Dwivedi

Global variables reside in maps accessible using direct_value_addr callbacks, so giving each load instruction's rewrite a unique reg->id disallows us from holding locks which are global.

This is not great, so refactor the active_spin_lock into two separate fields, active_spin_lock_ptr and active_spin_lock_id, which is generic enough to allow it for global variables, map lookups, and local kptr registers at the same time.

Quoted 9 lines

Held vs non-held is indicated by active_spin_lock_ptr, which stores the reg->map_ptr or reg->btf pointer of the register used for locking spin lock. But the active_spin_lock_id also needs to be compared to ensure whether bpf_spin_unlock is for the same register.

Next, pseudo load instructions are not given a unique reg->id, as they are doing lookup for the same map value (max_entries is never greater than 1).

For libbpf-style "internal maps" - like .bss.private further in this series - all the SEC(".bss.private") vars are globbed together into one map_value. e.g.

  struct bpf_spin_lock lock1 SEC(".bss.private");
  struct bpf_spin_lock lock2 SEC(".bss.private");
  ...
  spin_lock(&lock1);
  ...
  spin_lock(&lock2);
Quoted 4 lines

will result in same map but different offsets for the direct read (and different aux->map_off set in resolve_pseudo_ldimm64 for use in check_ld_imm). Seems like this patch would assign both same (active_spin_lock_ptr, active_spin_lock_id).

That won't be a problem. Two spin locks in a map value or datasec are already rejected on BPF_MAP_CREATE,
so there is no bug. See idx >= info_cnt check in btf_find_struct_field, btf_find_datasec_var.

Quoted 5 lines

I can include offset as the third part of the tuple. The problem then is figuring out which lock protects which bpf_list_head. We need another __guarded_by annotation and force users to use that to eliminate the ambiguity. So for now I just put it in the commit log and left it for the future.

Quoted 6 lines

Let's not go that far yet.
Extra annotations are just as confusing and non-obvious as putting locks in different sections.
Let's keep one lock per map value limitation for now. libbpf side needs to allow many non-mappable sections though. Single bss.private name is too limiting.

Quoted 7 lines

In that case,
Dave, since the libbpf patch is yours, would you be fine with reworking it to support multiple private maps?
Maybe it can just ignore the .XXX part in .bss.private.XXX? Also I think Andrii mentioned once that he wants to eventually merge data and bss, so it might be a good idea to call it .data.private from the start?

I'd probably make all non-canonical names to be not-mmapable. The compiler generates special sections already. Thankfully the code doesn't use them, but it will sooner or later. So libbpf has to create hidden maps for them eventually. They shouldn't be messed up from user space, since it will screw up compiler generated code.

Andrii, what's your take?

#58 lore raw

Re: [PATCH RFC bpf-next v1 21/32] bpf: Allow locking bpf_spin_lock global variables

Quoted 71 lines · Alexei Starovoitov
Quoted 61 lines · Kumar Kartikeya Dwivedi
Quoted 51 lines · Alexei Starovoitov
Quoted 43 lines · Kumar Kartikeya Dwivedi
Quoted 31 lines · Dave Marchevsky
Quoted 9 lines · Kumar Kartikeya Dwivedi

Global variables reside in maps accessible using direct_value_addr callbacks, so giving each load instruction's rewrite a unique reg->id disallows us from holding locks which are global.

This is not great, so refactor the active_spin_lock into two separate fields, active_spin_lock_ptr and active_spin_lock_id, which is generic enough to allow it for global variables, map lookups, and local kptr registers at the same time.

Quoted 9 lines

Held vs non-held is indicated by active_spin_lock_ptr, which stores the reg->map_ptr or reg->btf pointer of the register used for locking spin lock. But the active_spin_lock_id also needs to be compared to ensure whether bpf_spin_unlock is for the same register.

Next, pseudo load instructions are not given a unique reg->id, as they are doing lookup for the same map value (max_entries is never greater than 1).

For libbpf-style "internal maps" - like .bss.private further in this series - all the SEC(".bss.private") vars are globbed together into one map_value. e.g.

  struct bpf_spin_lock lock1 SEC(".bss.private");
  struct bpf_spin_lock lock2 SEC(".bss.private");
  ...
  spin_lock(&lock1);
  ...
  spin_lock(&lock2);
Quoted 4 lines

will result in same map but different offsets for the direct read (and different aux->map_off set in resolve_pseudo_ldimm64 for use in check_ld_imm). Seems like this patch would assign both same (active_spin_lock_ptr, active_spin_lock_id).

That won't be a problem. Two spin locks in a map value or datasec are already rejected on BPF_MAP_CREATE,
so there is no bug. See idx >= info_cnt check in btf_find_struct_field, btf_find_datasec_var.

Quoted 5 lines

I can include offset as the third part of the tuple. The problem then is figuring out which lock protects which bpf_list_head. We need another __guarded_by annotation and force users to use that to eliminate the ambiguity. So for now I just put it in the commit log and left it for the future.

Quoted 6 lines

Let's not go that far yet.
Extra annotations are just as confusing and non-obvious as putting locks in different sections.
Let's keep one lock per map value limitation for now. libbpf side needs to allow many non-mappable sections though. Single bss.private name is too limiting.

Quoted 7 lines

In that case,
Dave, since the libbpf patch is yours, would you be fine with reworking it to support multiple private maps?
Maybe it can just ignore the .XXX part in .bss.private.XXX? Also I think Andrii mentioned once that he wants to eventually merge data and bss, so it might be a good idea to call it .data.private from the start?

Quoted 8 lines

I'd probably make all non-canonical names to be not-mmapable. The compiler generates special sections already. Thankfully the code doesn't use them, but it will sooner or later. So libbpf has to create hidden maps for them eventually. They shouldn't be messed up from user space, since it will screw up compiler generated code.

Andrii, what's your take?

Ok, a bunch of things to unpack. We've also discussed a lot of this with Dave few weeks ago, but I have also few questions.

First, I'd like to not keep extending ".bss" with any custom ".bss.*" sections. This is why we have .data.* and .rodata.* and not .bss (bad, meaningless, historic name).

But I'm totally fine dedicating some other prefix to non-mmapable data sections that won't be exposed in skeleton and, well, not-mmapable. What to name it depends on what we anticipate putting in them?

If it's just for spinlocks, then having something like SEC(".locks") seems best to me. If it's for more stuff, like global kptrs, rbtrees and whatnot, then we'd need a bit more generic name (.private, or whatever, didn't think much on best name). We can also allow .locks.* or .private.* (i.e., keep it uniform with .data and .rodata handling, expect for mmapable aspect).

One benefit for having SEC(".locks") just for spin_locks is that we can teach libbpf to create a multi-element ARRAY map, where each lock variable is put into a separate element. From BPF verifier's perspective, there will be a single BTF type describing spin lock, but multiple "instances" of lock, one per each element. That seems a bit magical and I think, generally speaking, it's best to start supporting multiple lock declarations within single map element (and thus keep track of their offset within map_value); but at least that's an option.

Dave had some concerns about pinning such maps and whatnot, but for starters we decided to not worry about pinning for now. Dave, please bring up remaining issues, if you don't mind.

So to answer Alexei's specific option. I'm still not in favor of just saying "anything that's not .data or .rodata is non-mmapable map". I'd rather carve out naming prefixes with . (which are reserved for libbpf's own use) for these special purpose maps. I don't think that limits anyone, right?

#59 lore raw

Re: [PATCH RFC bpf-next v1 21/32] bpf: Allow locking bpf_spin_lock global variables

Quoted 100 lines · Andrii Nakryiko
Quoted 71 lines · Alexei Starovoitov
Quoted 61 lines · Kumar Kartikeya Dwivedi
Quoted 51 lines · Alexei Starovoitov
Quoted 43 lines · Kumar Kartikeya Dwivedi
Quoted 31 lines · Dave Marchevsky
Quoted 9 lines · Kumar Kartikeya Dwivedi

Global variables reside in maps accessible using direct_value_addr callbacks, so giving each load instruction's rewrite a unique reg->id disallows us from holding locks which are global.

This is not great, so refactor the active_spin_lock into two separate fields, active_spin_lock_ptr and active_spin_lock_id, which is generic enough to allow it for global variables, map lookups, and local kptr registers at the same time.

Quoted 9 lines

Held vs non-held is indicated by active_spin_lock_ptr, which stores the reg->map_ptr or reg->btf pointer of the register used for locking spin lock. But the active_spin_lock_id also needs to be compared to ensure whether bpf_spin_unlock is for the same register.

Next, pseudo load instructions are not given a unique reg->id, as they are doing lookup for the same map value (max_entries is never greater than 1).

For libbpf-style "internal maps" - like .bss.private further in this series - all the SEC(".bss.private") vars are globbed together into one map_value. e.g.

  struct bpf_spin_lock lock1 SEC(".bss.private");
  struct bpf_spin_lock lock2 SEC(".bss.private");
  ...
  spin_lock(&lock1);
  ...
  spin_lock(&lock2);
Quoted 4 lines

will result in same map but different offsets for the direct read (and different aux->map_off set in resolve_pseudo_ldimm64 for use in check_ld_imm). Seems like this patch would assign both same (active_spin_lock_ptr, active_spin_lock_id).

That won't be a problem. Two spin locks in a map value or datasec are already rejected on BPF_MAP_CREATE,
so there is no bug. See idx >= info_cnt check in btf_find_struct_field, btf_find_datasec_var.

Quoted 5 lines

I can include offset as the third part of the tuple. The problem then is figuring out which lock protects which bpf_list_head. We need another __guarded_by annotation and force users to use that to eliminate the ambiguity. So for now I just put it in the commit log and left it for the future.

Quoted 6 lines

Let's not go that far yet.
Extra annotations are just as confusing and non-obvious as putting locks in different sections.
Let's keep one lock per map value limitation for now. libbpf side needs to allow many non-mappable sections though. Single bss.private name is too limiting.

Quoted 7 lines

In that case,
Dave, since the libbpf patch is yours, would you be fine with reworking it to support multiple private maps?
Maybe it can just ignore the .XXX part in .bss.private.XXX? Also I think Andrii mentioned once that he wants to eventually merge data and bss, so it might be a good idea to call it .data.private from the start?

Quoted 8 lines

I'd probably make all non-canonical names to be not-mmapable. The compiler generates special sections already. Thankfully the code doesn't use them, but it will sooner or later. So libbpf has to create hidden maps for them eventually. They shouldn't be messed up from user space, since it will screw up compiler generated code.

Andrii, what's your take?

Ok, a bunch of things to unpack. We've also discussed a lot of this with Dave few weeks ago, but I have also few questions.

First, I'd like to not keep extending ".bss" with any custom ".bss.*" sections. This is why we have .data.* and .rodata.* and not .bss (bad, meaningless, historic name).

But I'm totally fine dedicating some other prefix to non-mmapable data sections that won't be exposed in skeleton and, well, not-mmapable. What to name it depends on what we anticipate putting in them?

If it's just for spinlocks, then having something like SEC(".locks") seems best to me. If it's for more stuff, like global kptrs, rbtrees and whatnot, then we'd need a bit more generic name (.private, or whatever, didn't think much on best name). We can also allow .locks.* or .private.* (i.e., keep it uniform with .data and .rodata handling, expect for mmapable aspect).

Quoted 9 lines

One benefit for having SEC(".locks") just for spin_locks is that we can teach libbpf to create a multi-element ARRAY map, where each lock variable is put into a separate element. From BPF verifier's perspective, there will be a single BTF type describing spin lock, but multiple "instances" of lock, one per each element. That seems a bit magical and I think, generally speaking, it's best to start supporting multiple lock declarations within single map element (and thus keep track of their offset within map_value); but at least that's an option.

".lock" won't work. We need lock+rb_root or lock+list_head to be in the same section.
It should be up to user to name that section with something meaningful. Ideally something like this should be supported:
SEC("enqueue") struct bpf_spin_lock enqueue_lock; SEC("enqueue") struct bpf_list_head enqueue_head __contains(foo, node); SEC("dequeue") struct bpf_spin_lock dequeue_lock; SEC("dequeue") struct bpf_list_head dequeue_head __contains(foo, node);

Quoted 3 lines

Dave had some concerns about pinning such maps and whatnot, but for starters we decided to not worry about pinning for now. Dave, please bring up remaining issues, if you don't mind.

Pinning shouldn't be an issue.
Only mmap is the problem. User space access if fine since kernel will mask out special fields on read/write.

Quoted 5 lines

So to answer Alexei's specific option. I'm still not in favor of just saying "anything that's not .data or .rodata is non-mmapable map". I'd rather carve out naming prefixes with . (which are reserved for libbpf's own use) for these special purpose maps. I don't think that limits anyone, right?

Is backward compat a concern?
Whether to mmap global data is a flag.
It can be opt-in or opt-out.
I'm proposing make all named section to be 'do not mmap'. If a section needs to be mmaped and appear in skeleton the user can do SEC("my_section.mmap")

What you're proposing is to do the other way around:
SEC("enqueue.nommap")
SEC("dequeue.nommap")
in the above example.
I guess it's fine, but more verbose.
The gut feeling is that the use case for naming section will be specifically for lock+rbtree. Everything else will go into common global .data or .rodata. Same thinking about compiler generated special sections with constants. They shouldn't be mmaped by default, but we're not going to hack llvm to add ".nommap" suffix to such sections.
Hence the proposal to avoid mmap by default for all non standard sections.

#60 lore raw

Re: [PATCH RFC bpf-next v1 21/32] bpf: Allow locking bpf_spin_lock global variables

Quoted 142 lines · Alexei Starovoitov
Quoted 100 lines · Andrii Nakryiko
Quoted 71 lines · Alexei Starovoitov
Quoted 61 lines · Kumar Kartikeya Dwivedi
Quoted 51 lines · Alexei Starovoitov
Quoted 43 lines · Kumar Kartikeya Dwivedi
Quoted 31 lines · Dave Marchevsky
Quoted 9 lines · Kumar Kartikeya Dwivedi

Global variables reside in maps accessible using direct_value_addr callbacks, so giving each load instruction's rewrite a unique reg->id disallows us from holding locks which are global.

This is not great, so refactor the active_spin_lock into two separate fields, active_spin_lock_ptr and active_spin_lock_id, which is generic enough to allow it for global variables, map lookups, and local kptr registers at the same time.

Quoted 9 lines

Held vs non-held is indicated by active_spin_lock_ptr, which stores the reg->map_ptr or reg->btf pointer of the register used for locking spin lock. But the active_spin_lock_id also needs to be compared to ensure whether bpf_spin_unlock is for the same register.

Next, pseudo load instructions are not given a unique reg->id, as they are doing lookup for the same map value (max_entries is never greater than 1).

For libbpf-style "internal maps" - like .bss.private further in this series - all the SEC(".bss.private") vars are globbed together into one map_value. e.g.

  struct bpf_spin_lock lock1 SEC(".bss.private");
  struct bpf_spin_lock lock2 SEC(".bss.private");
  ...
  spin_lock(&lock1);
  ...
  spin_lock(&lock2);
Quoted 4 lines

will result in same map but different offsets for the direct read (and different aux->map_off set in resolve_pseudo_ldimm64 for use in check_ld_imm). Seems like this patch would assign both same (active_spin_lock_ptr, active_spin_lock_id).

That won't be a problem. Two spin locks in a map value or datasec are already rejected on BPF_MAP_CREATE,
so there is no bug. See idx >= info_cnt check in btf_find_struct_field, btf_find_datasec_var.

Quoted 5 lines

I can include offset as the third part of the tuple. The problem then is figuring out which lock protects which bpf_list_head. We need another __guarded_by annotation and force users to use that to eliminate the ambiguity. So for now I just put it in the commit log and left it for the future.

Quoted 6 lines

Let's not go that far yet.
Extra annotations are just as confusing and non-obvious as putting locks in different sections.
Let's keep one lock per map value limitation for now. libbpf side needs to allow many non-mappable sections though. Single bss.private name is too limiting.

Quoted 7 lines

In that case,
Dave, since the libbpf patch is yours, would you be fine with reworking it to support multiple private maps?
Maybe it can just ignore the .XXX part in .bss.private.XXX? Also I think Andrii mentioned once that he wants to eventually merge data and bss, so it might be a good idea to call it .data.private from the start?

Quoted 8 lines

I'd probably make all non-canonical names to be not-mmapable. The compiler generates special sections already. Thankfully the code doesn't use them, but it will sooner or later. So libbpf has to create hidden maps for them eventually. They shouldn't be messed up from user space, since it will screw up compiler generated code.

Andrii, what's your take?

Ok, a bunch of things to unpack. We've also discussed a lot of this with Dave few weeks ago, but I have also few questions.

First, I'd like to not keep extending ".bss" with any custom ".bss.*" sections. This is why we have .data.* and .rodata.* and not .bss (bad, meaningless, historic name).

But I'm totally fine dedicating some other prefix to non-mmapable data sections that won't be exposed in skeleton and, well, not-mmapable. What to name it depends on what we anticipate putting in them?

If it's just for spinlocks, then having something like SEC(".locks") seems best to me. If it's for more stuff, like global kptrs, rbtrees and whatnot, then we'd need a bit more generic name (.private, or whatever, didn't think much on best name). We can also allow .locks.* or .private.* (i.e., keep it uniform with .data and .rodata handling, expect for mmapable aspect).

Quoted 9 lines

One benefit for having SEC(".locks") just for spin_locks is that we can teach libbpf to create a multi-element ARRAY map, where each lock variable is put into a separate element. From BPF verifier's perspective, there will be a single BTF type describing spin lock, but multiple "instances" of lock, one per each element. That seems a bit magical and I think, generally speaking, it's best to start supporting multiple lock declarations within single map element (and thus keep track of their offset within map_value); but at least that's an option.

".lock" won't work. We need lock+rb_root or lock+list_head to be in the same section.
It should be up to user to name that section with something meaningful. Ideally something like this should be supported:
SEC("enqueue") struct bpf_spin_lock enqueue_lock; SEC("enqueue") struct bpf_list_head enqueue_head __contains(foo, node); SEC("dequeue") struct bpf_spin_lock dequeue_lock; SEC("dequeue") struct bpf_list_head dequeue_head __contains(foo, node);

Quoted 3 lines

Dave had some concerns about pinning such maps and whatnot, but for starters we decided to not worry about pinning for now. Dave, please bring up remaining issues, if you don't mind.

Pinning shouldn't be an issue.
Only mmap is the problem. User space access if fine since kernel will mask out special fields on read/write.

Quoted 5 lines

So to answer Alexei's specific option. I'm still not in favor of just saying "anything that's not .data or .rodata is non-mmapable map". I'd rather carve out naming prefixes with . (which are reserved for libbpf's own use) for these special purpose maps. I don't think that limits anyone, right?

Is backward compat a concern?
Whether to mmap global data is a flag.
It can be opt-in or opt-out.
I'm proposing make all named section to be 'do not mmap'. If a section needs to be mmaped and appear in skeleton the user can do SEC("my_section.mmap")

Quoted 5 lines

What you're proposing is to do the other way around:
SEC("enqueue.nommap")
SEC("dequeue.nommap")
in the above example.
I guess it's fine, but more verbose.

Well, I didn't propose to use suffixes. Currently user can define SEC(".data.my_custom_use_case"). So I was proposing that we'll just define a different *prefix*, like SEC(".private.enqueue") and SEC(".private.dequeue") for your example above, which will be private to BPF program, not mmap'ed, not exposed in skeleton.

mmap is a bit orthogonal to exposing in skeleton, you can still imagine data section that will be allowed to be initialized from user-space before load but never mmaped afterwards. Just to say that .nommap doesn't necessarily imply that it shouldn't be in a skeleton.

So I still prefer special prefix (.private) and declare that this is both non-mmapable and not-exposed in skeleton.

As for allowing any section. It just feels unnecessary and long-term harmful to allow any section name at this point, tbh.

Quoted 6 lines

The gut feeling is that the use case for naming section will be specifically for lock+rbtree. Everything else will go into common global .data or .rodata. Same thinking about compiler generated special sections with constants. They shouldn't be mmaped by default, but we're not going to hack llvm to add ".nommap" suffix to such sections.
Hence the proposal to avoid mmap by default for all non standard sections.

#61 lore raw

Re: [PATCH RFC bpf-next v1 21/32] bpf: Allow locking bpf_spin_lock global variables

Quoted 144 lines · Andrii Nakryiko
Quoted 142 lines · Alexei Starovoitov
Quoted 100 lines · Andrii Nakryiko
Quoted 71 lines · Alexei Starovoitov
Quoted 61 lines · Kumar Kartikeya Dwivedi
Quoted 51 lines · Alexei Starovoitov
Quoted 43 lines · Kumar Kartikeya Dwivedi
Quoted 31 lines · Dave Marchevsky
Quoted 9 lines · Kumar Kartikeya Dwivedi

Global variables reside in maps accessible using direct_value_addr callbacks, so giving each load instruction's rewrite a unique reg->id disallows us from holding locks which are global.

This is not great, so refactor the active_spin_lock into two separate fields, active_spin_lock_ptr and active_spin_lock_id, which is generic enough to allow it for global variables, map lookups, and local kptr registers at the same time.

Quoted 9 lines

Held vs non-held is indicated by active_spin_lock_ptr, which stores the reg->map_ptr or reg->btf pointer of the register used for locking spin lock. But the active_spin_lock_id also needs to be compared to ensure whether bpf_spin_unlock is for the same register.

Next, pseudo load instructions are not given a unique reg->id, as they are doing lookup for the same map value (max_entries is never greater than 1).

For libbpf-style "internal maps" - like .bss.private further in this series - all the SEC(".bss.private") vars are globbed together into one map_value. e.g.

  struct bpf_spin_lock lock1 SEC(".bss.private");
  struct bpf_spin_lock lock2 SEC(".bss.private");
  ...
  spin_lock(&lock1);
  ...
  spin_lock(&lock2);
Quoted 4 lines

will result in same map but different offsets for the direct read (and different aux->map_off set in resolve_pseudo_ldimm64 for use in check_ld_imm). Seems like this patch would assign both same (active_spin_lock_ptr, active_spin_lock_id).

That won't be a problem. Two spin locks in a map value or datasec are already rejected on BPF_MAP_CREATE,
so there is no bug. See idx >= info_cnt check in btf_find_struct_field, btf_find_datasec_var.

Quoted 5 lines

I can include offset as the third part of the tuple. The problem then is figuring out which lock protects which bpf_list_head. We need another __guarded_by annotation and force users to use that to eliminate the ambiguity. So for now I just put it in the commit log and left it for the future.

Quoted 6 lines

Let's not go that far yet.
Extra annotations are just as confusing and non-obvious as putting locks in different sections.
Let's keep one lock per map value limitation for now. libbpf side needs to allow many non-mappable sections though. Single bss.private name is too limiting.

Quoted 7 lines

In that case,
Dave, since the libbpf patch is yours, would you be fine with reworking it to support multiple private maps?
Maybe it can just ignore the .XXX part in .bss.private.XXX? Also I think Andrii mentioned once that he wants to eventually merge data and bss, so it might be a good idea to call it .data.private from the start?

Quoted 8 lines

I'd probably make all non-canonical names to be not-mmapable. The compiler generates special sections already. Thankfully the code doesn't use them, but it will sooner or later. So libbpf has to create hidden maps for them eventually. They shouldn't be messed up from user space, since it will screw up compiler generated code.

Andrii, what's your take?

Ok, a bunch of things to unpack. We've also discussed a lot of this with Dave few weeks ago, but I have also few questions.

First, I'd like to not keep extending ".bss" with any custom ".bss.*" sections. This is why we have .data.* and .rodata.* and not .bss (bad, meaningless, historic name).

But I'm totally fine dedicating some other prefix to non-mmapable data sections that won't be exposed in skeleton and, well, not-mmapable. What to name it depends on what we anticipate putting in them?

If it's just for spinlocks, then having something like SEC(".locks") seems best to me. If it's for more stuff, like global kptrs, rbtrees and whatnot, then we'd need a bit more generic name (.private, or whatever, didn't think much on best name). We can also allow .locks.* or .private.* (i.e., keep it uniform with .data and .rodata handling, expect for mmapable aspect).

Quoted 9 lines

One benefit for having SEC(".locks") just for spin_locks is that we can teach libbpf to create a multi-element ARRAY map, where each lock variable is put into a separate element. From BPF verifier's perspective, there will be a single BTF type describing spin lock, but multiple "instances" of lock, one per each element. That seems a bit magical and I think, generally speaking, it's best to start supporting multiple lock declarations within single map element (and thus keep track of their offset within map_value); but at least that's an option.

".lock" won't work. We need lock+rb_root or lock+list_head to be in the same section.
It should be up to user to name that section with something meaningful. Ideally something like this should be supported:
SEC("enqueue") struct bpf_spin_lock enqueue_lock; SEC("enqueue") struct bpf_list_head enqueue_head __contains(foo, node); SEC("dequeue") struct bpf_spin_lock dequeue_lock; SEC("dequeue") struct bpf_list_head dequeue_head __contains(foo, node);

Quoted 3 lines

Dave had some concerns about pinning such maps and whatnot, but for starters we decided to not worry about pinning for now. Dave, please bring up remaining issues, if you don't mind.

Pinning shouldn't be an issue.
Only mmap is the problem. User space access if fine since kernel will mask out special fields on read/write.

Quoted 5 lines

So to answer Alexei's specific option. I'm still not in favor of just saying "anything that's not .data or .rodata is non-mmapable map". I'd rather carve out naming prefixes with . (which are reserved for libbpf's own use) for these special purpose maps. I don't think that limits anyone, right?

Is backward compat a concern?
Whether to mmap global data is a flag.
It can be opt-in or opt-out.
I'm proposing make all named section to be 'do not mmap'. If a section needs to be mmaped and appear in skeleton the user can do SEC("my_section.mmap")

Quoted 8 lines
Quoted 5 lines

What you're proposing is to do the other way around:
SEC("enqueue.nommap")
SEC("dequeue.nommap")
in the above example.
I guess it's fine, but more verbose.

Well, I didn't propose to use suffixes. Currently user can define SEC(".data.my_custom_use_case").

... and libbpf will mmap such maps and will expose them in skeleton. My point that it's an existing bug.
Compiler generated .rodata.str1.1 sections should not be messed by user space. There is no BTF for them either.
mmap and subsequent write by user space won't cause a crash for bpf prog, but it won't be doing what C code intended.
There is nothing in there for skeleton and user space to see,
but such map should be created, populated and map_fd provided to the prog to use.

Quoted 9 lines

So I was proposing that we'll just
define a different *prefix*, like SEC(".private.enqueue") and SEC(".private.dequeue") for your example above, which will be private to BPF program, not mmap'ed, not exposed in skeleton.

mmap is a bit orthogonal to exposing in skeleton, you can still imagine data section that will be allowed to be initialized from user-space before load but never mmaped afterwards. Just to say that .nommap doesn't necessarily imply that it shouldn't be in a skeleton.

Well. That's true for normal skeleton and for lskel, but not the case for kernel skel that doesn't have mmap. Exposing a map in skel implies that it will be accessed not only after _open and before _load, but after load as well. We can say that mmap != expose in skeleton, but I really don't see such feature being useful.

Quoted 5 lines

So I still prefer special prefix (.private) and declare that this is both non-mmapable and not-exposed in skeleton.

As for allowing any section. It just feels unnecessary and long-term harmful to allow any section name at this point, tbh.

Fine. How about a single new character instead of '.private' prefix ? Like SEC("#enqueue") that would mean no-skel and no-mmap ?

Or double dot SEC("..enqueue") ?

'.private' is too verbose and when it's read in the context of C file looks out of place and confusing.

#62 lore raw

Re: [PATCH RFC bpf-next v1 21/32] bpf: Allow locking bpf_spin_lock global variables

Quoted 152 lines · Alexei Starovoitov
Quoted 150 lines · Andrii Nakryiko
Quoted 147 lines · Alexei Starovoitov
Quoted 100 lines · Andrii Nakryiko
Quoted 71 lines · Alexei Starovoitov
Quoted 61 lines · Kumar Kartikeya Dwivedi
Quoted 51 lines · Alexei Starovoitov
Quoted 43 lines · Kumar Kartikeya Dwivedi
Quoted 31 lines · Dave Marchevsky
Quoted 9 lines · Kumar Kartikeya Dwivedi

Global variables reside in maps accessible using direct_value_addr callbacks, so giving each load instruction's rewrite a unique reg->id disallows us from holding locks which are global.

This is not great, so refactor the active_spin_lock into two separate fields, active_spin_lock_ptr and active_spin_lock_id, which is generic enough to allow it for global variables, map lookups, and local kptr registers at the same time.

Quoted 9 lines

Held vs non-held is indicated by active_spin_lock_ptr, which stores the reg->map_ptr or reg->btf pointer of the register used for locking spin lock. But the active_spin_lock_id also needs to be compared to ensure whether bpf_spin_unlock is for the same register.

Next, pseudo load instructions are not given a unique reg->id, as they are doing lookup for the same map value (max_entries is never greater than 1).

For libbpf-style "internal maps" - like .bss.private further in this series - all the SEC(".bss.private") vars are globbed together into one map_value. e.g.

  struct bpf_spin_lock lock1 SEC(".bss.private");
  struct bpf_spin_lock lock2 SEC(".bss.private");
  ...
  spin_lock(&lock1);
  ...
  spin_lock(&lock2);
Quoted 4 lines

will result in same map but different offsets for the direct read (and different aux->map_off set in resolve_pseudo_ldimm64 for use in check_ld_imm). Seems like this patch would assign both same (active_spin_lock_ptr, active_spin_lock_id).

That won't be a problem. Two spin locks in a map value or datasec are already rejected on BPF_MAP_CREATE,
so there is no bug. See idx >= info_cnt check in btf_find_struct_field, btf_find_datasec_var.

Quoted 5 lines

I can include offset as the third part of the tuple. The problem then is figuring out which lock protects which bpf_list_head. We need another __guarded_by annotation and force users to use that to eliminate the ambiguity. So for now I just put it in the commit log and left it for the future.

Quoted 6 lines

Let's not go that far yet.
Extra annotations are just as confusing and non-obvious as putting locks in different sections.
Let's keep one lock per map value limitation for now. libbpf side needs to allow many non-mappable sections though. Single bss.private name is too limiting.

Quoted 7 lines

In that case,
Dave, since the libbpf patch is yours, would you be fine with reworking it to support multiple private maps?
Maybe it can just ignore the .XXX part in .bss.private.XXX? Also I think Andrii mentioned once that he wants to eventually merge data and bss, so it might be a good idea to call it .data.private from the start?

Quoted 8 lines

I'd probably make all non-canonical names to be not-mmapable. The compiler generates special sections already. Thankfully the code doesn't use them, but it will sooner or later. So libbpf has to create hidden maps for them eventually. They shouldn't be messed up from user space, since it will screw up compiler generated code.

Andrii, what's your take?

Ok, a bunch of things to unpack. We've also discussed a lot of this with Dave few weeks ago, but I have also few questions.

First, I'd like to not keep extending ".bss" with any custom ".bss.*" sections. This is why we have .data.* and .rodata.* and not .bss (bad, meaningless, historic name).

But I'm totally fine dedicating some other prefix to non-mmapable data sections that won't be exposed in skeleton and, well, not-mmapable. What to name it depends on what we anticipate putting in them?

If it's just for spinlocks, then having something like SEC(".locks") seems best to me. If it's for more stuff, like global kptrs, rbtrees and whatnot, then we'd need a bit more generic name (.private, or whatever, didn't think much on best name). We can also allow .locks.* or .private.* (i.e., keep it uniform with .data and .rodata handling, expect for mmapable aspect).

Quoted 9 lines

One benefit for having SEC(".locks") just for spin_locks is that we can teach libbpf to create a multi-element ARRAY map, where each lock variable is put into a separate element. From BPF verifier's perspective, there will be a single BTF type describing spin lock, but multiple "instances" of lock, one per each element. That seems a bit magical and I think, generally speaking, it's best to start supporting multiple lock declarations within single map element (and thus keep track of their offset within map_value); but at least that's an option.

".lock" won't work. We need lock+rb_root or lock+list_head to be in the same section.
It should be up to user to name that section with something meaningful. Ideally something like this should be supported:
SEC("enqueue") struct bpf_spin_lock enqueue_lock; SEC("enqueue") struct bpf_list_head enqueue_head __contains(foo, node); SEC("dequeue") struct bpf_spin_lock dequeue_lock; SEC("dequeue") struct bpf_list_head dequeue_head __contains(foo, node);

Quoted 3 lines

Dave had some concerns about pinning such maps and whatnot, but for starters we decided to not worry about pinning for now. Dave, please bring up remaining issues, if you don't mind.

Pinning shouldn't be an issue.
Only mmap is the problem. User space access if fine since kernel will mask out special fields on read/write.

Quoted 5 lines

So to answer Alexei's specific option. I'm still not in favor of just saying "anything that's not .data or .rodata is non-mmapable map". I'd rather carve out naming prefixes with . (which are reserved for libbpf's own use) for these special purpose maps. I don't think that limits anyone, right?

Is backward compat a concern?
Whether to mmap global data is a flag.
It can be opt-in or opt-out.
I'm proposing make all named section to be 'do not mmap'. If a section needs to be mmaped and appear in skeleton the user can do SEC("my_section.mmap")

What you're proposing is to do the other way around:
SEC("enqueue.nommap")
SEC("dequeue.nommap")
in the above example.
I guess it's fine, but more verbose.

Quoted 5 lines
Quoted 2 lines

Well, I didn't propose to use suffixes. Currently user can define SEC(".data.my_custom_use_case").

... and libbpf will mmap such maps and will expose them in skeleton. My point that it's an existing bug.

hm... it's not a bug, it's a desired feature. I wanted

int my_var SEC(".data.mine");

to be just like .data but in a separate map. So no bug here.

Quoted 2 lines

Compiler generated .rodata.str1.1 sections should not be messed by user space. There is no BTF for them either.

Shouldn't but could if they wanted to without skeleton as well. In generated skeleton there will be an empty struct for this and no field for each of compiler's constant. User has to intentionally do something to harm themselves, which we can never stop either way.

So stuff like .rodata.str1.1 exposes a bit of compiler implementation details, but overall idea of allowing custom .data.xxx and .rodata.xxx sections was to make them mmapable and readable/writable through skeleton.

Carving out some sub-namespace based on special suffix feels wrong.

Quoted 15 lines

mmap and subsequent write by user space won't cause a crash for bpf prog, but it won't be doing what C code intended.
There is nothing in there for skeleton and user space to see,
but such map should be created, populated and map_fd provided to the prog to use.

Quoted 9 lines

So I was proposing that we'll just
define a different *prefix*, like SEC(".private.enqueue") and SEC(".private.dequeue") for your example above, which will be private to BPF program, not mmap'ed, not exposed in skeleton.

mmap is a bit orthogonal to exposing in skeleton, you can still imagine data section that will be allowed to be initialized from user-space before load but never mmaped afterwards. Just to say that .nommap doesn't necessarily imply that it shouldn't be in a skeleton.

Quoted 6 lines

Well. That's true for normal skeleton and for lskel, but not the case for kernel skel that doesn't have mmap. Exposing a map in skel implies that it will be accessed not only after _open and before _load, but after load as well. We can say that mmap != expose in skeleton, but I really don't see such feature being useful.

It's basically what .rodata is, actually. It's something that's settable from user-space before load and that's it. Yes, you can read it after load, but no one does it in practice. But we are digressing, I understand you want to make this short and sweet and I agree with you. I just disagree about wildcard rule for any non-dotted ELF section or using special suffix. See below.

Quoted 7 lines
Quoted 6 lines

So I still prefer special prefix (.private) and declare that this is both non-mmapable and not-exposed in skeleton.

As for allowing any section. It just feels unnecessary and long-term harmful to allow any section name at this point, tbh.

Quoted 7 lines

Fine. How about a single new character instead of '.private' prefix ? Like SEC("#enqueue") that would mean no-skel and no-mmap ?

Or double dot SEC("..enqueue") ?

'.private' is too verbose and when it's read in the context of C file looks out of place and confusing.

As I said, I gave zero thought to .private, I just took it from ".bss.private". I'd like to keep it "dotted", so SEC("#something") is very "unusual". Me not like.

For double-dot, could be just SEC("..data") and generalized to SEC("..data.<custom>")? BTW, we can add a macro, similar to __kconfig, to hide more descriptive and longer name. E.g.,

struct bpf_spin_lock my_lock __internal;

__internal, __private, __secret, don't know, naming is hard.

#63 lore raw

Re: [PATCH RFC bpf-next v1 21/32] bpf: Allow locking bpf_spin_lock global variables

Quoted 4 lines · Andrii Nakryiko
Quoted 4 lines
Quoted 3 lines

Well, I didn't propose to use suffixes. Currently user can define SEC(".data.my_custom_use_case").

Quoted 8 lines
Quoted 2 lines

... and libbpf will mmap such maps and will expose them in skeleton. My point that it's an existing bug.

hm... it's not a bug, it's a desired feature. I wanted

int my_var SEC(".data.mine");

to be just like .data but in a separate map. So no bug here.

".rodata.*" and ".data.*" section names are effectively reserved by the compiler. Sooner or later there will be trouble if users start mixing compiler sections with their own section names like ".data.mine".
In bpf backend we specicialy check for '.rodata' prefix and avoid emitting BTF. llvm can emit .rodata.cst%d, .rodata.str%d.%d, .data.rel, etc
Not sure how many such special sections will be generated once
bpf progs will get bigger, but creating them as maps will waste
plenty of kernel memory due to page align in bpf-array.
llvm should probably combine them when possible and minimize section usage in general, but that's orthogonal.
Mixing user and compiler sections under the same prefix is just asking for trouble.

Quoted 8 lines
Quoted 2 lines

Compiler generated .rodata.str1.1 sections should not be messed by user space. There is no BTF for them either.

Shouldn't but could if they wanted to without skeleton as well. In generated skeleton there will be an empty struct for this and no field for each of compiler's constant. User has to intentionally do something to harm themselves, which we can never stop either way.

Quoted 4 lines

So stuff like .rodata.str1.1 exposes a bit of compiler implementation details, but overall idea of allowing custom .data.xxx and .rodata.xxx sections was to make them mmapable and readable/writable through skeleton.

It's not a good idea to expose compiler internals into skeleton and even worse to ask users to operate in the compiler's namespace.

Quoted 1 line

Carving out some sub-namespace based on special suffix feels wrong.

agree. suffix doesn't work, since prefix is already owned by the compiler.

Quoted 39 lines
Quoted 15 lines

mmap and subsequent write by user space won't cause a crash for bpf prog, but it won't be doing what C code intended.
There is nothing in there for skeleton and user space to see,
but such map should be created, populated and map_fd provided to the prog to use.

Quoted 9 lines

So I was proposing that we'll just
define a different *prefix*, like SEC(".private.enqueue") and SEC(".private.dequeue") for your example above, which will be private to BPF program, not mmap'ed, not exposed in skeleton.

mmap is a bit orthogonal to exposing in skeleton, you can still imagine data section that will be allowed to be initialized from user-space before load but never mmaped afterwards. Just to say that .nommap doesn't necessarily imply that it shouldn't be in a skeleton.

Quoted 6 lines

Well. That's true for normal skeleton and for lskel, but not the case for kernel skel that doesn't have mmap. Exposing a map in skel implies that it will be accessed not only after _open and before _load, but after load as well. We can say that mmap != expose in skeleton, but I really don't see such feature being useful.

It's basically what .rodata is, actually. It's something that's settable from user-space before load and that's it. Yes, you can read it after load, but no one does it in practice. But we are digressing, I understand you want to make this short and sweet and I agree with you. I just disagree about wildcard rule for any non-dotted ELF section or using special suffix. See below.

Quoted 10 lines
Quoted 6 lines

So I still prefer special prefix (.private) and declare that this is both non-mmapable and not-exposed in skeleton.

As for allowing any section. It just feels unnecessary and long-term harmful to allow any section name at this point, tbh.

Fine. How about a single new character instead of '.private' prefix ? Like SEC("#enqueue") that would mean no-skel and no-mmap ?

Quoted 8 lines
Quoted 4 lines

Or double dot SEC("..enqueue") ?

'.private' is too verbose and when it's read in the context of C file looks out of place and confusing.

As I said, I gave zero thought to .private, I just took it from ".bss.private". I'd like to keep it "dotted", so SEC("#something") is very "unusual". Me not like.

Why is this unusual? We have SEC("?tc") already. SEC("#foo") is very similar.
Dot prefix is special. Something compiler will generate whereas the section that starts with [A-z] is user's. So reserving a prefix that starts from [A-z] would be wrong.

Quoted 5 lines

For double-dot, could be just SEC("..data") and generalized to SEC("..data.<custom>")? BTW, we can add a macro, similar to __kconfig, to hide more descriptive and longer name. E.g.,

struct bpf_spin_lock my_lock __internal;

Macro doesn't help, since the namespace is broken anyway. '..data' is dangerous because something might be doing strstr(".data") instead of strcmp(".data") and will match that section erroneously.

Quoted 1 line

__internal, __private, __secret, don't know, naming is hard.

Right. We already use special meaning for text section names: tc, xdp which is also far from ideal, but that's too late to change. For data I'm arguing that only '.[ro]data' should appear in skel and compiler internals '.[ro]data.*' should not leak to users. Then emit all of [A-z] starting sections, since that's what compilers and linkers will do, but reserve a single character like '#' or whatever other char to mean that this section shouldn't be mmapable.

#64 lore raw

Re: [PATCH RFC bpf-next v1 21/32] bpf: Allow locking bpf_spin_lock global variables

Quoted 6 lines · Alexei Starovoitov
Quoted 4 lines · Andrii Nakryiko
Quoted 4 lines
Quoted 3 lines

Well, I didn't propose to use suffixes. Currently user can define SEC(".data.my_custom_use_case").

Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

... and libbpf will mmap such maps and will expose them in skeleton. My point that it's an existing bug.

hm... it's not a bug, it's a desired feature. I wanted

int my_var SEC(".data.mine");

to be just like .data but in a separate map. So no bug here.

Not to bury the actual proposal at the end of this email, I'll put it here upfront, as I think it's a better compromise.

Given the initial problem was that libbpf creates an mmap-able array for data sections, how about we make libbpf smarter.

The rule is simple and unambiguous: if ELF data section doesn't contain any global variable, libbpf will not add MMAPABLE flag? I.e., if it's special compiler sections which have no variables, or if it's user data section that only has static variables (which explicitly are not to be exposed in BPF skeleton), libbpf just creates non-mmapable array and we don't expose such sections as skeleton structs.

User can still enforce MMAPABLE flag with explicit bpf_map__set_map_flags(), if necessary, so if libbpf's default behavior isn't sufficient and user intended mmapable array, they can still get this working.

That would cover your use case and won't require any new naming conventions. WDYT?

Quoted 10 lines

".rodata.*" and ".data.*" section names are effectively reserved by the compiler. Sooner or later there will be trouble if users start mixing compiler sections with their own section names like ".data.mine".
In bpf backend we specicialy check for '.rodata' prefix and avoid emitting BTF. llvm can emit .rodata.cst%d, .rodata.str%d.%d, .data.rel, etc
Not sure how many such special sections will be generated once
bpf progs will get bigger, but creating them as maps will waste plenty of kernel memory due to page align in bpf-array.
llvm should probably combine them when possible and minimize section usage in general, but that's orthogonal.

agree about the combining in LLVM and it's an optimization that libbpf should be oblivious to

Quoted 2 lines

Mixing user and compiler sections under the same prefix is just asking for trouble.

ELF spec specifies that .data/.data1 and .rodata/.rodata1 are special. And also adds:

Section names with a dot (.) prefix are reserved for the system, although applications may use these sections if their existing meanings are satisfactory. Applications may use names without the prefix to avoid conflicts with system sections. The object file format lets one define sections not in the list above. An object file may have more than one section with the same name.

I treat libbpf as "a system" and thus treat .* is reserved for libbpf use.

As for not creating a map for .rodata.cst* and others. Isn't it exactly the same as for .rodata? Compiler might generate relocation against that section and will expect to be able to access contents of such map at runtime (e.g., for struct literals, string constants, etc). So I don't think we can do that.

Quoted 64 lines
Quoted 8 lines
Quoted 2 lines

Compiler generated .rodata.str1.1 sections should not be messed by user space. There is no BTF for them either.

Shouldn't but could if they wanted to without skeleton as well. In generated skeleton there will be an empty struct for this and no field for each of compiler's constant. User has to intentionally do something to harm themselves, which we can never stop either way.

Quoted 4 lines

So stuff like .rodata.str1.1 exposes a bit of compiler implementation details, but overall idea of allowing custom .data.xxx and .rodata.xxx sections was to make them mmapable and readable/writable through skeleton.

It's not a good idea to expose compiler internals into skeleton and even worse to ask users to operate in the compiler's namespace.

Quoted 1 line

Carving out some sub-namespace based on special suffix feels wrong.

agree. suffix doesn't work, since prefix is already owned by the compiler.

Quoted 44 lines
Quoted 15 lines

mmap and subsequent write by user space won't cause a crash for bpf prog, but it won't be doing what C code intended.
There is nothing in there for skeleton and user space to see,
but such map should be created, populated and map_fd provided to the prog to use.

Quoted 9 lines

So I was proposing that we'll just
define a different *prefix*, like SEC(".private.enqueue") and SEC(".private.dequeue") for your example above, which will be private to BPF program, not mmap'ed, not exposed in skeleton.

mmap is a bit orthogonal to exposing in skeleton, you can still imagine data section that will be allowed to be initialized from user-space before load but never mmaped afterwards. Just to say that .nommap doesn't necessarily imply that it shouldn't be in a skeleton.

Quoted 6 lines

Well. That's true for normal skeleton and for lskel, but not the case for kernel skel that doesn't have mmap. Exposing a map in skel implies that it will be accessed not only after _open and before _load, but after load as well. We can say that mmap != expose in skeleton, but I really don't see such feature being useful.

It's basically what .rodata is, actually. It's something that's settable from user-space before load and that's it. Yes, you can read it after load, but no one does it in practice. But we are digressing, I understand you want to make this short and sweet and I agree with you. I just disagree about wildcard rule for any non-dotted ELF section or using special suffix. See below.

Quoted 14 lines
Quoted 6 lines

So I still prefer special prefix (.private) and declare that this is both non-mmapable and not-exposed in skeleton.

As for allowing any section. It just feels unnecessary and long-term harmful to allow any section name at this point, tbh.

Fine. How about a single new character instead of '.private' prefix ? Like SEC("#enqueue") that would mean no-skel and no-mmap ?

Or double dot SEC("..enqueue") ?

'.private' is too verbose and when it's read in the context of C file looks out of place and confusing.

Quoted 6 lines
Quoted 3 lines

As I said, I gave zero thought to .private, I just took it from ".bss.private". I'd like to keep it "dotted", so SEC("#something") is very "unusual". Me not like.

Why is this unusual? We have SEC("?tc") already.
SEC("#foo") is very similar.

Unusual because these sections were so far used for BPF programs, not for data. It's not the end of the world, but just not something I'd like to do.

Quoted 3 lines

Dot prefix is special. Something compiler will generate whereas the section that starts with [A-z] is user's. So reserving a prefix that starts from [A-z] would be wrong.

If we allow [a-zA-Z]* sections for data, we introduce potential conflict for new SEC("abc") program annotations. Why would we do this and cause more problems?

Quoted 11 lines
Quoted 6 lines

For double-dot, could be just SEC("..data") and generalized to SEC("..data.<custom>")? BTW, we can add a macro, similar to __kconfig, to hide more descriptive and longer name. E.g.,

struct bpf_spin_lock my_lock __internal;

Macro doesn't help, since the namespace is broken anyway. '..data' is dangerous because something might be doing strstr(".data") instead of strcmp(".data") and will match that section erroneously.

Not breaking broken/naive code is hardly a reason for anything. If someone is doing strstr() and expects it to work as a prefix check, well, it's a bug that should be fixed.

Quoted 6 lines
Quoted 1 line

__internal, __private, __secret, don't know, naming is hard.

Right. We already use special meaning for text section names: tc, xdp which is also far from ideal, but that's too late to change. For data I'm arguing that only '.[ro]data' should appear in skel and compiler internals '.[ro]data.*' should not leak to users.

I disagree. It's an already supported libbpf feature and I think it's fine to keep it this way. It's not hard to avoid compiler "special" sections, if it's at all a problem to share that section with the compiler.

Quoted 3 lines

Then emit all of [A-z] starting sections, since that's what compilers and linkers will do, but reserve a single character like '#' or whatever other char to mean that this section shouldn't be mmapable.

#65 lore raw

Re: [PATCH RFC bpf-next v1 21/32] bpf: Allow locking bpf_spin_lock global variables

Quoted 32 lines · Andrii Nakryiko
Quoted 6 lines · Alexei Starovoitov
Quoted 4 lines · Andrii Nakryiko
Quoted 4 lines
Quoted 3 lines

Well, I didn't propose to use suffixes. Currently user can define SEC(".data.my_custom_use_case").

Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

... and libbpf will mmap such maps and will expose them in skeleton. My point that it's an existing bug.

hm... it's not a bug, it's a desired feature. I wanted

int my_var SEC(".data.mine");

to be just like .data but in a separate map. So no bug here.

Not to bury the actual proposal at the end of this email, I'll put it here upfront, as I think it's a better compromise.

Given the initial problem was that libbpf creates an mmap-able array for data sections, how about we make libbpf smarter.

The rule is simple and unambiguous: if ELF data section doesn't contain any global variable, libbpf will not add MMAPABLE flag? I.e., if it's special compiler sections which have no variables, or if it's user data section that only has static variables (which explicitly are not to be exposed in BPF skeleton), libbpf just creates non-mmapable array and we don't expose such sections as skeleton structs.

Quoted 9 lines

User can still enforce MMAPABLE flag with explicit bpf_map__set_map_flags(), if necessary, so if libbpf's default behavior isn't sufficient and user intended mmapable array, they can still get this working.

That would cover your use case and won't require any new naming conventions. WDYT?

To close the loop, I went ahead and implemented this proposal in code. See [0]. I think it should be a good first step and should unblock all the linked list and rbtree_node work. Please give it a try.

[0] https://patchwork.kernel.org/project/netdevbpf/list/?series=686066&state=*

[...]

#66 lore raw

Re: [PATCH RFC bpf-next v1 21/32] bpf: Allow locking bpf_spin_lock global variables

In reply to Alexei Starovoitov
Quoted 111 lines · Alexei Starovoitov
Quoted 100 lines · Andrii Nakryiko
Quoted 71 lines · Alexei Starovoitov
Quoted 61 lines · Kumar Kartikeya Dwivedi
Quoted 51 lines · Alexei Starovoitov
Quoted 43 lines · Kumar Kartikeya Dwivedi
Quoted 31 lines · Dave Marchevsky
Quoted 9 lines · Kumar Kartikeya Dwivedi

Global variables reside in maps accessible using direct_value_addr callbacks, so giving each load instruction's rewrite a unique reg->id disallows us from holding locks which are global.

This is not great, so refactor the active_spin_lock into two separate fields, active_spin_lock_ptr and active_spin_lock_id, which is generic enough to allow it for global variables, map lookups, and local kptr registers at the same time.

Quoted 9 lines

Held vs non-held is indicated by active_spin_lock_ptr, which stores the reg->map_ptr or reg->btf pointer of the register used for locking spin lock. But the active_spin_lock_id also needs to be compared to ensure whether bpf_spin_unlock is for the same register.

Next, pseudo load instructions are not given a unique reg->id, as they are doing lookup for the same map value (max_entries is never greater than 1).

For libbpf-style "internal maps" - like .bss.private further in this series - all the SEC(".bss.private") vars are globbed together into one map_value. e.g.

  struct bpf_spin_lock lock1 SEC(".bss.private");
  struct bpf_spin_lock lock2 SEC(".bss.private");
  ...
  spin_lock(&lock1);
  ...
  spin_lock(&lock2);
Quoted 4 lines

will result in same map but different offsets for the direct read (and different aux->map_off set in resolve_pseudo_ldimm64 for use in check_ld_imm). Seems like this patch would assign both same (active_spin_lock_ptr, active_spin_lock_id).

That won't be a problem. Two spin locks in a map value or datasec are already rejected on BPF_MAP_CREATE,
so there is no bug. See idx >= info_cnt check in btf_find_struct_field, btf_find_datasec_var.

Quoted 5 lines

I can include offset as the third part of the tuple. The problem then is figuring out which lock protects which bpf_list_head. We need another __guarded_by annotation and force users to use that to eliminate the ambiguity. So for now I just put it in the commit log and left it for the future.

Quoted 6 lines

Let's not go that far yet.
Extra annotations are just as confusing and non-obvious as putting locks in different sections.
Let's keep one lock per map value limitation for now. libbpf side needs to allow many non-mappable sections though. Single bss.private name is too limiting.

Quoted 7 lines

In that case,
Dave, since the libbpf patch is yours, would you be fine with reworking it to support multiple private maps?
Maybe it can just ignore the .XXX part in .bss.private.XXX? Also I think Andrii mentioned once that he wants to eventually merge data and bss, so it might be a good idea to call it .data.private from the start?

Quoted 8 lines

I'd probably make all non-canonical names to be not-mmapable. The compiler generates special sections already. Thankfully the code doesn't use them, but it will sooner or later. So libbpf has to create hidden maps for them eventually. They shouldn't be messed up from user space, since it will screw up compiler generated code.

Andrii, what's your take?

Ok, a bunch of things to unpack. We've also discussed a lot of this with Dave few weeks ago, but I have also few questions.

First, I'd like to not keep extending ".bss" with any custom ".bss.*" sections. This is why we have .data.* and .rodata.* and not .bss (bad, meaningless, historic name).

But I'm totally fine dedicating some other prefix to non-mmapable data sections that won't be exposed in skeleton and, well, not-mmapable. What to name it depends on what we anticipate putting in them?

If it's just for spinlocks, then having something like SEC(".locks") seems best to me. If it's for more stuff, like global kptrs, rbtrees and whatnot, then we'd need a bit more generic name (.private, or whatever, didn't think much on best name). We can also allow .locks.* or .private.* (i.e., keep it uniform with .data and .rodata handling, expect for mmapable aspect).

Quoted 9 lines

One benefit for having SEC(".locks") just for spin_locks is that we can teach libbpf to create a multi-element ARRAY map, where each lock variable is put into a separate element. From BPF verifier's perspective, there will be a single BTF type describing spin lock, but multiple "instances" of lock, one per each element. That seems a bit magical and I think, generally speaking, it's best to start supporting multiple lock declarations within single map element (and thus keep track of their offset within map_value); but at least that's an option.

Quoted 9 lines

".lock" won't work. We need lock+rb_root or lock+list_head to be in the same section.
It should be up to user to name that section with something meaningful. Ideally something like this should be supported:
SEC("enqueue") struct bpf_spin_lock enqueue_lock; SEC("enqueue") struct bpf_list_head enqueue_head __contains(foo, node); SEC("dequeue") struct bpf_spin_lock dequeue_lock; SEC("dequeue") struct bpf_list_head dequeue_head __contains(foo, node);

Isn't the "head and lock must be in same section / map_value" desired, or just a consequence of this implementation? I don't see why it's desirable from user perspective. Seems to have same problem as rbtree RFCv1's rbtree_map struct creating its own bpf_spin_lock, namely not providing a way for multiple datastructures to share same lock in a way that makes sense to the verifier for enforcement.

Quoted 4 lines
Quoted 3 lines

Dave had some concerns about pinning such maps and whatnot, but for starters we decided to not worry about pinning for now. Dave, please bring up remaining issues, if you don't mind.

@Andrii, aside from vague pinning concerns from our last discussion about this, I don't have any specific concerns. A multi-element ".locks" is more appealing to me now, actually, as I think it enables best-of-both-worlds for this impl and my rbtree RFCv2 experiments:

* This series uses (map_ptr, map_value_offset) as lock identity for verification purposes and expects map_ptr for list_head and lock to be the same.
* If my logic in comment preceding this one is correct, downside is no lock sharing between datastructures.

* rbtree RFCv2 uses lock address as lock identity
for verification purposes and requires lock address to be known when verifying program using the lock.
* Downside: no clear path forward for map_in_map general case, can make it work for some specific cases but kludgey.

* If ".locks" exists, supporting multiple lock definitions, we can use locks_sec_offset or locks_sec_map_{key,idx} as lock identity for verification purposes.
* As a result "head and lock must be in same section" requirement is removed, and there's a path forward for map_in_map inner maps to share locks arbitrarily without losing verifiability.
* But I suspect this requires some special handling of the map backing ".locks" on kernel side.

I have some hacks on top of rbtree RFCv2 that are moving in this ".locks" direction, happy to fix them up and send something if I didn't miss anything above.

Regardless, @Kumar, happy to iterate on .bss.private patch until it's in a shape that satisfies everyone.

Quoted 28 lines

Pinning shouldn't be an issue.
Only mmap is the problem. User space access if fine since kernel will mask out special fields on read/write.

Quoted 5 lines

So to answer Alexei's specific option. I'm still not in favor of just saying "anything that's not .data or .rodata is non-mmapable map". I'd rather carve out naming prefixes with . (which are reserved for libbpf's own use) for these special purpose maps. I don't think that limits anyone, right?

Is backward compat a concern?
Whether to mmap global data is a flag.
It can be opt-in or opt-out.
I'm proposing make all named section to be 'do not mmap'. If a section needs to be mmaped and appear in skeleton the user can do SEC("my_section.mmap")

What you're proposing is to do the other way around:
SEC("enqueue.nommap")
SEC("dequeue.nommap")
in the above example.
I guess it's fine, but more verbose.
The gut feeling is that the use case for naming section will be specifically for lock+rbtree. Everything else will go into common global .data or .rodata. Same thinking about compiler generated special sections with constants. They shouldn't be mmaped by default, but we're not going to hack llvm to add ".nommap" suffix to such sections.
Hence the proposal to avoid mmap by default for all non standard sections.

author maintainer
#67 lore raw

Re: [PATCH RFC bpf-next v1 21/32] bpf: Allow locking bpf_spin_lock global variables

Quoted 123 lines · Dave Marchevsky
Quoted 111 lines · Alexei Starovoitov
Quoted 100 lines · Andrii Nakryiko
Quoted 71 lines · Alexei Starovoitov
Quoted 61 lines · Kumar Kartikeya Dwivedi
Quoted 51 lines · Alexei Starovoitov
Quoted 43 lines · Kumar Kartikeya Dwivedi
Quoted 31 lines · Dave Marchevsky
Quoted 9 lines · Kumar Kartikeya Dwivedi

Global variables reside in maps accessible using direct_value_addr callbacks, so giving each load instruction's rewrite a unique reg->id disallows us from holding locks which are global.

This is not great, so refactor the active_spin_lock into two separate fields, active_spin_lock_ptr and active_spin_lock_id, which is generic enough to allow it for global variables, map lookups, and local kptr registers at the same time.

Quoted 9 lines

Held vs non-held is indicated by active_spin_lock_ptr, which stores the reg->map_ptr or reg->btf pointer of the register used for locking spin lock. But the active_spin_lock_id also needs to be compared to ensure whether bpf_spin_unlock is for the same register.

Next, pseudo load instructions are not given a unique reg->id, as they are doing lookup for the same map value (max_entries is never greater than 1).

For libbpf-style "internal maps" - like .bss.private further in this series - all the SEC(".bss.private") vars are globbed together into one map_value. e.g.

  struct bpf_spin_lock lock1 SEC(".bss.private");
  struct bpf_spin_lock lock2 SEC(".bss.private");
  ...
  spin_lock(&lock1);
  ...
  spin_lock(&lock2);
Quoted 4 lines

will result in same map but different offsets for the direct read (and different aux->map_off set in resolve_pseudo_ldimm64 for use in check_ld_imm). Seems like this patch would assign both same (active_spin_lock_ptr, active_spin_lock_id).

That won't be a problem. Two spin locks in a map value or datasec are already rejected on BPF_MAP_CREATE,
so there is no bug. See idx >= info_cnt check in btf_find_struct_field, btf_find_datasec_var.

Quoted 5 lines

I can include offset as the third part of the tuple. The problem then is figuring out which lock protects which bpf_list_head. We need another __guarded_by annotation and force users to use that to eliminate the ambiguity. So for now I just put it in the commit log and left it for the future.

Quoted 6 lines

Let's not go that far yet.
Extra annotations are just as confusing and non-obvious as putting locks in different sections.
Let's keep one lock per map value limitation for now. libbpf side needs to allow many non-mappable sections though. Single bss.private name is too limiting.

Quoted 7 lines

In that case,
Dave, since the libbpf patch is yours, would you be fine with reworking it to support multiple private maps?
Maybe it can just ignore the .XXX part in .bss.private.XXX? Also I think Andrii mentioned once that he wants to eventually merge data and bss, so it might be a good idea to call it .data.private from the start?

Quoted 8 lines

I'd probably make all non-canonical names to be not-mmapable. The compiler generates special sections already. Thankfully the code doesn't use them, but it will sooner or later. So libbpf has to create hidden maps for them eventually. They shouldn't be messed up from user space, since it will screw up compiler generated code.

Andrii, what's your take?

Ok, a bunch of things to unpack. We've also discussed a lot of this with Dave few weeks ago, but I have also few questions.

First, I'd like to not keep extending ".bss" with any custom ".bss.*" sections. This is why we have .data.* and .rodata.* and not .bss (bad, meaningless, historic name).

But I'm totally fine dedicating some other prefix to non-mmapable data sections that won't be exposed in skeleton and, well, not-mmapable. What to name it depends on what we anticipate putting in them?

If it's just for spinlocks, then having something like SEC(".locks") seems best to me. If it's for more stuff, like global kptrs, rbtrees and whatnot, then we'd need a bit more generic name (.private, or whatever, didn't think much on best name). We can also allow .locks.* or .private.* (i.e., keep it uniform with .data and .rodata handling, expect for mmapable aspect).

Quoted 9 lines

One benefit for having SEC(".locks") just for spin_locks is that we can teach libbpf to create a multi-element ARRAY map, where each lock variable is put into a separate element. From BPF verifier's perspective, there will be a single BTF type describing spin lock, but multiple "instances" of lock, one per each element. That seems a bit magical and I think, generally speaking, it's best to start supporting multiple lock declarations within single map element (and thus keep track of their offset within map_value); but at least that's an option.

Quoted 9 lines

".lock" won't work. We need lock+rb_root or lock+list_head to be in the same section.
It should be up to user to name that section with something meaningful. Ideally something like this should be supported:
SEC("enqueue") struct bpf_spin_lock enqueue_lock; SEC("enqueue") struct bpf_list_head enqueue_head __contains(foo, node); SEC("dequeue") struct bpf_spin_lock dequeue_lock; SEC("dequeue") struct bpf_list_head dequeue_head __contains(foo, node);

Quoted 7 lines

Isn't the "head and lock must be in same section / map_value" desired, or just a consequence of this implementation? I don't see why it's desirable from user perspective. Seems to have same problem as rbtree RFCv1's rbtree_map struct creating its own bpf_spin_lock, namely not providing a way for multiple datastructures to share same lock in a way that makes sense to the verifier for enforcement.

There is no such restriction here. You just put a lock and every list or rbtree protected by that lock in the same section.
Then all of them share the same lock for the special section.

#define __private(X) SEC("map" #X)
struct bpf_spin_lock lock __private(a);
struct bpf_list_head head __contains(...) __private(a); struct bpf_rb_root root __contains(...) __private(a);

As I said already, it's also possible to do a more fine grained approach by having multiple of them globally.
Then this multiple separate section based approach is not needed at all. You can have just one private section for such bpf special structures, maybe even by default from libbpf side, as they can't be mmap'd anyway.

libbpf will see that you have bpf_spin_lock, bpf_list_head, bpf_rb_root, it will put them in .data.nommap.

But then the verifier needs to know which lock protects which data. You always need that info, in any approach. Here we assume by default just one bpf_spin_lock so the answer is known.
We can 'learn' that implicitly (storing what we see first in the verifier, e.g. if you added to head while holding lockA, we assume this is the one you'll be using to protect it). Later the same head cannot be added to using lockB.
Or we can just make the user annotate that explicitly, like clang's thread safety annotations (GUARDED_BY(lock) etc.).
Then the spin_lock_off protecting it is stored with other info in bpf_map_value_off_desc.

So compared to the example above, user will just do:
struct bpf_spin_lock lock1;
struct bpf_spin_lock lock2;
struct bpf_list_head head __contains(...) __guarded_by(lock1); struct bpf_list_head head2 __contains(...) __guarded_by(lock2); struct bpf_rb_root root __contains(...) __guarded_by(lock2);

It looks much cleaner to me from a user perspective. Just define what protects what, which also doubles as great documentation.

Regardless, the point is there are no limitations regarding coarse-grained/fine-grained locking or lock sharing. The question is more about how to expose it to the user.

Quoted 10 lines
Quoted 4 lines
Quoted 3 lines

Dave had some concerns about pinning such maps and whatnot, but for starters we decided to not worry about pinning for now. Dave, please bring up remaining issues, if you don't mind.

@Andrii, aside from vague pinning concerns from our last discussion about this, I don't have any specific concerns. A multi-element ".locks" is more appealing to me now, actually, as I think it enables best-of-both-worlds for this impl and my rbtree RFCv2 experiments:

Quoted 6 lines

* This series uses (map_ptr, map_value_offset) as lock identity for verification purposes and expects map_ptr for list_head and lock to be the same.
* If my logic in comment preceding this one is correct, downside is no lock sharing between datastructures.

See above.

Quoted 15 lines

* rbtree RFCv2 uses lock address as lock identity
for verification purposes and requires lock address to be known when verifying program using the lock.
* Downside: no clear path forward for map_in_map general case, can make it work for some specific cases but kludgey.

* If ".locks" exists, supporting multiple lock definitions, we can use locks_sec_offset or locks_sec_map_{key,idx} as lock identity for verification purposes.
* As a result "head and lock must be in same section" requirement is removed, and there's a path forward for map_in_map inner maps to share locks arbitrarily without losing verifiability.
* But I suspect this requires some special handling of the map backing ".locks" on kernel side.

Quoted 3 lines

I have some hacks on top of rbtree RFCv2 that are moving in this ".locks" direction, happy to fix them up and send something if I didn't miss anything above.

I don't really like the ".locks" section or the idea in general. There is nothing really special about locks in particular.
Same problem with bpf_timer. A nommap map approach also allows having more than one bpf_timer globally.

Quoted 4 lines

Regardless, @Kumar, happy to iterate on .bss.private patch until it's in a shape that satisfies everyone.

Great, once the discussion concludes it would be great if you send it out as its own patch, easier for me too.

#68 lore raw

Re: [PATCH RFC bpf-next v1 21/32] bpf: Allow locking bpf_spin_lock global variables

Quoted 10 lines · Kumar Kartikeya Dwivedi

So compared to the example above, user will just do:
struct bpf_spin_lock lock1;
struct bpf_spin_lock lock2;
struct bpf_list_head head __contains(...) __guarded_by(lock1); struct bpf_list_head head2 __contains(...) __guarded_by(lock2); struct bpf_rb_root root __contains(...) __guarded_by(lock2);

It looks much cleaner to me from a user perspective. Just define what protects what, which also doubles as great documentation.

Unfortunately that doesn't work.

We cannot magically exclude the locks from global data because of skel/mmap requirements.
We cannot move the locks automatically, because it involves massive analysis of the code and fixing all offsets in libbpf. So users have to use a different section when using global locks, rb_root, list_head.
Since a different section is needed anyway, it's better to keep one-lock-per-map-value for now.

author maintainer
#69 lore raw

Re: [PATCH RFC bpf-next v1 21/32] bpf: Allow locking bpf_spin_lock global variables

Quoted 15 lines · Alexei Starovoitov
Quoted 10 lines · Kumar Kartikeya Dwivedi

So compared to the example above, user will just do:
struct bpf_spin_lock lock1;
struct bpf_spin_lock lock2;
struct bpf_list_head head __contains(...) __guarded_by(lock1); struct bpf_list_head head2 __contains(...) __guarded_by(lock2); struct bpf_rb_root root __contains(...) __guarded_by(lock2);

It looks much cleaner to me from a user perspective. Just define what protects what, which also doubles as great documentation.

Unfortunately that doesn't work.

Quoted 8 lines

We cannot magically exclude the locks from global data because of skel/mmap requirements.
We cannot move the locks automatically, because it involves massive analysis of the code and fixing all offsets in libbpf. So users have to use a different section when using global locks, rb_root, list_head.
Since a different section is needed anyway, it's better to keep one-lock-per-map-value for now.

Argh, right. Then they need one extra 'annotation', at that point it's a bit questionable.
Also just realized reading your other reply that there are also unanswered questions wrt what we do for BPF_F_LOCK.

#70 lore raw

Re: [PATCH RFC bpf-next v1 21/32] bpf: Allow locking bpf_spin_lock global variables

In reply to Dave Marchevsky
Quoted 11 lines · Dave Marchevsky
Quoted 10 lines

".lock" won't work. We need lock+rb_root or lock+list_head to be in the same section.
It should be up to user to name that section with something meaningful. Ideally something like this should be supported:
SEC("enqueue") struct bpf_spin_lock enqueue_lock; SEC("enqueue") struct bpf_list_head enqueue_head __contains(foo, node); SEC("dequeue") struct bpf_spin_lock dequeue_lock; SEC("dequeue") struct bpf_list_head dequeue_head __contains(foo, node);

Quoted 6 lines

Isn't the "head and lock must be in same section / map_value" desired, or just a consequence of this implementation? I don't see why it's desirable from user perspective. Seems to have same problem as rbtree RFCv1's rbtree_map struct creating its own bpf_spin_lock, namely not providing a way for multiple datastructures to share same lock in a way that makes sense to the verifier for enforcement.

The requirement to have only one lock in an "allocation" (which is map value or one global section) comes from the need to take that lock when doing map updates.
Shared lists/rbtree might require the verifier to take that lock too. We can improve things with __guarded_by() tags in the future, but I prefer to start with simple one-lock-per-map-value.

author maintainer
#71 lore raw

RFC PATCH 22/32 bpf: Bump BTF_KFUNC_SET_MAX_CNT

In reply to Kumar Kartikeya Dwivedi

The current number of kfuncs limit in the btf_id_set8 wouldn't suffice any longer, especially keeping in mind the future patches in this series, hence bump it to 64.

---
 kernel/bpf/btf.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/bpf/btf.c +1 -1
index c8d4513cc73e..439c980419b9 100644
@@ -208,7 +208,7 @@ enum btf_kfunc_hook {
 };
 
 enum {
-	BTF_KFUNC_SET_MAX_CNT = 32,
+	BTF_KFUNC_SET_MAX_CNT = 64,
 	BTF_DTOR_KFUNC_MAX_CNT = 256,
 };
 
Signature · 1 line
2.34.1
author maintainer
#72 lore raw

RFC PATCH 23/32 bpf: Add single ownership BPF linked list API

In reply to Kumar Kartikeya Dwivedi

Add a linked list API for use in BPF programs, where it expects protection from the bpf_spin_lock in the same allocation as the bpf_list_head. Future patches will extend the same infrastructure to have different flavors with varying protection domains and visibility (e.g. percpu variant with local_t protection, usable in NMI progs).

The following functions are added to kick things off:

bpf_list_add
bpf_list_add_tail
bpf_list_del
bpf_list_pop_front
bpf_list_pop_back

The lock protecting the bpf_list_head needs to be taken for all operations.

Once a node has been added to the list, it's pointer changes to PTR_UNTRUSTED. However, it is only released once the lock protecting the list is unlocked. For such local kptrs with PTR_UNTRUSTED set but an active ref_obj_id, it is still permitted to read and write to them as long as the lock is held. However, they cannot be deleted using bpf_list_del after addition directly. bpf_list_del will only be permitted inside for_each helpers for lists which will be added in later patches. This is unlikely to be a problem as deleting right after addition in the same lock section is quite uncommon.

For now, bpf_list_del is hence unusable, unless a for_each helper is added, but it is still necessary to ensure it works correctly in presence of the rest of the API, and has thus been included with this change.

bpf_list_pop_front and bpf_list_pop_back delete the first or last item of the list respectively, and return pointer to the element at the list_node offset. The user can then use container_of style macro to get the actual entry type. The verifier however statically knows the actual type, so the safety properties are still preserved.

With these additions, programs can now manage their own linked lists and store their objects in them.

---
 include/linux/bpf_verifier.h                  |   5 +
 include/linux/btf.h                           |  11 +
 kernel/bpf/btf.c                              |  47 +-
 kernel/bpf/helpers.c                          |  55 ++
 kernel/bpf/verifier.c                         | 489 ++++++++++++++++--
 .../testing/selftests/bpf/bpf_experimental.h  |  35 ++
 6 files changed, 599 insertions(+), 43 deletions(-)
include/linux/bpf_verifier.h +5 -0
index 00c21ad6f61c..3cce796c4d76 100644
@@ -249,6 +249,11 @@ struct bpf_reference_state {
 	 * exiting a callback function.
 	 */
 	int callback_ref;
+	/* Mark the reference state to release the registers sharing the same id
+	 * on bpf_spin_unlock (for nodes that we will lose ownership to but are
+	 * safe to access inside the critical section).
+	 */
+	bool release_on_unlock;
 };
 
 /* state of the program:
include/linux/btf.h +11 -0
index 42c7f0283887..bd57a9cae12c 100644
@@ -437,6 +437,9 @@ int btf_local_type_has_bpf_list_node(const struct btf *btf,
 				     const struct btf_type *t, u32 *offsetp);
 int btf_local_type_has_bpf_spin_lock(const struct btf *btf,
 				     const struct btf_type *t, u32 *offsetp);
+int __btf_local_type_has_bpf_list_head(const struct btf *btf,
+				       const struct btf_type *t, u32 *offsetp,
+				       u32 *value_type_idp, u32 *list_node_offp);
 int btf_local_type_has_bpf_list_head(const struct btf *btf,
 				     const struct btf_type *t, u32 *offsetp);
 bool btf_local_type_has_special_fields(const struct btf *btf,
@@ -491,6 +494,14 @@ static inline int btf_local_type_has_bpf_spin_lock(const struct btf *btf,
 {
 	return -ENOENT;
 }
+static inline int __btf_local_type_has_bpf_list_head(const struct btf *btf,
+						     const struct btf_type *t,
+						     u32 *offsetp,
+						     u32 *value_type_idp,
+						     u32 *list_node_offp)
+{
+	return -ENOENT;
+}
 static inline int btf_local_type_has_bpf_list_head(const struct btf *btf,
 					           const struct btf_type *t,
 					           u32 *offsetp)
kernel/bpf/btf.c +43 -4

78 lines not on this page · raw message

kernel/bpf/helpers.c +55 -0

67 lines not on this page · raw message

kernel/bpf/verifier.c +450 -39

587 lines not on this page · raw message

tools/testing/selftests/bpf/bpf_experimental.h +35 -0
index 595e99d5cbc2..a8f7a5af8ee3 100644
@@ -67,4 +67,39 @@ void bpf_list_head_init(struct bpf_list_head *node) __ksym;
  */
 void bpf_kptr_free(void *kptr) __ksym;
 
+/* Description
+ *	Add a new entry to the head of a BPF linked list.
+ * Returns
+ *	Void.
+ */
+void bpf_list_add(struct bpf_list_node *node, struct bpf_list_head *head) __ksym;
+
+/* Description
+ *	Add a new entry to the tail of a BPF linked list.
+ * Returns
+ *	Void.
+ */
+void bpf_list_add_tail(struct bpf_list_node *node, struct bpf_list_head *head) __ksym;
+
+/* Description
+ *	Remove an entry already part of a BPF linked list.
+ * Returns
+ *	Void.
+ */
+void bpf_list_del(struct bpf_list_node *node) __ksym;
+
+/* Description
+ *	Remove the first entry of a BPF linked list.
+ * Returns
+ *	Pointer to bpf_list_node of deleted entry, or NULL if list is empty.
+ */
+struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head) __ksym;
+
+/* Description
+ *	Remove the last entry of a BPF linked list.
+ * Returns
+ *	Pointer to bpf_list_node of deleted entry, or NULL if list is empty.
+ */
+struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head) __ksym;
+
 #endif
Signature · 1 line
2.34.1
author maintainer
#73 lore raw

RFC PATCH 24/32 bpf: Permit NULL checking pointer with non-zero fixed offset

In reply to Kumar Kartikeya Dwivedi

Pointer increment on seeing PTR_MAYBE_NULL is already protected against, hence make an exception for local kptrs while still keeping the warning for other unintended cases that might creep in.

bpf_list_pop_{front,back} helpers return a local kptr with incremented offset pointing to bpf_list_node field. The user is supposed to then obtain the pointer to the entry using container_of after NULL checking it. The current restrictions trigger a warning when doing the NULL checking. Revisiting the reason, it is meant as an assertion which seems to actually work and catch the bad case! Good job verifier!

Hence, under no other circumstances can reg->off be non-zero for a register that has the PTR_MAYBE_NULL type flag set.

---
 kernel/bpf/verifier.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
kernel/bpf/verifier.c +9 -4
index dcbeb503c25c..5e0044796671 100644
@@ -11173,15 +11173,20 @@ static void mark_ptr_or_null_reg(struct bpf_func_state *state,
 {
 	if (type_may_be_null(reg->type) && reg->id == id &&
 	    !WARN_ON_ONCE(!reg->id)) {
-		if (WARN_ON_ONCE(reg->smin_value || reg->smax_value ||
-				 !tnum_equals_const(reg->var_off, 0) ||
-				 reg->off)) {
+		if (reg->smin_value || reg->smax_value || !tnum_equals_const(reg->var_off, 0) || reg->off) {
 			/* Old offset (both fixed and variable parts) should
 			 * have been known-zero, because we don't allow pointer
 			 * arithmetic on pointers that might be NULL. If we
 			 * see this happening, don't convert the register.
+			 *
+			 * But in some cases, some helpers that return local
+			 * kptrs advance offset for the returned pointer.
+			 * In those cases, it is fine to expect to see reg->off.
 			 */
-			return;
+			if (WARN_ON_ONCE(reg->type != (PTR_TO_BTF_ID | MEM_TYPE_LOCAL | PTR_MAYBE_NULL)))
+				return;
+			if (WARN_ON_ONCE(reg->smin_value || reg->smax_value || !tnum_equals_const(reg->var_off, 0)))
+				return;
 		}
 		if (is_null) {
 			reg->type = SCALAR_VALUE;
Signature · 1 line
2.34.1
author maintainer
#74 lore raw

RFC PATCH 25/32 bpf: Allow storing local kptrs in BPF maps

In reply to Kumar Kartikeya Dwivedi

Enable users to store local kptrs allocated from bpf_kptr_alloc into BPF maps. These are always referenced, hence they can only be stored in fields with kptr_ref type tag.

However, compared to normal kptrs, local kptrs point to types in program BTF. To tell the verifier which BTF to search the target of the pointer type in map value, introduce a new "local" type tag that can be applied with kptr_ref type tag.

When both are combined, this means we will not search the type in kernel BTF, and stash reference to the map BTF instead. When program uses this map, this map BTF must match program BTF for it to be able to interact with this local kptr in map value.

Later, more complex type matching (field by field) can be introduced to allow the case where program BTF differs from map BTF, but for now that is not done.

Note that for these local kptr fields, bpf_ktpr_xchg will set the type flag 'MEM_TYPE_LOCAL' so that the type information is preserved when moving value from program to map and vice versa.

Only fully constructed local kptrs can be moved into maps, i.e. escaping the program into the map is not possible in constructing or destructing phase. We may allow allocated but not initialized local kptrs in maps in the future. This would require that all fields are either in unknown or destructed state (i.e. state right after allocation or right after destruction of last field needing destruction). It would allow holding ownership of storage while the object on it has been destroyed. This would allow reusing same storage for multiple types that may fit in it.

Later commit will wire up freeing of these local kptr fields from the map value.

---
 include/linux/bpf.h   |  7 +++-
 include/linux/btf.h   |  2 +-
 kernel/bpf/btf.c      | 75 ++++++++++++++++++++++++++++---------------
 kernel/bpf/helpers.c  |  4 +--
 kernel/bpf/syscall.c  | 40 +++++++++++++++++++++--
 kernel/bpf/verifier.c | 63 ++++++++++++++++++++++++++----------
 6 files changed, 142 insertions(+), 49 deletions(-)
include/linux/bpf.h +6 -1
index 910aa891b97a..3353c47fefa9 100644
@@ -182,7 +182,9 @@ enum {
 enum bpf_off_type {
 	BPF_KPTR_UNREF,
 	BPF_KPTR_REF,
+	BPF_LOCAL_KPTR_REF,
 	BPF_LIST_HEAD,
+	BPF_OFF_TYPE_MAX,
 };
 
 struct bpf_map_value_off_desc {
@@ -546,6 +548,7 @@ enum bpf_arg_type {
 	ARG_PTR_TO_LONG,	/* pointer to long */
 	ARG_PTR_TO_SOCKET,	/* pointer to bpf_sock (fullsock) */
 	ARG_PTR_TO_BTF_ID,	/* pointer to in-kernel struct */
+	ARG_PTR_TO_DYN_BTF_ID,  /* pointer to in-kernel or local struct */
 	ARG_PTR_TO_ALLOC_MEM,	/* pointer to dynamically allocated memory */
 	ARG_CONST_ALLOC_SIZE_OR_ZERO,	/* number of allocated bytes requested */
 	ARG_PTR_TO_BTF_ID_SOCK_COMMON,	/* pointer to in-kernel sock_common or bpf-mirrored bpf_sock */
@@ -565,7 +568,7 @@ enum bpf_arg_type {
 	ARG_PTR_TO_SOCKET_OR_NULL	= PTR_MAYBE_NULL | ARG_PTR_TO_SOCKET,
 	ARG_PTR_TO_ALLOC_MEM_OR_NULL	= PTR_MAYBE_NULL | ARG_PTR_TO_ALLOC_MEM,
 	ARG_PTR_TO_STACK_OR_NULL	= PTR_MAYBE_NULL | ARG_PTR_TO_STACK,
-	ARG_PTR_TO_BTF_ID_OR_NULL	= PTR_MAYBE_NULL | ARG_PTR_TO_BTF_ID,
+	ARG_PTR_TO_DYN_BTF_ID_OR_NULL	= PTR_MAYBE_NULL | ARG_PTR_TO_DYN_BTF_ID,
 	/* pointer to memory does not need to be initialized, helper function must fill
 	 * all bytes or clear them in error case.
 	 */
@@ -591,6 +594,7 @@ enum bpf_return_type {
 	RET_PTR_TO_ALLOC_MEM,		/* returns a pointer to dynamically allocated memory */
 	RET_PTR_TO_MEM_OR_BTF_ID,	/* returns a pointer to a valid memory or a btf_id */
 	RET_PTR_TO_BTF_ID,		/* returns a pointer to a btf_id */
+	RET_PTR_TO_DYN_BTF_ID,		/* returns a pointer to a btf_id determined dynamically */
 	__BPF_RET_TYPE_MAX,
 
 	/* Extended ret_types. */
@@ -601,6 +605,7 @@ enum bpf_return_type {
 	RET_PTR_TO_ALLOC_MEM_OR_NULL	= PTR_MAYBE_NULL | MEM_ALLOC | RET_PTR_TO_ALLOC_MEM,
 	RET_PTR_TO_DYNPTR_MEM_OR_NULL	= PTR_MAYBE_NULL | RET_PTR_TO_ALLOC_MEM,
 	RET_PTR_TO_BTF_ID_OR_NULL	= PTR_MAYBE_NULL | RET_PTR_TO_BTF_ID,
+	RET_PTR_TO_DYN_BTF_ID_OR_NULL	= PTR_MAYBE_NULL | RET_PTR_TO_DYN_BTF_ID,
 
 	/* This must be the last entry. Its purpose is to ensure the enum is
 	 * wide enough to hold the higher bits reserved for bpf_type_flag.
include/linux/btf.h +1 -1
index bd57a9cae12c..b7d704f730c2 100644
@@ -157,7 +157,7 @@ bool btf_member_is_reg_int(const struct btf *btf, const struct btf_type *s,
 			   u32 expected_offset, u32 expected_size);
 int btf_find_spin_lock(const struct btf *btf, const struct btf_type *t);
 int btf_find_timer(const struct btf *btf, const struct btf_type *t);
-struct bpf_map_value_off *btf_parse_kptrs(const struct btf *btf,
+struct bpf_map_value_off *btf_parse_kptrs(struct btf *btf,
 					  const struct btf_type *t);
 struct bpf_map_value_off *btf_parse_list_heads(struct btf *btf,
 					       const struct btf_type *t);
kernel/bpf/btf.c +49 -26

139 lines not on this page · raw message

kernel/bpf/helpers.c +2 -2
index 9d5709441800..168460a03ec3 100644
@@ -1385,10 +1385,10 @@ BPF_CALL_2(bpf_kptr_xchg, void *, map_value, void *, ptr)
 static const struct bpf_func_proto bpf_kptr_xchg_proto = {
 	.func         = bpf_kptr_xchg,
 	.gpl_only     = false,
-	.ret_type     = RET_PTR_TO_BTF_ID_OR_NULL,
+	.ret_type     = RET_PTR_TO_DYN_BTF_ID_OR_NULL,
 	.ret_btf_id   = BPF_PTR_POISON,
 	.arg1_type    = ARG_PTR_TO_KPTR,
-	.arg2_type    = ARG_PTR_TO_BTF_ID_OR_NULL | OBJ_RELEASE,
+	.arg2_type    = ARG_PTR_TO_DYN_BTF_ID_OR_NULL | OBJ_RELEASE,
 	.arg2_btf_id  = BPF_PTR_POISON,
 };
 
kernel/bpf/syscall.c +38 -2
index e1749e0d2143..1af9a7cba08c 100644
@@ -590,6 +590,39 @@ bool bpf_map_equal_kptr_off_tab(const struct bpf_map *map_a, const struct bpf_ma
 				       map_value_has_kptrs(map_b));
 }
 
+static void bpf_free_local_kptr(const struct btf *btf, u32 btf_id, void *kptr)
+{
+	struct list_head *list, *olist;
+	u32 offset, list_node_off;
+	const struct btf_type *t;
+	void *entry;
+	int ret;
+
+	if (!kptr)
+		return;
+	/* We must free bpf_list_head in local kptr */
+	t = btf_type_by_id(btf, btf_id);
+	/* TODO: We should just populate this info once in struct btf, and then
+	 * do quick lookups into it. Instead of offset, table would be keyed by
+	 * btf_id.
+	 */
+	ret = __btf_local_type_has_bpf_list_head(btf, t, &offset, NULL, &list_node_off);
+	if (ret <= 0)
+		goto free_kptr;
+	/* List elements for bpf_list_head in local kptr cannot have
+	 * bpf_list_head again. Hence, just iterate and kfree them.
+	 */
+	olist = list = kptr + offset;
+	list = list->next;
+	while (list != olist) {
+		entry = list - list_node_off;
+		list = list->next;
+		kfree(entry);
+	}
+free_kptr:
+	kfree(kptr);
+}
+
 /* Caller must ensure map_value_has_kptrs is true. Note that this function can
  * be called on a map value while the map_value is visible to BPF programs, as
  * it ensures the correct synchronization, and we already enforce the same using
@@ -613,7 +646,10 @@ void bpf_map_free_kptrs(struct bpf_map *map, void *map_value)
 			continue;
 		}
 		old_ptr = xchg(btf_id_ptr, 0);
-		off_desc->kptr.dtor((void *)old_ptr);
+		if (off_desc->type == BPF_LOCAL_KPTR_REF)
+			bpf_free_local_kptr(off_desc->kptr.btf, off_desc->kptr.btf_id, (void *)old_ptr);
+		else
+			off_desc->kptr.dtor((void *)old_ptr);
 	}
 }
 
@@ -1102,7 +1138,7 @@ static int map_check_btf(struct bpf_map *map, const struct btf *btf,
 			return -EOPNOTSUPP;
 	}
 
-	map->kptr_off_tab = btf_parse_kptrs(btf, value_type);
+	map->kptr_off_tab = btf_parse_kptrs((struct btf *)btf, value_type);
 	if (map_value_has_kptrs(map)) {
 		if (!bpf_capable()) {
 			ret = -EPERM;
kernel/bpf/verifier.c +46 -17

164 lines not on this page · raw message

Signature · 1 line
2.34.1
author maintainer
#75 lore raw

RFC PATCH 26/32 bpf: Wire up freeing of bpf_list_heads in maps

In reply to Kumar Kartikeya Dwivedi

Until now, bpf_list_heads in maps were not being freed. Wire up code needed to release them when found in map value.
This will also handle freeling local kptr with bpf_list_head in them.

Note that bpf_list_head in map value requires appropriate locking during the draining operation.

---
 include/linux/bpf.h            |  3 +++
 kernel/bpf/arraymap.c          |  8 +++++++
 kernel/bpf/bpf_local_storage.c | 11 ++++++----
 kernel/bpf/hashtab.c           | 22 +++++++++++++++++++
 kernel/bpf/helpers.c           | 14 ++++++++++++
 kernel/bpf/syscall.c           | 39 ++++++++++++++++++++++++++++++++++
 6 files changed, 93 insertions(+), 4 deletions(-)
include/linux/bpf.h +3 -0
index 3353c47fefa9..ad18408ba442 100644
@@ -381,6 +381,8 @@ static inline void zero_map_value(struct bpf_map *map, void *dst)
 
 void copy_map_value_locked(struct bpf_map *map, void *dst, void *src,
 			   bool lock_src);
+void bpf_map_value_lock(struct bpf_map *map, void *map_value);
+void bpf_map_value_unlock(struct bpf_map *map, void *map_value);
 void bpf_timer_cancel_and_free(void *timer);
 int bpf_obj_name_cpy(char *dst, const char *src, unsigned int size);
 
@@ -1736,6 +1738,7 @@ struct bpf_map_value_off_desc *bpf_map_list_head_off_contains(struct bpf_map *ma
 void bpf_map_free_list_head_off_tab(struct bpf_map *map);
 struct bpf_map_value_off *bpf_map_copy_list_head_off_tab(const struct bpf_map *map);
 bool bpf_map_equal_list_head_off_tab(const struct bpf_map *map_a, const struct bpf_map *map_b);
+void bpf_map_free_list_heads(struct bpf_map *map, void *map_value);
 
 struct bpf_map *bpf_map_get(u32 ufd);
 struct bpf_map *bpf_map_get_with_uref(u32 ufd);
kernel/bpf/arraymap.c +8 -0
index c7263ee3a35f..5412fa66d659 100644
@@ -312,6 +312,8 @@ static void check_and_free_fields(struct bpf_array *arr, void *val)
 		bpf_timer_cancel_and_free(val + arr->map.timer_off);
 	if (map_value_has_kptrs(&arr->map))
 		bpf_map_free_kptrs(&arr->map, val);
+	if (map_value_has_list_heads(&arr->map))
+		bpf_map_free_list_heads(&arr->map, val);
 }
 
 /* Called from syscall or from eBPF program */
@@ -443,6 +445,12 @@ static void array_map_free(struct bpf_map *map)
 		bpf_map_free_kptr_off_tab(map);
 	}
 
+	if (map_value_has_list_heads(map)) {
+		for (i = 0; i < array->map.max_entries; i++)
+			bpf_map_free_list_heads(map, array_map_elem_ptr(array, i));
+		bpf_map_free_list_head_off_tab(map);
+	}
+
 	if (array->map.map_type == BPF_MAP_TYPE_PERCPU_ARRAY)
 		bpf_array_free_percpu(array);
 
kernel/bpf/bpf_local_storage.c +7 -4
index b5ccd76026b6..e89c6aa5d782 100644
@@ -107,6 +107,8 @@ static void check_and_free_fields(struct bpf_local_storage_elem *selem)
 {
 	if (map_value_has_kptrs(selem->map))
 		bpf_map_free_kptrs(selem->map, SDATA(selem));
+	if (map_value_has_list_heads(selem->map))
+		bpf_map_free_list_heads(selem->map, SDATA(selem));
 }
 
 static void bpf_selem_free_rcu(struct rcu_head *rcu)
@@ -608,13 +610,14 @@ void bpf_local_storage_map_free(struct bpf_local_storage_map *smap,
 	 */
 	synchronize_rcu();
 
-	/* When local storage map has kptrs, the call_rcu callback accesses
-	 * kptr_off_tab, hence we need the bpf_selem_free_rcu callbacks to
-	 * finish before we free it.
+	/* When local storage map has kptrs or bpf_list_heads, the call_rcu
+	 * callback accesses kptr_off_tab or list_head_off_tab, hence we need
+	 * the bpf_selem_free_rcu callbacks to finish before we free it.
 	 */
-	if (map_value_has_kptrs(&smap->map)) {
+	if (map_value_has_kptrs(&smap->map) || map_value_has_list_heads(&smap->map)) {
 		rcu_barrier();
 		bpf_map_free_kptr_off_tab(&smap->map);
+		bpf_map_free_list_head_off_tab(&smap->map);
 	}
 	bpf_map_free_list_head_off_tab(&smap->map);
 	kvfree(smap->buckets);
kernel/bpf/hashtab.c +22 -0
index 270e0ecf4ba3..bd1637fa7e3b 100644
@@ -297,6 +297,25 @@ static void htab_free_prealloced_kptrs(struct bpf_htab *htab)
 	}
 }
 
+static void htab_free_prealloced_list_heads(struct bpf_htab *htab)
+{
+	u32 num_entries = htab->map.max_entries;
+	int i;
+
+	if (!map_value_has_list_heads(&htab->map))
+		return;
+	if (htab_has_extra_elems(htab))
+		num_entries += num_possible_cpus();
+
+	for (i = 0; i < num_entries; i++) {
+		struct htab_elem *elem;
+
+		elem = get_htab_elem(htab, i);
+		bpf_map_free_list_heads(&htab->map, elem->key + round_up(htab->map.key_size, 8));
+		cond_resched();
+	}
+}
+
 static void htab_free_elems(struct bpf_htab *htab)
 {
 	int i;
@@ -782,6 +801,8 @@ static void check_and_free_fields(struct bpf_htab *htab,
 			bpf_map_free_kptrs(&htab->map, map_value);
 		}
 	}
+	if (map_value_has_list_heads(&htab->map))
+		bpf_map_free_list_heads(&htab->map, map_value);
 }
 
 /* It is called from the bpf_lru_list when the LRU needs to delete
@@ -1514,6 +1535,7 @@ static void htab_map_free(struct bpf_map *map)
 	if (!htab_is_prealloc(htab)) {
 		delete_all_elements(htab);
 	} else {
+		htab_free_prealloced_list_heads(htab);
 		htab_free_prealloced_kptrs(htab);
 		prealloc_destroy(htab);
 	}
kernel/bpf/helpers.c +14 -0
index 168460a03ec3..832dd57ae608 100644
@@ -377,6 +377,20 @@ void copy_map_value_locked(struct bpf_map *map, void *dst, void *src,
 	preempt_enable();
 }
 
+void bpf_map_value_lock(struct bpf_map *map, void *map_value)
+{
+	WARN_ON_ONCE(map->spin_lock_off < 0);
+	preempt_disable();
+	__bpf_spin_lock_irqsave(map_value + map->spin_lock_off);
+}
+
+void bpf_map_value_unlock(struct bpf_map *map, void *map_value)
+{
+	WARN_ON_ONCE(map->spin_lock_off < 0);
+	__bpf_spin_unlock_irqrestore(map_value + map->spin_lock_off);
+	preempt_enable();
+}
+
 BPF_CALL_0(bpf_jiffies64)
 {
 	return get_jiffies_64();
kernel/bpf/syscall.c +39 -0
index 1af9a7cba08c..f1e244b03382 100644
@@ -600,6 +600,13 @@ static void bpf_free_local_kptr(const struct btf *btf, u32 btf_id, void *kptr)
 
 	if (!kptr)
 		return;
+	/* There is no requirement to lock the bpf_spin_lock protecting
+	 * bpf_list_head in local kptr, as these are single ownership,
+	 * so if we have access to the kptr through xchg, we own it.
+	 *
+	 * If iterating elements of bpf_list_head in map value we are
+	 * already holding the lock for it.
+	 */
 	/* We must free bpf_list_head in local kptr */
 	t = btf_type_by_id(btf, btf_id);
 	/* TODO: We should just populate this info once in struct btf, and then
@@ -697,6 +704,38 @@ bool bpf_map_equal_list_head_off_tab(const struct bpf_map *map_a, const struct b
 				       map_value_has_list_heads(map_b));
 }
 
+void bpf_map_free_list_heads(struct bpf_map *map, void *map_value)
+{
+	struct bpf_map_value_off *tab = map->list_head_off_tab;
+	int i;
+
+	/* TODO: Should we error when bpf_list_head is alone in map value,
+	 * during BTF parsing, instead of ignoring it?
+	 */
+	if (map->spin_lock_off < 0)
+		return;
+
+	bpf_map_value_lock(map, map_value);
+	for (i = 0; i < tab->nr_off; i++) {
+		struct bpf_map_value_off_desc *off_desc = &tab->off[i];
+		struct list_head *list, *olist;
+		void *entry;
+
+		olist = list = map_value + off_desc->offset;
+		list = list->next;
+		if (!list)
+			goto init;
+		while (list != olist) {
+			entry = list - off_desc->list_head.list_node_off;
+			list = list->next;
+			bpf_free_local_kptr(off_desc->list_head.btf, off_desc->list_head.value_type_id, entry);
+		}
+	init:
+		INIT_LIST_HEAD(olist);
+	}
+	bpf_map_value_unlock(map, map_value);
+}
+
 /* called from workqueue */
 static void bpf_map_free_deferred(struct work_struct *work)
 {
Signature · 1 line
2.34.1
author maintainer
#76 lore raw

RFC PATCH 27/32 bpf: Add destructor for bpf_list_head in local kptr

In reply to Kumar Kartikeya Dwivedi

Refactor bpf_free_local_kptr's bpf_list_head handling logic to introduce the destructor for bpf_list_head inside local kptrs. The first argument is pointer to the bpf_list_head inside local kptr, while the second argument is the node offset of the value type of the list head.

It is possible to only take one argument and pass 'hidden' argument from verifier side, but unlike helpers which always take 5 arguments at C ABI level, kfuncs are more strongly checked from their prototype in kernel BTF. So hidden arguments are more work to support.

Secondly, it would again require rewriting arguments and bpf_patch_insn_data, which is expensive and slow. Hence, just force user to pass the offset, but check that it is the right one from verifier side, which turns out to be much more easier.

Ofcourse, this is a little bit inconvenient, we can explore improving it later though.

---
 include/linux/bpf.h                           |  1 +
 kernel/bpf/helpers.c                          |  6 +++
 kernel/bpf/syscall.c                          | 39 ++++++++++++-------
 kernel/bpf/verifier.c                         | 26 ++++++++++++-
 .../testing/selftests/bpf/bpf_experimental.h  | 11 ++++++
 5 files changed, 69 insertions(+), 14 deletions(-)
include/linux/bpf.h +1 -0
index ad18408ba442..9279e453528c 100644
@@ -1733,6 +1733,7 @@ void bpf_map_free_kptr_off_tab(struct bpf_map *map);
 struct bpf_map_value_off *bpf_map_copy_kptr_off_tab(const struct bpf_map *map);
 bool bpf_map_equal_kptr_off_tab(const struct bpf_map *map_a, const struct bpf_map *map_b);
 void bpf_map_free_kptrs(struct bpf_map *map, void *map_value);
+void bpf_free_local_kptr_list_head(struct list_head *list, u32 list_node_off);
 
 struct bpf_map_value_off_desc *bpf_map_list_head_off_contains(struct bpf_map *map, u32 offset);
 void bpf_map_free_list_head_off_tab(struct bpf_map *map);
kernel/bpf/helpers.c +6 -0
index 832dd57ae608..030c35bf030d 100644
@@ -1800,6 +1800,11 @@ struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head)
 	return (struct bpf_list_node *)node;
 }
 
+void bpf_list_head_fini(struct bpf_list_head *head__dlkptr, u64 node_off__k)
+{
+	bpf_free_local_kptr_list_head((struct list_head *)head__dlkptr, node_off__k);
+}
+
 __diag_pop();
 
 BTF_SET8_START(tracing_btf_ids)
@@ -1816,6 +1821,7 @@ BTF_ID_FLAGS(func, bpf_list_add_tail)
 BTF_ID_FLAGS(func, bpf_list_del)
 BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL | __KF_RET_DYN_BTF)
 BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL | __KF_RET_DYN_BTF)
+BTF_ID_FLAGS(func, bpf_list_head_fini)
 BTF_SET8_END(tracing_btf_ids)
 
 static const struct btf_kfunc_id_set tracing_kfunc_set = {
kernel/bpf/syscall.c +26 -13
index f1e244b03382..feaf4351345b 100644
@@ -590,12 +590,31 @@ bool bpf_map_equal_kptr_off_tab(const struct bpf_map *map_a, const struct bpf_ma
 				       map_value_has_kptrs(map_b));
 }
 
+void bpf_free_local_kptr_list_head(struct list_head *list, u32 list_node_off)
+{
+	struct list_head *olist;
+	void *entry;
+
+	/* List elements for bpf_list_head in local kptr cannot have
+	 * bpf_list_head again. Hence, just iterate and kfree them.
+	 */
+	olist = list;
+	list = list->next;
+	if (!list)
+		goto init;
+	while (list != olist) {
+		entry = list - list_node_off;
+		list = list->next;
+		kfree(entry);
+	}
+init:
+	INIT_LIST_HEAD(olist);
+}
+
 static void bpf_free_local_kptr(const struct btf *btf, u32 btf_id, void *kptr)
 {
-	struct list_head *list, *olist;
-	u32 offset, list_node_off;
+	u32 list_head_off, list_node_off;
 	const struct btf_type *t;
-	void *entry;
 	int ret;
 
 	if (!kptr)
@@ -613,19 +632,13 @@ static void bpf_free_local_kptr(const struct btf *btf, u32 btf_id, void *kptr)
 	 * do quick lookups into it. Instead of offset, table would be keyed by
 	 * btf_id.
 	 */
-	ret = __btf_local_type_has_bpf_list_head(btf, t, &offset, NULL, &list_node_off);
+	ret = __btf_local_type_has_bpf_list_head(btf, t, &list_head_off, NULL, &list_node_off);
 	if (ret <= 0)
 		goto free_kptr;
 	/* List elements for bpf_list_head in local kptr cannot have
-	 * bpf_list_head again. Hence, just iterate and kfree them.
-	 */
-	olist = list = kptr + offset;
-	list = list->next;
-	while (list != olist) {
-		entry = list - list_node_off;
-		list = list->next;
-		kfree(entry);
-	}
+         * bpf_list_head again. Hence, just iterate and kfree them.
+         */
+	bpf_free_local_kptr_list_head(kptr + list_head_off, list_node_off);
 free_kptr:
 	kfree(kptr);
 }
kernel/bpf/verifier.c +25 -1
index d2c4ffc80f4d..b795fe9a88da 100644
@@ -7911,6 +7911,7 @@ BTF_ID(func, bpf_list_add_tail)
 BTF_ID(func, bpf_list_del)
 BTF_ID(func, bpf_list_pop_front)
 BTF_ID(func, bpf_list_pop_back)
+BTF_ID(func, bpf_list_head_fini)
 BTF_ID(struct, btf) /* empty entry */
 
 enum bpf_special_kfuncs {
@@ -7924,6 +7925,7 @@ enum bpf_special_kfuncs {
 	KF_SPECIAL_bpf_list_del,
 	KF_SPECIAL_bpf_list_pop_front,
 	KF_SPECIAL_bpf_list_pop_back,
+	KF_SPECIAL_bpf_list_head_fini,
 	KF_SPECIAL_bpf_empty,
 	KF_SPECIAL_MAX = KF_SPECIAL_bpf_empty,
 };
@@ -8156,7 +8158,7 @@ static int find_local_type_fields(const struct btf *btf, u32 btf_id, struct loca
 
 	FILL_LOCAL_TYPE_FIELD(bpf_list_node, bpf_list_node_init, bpf_empty, false);
 	FILL_LOCAL_TYPE_FIELD(bpf_spin_lock, bpf_spin_lock_init, bpf_empty, false);
-	FILL_LOCAL_TYPE_FIELD(bpf_list_head, bpf_list_head_init, bpf_empty, true);
+	FILL_LOCAL_TYPE_FIELD(bpf_list_head, bpf_list_head_init, bpf_list_head_fini, true);
 
 #undef FILL_LOCAL_TYPE_FIELD
 
@@ -8391,6 +8393,19 @@ process_kf_arg_destructing_local_kptr(struct bpf_verifier_env *env,
 			if (mark_dtor)
 				ireg->type |= OBJ_DESTRUCTING;
 		}));
+
+		/* Stash the list_node offset in value type of the
+		 * bpf_list_head, so that offset of node in next argument can be
+		 * checked for bpf_list_head_fini.
+		 */
+		if (fields[i].type == FIELD_bpf_list_head) {
+			ret = __btf_local_type_has_bpf_list_head(reg->btf, btf_type_by_id(reg->btf, reg->btf_id),
+								 NULL, NULL, &meta->list_node.off);
+			if (ret <= 0) {
+				verbose(env, "verifier internal error: bpf_list_head not found\n");
+				return -EFAULT;
+			}
+		}
 		return 0;
 	}
 	verbose(env, "no destructible field at offset: %d\n", reg->off);
@@ -8875,6 +8890,15 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_arg_m
 		return -EINVAL;
 	}
 
+	/* Special semantic checks for some functions */
+	if (is_kfunc_special(meta->btf, meta->func_id, bpf_list_head_fini)) {
+		if (!meta->arg_constant.found || meta->list_node.off != meta->arg_constant.value) {
+			verbose(env, "arg#1 to bpf_list_head_fini must be constant %d\n",
+				meta->list_node.off);
+			return -EINVAL;
+		}
+	}
+
 	return 0;
 }
 
tools/testing/selftests/bpf/bpf_experimental.h +11 -0
index a8f7a5af8ee3..60fe48df4f68 100644
@@ -102,4 +102,15 @@ struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head) __ksym;
  */
 struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head) __ksym;
 
+/* Description
+ *	Destruct bpf_list_head field in a local kptr. This kfunc has destructor
+ *	semantics, and marks local kptr as destructing if it isn't already.
+ *
+ *	Note that value_node_offset is the offset of bpf_list_node inside the
+ *	value type of local kptr's bpf_list_head. It must be a known constant.
+ * Returns
+ *	Void.
+ */
+void bpf_list_head_fini(struct bpf_list_head *node, u64 value_node_offset) __ksym;
+
 #endif
Signature · 1 line
2.34.1
author maintainer
#77 lore raw

RFC PATCH 28/32 bpf: Remove duplicate PTR_TO_BTF_ID RO check

In reply to Kumar Kartikeya Dwivedi

From: Daniel Xu <[email protected]>

Since commit 27ae7997a661 ("bpf: Introduce BPF_PROG_TYPE_STRUCT_OPS") there has existed bpf_verifier_ops:btf_struct_access. When btf_struct_access is _unset_ for a prog type, the verifier runs the default implementation, which is to enforce read only:

        if (env->ops->btf_struct_access) {
                [...]
        } else {
                if (atype != BPF_READ) {
                        verbose(env, "only read is supported\n");
                        return -EACCES;
                }

                [...]
        }

When btf_struct_access is _set_, the expectation is that btf_struct_access has full control over accesses, including if writes are allowed.

Rather than carve out an exception for each prog type that may write to BTF ptrs, delete the redundant check and give full control to btf_struct_access.

[
Kartikeya: We also require to remove this check, as we are enabling writes to local kptrs, which are a special type of PTR_TO_BTF_ID pointing to btf_id in program BTF.

Note that probe_mem conversions, we only need then when such local kptr is marked with PTR_UNTRUSTED.

There are two cases when it is so. One is when node is marked for expiry on the end of critical section, it is marked as PTR_UNTRUSTED but with a non-zero ref_obj_id. This means that writing is still permitted to it, as is reading, and technically PROBE_MEM load conversion is not needed. It is just used to prevent passing this local kptr elsewhere.

The second case is loading reference local kptr from a map. In this case the pointer may well be invalid by the time we access it. Hence, writing to is disallowed but reading isn't. Here, PROBE_MEM conversion is crucial.

We could discern between ref_obj_id set vs unset case, but for it's left out of the current series.
]

Cc Martin KaFai Lau <[email protected]>
[ Kartikeya: Expanded commit message ]
---
 kernel/bpf/verifier.c | 3 ---
 1 file changed, 3 deletions(-)
kernel/bpf/verifier.c +0 -3
index b795fe9a88da..2897f780e8be 100644
@@ -14889,9 +14889,6 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)
 				insn->code = BPF_LDX | BPF_PROBE_MEM |
 					BPF_SIZE((insn)->code);
 				env->prog->aux->num_exentries++;
-			} else if (resolve_prog_type(env->prog) != BPF_PROG_TYPE_STRUCT_OPS) {
-				verbose(env, "Writes through BTF pointers are not allowed\n");
-				return -EINVAL;
 			}
 			continue;
 		default:
Signature · 1 line
2.34.1
author maintainer
#78 lore raw

RFC PATCH 29/32 libbpf: Add support for private BSS map section

In reply to Kumar Kartikeya Dwivedi

From: Dave Marchevsky <[email protected]>

Currently libbpf does not allow declaration of a struct bpf_spin_lock in global scope. Attempting to do so results in "failed to re-mmap" error, as .bss arraymap containing spinlock is not allowed to be mmap'd.

This patch adds support for a .bss.private section. The maps contained in this section will not be mmaped into userspace by libbpf, nor will they be exposed via bpftool-generated skeleton.

Intent here is to allow more natural programming pattern for global-scope spinlocks which will be used by rbtree locking mechanism in further patches in this series.

Notes:

* Initially I called the section .bss.no_mmap, but the broader 'private' term better indicates that skeleton shouldn't expose these maps at all, IMO.

* bpftool/gen.c's is_internal_mmapable_map function checks whether the map flags have BPF_F_MMAPABLE, so no bpftool changes were necessary to remove .bss.private maps from skeleton

---
 tools/lib/bpf/libbpf.c | 65 ++++++++++++++++++++++++++++--------------
 1 file changed, 44 insertions(+), 21 deletions(-)
tools/lib/bpf/libbpf.c +44 -21
index 3ad139285fad..17989dd49179 100644
@@ -464,6 +464,7 @@ struct bpf_struct_ops {
 #define KCONFIG_SEC ".kconfig"
 #define KSYMS_SEC ".ksyms"
 #define STRUCT_OPS_SEC ".struct_ops"
+#define BSS_SEC_PRIVATE ".bss.private"
 
 enum libbpf_map_type {
 	LIBBPF_MAP_UNSPEC,
@@ -577,6 +578,7 @@ enum sec_type {
 	SEC_BSS,
 	SEC_DATA,
 	SEC_RODATA,
+	SEC_BSS_PRIVATE,
 };
 
 struct elf_sec_desc {
@@ -1581,7 +1583,8 @@ bpf_map_find_btf_info(struct bpf_object *obj, struct bpf_map *map);
 
 static int
 bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
-			      const char *real_name, int sec_idx, void *data, size_t data_sz)
+			      const char *real_name, int sec_idx, void *data,
+			      size_t data_sz, bool do_mmap)
 {
 	struct bpf_map_def *def;
 	struct bpf_map *map;
@@ -1609,27 +1612,31 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
 	def->max_entries = 1;
 	def->map_flags = type == LIBBPF_MAP_RODATA || type == LIBBPF_MAP_KCONFIG
 			 ? BPF_F_RDONLY_PROG : 0;
-	def->map_flags |= BPF_F_MMAPABLE;
+	if (do_mmap)
+		def->map_flags |= BPF_F_MMAPABLE;
 
 	pr_debug("map '%s' (global data): at sec_idx %d, offset %zu, flags %x.\n",
 		 map->name, map->sec_idx, map->sec_offset, def->map_flags);
 
-	map->mmaped = mmap(NULL, bpf_map_mmap_sz(map), PROT_READ | PROT_WRITE,
-			   MAP_SHARED | MAP_ANONYMOUS, -1, 0);
-	if (map->mmaped == MAP_FAILED) {
-		err = -errno;
-		map->mmaped = NULL;
-		pr_warn("failed to alloc map '%s' content buffer: %d\n",
-			map->name, err);
-		zfree(&map->real_name);
-		zfree(&map->name);
-		return err;
+	map->mmaped = NULL;
+	if (do_mmap) {
+		map->mmaped = mmap(NULL, bpf_map_mmap_sz(map), PROT_READ | PROT_WRITE,
+				   MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+		if (map->mmaped == MAP_FAILED) {
+			err = -errno;
+			map->mmaped = NULL;
+			pr_warn("failed to alloc map '%s' content buffer: %d\n",
+				map->name, err);
+			zfree(&map->real_name);
+			zfree(&map->name);
+			return err;
+		}
 	}
 
 	/* failures are fine because of maps like .rodata.str1.1 */
 	(void) bpf_map_find_btf_info(obj, map);
 
-	if (data)
+	if (do_mmap && data)
 		memcpy(map->mmaped, data, data_sz);
 
 	pr_debug("map %td is \"%s\"\n", map - obj->maps, map->name);
@@ -1641,12 +1648,14 @@ static int bpf_object__init_global_data_maps(struct bpf_object *obj)
 	struct elf_sec_desc *sec_desc;
 	const char *sec_name;
 	int err = 0, sec_idx;
+	bool do_mmap;
 
 	/*
 	 * Populate obj->maps with libbpf internal maps.
 	 */
 	for (sec_idx = 1; sec_idx < obj->efile.sec_cnt; sec_idx++) {
 		sec_desc = &obj->efile.secs[sec_idx];
+		do_mmap = true;
 
 		/* Skip recognized sections with size 0. */
 		if (!sec_desc->data || sec_desc->data->d_size == 0)
@@ -1658,7 +1667,8 @@ static int bpf_object__init_global_data_maps(struct bpf_object *obj)
 			err = bpf_object__init_internal_map(obj, LIBBPF_MAP_DATA,
 							    sec_name, sec_idx,
 							    sec_desc->data->d_buf,
-							    sec_desc->data->d_size);
+							    sec_desc->data->d_size,
+							    do_mmap);
 			break;
 		case SEC_RODATA:
 			obj->has_rodata = true;
@@ -1666,14 +1676,18 @@ static int bpf_object__init_global_data_maps(struct bpf_object *obj)
 			err = bpf_object__init_internal_map(obj, LIBBPF_MAP_RODATA,
 							    sec_name, sec_idx,
 							    sec_desc->data->d_buf,
-							    sec_desc->data->d_size);
+							    sec_desc->data->d_size,
+							    do_mmap);
 			break;
+		case SEC_BSS_PRIVATE:
+			do_mmap = false;
 		case SEC_BSS:
 			sec_name = elf_sec_name(obj, elf_sec_by_idx(obj, sec_idx));
 			err = bpf_object__init_internal_map(obj, LIBBPF_MAP_BSS,
 							    sec_name, sec_idx,
 							    NULL,
-							    sec_desc->data->d_size);
+							    sec_desc->data->d_size,
+							    do_mmap);
 			break;
 		default:
 			/* skip */
@@ -1987,7 +2001,7 @@ static int bpf_object__init_kconfig_map(struct bpf_object *obj)
 	map_sz = last_ext->kcfg.data_off + last_ext->kcfg.sz;
 	err = bpf_object__init_internal_map(obj, LIBBPF_MAP_KCONFIG,
 					    ".kconfig", obj->efile.symbols_shndx,
-					    NULL, map_sz);
+					    NULL, map_sz, true);
 	if (err)
 		return err;
 
@@ -3431,6 +3445,10 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
 			sec_desc->sec_type = SEC_BSS;
 			sec_desc->shdr = sh;
 			sec_desc->data = data;
+		} else if (sh->sh_type == SHT_NOBITS && strcmp(name, BSS_SEC_PRIVATE) == 0) {
+			sec_desc->sec_type = SEC_BSS_PRIVATE;
+			sec_desc->shdr = sh;
+			sec_desc->data = data;
 		} else {
 			pr_info("elf: skipping section(%d) %s (size %zu)\n", idx, name,
 				(size_t)sh->sh_size);
@@ -3893,6 +3911,7 @@ static bool bpf_object__shndx_is_data(const struct bpf_object *obj,
 	case SEC_BSS:
 	case SEC_DATA:
 	case SEC_RODATA:
+	case SEC_BSS_PRIVATE:
 		return true;
 	default:
 		return false;
@@ -3912,6 +3931,7 @@ bpf_object__section_to_libbpf_map_type(const struct bpf_object *obj, int shndx)
 		return LIBBPF_MAP_KCONFIG;
 
 	switch (obj->efile.secs[shndx].sec_type) {
+	case SEC_BSS_PRIVATE:
 	case SEC_BSS:
 		return LIBBPF_MAP_BSS;
 	case SEC_DATA:
@@ -4901,16 +4921,19 @@ bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map)
 {
 	enum libbpf_map_type map_type = map->libbpf_type;
 	char *cp, errmsg[STRERR_BUFSIZE];
-	int err, zero = 0;
+	int err = 0, zero = 0;
 
 	if (obj->gen_loader) {
-		bpf_gen__map_update_elem(obj->gen_loader, map - obj->maps,
-					 map->mmaped, map->def.value_size);
+		if (map->mmaped)
+			bpf_gen__map_update_elem(obj->gen_loader, map - obj->maps,
+						 map->mmaped, map->def.value_size);
 		if (map_type == LIBBPF_MAP_RODATA || map_type == LIBBPF_MAP_KCONFIG)
 			bpf_gen__map_freeze(obj->gen_loader, map - obj->maps);
 		return 0;
 	}
-	err = bpf_map_update_elem(map->fd, &zero, map->mmaped, 0);
+
+	if (map->mmaped)
+		err = bpf_map_update_elem(map->fd, &zero, map->mmaped, 0);
 	if (err) {
 		err = -errno;
 		cp = libbpf_strerror_r(err, errmsg, sizeof(errmsg));
Signature · 1 line
2.34.1
author maintainer
#79 lore raw

RFC PATCH 30/32 selftests/bpf: Add BTF tag macros for local kptrs, BPF linked lists

In reply to Kumar Kartikeya Dwivedi

Since this is an experimental API, all of these tags are not exposed in libbpf, but hidden in the bpf_experimental.h header in the selftests directory. Once enough field experience has been gained with this API, it can graduate to stable and can be baked into UAPI.s

---
 tools/testing/selftests/bpf/bpf_experimental.h | 4 ++++
 1 file changed, 4 insertions(+)
tools/testing/selftests/bpf/bpf_experimental.h +4 -0
index 60fe48df4f68..21f12c510db4 100644
@@ -3,6 +3,10 @@
 #include <bpf/bpf_tracing.h>
 #include <bpf/bpf_helpers.h>
 
+#define __contains(kind, name, node) __attribute__((btf_decl_tag("contains:" #kind ":" #name ":" #node)))
+#define __kernel __attribute__((btf_decl_tag("kernel")))
+#define __local __attribute__((btf_type_tag("local")))
+
 #else
 
 struct bpf_list_head {
Signature · 1 line
2.34.1
author maintainer
#80 lore raw

RFC PATCH 31/32 selftests/bpf: Add BPF linked list API tests

In reply to Kumar Kartikeya Dwivedi

Include various tests covering the success and failure cases. Also, run the success cases at runtime to verify correctness of linked list manipulation routines, in addition to ensuring successful verification.

---
 kernel/bpf/helpers.c                          |   5 +-
 .../selftests/bpf/prog_tests/linked_list.c    |  88 +++++
 .../testing/selftests/bpf/progs/linked_list.c | 347 ++++++++++++++++++
 3 files changed, 439 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/linked_list.c
 create mode 100644 tools/testing/selftests/bpf/progs/linked_list.c
kernel/bpf/helpers.c +4 -1
index 030c35bf030d..928466f83cca 100644
@@ -1831,7 +1831,10 @@ static const struct btf_kfunc_id_set tracing_kfunc_set = {
 
 static int __init kfunc_init(void)
 {
-	return register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &tracing_kfunc_set);
+	int ret;
+
+	ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &tracing_kfunc_set);
+	return ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &tracing_kfunc_set);
 }
 
 late_initcall(kfunc_init);
tools/testing/selftests/bpf/prog_tests/linked_list.c new file +88 -0

88 lines not on this page · raw message

tools/testing/selftests/bpf/progs/linked_list.c new file +347 -0

347 lines not on this page · raw message

Signature · 1 line
2.34.1
author maintainer
#81 lore raw

RFC PATCH 32/32 selftests/bpf: Add referenced local kptr tests

In reply to Kumar Kartikeya Dwivedi

Add some cases where success and failure at verification time is tested.

---
 .../selftests/bpf/prog_tests/map_kptr.c       |  2 +-
 tools/testing/selftests/bpf/progs/map_kptr.c  | 38 +++++++++++++++++++
 2 files changed, 39 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/prog_tests/map_kptr.c +1 -1
index fdcea7a61491..f2608a3e4e0d 100644
@@ -91,7 +91,7 @@ static void test_map_kptr_success(bool test_run)
 	);
 	struct map_kptr *skel;
 	int key = 0, ret;
-	char buf[16];
+	char buf[24];
 
 	skel = map_kptr__open_and_load();
 	if (!ASSERT_OK_PTR(skel, "map_kptr__open_and_load"))
tools/testing/selftests/bpf/progs/map_kptr.c +38 -0
index eb8217803493..30c981be008b 100644
@@ -2,10 +2,17 @@
 #include <vmlinux.h>
 #include <bpf/bpf_tracing.h>
 #include <bpf/bpf_helpers.h>
+#include <bpf/bpf_core_read.h>
+#include "bpf_experimental.h"
+
+struct foo {
+	int data;
+};
 
 struct map_value {
 	struct prog_test_ref_kfunc __kptr *unref_ptr;
 	struct prog_test_ref_kfunc __kptr_ref *ref_ptr;
+	struct foo __kptr_ref __local *lref_ptr;
 };
 
 struct array_map {
@@ -130,11 +137,42 @@ static void test_kptr_get(struct map_value *v)
 	bpf_kfunc_call_test_release(p);
 }
 
+static void test_local_kptr_ref(struct map_value *v)
+{
+	struct foo *p;
+
+	p = v->lref_ptr;
+	if (!p)
+		return;
+	if (p->data > 100)
+		return;
+	/* store NULL */
+	p = bpf_kptr_xchg(&v->lref_ptr, NULL);
+	if (!p)
+		return;
+	if (p->data > 100) {
+		p->data = 0;
+		bpf_kptr_free(p);
+		return;
+	}
+	bpf_kptr_free(p);
+
+	p = bpf_kptr_alloc(bpf_core_type_id_local(struct foo), 0);
+	if (!p)
+		return;
+	/* store ptr_ */
+	p = bpf_kptr_xchg(&v->lref_ptr, p);
+	if (!p)
+		return;
+	bpf_kptr_free(p);
+}
+
 static void test_kptr(struct map_value *v)
 {
 	test_kptr_unref(v);
 	test_kptr_ref(v);
 	test_kptr_get(v);
+	test_local_kptr_ref(v);
 }
 
 SEC("tc")
Signature · 1 line
2.34.1