Skip to contentSkip to conversation

bpf: Use bpf_mem_cache_alloc/free in bpf_local_storage

PATCH Superseded Martin KaFai Lau 4 participants 23 messages Mar 6, 2023 – Mar 8, 2023

Subsystems (5)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Martin KaFai Lau

author reviewer
#0 lore raw

cover bpf: Use bpf_mem_cache_alloc/free in bpf_local_storage

From: Martin KaFai Lau <[email protected]>

This set is to use bpf_mem_cache_alloc/free in bpf_local_storage. The primary motivation is to solve the deadlock/recursion issue when bpf_task_storage is used in a bpf tracing prog [1]. This set also comes with a micro-benchmark to test the storage creation.

Patch 1 to 4 are some general cleanup after bpf_local_storage has been extended to multiple kernel objects (sk, task, inode, and then cgrp).

Patch 5 to 11 is to refactor the memory free logic into the new bpf_selem_free() and bpf_local_storage_free() functions. Together with the existing bpf_selem_alloc() and bpf_local_storage_alloc(), it should provide an easier way to change the alloc/free path in the future.

Patch 12 to 13 is to use bpf_mem_cache_alloc/free.

The remaining patches are selftests and benchmark.

[1]: https://lore.kernel.org/bpf/[email protected]/

Martin KaFai Lau (16):
bpf: Move a few bpf_local_storage functions to static scope
bpf: Refactor codes into bpf_local_storage_destroy
bpf: Remove __bpf_local_storage_map_alloc
bpf: Remove the preceding __ from __bpf_selem_unlink_storage
bpf: Remember smap in bpf_local_storage
bpf: Repurpose use_trace_rcu to reuse_now in bpf_local_storage
bpf: Remove bpf_selem_free_fields*_rcu
bpf: Add bpf_selem_free_rcu callback
bpf: Add bpf_selem_free()
bpf: Add bpf_local_storage_rcu callback
bpf: Add bpf_local_storage_free()
bpf: Use bpf_mem_cache_alloc/free in bpf_selem_alloc/free
bpf: Use bpf_mem_cache_alloc/free for bpf_local_storage selftests/bpf: Replace CHECK with ASSERT in test_local_storage selftests/bpf: Check freeing sk->sk_local_storage with

    sk_local_storage->smap is NULL

selftests/bpf: Add local-storage-create benchmark

 include/linux/bpf_local_storage.h             |  22 +-
 include/linux/bpf_mem_alloc.h                 |   5 +
 kernel/bpf/bpf_cgrp_storage.c                 |  11 +-
 kernel/bpf/bpf_inode_storage.c                |  10 +-
 kernel/bpf/bpf_local_storage.c                | 292 ++++++++++--------
 kernel/bpf/bpf_task_storage.c                 |  11 +-
 net/core/bpf_sk_storage.c                     |  12 +-
 tools/testing/selftests/bpf/Makefile          |   2 +
 tools/testing/selftests/bpf/bench.c           |   2 +
 .../bpf/benchs/bench_local_storage_create.c   | 141 +++++++++
 .../bpf/prog_tests/test_local_storage.c       |  49 ++-
 .../bpf/progs/bench_local_storage_create.c    |  57 ++++
 .../selftests/bpf/progs/local_storage.c       |  29 +-
 13 files changed, 429 insertions(+), 214 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/benchs/bench_local_storage_create.c
 create mode 100644 tools/testing/selftests/bpf/progs/bench_local_storage_create.c
Signature · 1 line
2.30.2
author reviewer
#1 lore raw

PATCH 1/16 bpf: Move a few bpf_local_storage functions to static scope

From: Martin KaFai Lau <[email protected]>

This patch moves the bpf_local_storage_free_rcu() and bpf_selem_unlink_map() to static because they are not used outside of bpf_local_storage.c.

---
 include/linux/bpf_local_storage.h | 4 ----
 kernel/bpf/bpf_local_storage.c    | 4 ++--
 2 files changed, 2 insertions(+), 6 deletions(-)
include/linux/bpf_local_storage.h +0 -4
index 6d37a40cd90e..6917c9a408a1 100644
@@ -147,8 +147,6 @@ void bpf_selem_unlink(struct bpf_local_storage_elem *selem, bool use_trace_rcu);
 void bpf_selem_link_map(struct bpf_local_storage_map *smap,
 			struct bpf_local_storage_elem *selem);
 
-void bpf_selem_unlink_map(struct bpf_local_storage_elem *selem);
-
 struct bpf_local_storage_elem *
 bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner, void *value,
 		bool charge_mem, gfp_t gfp_flags);
@@ -163,6 +161,4 @@ struct bpf_local_storage_data *
 bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap,
 			 void *value, u64 map_flags, gfp_t gfp_flags);
 
-void bpf_local_storage_free_rcu(struct rcu_head *rcu);
-
 #endif /* _BPF_LOCAL_STORAGE_H */
kernel/bpf/bpf_local_storage.c +2 -2
index 3d320393a12c..0510f50bd3ea 100644
@@ -95,7 +95,7 @@ bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner,
 	return NULL;
 }
 
-void bpf_local_storage_free_rcu(struct rcu_head *rcu)
+static void bpf_local_storage_free_rcu(struct rcu_head *rcu)
 {
 	struct bpf_local_storage *local_storage;
 
@@ -251,7 +251,7 @@ void bpf_selem_link_storage_nolock(struct bpf_local_storage *local_storage,
 	hlist_add_head_rcu(&selem->snode, &local_storage->list);
 }
 
-void bpf_selem_unlink_map(struct bpf_local_storage_elem *selem)
+static void bpf_selem_unlink_map(struct bpf_local_storage_elem *selem)
 {
 	struct bpf_local_storage_map *smap;
 	struct bpf_local_storage_map_bucket *b;
Signature · 1 line
2.30.2
author reviewer
#2 lore raw

PATCH 2/16 bpf: Refactor codes into bpf_local_storage_destroy

In reply to Martin KaFai Lau

From: Martin KaFai Lau <[email protected]>

This patch first renames bpf_local_storage_unlink_nolock to bpf_local_storage_destroy(). It better reflects that it is only used when the storage's owner (sk/task/cgrp/inode) is being kfree().

All bpf_local_storage_destroy's caller is taking the spin lock and then free the storage. This patch also moves these two steps into the bpf_local_storage_destroy.

This is a preparation work for a later patch that uses bpf_mem_cache_alloc/free in the bpf_local_storage.

---
 include/linux/bpf_local_storage.h | 2 +-
 kernel/bpf/bpf_cgrp_storage.c     | 9 +--------
 kernel/bpf/bpf_inode_storage.c    | 8 +-------
 kernel/bpf/bpf_local_storage.c    | 8 ++++++--
 kernel/bpf/bpf_task_storage.c     | 9 +--------
 net/core/bpf_sk_storage.c         | 8 +-------
 6 files changed, 11 insertions(+), 33 deletions(-)
include/linux/bpf_local_storage.h +1 -1
index 6917c9a408a1..c8dcf6f40497 100644
@@ -128,7 +128,7 @@ bpf_local_storage_lookup(struct bpf_local_storage *local_storage,
 			 struct bpf_local_storage_map *smap,
 			 bool cacheit_lockit);
 
-bool bpf_local_storage_unlink_nolock(struct bpf_local_storage *local_storage);
+void bpf_local_storage_destroy(struct bpf_local_storage *local_storage);
 
 void bpf_local_storage_map_free(struct bpf_map *map,
 				struct bpf_local_storage_cache *cache,
kernel/bpf/bpf_cgrp_storage.c +1 -8
index 6cdf6d9ed91d..1d00f1d9bdb7 100644
@@ -46,8 +46,6 @@ static struct bpf_local_storage __rcu **cgroup_storage_ptr(void *owner)
 void bpf_cgrp_storage_free(struct cgroup *cgroup)
 {
 	struct bpf_local_storage *local_storage;
-	bool free_cgroup_storage = false;
-	unsigned long flags;
 
 	rcu_read_lock();
 	local_storage = rcu_dereference(cgroup->bpf_cgrp_storage);
@@ -57,14 +55,9 @@ void bpf_cgrp_storage_free(struct cgroup *cgroup)
 	}
 
 	bpf_cgrp_storage_lock();
-	raw_spin_lock_irqsave(&local_storage->lock, flags);
-	free_cgroup_storage = bpf_local_storage_unlink_nolock(local_storage);
-	raw_spin_unlock_irqrestore(&local_storage->lock, flags);
+	bpf_local_storage_destroy(local_storage);
 	bpf_cgrp_storage_unlock();
 	rcu_read_unlock();
-
-	if (free_cgroup_storage)
-		kfree_rcu(local_storage, rcu);
 }
 
 static struct bpf_local_storage_data *
kernel/bpf/bpf_inode_storage.c +1 -7
index 05f4c66c9089..b4a9904df54e 100644
@@ -57,7 +57,6 @@ static struct bpf_local_storage_data *inode_storage_lookup(struct inode *inode,
 void bpf_inode_storage_free(struct inode *inode)
 {
 	struct bpf_local_storage *local_storage;
-	bool free_inode_storage = false;
 	struct bpf_storage_blob *bsb;
 
 	bsb = bpf_inode(inode);
@@ -72,13 +71,8 @@ void bpf_inode_storage_free(struct inode *inode)
 		return;
 	}
 
-	raw_spin_lock_bh(&local_storage->lock);
-	free_inode_storage = bpf_local_storage_unlink_nolock(local_storage);
-	raw_spin_unlock_bh(&local_storage->lock);
+	bpf_local_storage_destroy(local_storage);
 	rcu_read_unlock();
-
-	if (free_inode_storage)
-		kfree_rcu(local_storage, rcu);
 }
 
 static void *bpf_fd_inode_storage_lookup_elem(struct bpf_map *map, void *key)
kernel/bpf/bpf_local_storage.c +6 -2
index 0510f50bd3ea..4d2bc7c97f7d 100644
@@ -652,11 +652,12 @@ int bpf_local_storage_map_check_btf(const struct bpf_map *map,
 	return 0;
 }
 
-bool bpf_local_storage_unlink_nolock(struct bpf_local_storage *local_storage)
+void bpf_local_storage_destroy(struct bpf_local_storage *local_storage)
 {
 	struct bpf_local_storage_elem *selem;
 	bool free_storage = false;
 	struct hlist_node *n;
+	unsigned long flags;
 
 	/* Neither the bpf_prog nor the bpf_map's syscall
 	 * could be modifying the local_storage->list now.
@@ -667,6 +668,7 @@ bool bpf_local_storage_unlink_nolock(struct bpf_local_storage *local_storage)
 	 * when unlinking elem from the local_storage->list and
 	 * the map's bucket->list.
 	 */
+	raw_spin_lock_irqsave(&local_storage->lock, flags);
 	hlist_for_each_entry_safe(selem, n, &local_storage->list, snode) {
 		/* Always unlink from map before unlinking from
 		 * local_storage.
@@ -681,8 +683,10 @@ bool bpf_local_storage_unlink_nolock(struct bpf_local_storage *local_storage)
 		free_storage = bpf_selem_unlink_storage_nolock(
 			local_storage, selem, false, false);
 	}
+	raw_spin_unlock_irqrestore(&local_storage->lock, flags);
 
-	return free_storage;
+	if (free_storage)
+		kfree_rcu(local_storage, rcu);
 }
 
 struct bpf_map *
kernel/bpf/bpf_task_storage.c +1 -8
index 1e486055a523..b5f404fe146c 100644
@@ -72,8 +72,6 @@ task_storage_lookup(struct task_struct *task, struct bpf_map *map,
 void bpf_task_storage_free(struct task_struct *task)
 {
 	struct bpf_local_storage *local_storage;
-	bool free_task_storage = false;
-	unsigned long flags;
 
 	rcu_read_lock();
 
@@ -84,14 +82,9 @@ void bpf_task_storage_free(struct task_struct *task)
 	}
 
 	bpf_task_storage_lock();
-	raw_spin_lock_irqsave(&local_storage->lock, flags);
-	free_task_storage = bpf_local_storage_unlink_nolock(local_storage);
-	raw_spin_unlock_irqrestore(&local_storage->lock, flags);
+	bpf_local_storage_destroy(local_storage);
 	bpf_task_storage_unlock();
 	rcu_read_unlock();
-
-	if (free_task_storage)
-		kfree_rcu(local_storage, rcu);
 }
 
 static void *bpf_pid_task_storage_lookup_elem(struct bpf_map *map, void *key)
net/core/bpf_sk_storage.c +1 -7
index bb378c33f542..42569d0904a5 100644
@@ -49,7 +49,6 @@ static int bpf_sk_storage_del(struct sock *sk, struct bpf_map *map)
 void bpf_sk_storage_free(struct sock *sk)
 {
 	struct bpf_local_storage *sk_storage;
-	bool free_sk_storage = false;
 
 	rcu_read_lock();
 	sk_storage = rcu_dereference(sk->sk_bpf_storage);
@@ -58,13 +57,8 @@ void bpf_sk_storage_free(struct sock *sk)
 		return;
 	}
 
-	raw_spin_lock_bh(&sk_storage->lock);
-	free_sk_storage = bpf_local_storage_unlink_nolock(sk_storage);
-	raw_spin_unlock_bh(&sk_storage->lock);
+	bpf_local_storage_destroy(sk_storage);
 	rcu_read_unlock();
-
-	if (free_sk_storage)
-		kfree_rcu(sk_storage, rcu);
 }
 
 static void bpf_sk_storage_map_free(struct bpf_map *map)
Signature · 1 line
2.30.2
author reviewer
#3 lore raw

PATCH 3/16 bpf: Remove __bpf_local_storage_map_alloc

In reply to Martin KaFai Lau

From: Martin KaFai Lau <[email protected]>

bpf_local_storage_map_alloc() is the only caller of __bpf_local_storage_map_alloc(). The remaining logic in bpf_local_storage_map_alloc() is only a one liner setting the smap->cache_idx.

Remove __bpf_local_storage_map_alloc() to simplify code.

---
 kernel/bpf/bpf_local_storage.c | 63 ++++++++++++++--------------------
 1 file changed, 26 insertions(+), 37 deletions(-)
kernel/bpf/bpf_local_storage.c +26 -37
index 4d2bc7c97f7d..acedf6b07c54 100644
@@ -601,40 +601,6 @@ int bpf_local_storage_map_alloc_check(union bpf_attr *attr)
 	return 0;
 }
 
-static struct bpf_local_storage_map *__bpf_local_storage_map_alloc(union bpf_attr *attr)
-{
-	struct bpf_local_storage_map *smap;
-	unsigned int i;
-	u32 nbuckets;
-
-	smap = bpf_map_area_alloc(sizeof(*smap), NUMA_NO_NODE);
-	if (!smap)
-		return ERR_PTR(-ENOMEM);
-	bpf_map_init_from_attr(&smap->map, attr);
-
-	nbuckets = roundup_pow_of_two(num_possible_cpus());
-	/* Use at least 2 buckets, select_bucket() is undefined behavior with 1 bucket */
-	nbuckets = max_t(u32, 2, nbuckets);
-	smap->bucket_log = ilog2(nbuckets);
-
-	smap->buckets = bpf_map_kvcalloc(&smap->map, sizeof(*smap->buckets),
-					 nbuckets, GFP_USER | __GFP_NOWARN);
-	if (!smap->buckets) {
-		bpf_map_area_free(smap);
-		return ERR_PTR(-ENOMEM);
-	}
-
-	for (i = 0; i < nbuckets; i++) {
-		INIT_HLIST_HEAD(&smap->buckets[i].list);
-		raw_spin_lock_init(&smap->buckets[i].lock);
-	}
-
-	smap->elem_size = offsetof(struct bpf_local_storage_elem,
-				   sdata.data[attr->value_size]);
-
-	return smap;
-}
-
 int bpf_local_storage_map_check_btf(const struct bpf_map *map,
 				    const struct btf *btf,
 				    const struct btf_type *key_type,
@@ -694,10 +660,33 @@ bpf_local_storage_map_alloc(union bpf_attr *attr,
 			    struct bpf_local_storage_cache *cache)
 {
 	struct bpf_local_storage_map *smap;
+	unsigned int i;
+	u32 nbuckets;
+
+	smap = bpf_map_area_alloc(sizeof(*smap), NUMA_NO_NODE);
+	if (!smap)
+		return ERR_PTR(-ENOMEM);
+	bpf_map_init_from_attr(&smap->map, attr);
+
+	nbuckets = roundup_pow_of_two(num_possible_cpus());
+	/* Use at least 2 buckets, select_bucket() is undefined behavior with 1 bucket */
+	nbuckets = max_t(u32, 2, nbuckets);
+	smap->bucket_log = ilog2(nbuckets);
 
-	smap = __bpf_local_storage_map_alloc(attr);
-	if (IS_ERR(smap))
-		return ERR_CAST(smap);
+	smap->buckets = bpf_map_kvcalloc(&smap->map, sizeof(*smap->buckets),
+					 nbuckets, GFP_USER | __GFP_NOWARN);
+	if (!smap->buckets) {
+		bpf_map_area_free(smap);
+		return ERR_PTR(-ENOMEM);
+	}
+
+	for (i = 0; i < nbuckets; i++) {
+		INIT_HLIST_HEAD(&smap->buckets[i].list);
+		raw_spin_lock_init(&smap->buckets[i].lock);
+	}
+
+	smap->elem_size = offsetof(struct bpf_local_storage_elem,
+				   sdata.data[attr->value_size]);
 
 	smap->cache_idx = bpf_local_storage_cache_idx_get(cache);
 	return &smap->map;
Signature · 1 line
2.30.2
author reviewer
#4 lore raw

PATCH 4/16 bpf: Remove the preceding __ from __bpf_selem_unlink_storage

In reply to Martin KaFai Lau

From: Martin KaFai Lau <[email protected]>

__bpf_selem_unlink_storage is taking the spin lock and there is no name collision also. Having the preceding '__' is confusing when reviewing the later patch.

---
 kernel/bpf/bpf_local_storage.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
kernel/bpf/bpf_local_storage.c +3 -3
index acedf6b07c54..fef75beaf66d 100644
@@ -216,8 +216,8 @@ static bool bpf_selem_unlink_storage_nolock(struct bpf_local_storage *local_stor
 	return free_local_storage;
 }
 
-static void __bpf_selem_unlink_storage(struct bpf_local_storage_elem *selem,
-				       bool use_trace_rcu)
+static void bpf_selem_unlink_storage(struct bpf_local_storage_elem *selem,
+				     bool use_trace_rcu)
 {
 	struct bpf_local_storage *local_storage;
 	bool free_local_storage = false;
@@ -288,7 +288,7 @@ void bpf_selem_unlink(struct bpf_local_storage_elem *selem, bool use_trace_rcu)
 	 * the local_storage.
 	 */
 	bpf_selem_unlink_map(selem);
-	__bpf_selem_unlink_storage(selem, use_trace_rcu);
+	bpf_selem_unlink_storage(selem, use_trace_rcu);
 }
 
 /* If cacheit_lockit is false, this lookup function is lockless */
Signature · 1 line
2.30.2
author reviewer
#5 lore raw

PATCH 5/16 bpf: Remember smap in bpf_local_storage

In reply to Martin KaFai Lau

From: Martin KaFai Lau <[email protected]>

This patch remembers which smap triggers the allocation of a 'struct bpf_local_storage' object. The local_storage is allocated during the very first selem added to the owner. The smap pointer is needed when using the bpf_mem_cache_free in a later patch because it needs to free to the correct smap's bpf_mem_alloc object.

When a selem is being removed, it needs to check if it is the selem that triggers the creation of the local_storage. If it is, the local_storage->smap pointer will be reset to NULL. This NULL reset is done under the local_storage->lock in bpf_selem_unlink_storage_nolock() when a selem is being removed. Also note that the local_storage may not go away even local_storage->smap is NULL because there may be other selem still stored in the local_storage.

---
 include/linux/bpf_local_storage.h | 1 +
 kernel/bpf/bpf_local_storage.c    | 4 ++++
 2 files changed, 5 insertions(+)
include/linux/bpf_local_storage.h +1 -0
index c8dcf6f40497..31ee681b4c65 100644
@@ -83,6 +83,7 @@ struct bpf_local_storage_elem {
 
 struct bpf_local_storage {
 	struct bpf_local_storage_data __rcu *cache[BPF_LOCAL_STORAGE_CACHE_SIZE];
+	struct bpf_local_storage_map __rcu *smap;
 	struct hlist_head list; /* List of bpf_local_storage_elem */
 	void *owner;		/* The object that owns the above "list" of
 				 * bpf_local_storage_elem.
kernel/bpf/bpf_local_storage.c +4 -0
index fef75beaf66d..c8ca1886374e 100644
@@ -213,6 +213,9 @@ static bool bpf_selem_unlink_storage_nolock(struct bpf_local_storage *local_stor
 			kfree_rcu(selem, rcu);
 	}
 
+	if (rcu_access_pointer(local_storage->smap) == smap)
+		RCU_INIT_POINTER(local_storage->smap, NULL);
+
 	return free_local_storage;
 }
 
@@ -368,6 +371,7 @@ int bpf_local_storage_alloc(void *owner,
 		goto uncharge;
 	}
 
+	RCU_INIT_POINTER(storage->smap, smap);
 	INIT_HLIST_HEAD(&storage->list);
 	raw_spin_lock_init(&storage->lock);
 	storage->owner = owner;
Signature · 1 line
2.30.2
author reviewer
#6 lore raw

PATCH 6/16 bpf: Repurpose use_trace_rcu to reuse_now in bpf_local_storage

In reply to Martin KaFai Lau

From: Martin KaFai Lau <[email protected]>

This patch re-purpose the use_trace_rcu to mean if the freed memory can be reused immediately or not. The use_trace_rcu is renamed to reuse_now. Other than the boolean test is reversed, it should be a no-op.

The following explains the reason for the rename and how it will be used in a later patch.

In a later patch, bpf_mem_cache_alloc/free will be used
in the bpf_local_storage. The bpf mem allocator will reuse the freed memory immediately. Some of the free paths in bpf_local_storage does not support memory to be reused immediately. These paths are the "delete" elem cases from the bpf_*_storage_delete() helper and the map_delete_elem() syscall. Note that "delete" elem before the owner's (sk/task/cgrp/inode) lifetime ended is not the common usage for the local storage.

The common free path, bpf_local_storage_destroy(), can reuse the memory immediately. This common path means the storage stays with its owner until the owner is destroyed.

The above mentioned "delete" elem paths that cannot reuse immediately always has the 'use_trace_rcu == true'. The cases that is safe for immediate reuse always have 'use_trace_rcu == false'. Instead of adding another arg in a later patch, this patch re-purpose this arg to reuse_now and have the test logic reversed.

In a later patch, 'reuse_now == true' will free to the bpf_mem_cache_free() where the memory can be reused immediately. 'reuse_now == false' will go through the call_rcu_tasks_trace().

---
 include/linux/bpf_local_storage.h |  2 +-
 kernel/bpf/bpf_cgrp_storage.c     |  2 +-
 kernel/bpf/bpf_inode_storage.c    |  2 +-
 kernel/bpf/bpf_local_storage.c    | 24 ++++++++++++------------
 kernel/bpf/bpf_task_storage.c     |  2 +-
 net/core/bpf_sk_storage.c         |  2 +-
 6 files changed, 17 insertions(+), 17 deletions(-)
include/linux/bpf_local_storage.h +1 -1
index 31ee681b4c65..fad09f42a2f4 100644
@@ -143,7 +143,7 @@ int bpf_local_storage_map_check_btf(const struct bpf_map *map,
 void bpf_selem_link_storage_nolock(struct bpf_local_storage *local_storage,
 				   struct bpf_local_storage_elem *selem);
 
-void bpf_selem_unlink(struct bpf_local_storage_elem *selem, bool use_trace_rcu);
+void bpf_selem_unlink(struct bpf_local_storage_elem *selem, bool reuse_now);
 
 void bpf_selem_link_map(struct bpf_local_storage_map *smap,
 			struct bpf_local_storage_elem *selem);
kernel/bpf/bpf_cgrp_storage.c +1 -1
index 1d00f1d9bdb7..617e6111a51f 100644
@@ -121,7 +121,7 @@ static int cgroup_storage_delete(struct cgroup *cgroup, struct bpf_map *map)
 	if (!sdata)
 		return -ENOENT;
 
-	bpf_selem_unlink(SELEM(sdata), true);
+	bpf_selem_unlink(SELEM(sdata), false);
 	return 0;
 }
 
kernel/bpf/bpf_inode_storage.c +1 -1
index b4a9904df54e..e37d9d2e28f4 100644
@@ -122,7 +122,7 @@ static int inode_storage_delete(struct inode *inode, struct bpf_map *map)
 	if (!sdata)
 		return -ENOENT;
 
-	bpf_selem_unlink(SELEM(sdata), true);
+	bpf_selem_unlink(SELEM(sdata), false);
 
 	return 0;
 }
kernel/bpf/bpf_local_storage.c +12 -12
index c8ca1886374e..8c1401a24c7d 100644
@@ -147,7 +147,7 @@ static void bpf_selem_free_trace_rcu(struct rcu_head *rcu)
  */
 static bool bpf_selem_unlink_storage_nolock(struct bpf_local_storage *local_storage,
 					    struct bpf_local_storage_elem *selem,
-					    bool uncharge_mem, bool use_trace_rcu)
+					    bool uncharge_mem, bool reuse_now)
 {
 	struct bpf_local_storage_map *smap;
 	bool free_local_storage;
@@ -201,7 +201,7 @@ static bool bpf_selem_unlink_storage_nolock(struct bpf_local_storage *local_stor
 	 * any special fields.
 	 */
 	rec = smap->map.record;
-	if (use_trace_rcu) {
+	if (!reuse_now) {
 		if (!IS_ERR_OR_NULL(rec))
 			call_rcu_tasks_trace(&selem->rcu, bpf_selem_free_fields_trace_rcu);
 		else
@@ -220,7 +220,7 @@ static bool bpf_selem_unlink_storage_nolock(struct bpf_local_storage *local_stor
 }
 
 static void bpf_selem_unlink_storage(struct bpf_local_storage_elem *selem,
-				     bool use_trace_rcu)
+				     bool reuse_now)
 {
 	struct bpf_local_storage *local_storage;
 	bool free_local_storage = false;
@@ -235,11 +235,11 @@ static void bpf_selem_unlink_storage(struct bpf_local_storage_elem *selem,
 	raw_spin_lock_irqsave(&local_storage->lock, flags);
 	if (likely(selem_linked_to_storage(selem)))
 		free_local_storage = bpf_selem_unlink_storage_nolock(
-			local_storage, selem, true, use_trace_rcu);
+			local_storage, selem, true, reuse_now);
 	raw_spin_unlock_irqrestore(&local_storage->lock, flags);
 
 	if (free_local_storage) {
-		if (use_trace_rcu)
+		if (!reuse_now)
 			call_rcu_tasks_trace(&local_storage->rcu,
 				     bpf_local_storage_free_rcu);
 		else
@@ -284,14 +284,14 @@ void bpf_selem_link_map(struct bpf_local_storage_map *smap,
 	raw_spin_unlock_irqrestore(&b->lock, flags);
 }
 
-void bpf_selem_unlink(struct bpf_local_storage_elem *selem, bool use_trace_rcu)
+void bpf_selem_unlink(struct bpf_local_storage_elem *selem, bool reuse_now)
 {
 	/* Always unlink from map before unlinking from local_storage
 	 * because selem will be freed after successfully unlinked from
 	 * the local_storage.
 	 */
 	bpf_selem_unlink_map(selem);
-	bpf_selem_unlink_storage(selem, use_trace_rcu);
+	bpf_selem_unlink_storage(selem, reuse_now);
 }
 
 /* If cacheit_lockit is false, this lookup function is lockless */
@@ -538,7 +538,7 @@ bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap,
 	if (old_sdata) {
 		bpf_selem_unlink_map(SELEM(old_sdata));
 		bpf_selem_unlink_storage_nolock(local_storage, SELEM(old_sdata),
-						false, true);
+						false, false);
 	}
 
 unlock:
@@ -651,7 +651,7 @@ void bpf_local_storage_destroy(struct bpf_local_storage *local_storage)
 		 * of the loop will set the free_cgroup_storage to true.
 		 */
 		free_storage = bpf_selem_unlink_storage_nolock(
-			local_storage, selem, false, false);
+			local_storage, selem, false, true);
 	}
 	raw_spin_unlock_irqrestore(&local_storage->lock, flags);
 
@@ -735,7 +735,7 @@ void bpf_local_storage_map_free(struct bpf_map *map,
 				migrate_disable();
 				this_cpu_inc(*busy_counter);
 			}
-			bpf_selem_unlink(selem, false);
+			bpf_selem_unlink(selem, true);
 			if (busy_counter) {
 				this_cpu_dec(*busy_counter);
 				migrate_enable();
@@ -773,8 +773,8 @@ void bpf_local_storage_map_free(struct bpf_map *map,
 		/* We cannot skip rcu_barrier() when rcu_trace_implies_rcu_gp()
 		 * is true, because while call_rcu invocation is skipped in that
 		 * case in bpf_selem_free_fields_trace_rcu (and all local
-		 * storage maps pass use_trace_rcu = true), there can be
-		 * call_rcu callbacks based on use_trace_rcu = false in the
+		 * storage maps pass reuse_now = false), there can be
+		 * call_rcu callbacks based on reuse_now = true in the
 		 * while ((selem = ...)) loop above or when owner's free path
 		 * calls bpf_local_storage_unlink_nolock.
 		 */
kernel/bpf/bpf_task_storage.c +1 -1
index b5f404fe146c..65aeb042c263 100644
@@ -168,7 +168,7 @@ static int task_storage_delete(struct task_struct *task, struct bpf_map *map,
 	if (!nobusy)
 		return -EBUSY;
 
-	bpf_selem_unlink(SELEM(sdata), true);
+	bpf_selem_unlink(SELEM(sdata), false);
 
 	return 0;
 }
net/core/bpf_sk_storage.c +1 -1
index 42569d0904a5..8b0c9e4341eb 100644
@@ -40,7 +40,7 @@ static int bpf_sk_storage_del(struct sock *sk, struct bpf_map *map)
 	if (!sdata)
 		return -ENOENT;
 
-	bpf_selem_unlink(SELEM(sdata), true);
+	bpf_selem_unlink(SELEM(sdata), false);
 
 	return 0;
 }
Signature · 1 line
2.30.2
author reviewer
#7 lore raw

PATCH 7/16 bpf: Remove bpf_selem_free_fields*_rcu

In reply to Martin KaFai Lau

From: Martin KaFai Lau <[email protected]>

This patch removes the bpf_selem_free_fields*_rcu. The bpf_obj_free_fields() can be done before the call_rcu_trasks_trace() and kfree_rcu(). It is needed when a later patch uses bpf_mem_cache_alloc/free. In bpf hashtab, bpf_obj_free_fields() is also called before calling bpf_mem_cache_free. The discussion can be found in
https://lore.kernel.org/bpf/[email protected]/

Cc Kumar Kartikeya Dwivedi <[email protected]>
---
 kernel/bpf/bpf_local_storage.c | 67 +++-------------------------------
 1 file changed, 5 insertions(+), 62 deletions(-)
kernel/bpf/bpf_local_storage.c +5 -62
index 8c1401a24c7d..9a1febcc9565 100644
@@ -109,27 +109,6 @@ static void bpf_local_storage_free_rcu(struct rcu_head *rcu)
 		kfree_rcu(local_storage, rcu);
 }
 
-static void bpf_selem_free_fields_rcu(struct rcu_head *rcu)
-{
-	struct bpf_local_storage_elem *selem;
-	struct bpf_local_storage_map *smap;
-
-	selem = container_of(rcu, struct bpf_local_storage_elem, rcu);
-	/* protected by the rcu_barrier*() */
-	smap = rcu_dereference_protected(SDATA(selem)->smap, true);
-	bpf_obj_free_fields(smap->map.record, SDATA(selem)->data);
-	kfree(selem);
-}
-
-static void bpf_selem_free_fields_trace_rcu(struct rcu_head *rcu)
-{
-	/* Free directly if Tasks Trace RCU GP also implies RCU GP */
-	if (rcu_trace_implies_rcu_gp())
-		bpf_selem_free_fields_rcu(rcu);
-	else
-		call_rcu(rcu, bpf_selem_free_fields_rcu);
-}
-
 static void bpf_selem_free_trace_rcu(struct rcu_head *rcu)
 {
 	struct bpf_local_storage_elem *selem;
@@ -151,7 +130,6 @@ static bool bpf_selem_unlink_storage_nolock(struct bpf_local_storage *local_stor
 {
 	struct bpf_local_storage_map *smap;
 	bool free_local_storage;
-	struct btf_record *rec;
 	void *owner;
 
 	smap = rcu_dereference_check(SDATA(selem)->smap, bpf_rcu_lock_held());
@@ -192,26 +170,11 @@ static bool bpf_selem_unlink_storage_nolock(struct bpf_local_storage *local_stor
 	    SDATA(selem))
 		RCU_INIT_POINTER(local_storage->cache[smap->cache_idx], NULL);
 
-	/* A different RCU callback is chosen whenever we need to free
-	 * additional fields in selem data before freeing selem.
-	 * bpf_local_storage_map_free only executes rcu_barrier to wait for RCU
-	 * callbacks when it has special fields, hence we can only conditionally
-	 * dereference smap, as by this time the map might have already been
-	 * freed without waiting for our call_rcu callback if it did not have
-	 * any special fields.
-	 */
-	rec = smap->map.record;
-	if (!reuse_now) {
-		if (!IS_ERR_OR_NULL(rec))
-			call_rcu_tasks_trace(&selem->rcu, bpf_selem_free_fields_trace_rcu);
-		else
-			call_rcu_tasks_trace(&selem->rcu, bpf_selem_free_trace_rcu);
-	} else {
-		if (!IS_ERR_OR_NULL(rec))
-			call_rcu(&selem->rcu, bpf_selem_free_fields_rcu);
-		else
-			kfree_rcu(selem, rcu);
-	}
+	bpf_obj_free_fields(smap->map.record, SDATA(selem)->data);
+	if (!reuse_now)
+		call_rcu_tasks_trace(&selem->rcu, bpf_selem_free_trace_rcu);
+	else
+		kfree_rcu(selem, rcu);
 
 	if (rcu_access_pointer(local_storage->smap) == smap)
 		RCU_INIT_POINTER(local_storage->smap, NULL);
@@ -759,26 +722,6 @@ void bpf_local_storage_map_free(struct bpf_map *map,
 	 */
 	synchronize_rcu();
 
-	/* Only delay freeing of smap, buckets are not needed anymore */
 	kvfree(smap->buckets);
-
-	/* When local storage has special fields, callbacks for
-	 * bpf_selem_free_fields_rcu and bpf_selem_free_fields_trace_rcu will
-	 * keep using the map BTF record, we need to execute an RCU barrier to
-	 * wait for them as the record will be freed right after our map_free
-	 * callback.
-	 */
-	if (!IS_ERR_OR_NULL(smap->map.record)) {
-		rcu_barrier_tasks_trace();
-		/* We cannot skip rcu_barrier() when rcu_trace_implies_rcu_gp()
-		 * is true, because while call_rcu invocation is skipped in that
-		 * case in bpf_selem_free_fields_trace_rcu (and all local
-		 * storage maps pass reuse_now = false), there can be
-		 * call_rcu callbacks based on reuse_now = true in the
-		 * while ((selem = ...)) loop above or when owner's free path
-		 * calls bpf_local_storage_unlink_nolock.
-		 */
-		rcu_barrier();
-	}
 	bpf_map_area_free(smap);
 }
Signature · 1 line
2.30.2
maintainer Acked-by on v1 7/16
#8 lore raw

Re: [PATCH bpf-next 07/16] bpf: Remove bpf_selem_free_fields*_rcu

Quoted 10 lines · Martin KaFai Lau

From: Martin KaFai Lau <[email protected]>

This patch removes the bpf_selem_free_fields*_rcu. The bpf_obj_free_fields() can be done before the call_rcu_trasks_trace() and kfree_rcu(). It is needed when a later patch uses bpf_mem_cache_alloc/free. In bpf hashtab, bpf_obj_free_fields() is also called before calling bpf_mem_cache_free. The discussion can be found in
https://lore.kernel.org/bpf/[email protected]/

Quoted 3 lines
Cc Kumar Kartikeya Dwivedi <[email protected]>

---

author reviewer
#9 lore raw

PATCH 8/16 bpf: Add bpf_selem_free_rcu callback

In reply to Martin KaFai Lau

From: Martin KaFai Lau <[email protected]>

Add bpf_selem_free_rcu() callback to do the kfree() instead of using kfree_rcu. It is a preparation work for using bpf_mem_cache_alloc/free in a later patch.

---
 kernel/bpf/bpf_local_storage.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
kernel/bpf/bpf_local_storage.c +9 -4
index 9a1febcc9565..528579c9f60f 100644
@@ -109,15 +109,20 @@ static void bpf_local_storage_free_rcu(struct rcu_head *rcu)
 		kfree_rcu(local_storage, rcu);
 }
 
-static void bpf_selem_free_trace_rcu(struct rcu_head *rcu)
+static void bpf_selem_free_rcu(struct rcu_head *rcu)
 {
 	struct bpf_local_storage_elem *selem;
 
 	selem = container_of(rcu, struct bpf_local_storage_elem, rcu);
+	kfree(selem);
+}
+
+static void bpf_selem_free_trace_rcu(struct rcu_head *rcu)
+{
 	if (rcu_trace_implies_rcu_gp())
-		kfree(selem);
+		bpf_selem_free_rcu(rcu);
 	else
-		kfree_rcu(selem, rcu);
+		call_rcu(rcu, bpf_selem_free_rcu);
 }
 
 /* local_storage->lock must be held and selem->local_storage == local_storage.
@@ -174,7 +179,7 @@ static bool bpf_selem_unlink_storage_nolock(struct bpf_local_storage *local_stor
 	if (!reuse_now)
 		call_rcu_tasks_trace(&selem->rcu, bpf_selem_free_trace_rcu);
 	else
-		kfree_rcu(selem, rcu);
+		call_rcu(&selem->rcu, bpf_selem_free_rcu);
 
 	if (rcu_access_pointer(local_storage->smap) == smap)
 		RCU_INIT_POINTER(local_storage->smap, NULL);
Signature · 1 line
2.30.2
author reviewer
#10 lore raw

PATCH 9/16 bpf: Add bpf_selem_free()

In reply to Martin KaFai Lau

From: Martin KaFai Lau <[email protected]>

This patch refactors the selem freeing logic into bpf_selem_free(). It is a preparation work for a later patch using bpf_mem_cache_alloc/free. The other kfree(selem) cases are also changed to bpf_selem_free(..., reuse_now = true).

---
 include/linux/bpf_local_storage.h |  4 ++++
 kernel/bpf/bpf_local_storage.c    | 21 ++++++++++++++-------
 net/core/bpf_sk_storage.c         |  2 +-
 3 files changed, 19 insertions(+), 8 deletions(-)
include/linux/bpf_local_storage.h +4 -0
index fad09f42a2f4..adb5023a1af5 100644
@@ -152,6 +152,10 @@ struct bpf_local_storage_elem *
 bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner, void *value,
 		bool charge_mem, gfp_t gfp_flags);
 
+void bpf_selem_free(struct bpf_local_storage_elem *selem,
+		    struct bpf_local_storage_map *smap,
+		    bool reuse_now);
+
 int
 bpf_local_storage_alloc(void *owner,
 			struct bpf_local_storage_map *smap,
kernel/bpf/bpf_local_storage.c +14 -7
index 528579c9f60f..f611668f8a0b 100644
@@ -125,6 +125,17 @@ static void bpf_selem_free_trace_rcu(struct rcu_head *rcu)
 		call_rcu(rcu, bpf_selem_free_rcu);
 }
 
+void bpf_selem_free(struct bpf_local_storage_elem *selem,
+		    struct bpf_local_storage_map *smap,
+		    bool reuse_now)
+{
+	bpf_obj_free_fields(smap->map.record, SDATA(selem)->data);
+	if (!reuse_now)
+		call_rcu_tasks_trace(&selem->rcu, bpf_selem_free_trace_rcu);
+	else
+		call_rcu(&selem->rcu, bpf_selem_free_rcu);
+}
+
 /* local_storage->lock must be held and selem->local_storage == local_storage.
  * The caller must ensure selem->smap is still valid to be
  * dereferenced for its smap->elem_size and smap->cache_idx.
@@ -175,11 +186,7 @@ static bool bpf_selem_unlink_storage_nolock(struct bpf_local_storage *local_stor
 	    SDATA(selem))
 		RCU_INIT_POINTER(local_storage->cache[smap->cache_idx], NULL);
 
-	bpf_obj_free_fields(smap->map.record, SDATA(selem)->data);
-	if (!reuse_now)
-		call_rcu_tasks_trace(&selem->rcu, bpf_selem_free_trace_rcu);
-	else
-		call_rcu(&selem->rcu, bpf_selem_free_rcu);
+	bpf_selem_free(selem, smap, reuse_now);
 
 	if (rcu_access_pointer(local_storage->smap) == smap)
 		RCU_INIT_POINTER(local_storage->smap, NULL);
@@ -423,7 +430,7 @@ bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap,
 
 		err = bpf_local_storage_alloc(owner, smap, selem, gfp_flags);
 		if (err) {
-			kfree(selem);
+			bpf_selem_free(selem, smap, true);
 			mem_uncharge(smap, owner, smap->elem_size);
 			return ERR_PTR(err);
 		}
@@ -517,7 +524,7 @@ bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap,
 	raw_spin_unlock_irqrestore(&local_storage->lock, flags);
 	if (selem) {
 		mem_uncharge(smap, owner, smap->elem_size);
-		kfree(selem);
+		bpf_selem_free(selem, smap, true);
 	}
 	return ERR_PTR(err);
 }
net/core/bpf_sk_storage.c +1 -1
index 8b0c9e4341eb..4fc078e8e9ca 100644
@@ -197,7 +197,7 @@ int bpf_sk_storage_clone(const struct sock *sk, struct sock *newsk)
 		} else {
 			ret = bpf_local_storage_alloc(newsk, smap, copy_selem, GFP_ATOMIC);
 			if (ret) {
-				kfree(copy_selem);
+				bpf_selem_free(selem, smap, true);
 				atomic_sub(smap->elem_size,
 					   &newsk->sk_omem_alloc);
 				bpf_map_put(map);
Signature · 1 line
2.30.2
author reviewer
#11 lore raw

Re: [PATCH bpf-next 09/16] bpf: Add bpf_selem_free()

net/core/bpf_sk_storage.c int bpf_sk_storage_clone(const struct sock *sk, struct sock *newsk) · line 200

Quoted 10 lines · Martin KaFai Lau
net/core/bpf_sk_storage.c +1 -1
index 8b0c9e4341eb..4fc078e8e9ca 100644
@@ -197,7 +197,7 @@ int bpf_sk_storage_clone(const struct sock *sk, struct sock *newsk)
  		} else {
  			ret = bpf_local_storage_alloc(newsk, smap, copy_selem, GFP_ATOMIC);
  			if (ret) {
-				kfree(copy_selem);
+				bpf_selem_free(selem, smap, true);

noticed there is a bug here, should be copy_selem. will fix.

Quoted 3 lines
net/core/bpf_sk_storage.c +0 -0
  				atomic_sub(smap->elem_size,
  					   &newsk->sk_omem_alloc);
  				bpf_map_put(map);
author reviewer
#12 lore raw

PATCH 10/16 bpf: Add bpf_local_storage_rcu callback

In reply to Martin KaFai Lau

From: Martin KaFai Lau <[email protected]>

The existing bpf_local_storage_free_rcu is renamed to bpf_local_storage_free_trace_rcu. A new bpf_local_storage_rcu callback is added to do the kfree instead of using kfree_rcu. It is a preparation work for a later patch using bpf_mem_cache_alloc/free.

---
 kernel/bpf/bpf_local_storage.c | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)
kernel/bpf/bpf_local_storage.c +11 -6
index f611668f8a0b..0568b479bdb0 100644
@@ -99,14 +99,19 @@ static void bpf_local_storage_free_rcu(struct rcu_head *rcu)
 {
 	struct bpf_local_storage *local_storage;
 
+	local_storage = container_of(rcu, struct bpf_local_storage, rcu);
+	kfree(local_storage);
+}
+
+static void bpf_local_storage_free_trace_rcu(struct rcu_head *rcu)
+{
 	/* If RCU Tasks Trace grace period implies RCU grace period, do
 	 * kfree(), else do kfree_rcu().
 	 */
-	local_storage = container_of(rcu, struct bpf_local_storage, rcu);
 	if (rcu_trace_implies_rcu_gp())
-		kfree(local_storage);
+		bpf_local_storage_free_rcu(rcu);
 	else
-		kfree_rcu(local_storage, rcu);
+		call_rcu(rcu, bpf_local_storage_free_rcu);
 }
 
 static void bpf_selem_free_rcu(struct rcu_head *rcu)
@@ -216,9 +221,9 @@ static void bpf_selem_unlink_storage(struct bpf_local_storage_elem *selem,
 	if (free_local_storage) {
 		if (!reuse_now)
 			call_rcu_tasks_trace(&local_storage->rcu,
-				     bpf_local_storage_free_rcu);
+				     bpf_local_storage_free_trace_rcu);
 		else
-			kfree_rcu(local_storage, rcu);
+			call_rcu(&local_storage->rcu, bpf_local_storage_free_rcu);
 	}
 }
 
@@ -631,7 +636,7 @@ void bpf_local_storage_destroy(struct bpf_local_storage *local_storage)
 	raw_spin_unlock_irqrestore(&local_storage->lock, flags);
 
 	if (free_storage)
-		kfree_rcu(local_storage, rcu);
+		call_rcu(&local_storage->rcu, bpf_local_storage_free_rcu);
 }
 
 struct bpf_map *
Signature · 1 line
2.30.2
author reviewer
#13 lore raw

PATCH 11/16 bpf: Add bpf_local_storage_free()

In reply to Martin KaFai Lau

From: Martin KaFai Lau <[email protected]>

This patch refactors local_storage freeing logic into bpf_local_storage_free(). It is a preparation work for a later patch that uses bpf_mem_cache_alloc/free. The other kfree(local_storage) cases are also changed to bpf_local_storage_free(..., reuse_now = true).

---
 kernel/bpf/bpf_local_storage.c | 23 ++++++++++++++---------
 1 file changed, 14 insertions(+), 9 deletions(-)
kernel/bpf/bpf_local_storage.c +14 -9
index 0568b479bdb0..532b82084ba7 100644
@@ -114,6 +114,16 @@ static void bpf_local_storage_free_trace_rcu(struct rcu_head *rcu)
 		call_rcu(rcu, bpf_local_storage_free_rcu);
 }
 
+static void bpf_local_storage_free(struct bpf_local_storage *local_storage,
+				   bool reuse_now)
+{
+	if (!reuse_now)
+		call_rcu_tasks_trace(&local_storage->rcu,
+				     bpf_local_storage_free_trace_rcu);
+	else
+		call_rcu(&local_storage->rcu, bpf_local_storage_free_rcu);
+}
+
 static void bpf_selem_free_rcu(struct rcu_head *rcu)
 {
 	struct bpf_local_storage_elem *selem;
@@ -218,13 +228,8 @@ static void bpf_selem_unlink_storage(struct bpf_local_storage_elem *selem,
 			local_storage, selem, true, reuse_now);
 	raw_spin_unlock_irqrestore(&local_storage->lock, flags);
 
-	if (free_local_storage) {
-		if (!reuse_now)
-			call_rcu_tasks_trace(&local_storage->rcu,
-				     bpf_local_storage_free_trace_rcu);
-		else
-			call_rcu(&local_storage->rcu, bpf_local_storage_free_rcu);
-	}
+	if (free_local_storage)
+		bpf_local_storage_free(local_storage, reuse_now);
 }
 
 void bpf_selem_link_storage_nolock(struct bpf_local_storage *local_storage,
@@ -391,7 +396,7 @@ int bpf_local_storage_alloc(void *owner,
 	return 0;
 
 uncharge:
-	kfree(storage);
+	bpf_local_storage_free(storage, true);
 	mem_uncharge(smap, owner, sizeof(*storage));
 	return err;
 }
@@ -636,7 +641,7 @@ void bpf_local_storage_destroy(struct bpf_local_storage *local_storage)
 	raw_spin_unlock_irqrestore(&local_storage->lock, flags);
 
 	if (free_storage)
-		call_rcu(&local_storage->rcu, bpf_local_storage_free_rcu);
+		bpf_local_storage_free(local_storage, true);
 }
 
 struct bpf_map *
Signature · 1 line
2.30.2
author reviewer
#14 lore raw

PATCH 12/16 bpf: Use bpf_mem_cache_alloc/free in bpf_selem_alloc/free

In reply to Martin KaFai Lau

From: Martin KaFai Lau <[email protected]>

This patch uses bpf_mem_cache_alloc/free in bpf_selem_alloc/free.

The ____cacheline_aligned attribute is no longer needed
in 'struct bpf_local_storage_elem'. bpf_mem_cache_alloc will have 'struct llist_node' in front of the 'struct bpf_local_storage_elem'. It will use the 8bytes hole in the bpf_local_storage_elem.

After bpf_mem_cache_alloc(), the SDATA(selem)->data is zero-ed because bpf_mem_cache_alloc() could return a reused selem. It is to keep the existing bpf_map_kzalloc() behavior. Only SDATA(selem)->data is zero-ed. SDATA(selem)->data is the visible part to the bpf prog. No need to use zero_map_value() to do the zeroing because bpf_selem_free() ensures no bpf prog is using the selem before returning the selem through bpf_mem_cache_free(). For the internal fields of selem, they will be initialized when linking to the new smap and the new local_storage.

When bpf_mem_cache_alloc() fails, bpf_selem_alloc() will try to fallback to kzalloc only if the caller has GFP_KERNEL flag set (ie. from sleepable bpf prog that should not cause deadlock). BPF_MA_SIZE and BPF_MA_PTR macro are added for that.

For the common selem free path where the selem is freed when its owner is also being freed, reuse_now == true and selem can be reused immediately. bpf_selem_free() uses bpf_mem_cache_free() where selem will be considered for immediate reuse.

For the uncommon path that the bpf prog explicitly deletes the selem (by using the helper bpf_*_storage_delete), the selem cannot be reused immediately. reuse_now == false and bpf_selem_free() will stay with the current call_rcu_tasks_trace. BPF_MA_NODE macro is added to get the correct address for the kfree.

mem_charge and mem_uncharge are changed to use the BPF_MA_SIZE macro. It will have a temporarily over-charge for the bpf_local_storage_alloc() because bpf_local_storage is not moved to bpf_mem_cache_alloc in this patch but it will be done in the next patch.

---
 include/linux/bpf_local_storage.h |  8 ++---
 include/linux/bpf_mem_alloc.h     |  5 +++
 kernel/bpf/bpf_local_storage.c    | 56 +++++++++++++++++++++++++------
 3 files changed, 53 insertions(+), 16 deletions(-)
include/linux/bpf_local_storage.h +3 -5
index adb5023a1af5..a236c9b964cf 100644
@@ -13,6 +13,7 @@
 #include <linux/list.h>
 #include <linux/hash.h>
 #include <linux/types.h>
+#include <linux/bpf_mem_alloc.h>
 #include <uapi/linux/btf.h>
 
 #define BPF_LOCAL_STORAGE_CACHE_SIZE	16
@@ -55,6 +56,7 @@ struct bpf_local_storage_map {
 	u32 bucket_log;
 	u16 elem_size;
 	u16 cache_idx;
+	struct bpf_mem_alloc selem_ma;
 };
 
 struct bpf_local_storage_data {
@@ -74,11 +76,7 @@ struct bpf_local_storage_elem {
 	struct hlist_node snode;	/* Linked to bpf_local_storage */
 	struct bpf_local_storage __rcu *local_storage;
 	struct rcu_head rcu;
-	/* 8 bytes hole */
-	/* The data is stored in another cacheline to minimize
-	 * the number of cachelines access during a cache hit.
-	 */
-	struct bpf_local_storage_data sdata ____cacheline_aligned;
+	struct bpf_local_storage_data sdata;
 };
 
 struct bpf_local_storage {
include/linux/bpf_mem_alloc.h +5 -0
index a7104af61ab4..0ab16fb0ab50 100644
@@ -5,6 +5,11 @@
 #include <linux/compiler_types.h>
 #include <linux/workqueue.h>
 
+#define BPF_MA_NODE_SZ sizeof(struct llist_node)
+#define BPF_MA_SIZE(_size) ((_size) + BPF_MA_NODE_SZ)
+#define BPF_MA_PTR(_node) ((void *)(_node) + BPF_MA_NODE_SZ)
+#define BPF_MA_NODE(_ptr) ((void *)(_ptr) - BPF_MA_NODE_SZ)
+
 struct bpf_mem_cache;
 struct bpf_mem_caches;
 
kernel/bpf/bpf_local_storage.c +45 -11

116 lines not on this page · raw message

Signature · 1 line
2.30.2
#15 lore raw

Re: [PATCH bpf-next 12/16] bpf: Use bpf_mem_cache_alloc/free in bpf_selem_alloc/free

Quoted 129 lines · Martin KaFai Lau

From: Martin KaFai Lau <[email protected]>

This patch uses bpf_mem_cache_alloc/free in bpf_selem_alloc/free.

The ____cacheline_aligned attribute is no longer needed
in 'struct bpf_local_storage_elem'. bpf_mem_cache_alloc will have 'struct llist_node' in front of the 'struct bpf_local_storage_elem'. It will use the 8bytes hole in the bpf_local_storage_elem.

After bpf_mem_cache_alloc(), the SDATA(selem)->data is zero-ed because bpf_mem_cache_alloc() could return a reused selem. It is to keep the existing bpf_map_kzalloc() behavior. Only SDATA(selem)->data is zero-ed. SDATA(selem)->data is the visible part to the bpf prog. No need to use zero_map_value() to do the zeroing because bpf_selem_free() ensures no bpf prog is using the selem before returning the selem through bpf_mem_cache_free(). For the internal fields of selem, they will be initialized when linking to the new smap and the new local_storage.

When bpf_mem_cache_alloc() fails, bpf_selem_alloc() will try to fallback to kzalloc only if the caller has GFP_KERNEL flag set (ie. from sleepable bpf prog that should not cause deadlock). BPF_MA_SIZE and BPF_MA_PTR macro are added for that.

For the common selem free path where the selem is freed when its owner is also being freed, reuse_now == true and selem can be reused immediately. bpf_selem_free() uses bpf_mem_cache_free() where selem will be considered for immediate reuse.

For the uncommon path that the bpf prog explicitly deletes the selem (by using the helper bpf_*_storage_delete), the selem cannot be reused immediately. reuse_now == false and bpf_selem_free() will stay with the current call_rcu_tasks_trace. BPF_MA_NODE macro is added to get the correct address for the kfree.

mem_charge and mem_uncharge are changed to use the BPF_MA_SIZE macro. It will have a temporarily over-charge for the bpf_local_storage_alloc() because bpf_local_storage is not moved to bpf_mem_cache_alloc in this patch but it will be done in the next patch.

---
 include/linux/bpf_local_storage.h |  8 ++---
 include/linux/bpf_mem_alloc.h     |  5 +++
 kernel/bpf/bpf_local_storage.c    | 56 +++++++++++++++++++++++++------
 3 files changed, 53 insertions(+), 16 deletions(-)
include/linux/bpf_local_storage.h +3 -5
index adb5023a1af5..a236c9b964cf 100644
@@ -13,6 +13,7 @@
 #include <linux/list.h>
 #include <linux/hash.h>
 #include <linux/types.h>
+#include <linux/bpf_mem_alloc.h>
 #include <uapi/linux/btf.h>
 
 #define BPF_LOCAL_STORAGE_CACHE_SIZE   16
@@ -55,6 +56,7 @@ struct bpf_local_storage_map {
        u32 bucket_log;
        u16 elem_size;
        u16 cache_idx;
+       struct bpf_mem_alloc selem_ma;
 };
 
 struct bpf_local_storage_data {
@@ -74,11 +76,7 @@ struct bpf_local_storage_elem {
        struct hlist_node snode;        /* Linked to bpf_local_storage */
        struct bpf_local_storage __rcu *local_storage;
        struct rcu_head rcu;
-       /* 8 bytes hole */
-       /* The data is stored in another cacheline to minimize
-        * the number of cachelines access during a cache hit.
-        */
-       struct bpf_local_storage_data sdata ____cacheline_aligned;
+       struct bpf_local_storage_data sdata;
 };
 
 struct bpf_local_storage {
include/linux/bpf_mem_alloc.h +5 -0
index a7104af61ab4..0ab16fb0ab50 100644
@@ -5,6 +5,11 @@
 #include <linux/compiler_types.h>
 #include <linux/workqueue.h>
 
+#define BPF_MA_NODE_SZ sizeof(struct llist_node)
+#define BPF_MA_SIZE(_size) ((_size) + BPF_MA_NODE_SZ)
+#define BPF_MA_PTR(_node) ((void *)(_node) + BPF_MA_NODE_SZ)
+#define BPF_MA_NODE(_ptr) ((void *)(_ptr) - BPF_MA_NODE_SZ)
+
 struct bpf_mem_cache;
 struct bpf_mem_caches;
 
kernel/bpf/bpf_local_storage.c +4 -4
index 532b82084ba7..d3c0dd5737d6 100644
@@ -31,7 +31,7 @@ static int mem_charge(struct bpf_local_storage_map *smap, void *owner, u32 size)
        if (!map->ops->map_local_storage_charge)
                return 0;
 
-       return map->ops->map_local_storage_charge(smap, owner, size);
+       return map->ops->map_local_storage_charge(smap, owner, BPF_MA_SIZE(size));
 }
 
 static void mem_uncharge(struct bpf_local_storage_map *smap, void *owner,
@@ -40,7 +40,7 @@ static void mem_uncharge(struct bpf_local_storage_map *smap, void *owner,
        struct bpf_map *map = &smap->map;
 
        if (map->ops->map_local_storage_uncharge)
-               map->ops->map_local_storage_uncharge(smap, owner, size);
+               map->ops->map_local_storage_uncharge(smap, owner, BPF_MA_SIZE(size));
 }
 
 static struct bpf_local_storage __rcu **
@@ -80,12 +80,32 @@ bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner,
        if (charge_mem && mem_charge(smap, owner, smap->elem_size))
                return NULL;
 
-       selem = bpf_map_kzalloc(&smap->map, smap->elem_size,
-                               gfp_flags | __GFP_NOWARN);
+       migrate_disable();
+       selem = bpf_mem_cache_alloc(&smap->selem_ma);

kernel/bpf/bpf_local_storage.c bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner, · line 94

Quoted 10 lines
kernel/bpf/bpf_local_storage.c +10 -0
+       migrate_enable();
+       if (!selem && (gfp_flags & GFP_KERNEL)) {
+               void *ma_node;
+
+               ma_node = bpf_map_kzalloc(&smap->map,
+                                         BPF_MA_SIZE(smap->elem_size),
+                                         gfp_flags | __GFP_NOWARN);
+               if (ma_node)
+                       selem = BPF_MA_PTR(ma_node);
+       }

If I understand it correctly the code is not trying
to free selem the same way it allocated it.
So we can have kzalloc-ed selems freed into bpf_mem_cache_alloc free-list. That feels dangerous.
I don't think we can do such things in local storage,
but if we add this api to bpf_mem_alloc it might be acceptable. I mean mem alloc will try to take from the free list and if empty and GFP_KERNEL it will kzalloc it.
The knowledge of hidden llist_node shouldn't leave the bpf/memalloc.c file. reuse_now should probably be a memalloc api flag too.
The implementation detail that it's scary but ok-ish to kfree or bpf_mem_cache_free depending on circumstances should stay in memalloc.c

author reviewer
#16 lore raw

Re: [PATCH bpf-next 12/16] bpf: Use bpf_mem_cache_alloc/free in bpf_selem_alloc/free

Quoted 23 lines · Alexei Starovoitov
Quoted 8 lines
kernel/bpf/bpf_local_storage.c +2 -2
@@ -80,12 +80,32 @@ bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner,
         if (charge_mem && mem_charge(smap, owner, smap->elem_size))
                 return NULL;
 
-       selem = bpf_map_kzalloc(&smap->map, smap->elem_size,
-                               gfp_flags | __GFP_NOWARN);
+       migrate_disable();
+       selem = bpf_mem_cache_alloc(&smap->selem_ma);

kernel/bpf/bpf_local_storage.c bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner, · line 94

Quoted 10 lines
kernel/bpf/bpf_local_storage.c +10 -0
+       migrate_enable();
+       if (!selem && (gfp_flags & GFP_KERNEL)) {
+               void *ma_node;
+
+               ma_node = bpf_map_kzalloc(&smap->map,
+                                         BPF_MA_SIZE(smap->elem_size),
+                                         gfp_flags | __GFP_NOWARN);
+               if (ma_node)
+                       selem = BPF_MA_PTR(ma_node);
+       }

If I understand it correctly the code is not trying
to free selem the same way it allocated it.
So we can have kzalloc-ed selems freed into bpf_mem_cache_alloc free-list. That feels dangerous.

Quoted 8 lines

I don't think we can do such things in local storage,
but if we add this api to bpf_mem_alloc it might be acceptable. I mean mem alloc will try to take from the free list and if empty and GFP_KERNEL it will kzalloc it.
The knowledge of hidden llist_node shouldn't leave the bpf/memalloc.c file. reuse_now should probably be a memalloc api flag too.
The implementation detail that it's scary but ok-ish to kfree or bpf_mem_cache_free depending on circumstances should stay in memalloc.c

All make sense. I will create a bpf_mem_cache_alloc_flags(..., gfp_t flags) to hide the llist_node and kzalloc details. For free, local storage still needs to use the selem->rcu head in its call_rcu_tasks_trace(), so I will create a bpf_mem_cache_raw_free(void *ptr) to hide the llist_node details, like:

/* 'struct bpf_mem_alloc *ma' is not available at this
* point but the caller knows it is percpu or not and
* call different raw_free function.
*/
void bpf_mem_cache_raw_free(void *ptr)

{
        kfree(ptr - LLIST_NODE_SZ);
}
author reviewer
#17 lore raw

PATCH 13/16 bpf: Use bpf_mem_cache_alloc/free for bpf_local_storage

In reply to Martin KaFai Lau

From: Martin KaFai Lau <[email protected]>

This patch uses bpf_mem_cache_alloc/free for allocating and freeing bpf_local_storage.

The changes are similar to the previous patch when switching bpf_local_storage_elem to bpf_mem_cache_alloc/free.

A few things that worth to mention for bpf_local_storage:

The local_storage is deleted when the last selem is deleted. Before deleting a selem from local_storage, it needs to retrieve the local_storage->smap because the bpf_selem_unlink_storage_nolock() may have set it to NULL. Note that local_storage->smap may have already been NULL when the selem created this local_storage has been removed. In this case, call_rcu will be used to free the local_storage.

When bpf_local_storage_alloc getting a reused memory, all fields are either in the correct values or will be initialized. 'cache[]' must already be all NULLs. 'list' must be empty. Others will be initialized.

---
 include/linux/bpf_local_storage.h |  1 +
 kernel/bpf/bpf_local_storage.c    | 54 ++++++++++++++++++++++++++-----
 2 files changed, 47 insertions(+), 8 deletions(-)
include/linux/bpf_local_storage.h +1 -0
index a236c9b964cf..d6d4ec248c00 100644
@@ -57,6 +57,7 @@ struct bpf_local_storage_map {
 	u16 elem_size;
 	u16 cache_idx;
 	struct bpf_mem_alloc selem_ma;
+	struct bpf_mem_alloc storage_ma;
 };
 
 struct bpf_local_storage_data {
kernel/bpf/bpf_local_storage.c +46 -8

130 lines not on this page · raw message

Signature · 1 line
2.30.2
author reviewer
#18 lore raw

PATCH 14/16 selftests/bpf: Replace CHECK with ASSERT in test_local_storage

In reply to Martin KaFai Lau

From: Martin KaFai Lau <[email protected]>

This patch migrates the CHECK macro to ASSERT macro.

---
 .../bpf/prog_tests/test_local_storage.c       | 49 +++++++------------
 1 file changed, 19 insertions(+), 30 deletions(-)
tools/testing/selftests/bpf/prog_tests/test_local_storage.c +19 -30
index 9c77cd6b1eaf..c33f840f4880 100644
@@ -13,8 +13,6 @@
 #include "network_helpers.h"
 #include "task_local_storage_helpers.h"
 
-static unsigned int duration;
-
 #define TEST_STORAGE_VALUE 0xbeefdead
 
 struct storage {
@@ -60,36 +58,32 @@ static bool check_syscall_operations(int map_fd, int obj_fd)
 
 	/* Looking up an existing element should fail initially */
 	err = bpf_map_lookup_elem_flags(map_fd, &obj_fd, &lookup_val, 0);
-	if (CHECK(!err || errno != ENOENT, "bpf_map_lookup_elem",
-		  "err:%d errno:%d\n", err, errno))
+	if (!ASSERT_ERR(err, "bpf_map_lookup_elem") ||
+	    !ASSERT_EQ(errno, ENOENT, "errno"))
 		return false;
 
 	/* Create a new element */
 	err = bpf_map_update_elem(map_fd, &obj_fd, &val, BPF_NOEXIST);
-	if (CHECK(err < 0, "bpf_map_update_elem", "err:%d errno:%d\n", err,
-		  errno))
+	if (!ASSERT_OK(err, "bpf_map_update_elem"))
 		return false;
 
 	/* Lookup the newly created element */
 	err = bpf_map_lookup_elem_flags(map_fd, &obj_fd, &lookup_val, 0);
-	if (CHECK(err < 0, "bpf_map_lookup_elem", "err:%d errno:%d", err,
-		  errno))
+	if (!ASSERT_OK(err, "bpf_map_lookup_elem"))
 		return false;
 
 	/* Check the value of the newly created element */
-	if (CHECK(lookup_val.value != val.value, "bpf_map_lookup_elem",
-		  "value got = %x errno:%d", lookup_val.value, val.value))
+	if (!ASSERT_EQ(lookup_val.value, val.value, "bpf_map_lookup_elem"))
 		return false;
 
 	err = bpf_map_delete_elem(map_fd, &obj_fd);
-	if (CHECK(err, "bpf_map_delete_elem()", "err:%d errno:%d\n", err,
-		  errno))
+	if (!ASSERT_OK(err, "bpf_map_delete_elem()"))
 		return false;
 
 	/* The lookup should fail, now that the element has been deleted */
 	err = bpf_map_lookup_elem_flags(map_fd, &obj_fd, &lookup_val, 0);
-	if (CHECK(!err || errno != ENOENT, "bpf_map_lookup_elem",
-		  "err:%d errno:%d\n", err, errno))
+	if (!ASSERT_ERR(err, "bpf_map_lookup_elem") ||
+	    !ASSERT_EQ(errno, ENOENT, "errno"))
 		return false;
 
 	return true;
@@ -104,35 +98,32 @@ void test_test_local_storage(void)
 	char cmd[256];
 
 	skel = local_storage__open_and_load();
-	if (CHECK(!skel, "skel_load", "lsm skeleton failed\n"))
+	if (!ASSERT_OK_PTR(skel, "skel_load"))
 		goto close_prog;
 
 	err = local_storage__attach(skel);
-	if (CHECK(err, "attach", "lsm attach failed: %d\n", err))
+	if (!ASSERT_OK(err, "attach"))
 		goto close_prog;
 
 	task_fd = sys_pidfd_open(getpid(), 0);
-	if (CHECK(task_fd < 0, "pidfd_open",
-		  "failed to get pidfd err:%d, errno:%d", task_fd, errno))
+	if (!ASSERT_GE(task_fd, 0, "pidfd_open"))
 		goto close_prog;
 
 	if (!check_syscall_operations(bpf_map__fd(skel->maps.task_storage_map),
 				      task_fd))
 		goto close_prog;
 
-	if (CHECK(!mkdtemp(tmp_dir_path), "mkdtemp",
-		  "unable to create tmpdir: %d\n", errno))
+	if (!ASSERT_OK_PTR(mkdtemp(tmp_dir_path), "mkdtemp"))
 		goto close_prog;
 
 	snprintf(tmp_exec_path, sizeof(tmp_exec_path), "%s/copy_of_rm",
 		 tmp_dir_path);
 	snprintf(cmd, sizeof(cmd), "cp /bin/rm %s", tmp_exec_path);
-	if (CHECK_FAIL(system(cmd)))
+	if (!ASSERT_OK(system(cmd), "system(cp)"))
 		goto close_prog_rmdir;
 
 	rm_fd = open(tmp_exec_path, O_RDONLY);
-	if (CHECK(rm_fd < 0, "open", "failed to open %s err:%d, errno:%d",
-		  tmp_exec_path, rm_fd, errno))
+	if (!ASSERT_GE(rm_fd, 0, "open(tmp_exec_path)"))
 		goto close_prog_rmdir;
 
 	if (!check_syscall_operations(bpf_map__fd(skel->maps.inode_storage_map),
@@ -145,7 +136,7 @@ void test_test_local_storage(void)
 	 * LSM program.
 	 */
 	err = run_self_unlink(&skel->bss->monitored_pid, tmp_exec_path);
-	if (CHECK(err != EPERM, "run_self_unlink", "err %d want EPERM\n", err))
+	if (!ASSERT_EQ(err, EPERM, "run_self_unlink"))
 		goto close_prog_rmdir;
 
 	/* Set the process being monitored to be the current process */
@@ -156,18 +147,16 @@ void test_test_local_storage(void)
 	 */
 	snprintf(cmd, sizeof(cmd), "mv %s/copy_of_rm %s/check_null_ptr",
 		 tmp_dir_path, tmp_dir_path);
-	if (CHECK_FAIL(system(cmd)))
+	if (!ASSERT_OK(system(cmd), "system(mv)"))
 		goto close_prog_rmdir;
 
-	CHECK(skel->data->inode_storage_result != 0, "inode_storage_result",
-	      "inode_local_storage not set\n");
+	ASSERT_EQ(skel->data->inode_storage_result, 0, "inode_storage_result");
 
 	serv_sk = start_server(AF_INET6, SOCK_STREAM, NULL, 0, 0);
-	if (CHECK(serv_sk < 0, "start_server", "failed to start server\n"))
+	if (!ASSERT_GE(serv_sk, 0, "start_server"))
 		goto close_prog_rmdir;
 
-	CHECK(skel->data->sk_storage_result != 0, "sk_storage_result",
-	      "sk_local_storage not set\n");
+	ASSERT_EQ(skel->data->sk_storage_result, 0, "sk_storage_result");
 
 	if (!check_syscall_operations(bpf_map__fd(skel->maps.sk_storage_map),
 				      serv_sk))
Signature · 1 line
2.30.2
#19 lore raw

Re: [PATCH bpf-next 14/16] selftests/bpf: Replace CHECK with ASSERT in test_local_storage

Quoted 7 lines · Martin KaFai Lau

From: Martin KaFai Lau <[email protected]>

This patch migrates the CHECK macro to ASSERT macro.

---

Thanks for the cleanup!

Quoted 17 lines
 .../bpf/prog_tests/test_local_storage.c       | 49 +++++++------------
 1 file changed, 19 insertions(+), 30 deletions(-)
tools/testing/selftests/bpf/prog_tests/test_local_storage.c +0 -2
index 9c77cd6b1eaf..c33f840f4880 100644
@@ -13,8 +13,6 @@
 #include "network_helpers.h"
 #include "task_local_storage_helpers.h"
 
-static unsigned int duration;
-
 #define TEST_STORAGE_VALUE 0xbeefdead
 
 struct storage {

tools/testing/selftests/bpf/prog_tests/test_local_storage.c static bool check_syscall_operations(int map_fd, int obj_fd) · line 62

Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/test_local_storage.c +2 -2
@@ -60,36 +58,32 @@ static bool check_syscall_operations(int map_fd, int obj_fd)
 
        /* Looking up an existing element should fail initially */
        err = bpf_map_lookup_elem_flags(map_fd, &obj_fd, &lookup_val, 0);
-       if (CHECK(!err || errno != ENOENT, "bpf_map_lookup_elem",
-                 "err:%d errno:%d\n", err, errno))
+       if (!ASSERT_ERR(err, "bpf_map_lookup_elem") ||
+           !ASSERT_EQ(errno, ENOENT, "errno"))

all libbpf APIs since v1.0 always return actual error number directly, so no need to check errno anymore, you can simplify this further

Quoted 24 lines
tools/testing/selftests/bpf/prog_tests/test_local_storage.c +3 -7
                return false;
 
        /* Create a new element */
        err = bpf_map_update_elem(map_fd, &obj_fd, &val, BPF_NOEXIST);
-       if (CHECK(err < 0, "bpf_map_update_elem", "err:%d errno:%d\n", err,
-                 errno))
+       if (!ASSERT_OK(err, "bpf_map_update_elem"))
                return false;
 
        /* Lookup the newly created element */
        err = bpf_map_lookup_elem_flags(map_fd, &obj_fd, &lookup_val, 0);
-       if (CHECK(err < 0, "bpf_map_lookup_elem", "err:%d errno:%d", err,
-                 errno))
+       if (!ASSERT_OK(err, "bpf_map_lookup_elem"))
                return false;
 
        /* Check the value of the newly created element */
-       if (CHECK(lookup_val.value != val.value, "bpf_map_lookup_elem",
-                 "value got = %x errno:%d", lookup_val.value, val.value))
+       if (!ASSERT_EQ(lookup_val.value, val.value, "bpf_map_lookup_elem"))
                return false;
 
        err = bpf_map_delete_elem(map_fd, &obj_fd);
-       if (CHECK(err, "bpf_map_delete_elem()", "err:%d errno:%d\n", err,

tools/testing/selftests/bpf/prog_tests/test_local_storage.c static bool check_syscall_operations(int map_fd, int obj_fd) · line 86

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/test_local_storage.c +3 -3
-                 errno))
+       if (!ASSERT_OK(err, "bpf_map_delete_elem()"))
                return false;
 
        /* The lookup should fail, now that the element has been deleted */
        err = bpf_map_lookup_elem_flags(map_fd, &obj_fd, &lookup_val, 0);
-       if (CHECK(!err || errno != ENOENT, "bpf_map_lookup_elem",
-                 "err:%d errno:%d\n", err, errno))
+       if (!ASSERT_ERR(err, "bpf_map_lookup_elem") ||
+           !ASSERT_EQ(errno, ENOENT, "errno"))

same here and probably in other places (I haven't checked everything)

Quoted 80 lines
tools/testing/selftests/bpf/prog_tests/test_local_storage.c +11 -17
                return false;
 
        return true;
@@ -104,35 +98,32 @@ void test_test_local_storage(void)
        char cmd[256];
 
        skel = local_storage__open_and_load();
-       if (CHECK(!skel, "skel_load", "lsm skeleton failed\n"))
+       if (!ASSERT_OK_PTR(skel, "skel_load"))
                goto close_prog;
 
        err = local_storage__attach(skel);
-       if (CHECK(err, "attach", "lsm attach failed: %d\n", err))
+       if (!ASSERT_OK(err, "attach"))
                goto close_prog;
 
        task_fd = sys_pidfd_open(getpid(), 0);
-       if (CHECK(task_fd < 0, "pidfd_open",
-                 "failed to get pidfd err:%d, errno:%d", task_fd, errno))
+       if (!ASSERT_GE(task_fd, 0, "pidfd_open"))
                goto close_prog;
 
        if (!check_syscall_operations(bpf_map__fd(skel->maps.task_storage_map),
                                      task_fd))
                goto close_prog;
 
-       if (CHECK(!mkdtemp(tmp_dir_path), "mkdtemp",
-                 "unable to create tmpdir: %d\n", errno))
+       if (!ASSERT_OK_PTR(mkdtemp(tmp_dir_path), "mkdtemp"))
                goto close_prog;
 
        snprintf(tmp_exec_path, sizeof(tmp_exec_path), "%s/copy_of_rm",
                 tmp_dir_path);
        snprintf(cmd, sizeof(cmd), "cp /bin/rm %s", tmp_exec_path);
-       if (CHECK_FAIL(system(cmd)))
+       if (!ASSERT_OK(system(cmd), "system(cp)"))
                goto close_prog_rmdir;
 
        rm_fd = open(tmp_exec_path, O_RDONLY);
-       if (CHECK(rm_fd < 0, "open", "failed to open %s err:%d, errno:%d",
-                 tmp_exec_path, rm_fd, errno))
+       if (!ASSERT_GE(rm_fd, 0, "open(tmp_exec_path)"))
                goto close_prog_rmdir;
 
        if (!check_syscall_operations(bpf_map__fd(skel->maps.inode_storage_map),
@@ -145,7 +136,7 @@ void test_test_local_storage(void)
         * LSM program.
         */
        err = run_self_unlink(&skel->bss->monitored_pid, tmp_exec_path);
-       if (CHECK(err != EPERM, "run_self_unlink", "err %d want EPERM\n", err))
+       if (!ASSERT_EQ(err, EPERM, "run_self_unlink"))
                goto close_prog_rmdir;
 
        /* Set the process being monitored to be the current process */
@@ -156,18 +147,16 @@ void test_test_local_storage(void)
         */
        snprintf(cmd, sizeof(cmd), "mv %s/copy_of_rm %s/check_null_ptr",
                 tmp_dir_path, tmp_dir_path);
-       if (CHECK_FAIL(system(cmd)))
+       if (!ASSERT_OK(system(cmd), "system(mv)"))
                goto close_prog_rmdir;
 
-       CHECK(skel->data->inode_storage_result != 0, "inode_storage_result",
-             "inode_local_storage not set\n");
+       ASSERT_EQ(skel->data->inode_storage_result, 0, "inode_storage_result");
 
        serv_sk = start_server(AF_INET6, SOCK_STREAM, NULL, 0, 0);
-       if (CHECK(serv_sk < 0, "start_server", "failed to start server\n"))
+       if (!ASSERT_GE(serv_sk, 0, "start_server"))
                goto close_prog_rmdir;
 
-       CHECK(skel->data->sk_storage_result != 0, "sk_storage_result",
-             "sk_local_storage not set\n");
+       ASSERT_EQ(skel->data->sk_storage_result, 0, "sk_storage_result");
 
        if (!check_syscall_operations(bpf_map__fd(skel->maps.sk_storage_map),
                                      serv_sk))
--

2.30.2

author reviewer
#20 lore raw

Re: [PATCH bpf-next 14/16] selftests/bpf: Replace CHECK with ASSERT in test_local_storage

Quoted 40 lines · Andrii Nakryiko

tools/testing/selftests/bpf/prog_tests/test_local_storage.c static bool check_syscall_operations(int map_fd, int obj_fd) · line 62

Quoted 8 lines
tools/testing/selftests/bpf/prog_tests/test_local_storage.c +2 -2
@@ -60,36 +58,32 @@ static bool check_syscall_operations(int map_fd, int obj_fd)
 
         /* Looking up an existing element should fail initially */
         err = bpf_map_lookup_elem_flags(map_fd, &obj_fd, &lookup_val, 0);
-       if (CHECK(!err || errno != ENOENT, "bpf_map_lookup_elem",
-                 "err:%d errno:%d\n", err, errno))
+       if (!ASSERT_ERR(err, "bpf_map_lookup_elem") ||
+           !ASSERT_EQ(errno, ENOENT, "errno"))

all libbpf APIs since v1.0 always return actual error number directly, so no need to check errno anymore, you can simplify this further

Quoted 28 lines
tools/testing/selftests/bpf/prog_tests/test_local_storage.c +4 -8
                 return false;
 
         /* Create a new element */
         err = bpf_map_update_elem(map_fd, &obj_fd, &val, BPF_NOEXIST);
-       if (CHECK(err < 0, "bpf_map_update_elem", "err:%d errno:%d\n", err,
-                 errno))
+       if (!ASSERT_OK(err, "bpf_map_update_elem"))
                 return false;
 
         /* Lookup the newly created element */
         err = bpf_map_lookup_elem_flags(map_fd, &obj_fd, &lookup_val, 0);
-       if (CHECK(err < 0, "bpf_map_lookup_elem", "err:%d errno:%d", err,
-                 errno))
+       if (!ASSERT_OK(err, "bpf_map_lookup_elem"))
                 return false;
 
         /* Check the value of the newly created element */
-       if (CHECK(lookup_val.value != val.value, "bpf_map_lookup_elem",
-                 "value got = %x errno:%d", lookup_val.value, val.value))
+       if (!ASSERT_EQ(lookup_val.value, val.value, "bpf_map_lookup_elem"))
                 return false;
 
         err = bpf_map_delete_elem(map_fd, &obj_fd);
-       if (CHECK(err, "bpf_map_delete_elem()", "err:%d errno:%d\n", err,
-                 errno))
+       if (!ASSERT_OK(err, "bpf_map_delete_elem()"))
                 return false;
 
Quoted 8 lines

tools/testing/selftests/bpf/prog_tests/test_local_storage.c static bool check_syscall_operations(int map_fd, int obj_fd) · line 86

Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/test_local_storage.c +2 -2
         /* The lookup should fail, now that the element has been deleted */
         err = bpf_map_lookup_elem_flags(map_fd, &obj_fd, &lookup_val, 0);
-       if (CHECK(!err || errno != ENOENT, "bpf_map_lookup_elem",
-                 "err:%d errno:%d\n", err, errno))
+       if (!ASSERT_ERR(err, "bpf_map_lookup_elem") ||
+           !ASSERT_EQ(errno, ENOENT, "errno"))

same here and probably in other places (I haven't checked everything)

Ack. will simplify.

author reviewer
#21 lore raw

PATCH 15/16 selftests/bpf: Check freeing sk->sk_local_storage with sk_local_storage->smap is NULL

In reply to Martin KaFai Lau

From: Martin KaFai Lau <[email protected]>

This patch tweats the socket_bind bpf prog to test the local_storage->smap == NULL case in the bpf_local_storage_free() code path. The idea is to create the local_storage with
the sk_storage_map's selem first. Then add the sk_storage_map2's selem and then delete the earlier sk_storeage_map's selem.

---
 .../selftests/bpf/progs/local_storage.c       | 29 +++++++++++++------
 1 file changed, 20 insertions(+), 9 deletions(-)
tools/testing/selftests/bpf/progs/local_storage.c +20 -9
index 19423ed862e3..797c81655a47 100644
@@ -109,18 +109,17 @@ int BPF_PROG(socket_bind, struct socket *sock, struct sockaddr *address,
 {
 	__u32 pid = bpf_get_current_pid_tgid() >> 32;
 	struct local_storage *storage;
-	int err;
 
 	if (pid != monitored_pid)
 		return 0;
 
-	storage = bpf_sk_storage_get(&sk_storage_map, sock->sk, 0,
-				     BPF_LOCAL_STORAGE_GET_F_CREATE);
+	storage = bpf_sk_storage_get(&sk_storage_map, sock->sk, 0, 0);
 	if (!storage)
 		return 0;
 
+	sk_storage_result = -1;
 	if (storage->value != DUMMY_STORAGE_VALUE)
-		sk_storage_result = -1;
+		return 0;
 
 	/* This tests that we can associate multiple elements
 	 * with the local storage.
@@ -130,14 +129,26 @@ int BPF_PROG(socket_bind, struct socket *sock, struct sockaddr *address,
 	if (!storage)
 		return 0;
 
-	err = bpf_sk_storage_delete(&sk_storage_map, sock->sk);
-	if (err)
+	if (bpf_sk_storage_delete(&sk_storage_map2, sock->sk))
 		return 0;
 
-	err = bpf_sk_storage_delete(&sk_storage_map2, sock->sk);
-	if (!err)
-		sk_storage_result = err;
+	storage = bpf_sk_storage_get(&sk_storage_map2, sock->sk, 0,
+				     BPF_LOCAL_STORAGE_GET_F_CREATE);
+	if (!storage)
+		return 0;
+
+	if (bpf_sk_storage_delete(&sk_storage_map, sock->sk))
+		return 0;
+
+	/* Ensure that the sk_storage_map is disconnected from the storage.
+	 * The storage memory should not be freed back to the
+	 * bpf_mem_alloc of the sk_bpf_storage_map because
+	 * sk_bpf_storage_map may have been gone.
+	 */
+	if (!sock->sk->sk_bpf_storage || sock->sk->sk_bpf_storage->smap)
+		return 0;
 
+	sk_storage_result = 0;
 	return 0;
 }
 
Signature · 1 line
2.30.2
author reviewer
#22 lore raw

PATCH 16/16 selftests/bpf: Add local-storage-create benchmark

In reply to Martin KaFai Lau

From: Martin KaFai Lau <[email protected]>

This patch tests how many kmallocs is needed to create and free a batch of UDP sockets and each socket has a 64bytes bpf storage. It also measures how fast the UDP sockets can be created.

The result is from my qemu setup.

Before bpf_mem_alloc/free:
./bench -p 1 local-storage-create
Setting up benchmark 'local-storage-create'...
Benchmark 'local-storage-create' started.
Iter 0 ( 73.193us): creates 213.552k/s (213.552k/prod), 3.09 kmallocs/create
Iter 1 (-20.724us): creates 211.908k/s (211.908k/prod), 3.09 kmallocs/create
Iter 2 ( 9.280us): creates 212.574k/s (212.574k/prod), 3.12 kmallocs/create
Iter 3 ( 11.039us): creates 213.209k/s (213.209k/prod), 3.12 kmallocs/create
Iter 4 (-11.411us): creates 213.351k/s (213.351k/prod), 3.12 kmallocs/create
Iter 5 ( -7.915us): creates 214.754k/s (214.754k/prod), 3.12 kmallocs/create
Iter 6 ( 11.317us): creates 210.942k/s (210.942k/prod), 3.12 kmallocs/create
Summary: creates 212.789 ± 1.310k/s (212.789k/prod), 3.12 kmallocs/create

After bpf_mem_alloc/free:
./bench -p 1 local-storage-create
Setting up benchmark 'local-storage-create'...
Benchmark 'local-storage-create' started.
Iter 0 ( 68.265us): creates 243.984k/s (243.984k/prod), 1.04 kmallocs/create
Iter 1 ( 30.357us): creates 238.424k/s (238.424k/prod), 1.04 kmallocs/create
Iter 2 (-18.712us): creates 232.963k/s (232.963k/prod), 1.04 kmallocs/create
Iter 3 (-15.885us): creates 238.879k/s (238.879k/prod), 1.04 kmallocs/create
Iter 4 ( 5.590us): creates 237.490k/s (237.490k/prod), 1.04 kmallocs/create
Iter 5 ( 8.577us): creates 237.521k/s (237.521k/prod), 1.04 kmallocs/create
Iter 6 ( -6.263us): creates 238.508k/s (238.508k/prod), 1.04 kmallocs/create
Summary: creates 237.298 ± 2.198k/s (237.298k/prod), 1.04 kmallocs/create

---
 tools/testing/selftests/bpf/Makefile          |   2 +
 tools/testing/selftests/bpf/bench.c           |   2 +
 .../bpf/benchs/bench_local_storage_create.c   | 141 ++++++++++++++++++
 .../bpf/progs/bench_local_storage_create.c    |  57 +++++++
 4 files changed, 202 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/benchs/bench_local_storage_create.c
 create mode 100644 tools/testing/selftests/bpf/progs/bench_local_storage_create.c
tools/testing/selftests/bpf/Makefile +2 -0
index 16f404aa1b23..545fff0a6dfc 100644
@@ -639,6 +639,7 @@ $(OUTPUT)/bench_strncmp.o: $(OUTPUT)/strncmp_bench.skel.h
 $(OUTPUT)/bench_bpf_hashmap_full_update.o: $(OUTPUT)/bpf_hashmap_full_update_bench.skel.h
 $(OUTPUT)/bench_local_storage.o: $(OUTPUT)/local_storage_bench.skel.h
 $(OUTPUT)/bench_local_storage_rcu_tasks_trace.o: $(OUTPUT)/local_storage_rcu_tasks_trace_bench.skel.h
+$(OUTPUT)/bench_local_storage_create.o: $(OUTPUT)/bench_local_storage_create.skel.h
 $(OUTPUT)/bench_bpf_hashmap_lookup.o: $(OUTPUT)/bpf_hashmap_lookup.skel.h
 $(OUTPUT)/bench.o: bench.h testing_helpers.h $(BPFOBJ)
 $(OUTPUT)/bench: LDLIBS += -lm
@@ -656,6 +657,7 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \
 		 $(OUTPUT)/bench_local_storage.o \
 		 $(OUTPUT)/bench_local_storage_rcu_tasks_trace.o \
 		 $(OUTPUT)/bench_bpf_hashmap_lookup.o \
+		 $(OUTPUT)/bench_local_storage_create.o \
 		 #
 	$(call msg,BINARY,,$@)
 	$(Q)$(CC) $(CFLAGS) $(LDFLAGS) $(filter %.a %.o,$^) $(LDLIBS) -o $@
tools/testing/selftests/bpf/bench.c +2 -0
index 0b2a53bb8460..dc3827c1f139 100644
@@ -515,6 +515,7 @@ extern const struct bench bench_local_storage_cache_interleaved_get;
 extern const struct bench bench_local_storage_cache_hashmap_control;
 extern const struct bench bench_local_storage_tasks_trace;
 extern const struct bench bench_bpf_hashmap_lookup;
+extern const struct bench bench_local_storage_create;
 
 static const struct bench *benchs[] = {
 	&bench_count_global,
@@ -555,6 +556,7 @@ static const struct bench *benchs[] = {
 	&bench_local_storage_cache_hashmap_control,
 	&bench_local_storage_tasks_trace,
 	&bench_bpf_hashmap_lookup,
+	&bench_local_storage_create,
 };
 
 static void find_benchmark(void)
tools/testing/selftests/bpf/benchs/bench_local_storage_create.c new file +141 -0

141 lines not on this page · raw message

tools/testing/selftests/bpf/progs/bench_local_storage_create.c new file +57 -0

57 lines not on this page · raw message

Signature · 1 line
2.30.2