Skip to contentSkip to conversation

bpf: add percpu stats for bpf_map

PATCH Superseded Anton Protopopov 3 participants 20 messages Jun 30, 2023 – Jul 6, 2023

Subsystems (5)
Kselftest BPF BPF selftests BPF core +1 more subsystems

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v3 cover bpf: add percpu stats for bpf_map

This series adds a mechanism for maps to populate per-cpu counters on insertions/deletions. The sum of these counters can be accessed by a new kfunc from map iterator and tracing programs.

The following patches are present in the series:

* Patch 1 adds a generic per-cpu counter to struct bpf_map
* Patch 2 adds a new kfunc to access the sum of per-cpu counters
* Patch 3 utilizes this mechanism for hash-based maps
* Patch 4 extends the preloaded map iterator to dump the sum
* Patch 5 adds a self-test for the change
* Patch 6 patches map_ptr selftest to check that elem_count was initialized

The reason for adding this functionality in our case (Cilium) is to get signals about how full some heavy-used maps are and what the actual dynamic profile of map capacity is. In the case of LRU maps this is impossible to get this information anyhow else. The original presentation can be found here [1].

[1] https://lpc.events/event/16/contributions/1368/

v2 -> v3:
- split commits to better represent update logic
- remove filter from kfunc to allow all tracing programs
- extend selftests

v1 -> v2:
- make the counters generic part of struct bpf_map
- don't use map_info and /proc/self/fdinfo in favor of a kfunc

Anton Protopopov (6):
bpf: add percpu stats for bpf_map elements insertions/deletions
bpf: add a new kfunc to return current bpf_map elements count
bpf: populate the per-cpu insertions/deletions counters for hashmaps
bpf: make preloaded map iterators to display map elements count selftests/bpf: test map percpu stats
selftests/bpf: check that ->elem_count is non-zero for the hash map

 include/linux/bpf.h                           |  30 +
 kernel/bpf/hashtab.c                          |  23 +-
 kernel/bpf/map_iter.c                         |  39 +-
 kernel/bpf/preload/iterators/iterators.bpf.c  |   9 +-
 .../iterators/iterators.lskel-little-endian.h | 526 +++++++++---------
 .../bpf/map_tests/map_percpu_stats.c          | 336 +++++++++++
 .../selftests/bpf/progs/map_percpu_stats.c    |  24 +
 .../selftests/bpf/progs/map_ptr_kern.c        |   3 +
 8 files changed, 726 insertions(+), 264 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/map_tests/map_percpu_stats.c
 create mode 100644 tools/testing/selftests/bpf/progs/map_percpu_stats.c
Signature · 1 line
2.34.1
#1 lore raw

PATCH v3 1/6 bpf: add percpu stats for bpf_map elements insertions/deletions

Add a generic percpu stats for bpf_map elements insertions/deletions in order to keep track of both, the current (approximate) number of elements in a map and per-cpu statistics on update/delete operations.

To expose these stats a particular map implementation should initialize the counter and adjust it as needed using the 'bpf_map_*_elem_count' helpers provided by this commit.

---
 include/linux/bpf.h | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
include/linux/bpf.h +30 -0
index f58895830ada..360433f14496 100644
@@ -275,6 +275,7 @@ struct bpf_map {
 	} owner;
 	bool bypass_spec_v1;
 	bool frozen; /* write-once; write-protected by freeze_mutex */
+	s64 __percpu *elem_count;
 };
 
 static inline const char *btf_field_type_name(enum btf_field_type type)
@@ -2040,6 +2041,35 @@ bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, size_t align,
 }
 #endif
 
+static inline int
+bpf_map_init_elem_count(struct bpf_map *map)
+{
+	size_t size = sizeof(*map->elem_count), align = size;
+	gfp_t flags = GFP_USER | __GFP_NOWARN;
+
+	map->elem_count = bpf_map_alloc_percpu(map, size, align, flags);
+	if (!map->elem_count)
+		return -ENOMEM;
+
+	return 0;
+}
+
+static inline void
+bpf_map_free_elem_count(struct bpf_map *map)
+{
+	free_percpu(map->elem_count);
+}
+
+static inline void bpf_map_inc_elem_count(struct bpf_map *map)
+{
+	this_cpu_inc(*map->elem_count);
+}
+
+static inline void bpf_map_dec_elem_count(struct bpf_map *map)
+{
+	this_cpu_dec(*map->elem_count);
+}
+
 extern int sysctl_unprivileged_bpf_disabled;
 
 static inline bool bpf_allow_ptr_leaks(void)
Signature · 1 line
2.34.1
#2 lore raw

PATCH v3 2/6 bpf: add a new kfunc to return current bpf_map elements count

In reply to Anton Protopopov

A bpf_map_sum_elem_count kfunc was added to simplify getting the sum of the map per-cpu element counters. If a map doesn't implement the counter, then the function will always return 0.

---
 kernel/bpf/map_iter.c | 39 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 38 insertions(+), 1 deletion(-)
kernel/bpf/map_iter.c +38 -1
index b0fa190b0979..d06d3b7150e5 100644
@@ -93,7 +93,7 @@ static struct bpf_iter_reg bpf_map_reg_info = {
 	.ctx_arg_info_size	= 1,
 	.ctx_arg_info		= {
 		{ offsetof(struct bpf_iter__bpf_map, map),
-		  PTR_TO_BTF_ID_OR_NULL },
+		  PTR_TO_BTF_ID_OR_NULL | PTR_TRUSTED },
 	},
 	.seq_info		= &bpf_map_seq_info,
 };
@@ -193,3 +193,40 @@ static int __init bpf_map_iter_init(void)
 }
 
 late_initcall(bpf_map_iter_init);
+
+__diag_push();
+__diag_ignore_all("-Wmissing-prototypes",
+		  "Global functions as their definitions will be in vmlinux BTF");
+
+__bpf_kfunc s64 bpf_map_sum_elem_count(struct bpf_map *map)
+{
+	s64 *pcount;
+	s64 ret = 0;
+	int cpu;
+
+	if (!map || !map->elem_count)
+		return 0;
+
+	for_each_possible_cpu(cpu) {
+		pcount = per_cpu_ptr(map->elem_count, cpu);
+		ret += READ_ONCE(*pcount);
+	}
+	return ret;
+}
+
+__diag_pop();
+
+BTF_SET8_START(bpf_map_iter_kfunc_ids)
+BTF_ID_FLAGS(func, bpf_map_sum_elem_count, KF_TRUSTED_ARGS)
+BTF_SET8_END(bpf_map_iter_kfunc_ids)
+
+static const struct btf_kfunc_id_set bpf_map_iter_kfunc_set = {
+	.owner = THIS_MODULE,
+	.set   = &bpf_map_iter_kfunc_ids,
+};
+
+static int init_subsystem(void)
+{
+	return register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &bpf_map_iter_kfunc_set);
+}
+late_initcall(init_subsystem);
Signature · 1 line
2.34.1
#3 lore raw

PATCH v3 3/6 bpf: populate the per-cpu insertions/deletions counters for hashmaps

In reply to Anton Protopopov

Initialize and utilize the per-cpu insertions/deletions counters for hash-based maps. Non-trivial changes only apply to the preallocated maps for which the {inc,dec}_elem_count functions are not called, as there's no need in counting elements to sustain proper map operations.

To increase/decrease percpu counters for preallocated maps we add raw calls to the bpf_map_{inc,dec}_elem_count functions so that the impact is minimal. For dynamically allocated maps we add corresponding calls to the existing {inc,dec}_elem_count functions.

---
 kernel/bpf/hashtab.c | 23 ++++++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)
kernel/bpf/hashtab.c +20 -3
index 56d3da7d0bc6..faaef4fd3df0 100644
@@ -581,8 +581,14 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr)
 		}
 	}
 
+	err = bpf_map_init_elem_count(&htab->map);
+	if (err)
+		goto free_extra_elements;
+
 	return &htab->map;
 
+free_extra_elements:
+	free_percpu(htab->extra_elems);
 free_prealloc:
 	prealloc_destroy(htab);
 free_map_locked:
@@ -804,6 +810,7 @@ static bool htab_lru_map_delete_node(void *arg, struct bpf_lru_node *node)
 		if (l == tgt_l) {
 			hlist_nulls_del_rcu(&l->hash_node);
 			check_and_free_fields(htab, l);
+			bpf_map_dec_elem_count(&htab->map);
 			break;
 		}
 
@@ -900,6 +907,8 @@ static bool is_map_full(struct bpf_htab *htab)
 
 static void inc_elem_count(struct bpf_htab *htab)
 {
+	bpf_map_inc_elem_count(&htab->map);
+
 	if (htab->use_percpu_counter)
 		percpu_counter_add_batch(&htab->pcount, 1, PERCPU_COUNTER_BATCH);
 	else
@@ -908,6 +917,8 @@ static void inc_elem_count(struct bpf_htab *htab)
 
 static void dec_elem_count(struct bpf_htab *htab)
 {
+	bpf_map_dec_elem_count(&htab->map);
+
 	if (htab->use_percpu_counter)
 		percpu_counter_add_batch(&htab->pcount, -1, PERCPU_COUNTER_BATCH);
 	else
@@ -920,6 +931,7 @@ static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l)
 	htab_put_fd_value(htab, l);
 
 	if (htab_is_prealloc(htab)) {
+		bpf_map_dec_elem_count(&htab->map);
 		check_and_free_fields(htab, l);
 		__pcpu_freelist_push(&htab->freelist, &l->fnode);
 	} else {
@@ -1000,6 +1012,7 @@ static struct htab_elem *alloc_htab_elem(struct bpf_htab *htab, void *key,
 			if (!l)
 				return ERR_PTR(-E2BIG);
 			l_new = container_of(l, struct htab_elem, fnode);
+			bpf_map_inc_elem_count(&htab->map);
 		}
 	} else {
 		if (is_map_full(htab))
@@ -1224,7 +1237,8 @@ static long htab_lru_map_update_elem(struct bpf_map *map, void *key, void *value
 	if (l_old) {
 		bpf_lru_node_set_ref(&l_new->lru_node);
 		hlist_nulls_del_rcu(&l_old->hash_node);
-	}
+	} else
+		bpf_map_inc_elem_count(&htab->map);
 	ret = 0;
 
 err:
@@ -1351,6 +1365,7 @@ static long __htab_lru_percpu_map_update_elem(struct bpf_map *map, void *key,
 		pcpu_init_value(htab, htab_elem_get_ptr(l_new, key_size),
 				value, onallcpus);
 		hlist_nulls_add_head_rcu(&l_new->hash_node, head);
+		bpf_map_inc_elem_count(&htab->map);
 		l_new = NULL;
 	}
 	ret = 0;
@@ -1437,9 +1452,10 @@ static long htab_lru_map_delete_elem(struct bpf_map *map, void *key)
 
 	l = lookup_elem_raw(head, hash, key, key_size);
 
-	if (l)
+	if (l) {
+		bpf_map_dec_elem_count(&htab->map);
 		hlist_nulls_del_rcu(&l->hash_node);
-	else
+	} else
 		ret = -ENOENT;
 
 	htab_unlock_bucket(htab, b, hash, flags);
@@ -1523,6 +1539,7 @@ static void htab_map_free(struct bpf_map *map)
 		prealloc_destroy(htab);
 	}
 
+	bpf_map_free_elem_count(map);
 	free_percpu(htab->extra_elems);
 	bpf_map_area_free(htab->buckets);
 	bpf_mem_alloc_destroy(&htab->pcpu_ma);
Signature · 1 line
2.34.1
#4 lore raw

Re: [v3 PATCH bpf-next 3/6] bpf: populate the per-cpu insertions/deletions counters for hashmaps

Hi,

Quoted 20 lines · Anton Protopopov

Initialize and utilize the per-cpu insertions/deletions counters for hash-based maps. Non-trivial changes only apply to the preallocated maps for which the {inc,dec}_elem_count functions are not called, as there's no need in counting elements to sustain proper map operations.

To increase/decrease percpu counters for preallocated maps we add raw calls to the bpf_map_{inc,dec}_elem_count functions so that the impact is minimal. For dynamically allocated maps we add corresponding calls to the existing {inc,dec}_elem_count functions.

---
 kernel/bpf/hashtab.c | 23 ++++++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)
kernel/bpf/hashtab.c +0 -0
index 56d3da7d0bc6..faaef4fd3df0 100644

kernel/bpf/hashtab.c static struct bpf_map *htab_map_alloc(union bpf_attr *attr) · line 586

Quoted 6 lines
kernel/bpf/hashtab.c +3 -0
@@ -581,8 +581,14 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr)
 		}
 	}
 
+	err = bpf_map_init_elem_count(&htab->map);
+	if (err)
+		goto free_extra_elements;

Considering the per-cpu counter is not always needed, is it a good idea to make the elem_count being optional by introducing a new map flag ?

kernel/bpf/hashtab.c static struct bpf_map *htab_map_alloc(union bpf_attr *attr) · line 593

Quoted 7 lines
kernel/bpf/hashtab.c +3 -0
+
 	return &htab->map;
 
+free_extra_elements:
+	free_percpu(htab->extra_elems);
 free_prealloc:
 	prealloc_destroy(htab);

Need to check prealloc before calling prealloc_destroy(htab), otherwise for non-preallocated percpu htab prealloc_destroy() will trigger invalid memory dereference.

Quoted 62 lines
kernel/bpf/hashtab.c +10 -1
 free_map_locked:
@@ -804,6 +810,7 @@ static bool htab_lru_map_delete_node(void *arg, struct bpf_lru_node *node)
 		if (l == tgt_l) {
 			hlist_nulls_del_rcu(&l->hash_node);
 			check_and_free_fields(htab, l);
+			bpf_map_dec_elem_count(&htab->map);
 			break;
 		}
 
@@ -900,6 +907,8 @@ static bool is_map_full(struct bpf_htab *htab)
 
 static void inc_elem_count(struct bpf_htab *htab)
 {
+	bpf_map_inc_elem_count(&htab->map);
+
 	if (htab->use_percpu_counter)
 		percpu_counter_add_batch(&htab->pcount, 1, PERCPU_COUNTER_BATCH);
 	else
@@ -908,6 +917,8 @@ static void inc_elem_count(struct bpf_htab *htab)
 
 static void dec_elem_count(struct bpf_htab *htab)
 {
+	bpf_map_dec_elem_count(&htab->map);
+
 	if (htab->use_percpu_counter)
 		percpu_counter_add_batch(&htab->pcount, -1, PERCPU_COUNTER_BATCH);
 	else
@@ -920,6 +931,7 @@ static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l)
 	htab_put_fd_value(htab, l);
 
 	if (htab_is_prealloc(htab)) {
+		bpf_map_dec_elem_count(&htab->map);
 		check_and_free_fields(htab, l);
 		__pcpu_freelist_push(&htab->freelist, &l->fnode);
 	} else {
@@ -1000,6 +1012,7 @@ static struct htab_elem *alloc_htab_elem(struct bpf_htab *htab, void *key,
 			if (!l)
 				return ERR_PTR(-E2BIG);
 			l_new = container_of(l, struct htab_elem, fnode);
+			bpf_map_inc_elem_count(&htab->map);
 		}
 	} else {
 		if (is_map_full(htab))
@@ -1224,7 +1237,8 @@ static long htab_lru_map_update_elem(struct bpf_map *map, void *key, void *value
 	if (l_old) {
 		bpf_lru_node_set_ref(&l_new->lru_node);
 		hlist_nulls_del_rcu(&l_old->hash_node);
-	}
+	} else
+		bpf_map_inc_elem_count(&htab->map);
 	ret = 0;
 
 err:
@@ -1351,6 +1365,7 @@ static long __htab_lru_percpu_map_update_elem(struct bpf_map *map, void *key,
 		pcpu_init_value(htab, htab_elem_get_ptr(l_new, key_size),
 				value, onallcpus);
 		hlist_nulls_add_head_rcu(&l_new->hash_node, head);
+		bpf_map_inc_elem_count(&htab->map);
 		l_new = NULL;
 	}
 	ret = 0;

kernel/bpf/hashtab.c static long htab_lru_map_delete_elem(struct bpf_map *map, void *key) · line 1459

Quoted 10 lines
kernel/bpf/hashtab.c +3 -2
@@ -1437,9 +1452,10 @@ static long htab_lru_map_delete_elem(struct bpf_map *map, void *key)
 
 	l = lookup_elem_raw(head, hash, key, key_size);
 
-	if (l)
+	if (l) {
+		bpf_map_dec_elem_count(&htab->map);
 		hlist_nulls_del_rcu(&l->hash_node);
-	else
+	} else
 		ret = -ENOENT;

Also need to decrease elem_count for __htab_map_lookup_and_delete_batch() and __htab_map_lookup_and_delete_elem() when is_lru_map is true. Maybe for LRU map, we could just do bpf_map_dec_elem_count() in htab_lru_push_free() and do bpf_map_inc_elem_count() in prealloc_lru_pop().

Quoted 10 lines
kernel/bpf/hashtab.c +1 -0
 	htab_unlock_bucket(htab, b, hash, flags);
@@ -1523,6 +1539,7 @@ static void htab_map_free(struct bpf_map *map)
 		prealloc_destroy(htab);
 	}
 
+	bpf_map_free_elem_count(map);
 	free_percpu(htab->extra_elems);
 	bpf_map_area_free(htab->buckets);
 	bpf_mem_alloc_destroy(&htab->pcpu_ma);
#5 lore raw

Re: [v3 PATCH bpf-next 3/6] bpf: populate the per-cpu insertions/deletions counters for hashmaps

Quoted 26 lines · Hou Tao

Hi,

Quoted 23 lines · Anton Protopopov

Initialize and utilize the per-cpu insertions/deletions counters for hash-based maps. Non-trivial changes only apply to the preallocated maps for which the {inc,dec}_elem_count functions are not called, as there's no need in counting elements to sustain proper map operations.

To increase/decrease percpu counters for preallocated maps we add raw calls to the bpf_map_{inc,dec}_elem_count functions so that the impact is minimal. For dynamically allocated maps we add corresponding calls to the existing {inc,dec}_elem_count functions.

---
 kernel/bpf/hashtab.c | 23 ++++++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)
kernel/bpf/hashtab.c +0 -0
index 56d3da7d0bc6..faaef4fd3df0 100644
@@ -581,8 +581,14 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr)
 		}
 	}
 
Quoted 5 lines

kernel/bpf/hashtab.c static struct bpf_map *htab_map_alloc(union bpf_attr *attr) · line 586

Quoted 3 lines
kernel/bpf/hashtab.c +3 -0
+	err = bpf_map_init_elem_count(&htab->map);
+	if (err)
+		goto free_extra_elements;

Considering the per-cpu counter is not always needed, is it a good idea to make the elem_count being optional by introducing a new map flag ?

Per-map-flag or a static key? For me it looked like just doing an unconditional inc for a per-cpu variable is better vs. doing a check then inc or an unconditional jump.

Quoted 10 lines

kernel/bpf/hashtab.c static struct bpf_map *htab_map_alloc(union bpf_attr *attr) · line 593

Quoted 7 lines
kernel/bpf/hashtab.c +3 -0
+
 	return &htab->map;
 
+free_extra_elements:
+	free_percpu(htab->extra_elems);
 free_prealloc:
 	prealloc_destroy(htab);

Need to check prealloc before calling prealloc_destroy(htab), otherwise for non-preallocated percpu htab prealloc_destroy() will trigger invalid memory dereference.

Thanks!

Quoted 65 lines
Quoted 65 lines
kernel/bpf/hashtab.c +10 -1
 free_map_locked:
@@ -804,6 +810,7 @@ static bool htab_lru_map_delete_node(void *arg, struct bpf_lru_node *node)
 		if (l == tgt_l) {
 			hlist_nulls_del_rcu(&l->hash_node);
 			check_and_free_fields(htab, l);
+			bpf_map_dec_elem_count(&htab->map);
 			break;
 		}
 
@@ -900,6 +907,8 @@ static bool is_map_full(struct bpf_htab *htab)
 
 static void inc_elem_count(struct bpf_htab *htab)
 {
+	bpf_map_inc_elem_count(&htab->map);
+
 	if (htab->use_percpu_counter)
 		percpu_counter_add_batch(&htab->pcount, 1, PERCPU_COUNTER_BATCH);
 	else
@@ -908,6 +917,8 @@ static void inc_elem_count(struct bpf_htab *htab)
 
 static void dec_elem_count(struct bpf_htab *htab)
 {
+	bpf_map_dec_elem_count(&htab->map);
+
 	if (htab->use_percpu_counter)
 		percpu_counter_add_batch(&htab->pcount, -1, PERCPU_COUNTER_BATCH);
 	else
@@ -920,6 +931,7 @@ static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l)
 	htab_put_fd_value(htab, l);
 
 	if (htab_is_prealloc(htab)) {
+		bpf_map_dec_elem_count(&htab->map);
 		check_and_free_fields(htab, l);
 		__pcpu_freelist_push(&htab->freelist, &l->fnode);
 	} else {
@@ -1000,6 +1012,7 @@ static struct htab_elem *alloc_htab_elem(struct bpf_htab *htab, void *key,
 			if (!l)
 				return ERR_PTR(-E2BIG);
 			l_new = container_of(l, struct htab_elem, fnode);
+			bpf_map_inc_elem_count(&htab->map);
 		}
 	} else {
 		if (is_map_full(htab))
@@ -1224,7 +1237,8 @@ static long htab_lru_map_update_elem(struct bpf_map *map, void *key, void *value
 	if (l_old) {
 		bpf_lru_node_set_ref(&l_new->lru_node);
 		hlist_nulls_del_rcu(&l_old->hash_node);
-	}
+	} else
+		bpf_map_inc_elem_count(&htab->map);
 	ret = 0;
 
 err:
@@ -1351,6 +1365,7 @@ static long __htab_lru_percpu_map_update_elem(struct bpf_map *map, void *key,
 		pcpu_init_value(htab, htab_elem_get_ptr(l_new, key_size),
 				value, onallcpus);
 		hlist_nulls_add_head_rcu(&l_new->hash_node, head);
+		bpf_map_inc_elem_count(&htab->map);
 		l_new = NULL;
 	}
 	ret = 0;
@@ -1437,9 +1452,10 @@ static long htab_lru_map_delete_elem(struct bpf_map *map, void *key)
 
 	l = lookup_elem_raw(head, hash, key, key_size);
 
Quoted 12 lines

kernel/bpf/hashtab.c static long htab_lru_map_delete_elem(struct bpf_map *map, void *key) · line 1459

Quoted 7 lines
kernel/bpf/hashtab.c +3 -2
-	if (l)
+	if (l) {
+		bpf_map_dec_elem_count(&htab->map);
 		hlist_nulls_del_rcu(&l->hash_node);
-	else
+	} else
 		ret = -ENOENT;

Also need to decrease elem_count for __htab_map_lookup_and_delete_batch() and __htab_map_lookup_and_delete_elem() when is_lru_map is true. Maybe for LRU map, we could just do bpf_map_dec_elem_count() in htab_lru_push_free() and do bpf_map_inc_elem_count() in prealloc_lru_pop().

Thanks. I will fix the logic and extend the selftest to test the batch ops as well.

Quoted 11 lines
Quoted 10 lines
kernel/bpf/hashtab.c +1 -0
 	htab_unlock_bucket(htab, b, hash, flags);
@@ -1523,6 +1539,7 @@ static void htab_map_free(struct bpf_map *map)
 		prealloc_destroy(htab);
 	}
 
+	bpf_map_free_elem_count(map);
 	free_percpu(htab->extra_elems);
 	bpf_map_area_free(htab->buckets);
 	bpf_mem_alloc_destroy(&htab->pcpu_ma);
#6 lore raw

Re: [v3 PATCH bpf-next 3/6] bpf: populate the per-cpu insertions/deletions counters for hashmaps

Hi,

Quoted 27 lines · Anton Protopopov
Quoted 26 lines · Hou Tao

Hi,

Quoted 23 lines · Anton Protopopov

Initialize and utilize the per-cpu insertions/deletions counters for hash-based maps. Non-trivial changes only apply to the preallocated maps for which the {inc,dec}_elem_count functions are not called, as there's no need in counting elements to sustain proper map operations.

To increase/decrease percpu counters for preallocated maps we add raw calls to the bpf_map_{inc,dec}_elem_count functions so that the impact is minimal. For dynamically allocated maps we add corresponding calls to the existing {inc,dec}_elem_count functions.

---
 kernel/bpf/hashtab.c | 23 ++++++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)
kernel/bpf/hashtab.c +0 -0
index 56d3da7d0bc6..faaef4fd3df0 100644
@@ -581,8 +581,14 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr)
 		}
 	}
 
Quoted 8 lines
Quoted 5 lines

kernel/bpf/hashtab.c static struct bpf_map *htab_map_alloc(union bpf_attr *attr) · line 586

Quoted 3 lines
kernel/bpf/hashtab.c +3 -0
+	err = bpf_map_init_elem_count(&htab->map);
+	if (err)
+		goto free_extra_elements;

Considering the per-cpu counter is not always needed, is it a good idea to make the elem_count being optional by introducing a new map flag ?

Per-map-flag or a static key? For me it looked like just doing an unconditional inc for a per-cpu variable is better vs. doing a check then inc or an unconditional jump.

Sorry I didn't make it clear that I was worried about the allocated per-cpu memory. Previous I thought the per-cpu memory is limited, but after did some experiments I found it was almost the same as kmalloc() which could use all available memory to fulfill the allocation request. For a host with 72-cpus, the memory overhead for 10k hash map is about ~6MB. The overhead is tiny compared with the total available memory, but it is avoidable.

#7 lore raw

Re: [v3 PATCH bpf-next 3/6] bpf: populate the per-cpu insertions/deletions counters for hashmaps

Quoted 39 lines · Hou Tao

Hi,

Quoted 27 lines · Anton Protopopov
Quoted 26 lines · Hou Tao

Hi,

Quoted 23 lines · Anton Protopopov

Initialize and utilize the per-cpu insertions/deletions counters for hash-based maps. Non-trivial changes only apply to the preallocated maps for which the {inc,dec}_elem_count functions are not called, as there's no need in counting elements to sustain proper map operations.

To increase/decrease percpu counters for preallocated maps we add raw calls to the bpf_map_{inc,dec}_elem_count functions so that the impact is minimal. For dynamically allocated maps we add corresponding calls to the existing {inc,dec}_elem_count functions.

---
 kernel/bpf/hashtab.c | 23 ++++++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)
kernel/bpf/hashtab.c +0 -0
index 56d3da7d0bc6..faaef4fd3df0 100644
@@ -581,8 +581,14 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr)
 		}
 	}
 
Quoted 8 lines
Quoted 5 lines

kernel/bpf/hashtab.c static struct bpf_map *htab_map_alloc(union bpf_attr *attr) · line 586

Quoted 3 lines
kernel/bpf/hashtab.c +3 -0
+	err = bpf_map_init_elem_count(&htab->map);
+	if (err)
+		goto free_extra_elements;

Considering the per-cpu counter is not always needed, is it a good idea to make the elem_count being optional by introducing a new map flag ?

Per-map-flag or a static key? For me it looked like just doing an unconditional inc for a per-cpu variable is better vs. doing a check then inc or an unconditional jump.

Quoted 7 lines

Sorry I didn't make it clear that I was worried about the allocated per-cpu memory. Previous I thought the per-cpu memory is limited, but after did some experiments I found it was almost the same as kmalloc() which could use all available memory to fulfill the allocation request. For a host with 72-cpus, the memory overhead for 10k hash map is about ~6MB. The overhead is tiny compared with the total available memory, but it is avoidable.

So, in my first patch I've only added new counters for preallocated maps. But then the feedback was that we need a generic percpu inc/dec counters, so I added them by default. For me a percpu s64 looks cheap enough for a hash map...

#8 lore raw

Re: [v3 PATCH bpf-next 3/6] bpf: populate the per-cpu insertions/deletions counters for hashmaps

Hi,

Quoted 5 lines · Anton Protopopov
Quoted 4 lines · Hou Tao

Hi,

Quoted 1 line · Anton Protopopov

On Tue, Jul 04, 2023 at 09:56:36PM +0800, Hou Tao wrote:

SNIP

Quoted 11 lines
Quoted 11 lines
Quoted 4 lines
Quoted 1 line

by introducing a new map flag ?

Per-map-flag or a static key? For me it looked like just doing an unconditional inc for a per-cpu variable is better vs. doing a check then inc or an unconditional jump.

Sorry I didn't make it clear that I was worried about the allocated per-cpu memory. Previous I thought the per-cpu memory is limited, but after did some experiments I found it was almost the same as kmalloc() which could use all available memory to fulfill the allocation request. For a host with 72-cpus, the memory overhead for 10k hash map is about ~6MB. The overhead is tiny compared with the total available memory, but it is avoidable.

Quoted 3 lines

So, in my first patch I've only added new counters for preallocated maps. But then the feedback was that we need a generic percpu inc/dec counters, so I added them by default. For me a percpu s64 looks cheap enough for a hash map...

Thanks for the explanation. Let's just allocate the per-cpu elem_count in hash map. If there are use cases which need to make it optional, we can revise that later.

Quoted 1 line

.

#9 lore raw

PATCH v3 4/6 bpf: make preloaded map iterators to display map elements count

In reply to Anton Protopopov

Add another column to the /sys/fs/bpf/maps.debug iterator to display cur_entries, the current number of entries in the map as is returned by the bpf_map_sum_elem_count kfunc. Also fix formatting.

Example:

    # cat /sys/fs/bpf/maps.debug
      id name             max_entries  cur_entries
       2 iterator.rodata            1            0
     125 cilium_auth_map       524288          666
     126 cilium_runtime_          256            0
     127 cilium_signals            32            0
     128 cilium_node_map        16384         1344
     129 cilium_events             32            0
     ...
---
 kernel/bpf/preload/iterators/iterators.bpf.c  |   9 +-
 .../iterators/iterators.lskel-little-endian.h | 526 +++++++++---------
 2 files changed, 275 insertions(+), 260 deletions(-)
kernel/bpf/preload/iterators/iterators.bpf.c +7 -2
index 03af863314ea..b78968b63fab 100644
@@ -73,6 +73,8 @@ static const char *get_name(struct btf *btf, long btf_id, const char *fallback)
 	return str + name_off;
 }
 
+__s64 bpf_map_sum_elem_count(struct bpf_map *map) __ksym;
+
 SEC("iter/bpf_map")
 int dump_bpf_map(struct bpf_iter__bpf_map *ctx)
 {
@@ -84,9 +86,12 @@ int dump_bpf_map(struct bpf_iter__bpf_map *ctx)
 		return 0;
 
 	if (seq_num == 0)
-		BPF_SEQ_PRINTF(seq, "  id name             max_entries\n");
+		BPF_SEQ_PRINTF(seq, "  id name             max_entries  cur_entries\n");
+
+	BPF_SEQ_PRINTF(seq, "%4u %-16s  %10d   %10lld\n",
+		       map->id, map->name, map->max_entries,
+		       bpf_map_sum_elem_count(map));
 
-	BPF_SEQ_PRINTF(seq, "%4u %-16s%6d\n", map->id, map->name, map->max_entries);
 	return 0;
 }
 
kernel/bpf/preload/iterators/iterators.lskel-little-endian.h +268 -258

583 lines not on this page · raw message

Signature · 1 line
2.34.1
#10 lore raw

PATCH v3 5/6 selftests/bpf: test map percpu stats

In reply to Anton Protopopov

Add a new map test, map_percpu_stats.c, which is checking the correctness of map's percpu elements counters. For supported maps the test upserts a number of elements, checks the correctness of the counters, then deletes all the elements and checks again that the counters sum drops down to zero.

The following map types are tested:

* BPF_MAP_TYPE_HASH, BPF_F_NO_PREALLOC
* BPF_MAP_TYPE_PERCPU_HASH, BPF_F_NO_PREALLOC
* BPF_MAP_TYPE_HASH,
* BPF_MAP_TYPE_PERCPU_HASH,
* BPF_MAP_TYPE_LRU_HASH
* BPF_MAP_TYPE_LRU_PERCPU_HASH

---
 .../bpf/map_tests/map_percpu_stats.c          | 336 ++++++++++++++++++
 .../selftests/bpf/progs/map_percpu_stats.c    |  24 ++
 2 files changed, 360 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/map_tests/map_percpu_stats.c
 create mode 100644 tools/testing/selftests/bpf/progs/map_percpu_stats.c
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +336 -0
new file mode 100644
index 000000000000..5b45af230368
@@ -0,0 +1,336 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2023 Isovalent */
+
+#include <errno.h>
+#include <unistd.h>
+#include <pthread.h>
+
+#include <bpf/bpf.h>
+#include <bpf/libbpf.h>
+
+#include <bpf_util.h>
+#include <test_maps.h>
+
+#include "map_percpu_stats.skel.h"
+
+#define MAX_ENTRIES 16384
+#define N_THREADS 37
+
+#define MAX_MAP_KEY_SIZE 4
+
+static void map_info(int map_fd, struct bpf_map_info *info)
+{
+	__u32 len = sizeof(*info);
+	int ret;
+
+	memset(info, 0, sizeof(*info));
+
+	ret = bpf_obj_get_info_by_fd(map_fd, info, &len);
+	CHECK(ret < 0, "bpf_obj_get_info_by_fd", "error: %s\n", strerror(errno));
+}
+
+static const char *map_type_to_s(__u32 type)
+{
+	switch (type) {
+	case BPF_MAP_TYPE_HASH:
+		return "HASH";
+	case BPF_MAP_TYPE_PERCPU_HASH:
+		return "PERCPU_HASH";
+	case BPF_MAP_TYPE_LRU_HASH:
+		return "LRU_HASH";
+	case BPF_MAP_TYPE_LRU_PERCPU_HASH:
+		return "LRU_PERCPU_HASH";
+	default:
+		return "<define-me>";
+	}
+}
+
+/* Map i -> map-type-specific-key */
+static void *map_key(__u32 type, __u32 i)
+{
+	static __thread __u8 key[MAX_MAP_KEY_SIZE];
+
+	*(__u32 *)key = i;
+	return key;
+}
+
+static __u32 map_count_elements(__u32 type, int map_fd)
+{
+	void *key = map_key(type, -1);
+	int n = 0;
+
+	while (!bpf_map_get_next_key(map_fd, key, key))
+		n++;
+	return n;
+}
+
+static void delete_all_elements(__u32 type, int map_fd)
+{
+	void *key = map_key(type, -1);
+	void *keys;
+	int n = 0;
+	int ret;
+
+	keys = calloc(MAX_MAP_KEY_SIZE, MAX_ENTRIES);
+	CHECK(!keys, "calloc", "error: %s\n", strerror(errno));
+
+	for (; !bpf_map_get_next_key(map_fd, key, key); n++)
+		memcpy(keys + n*MAX_MAP_KEY_SIZE, key, MAX_MAP_KEY_SIZE);
+
+	while (--n >= 0) {
+		ret = bpf_map_delete_elem(map_fd, keys + n*MAX_MAP_KEY_SIZE);
+		CHECK(ret < 0, "bpf_map_delete_elem", "error: %s\n", strerror(errno));
+	}
+}
+
+static bool is_lru(__u32 map_type)
+{
+	return map_type == BPF_MAP_TYPE_LRU_HASH ||
+	       map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH;
+}
+
+struct upsert_opts {
+	__u32 map_type;
+	int map_fd;
+	__u32 n;
+};
+
+static void *patch_map_thread(void *arg)
+{
+	struct upsert_opts *opts = arg;
+	void *key;
+	int val;
+	int ret;
+	int i;
+
+	for (i = 0; i < opts->n; i++) {
+		key = map_key(opts->map_type, i);
+		val = rand();
+		ret = bpf_map_update_elem(opts->map_fd, key, &val, 0);
+		CHECK(ret < 0, "bpf_map_update_elem", "error: %s\n", strerror(errno));
+	}
+	return NULL;
+}
+
+static void upsert_elements(struct upsert_opts *opts)
+{
+	pthread_t threads[N_THREADS];
+	int ret;
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(threads); i++) {
+		ret = pthread_create(&i[threads], NULL, patch_map_thread, opts);
+		CHECK(ret != 0, "pthread_create", "error: %s\n", strerror(ret));
+	}
+
+	for (i = 0; i < ARRAY_SIZE(threads); i++) {
+		ret = pthread_join(i[threads], NULL);
+		CHECK(ret != 0, "pthread_join", "error: %s\n", strerror(ret));
+	}
+}
+
+static __u32 read_cur_elements(int iter_fd)
+{
+	char buf[64];
+	ssize_t n;
+	__u32 ret;
+
+	n = read(iter_fd, buf, sizeof(buf)-1);
+	CHECK(n <= 0, "read", "error: %s\n", strerror(errno));
+	buf[n] = '\0';
+
+	errno = 0;
+	ret = (__u32)strtol(buf, NULL, 10);
+	CHECK(errno != 0, "strtol", "error: %s\n", strerror(errno));
+
+	return ret;
+}
+
+static __u32 get_cur_elements(int map_id)
+{
+	LIBBPF_OPTS(bpf_iter_attach_opts, opts);
+	union bpf_iter_link_info linfo;
+	struct map_percpu_stats *skel;
+	struct bpf_link *link;
+	int iter_fd;
+	int ret;
+
+	opts.link_info = &linfo;
+	opts.link_info_len = sizeof(linfo);
+
+	skel = map_percpu_stats__open();
+	CHECK(skel == NULL, "map_percpu_stats__open", "error: %s", strerror(errno));
+
+	skel->bss->target_id = map_id;
+
+	ret = map_percpu_stats__load(skel);
+	CHECK(ret != 0, "map_percpu_stats__load", "error: %s", strerror(errno));
+
+	link = bpf_program__attach_iter(skel->progs.dump_bpf_map, &opts);
+	CHECK(!link, "bpf_program__attach_iter", "error: %s\n", strerror(errno));
+
+	iter_fd = bpf_iter_create(bpf_link__fd(link));
+	CHECK(iter_fd < 0, "bpf_iter_create", "error: %s\n", strerror(errno));
+
+	return read_cur_elements(iter_fd);
+}
+
+static void __test(int map_fd)
+{
+	__u32 n = MAX_ENTRIES - 1000;
+	__u32 real_current_elements;
+	__u32 iter_current_elements;
+	struct upsert_opts opts = {
+		.map_fd = map_fd,
+		.n = n,
+	};
+	struct bpf_map_info info;
+
+	map_info(map_fd, &info);
+	opts.map_type = info.type;
+
+	/*
+	 * Upsert keys [0, n) under some competition: with random values from
+	 * N_THREADS threads
+	 */
+	upsert_elements(&opts);
+
+	/*
+	 * The sum of percpu elements counters for all hashtable-based maps
+	 * should be equal to the number of elements present in the map. For
+	 * non-lru maps this number should be the number n of upserted
+	 * elements. For lru maps some elements might have been evicted. Check
+	 * that all numbers make sense
+	 */
+	map_info(map_fd, &info);
+	real_current_elements = map_count_elements(info.type, map_fd);
+	if (!is_lru(info.type))
+		CHECK(n != real_current_elements, "map_count_elements",
+		      "real_current_elements(%u) != expected(%u)\n", real_current_elements, n);
+
+	iter_current_elements = get_cur_elements(info.id);
+	CHECK(iter_current_elements != real_current_elements, "get_cur_elements",
+	      "iter_current_elements=%u, expected %u (map_type=%s,map_flags=%08x)\n",
+	      iter_current_elements, real_current_elements, map_type_to_s(info.type), info.map_flags);
+
+	/*
+	 * Cleanup the map and check that all elements are actually gone and
+	 * that the sum of percpu elements counters is back to 0 as well
+	 */
+	delete_all_elements(info.type, map_fd);
+	map_info(map_fd, &info);
+	real_current_elements = map_count_elements(info.type, map_fd);
+	CHECK(real_current_elements, "map_count_elements",
+	      "expected real_current_elements=0, got %u", real_current_elements);
+
+	iter_current_elements = get_cur_elements(info.id);
+	CHECK(iter_current_elements != 0, "get_cur_elements",
+	      "iter_current_elements=%u, expected 0 (map_type=%s,map_flags=%08x)\n",
+	      iter_current_elements, map_type_to_s(info.type), info.map_flags);
+
+	close(map_fd);
+}
+
+static int map_create_opts(__u32 type, const char *name,
+			   struct bpf_map_create_opts *map_opts,
+			   __u32 key_size, __u32 val_size)
+{
+	int map_fd;
+
+	map_fd = bpf_map_create(type, name, key_size, val_size, MAX_ENTRIES, map_opts);
+	CHECK(map_fd < 0, "bpf_map_create()", "error:%s (name=%s)\n",
+			strerror(errno), name);
+
+	return map_fd;
+}
+
+static int map_create(__u32 type, const char *name, struct bpf_map_create_opts *map_opts)
+{
+	return map_create_opts(type, name, map_opts, sizeof(int), sizeof(int));
+}
+
+static int create_hash(void)
+{
+	struct bpf_map_create_opts map_opts = {
+		.sz = sizeof(map_opts),
+		.map_flags = BPF_F_NO_PREALLOC,
+	};
+
+	return map_create(BPF_MAP_TYPE_HASH, "hash", &map_opts);
+}
+
+static int create_percpu_hash(void)
+{
+	struct bpf_map_create_opts map_opts = {
+		.sz = sizeof(map_opts),
+		.map_flags = BPF_F_NO_PREALLOC,
+	};
+
+	return map_create(BPF_MAP_TYPE_PERCPU_HASH, "percpu_hash", &map_opts);
+}
+
+static int create_hash_prealloc(void)
+{
+	return map_create(BPF_MAP_TYPE_HASH, "hash", NULL);
+}
+
+static int create_percpu_hash_prealloc(void)
+{
+	return map_create(BPF_MAP_TYPE_PERCPU_HASH, "percpu_hash_prealloc", NULL);
+}
+
+static int create_lru_hash(void)
+{
+	return map_create(BPF_MAP_TYPE_LRU_HASH, "lru_hash", NULL);
+}
+
+static int create_percpu_lru_hash(void)
+{
+	return map_create(BPF_MAP_TYPE_LRU_PERCPU_HASH, "lru_hash_percpu", NULL);
+}
+
+static void map_percpu_stats_hash(void)
+{
+	__test(create_hash());
+	printf("test_%s:PASS\n", __func__);
+}
+
+static void map_percpu_stats_percpu_hash(void)
+{
+	__test(create_percpu_hash());
+	printf("test_%s:PASS\n", __func__);
+}
+
+static void map_percpu_stats_hash_prealloc(void)
+{
+	__test(create_hash_prealloc());
+	printf("test_%s:PASS\n", __func__);
+}
+
+static void map_percpu_stats_percpu_hash_prealloc(void)
+{
+	__test(create_percpu_hash_prealloc());
+	printf("test_%s:PASS\n", __func__);
+}
+
+static void map_percpu_stats_lru_hash(void)
+{
+	__test(create_lru_hash());
+	printf("test_%s:PASS\n", __func__);
+}
+
+static void map_percpu_stats_percpu_lru_hash(void)
+{
+	__test(create_percpu_lru_hash());
+	printf("test_%s:PASS\n", __func__);
+}
+
+void test_map_percpu_stats(void)
+{
+	map_percpu_stats_hash();
+	map_percpu_stats_percpu_hash();
+	map_percpu_stats_hash_prealloc();
+	map_percpu_stats_percpu_hash_prealloc();
+	map_percpu_stats_lru_hash();
+	map_percpu_stats_percpu_lru_hash();
+}
tools/testing/selftests/bpf/progs/map_percpu_stats.c new file +24 -0
new file mode 100644
index 000000000000..10b2325c1720
@@ -0,0 +1,24 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2023 Isovalent */
+
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+__u32 target_id;
+
+__s64 bpf_map_sum_elem_count(struct bpf_map *map) __ksym;
+
+SEC("iter/bpf_map")
+int dump_bpf_map(struct bpf_iter__bpf_map *ctx)
+{
+	struct seq_file *seq = ctx->meta->seq;
+	struct bpf_map *map = ctx->map;
+
+	if (map && map->id == target_id)
+		BPF_SEQ_PRINTF(seq, "%lld", bpf_map_sum_elem_count(map));
+
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
Signature · 1 line
2.34.1
#11 lore raw

Re: [v3 PATCH bpf-next 5/6] selftests/bpf: test map percpu stats

Hi,

Quoted 5 lines · Anton Protopopov

Add a new map test, map_percpu_stats.c, which is checking the correctness of map's percpu elements counters. For supported maps the test upserts a number of elements, checks the correctness of the counters, then deletes all the elements and checks again that the counters sum drops down to zero.

Quoted 8 lines

The following map types are tested:

* BPF_MAP_TYPE_HASH, BPF_F_NO_PREALLOC
* BPF_MAP_TYPE_PERCPU_HASH, BPF_F_NO_PREALLOC
* BPF_MAP_TYPE_HASH,
* BPF_MAP_TYPE_PERCPU_HASH,
* BPF_MAP_TYPE_LRU_HASH
* BPF_MAP_TYPE_LRU_PERCPU_HASH

A test for BPF_MAP_TYPE_HASH_OF_MAPS is also needed.

Quoted 22 lines
---
 .../bpf/map_tests/map_percpu_stats.c          | 336 ++++++++++++++++++
 .../selftests/bpf/progs/map_percpu_stats.c    |  24 ++
 2 files changed, 360 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/map_tests/map_percpu_stats.c
 create mode 100644 tools/testing/selftests/bpf/progs/map_percpu_stats.c
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +7 -0
new file mode 100644
index 000000000000..5b45af230368
@@ -0,0 +1,336 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2023 Isovalent */
+
+#include <errno.h>
+#include <unistd.h>
+#include <pthread.h>
+

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 17

Quoted 10 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +10 -0
+#include <bpf/bpf.h>
+#include <bpf/libbpf.h>
+
+#include <bpf_util.h>
+#include <test_maps.h>
+
+#include "map_percpu_stats.skel.h"
+
+#define MAX_ENTRIES 16384
+#define N_THREADS 37

Why 37 thread is needed here ? Does a small number of threads work as well ?

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 29

Quoted 12 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +12 -0
+
+#define MAX_MAP_KEY_SIZE 4
+
+static void map_info(int map_fd, struct bpf_map_info *info)
+{
+	__u32 len = sizeof(*info);
+	int ret;
+
+	memset(info, 0, sizeof(*info));
+
+	ret = bpf_obj_get_info_by_fd(map_fd, info, &len);
+	CHECK(ret < 0, "bpf_obj_get_info_by_fd", "error: %s\n", strerror(errno));

Please use ASSERT_OK instead.

Quoted 12 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +12 -0
+}
+
+static const char *map_type_to_s(__u32 type)
+{
+	switch (type) {
+	case BPF_MAP_TYPE_HASH:
+		return "HASH";
+	case BPF_MAP_TYPE_PERCPU_HASH:
+		return "PERCPU_HASH";
+	case BPF_MAP_TYPE_LRU_HASH:
+		return "LRU_HASH";
+	case BPF_MAP_TYPE_LRU_PERCPU_HASH:

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 51

Quoted 10 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +10 -0
+		return "LRU_PERCPU_HASH";
+	default:
+		return "<define-me>";
+	}
+}
+
+/* Map i -> map-type-specific-key */
+static void *map_key(__u32 type, __u32 i)
+{
+	static __thread __u8 key[MAX_MAP_KEY_SIZE];

Why a per-thread key is necessary here ? Could we just define it when the key is needed ?

Quoted 23 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +23 -0
+
+	*(__u32 *)key = i;
+	return key;
+}
+
+static __u32 map_count_elements(__u32 type, int map_fd)
+{
+	void *key = map_key(type, -1);
+	int n = 0;
+
+	while (!bpf_map_get_next_key(map_fd, key, key))
+		n++;
+	return n;
+}
+
+static void delete_all_elements(__u32 type, int map_fd)
+{
+	void *key = map_key(type, -1);
+	void *keys;
+	int n = 0;
+	int ret;
+
+	keys = calloc(MAX_MAP_KEY_SIZE, MAX_ENTRIES);

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 84

Quoted 10 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +10 -0
+	CHECK(!keys, "calloc", "error: %s\n", strerror(errno));
+
+	for (; !bpf_map_get_next_key(map_fd, key, key); n++)
+		memcpy(keys + n*MAX_MAP_KEY_SIZE, key, MAX_MAP_KEY_SIZE);
+
+	while (--n >= 0) {
+		ret = bpf_map_delete_elem(map_fd, keys + n*MAX_MAP_KEY_SIZE);
+		CHECK(ret < 0, "bpf_map_delete_elem", "error: %s\n", strerror(errno));
+	}
+}

Please use ASSERT_xxx() to replace CHECK().

Quoted 81 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +81 -0
+
+static bool is_lru(__u32 map_type)
+{
+	return map_type == BPF_MAP_TYPE_LRU_HASH ||
+	       map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH;
+}
+
+struct upsert_opts {
+	__u32 map_type;
+	int map_fd;
+	__u32 n;
+};
+
+static void *patch_map_thread(void *arg)
+{
+	struct upsert_opts *opts = arg;
+	void *key;
+	int val;
+	int ret;
+	int i;
+
+	for (i = 0; i < opts->n; i++) {
+		key = map_key(opts->map_type, i);
+		val = rand();
+		ret = bpf_map_update_elem(opts->map_fd, key, &val, 0);
+		CHECK(ret < 0, "bpf_map_update_elem", "error: %s\n", strerror(errno));
+	}
+	return NULL;
+}
+
+static void upsert_elements(struct upsert_opts *opts)
+{
+	pthread_t threads[N_THREADS];
+	int ret;
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(threads); i++) {
+		ret = pthread_create(&i[threads], NULL, patch_map_thread, opts);
+		CHECK(ret != 0, "pthread_create", "error: %s\n", strerror(ret));
+	}
+
+	for (i = 0; i < ARRAY_SIZE(threads); i++) {
+		ret = pthread_join(i[threads], NULL);
+		CHECK(ret != 0, "pthread_join", "error: %s\n", strerror(ret));
+	}
+}
+
+static __u32 read_cur_elements(int iter_fd)
+{
+	char buf[64];
+	ssize_t n;
+	__u32 ret;
+
+	n = read(iter_fd, buf, sizeof(buf)-1);
+	CHECK(n <= 0, "read", "error: %s\n", strerror(errno));
+	buf[n] = '\0';
+
+	errno = 0;
+	ret = (__u32)strtol(buf, NULL, 10);
+	CHECK(errno != 0, "strtol", "error: %s\n", strerror(errno));
+
+	return ret;
+}
+
+static __u32 get_cur_elements(int map_id)
+{
+	LIBBPF_OPTS(bpf_iter_attach_opts, opts);
+	union bpf_iter_link_info linfo;
+	struct map_percpu_stats *skel;
+	struct bpf_link *link;
+	int iter_fd;
+	int ret;
+
+	opts.link_info = &linfo;
+	opts.link_info_len = sizeof(linfo);
+
+	skel = map_percpu_stats__open();
+	CHECK(skel == NULL, "map_percpu_stats__open", "error: %s", strerror(errno));
+
+	skel->bss->target_id = map_id;
+

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 175

Quoted 10 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +10 -0
+	ret = map_percpu_stats__load(skel);
+	CHECK(ret != 0, "map_percpu_stats__load", "error: %s", strerror(errno));
+
+	link = bpf_program__attach_iter(skel->progs.dump_bpf_map, &opts);
+	CHECK(!link, "bpf_program__attach_iter", "error: %s\n", strerror(errno));
+
+	iter_fd = bpf_iter_create(bpf_link__fd(link));
+	CHECK(iter_fd < 0, "bpf_iter_create", "error: %s\n", strerror(errno));
+
+	return read_cur_elements(iter_fd);

Need to do close(iter_fd), bpf_link__destroy(link) and map__percpu_stats__destroy() before return, otherwise there will be resource leak.

Quoted 20 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +20 -0
+}
+
+static void __test(int map_fd)
+{
+	__u32 n = MAX_ENTRIES - 1000;
+	__u32 real_current_elements;
+	__u32 iter_current_elements;
+	struct upsert_opts opts = {
+		.map_fd = map_fd,
+		.n = n,
+	};
+	struct bpf_map_info info;
+
+	map_info(map_fd, &info);
+	opts.map_type = info.type;
+
+	/*
+	 * Upsert keys [0, n) under some competition: with random values from
+	 * N_THREADS threads
+	 */

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 205

Quoted 10 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +10 -0
+	upsert_elements(&opts);
+
+	/*
+	 * The sum of percpu elements counters for all hashtable-based maps
+	 * should be equal to the number of elements present in the map. For
+	 * non-lru maps this number should be the number n of upserted
+	 * elements. For lru maps some elements might have been evicted. Check
+	 * that all numbers make sense
+	 */
+	map_info(map_fd, &info);

I think there is no need to call map_info() multiple times because the needed type and id will not be changed after creation.

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 209

Quoted 4 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +4 -0
+	real_current_elements = map_count_elements(info.type, map_fd);
+	if (!is_lru(info.type))
+		CHECK(n != real_current_elements, "map_count_elements",
+		      "real_current_elements(%u) != expected(%u)\n", real_current_elements, n);

For LRU map, please use ASSERT_EQ(). For LRU map, should we check "n >= real_current_elements" instead ?

Quoted 5 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +5 -0
+
+	iter_current_elements = get_cur_elements(info.id);
+	CHECK(iter_current_elements != real_current_elements, "get_cur_elements",
+	      "iter_current_elements=%u, expected %u (map_type=%s,map_flags=%08x)\n",
+	      iter_current_elements, real_current_elements, map_type_to_s(info.type), info.map_flags);

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 224

Quoted 10 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +10 -0
+
+	/*
+	 * Cleanup the map and check that all elements are actually gone and
+	 * that the sum of percpu elements counters is back to 0 as well
+	 */
+	delete_all_elements(info.type, map_fd);
+	map_info(map_fd, &info);
+	real_current_elements = map_count_elements(info.type, map_fd);
+	CHECK(real_current_elements, "map_count_elements",
+	      "expected real_current_elements=0, got %u", real_current_elements);

ASSERT_EQ

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 230

Quoted 6 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +6 -0
+
+	iter_current_elements = get_cur_elements(info.id);
+	CHECK(iter_current_elements != 0, "get_cur_elements",
+	      "iter_current_elements=%u, expected 0 (map_type=%s,map_flags=%08x)\n",
+	      iter_current_elements, map_type_to_s(info.type), info.map_flags);
+

ASSERT_NEQ

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 242

Quoted 12 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +12 -0
+	close(map_fd);
+}
+
+static int map_create_opts(__u32 type, const char *name,
+			   struct bpf_map_create_opts *map_opts,
+			   __u32 key_size, __u32 val_size)
+{
+	int map_fd;
+
+	map_fd = bpf_map_create(type, name, key_size, val_size, MAX_ENTRIES, map_opts);
+	CHECK(map_fd < 0, "bpf_map_create()", "error:%s (name=%s)\n",
+			strerror(errno), name);

Please use ASSERT_GE instead.

Quoted 73 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +73 -0
+
+	return map_fd;
+}
+
+static int map_create(__u32 type, const char *name, struct bpf_map_create_opts *map_opts)
+{
+	return map_create_opts(type, name, map_opts, sizeof(int), sizeof(int));
+}
+
+static int create_hash(void)
+{
+	struct bpf_map_create_opts map_opts = {
+		.sz = sizeof(map_opts),
+		.map_flags = BPF_F_NO_PREALLOC,
+	};
+
+	return map_create(BPF_MAP_TYPE_HASH, "hash", &map_opts);
+}
+
+static int create_percpu_hash(void)
+{
+	struct bpf_map_create_opts map_opts = {
+		.sz = sizeof(map_opts),
+		.map_flags = BPF_F_NO_PREALLOC,
+	};
+
+	return map_create(BPF_MAP_TYPE_PERCPU_HASH, "percpu_hash", &map_opts);
+}
+
+static int create_hash_prealloc(void)
+{
+	return map_create(BPF_MAP_TYPE_HASH, "hash", NULL);
+}
+
+static int create_percpu_hash_prealloc(void)
+{
+	return map_create(BPF_MAP_TYPE_PERCPU_HASH, "percpu_hash_prealloc", NULL);
+}
+
+static int create_lru_hash(void)
+{
+	return map_create(BPF_MAP_TYPE_LRU_HASH, "lru_hash", NULL);
+}
+
+static int create_percpu_lru_hash(void)
+{
+	return map_create(BPF_MAP_TYPE_LRU_PERCPU_HASH, "lru_hash_percpu", NULL);
+}
+
+static void map_percpu_stats_hash(void)
+{
+	__test(create_hash());
+	printf("test_%s:PASS\n", __func__);
+}
+
+static void map_percpu_stats_percpu_hash(void)
+{
+	__test(create_percpu_hash());
+	printf("test_%s:PASS\n", __func__);
+}
+
+static void map_percpu_stats_hash_prealloc(void)
+{
+	__test(create_hash_prealloc());
+	printf("test_%s:PASS\n", __func__);
+}
+
+static void map_percpu_stats_percpu_hash_prealloc(void)
+{
+	__test(create_percpu_hash_prealloc());
+	printf("test_%s:PASS\n", __func__);
+}
+

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 325

Quoted 10 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +10 -0
+static void map_percpu_stats_lru_hash(void)
+{
+	__test(create_lru_hash());
+	printf("test_%s:PASS\n", __func__);
+}
+
+static void map_percpu_stats_percpu_lru_hash(void)
+{
+	__test(create_percpu_lru_hash());
+	printf("test_%s:PASS\n", __func__);

After switch to subtest, the printf() can be removed.

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 336

Quoted 11 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +11 -0
+}
+
+void test_map_percpu_stats(void)
+{
+	map_percpu_stats_hash();
+	map_percpu_stats_percpu_hash();
+	map_percpu_stats_hash_prealloc();
+	map_percpu_stats_percpu_hash_prealloc();
+	map_percpu_stats_lru_hash();
+	map_percpu_stats_percpu_lru_hash();
+}

Please use test__start_subtest() to create multiple subtests.

Quoted 30 lines
tools/testing/selftests/bpf/progs/map_percpu_stats.c new file +24 -0
new file mode 100644
index 000000000000..10b2325c1720
@@ -0,0 +1,24 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2023 Isovalent */
+
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+__u32 target_id;
+
+__s64 bpf_map_sum_elem_count(struct bpf_map *map) __ksym;
+
+SEC("iter/bpf_map")
+int dump_bpf_map(struct bpf_iter__bpf_map *ctx)
+{
+	struct seq_file *seq = ctx->meta->seq;
+	struct bpf_map *map = ctx->map;
+
+	if (map && map->id == target_id)
+		BPF_SEQ_PRINTF(seq, "%lld", bpf_map_sum_elem_count(map));
+
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
#12 lore raw

Re: [v3 PATCH bpf-next 5/6] selftests/bpf: test map percpu stats

Quoted 10 lines · Hou Tao

Hi,

Quoted 7 lines · Anton Protopopov

Add a new map test, map_percpu_stats.c, which is checking the correctness of map's percpu elements counters. For supported maps the test upserts a number of elements, checks the correctness of the counters, then deletes all the elements and checks again that the counters sum drops down to zero.

The following map types are tested:

Quoted 8 lines
Quoted 6 lines

* BPF_MAP_TYPE_HASH, BPF_F_NO_PREALLOC
* BPF_MAP_TYPE_PERCPU_HASH, BPF_F_NO_PREALLOC
* BPF_MAP_TYPE_HASH,
* BPF_MAP_TYPE_PERCPU_HASH,
* BPF_MAP_TYPE_LRU_HASH
* BPF_MAP_TYPE_LRU_PERCPU_HASH

A test for BPF_MAP_TYPE_HASH_OF_MAPS is also needed.

I will add it.

Quoted 26 lines
Quoted 26 lines
---
 .../bpf/map_tests/map_percpu_stats.c          | 336 ++++++++++++++++++
 .../selftests/bpf/progs/map_percpu_stats.c    |  24 ++
 2 files changed, 360 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/map_tests/map_percpu_stats.c
 create mode 100644 tools/testing/selftests/bpf/progs/map_percpu_stats.c
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +11 -0
new file mode 100644
index 000000000000..5b45af230368
@@ -0,0 +1,336 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2023 Isovalent */
+
+#include <errno.h>
+#include <unistd.h>
+#include <pthread.h>
+
+#include <bpf/bpf.h>
+#include <bpf/libbpf.h>
+
+#include <bpf_util.h>
Quoted 8 lines

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 17

Quoted 6 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +6 -0
+#include <test_maps.h>
+
+#include "map_percpu_stats.skel.h"
+
+#define MAX_ENTRIES 16384
+#define N_THREADS 37

Why 37 thread is needed here ? Does a small number of threads work as well ?

This was used to evict more elements from LRU maps when they are full.

Quoted 5 lines
Quoted 5 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +5 -0
+
+#define MAX_MAP_KEY_SIZE 4
+
+static void map_info(int map_fd, struct bpf_map_info *info)
+{
Quoted 8 lines

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 29

Quoted 7 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +7 -0
+	__u32 len = sizeof(*info);
+	int ret;
+
+	memset(info, 0, sizeof(*info));
+
+	ret = bpf_obj_get_info_by_fd(map_fd, info, &len);
+	CHECK(ret < 0, "bpf_obj_get_info_by_fd", "error: %s\n", strerror(errno));

Please use ASSERT_OK instead.

Ok, thanks, will do (and for all other similar cases you've mentioned below).

Quoted 17 lines
Quoted 17 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +17 -0
+}
+
+static const char *map_type_to_s(__u32 type)
+{
+	switch (type) {
+	case BPF_MAP_TYPE_HASH:
+		return "HASH";
+	case BPF_MAP_TYPE_PERCPU_HASH:
+		return "PERCPU_HASH";
+	case BPF_MAP_TYPE_LRU_HASH:
+		return "LRU_HASH";
+	case BPF_MAP_TYPE_LRU_PERCPU_HASH:
+		return "LRU_PERCPU_HASH";
+	default:
+		return "<define-me>";
+	}
+}
Quoted 8 lines

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 51

Quoted 5 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +5 -0
+
+/* Map i -> map-type-specific-key */
+static void *map_key(__u32 type, __u32 i)
+{
+	static __thread __u8 key[MAX_MAP_KEY_SIZE];

Why a per-thread key is necessary here ? Could we just define it when the key is needed ?

Thanks, I will remove it to simplify code. (This was used to create keys for non-hash maps, e.g., LPM-trie in the first version of the patch.)

Quoted 122 lines
Quoted 23 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +23 -0
+
+	*(__u32 *)key = i;
+	return key;
+}
+
+static __u32 map_count_elements(__u32 type, int map_fd)
+{
+	void *key = map_key(type, -1);
+	int n = 0;
+
+	while (!bpf_map_get_next_key(map_fd, key, key))
+		n++;
+	return n;
+}
+
+static void delete_all_elements(__u32 type, int map_fd)
+{
+	void *key = map_key(type, -1);
+	void *keys;
+	int n = 0;
+	int ret;
+
+	keys = calloc(MAX_MAP_KEY_SIZE, MAX_ENTRIES);

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 84

Quoted 10 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +10 -0
+	CHECK(!keys, "calloc", "error: %s\n", strerror(errno));
+
+	for (; !bpf_map_get_next_key(map_fd, key, key); n++)
+		memcpy(keys + n*MAX_MAP_KEY_SIZE, key, MAX_MAP_KEY_SIZE);
+
+	while (--n >= 0) {
+		ret = bpf_map_delete_elem(map_fd, keys + n*MAX_MAP_KEY_SIZE);
+		CHECK(ret < 0, "bpf_map_delete_elem", "error: %s\n", strerror(errno));
+	}
+}

Please use ASSERT_xxx() to replace CHECK().

Quoted 87 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +87 -0
+
+static bool is_lru(__u32 map_type)
+{
+	return map_type == BPF_MAP_TYPE_LRU_HASH ||
+	       map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH;
+}
+
+struct upsert_opts {
+	__u32 map_type;
+	int map_fd;
+	__u32 n;
+};
+
+static void *patch_map_thread(void *arg)
+{
+	struct upsert_opts *opts = arg;
+	void *key;
+	int val;
+	int ret;
+	int i;
+
+	for (i = 0; i < opts->n; i++) {
+		key = map_key(opts->map_type, i);
+		val = rand();
+		ret = bpf_map_update_elem(opts->map_fd, key, &val, 0);
+		CHECK(ret < 0, "bpf_map_update_elem", "error: %s\n", strerror(errno));
+	}
+	return NULL;
+}
+
+static void upsert_elements(struct upsert_opts *opts)
+{
+	pthread_t threads[N_THREADS];
+	int ret;
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(threads); i++) {
+		ret = pthread_create(&i[threads], NULL, patch_map_thread, opts);
+		CHECK(ret != 0, "pthread_create", "error: %s\n", strerror(ret));
+	}
+
+	for (i = 0; i < ARRAY_SIZE(threads); i++) {
+		ret = pthread_join(i[threads], NULL);
+		CHECK(ret != 0, "pthread_join", "error: %s\n", strerror(ret));
+	}
+}
+
+static __u32 read_cur_elements(int iter_fd)
+{
+	char buf[64];
+	ssize_t n;
+	__u32 ret;
+
+	n = read(iter_fd, buf, sizeof(buf)-1);
+	CHECK(n <= 0, "read", "error: %s\n", strerror(errno));
+	buf[n] = '\0';
+
+	errno = 0;
+	ret = (__u32)strtol(buf, NULL, 10);
+	CHECK(errno != 0, "strtol", "error: %s\n", strerror(errno));
+
+	return ret;
+}
+
+static __u32 get_cur_elements(int map_id)
+{
+	LIBBPF_OPTS(bpf_iter_attach_opts, opts);
+	union bpf_iter_link_info linfo;
+	struct map_percpu_stats *skel;
+	struct bpf_link *link;
+	int iter_fd;
+	int ret;
+
+	opts.link_info = &linfo;
+	opts.link_info_len = sizeof(linfo);
+
+	skel = map_percpu_stats__open();
+	CHECK(skel == NULL, "map_percpu_stats__open", "error: %s", strerror(errno));
+
+	skel->bss->target_id = map_id;
+
+	ret = map_percpu_stats__load(skel);
+	CHECK(ret != 0, "map_percpu_stats__load", "error: %s", strerror(errno));
+
+	link = bpf_program__attach_iter(skel->progs.dump_bpf_map, &opts);
+	CHECK(!link, "bpf_program__attach_iter", "error: %s\n", strerror(errno));
+
Quoted 8 lines

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 175

Quoted 4 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +4 -0
+	iter_fd = bpf_iter_create(bpf_link__fd(link));
+	CHECK(iter_fd < 0, "bpf_iter_create", "error: %s\n", strerror(errno));
+
+	return read_cur_elements(iter_fd);

Need to do close(iter_fd), bpf_link__destroy(link) and map__percpu_stats__destroy() before return, otherwise there will be resource leak.

Yes, thanks.

Quoted 25 lines
Quoted 25 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +25 -0
+}
+
+static void __test(int map_fd)
+{
+	__u32 n = MAX_ENTRIES - 1000;
+	__u32 real_current_elements;
+	__u32 iter_current_elements;
+	struct upsert_opts opts = {
+		.map_fd = map_fd,
+		.n = n,
+	};
+	struct bpf_map_info info;
+
+	map_info(map_fd, &info);
+	opts.map_type = info.type;
+
+	/*
+	 * Upsert keys [0, n) under some competition: with random values from
+	 * N_THREADS threads
+	 */
+	upsert_elements(&opts);
+
+	/*
+	 * The sum of percpu elements counters for all hashtable-based maps
+	 * should be equal to the number of elements present in the map. For
Quoted 8 lines

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 205

Quoted 5 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +5 -0
+	 * non-lru maps this number should be the number n of upserted
+	 * elements. For lru maps some elements might have been evicted. Check
+	 * that all numbers make sense
+	 */
+	map_info(map_fd, &info);

I think there is no need to call map_info() multiple times because the needed type and id will not be changed after creation.

Thanks, will fix. (This code left from the first version when the counter was returned by map_info().)

Quoted 129 lines

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 209

Quoted 4 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +4 -0
+	real_current_elements = map_count_elements(info.type, map_fd);
+	if (!is_lru(info.type))
+		CHECK(n != real_current_elements, "map_count_elements",
+		      "real_current_elements(%u) != expected(%u)\n", real_current_elements, n);

For LRU map, please use ASSERT_EQ(). For LRU map, should we check "n >= real_current_elements" instead ?

Quoted 5 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +5 -0
+
+	iter_current_elements = get_cur_elements(info.id);
+	CHECK(iter_current_elements != real_current_elements, "get_cur_elements",
+	      "iter_current_elements=%u, expected %u (map_type=%s,map_flags=%08x)\n",
+	      iter_current_elements, real_current_elements, map_type_to_s(info.type), info.map_flags);

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 224

Quoted 10 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +10 -0
+
+	/*
+	 * Cleanup the map and check that all elements are actually gone and
+	 * that the sum of percpu elements counters is back to 0 as well
+	 */
+	delete_all_elements(info.type, map_fd);
+	map_info(map_fd, &info);
+	real_current_elements = map_count_elements(info.type, map_fd);
+	CHECK(real_current_elements, "map_count_elements",
+	      "expected real_current_elements=0, got %u", real_current_elements);

ASSERT_EQ

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 230

Quoted 6 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +6 -0
+
+	iter_current_elements = get_cur_elements(info.id);
+	CHECK(iter_current_elements != 0, "get_cur_elements",
+	      "iter_current_elements=%u, expected 0 (map_type=%s,map_flags=%08x)\n",
+	      iter_current_elements, map_type_to_s(info.type), info.map_flags);
+

ASSERT_NEQ

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 242

Quoted 12 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +12 -0
+	close(map_fd);
+}
+
+static int map_create_opts(__u32 type, const char *name,
+			   struct bpf_map_create_opts *map_opts,
+			   __u32 key_size, __u32 val_size)
+{
+	int map_fd;
+
+	map_fd = bpf_map_create(type, name, key_size, val_size, MAX_ENTRIES, map_opts);
+	CHECK(map_fd < 0, "bpf_map_create()", "error:%s (name=%s)\n",
+			strerror(errno), name);

Please use ASSERT_GE instead.

Quoted 73 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +73 -0
+
+	return map_fd;
+}
+
+static int map_create(__u32 type, const char *name, struct bpf_map_create_opts *map_opts)
+{
+	return map_create_opts(type, name, map_opts, sizeof(int), sizeof(int));
+}
+
+static int create_hash(void)
+{
+	struct bpf_map_create_opts map_opts = {
+		.sz = sizeof(map_opts),
+		.map_flags = BPF_F_NO_PREALLOC,
+	};
+
+	return map_create(BPF_MAP_TYPE_HASH, "hash", &map_opts);
+}
+
+static int create_percpu_hash(void)
+{
+	struct bpf_map_create_opts map_opts = {
+		.sz = sizeof(map_opts),
+		.map_flags = BPF_F_NO_PREALLOC,
+	};
+
+	return map_create(BPF_MAP_TYPE_PERCPU_HASH, "percpu_hash", &map_opts);
+}
+
+static int create_hash_prealloc(void)
+{
+	return map_create(BPF_MAP_TYPE_HASH, "hash", NULL);
+}
+
+static int create_percpu_hash_prealloc(void)
+{
+	return map_create(BPF_MAP_TYPE_PERCPU_HASH, "percpu_hash_prealloc", NULL);
+}
+
+static int create_lru_hash(void)
+{
+	return map_create(BPF_MAP_TYPE_LRU_HASH, "lru_hash", NULL);
+}
+
+static int create_percpu_lru_hash(void)
+{
+	return map_create(BPF_MAP_TYPE_LRU_PERCPU_HASH, "lru_hash_percpu", NULL);
+}
+
+static void map_percpu_stats_hash(void)
+{
+	__test(create_hash());
+	printf("test_%s:PASS\n", __func__);
+}
+
+static void map_percpu_stats_percpu_hash(void)
+{
+	__test(create_percpu_hash());
+	printf("test_%s:PASS\n", __func__);
+}
+
+static void map_percpu_stats_hash_prealloc(void)
+{
+	__test(create_hash_prealloc());
+	printf("test_%s:PASS\n", __func__);
+}
+
+static void map_percpu_stats_percpu_hash_prealloc(void)
+{
+	__test(create_percpu_hash_prealloc());
+	printf("test_%s:PASS\n", __func__);
+}
+

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 325

Quoted 10 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +10 -0
+static void map_percpu_stats_lru_hash(void)
+{
+	__test(create_lru_hash());
+	printf("test_%s:PASS\n", __func__);
+}
+
+static void map_percpu_stats_percpu_lru_hash(void)
+{
+	__test(create_percpu_lru_hash());
+	printf("test_%s:PASS\n", __func__);

After switch to subtest, the printf() can be removed.

Quoted 13 lines

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 336

Quoted 11 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +11 -0
+}
+
+void test_map_percpu_stats(void)
+{
+	map_percpu_stats_hash();
+	map_percpu_stats_percpu_hash();
+	map_percpu_stats_hash_prealloc();
+	map_percpu_stats_percpu_hash_prealloc();
+	map_percpu_stats_lru_hash();
+	map_percpu_stats_percpu_lru_hash();
+}

Please use test__start_subtest() to create multiple subtests.

Thanks.

I will update this selftest in v4 with your comments addressed + batch ops tests.

Quoted 31 lines
Quoted 30 lines
tools/testing/selftests/bpf/progs/map_percpu_stats.c new file +24 -0
new file mode 100644
index 000000000000..10b2325c1720
@@ -0,0 +1,24 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2023 Isovalent */
+
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+__u32 target_id;
+
+__s64 bpf_map_sum_elem_count(struct bpf_map *map) __ksym;
+
+SEC("iter/bpf_map")
+int dump_bpf_map(struct bpf_iter__bpf_map *ctx)
+{
+	struct seq_file *seq = ctx->meta->seq;
+	struct bpf_map *map = ctx->map;
+
+	if (map && map->id == target_id)
+		BPF_SEQ_PRINTF(seq, "%lld", bpf_map_sum_elem_count(map));
+
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
#13 lore raw

Re: [v3 PATCH bpf-next 5/6] selftests/bpf: test map percpu stats

Quoted 5 lines · Anton Protopopov
Quoted 4 lines · Hou Tao

Hi,

On 6/30/2023 4:25 PM, Anton Protopopov wrote:
[...]

Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +5 -0
+}
+
+void test_map_percpu_stats(void)
+{
+	map_percpu_stats_hash();
Quoted 8 lines

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 336

Quoted 6 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +6 -0
+	map_percpu_stats_percpu_hash();
+	map_percpu_stats_hash_prealloc();
+	map_percpu_stats_percpu_hash_prealloc();
+	map_percpu_stats_lru_hash();
+	map_percpu_stats_percpu_lru_hash();
+}

Please use test__start_subtest() to create multiple subtests.

After looking at code, I think that I will leave the individual functions here, as the test__start_subtest() function is only implemented in test_progs (not test_maps), and adding it here looks like out of scope for this patch.

#14 lore raw

Re: [v3 PATCH bpf-next 5/6] selftests/bpf: test map percpu stats

Quoted 20 lines · Anton Protopopov
Quoted 5 lines · Anton Protopopov
Quoted 4 lines · Hou Tao

Hi,

On 6/30/2023 4:25 PM, Anton Protopopov wrote:
[...]

Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +5 -0
+}
+
+void test_map_percpu_stats(void)
+{
+	map_percpu_stats_hash();
Quoted 8 lines

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 336

Quoted 6 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +6 -0
+	map_percpu_stats_percpu_hash();
+	map_percpu_stats_hash_prealloc();
+	map_percpu_stats_percpu_hash_prealloc();
+	map_percpu_stats_lru_hash();
+	map_percpu_stats_percpu_lru_hash();
+}

Please use test__start_subtest() to create multiple subtests.

Quoted 3 lines

After looking at code, I think that I will leave the individual functions here, as the test__start_subtest() function is only implemented in test_progs (not test_maps), and adding it here looks like out of scope for this patch.

Ah, sorry, looks like the same stands for ASSERT* macros as well, as they are only used in test_progs. (I will still fix the checks where you commented on specific values, like n <= cur_elems for LRUs.)

#15 lore raw

Re: [v3 PATCH bpf-next 5/6] selftests/bpf: test map percpu stats

In reply to Anton Protopopov

Hi,

Quoted 18 lines · Anton Protopopov
Quoted 5 lines · Anton Protopopov
Quoted 4 lines · Hou Tao

Hi,

On 6/30/2023 4:25 PM, Anton Protopopov wrote:
[...]

Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +4 -0
+}
+
+void test_map_percpu_stats(void)
+{
Quoted 8 lines

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 336

Quoted 7 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +7 -0
+	map_percpu_stats_hash();
+	map_percpu_stats_percpu_hash();
+	map_percpu_stats_hash_prealloc();
+	map_percpu_stats_percpu_hash_prealloc();
+	map_percpu_stats_lru_hash();
+	map_percpu_stats_percpu_lru_hash();
+}

Please use test__start_subtest() to create multiple subtests.

Quoted 4 lines

After looking at code, I think that I will leave the individual functions here, as the test__start_subtest() function is only implemented in test_progs (not test_maps), and adding it here looks like out of scope for this patch. .

I see. But can we just add these tests in test_progs instead which is more flexible ?

#16 lore raw

Re: [v3 PATCH bpf-next 5/6] selftests/bpf: test map percpu stats

Quoted 21 lines · Hou Tao

Hi,

Quoted 18 lines · Anton Protopopov
Quoted 17 lines · Anton Protopopov
Quoted 16 lines · Hou Tao

Hi,

On 6/30/2023 4:25 PM, Anton Protopopov wrote:
[...]

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 336

Quoted 11 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +11 -0
+}
+
+void test_map_percpu_stats(void)
+{
+	map_percpu_stats_hash();
+	map_percpu_stats_percpu_hash();
+	map_percpu_stats_hash_prealloc();
+	map_percpu_stats_percpu_hash_prealloc();
+	map_percpu_stats_lru_hash();
+	map_percpu_stats_percpu_lru_hash();
+}

Please use test__start_subtest() to create multiple subtests.

Quoted 6 lines
Quoted 4 lines

After looking at code, I think that I will leave the individual functions here, as the test__start_subtest() function is only implemented in test_progs (not test_maps), and adding it here looks like out of scope for this patch. .

I see. But can we just add these tests in test_progs instead which is more flexible ?

I think that it makes more sense to port this test_prog flexibility to the test_maps program. I can volunteer to do this (but not right away).

#17 lore raw

Re: [v3 PATCH bpf-next 5/6] selftests/bpf: test map percpu stats

In reply to Anton Protopopov

Hi,

Quoted 11 lines · Anton Protopopov
Quoted 10 lines · Hou Tao

Hi,

Quoted 7 lines · Anton Protopopov

Add a new map test, map_percpu_stats.c, which is checking the correctness of map's percpu elements counters. For supported maps the test upserts a number of elements, checks the correctness of the counters, then deletes all the elements and checks again that the counters sum drops down to zero.

The following map types are tested:

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

* BPF_MAP_TYPE_HASH, BPF_F_NO_PREALLOC
* BPF_MAP_TYPE_PERCPU_HASH, BPF_F_NO_PREALLOC
* BPF_MAP_TYPE_HASH,
* BPF_MAP_TYPE_PERCPU_HASH,
* BPF_MAP_TYPE_LRU_HASH
* BPF_MAP_TYPE_LRU_PERCPU_HASH

A test for BPF_MAP_TYPE_HASH_OF_MAPS is also needed.

We could also exercise the test for LRU map with BPF_F_NO_COMMON_LRU.

Quoted 1 line

SNIP

Quoted 24 lines
Quoted 16 lines
Quoted 16 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +10 -0
new file mode 100644
index 000000000000..5b45af230368
@@ -0,0 +1,336 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2023 Isovalent */
+
+#include <errno.h>
+#include <unistd.h>
+#include <pthread.h>
+
+#include <bpf/bpf.h>
+#include <bpf/libbpf.h>
+
Quoted 8 lines

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 17

Quoted 7 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +7 -0
+#include <bpf_util.h>
+#include <test_maps.h>
+
+#include "map_percpu_stats.skel.h"
+
+#define MAX_ENTRIES 16384
+#define N_THREADS 37

Why 37 thread is needed here ? Does a small number of threads work as well ?

Quoted 1 line

This was used to evict more elements from LRU maps when they are full.

I see. But in my understanding, for the global LRU list, the eviction (the invocation of htab_lru_map_delete_node) will be possible if the free element is less than LOCAL_FREE_TARGET(128) * nr_running_cpus. Now the number of free elements is 1000 as defined in __test(), the number of vCPU is 8 in my local VM setup (BPF CI also uses 8 vCPUs) and it is hard to trigger the eviction because 8 * 128 is roughly equal with 1000. So I suggest to decrease the number of free elements to 512 and the number of threads to 8, or adjust the number of running thread and free elements according to the number of online CPUs.

#18 lore raw

Re: [v3 PATCH bpf-next 5/6] selftests/bpf: test map percpu stats

Quoted 14 lines · Hou Tao

Hi,

Quoted 11 lines · Anton Protopopov
Quoted 10 lines · Hou Tao

Hi,

Quoted 7 lines · Anton Protopopov

Add a new map test, map_percpu_stats.c, which is checking the correctness of map's percpu elements counters. For supported maps the test upserts a number of elements, checks the correctness of the counters, then deletes all the elements and checks again that the counters sum drops down to zero.

The following map types are tested:

Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

* BPF_MAP_TYPE_HASH, BPF_F_NO_PREALLOC
* BPF_MAP_TYPE_PERCPU_HASH, BPF_F_NO_PREALLOC
* BPF_MAP_TYPE_HASH,
* BPF_MAP_TYPE_PERCPU_HASH,
* BPF_MAP_TYPE_LRU_HASH
* BPF_MAP_TYPE_LRU_PERCPU_HASH

A test for BPF_MAP_TYPE_HASH_OF_MAPS is also needed.

We could also exercise the test for LRU map with BPF_F_NO_COMMON_LRU.

Thanks, added.

Quoted 28 lines
Quoted 1 line

SNIP

Quoted 24 lines
Quoted 16 lines
Quoted 16 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +10 -0
new file mode 100644
index 000000000000..5b45af230368
@@ -0,0 +1,336 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2023 Isovalent */
+
+#include <errno.h>
+#include <unistd.h>
+#include <pthread.h>
+
+#include <bpf/bpf.h>
+#include <bpf/libbpf.h>
+
Quoted 8 lines

tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 17

Quoted 7 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +7 -0
+#include <bpf_util.h>
+#include <test_maps.h>
+
+#include "map_percpu_stats.skel.h"
+
+#define MAX_ENTRIES 16384
+#define N_THREADS 37

Why 37 thread is needed here ? Does a small number of threads work as well ?

Quoted 1 line

This was used to evict more elements from LRU maps when they are full.

Quoted 9 lines

I see. But in my understanding, for the global LRU list, the eviction (the invocation of htab_lru_map_delete_node) will be possible if the free element is less than LOCAL_FREE_TARGET(128) * nr_running_cpus. Now the number of free elements is 1000 as defined in __test(), the number of vCPU is 8 in my local VM setup (BPF CI also uses 8 vCPUs) and it is hard to trigger the eviction because 8 * 128 is roughly equal with 1000. So I suggest to decrease the number of free elements to 512 and the number of threads to 8, or adjust the number of running thread and free elements according to the number of online CPUs.

Yes, makes sense. I've changed the test to use 8 threads and offset of 512.

#19 lore raw

PATCH v3 6/6 selftests/bpf: check that ->elem_count is non-zero for the hash map

In reply to Anton Protopopov

Previous commits populated the ->elem_count per-cpu pointer for hash maps. Check that this pointer is non-NULL in an existing map.

---
 tools/testing/selftests/bpf/progs/map_ptr_kern.c | 3 +++
 1 file changed, 3 insertions(+)
tools/testing/selftests/bpf/progs/map_ptr_kern.c +3 -0
index db388f593d0a..d6e234a37ccb 100644
@@ -33,6 +33,7 @@ struct bpf_map {
 	__u32 value_size;
 	__u32 max_entries;
 	__u32 id;
+	__s64 *elem_count;
 } __attribute__((preserve_access_index));
 
 static inline int check_bpf_map_fields(struct bpf_map *map, __u32 key_size,
@@ -111,6 +112,8 @@ static inline int check_hash(void)
 
 	VERIFY(check_default_noinline(&hash->map, map));
 
+	VERIFY(map->elem_count != NULL);
+
 	VERIFY(hash->n_buckets == MAX_ENTRIES);
 	VERIFY(hash->elem_size == 64);
 
Signature · 1 line
2.34.1