bpf: add percpu stats for bpf_map
Subsystems (5)
v3 cover bpf: add percpu stats for bpf_map
This series adds a mechanism for maps to populate per-cpu counters on insertions/deletions. The sum of these counters can be accessed by a new kfunc from map iterator and tracing programs.
The following patches are present in the series:
* Patch 1 adds a generic per-cpu counter to struct bpf_map
* Patch 2 adds a new kfunc to access the sum of per-cpu counters
* Patch 3 utilizes this mechanism for hash-based maps
* Patch 4 extends the preloaded map iterator to dump the sum
* Patch 5 adds a self-test for the change
* Patch 6 patches map_ptr selftest to check that elem_count was initialized
The reason for adding this functionality in our case (Cilium) is to get signals about how full some heavy-used maps are and what the actual dynamic profile of map capacity is. In the case of LRU maps this is impossible to get this information anyhow else. The original presentation can be found here [1].
[1] https://lpc.events/event/16/contributions/1368/
v2 -> v3:
- split commits to better represent update logic
- remove filter from kfunc to allow all tracing programs
- extend selftests
v1 -> v2:
- make the counters generic part of struct bpf_map
- don't use map_info and /proc/self/fdinfo in favor of a kfunc
Anton Protopopov (6):
bpf: add percpu stats for bpf_map elements insertions/deletions
bpf: add a new kfunc to return current bpf_map elements count
bpf: populate the per-cpu insertions/deletions counters for hashmaps
bpf: make preloaded map iterators to display map elements count selftests/bpf: test map percpu stats
selftests/bpf: check that ->elem_count is non-zero for the hash map
include/linux/bpf.h | 30 + kernel/bpf/hashtab.c | 23 +- kernel/bpf/map_iter.c | 39 +- kernel/bpf/preload/iterators/iterators.bpf.c | 9 +- .../iterators/iterators.lskel-little-endian.h | 526 +++++++++--------- .../bpf/map_tests/map_percpu_stats.c | 336 +++++++++++ .../selftests/bpf/progs/map_percpu_stats.c | 24 + .../selftests/bpf/progs/map_ptr_kern.c | 3 + 8 files changed, 726 insertions(+), 264 deletions(-) create mode 100644 tools/testing/selftests/bpf/map_tests/map_percpu_stats.c create mode 100644 tools/testing/selftests/bpf/progs/map_percpu_stats.c
Signature · 1 line
2.34.1
PATCH v3 1/6 bpf: add percpu stats for bpf_map elements insertions/deletions
Add a generic percpu stats for bpf_map elements insertions/deletions in order to keep track of both, the current (approximate) number of elements in a map and per-cpu statistics on update/delete operations.
To expose these stats a particular map implementation should initialize the counter and adjust it as needed using the 'bpf_map_*_elem_count' helpers provided by this commit.
--- include/linux/bpf.h | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+)
include/linux/bpf.h +30 -0
@@ -275,6 +275,7 @@ struct bpf_map { } owner;bool bypass_spec_v1;bool frozen;/* write-once; write-protected by freeze_mutex */ +s64 __percpu *elem_count; };static inline const char *btf_field_type_name (enum btf_field_type type)
@@ -2040,6 +2041,35 @@ bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, size_t align, }#endif +static inline int +bpf_map_init_elem_count (struct bpf_map *map) +{ +size_t size =sizeof (*map->elem_count), align = size; +gfp_t flags =GFP_USER | __GFP_NOWARN; + + map->elem_count =bpf_map_alloc_percpu (map, size, align, flags); +if (!map->elem_count) +return -ENOMEM ; + +return 0 ; +} + +static inline void +bpf_map_free_elem_count (struct bpf_map *map) +{ +free_percpu (map->elem_count); +} + +static inline void bpf_map_inc_elem_count (struct bpf_map *map) +{ +this_cpu_inc (*map->elem_count); +} + +static inline void bpf_map_dec_elem_count (struct bpf_map *map) +{ +this_cpu_dec (*map->elem_count); +} +extern int sysctl_unprivileged_bpf_disabled;static inline bool bpf_allow_ptr_leaks (void )
Signature · 1 line
2.34.1
PATCH v3 2/6 bpf: add a new kfunc to return current bpf_map elements count
In reply to Anton ProtopopovA bpf_map_sum_elem_count kfunc was added to simplify getting the sum of the map per-cpu element counters. If a map doesn't implement the counter, then the function will always return 0.
--- kernel/bpf/map_iter.c | 39 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-)
kernel/bpf/map_iter.c +38 -1
@@ -93,7 +93,7 @@ static struct bpf_iter_reg bpf_map_reg_info = { .ctx_arg_info_size = 1, .ctx_arg_info = { { offsetof(struct bpf_iter__bpf_map, map), - PTR_TO_BTF_ID_OR_NULL }, + PTR_TO_BTF_ID_OR_NULL | PTR_TRUSTED }, }, .seq_info = &bpf_map_seq_info, };
@@ -193,3 +193,40 @@ static int __init bpf_map_iter_init(void) }late_initcall (bpf_map_iter_init); + +__diag_push (); +__diag_ignore_all ("-Wmissing-prototypes" , +"Global functions as their definitions will be in vmlinux BTF" ); + +__bpf_kfunc s64 bpf_map_sum_elem_count(struct bpf_map *map) +{ +s64 *pcount; +s64 ret =0 ; +int cpu; + +if (!map || !map->elem_count) +return 0 ; + +for_each_possible_cpu (cpu) { + pcount =per_cpu_ptr (map->elem_count, cpu); + ret +=READ_ONCE (*pcount); + } +return ret; +} + +__diag_pop (); + +BTF_SET8_START (bpf_map_iter_kfunc_ids) +BTF_ID_FLAGS (func ,bpf_map_sum_elem_count ,KF_TRUSTED_ARGS ) +BTF_SET8_END (bpf_map_iter_kfunc_ids) + +static const struct btf_kfunc_id_set bpf_map_iter_kfunc_set = { + .owner =THIS_MODULE , + .set = &bpf_map_iter_kfunc_ids, +}; + +static int init_subsystem (void ) +{ +return register_btf_kfunc_id_set (BPF_PROG_TYPE_TRACING , &bpf_map_iter_kfunc_set); +} +late_initcall (init_subsystem);
Signature · 1 line
2.34.1
PATCH v3 3/6 bpf: populate the per-cpu insertions/deletions counters for hashmaps
In reply to Anton ProtopopovInitialize and utilize the per-cpu insertions/deletions counters for hash-based maps. Non-trivial changes only apply to the preallocated maps for which the {inc,dec}_elem_count functions are not called, as there's no need in counting elements to sustain proper map operations.
To increase/decrease percpu counters for preallocated maps we add raw calls to the bpf_map_{inc,dec}_elem_count functions so that the impact is minimal. For dynamically allocated maps we add corresponding calls to the existing {inc,dec}_elem_count functions.
--- kernel/bpf/hashtab.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)
kernel/bpf/hashtab.c +20 -3
@@ -581,8 +581,14 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr) } } + err =bpf_map_init_elem_count (&htab->map); +if (err) + goto free_extra_elements; +return &htab->map; +free_extra_elements: +free_percpu (htab->extra_elems); free_prealloc:prealloc_destroy (htab); free_map_locked:
@@ -804,6 +810,7 @@ static bool htab_lru_map_delete_node(void *arg, struct bpf_lru_node *node)if (l == tgt_l) {hlist_nulls_del_rcu (&l->hash_node);check_and_free_fields (htab, l); +bpf_map_dec_elem_count (&htab->map);break ; }
@@ -900,6 +907,8 @@ static bool is_map_full(struct bpf_htab *htab)static void inc_elem_count (struct bpf_htab *htab) { +bpf_map_inc_elem_count (&htab->map); +if (htab->use_percpu_counter)percpu_counter_add_batch (&htab->pcount,1 ,PERCPU_COUNTER_BATCH );else
@@ -908,6 +917,8 @@ static void inc_elem_count(struct bpf_htab *htab)static void dec_elem_count (struct bpf_htab *htab) { +bpf_map_dec_elem_count (&htab->map); +if (htab->use_percpu_counter)percpu_counter_add_batch (&htab->pcount,-1 ,PERCPU_COUNTER_BATCH );else
@@ -920,6 +931,7 @@ static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l)htab_put_fd_value (htab, l);if (htab_is_prealloc (htab)) { +bpf_map_dec_elem_count (&htab->map);check_and_free_fields (htab, l);__pcpu_freelist_push (&htab->freelist, &l->fnode); }else {
@@ -1000,6 +1012,7 @@ static struct htab_elem *alloc_htab_elem(struct bpf_htab *htab, void *key,if (!l)return ERR_PTR (-E2BIG ); l_new = container_of(l, struct htab_elem, fnode); +bpf_map_inc_elem_count (&htab->map); } }else {if (is_map_full (htab))
@@ -1224,7 +1237,8 @@ static long htab_lru_map_update_elem(struct bpf_map *map, void *key, void *valueif (l_old) {bpf_lru_node_set_ref (&l_new->lru_node);hlist_nulls_del_rcu (&l_old->hash_node); - } + }else +bpf_map_inc_elem_count (&htab->map); ret =0 ; err:
@@ -1351,6 +1365,7 @@ static long __htab_lru_percpu_map_update_elem(struct bpf_map *map, void *key,pcpu_init_value (htab,htab_elem_get_ptr (l_new, key_size), value, onallcpus);hlist_nulls_add_head_rcu (&l_new->hash_node, head); +bpf_map_inc_elem_count (&htab->map); l_new =NULL ; } ret =0 ;
@@ -1437,9 +1452,10 @@ static long htab_lru_map_delete_elem(struct bpf_map *map, void *key) l =lookup_elem_raw (head, hash, key, key_size); -if (l) +if (l) { +bpf_map_dec_elem_count (&htab->map);hlist_nulls_del_rcu (&l->hash_node); -else + }else ret = -ENOENT ;htab_unlock_bucket (htab, b, hash, flags);
@@ -1523,6 +1539,7 @@ static void htab_map_free(struct bpf_map *map)prealloc_destroy (htab); } +bpf_map_free_elem_count (map);free_percpu (htab->extra_elems);bpf_map_area_free (htab->buckets);bpf_mem_alloc_destroy (&htab->pcpu_ma);
Signature · 1 line
2.34.1
Re: [v3 PATCH bpf-next 3/6] bpf: populate the per-cpu insertions/deletions counters for hashmaps
Hi,
Quoted 20 lines · Anton Protopopov
Initialize and utilize the per-cpu insertions/deletions counters for hash-based maps. Non-trivial changes only apply to the preallocated maps for which the {inc,dec}_elem_count functions are not called, as there's no need in counting elements to sustain proper map operations.
To increase/decrease percpu counters for preallocated maps we add raw calls to the bpf_map_{inc,dec}_elem_count functions so that the impact is minimal. For dynamically allocated maps we add corresponding calls to the existing {inc,dec}_elem_count functions.
Signed-off-by Anton Protopopov <[email protected]>--- kernel/bpf/hashtab.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)kernel/bpf/hashtab.c +0
-0
kernel/bpf/hashtab.c static struct bpf_map *htab_map_alloc(union bpf_attr *attr) · line 586
Quoted 6 lines
kernel/bpf/hashtab.c +3
-0@@ -581,8 +581,14 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr) } } + err =bpf_map_init_elem_count (&htab->map); +if (err) + goto free_extra_elements;
Considering the per-cpu counter is not always needed, is it a good idea to make the elem_count being optional by introducing a new map flag ?
kernel/bpf/hashtab.c static struct bpf_map *htab_map_alloc(union bpf_attr *attr) · line 593
Quoted 7 lines
kernel/bpf/hashtab.c +3
-0+return &htab->map; +free_extra_elements: +free_percpu (htab->extra_elems); free_prealloc:prealloc_destroy (htab);
Need to check prealloc before calling prealloc_destroy(htab), otherwise for non-preallocated percpu htab prealloc_destroy() will trigger invalid memory dereference.
Quoted 62 lines
kernel/bpf/hashtab.c +10
-1free_map_locked:@@ -804,6 +810,7 @@ static bool htab_lru_map_delete_node(void *arg, struct bpf_lru_node *node)if (l == tgt_l) {hlist_nulls_del_rcu (&l->hash_node);check_and_free_fields (htab, l); +bpf_map_dec_elem_count (&htab->map);break ; }@@ -900,6 +907,8 @@ static bool is_map_full(struct bpf_htab *htab)static void inc_elem_count (struct bpf_htab *htab) { +bpf_map_inc_elem_count (&htab->map); +if (htab->use_percpu_counter)percpu_counter_add_batch (&htab->pcount,1 ,PERCPU_COUNTER_BATCH );else @@ -908,6 +917,8 @@ static void inc_elem_count(struct bpf_htab *htab)static void dec_elem_count (struct bpf_htab *htab) { +bpf_map_dec_elem_count (&htab->map); +if (htab->use_percpu_counter)percpu_counter_add_batch (&htab->pcount,-1 ,PERCPU_COUNTER_BATCH );else @@ -920,6 +931,7 @@ static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l)htab_put_fd_value (htab, l);if (htab_is_prealloc (htab)) { +bpf_map_dec_elem_count (&htab->map);check_and_free_fields (htab, l);__pcpu_freelist_push (&htab->freelist, &l->fnode); }else {@@ -1000,6 +1012,7 @@ static struct htab_elem *alloc_htab_elem(struct bpf_htab *htab, void *key,if (!l)return ERR_PTR (-E2BIG ); l_new = container_of(l, struct htab_elem, fnode); +bpf_map_inc_elem_count (&htab->map); } }else {if (is_map_full (htab))@@ -1224,7 +1237,8 @@ static long htab_lru_map_update_elem(struct bpf_map *map, void *key, void *valueif (l_old) {bpf_lru_node_set_ref (&l_new->lru_node);hlist_nulls_del_rcu (&l_old->hash_node); - } + }else +bpf_map_inc_elem_count (&htab->map); ret =0 ; err:@@ -1351,6 +1365,7 @@ static long __htab_lru_percpu_map_update_elem(struct bpf_map *map, void *key,pcpu_init_value (htab,htab_elem_get_ptr (l_new, key_size), value, onallcpus);hlist_nulls_add_head_rcu (&l_new->hash_node, head); +bpf_map_inc_elem_count (&htab->map); l_new =NULL ; } ret =0 ;
kernel/bpf/hashtab.c static long htab_lru_map_delete_elem(struct bpf_map *map, void *key) · line 1459
Quoted 10 lines
kernel/bpf/hashtab.c +3
-2@@ -1437,9 +1452,10 @@ static long htab_lru_map_delete_elem(struct bpf_map *map, void *key) l =lookup_elem_raw (head, hash, key, key_size); -if (l) +if (l) { +bpf_map_dec_elem_count (&htab->map);hlist_nulls_del_rcu (&l->hash_node); -else + }else ret = -ENOENT ;
Also need to decrease elem_count for __htab_map_lookup_and_delete_batch() and __htab_map_lookup_and_delete_elem() when is_lru_map is true. Maybe for LRU map, we could just do bpf_map_dec_elem_count() in htab_lru_push_free() and do bpf_map_inc_elem_count() in prealloc_lru_pop().
Quoted 10 lines
kernel/bpf/hashtab.c +1
-0htab_unlock_bucket (htab, b, hash, flags);@@ -1523,6 +1539,7 @@ static void htab_map_free(struct bpf_map *map)prealloc_destroy (htab); } +bpf_map_free_elem_count (map);free_percpu (htab->extra_elems);bpf_map_area_free (htab->buckets);bpf_mem_alloc_destroy (&htab->pcpu_ma);
Re: [v3 PATCH bpf-next 3/6] bpf: populate the per-cpu insertions/deletions counters for hashmaps
Quoted 26 lines · Hou Tao
Hi,
Quoted 23 lines · Anton Protopopov
Initialize and utilize the per-cpu insertions/deletions counters for hash-based maps. Non-trivial changes only apply to the preallocated maps for which the {inc,dec}_elem_count functions are not called, as there's no need in counting elements to sustain proper map operations.
To increase/decrease percpu counters for preallocated maps we add raw calls to the bpf_map_{inc,dec}_elem_count functions so that the impact is minimal. For dynamically allocated maps we add corresponding calls to the existing {inc,dec}_elem_count functions.
Signed-off-by Anton Protopopov <[email protected]>--- kernel/bpf/hashtab.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)kernel/bpf/hashtab.c +0
-0@@ -581,8 +581,14 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr) } }
Quoted 5 lines
kernel/bpf/hashtab.c static struct bpf_map *htab_map_alloc(union bpf_attr *attr) · line 586
Quoted 3 lines
kernel/bpf/hashtab.c +3
-0+ err =bpf_map_init_elem_count (&htab->map); +if (err) + goto free_extra_elements;Considering the per-cpu counter is not always needed, is it a good idea to make the
elem_countbeing optional by introducing a new map flag ?
Per-map-flag or a static key? For me it looked like just doing an unconditional inc for a per-cpu variable is better vs. doing a check then inc or an unconditional jump.
Quoted 10 lines
kernel/bpf/hashtab.c static struct bpf_map *htab_map_alloc(union bpf_attr *attr) · line 593
Quoted 7 lines
kernel/bpf/hashtab.c +3
-0+return &htab->map; +free_extra_elements: +free_percpu (htab->extra_elems); free_prealloc:prealloc_destroy (htab);Need to check prealloc before calling
prealloc_destroy(htab), otherwise for non-preallocated percpu htabprealloc_destroy()will trigger invalid memory dereference.
Thanks!
Quoted 65 lines
Quoted 65 lines
kernel/bpf/hashtab.c +10
-1free_map_locked:@@ -804,6 +810,7 @@ static bool htab_lru_map_delete_node(void *arg, struct bpf_lru_node *node)if (l == tgt_l) {hlist_nulls_del_rcu (&l->hash_node);check_and_free_fields (htab, l); +bpf_map_dec_elem_count (&htab->map);break ; }@@ -900,6 +907,8 @@ static bool is_map_full(struct bpf_htab *htab)static void inc_elem_count (struct bpf_htab *htab) { +bpf_map_inc_elem_count (&htab->map); +if (htab->use_percpu_counter)percpu_counter_add_batch (&htab->pcount,1 ,PERCPU_COUNTER_BATCH );else @@ -908,6 +917,8 @@ static void inc_elem_count(struct bpf_htab *htab)static void dec_elem_count (struct bpf_htab *htab) { +bpf_map_dec_elem_count (&htab->map); +if (htab->use_percpu_counter)percpu_counter_add_batch (&htab->pcount,-1 ,PERCPU_COUNTER_BATCH );else @@ -920,6 +931,7 @@ static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l)htab_put_fd_value (htab, l);if (htab_is_prealloc (htab)) { +bpf_map_dec_elem_count (&htab->map);check_and_free_fields (htab, l);__pcpu_freelist_push (&htab->freelist, &l->fnode); }else {@@ -1000,6 +1012,7 @@ static struct htab_elem *alloc_htab_elem(struct bpf_htab *htab, void *key,if (!l)return ERR_PTR (-E2BIG ); l_new = container_of(l, struct htab_elem, fnode); +bpf_map_inc_elem_count (&htab->map); } }else {if (is_map_full (htab))@@ -1224,7 +1237,8 @@ static long htab_lru_map_update_elem(struct bpf_map *map, void *key, void *valueif (l_old) {bpf_lru_node_set_ref (&l_new->lru_node);hlist_nulls_del_rcu (&l_old->hash_node); - } + }else +bpf_map_inc_elem_count (&htab->map); ret =0 ; err:@@ -1351,6 +1365,7 @@ static long __htab_lru_percpu_map_update_elem(struct bpf_map *map, void *key,pcpu_init_value (htab,htab_elem_get_ptr (l_new, key_size), value, onallcpus);hlist_nulls_add_head_rcu (&l_new->hash_node, head); +bpf_map_inc_elem_count (&htab->map); l_new =NULL ; } ret =0 ;@@ -1437,9 +1452,10 @@ static long htab_lru_map_delete_elem(struct bpf_map *map, void *key) l =lookup_elem_raw (head, hash, key, key_size);
Quoted 12 lines
kernel/bpf/hashtab.c static long htab_lru_map_delete_elem(struct bpf_map *map, void *key) · line 1459
Quoted 7 lines
kernel/bpf/hashtab.c +3
-2-if (l) +if (l) { +bpf_map_dec_elem_count (&htab->map);hlist_nulls_del_rcu (&l->hash_node); -else + }else ret = -ENOENT ;Also need to decrease
elem_countfor__htab_map_lookup_and_delete_batch()and__htab_map_lookup_and_delete_elem()whenis_lru_mapis true. Maybe for LRU map, we could just dobpf_map_dec_elem_count()inhtab_lru_push_free()and dobpf_map_inc_elem_count()inprealloc_lru_pop().
Thanks. I will fix the logic and extend the selftest to test the batch ops as well.
Quoted 11 lines
Quoted 10 lines
kernel/bpf/hashtab.c +1
-0htab_unlock_bucket (htab, b, hash, flags);@@ -1523,6 +1539,7 @@ static void htab_map_free(struct bpf_map *map)prealloc_destroy (htab); } +bpf_map_free_elem_count (map);free_percpu (htab->extra_elems);bpf_map_area_free (htab->buckets);bpf_mem_alloc_destroy (&htab->pcpu_ma);
Re: [v3 PATCH bpf-next 3/6] bpf: populate the per-cpu insertions/deletions counters for hashmaps
Hi,
Quoted 27 lines · Anton Protopopov
Quoted 26 lines · Hou Tao
Hi,
Quoted 23 lines · Anton Protopopov
Initialize and utilize the per-cpu insertions/deletions counters for hash-based maps. Non-trivial changes only apply to the preallocated maps for which the {inc,dec}_elem_count functions are not called, as there's no need in counting elements to sustain proper map operations.
To increase/decrease percpu counters for preallocated maps we add raw calls to the bpf_map_{inc,dec}_elem_count functions so that the impact is minimal. For dynamically allocated maps we add corresponding calls to the existing {inc,dec}_elem_count functions.
Signed-off-by Anton Protopopov <[email protected]>--- kernel/bpf/hashtab.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)kernel/bpf/hashtab.c +0
-0@@ -581,8 +581,14 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr) } }
Quoted 8 lines
Quoted 5 lines
kernel/bpf/hashtab.c static struct bpf_map *htab_map_alloc(union bpf_attr *attr) · line 586
Quoted 3 lines
kernel/bpf/hashtab.c +3
-0+ err =bpf_map_init_elem_count (&htab->map); +if (err) + goto free_extra_elements;Considering the per-cpu counter is not always needed, is it a good idea to make the
elem_countbeing optional by introducing a new map flag ?Per-map-flag or a static key? For me it looked like just doing an unconditional
incfor a per-cpu variable is better vs. doing a check thenincor an unconditional jump.
Sorry I didn't make it clear that I was worried about the allocated per-cpu memory. Previous I thought the per-cpu memory is limited, but after did some experiments I found it was almost the same as kmalloc() which could use all available memory to fulfill the allocation request. For a host with 72-cpus, the memory overhead for 10k hash map is about ~6MB. The overhead is tiny compared with the total available memory, but it is avoidable.
Re: [v3 PATCH bpf-next 3/6] bpf: populate the per-cpu insertions/deletions counters for hashmaps
Quoted 39 lines · Hou Tao
Hi,
Quoted 27 lines · Anton Protopopov
Quoted 26 lines · Hou Tao
Hi,
Quoted 23 lines · Anton Protopopov
Initialize and utilize the per-cpu insertions/deletions counters for hash-based maps. Non-trivial changes only apply to the preallocated maps for which the {inc,dec}_elem_count functions are not called, as there's no need in counting elements to sustain proper map operations.
To increase/decrease percpu counters for preallocated maps we add raw calls to the bpf_map_{inc,dec}_elem_count functions so that the impact is minimal. For dynamically allocated maps we add corresponding calls to the existing {inc,dec}_elem_count functions.
Signed-off-by Anton Protopopov <[email protected]>--- kernel/bpf/hashtab.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)kernel/bpf/hashtab.c +0
-0@@ -581,8 +581,14 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr) } }Quoted 8 lines
Quoted 5 lines
kernel/bpf/hashtab.c static struct bpf_map *htab_map_alloc(union bpf_attr *attr) · line 586
Quoted 3 lines
kernel/bpf/hashtab.c +3
-0+ err =bpf_map_init_elem_count (&htab->map); +if (err) + goto free_extra_elements;Considering the per-cpu counter is not always needed, is it a good idea to make the
elem_countbeing optional by introducing a new map flag ?Per-map-flag or a static key? For me it looked like just doing an unconditional
incfor a per-cpu variable is better vs. doing a check thenincor an unconditional jump.
Quoted 7 lines
Sorry I didn't make it clear that I was worried about the allocated per-cpu memory. Previous I thought the per-cpu memory is limited, but after did some experiments I found it was almost the same as
kmalloc()which could use all available memory to fulfill the allocation request. For a host with 72-cpus, the memory overhead for 10k hash map is about ~6MB. The overhead is tiny compared with the total available memory, but it is avoidable.
So, in my first patch I've only added new counters for preallocated maps. But then the feedback was that we need a generic percpu inc/dec counters, so I added them by default. For me a percpu s64 looks cheap enough for a hash map...
Re: [v3 PATCH bpf-next 3/6] bpf: populate the per-cpu insertions/deletions counters for hashmaps
Hi,
Quoted 5 lines · Anton Protopopov
Quoted 4 lines · Hou Tao
Hi,
Quoted 1 line · Anton Protopopov
On Tue, Jul 04, 2023 at 09:56:36PM +0800, Hou Tao wrote:
SNIP
Quoted 11 lines
Quoted 11 lines
Quoted 4 lines
Quoted 1 line
by introducing a new map flag ?
Per-map-flag or a static key? For me it looked like just doing an unconditional
incfor a per-cpu variable is better vs. doing a check thenincor an unconditional jump.Sorry I didn't make it clear that I was worried about the allocated per-cpu memory. Previous I thought the per-cpu memory is limited, but after did some experiments I found it was almost the same as
kmalloc()which could use all available memory to fulfill the allocation request. For a host with 72-cpus, the memory overhead for 10k hash map is about ~6MB. The overhead is tiny compared with the total available memory, but it is avoidable.
Quoted 3 lines
So, in my first patch I've only added new counters for preallocated maps. But then the feedback was that we need a generic percpu inc/dec counters, so I added them by default. For me a percpu s64 looks cheap enough for a hash map...
Thanks for the explanation. Let's just allocate the per-cpu elem_count in hash map. If there are use cases which need to make it optional, we can revise that later.
Quoted 1 line
.
PATCH v3 4/6 bpf: make preloaded map iterators to display map elements count
In reply to Anton ProtopopovAdd another column to the /sys/fs/bpf/maps.debug iterator to display cur_entries, the current number of entries in the map as is returned by the bpf_map_sum_elem_count kfunc. Also fix formatting.
Example:
# cat /sys/fs/bpf/maps.debug
id name max_entries cur_entries
2 iterator.rodata 1 0
125 cilium_auth_map 524288 666
126 cilium_runtime_ 256 0
127 cilium_signals 32 0
128 cilium_node_map 16384 1344
129 cilium_events 32 0
...--- kernel/bpf/preload/iterators/iterators.bpf.c | 9 +- .../iterators/iterators.lskel-little-endian.h | 526 +++++++++--------- 2 files changed, 275 insertions(+), 260 deletions(-)
kernel/bpf/preload/iterators/iterators.bpf.c +7 -2
@@ -73,6 +73,8 @@ static const char *get_name(struct btf *btf, long btf_id, const char *fallback)return str + name_off; } +__s64 bpf_map_sum_elem_count(struct bpf_map *map) __ksym; + SEC("iter/bpf_map")int dump_bpf_map (struct bpf_iter__bpf_map *ctx) {
@@ -84,9 +86,12 @@ int dump_bpf_map(struct bpf_iter__bpf_map *ctx)return 0 ;if (seq_num ==0 ) -BPF_SEQ_PRINTF (seq," id name max_entries\n" ); +BPF_SEQ_PRINTF (seq," id name max_entries cur_entries\n" ); + +BPF_SEQ_PRINTF (seq,"%4u %-16s %10d %10lld\n" , + map->id, map->name, map->max_entries, +bpf_map_sum_elem_count (map)); -BPF_SEQ_PRINTF (seq,"%4u %-16s%6d\n" , map->id, map->name, map->max_entries);return 0 ; }
kernel/bpf/preload/iterators/iterators.lskel-little-endian.h +268 -258
583 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v3 5/6 selftests/bpf: test map percpu stats
In reply to Anton ProtopopovAdd a new map test, map_percpu_stats.c, which is checking the correctness of map's percpu elements counters. For supported maps the test upserts a number of elements, checks the correctness of the counters, then deletes all the elements and checks again that the counters sum drops down to zero.
The following map types are tested:
* BPF_MAP_TYPE_HASH, BPF_F_NO_PREALLOC
* BPF_MAP_TYPE_PERCPU_HASH, BPF_F_NO_PREALLOC
* BPF_MAP_TYPE_HASH,
* BPF_MAP_TYPE_PERCPU_HASH,
* BPF_MAP_TYPE_LRU_HASH
* BPF_MAP_TYPE_LRU_PERCPU_HASH
--- .../bpf/map_tests/map_percpu_stats.c | 336 ++++++++++++++++++ .../selftests/bpf/progs/map_percpu_stats.c | 24 ++ 2 files changed, 360 insertions(+) create mode 100644 tools/testing/selftests/bpf/map_tests/map_percpu_stats.c create mode 100644 tools/testing/selftests/bpf/progs/map_percpu_stats.c
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +336 -0
@@ -0,0 +1,336 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2023 Isovalent */ + +#include <errno.h> +#include <unistd.h> +#include <pthread.h> + +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#include <bpf_util.h> +#include <test_maps.h> + +#include "map_percpu_stats.skel.h" + +#define MAX_ENTRIES 16384 +#define N_THREADS 37 + +#define MAX_MAP_KEY_SIZE 4 + +static void map_info (int map_fd,struct bpf_map_info *info) +{ +__u32 len =sizeof (*info); +int ret; + +memset (info,0 ,sizeof (*info)); + + ret =bpf_obj_get_info_by_fd (map_fd, info, &len); +CHECK (ret <0 ,"bpf_obj_get_info_by_fd" ,"error: %s\n" ,strerror (errno)); +} + +static const char *map_type_to_s (__u32 type) +{ +switch (type) { +case BPF_MAP_TYPE_HASH : +return "HASH" ; +case BPF_MAP_TYPE_PERCPU_HASH : +return "PERCPU_HASH" ; +case BPF_MAP_TYPE_LRU_HASH : +return "LRU_HASH" ; +case BPF_MAP_TYPE_LRU_PERCPU_HASH : +return "LRU_PERCPU_HASH" ; +default : +return "<define-me>" ; + } +} + +/* Map i -> map-type-specific-key */ +static void *map_key (__u32 type,__u32 i) +{ +static __thread__u8 key[MAX_MAP_KEY_SIZE ]; + + *(__u32 *)key = i; +return key; +} + +static __u32 map_count_elements (__u32 type,int map_fd) +{ +void *key =map_key (type,-1 ); +int n =0 ; + +while (!bpf_map_get_next_key (map_fd, key, key)) + n++; +return n; +} + +static void delete_all_elements (__u32 type,int map_fd) +{ +void *key =map_key (type,-1 ); +void *keys; +int n =0 ; +int ret; + + keys =calloc (MAX_MAP_KEY_SIZE ,MAX_ENTRIES ); +CHECK (!keys,"calloc" ,"error: %s\n" ,strerror (errno)); + +for (; !bpf_map_get_next_key (map_fd, key, key); n++) +memcpy (keys + n*MAX_MAP_KEY_SIZE , key,MAX_MAP_KEY_SIZE ); + +while (--n >=0 ) { + ret =bpf_map_delete_elem (map_fd, keys + n*MAX_MAP_KEY_SIZE ); +CHECK (ret <0 ,"bpf_map_delete_elem" ,"error: %s\n" ,strerror (errno)); + } +} + +static bool is_lru (__u32 map_type) +{ +return map_type ==BPF_MAP_TYPE_LRU_HASH || + map_type ==BPF_MAP_TYPE_LRU_PERCPU_HASH ; +} + +struct upsert_opts { +__u32 map_type; +int map_fd; +__u32 n; +}; + +static void *patch_map_thread (void *arg) +{ +struct upsert_opts *opts = arg; +void *key; +int val; +int ret; +int i; + +for (i =0 ; i < opts->n; i++) { + key =map_key (opts->map_type, i); + val =rand (); + ret =bpf_map_update_elem (opts->map_fd, key, &val,0 ); +CHECK (ret <0 ,"bpf_map_update_elem" ,"error: %s\n" ,strerror (errno)); + } +return NULL ; +} + +static void upsert_elements (struct upsert_opts *opts) +{ +pthread_t threads[N_THREADS ]; +int ret; +int i; + +for (i =0 ; i <ARRAY_SIZE (threads); i++) { + ret =pthread_create (&i[threads],NULL , patch_map_thread, opts); +CHECK (ret !=0 ,"pthread_create" ,"error: %s\n" ,strerror (ret)); + } + +for (i =0 ; i <ARRAY_SIZE (threads); i++) { + ret =pthread_join (i[threads],NULL ); +CHECK (ret !=0 ,"pthread_join" ,"error: %s\n" ,strerror (ret)); + } +} + +static __u32 read_cur_elements (int iter_fd) +{ +char buf[64 ]; +ssize_t n; +__u32 ret; + + n =read (iter_fd, buf,sizeof (buf)-1 ); +CHECK (n <=0 ,"read" ,"error: %s\n" ,strerror (errno)); + buf[n] ='\0' ; + + errno =0 ; + ret = (__u32 )strtol (buf,NULL ,10 ); +CHECK (errno !=0 ,"strtol" ,"error: %s\n" ,strerror (errno)); + +return ret; +} + +static __u32 get_cur_elements (int map_id) +{ +LIBBPF_OPTS (bpf_iter_attach_opts, opts); +union bpf_iter_link_info linfo; +struct map_percpu_stats *skel; +struct bpf_link *link; +int iter_fd; +int ret; + + opts.link_info = &linfo; + opts.link_info_len =sizeof (linfo); + + skel =map_percpu_stats__open (); +CHECK (skel ==NULL ,"map_percpu_stats__open" ,"error: %s" ,strerror (errno)); + + skel->bss->target_id = map_id; + + ret =map_percpu_stats__load (skel); +CHECK (ret !=0 ,"map_percpu_stats__load" ,"error: %s" ,strerror (errno)); + + link =bpf_program__attach_iter (skel->progs.dump_bpf_map, &opts); +CHECK (!link,"bpf_program__attach_iter" ,"error: %s\n" ,strerror (errno)); + + iter_fd =bpf_iter_create (bpf_link__fd (link)); +CHECK (iter_fd <0 ,"bpf_iter_create" ,"error: %s\n" ,strerror (errno)); + +return read_cur_elements (iter_fd); +} + +static void __test (int map_fd) +{ +__u32 n =MAX_ENTRIES -1000 ; +__u32 real_current_elements; +__u32 iter_current_elements; +struct upsert_opts opts = { + .map_fd = map_fd, + .n = n, + }; +struct bpf_map_info info; + +map_info (map_fd, &info); + opts.map_type = info.type; + +/* +* Upsert keys [0, n) under some competition: with random values from +* N_THREADS threads +*/ +upsert_elements (&opts); + +/* +* The sum of percpu elements counters for all hashtable-based maps +* should be equal to the number of elements present in the map. For +* non-lru maps this number should be the number n of upserted +* elements. For lru maps some elements might have been evicted. Check +* that all numbers make sense +*/ +map_info (map_fd, &info); + real_current_elements =map_count_elements (info.type, map_fd); +if (!is_lru (info.type)) +CHECK (n != real_current_elements,"map_count_elements" , +"real_current_elements(%u) != expected(%u)\n" , real_current_elements, n); + + iter_current_elements =get_cur_elements (info.id); +CHECK (iter_current_elements != real_current_elements,"get_cur_elements" , +"iter_current_elements=%u, expected %u (map_type=%s,map_flags=%08x)\n" , + iter_current_elements, real_current_elements,map_type_to_s (info.type), info.map_flags); + +/* +* Cleanup the map and check that all elements are actually gone and +* that the sum of percpu elements counters is back to 0 as well +*/ +delete_all_elements (info.type, map_fd); +map_info (map_fd, &info); + real_current_elements =map_count_elements (info.type, map_fd); +CHECK (real_current_elements,"map_count_elements" , +"expected real_current_elements=0, got %u" , real_current_elements); + + iter_current_elements =get_cur_elements (info.id); +CHECK (iter_current_elements !=0 ,"get_cur_elements" , +"iter_current_elements=%u, expected 0 (map_type=%s,map_flags=%08x)\n" , + iter_current_elements,map_type_to_s (info.type), info.map_flags); + +close (map_fd); +} + +static int map_create_opts (__u32 type,const char *name, +struct bpf_map_create_opts *map_opts, +__u32 key_size,__u32 val_size) +{ +int map_fd; + + map_fd =bpf_map_create (type, name, key_size, val_size,MAX_ENTRIES , map_opts); +CHECK (map_fd <0 ,"bpf_map_create()" ,"error:%s (name=%s)\n" , +strerror (errno), name); + +return map_fd; +} + +static int map_create (__u32 type,const char *name,struct bpf_map_create_opts *map_opts) +{ +return map_create_opts (type, name, map_opts,sizeof (int ),sizeof (int )); +} + +static int create_hash (void ) +{ +struct bpf_map_create_opts map_opts = { + .sz =sizeof (map_opts), + .map_flags =BPF_F_NO_PREALLOC , + }; + +return map_create (BPF_MAP_TYPE_HASH ,"hash" , &map_opts); +} + +static int create_percpu_hash (void ) +{ +struct bpf_map_create_opts map_opts = { + .sz =sizeof (map_opts), + .map_flags =BPF_F_NO_PREALLOC , + }; + +return map_create (BPF_MAP_TYPE_PERCPU_HASH ,"percpu_hash" , &map_opts); +} + +static int create_hash_prealloc (void ) +{ +return map_create (BPF_MAP_TYPE_HASH ,"hash" ,NULL ); +} + +static int create_percpu_hash_prealloc (void ) +{ +return map_create (BPF_MAP_TYPE_PERCPU_HASH ,"percpu_hash_prealloc" ,NULL ); +} + +static int create_lru_hash (void ) +{ +return map_create (BPF_MAP_TYPE_LRU_HASH ,"lru_hash" ,NULL ); +} + +static int create_percpu_lru_hash (void ) +{ +return map_create (BPF_MAP_TYPE_LRU_PERCPU_HASH ,"lru_hash_percpu" ,NULL ); +} + +static void map_percpu_stats_hash (void ) +{ +__test (create_hash ()); +printf ("test_%s:PASS\n" , __func__); +} + +static void map_percpu_stats_percpu_hash (void ) +{ +__test (create_percpu_hash ()); +printf ("test_%s:PASS\n" , __func__); +} + +static void map_percpu_stats_hash_prealloc (void ) +{ +__test (create_hash_prealloc ()); +printf ("test_%s:PASS\n" , __func__); +} + +static void map_percpu_stats_percpu_hash_prealloc (void ) +{ +__test (create_percpu_hash_prealloc ()); +printf ("test_%s:PASS\n" , __func__); +} + +static void map_percpu_stats_lru_hash (void ) +{ +__test (create_lru_hash ()); +printf ("test_%s:PASS\n" , __func__); +} + +static void map_percpu_stats_percpu_lru_hash (void ) +{ +__test (create_percpu_lru_hash ()); +printf ("test_%s:PASS\n" , __func__); +} + +void test_map_percpu_stats (void ) +{ +map_percpu_stats_hash (); +map_percpu_stats_percpu_hash (); +map_percpu_stats_hash_prealloc (); +map_percpu_stats_percpu_hash_prealloc (); +map_percpu_stats_lru_hash (); +map_percpu_stats_percpu_lru_hash (); +}
tools/testing/selftests/bpf/progs/map_percpu_stats.c new file +24 -0
@@ -0,0 +1,24 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2023 Isovalent */ + +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_tracing.h> + +__u32 target_id; + +__s64 bpf_map_sum_elem_count(struct bpf_map *map) __ksym; + +SEC("iter/bpf_map") +int dump_bpf_map (struct bpf_iter__bpf_map *ctx) +{ +struct seq_file *seq = ctx->meta->seq; +struct bpf_map *map = ctx->map; + +if (map && map->id == target_id) +BPF_SEQ_PRINTF (seq,"%lld" ,bpf_map_sum_elem_count (map)); + +return 0 ; +} + +char _license[] SEC("license") = "GPL";
Signature · 1 line
2.34.1
Re: [v3 PATCH bpf-next 5/6] selftests/bpf: test map percpu stats
Hi,
Quoted 5 lines · Anton Protopopov
Add a new map test,
map_percpu_stats.c, which is checking the correctness of map's percpu elements counters. For supported maps the test upserts a number of elements, checks the correctness of the counters, then deletes all the elements and checks again that the counters sum drops down to zero.
Quoted 8 lines
The following map types are tested:
*
BPF_MAP_TYPE_HASH,BPF_F_NO_PREALLOC
*BPF_MAP_TYPE_PERCPU_HASH,BPF_F_NO_PREALLOC
*BPF_MAP_TYPE_HASH,
*BPF_MAP_TYPE_PERCPU_HASH,
*BPF_MAP_TYPE_LRU_HASH
*BPF_MAP_TYPE_LRU_PERCPU_HASH
A test for BPF_MAP_TYPE_HASH_OF_MAPS is also needed.
Quoted 22 lines
Signed-off-by Anton Protopopov <[email protected]>--- .../bpf/map_tests/map_percpu_stats.c | 336 ++++++++++++++++++ .../selftests/bpf/progs/map_percpu_stats.c | 24 ++ 2 files changed, 360 insertions(+) create mode 100644 tools/testing/selftests/bpf/map_tests/map_percpu_stats.c create mode 100644 tools/testing/selftests/bpf/progs/map_percpu_stats.ctools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +7
-0@@ -0,0 +1,336 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2023 Isovalent */ + +#include <errno.h> +#include <unistd.h> +#include <pthread.h> +
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 17
Quoted 10 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +10
-0+#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#include <bpf_util.h> +#include <test_maps.h> + +#include "map_percpu_stats.skel.h" + +#define MAX_ENTRIES 16384 +#define N_THREADS 37
Why 37 thread is needed here ? Does a small number of threads work as well ?
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 29
Quoted 12 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +12
-0+ +#define MAX_MAP_KEY_SIZE 4 + +static void map_info (int map_fd,struct bpf_map_info *info) +{ +__u32 len =sizeof (*info); +int ret; + +memset (info,0 ,sizeof (*info)); + + ret =bpf_obj_get_info_by_fd (map_fd, info, &len); +CHECK (ret <0 ,"bpf_obj_get_info_by_fd" ,"error: %s\n" ,strerror (errno));
Please use ASSERT_OK instead.
Quoted 12 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +12
-0+} + +static const char *map_type_to_s (__u32 type) +{ +switch (type) { +case BPF_MAP_TYPE_HASH : +return "HASH" ; +case BPF_MAP_TYPE_PERCPU_HASH : +return "PERCPU_HASH" ; +case BPF_MAP_TYPE_LRU_HASH : +return "LRU_HASH" ; +case BPF_MAP_TYPE_LRU_PERCPU_HASH :
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 51
Quoted 10 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +10
-0+return "LRU_PERCPU_HASH" ; +default : +return "<define-me>" ; + } +} + +/* Map i -> map-type-specific-key */ +static void *map_key (__u32 type,__u32 i) +{ +static __thread__u8 key[MAX_MAP_KEY_SIZE ];
Why a per-thread key is necessary here ? Could we just define it when the key is needed ?
Quoted 23 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +23
-0+ + *(__u32 *)key = i; +return key; +} + +static __u32 map_count_elements (__u32 type,int map_fd) +{ +void *key =map_key (type,-1 ); +int n =0 ; + +while (!bpf_map_get_next_key (map_fd, key, key)) + n++; +return n; +} + +static void delete_all_elements (__u32 type,int map_fd) +{ +void *key =map_key (type,-1 ); +void *keys; +int n =0 ; +int ret; + + keys =calloc (MAX_MAP_KEY_SIZE ,MAX_ENTRIES );
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 84
Quoted 10 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +10
-0+CHECK (!keys,"calloc" ,"error: %s\n" ,strerror (errno)); + +for (; !bpf_map_get_next_key (map_fd, key, key); n++) +memcpy (keys + n*MAX_MAP_KEY_SIZE , key,MAX_MAP_KEY_SIZE ); + +while (--n >=0 ) { + ret =bpf_map_delete_elem (map_fd, keys + n*MAX_MAP_KEY_SIZE ); +CHECK (ret <0 ,"bpf_map_delete_elem" ,"error: %s\n" ,strerror (errno)); + } +}
Please use ASSERT_xxx() to replace CHECK().
Quoted 81 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +81
-0+ +static bool is_lru (__u32 map_type) +{ +return map_type ==BPF_MAP_TYPE_LRU_HASH || + map_type ==BPF_MAP_TYPE_LRU_PERCPU_HASH ; +} + +struct upsert_opts { +__u32 map_type; +int map_fd; +__u32 n; +}; + +static void *patch_map_thread (void *arg) +{ +struct upsert_opts *opts = arg; +void *key; +int val; +int ret; +int i; + +for (i =0 ; i < opts->n; i++) { + key =map_key (opts->map_type, i); + val =rand (); + ret =bpf_map_update_elem (opts->map_fd, key, &val,0 ); +CHECK (ret <0 ,"bpf_map_update_elem" ,"error: %s\n" ,strerror (errno)); + } +return NULL ; +} + +static void upsert_elements (struct upsert_opts *opts) +{ +pthread_t threads[N_THREADS ]; +int ret; +int i; + +for (i =0 ; i <ARRAY_SIZE (threads); i++) { + ret =pthread_create (&i[threads],NULL , patch_map_thread, opts); +CHECK (ret !=0 ,"pthread_create" ,"error: %s\n" ,strerror (ret)); + } + +for (i =0 ; i <ARRAY_SIZE (threads); i++) { + ret =pthread_join (i[threads],NULL ); +CHECK (ret !=0 ,"pthread_join" ,"error: %s\n" ,strerror (ret)); + } +} + +static __u32 read_cur_elements (int iter_fd) +{ +char buf[64 ]; +ssize_t n; +__u32 ret; + + n =read (iter_fd, buf,sizeof (buf)-1 ); +CHECK (n <=0 ,"read" ,"error: %s\n" ,strerror (errno)); + buf[n] ='\0' ; + + errno =0 ; + ret = (__u32 )strtol (buf,NULL ,10 ); +CHECK (errno !=0 ,"strtol" ,"error: %s\n" ,strerror (errno)); + +return ret; +} + +static __u32 get_cur_elements (int map_id) +{ +LIBBPF_OPTS (bpf_iter_attach_opts, opts); +union bpf_iter_link_info linfo; +struct map_percpu_stats *skel; +struct bpf_link *link; +int iter_fd; +int ret; + + opts.link_info = &linfo; + opts.link_info_len =sizeof (linfo); + + skel =map_percpu_stats__open (); +CHECK (skel ==NULL ,"map_percpu_stats__open" ,"error: %s" ,strerror (errno)); + + skel->bss->target_id = map_id; +
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 175
Quoted 10 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +10
-0+ ret =map_percpu_stats__load (skel); +CHECK (ret !=0 ,"map_percpu_stats__load" ,"error: %s" ,strerror (errno)); + + link =bpf_program__attach_iter (skel->progs.dump_bpf_map, &opts); +CHECK (!link,"bpf_program__attach_iter" ,"error: %s\n" ,strerror (errno)); + + iter_fd =bpf_iter_create (bpf_link__fd (link)); +CHECK (iter_fd <0 ,"bpf_iter_create" ,"error: %s\n" ,strerror (errno)); + +return read_cur_elements (iter_fd);
Need to do close(iter_fd), bpf_link__destroy(link) and map__percpu_stats__destroy() before return, otherwise there will be resource leak.
Quoted 20 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +20
-0+} + +static void __test (int map_fd) +{ +__u32 n =MAX_ENTRIES -1000 ; +__u32 real_current_elements; +__u32 iter_current_elements; +struct upsert_opts opts = { + .map_fd = map_fd, + .n = n, + }; +struct bpf_map_info info; + +map_info (map_fd, &info); + opts.map_type = info.type; + +/* +* Upsert keys [0, n) under some competition: with random values from +* N_THREADS threads +*/
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 205
Quoted 10 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +10
-0+upsert_elements (&opts); + +/* +* The sum of percpu elements counters for all hashtable-based maps +* should be equal to the number of elements present in the map. For +* non-lru maps this number should be the number n of upserted +* elements. For lru maps some elements might have been evicted. Check +* that all numbers make sense +*/ +map_info (map_fd, &info);
I think there is no need to call map_info() multiple times because the needed type and id will not be changed after creation.
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 209
Quoted 4 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +4
-0+ real_current_elements =map_count_elements (info.type, map_fd); +if (!is_lru (info.type)) +CHECK (n != real_current_elements,"map_count_elements" , +"real_current_elements(%u) != expected(%u)\n" , real_current_elements, n);
For LRU map, please use ASSERT_EQ(). For LRU map, should we check "n >= real_current_elements" instead ?
Quoted 5 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +5
-0+ + iter_current_elements =get_cur_elements (info.id); +CHECK (iter_current_elements != real_current_elements,"get_cur_elements" , +"iter_current_elements=%u, expected %u (map_type=%s,map_flags=%08x)\n" , + iter_current_elements, real_current_elements,map_type_to_s (info.type), info.map_flags);
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 224
Quoted 10 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +10
-0+ +/* +* Cleanup the map and check that all elements are actually gone and +* that the sum of percpu elements counters is back to 0 as well +*/ +delete_all_elements (info.type, map_fd); +map_info (map_fd, &info); + real_current_elements =map_count_elements (info.type, map_fd); +CHECK (real_current_elements,"map_count_elements" , +"expected real_current_elements=0, got %u" , real_current_elements);
ASSERT_EQ
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 230
Quoted 6 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +6
-0+ + iter_current_elements =get_cur_elements (info.id); +CHECK (iter_current_elements !=0 ,"get_cur_elements" , +"iter_current_elements=%u, expected 0 (map_type=%s,map_flags=%08x)\n" , + iter_current_elements,map_type_to_s (info.type), info.map_flags); +
ASSERT_NEQ
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 242
Quoted 12 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +12
-0+close (map_fd); +} + +static int map_create_opts (__u32 type,const char *name, +struct bpf_map_create_opts *map_opts, +__u32 key_size,__u32 val_size) +{ +int map_fd; + + map_fd =bpf_map_create (type, name, key_size, val_size,MAX_ENTRIES , map_opts); +CHECK (map_fd <0 ,"bpf_map_create()" ,"error:%s (name=%s)\n" , +strerror (errno), name);
Please use ASSERT_GE instead.
Quoted 73 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +73
-0+ +return map_fd; +} + +static int map_create (__u32 type,const char *name,struct bpf_map_create_opts *map_opts) +{ +return map_create_opts (type, name, map_opts,sizeof (int ),sizeof (int )); +} + +static int create_hash (void ) +{ +struct bpf_map_create_opts map_opts = { + .sz =sizeof (map_opts), + .map_flags =BPF_F_NO_PREALLOC , + }; + +return map_create (BPF_MAP_TYPE_HASH ,"hash" , &map_opts); +} + +static int create_percpu_hash (void ) +{ +struct bpf_map_create_opts map_opts = { + .sz =sizeof (map_opts), + .map_flags =BPF_F_NO_PREALLOC , + }; + +return map_create (BPF_MAP_TYPE_PERCPU_HASH ,"percpu_hash" , &map_opts); +} + +static int create_hash_prealloc (void ) +{ +return map_create (BPF_MAP_TYPE_HASH ,"hash" ,NULL ); +} + +static int create_percpu_hash_prealloc (void ) +{ +return map_create (BPF_MAP_TYPE_PERCPU_HASH ,"percpu_hash_prealloc" ,NULL ); +} + +static int create_lru_hash (void ) +{ +return map_create (BPF_MAP_TYPE_LRU_HASH ,"lru_hash" ,NULL ); +} + +static int create_percpu_lru_hash (void ) +{ +return map_create (BPF_MAP_TYPE_LRU_PERCPU_HASH ,"lru_hash_percpu" ,NULL ); +} + +static void map_percpu_stats_hash (void ) +{ +__test (create_hash ()); +printf ("test_%s:PASS\n" , __func__); +} + +static void map_percpu_stats_percpu_hash (void ) +{ +__test (create_percpu_hash ()); +printf ("test_%s:PASS\n" , __func__); +} + +static void map_percpu_stats_hash_prealloc (void ) +{ +__test (create_hash_prealloc ()); +printf ("test_%s:PASS\n" , __func__); +} + +static void map_percpu_stats_percpu_hash_prealloc (void ) +{ +__test (create_percpu_hash_prealloc ()); +printf ("test_%s:PASS\n" , __func__); +} +
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 325
Quoted 10 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +10
-0+static void map_percpu_stats_lru_hash (void ) +{ +__test (create_lru_hash ()); +printf ("test_%s:PASS\n" , __func__); +} + +static void map_percpu_stats_percpu_lru_hash (void ) +{ +__test (create_percpu_lru_hash ()); +printf ("test_%s:PASS\n" , __func__);
After switch to subtest, the printf() can be removed.
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 336
Quoted 11 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +11
-0+} + +void test_map_percpu_stats (void ) +{ +map_percpu_stats_hash (); +map_percpu_stats_percpu_hash (); +map_percpu_stats_hash_prealloc (); +map_percpu_stats_percpu_hash_prealloc (); +map_percpu_stats_lru_hash (); +map_percpu_stats_percpu_lru_hash (); +}
Please use test__start_subtest() to create multiple subtests.
Quoted 30 lines
tools/testing/selftests/bpf/progs/map_percpu_stats.c new file +24
-0@@ -0,0 +1,24 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2023 Isovalent */ + +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_tracing.h> + +__u32 target_id; + +__s64 bpf_map_sum_elem_count(struct bpf_map *map) __ksym; + +SEC("iter/bpf_map") +int dump_bpf_map (struct bpf_iter__bpf_map *ctx) +{ +struct seq_file *seq = ctx->meta->seq; +struct bpf_map *map = ctx->map; + +if (map && map->id == target_id) +BPF_SEQ_PRINTF (seq,"%lld" ,bpf_map_sum_elem_count (map)); + +return 0 ; +} + +char _license[] SEC("license") = "GPL";
Re: [v3 PATCH bpf-next 5/6] selftests/bpf: test map percpu stats
Quoted 10 lines · Hou Tao
Hi,
Quoted 7 lines · Anton Protopopov
Add a new map test,
map_percpu_stats.c, which is checking the correctness of map's percpu elements counters. For supported maps the test upserts a number of elements, checks the correctness of the counters, then deletes all the elements and checks again that the counters sum drops down to zero.The following map types are tested:
Quoted 8 lines
Quoted 6 lines
*
BPF_MAP_TYPE_HASH,BPF_F_NO_PREALLOC
*BPF_MAP_TYPE_PERCPU_HASH,BPF_F_NO_PREALLOC
*BPF_MAP_TYPE_HASH,
*BPF_MAP_TYPE_PERCPU_HASH,
*BPF_MAP_TYPE_LRU_HASH
*BPF_MAP_TYPE_LRU_PERCPU_HASHA test for
BPF_MAP_TYPE_HASH_OF_MAPSis also needed.
I will add it.
Quoted 26 lines
Quoted 26 lines
Signed-off-by Anton Protopopov <[email protected]>--- .../bpf/map_tests/map_percpu_stats.c | 336 ++++++++++++++++++ .../selftests/bpf/progs/map_percpu_stats.c | 24 ++ 2 files changed, 360 insertions(+) create mode 100644 tools/testing/selftests/bpf/map_tests/map_percpu_stats.c create mode 100644 tools/testing/selftests/bpf/progs/map_percpu_stats.ctools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +11
-0@@ -0,0 +1,336 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2023 Isovalent */ + +#include <errno.h> +#include <unistd.h> +#include <pthread.h> + +#include <bpf/bpf.h> +#include <bpf/libbpf.h> + +#include <bpf_util.h>
Quoted 8 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 17
Quoted 6 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +6
-0+#include <test_maps.h> + +#include "map_percpu_stats.skel.h" + +#define MAX_ENTRIES 16384 +#define N_THREADS 37Why 37 thread is needed here ? Does a small number of threads work as well ?
This was used to evict more elements from LRU maps when they are full.
Quoted 5 lines
Quoted 5 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +5
-0+ +#define MAX_MAP_KEY_SIZE 4 + +static void map_info (int map_fd,struct bpf_map_info *info) +{
Quoted 8 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 29
Quoted 7 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +7
-0+__u32 len =sizeof (*info); +int ret; + +memset (info,0 ,sizeof (*info)); + + ret =bpf_obj_get_info_by_fd (map_fd, info, &len); +CHECK (ret <0 ,"bpf_obj_get_info_by_fd" ,"error: %s\n" ,strerror (errno));Please use
ASSERT_OKinstead.
Ok, thanks, will do (and for all other similar cases you've mentioned below).
Quoted 17 lines
Quoted 17 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +17
-0+} + +static const char *map_type_to_s (__u32 type) +{ +switch (type) { +case BPF_MAP_TYPE_HASH : +return "HASH" ; +case BPF_MAP_TYPE_PERCPU_HASH : +return "PERCPU_HASH" ; +case BPF_MAP_TYPE_LRU_HASH : +return "LRU_HASH" ; +case BPF_MAP_TYPE_LRU_PERCPU_HASH : +return "LRU_PERCPU_HASH" ; +default : +return "<define-me>" ; + } +}
Quoted 8 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 51
Quoted 5 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +5
-0+ +/* Map i -> map-type-specific-key */ +static void *map_key (__u32 type,__u32 i) +{ +static __thread__u8 key[MAX_MAP_KEY_SIZE ];Why a per-thread key is necessary here ? Could we just define it when the key is needed ?
Thanks, I will remove it to simplify code. (This was used to create keys for non-hash maps, e.g., LPM-trie in the first version of the patch.)
Quoted 122 lines
Quoted 23 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +23
-0+ + *(__u32 *)key = i; +return key; +} + +static __u32 map_count_elements (__u32 type,int map_fd) +{ +void *key =map_key (type,-1 ); +int n =0 ; + +while (!bpf_map_get_next_key (map_fd, key, key)) + n++; +return n; +} + +static void delete_all_elements (__u32 type,int map_fd) +{ +void *key =map_key (type,-1 ); +void *keys; +int n =0 ; +int ret; + + keys =calloc (MAX_MAP_KEY_SIZE ,MAX_ENTRIES );tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 84
Quoted 10 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +10
-0+CHECK (!keys,"calloc" ,"error: %s\n" ,strerror (errno)); + +for (; !bpf_map_get_next_key (map_fd, key, key); n++) +memcpy (keys + n*MAX_MAP_KEY_SIZE , key,MAX_MAP_KEY_SIZE ); + +while (--n >=0 ) { + ret =bpf_map_delete_elem (map_fd, keys + n*MAX_MAP_KEY_SIZE ); +CHECK (ret <0 ,"bpf_map_delete_elem" ,"error: %s\n" ,strerror (errno)); + } +}Please use
ASSERT_xxx()to replaceCHECK().Quoted 87 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +87
-0+ +static bool is_lru (__u32 map_type) +{ +return map_type ==BPF_MAP_TYPE_LRU_HASH || + map_type ==BPF_MAP_TYPE_LRU_PERCPU_HASH ; +} + +struct upsert_opts { +__u32 map_type; +int map_fd; +__u32 n; +}; + +static void *patch_map_thread (void *arg) +{ +struct upsert_opts *opts = arg; +void *key; +int val; +int ret; +int i; + +for (i =0 ; i < opts->n; i++) { + key =map_key (opts->map_type, i); + val =rand (); + ret =bpf_map_update_elem (opts->map_fd, key, &val,0 ); +CHECK (ret <0 ,"bpf_map_update_elem" ,"error: %s\n" ,strerror (errno)); + } +return NULL ; +} + +static void upsert_elements (struct upsert_opts *opts) +{ +pthread_t threads[N_THREADS ]; +int ret; +int i; + +for (i =0 ; i <ARRAY_SIZE (threads); i++) { + ret =pthread_create (&i[threads],NULL , patch_map_thread, opts); +CHECK (ret !=0 ,"pthread_create" ,"error: %s\n" ,strerror (ret)); + } + +for (i =0 ; i <ARRAY_SIZE (threads); i++) { + ret =pthread_join (i[threads],NULL ); +CHECK (ret !=0 ,"pthread_join" ,"error: %s\n" ,strerror (ret)); + } +} + +static __u32 read_cur_elements (int iter_fd) +{ +char buf[64 ]; +ssize_t n; +__u32 ret; + + n =read (iter_fd, buf,sizeof (buf)-1 ); +CHECK (n <=0 ,"read" ,"error: %s\n" ,strerror (errno)); + buf[n] ='\0' ; + + errno =0 ; + ret = (__u32 )strtol (buf,NULL ,10 ); +CHECK (errno !=0 ,"strtol" ,"error: %s\n" ,strerror (errno)); + +return ret; +} + +static __u32 get_cur_elements (int map_id) +{ +LIBBPF_OPTS (bpf_iter_attach_opts, opts); +union bpf_iter_link_info linfo; +struct map_percpu_stats *skel; +struct bpf_link *link; +int iter_fd; +int ret; + + opts.link_info = &linfo; + opts.link_info_len =sizeof (linfo); + + skel =map_percpu_stats__open (); +CHECK (skel ==NULL ,"map_percpu_stats__open" ,"error: %s" ,strerror (errno)); + + skel->bss->target_id = map_id; + + ret =map_percpu_stats__load (skel); +CHECK (ret !=0 ,"map_percpu_stats__load" ,"error: %s" ,strerror (errno)); + + link =bpf_program__attach_iter (skel->progs.dump_bpf_map, &opts); +CHECK (!link,"bpf_program__attach_iter" ,"error: %s\n" ,strerror (errno)); +
Quoted 8 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 175
Quoted 4 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +4
-0+ iter_fd =bpf_iter_create (bpf_link__fd (link)); +CHECK (iter_fd <0 ,"bpf_iter_create" ,"error: %s\n" ,strerror (errno)); + +return read_cur_elements (iter_fd);Need to do
close(iter_fd),bpf_link__destroy(link)andmap__percpu_stats__destroy()before return, otherwise there will be resource leak.
Yes, thanks.
Quoted 25 lines
Quoted 25 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +25
-0+} + +static void __test (int map_fd) +{ +__u32 n =MAX_ENTRIES -1000 ; +__u32 real_current_elements; +__u32 iter_current_elements; +struct upsert_opts opts = { + .map_fd = map_fd, + .n = n, + }; +struct bpf_map_info info; + +map_info (map_fd, &info); + opts.map_type = info.type; + +/* +* Upsert keys [0, n) under some competition: with random values from +* N_THREADS threads +*/ +upsert_elements (&opts); + +/* +* The sum of percpu elements counters for all hashtable-based maps +* should be equal to the number of elements present in the map. For
Quoted 8 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 205
Quoted 5 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +5
-0+* non-lru maps this number should be the number n of upserted +* elements. For lru maps some elements might have been evicted. Check +* that all numbers make sense +*/ +map_info (map_fd, &info);I think there is no need to call
map_info()multiple times because the needed type and id will not be changed after creation.
Thanks, will fix. (This code left from the first version when the counter was returned by map_info().)
Quoted 129 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 209
Quoted 4 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +4
-0+ real_current_elements =map_count_elements (info.type, map_fd); +if (!is_lru (info.type)) +CHECK (n != real_current_elements,"map_count_elements" , +"real_current_elements(%u) != expected(%u)\n" , real_current_elements, n);For LRU map, please use
ASSERT_EQ(). For LRU map, should we check "n >=real_current_elements" instead ?Quoted 5 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +5
-0+ + iter_current_elements =get_cur_elements (info.id); +CHECK (iter_current_elements != real_current_elements,"get_cur_elements" , +"iter_current_elements=%u, expected %u (map_type=%s,map_flags=%08x)\n" , + iter_current_elements, real_current_elements,map_type_to_s (info.type), info.map_flags);tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 224
Quoted 10 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +10
-0+ +/* +* Cleanup the map and check that all elements are actually gone and +* that the sum of percpu elements counters is back to 0 as well +*/ +delete_all_elements (info.type, map_fd); +map_info (map_fd, &info); + real_current_elements =map_count_elements (info.type, map_fd); +CHECK (real_current_elements,"map_count_elements" , +"expected real_current_elements=0, got %u" , real_current_elements);
ASSERT_EQtools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 230
Quoted 6 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +6
-0+ + iter_current_elements =get_cur_elements (info.id); +CHECK (iter_current_elements !=0 ,"get_cur_elements" , +"iter_current_elements=%u, expected 0 (map_type=%s,map_flags=%08x)\n" , + iter_current_elements,map_type_to_s (info.type), info.map_flags); +
ASSERT_NEQtools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 242
Quoted 12 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +12
-0+close (map_fd); +} + +static int map_create_opts (__u32 type,const char *name, +struct bpf_map_create_opts *map_opts, +__u32 key_size,__u32 val_size) +{ +int map_fd; + + map_fd =bpf_map_create (type, name, key_size, val_size,MAX_ENTRIES , map_opts); +CHECK (map_fd <0 ,"bpf_map_create()" ,"error:%s (name=%s)\n" , +strerror (errno), name);Please use
ASSERT_GEinstead.Quoted 73 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +73
-0+ +return map_fd; +} + +static int map_create (__u32 type,const char *name,struct bpf_map_create_opts *map_opts) +{ +return map_create_opts (type, name, map_opts,sizeof (int ),sizeof (int )); +} + +static int create_hash (void ) +{ +struct bpf_map_create_opts map_opts = { + .sz =sizeof (map_opts), + .map_flags =BPF_F_NO_PREALLOC , + }; + +return map_create (BPF_MAP_TYPE_HASH ,"hash" , &map_opts); +} + +static int create_percpu_hash (void ) +{ +struct bpf_map_create_opts map_opts = { + .sz =sizeof (map_opts), + .map_flags =BPF_F_NO_PREALLOC , + }; + +return map_create (BPF_MAP_TYPE_PERCPU_HASH ,"percpu_hash" , &map_opts); +} + +static int create_hash_prealloc (void ) +{ +return map_create (BPF_MAP_TYPE_HASH ,"hash" ,NULL ); +} + +static int create_percpu_hash_prealloc (void ) +{ +return map_create (BPF_MAP_TYPE_PERCPU_HASH ,"percpu_hash_prealloc" ,NULL ); +} + +static int create_lru_hash (void ) +{ +return map_create (BPF_MAP_TYPE_LRU_HASH ,"lru_hash" ,NULL ); +} + +static int create_percpu_lru_hash (void ) +{ +return map_create (BPF_MAP_TYPE_LRU_PERCPU_HASH ,"lru_hash_percpu" ,NULL ); +} + +static void map_percpu_stats_hash (void ) +{ +__test (create_hash ()); +printf ("test_%s:PASS\n" , __func__); +} + +static void map_percpu_stats_percpu_hash (void ) +{ +__test (create_percpu_hash ()); +printf ("test_%s:PASS\n" , __func__); +} + +static void map_percpu_stats_hash_prealloc (void ) +{ +__test (create_hash_prealloc ()); +printf ("test_%s:PASS\n" , __func__); +} + +static void map_percpu_stats_percpu_hash_prealloc (void ) +{ +__test (create_percpu_hash_prealloc ()); +printf ("test_%s:PASS\n" , __func__); +} +tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 325
Quoted 10 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +10
-0+static void map_percpu_stats_lru_hash (void ) +{ +__test (create_lru_hash ()); +printf ("test_%s:PASS\n" , __func__); +} + +static void map_percpu_stats_percpu_lru_hash (void ) +{ +__test (create_percpu_lru_hash ()); +printf ("test_%s:PASS\n" , __func__);After switch to subtest, the
printf()can be removed.
Quoted 13 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 336
Quoted 11 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +11
-0+} + +void test_map_percpu_stats (void ) +{ +map_percpu_stats_hash (); +map_percpu_stats_percpu_hash (); +map_percpu_stats_hash_prealloc (); +map_percpu_stats_percpu_hash_prealloc (); +map_percpu_stats_lru_hash (); +map_percpu_stats_percpu_lru_hash (); +}Please use
test__start_subtest()to create multiple subtests.
Thanks.
I will update this selftest in v4 with your comments addressed + batch ops tests.
Quoted 31 lines
Quoted 30 lines
tools/testing/selftests/bpf/progs/map_percpu_stats.c new file +24
-0@@ -0,0 +1,24 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2023 Isovalent */ + +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_tracing.h> + +__u32 target_id; + +__s64 bpf_map_sum_elem_count(struct bpf_map *map) __ksym; + +SEC("iter/bpf_map") +int dump_bpf_map (struct bpf_iter__bpf_map *ctx) +{ +struct seq_file *seq = ctx->meta->seq; +struct bpf_map *map = ctx->map; + +if (map && map->id == target_id) +BPF_SEQ_PRINTF (seq,"%lld" ,bpf_map_sum_elem_count (map)); + +return 0 ; +} + +char _license[] SEC("license") = "GPL";
Re: [v3 PATCH bpf-next 5/6] selftests/bpf: test map percpu stats
Quoted 5 lines · Anton Protopopov
Quoted 4 lines · Hou Tao
Hi,
On 6/30/2023 4:25 PM, Anton Protopopov wrote:
[...]
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +5
-0+} + +void test_map_percpu_stats (void ) +{ +map_percpu_stats_hash ();Quoted 8 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 336
Quoted 6 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +6
-0+map_percpu_stats_percpu_hash (); +map_percpu_stats_hash_prealloc (); +map_percpu_stats_percpu_hash_prealloc (); +map_percpu_stats_lru_hash (); +map_percpu_stats_percpu_lru_hash (); +}Please use
test__start_subtest()to create multiple subtests.
After looking at code, I think that I will leave the individual functions here, as the test__start_subtest() function is only implemented in test_progs (not test_maps), and adding it here looks like out of scope for this patch.
Re: [v3 PATCH bpf-next 5/6] selftests/bpf: test map percpu stats
Quoted 20 lines · Anton Protopopov
Quoted 5 lines · Anton Protopopov
Quoted 4 lines · Hou Tao
Hi,
On 6/30/2023 4:25 PM, Anton Protopopov wrote:
[...]Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +5
-0+} + +void test_map_percpu_stats (void ) +{ +map_percpu_stats_hash ();Quoted 8 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 336
Quoted 6 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +6
-0+map_percpu_stats_percpu_hash (); +map_percpu_stats_hash_prealloc (); +map_percpu_stats_percpu_hash_prealloc (); +map_percpu_stats_lru_hash (); +map_percpu_stats_percpu_lru_hash (); +}Please use
test__start_subtest()to create multiple subtests.
Quoted 3 lines
After looking at code, I think that I will leave the individual functions here, as the
test__start_subtest()function is only implemented intest_progs(nottest_maps), and adding it here looks like out of scope for this patch.
Ah, sorry, looks like the same stands for ASSERT* macros as well, as they are only used in test_progs. (I will still fix the checks where you commented on specific values, like n <= cur_elems for LRUs.)
Re: [v3 PATCH bpf-next 5/6] selftests/bpf: test map percpu stats
In reply to Anton ProtopopovHi,
Quoted 18 lines · Anton Protopopov
Quoted 5 lines · Anton Protopopov
Quoted 4 lines · Hou Tao
Hi,
On 6/30/2023 4:25 PM, Anton Protopopov wrote:
[...]Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +4
-0+} + +void test_map_percpu_stats (void ) +{Quoted 8 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 336
Quoted 7 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +7
-0+map_percpu_stats_hash (); +map_percpu_stats_percpu_hash (); +map_percpu_stats_hash_prealloc (); +map_percpu_stats_percpu_hash_prealloc (); +map_percpu_stats_lru_hash (); +map_percpu_stats_percpu_lru_hash (); +}Please use
test__start_subtest()to create multiple subtests.
Quoted 4 lines
After looking at code, I think that I will leave the individual functions here, as the
test__start_subtest()function is only implemented intest_progs(nottest_maps), and adding it here looks like out of scope for this patch. .
I see. But can we just add these tests in test_progs instead which is more flexible ?
Re: [v3 PATCH bpf-next 5/6] selftests/bpf: test map percpu stats
Quoted 21 lines · Hou Tao
Hi,
Quoted 18 lines · Anton Protopopov
Quoted 17 lines · Anton Protopopov
Quoted 16 lines · Hou Tao
Hi,
On 6/30/2023 4:25 PM, Anton Protopopov wrote:
[...]tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 336
Quoted 11 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +11
-0+} + +void test_map_percpu_stats (void ) +{ +map_percpu_stats_hash (); +map_percpu_stats_percpu_hash (); +map_percpu_stats_hash_prealloc (); +map_percpu_stats_percpu_hash_prealloc (); +map_percpu_stats_lru_hash (); +map_percpu_stats_percpu_lru_hash (); +}Please use
test__start_subtest()to create multiple subtests.
Quoted 6 lines
Quoted 4 lines
After looking at code, I think that I will leave the individual functions here, as the
test__start_subtest()function is only implemented intest_progs(nottest_maps), and adding it here looks like out of scope for this patch. .I see. But can we just add these tests in
test_progsinstead which is more flexible ?
I think that it makes more sense to port this test_prog flexibility to the test_maps program. I can volunteer to do this (but not right away).
Re: [v3 PATCH bpf-next 5/6] selftests/bpf: test map percpu stats
In reply to Anton ProtopopovHi,
Quoted 11 lines · Anton Protopopov
Quoted 10 lines · Hou Tao
Hi,
Quoted 7 lines · Anton Protopopov
Add a new map test,
map_percpu_stats.c, which is checking the correctness of map's percpu elements counters. For supported maps the test upserts a number of elements, checks the correctness of the counters, then deletes all the elements and checks again that the counters sum drops down to zero.The following map types are tested:
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
*
BPF_MAP_TYPE_HASH,BPF_F_NO_PREALLOC
*BPF_MAP_TYPE_PERCPU_HASH,BPF_F_NO_PREALLOC
*BPF_MAP_TYPE_HASH,
*BPF_MAP_TYPE_PERCPU_HASH,
*BPF_MAP_TYPE_LRU_HASH
*BPF_MAP_TYPE_LRU_PERCPU_HASHA test for
BPF_MAP_TYPE_HASH_OF_MAPSis also needed.
We could also exercise the test for LRU map with BPF_F_NO_COMMON_LRU.
Quoted 1 line
SNIP
Quoted 24 lines
Quoted 16 lines
Quoted 16 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +10
-0@@ -0,0 +1,336 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2023 Isovalent */ + +#include <errno.h> +#include <unistd.h> +#include <pthread.h> + +#include <bpf/bpf.h> +#include <bpf/libbpf.h> +Quoted 8 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 17
Quoted 7 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +7
-0+#include <bpf_util.h> +#include <test_maps.h> + +#include "map_percpu_stats.skel.h" + +#define MAX_ENTRIES 16384 +#define N_THREADS 37Why 37 thread is needed here ? Does a small number of threads work as well ?
Quoted 1 line
This was used to evict more elements from LRU maps when they are full.
I see. But in my understanding, for the global LRU list, the eviction (the invocation of htab_lru_map_delete_node) will be possible if the free element is less than LOCAL_FREE_TARGET(128) * nr_running_cpus. Now the number of free elements is 1000 as defined in __test(), the number of vCPU is 8 in my local VM setup (BPF CI also uses 8 vCPUs) and it is hard to trigger the eviction because 8 * 128 is roughly equal with 1000. So I suggest to decrease the number of free elements to 512 and the number of threads to 8, or adjust the number of running thread and free elements according to the number of online CPUs.
Re: [v3 PATCH bpf-next 5/6] selftests/bpf: test map percpu stats
Quoted 14 lines · Hou Tao
Hi,
Quoted 11 lines · Anton Protopopov
Quoted 10 lines · Hou Tao
Hi,
Quoted 7 lines · Anton Protopopov
Add a new map test,
map_percpu_stats.c, which is checking the correctness of map's percpu elements counters. For supported maps the test upserts a number of elements, checks the correctness of the counters, then deletes all the elements and checks again that the counters sum drops down to zero.The following map types are tested:
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
*
BPF_MAP_TYPE_HASH,BPF_F_NO_PREALLOC
*BPF_MAP_TYPE_PERCPU_HASH,BPF_F_NO_PREALLOC
*BPF_MAP_TYPE_HASH,
*BPF_MAP_TYPE_PERCPU_HASH,
*BPF_MAP_TYPE_LRU_HASH
*BPF_MAP_TYPE_LRU_PERCPU_HASHA test for
BPF_MAP_TYPE_HASH_OF_MAPSis also needed.We could also exercise the test for LRU map with
BPF_F_NO_COMMON_LRU.
Thanks, added.
Quoted 28 lines
Quoted 1 line
SNIP
Quoted 24 lines
Quoted 16 lines
Quoted 16 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +10
-0@@ -0,0 +1,336 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2023 Isovalent */ + +#include <errno.h> +#include <unistd.h> +#include <pthread.h> + +#include <bpf/bpf.h> +#include <bpf/libbpf.h> +Quoted 8 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c line 17
Quoted 7 lines
tools/testing/selftests/bpf/map_tests/map_percpu_stats.c new file +7
-0+#include <bpf_util.h> +#include <test_maps.h> + +#include "map_percpu_stats.skel.h" + +#define MAX_ENTRIES 16384 +#define N_THREADS 37Why 37 thread is needed here ? Does a small number of threads work as well ?
Quoted 1 line
This was used to evict more elements from LRU maps when they are full.
Quoted 9 lines
I see. But in my understanding, for the global LRU list, the eviction (the invocation of
htab_lru_map_delete_node) will be possible if the free element is less thanLOCAL_FREE_TARGET(128)*nr_running_cpus. Now the number of free elements is 1000 as defined in__test(), the number of vCPU is 8 in my local VM setup (BPF CI also uses 8 vCPUs) and it is hard to trigger the eviction because 8 * 128 is roughly equal with 1000. So I suggest to decrease the number of free elements to 512 and the number of threads to 8, or adjust the number of running thread and free elements according to the number of online CPUs.
Yes, makes sense. I've changed the test to use 8 threads and offset of 512.
PATCH v3 6/6 selftests/bpf: check that ->elem_count is non-zero for the hash map
In reply to Anton ProtopopovPrevious commits populated the ->elem_count per-cpu pointer for hash maps. Check that this pointer is non-NULL in an existing map.
--- tools/testing/selftests/bpf/progs/map_ptr_kern.c | 3 +++ 1 file changed, 3 insertions(+)
tools/testing/selftests/bpf/progs/map_ptr_kern.c +3 -0
@@ -33,6 +33,7 @@ struct bpf_map {__u32 value_size;__u32 max_entries;__u32 id; +__s64 *elem_count; } __attribute__((preserve_access_index)); static inline int check_bpf_map_fields(struct bpf_map *map, __u32 key_size,
@@ -111,6 +112,8 @@ static inline int check_hash(void)VERIFY (check_default_noinline (&hash->map, map)); +VERIFY (map->elem_count !=NULL ); +VERIFY (hash->n_buckets ==MAX_ENTRIES );VERIFY (hash->elem_size ==64 );
Signature · 1 line
2.34.1