bpf: Introduce BPF_F_CPU and BPF_F_ALL_CPUS flags for percpu maps
Subsystems (5)
v9 cover bpf: Introduce BPF_F_CPU and BPF_F_ALL_CPUS flags for percpu maps
This patch set introduces the BPF_F_CPU and BPF_F_ALL_CPUS flags for percpu maps, as the requirement of BPF_F_ALL_CPUS flag for percpu_array maps was discussed in the thread of
"[PATCH bpf-next v3 0/4] bpf: Introduce global percpu data"[1].
The goal of BPF_F_ALL_CPUS flag is to reduce data caching overhead in light skeletons by allowing a single value to be reused to update values across all CPUs. This avoids the M:N problem where M cached values are used to update a map on N CPUs kernel.
The BPF_F_CPU flag is accompanied by *flags*-embedded cpu info, which specifies the target CPU for the operation:
* For lookup operations: the flag field alongside cpu info enable querying a value on the specified CPU.
* For update operations: the flag field alongside cpu info enable updating value for specified CPU.
Links:
[1] https://lore.kernel.org/bpf/[email protected]/
Changes:
v8 -> v9:
* Change value type from u64 to u32 in selftests.
* Address comments from Andrii:
* Keep value_size unaligned and update everywhere for consistency when cpu flags are specified.
* Update value by getting pointer for percpu hash and percpu cgroup_storage maps.
v7 -> v8:
* Address comments from Andrii:
* Check BPF_F_LOCK when update percpu_array, percpu_hash and lru_percpu_hash maps.
* Refactor flags check in __htab_map_lookup_and_delete_batch().
* Keep value_size unaligned and copy value using copy_map_value() in __htab_map_lookup_and_delete_batch() when BPF_F_CPU is specified.
* Update warn message in libbpf's validate_map_op().
* Update comment of libbpf's bpf_map__lookup_elem().
v6 -> v7:
* Get correct value size for percpu_hash and lru_percpu_hash in update_batch API.
* Set 'count' as 'max_entries' in test cases for lookup_batch API.
* Address comment from Alexei:
* Move cpu flags check into bpf_map_check_op_flags().
v5 -> v6:
* Move bpf_map_check_op_flags() from 'bpf.h' to 'syscall.c'.
* Address comments from Alexei:
* Drop the refactoring code of data copying logic for percpu maps.
* Drop bpf_map_check_op_flags() wrappers.
v4 -> v5:
* Address comments from Andrii:
* Refactor data copying logic for all percpu maps.
* Drop this_cpu_ptr() micro-optimization.
* Drop cpu check in libbpf's validate_map_op().
* Enhance bpf_map_check_op_flags() using *allowed flags* instead of
'extra_flags_mask'.
v3 -> v4:
* Address comments from Andrii:
* Remove unnecessary map_type check in bpf_map_value_size().
* Reduce code churn.
* Remove unnecessary do_delete check in
__htab_map_lookup_and_delete_batch().
* Introduce bpf_percpu_copy_to_user() and bpf_percpu_copy_from_user().
* Rename check_map_flags() to bpf_map_check_op_flags() with
extra_flags_mask.
* Add human-readable pr_warn() explanations in validate_map_op().
* Use flags in bpf_map__delete_elem() and
bpf_map__lookup_and_delete_elem(). * Drop "for alignment reasons".
v3 link: https://lore.kernel.org/bpf/[email protected]/
v2 -> v3:
* Address comments from Alexei:
* Use BPF_F_ALL_CPUS instead of BPF_ALL_CPUS magic.
* Introduce these two cpu flags for all percpu maps.
* Address comments from Jiri:
* Reduce some unnecessary u32 cast.
* Refactor more generic map flags check function.
* A code style issue.
v2 link: https://lore.kernel.org/bpf/[email protected]/
v1 -> v2:
* Address comments from Andrii:
* Embed cpu info as high 32 bits of *flags* totally.
* Use ERANGE instead of E2BIG.
* Few format issues.
Leon Hwang (7):
bpf: Introduce internal bpf_map_check_op_flags helper function
bpf: Introduce BPF_F_CPU and BPF_F_ALL_CPUS flags
bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_array
maps
bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_hash
and lru_percpu_hash maps
bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for
percpu_cgroup_storage maps
libbpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu maps selftests/bpf: Add cases to test BPF_F_CPU and BPF_F_ALL_CPUS flags
include/linux/bpf-cgroup.h | 4 +- include/linux/bpf.h | 44 +++- include/uapi/linux/bpf.h | 2 + kernel/bpf/arraymap.c | 25 +- kernel/bpf/hashtab.c | 82 ++++-- kernel/bpf/local_storage.c | 27 +- kernel/bpf/syscall.c | 65 ++--- tools/include/uapi/linux/bpf.h | 2 + tools/lib/bpf/bpf.h | 8 + tools/lib/bpf/libbpf.c | 26 +- tools/lib/bpf/libbpf.h | 21 +- .../selftests/bpf/prog_tests/percpu_alloc.c | 248 ++++++++++++++++++ .../selftests/bpf/progs/percpu_alloc_array.c | 32 +++ 13 files changed, 487 insertions(+), 99 deletions(-)
--
2.51.0
PATCH v9 1/7 bpf: Introduce internal bpf_map_check_op_flags helper function
It is to unify map flags checking for lookup_elem, update_elem, lookup_batch and update_batch APIs.
Therefore, it will be convenient to check BPF_F_CPU and BPF_F_ALL_CPUS flags in it for these APIs in next patch.
--- include/linux/bpf.h | 11 +++++++++++ kernel/bpf/syscall.c | 34 +++++++++++----------------------- 2 files changed, 22 insertions(+), 23 deletions(-)
include/linux/bpf.h +11 -0
@@ -3768,4 +3768,15 @@ int bpf_prog_get_file_line(struct bpf_prog *prog, unsigned long ip, const char * const char **linep, int *nump);struct bpf_prog *bpf_prog_find_from_stack (void ); +static inline int bpf_map_check_op_flags (struct bpf_map *map,u64 flags,u64 allowed_flags) +{ +if (flags & ~allowed_flags) +return -EINVAL ; + +if ((flags &BPF_F_LOCK ) && !btf_record_has_field (map->record,BPF_SPIN_LOCK )) +return -EINVAL ; + +return 0 ; +} +#endif /* _LINUX_BPF_H */
kernel/bpf/syscall.c +11 -23
@@ -1709,9 +1709,6 @@ static int map_lookup_elem(union bpf_attr *attr)if (CHECK_ATTR (BPF_MAP_LOOKUP_ELEM ))return -EINVAL ; -if (attr->flags & ~BPF_F_LOCK ) -return -EINVAL ; -CLASS (fd, f)(attr->map_fd); map =__bpf_map_get (f);if (IS_ERR (map))
@@ -1719,9 +1716,9 @@ static int map_lookup_elem(union bpf_attr *attr)if (!(map_get_sys_perms (map, f) &FMODE_CAN_READ ))return -EPERM ; -if ((attr->flags &BPF_F_LOCK ) && - !btf_record_has_field (map->record,BPF_SPIN_LOCK )) -return -EINVAL ; + err =bpf_map_check_op_flags (map, attr->flags,BPF_F_LOCK ); +if (err) +return err; key =__bpf_copy_key (ukey, map->key_size);if (IS_ERR (key))
@@ -1784,11 +1781,9 @@ static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr) goto err_put; } -if ((attr->flags &BPF_F_LOCK ) && - !btf_record_has_field (map->record,BPF_SPIN_LOCK )) { - err = -EINVAL ; + err =bpf_map_check_op_flags (map, attr->flags, ~0 ); +if (err) goto err_put; - } key =___bpf_copy_key (ukey, map->key_size);if (IS_ERR (key)) {
@@ -1992,13 +1987,9 @@ int generic_map_update_batch(struct bpf_map *map, struct file *map_file,void *key, *value;int err =0 ; -if (attr->batch.elem_flags & ~BPF_F_LOCK ) -return -EINVAL ; - -if ((attr->batch.elem_flags &BPF_F_LOCK ) && - !btf_record_has_field (map->record,BPF_SPIN_LOCK )) { -return -EINVAL ; - } + err =bpf_map_check_op_flags (map, attr->batch.elem_flags,BPF_F_LOCK ); +if (err) +return err; value_size =bpf_map_value_size (map);
@@ -2055,12 +2046,9 @@ int generic_map_lookup_batch(struct bpf_map *map,u32 value_size, cp, max_count;int err; -if (attr->batch.elem_flags & ~BPF_F_LOCK ) -return -EINVAL ; - -if ((attr->batch.elem_flags &BPF_F_LOCK ) && - !btf_record_has_field (map->record,BPF_SPIN_LOCK )) -return -EINVAL ; + err =bpf_map_check_op_flags (map, attr->batch.elem_flags,BPF_F_LOCK ); +if (err) +return err; value_size =bpf_map_value_size (map);
Signature · 1 line
2.51.0
PATCH v9 2/7 bpf: Introduce BPF_F_CPU and BPF_F_ALL_CPUS flags
In reply to Leon HwangIntroduce BPF_F_CPU and BPF_F_ALL_CPUS flags and check them for following APIs:
* 'map_lookup_elem()'
* 'map_update_elem()'
* 'generic_map_lookup_batch()'
* 'generic_map_update_batch()'
And, get the correct value size for these APIs.
--- include/linux/bpf.h | 23 ++++++++++++++++++++++- include/uapi/linux/bpf.h | 2 ++ kernel/bpf/syscall.c | 31 +++++++++++++++++-------------- tools/include/uapi/linux/bpf.h | 2 ++ 4 files changed, 43 insertions(+), 15 deletions(-)
include/linux/bpf.h +22 -1
@@ -3768,14 +3768,35 @@ int bpf_prog_get_file_line(struct bpf_prog *prog, unsigned long ip, const char * const char **linep, int *nump);struct bpf_prog *bpf_prog_find_from_stack (void ); +static inline bool bpf_map_supports_cpu_flags (enum bpf_map_type map_type) +{ +return false; +} +static inline int bpf_map_check_op_flags (struct bpf_map *map,u64 flags,u64 allowed_flags) { -if (flags & ~allowed_flags) +u32 cpu; + +if ((u32 )flags & ~allowed_flags)return -EINVAL ;if ((flags &BPF_F_LOCK ) && !btf_record_has_field (map->record,BPF_SPIN_LOCK ))return -EINVAL ; +if (!(flags &BPF_F_CPU ) && flags >>32 ) +return -EINVAL ; + +if (flags & (BPF_F_CPU |BPF_F_ALL_CPUS )) { +if (!bpf_map_supports_cpu_flags (map->map_type)) +return -EINVAL ; +if ((flags &BPF_F_CPU ) && (flags &BPF_F_ALL_CPUS )) +return -EINVAL ; + + cpu = flags >>32 ; +if ((flags &BPF_F_CPU ) && cpu >=num_possible_cpus ()) +return -ERANGE ; + } +return 0 ; }
include/uapi/linux/bpf.h +2 -0
@@ -1372,6 +1372,8 @@ enum {BPF_NOEXIST =1 ,/* create new element if it didn't exist */ BPF_EXIST =2 ,/* update existing element */ BPF_F_LOCK =4 ,/* spin_lock-ed map_lookup/map_update */ +BPF_F_CPU =8 ,/* cpu flag for percpu maps, upper 32-bit of flags is a cpu number */ + BPF_F_ALL_CPUS = 16, /* update value across all CPUs for percpu maps */ };/* flags for BPF_MAP_CREATE command */
kernel/bpf/syscall.c +17 -14
@@ -133,12 +133,14 @@ bool bpf_map_write_active(const struct bpf_map *map)return atomic64_read (&map->writecnt) !=0 ; } -static u32 bpf_map_value_size (const struct bpf_map *map) -{ -if (map->map_type ==BPF_MAP_TYPE_PERCPU_HASH || - map->map_type ==BPF_MAP_TYPE_LRU_PERCPU_HASH || - map->map_type ==BPF_MAP_TYPE_PERCPU_ARRAY || - map->map_type ==BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE ) +static u32 bpf_map_value_size (const struct bpf_map *map,u64 flags) +{ +if (flags & (BPF_F_CPU |BPF_F_ALL_CPUS )) +return map->value_size; +else if (map->map_type ==BPF_MAP_TYPE_PERCPU_HASH || + map->map_type ==BPF_MAP_TYPE_LRU_PERCPU_HASH || + map->map_type ==BPF_MAP_TYPE_PERCPU_ARRAY || + map->map_type ==BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE )return round_up (map->value_size,8 ) *num_possible_cpus ();else if (IS_FD_MAP (map))return sizeof (u32);
@@ -1716,7 +1718,7 @@ static int map_lookup_elem(union bpf_attr *attr)if (!(map_get_sys_perms (map, f) &FMODE_CAN_READ ))return -EPERM ; - err =bpf_map_check_op_flags (map, attr->flags,BPF_F_LOCK ); + err =bpf_map_check_op_flags (map, attr->flags,BPF_F_LOCK |BPF_F_CPU );if (err)return err;
@@ -1724,7 +1726,7 @@ static int map_lookup_elem(union bpf_attr *attr)if (IS_ERR (key))return PTR_ERR (key); - value_size =bpf_map_value_size (map); + value_size =bpf_map_value_size (map, attr->flags); err = -ENOMEM ; value =kvmalloc (value_size,GFP_USER | __GFP_NOWARN);
@@ -1791,7 +1793,7 @@ static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr) goto err_put; } - value_size =bpf_map_value_size (map); + value_size =bpf_map_value_size (map, attr->flags); value =kvmemdup_bpfptr (uvalue, value_size);if (IS_ERR (value)) { err =PTR_ERR (value);
@@ -1987,11 +1989,12 @@ int generic_map_update_batch(struct bpf_map *map, struct file *map_file,void *key, *value;int err =0 ; - err =bpf_map_check_op_flags (map, attr->batch.elem_flags,BPF_F_LOCK ); + err =bpf_map_check_op_flags (map, attr->batch.elem_flags, +BPF_F_LOCK |BPF_F_CPU |BPF_F_ALL_CPUS );if (err)return err; - value_size =bpf_map_value_size (map); + value_size =bpf_map_value_size (map, attr->batch.elem_flags); max_count = attr->batch.count;if (!max_count)
@@ -2046,11 +2049,11 @@ int generic_map_lookup_batch(struct bpf_map *map,u32 value_size, cp, max_count;int err; - err =bpf_map_check_op_flags (map, attr->batch.elem_flags,BPF_F_LOCK ); + err =bpf_map_check_op_flags (map, attr->batch.elem_flags,BPF_F_LOCK |BPF_F_CPU );if (err)return err; - value_size =bpf_map_value_size (map); + value_size =bpf_map_value_size (map, attr->batch.elem_flags); max_count = attr->batch.count;if (!max_count)
@@ -2172,7 +2175,7 @@ static int map_lookup_and_delete_elem(union bpf_attr *attr) goto err_put; } - value_size =bpf_map_value_size (map); + value_size =bpf_map_value_size (map,0 ); err = -ENOMEM ; value =kvmalloc (value_size,GFP_USER | __GFP_NOWARN);
tools/include/uapi/linux/bpf.h +2 -0
@@ -1372,6 +1372,8 @@ enum {BPF_NOEXIST =1 ,/* create new element if it didn't exist */ BPF_EXIST =2 ,/* update existing element */ BPF_F_LOCK =4 ,/* spin_lock-ed map_lookup/map_update */ +BPF_F_CPU =8 ,/* cpu flag for percpu maps, upper 32-bit of flags is a cpu number */ + BPF_F_ALL_CPUS = 16, /* update value across all CPUs for percpu maps */ };/* flags for BPF_MAP_CREATE command */
Signature · 1 line
2.51.0
PATCH v9 3/7 bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_array maps
In reply to Leon HwangIntroduce support for the BPF_F_ALL_CPUS flag in percpu_array maps to allow updating values for all CPUs with a single value for both update_elem and update_batch APIs.
Introduce support for the BPF_F_CPU flag in percpu_array maps to allow:
* update value for specified CPU for both update_elem and update_batch APIs.
* lookup value for specified CPU for both lookup_elem and lookup_batch APIs.
The BPF_F_CPU flag is passed via:
* map_flags of lookup_elem and update_elem APIs along with embedded cpu info.
* elem_flags of lookup_batch and update_batch APIs along with embedded cpu info.
--- include/linux/bpf.h | 9 +++++++-- kernel/bpf/arraymap.c | 25 ++++++++++++++++++++----- kernel/bpf/syscall.c | 2 +- 3 files changed, 28 insertions(+), 8 deletions(-)
include/linux/bpf.h +7 -2
@@ -2718,7 +2718,7 @@ int map_set_for_each_callback_args(struct bpf_verifier_env *env, struct bpf_func_state *callee);int bpf_percpu_hash_copy (struct bpf_map *map,void *key,void *value); -int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value); +int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value,u64 flags);int bpf_percpu_hash_update (struct bpf_map *map,void *key,void *value,u64 flags); int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value,
@@ -3770,7 +3770,12 @@ struct bpf_prog *bpf_prog_find_from_stack(void);static inline bool bpf_map_supports_cpu_flags (enum bpf_map_type map_type) { -return false; +switch (map_type) { +case BPF_MAP_TYPE_PERCPU_ARRAY : +return true; +default : +return false; + } }static inline int bpf_map_check_op_flags (struct bpf_map *map,u64 flags,u64 allowed_flags)
kernel/bpf/arraymap.c +20 -5
@@ -307,7 +307,7 @@ static void *percpu_array_map_lookup_percpu_elem(struct bpf_map *map, void *key,return per_cpu_ptr (array->pptrs[index & array->index_mask], cpu); } -int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value) +int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value,u64 map_flags) { struct bpf_array *array = container_of(map, struct bpf_array, map);u32 index = *(u32 *)key;
@@ -325,11 +325,18 @@ int bpf_percpu_array_copy(struct bpf_map *map, void *key, void *value) size = array->elem_size;rcu_read_lock (); pptr = array->pptrs[index & array->index_mask]; +if (map_flags &BPF_F_CPU ) { + cpu = map_flags >>32 ; +copy_map_value (map, value,per_cpu_ptr (pptr, cpu)); +check_and_init_map_value (map, value); + goto unlock; + }for_each_possible_cpu (cpu) {copy_map_value_long (map, value + off,per_cpu_ptr (pptr, cpu));check_and_init_map_value (map, value + off); off += size; } +unlock:rcu_read_unlock ();return 0 ; }
@@ -399,10 +406,11 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value, struct bpf_array *array = container_of(map, struct bpf_array, map);u32 index = *(u32 *)key;void __percpu *pptr; -int cpu, off =0 ; +void *ptr;u32 size; +int cpu; -if (unlikely (map_flags >BPF_EXIST )) +if (unlikely ((map_flags &BPF_F_LOCK ) || (u32 )map_flags >BPF_F_ALL_CPUS ))/* unknown flags */ return -EINVAL ;
@@ -423,11 +431,18 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value, size = array->elem_size;rcu_read_lock (); pptr = array->pptrs[index & array->index_mask]; +if (map_flags &BPF_F_CPU ) { + cpu = map_flags >>32 ; +copy_map_value (map,per_cpu_ptr (pptr, cpu), value); +bpf_obj_free_fields (array->map.record,per_cpu_ptr (pptr, cpu)); + goto unlock; + }for_each_possible_cpu (cpu) { -copy_map_value_long (map,per_cpu_ptr (pptr, cpu), value + off); + ptr = (map_flags &BPF_F_ALL_CPUS ) ? value : value + size * cpu; +copy_map_value (map,per_cpu_ptr (pptr, cpu), ptr);bpf_obj_free_fields (array->map.record,per_cpu_ptr (pptr, cpu)); - off += size; } +unlock:rcu_read_unlock ();return 0 ; }
kernel/bpf/syscall.c +1 -1
@@ -318,7 +318,7 @@ static int bpf_map_copy_value(struct bpf_map *map, void *key, void *value, map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH) { err =bpf_percpu_hash_copy (map, key, value); } else if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) { - err =bpf_percpu_array_copy (map, key, value); + err =bpf_percpu_array_copy (map, key, value, flags); } else if (map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE) { err =bpf_percpu_cgroup_storage_copy (map, key, value); } else if (map->map_type == BPF_MAP_TYPE_STACK_TRACE) {
Signature · 1 line
2.51.0
PATCH v9 4/7 bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_hash and lru_percpu_hash maps
In reply to Leon HwangIntroduce BPF_F_ALL_CPUS flag support for percpu_hash and lru_percpu_hash maps to allow updating values for all CPUs with a single value for both update_elem and update_batch APIs.
Introduce BPF_F_CPU flag support for percpu_hash and lru_percpu_hash maps to allow:
* update value for specified CPU for both update_elem and update_batch APIs.
* lookup value for specified CPU for both lookup_elem and lookup_batch APIs.
The BPF_F_CPU flag is passed via:
* map_flags along with embedded cpu info.
* elem_flags along with embedded cpu info.
--- include/linux/bpf.h | 4 ++- kernel/bpf/hashtab.c | 82 +++++++++++++++++++++++++++++--------------- kernel/bpf/syscall.c | 2 +- 3 files changed, 59 insertions(+), 29 deletions(-)
include/linux/bpf.h +3 -1
@@ -2717,7 +2717,7 @@ int map_set_for_each_callback_args(struct bpf_verifier_env *env,struct bpf_func_state *caller, struct bpf_func_state *callee); -int bpf_percpu_hash_copy (struct bpf_map *map,void *key,void *value); +int bpf_percpu_hash_copy (struct bpf_map *map,void *key,void *value,u64 flags);int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value,u64 flags);int bpf_percpu_hash_update (struct bpf_map *map,void *key,void *value,u64 flags);
@@ -3772,6 +3772,8 @@ static inline bool bpf_map_supports_cpu_flags(enum bpf_map_type map_type) {switch (map_type) {case BPF_MAP_TYPE_PERCPU_ARRAY : +case BPF_MAP_TYPE_PERCPU_HASH : +case BPF_MAP_TYPE_LRU_PERCPU_HASH :return true;default :return false;
kernel/bpf/hashtab.c +55 -27
207 lines not on this page · raw message
kernel/bpf/syscall.c +1 -1
@@ -316,7 +316,7 @@ static int bpf_map_copy_value(struct bpf_map *map, void *key, void *value,bpf_disable_instrumentation ();if (map->map_type ==BPF_MAP_TYPE_PERCPU_HASH || map->map_type ==BPF_MAP_TYPE_LRU_PERCPU_HASH ) { - err =bpf_percpu_hash_copy (map, key, value); + err =bpf_percpu_hash_copy (map, key, value, flags); }else if (map->map_type ==BPF_MAP_TYPE_PERCPU_ARRAY ) { err =bpf_percpu_array_copy (map, key, value, flags); }else if (map->map_type ==BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE ) {
Signature · 1 line
2.51.0
PATCH v9 5/7 bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_cgroup_storage maps
In reply to Leon HwangIntroduce BPF_F_ALL_CPUS flag support for percpu_cgroup_storage maps to allow updating values for all CPUs with a single value for update_elem API.
Introduce BPF_F_CPU flag support for percpu_cgroup_storage maps to allow:
* update value for specified CPU for update_elem API.
* lookup value for specified CPU for lookup_elem API.
The BPF_F_CPU flag is passed via map_flags along with embedded cpu info.
--- include/linux/bpf-cgroup.h | 4 ++-- include/linux/bpf.h | 1 + kernel/bpf/local_storage.c | 27 ++++++++++++++++++++------- kernel/bpf/syscall.c | 2 +- 4 files changed, 24 insertions(+), 10 deletions(-)
include/linux/bpf-cgroup.h +2 -2
@@ -172,7 +172,7 @@ void bpf_cgroup_storage_link(struct bpf_cgroup_storage *storage,void bpf_cgroup_storage_unlink (struct bpf_cgroup_storage *storage);int bpf_cgroup_storage_assign (struct bpf_prog_aux *aux,struct bpf_map *map); -int bpf_percpu_cgroup_storage_copy (struct bpf_map *map,void *key,void *value); +int bpf_percpu_cgroup_storage_copy (struct bpf_map *map,void *key,void *value,u64 flags);int bpf_percpu_cgroup_storage_update (struct bpf_map *map,void *key,void *value,u64 flags);
@@ -467,7 +467,7 @@ static inline struct bpf_cgroup_storage *bpf_cgroup_storage_alloc(static inline void bpf_cgroup_storage_free (struct bpf_cgroup_storage *storage) {}static inline int bpf_percpu_cgroup_storage_copy (struct bpf_map *map,void *key, -void *value) { +void *value,u64 flags) {return 0 ; } static inline int bpf_percpu_cgroup_storage_update(struct bpf_map *map,
include/linux/bpf.h +1 -0
@@ -3774,6 +3774,7 @@ static inline bool bpf_map_supports_cpu_flags(enum bpf_map_type map_type)case BPF_MAP_TYPE_PERCPU_ARRAY :case BPF_MAP_TYPE_PERCPU_HASH :case BPF_MAP_TYPE_LRU_PERCPU_HASH : +case BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE :return true;default :return false;
kernel/bpf/local_storage.c +20 -7
@@ -180,7 +180,7 @@ static long cgroup_storage_update_elem(struct bpf_map *map, void *key, }int bpf_percpu_cgroup_storage_copy (struct bpf_map *_map,void *key, -void *value) +void *value,u64 map_flags) {struct bpf_cgroup_storage_map *map =map_to_storage (_map);struct bpf_cgroup_storage *storage;
@@ -198,12 +198,18 @@ int bpf_percpu_cgroup_storage_copy(struct bpf_map *_map, void *key,* access 'value_size' of them, so copying rounded areas * will not leak any kernel data */ +if (map_flags &BPF_F_CPU ) { + cpu = map_flags >>32 ; +memcpy (value,per_cpu_ptr (storage->percpu_buf, cpu), _map->value_size); + goto unlock; + } size =round_up (_map->value_size,8 );for_each_possible_cpu (cpu) {bpf_long_memcpy (value + off,per_cpu_ptr (storage->percpu_buf, cpu), size); off += size; } +unlock:rcu_read_unlock ();return 0 ; }
@@ -213,10 +219,11 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map *_map, void *key, {struct bpf_cgroup_storage_map *map =map_to_storage (_map);struct bpf_cgroup_storage *storage; -int cpu, off =0 ; +void *ptr;u32 size; +int cpu; -if (map_flags !=BPF_ANY && map_flags !=BPF_EXIST ) +if ((u32 )map_flags & ~(BPF_ANY |BPF_EXIST |BPF_F_CPU |BPF_F_ALL_CPUS ))return -EINVAL ;rcu_read_lock ();
@@ -232,12 +239,18 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map *_map, void *key,* returned or zeros which were zero-filled by percpu_alloc, * so no kernel data leaks possible */ - size =round_up (_map->value_size,8 ); + size = (map_flags & (BPF_F_CPU |BPF_F_ALL_CPUS )) ? _map->value_size : +round_up (_map->value_size,8 ); +if (map_flags &BPF_F_CPU ) { + cpu = map_flags >>32 ; +memcpy (per_cpu_ptr (storage->percpu_buf, cpu), value, size); + goto unlock; + }for_each_possible_cpu (cpu) { -bpf_long_memcpy (per_cpu_ptr (storage->percpu_buf, cpu), - value + off, size); - off += size; + ptr = (map_flags &BPF_F_ALL_CPUS ) ? value : value + size * cpu; +memcpy (per_cpu_ptr (storage->percpu_buf, cpu), ptr, size); } +unlock:rcu_read_unlock ();return 0 ; }
kernel/bpf/syscall.c +1 -1
@@ -320,7 +320,7 @@ static int bpf_map_copy_value(struct bpf_map *map, void *key, void *value, }else if (map->map_type ==BPF_MAP_TYPE_PERCPU_ARRAY ) { err =bpf_percpu_array_copy (map, key, value, flags); }else if (map->map_type ==BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE ) { - err =bpf_percpu_cgroup_storage_copy (map, key, value); + err =bpf_percpu_cgroup_storage_copy (map, key, value, flags); }else if (map->map_type ==BPF_MAP_TYPE_STACK_TRACE ) { err =bpf_stackmap_extract (map, key, value, false); }else if (IS_FD_ARRAY (map) ||IS_FD_PROG_ARRAY (map)) {
Signature · 1 line
2.51.0
PATCH v9 6/7 libbpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu maps
In reply to Leon HwangAdd libbpf support for the BPF_F_CPU flag for percpu maps by embedding the cpu info into the high 32 bits of:
1. **flags**: bpf_map_lookup_elem_flags(), bpf_map__lookup_elem(), bpf_map_update_elem() and bpf_map__update_elem()
2. **opts->elem_flags**: bpf_map_lookup_batch() and bpf_map_update_batch()
And the flag can be BPF_F_ALL_CPUS, but cannot be
'BPF_F_CPU | BPF_F_ALL_CPUS'.
Behavior:
* If the flag is BPF_F_ALL_CPUS, the update is applied across all CPUs.
* If the flag is BPF_F_CPU, it updates value only to the specified CPU.
* If the flag is BPF_F_CPU, lookup value only from the specified CPU.
* lookup does not support BPF_F_ALL_CPUS.
--- tools/lib/bpf/bpf.h | 8 ++++++++ tools/lib/bpf/libbpf.c | 26 ++++++++++++++++++++------ tools/lib/bpf/libbpf.h | 21 ++++++++------------- 3 files changed, 36 insertions(+), 19 deletions(-)
tools/lib/bpf/bpf.h +8 -0
@@ -289,6 +289,14 @@ LIBBPF_API int bpf_map_lookup_and_delete_batch(int fd, void *in_batch,* Update spin_lock-ed map elements. This must be * specified if the map value contains a spinlock. * +* **BPF_F_CPU** +* As for percpu maps, update value on the specified CPU. And the cpu +* info is embedded into the high 32 bits of **opts->elem_flags**. +* +* **BPF_F_ALL_CPUS** +* As for percpu maps, update value across all CPUs. This flag cannot +* be used with BPF_F_CPU at the same time. +* * @param fd BPF map file descriptor * @param keys pointer to an array of *count* keys * @param values pointer to an array of *count* values
tools/lib/bpf/libbpf.c +20 -6
@@ -10671,7 +10671,7 @@ bpf_object__find_map_fd_by_name(const struct bpf_object *obj, const char *name) }static int validate_map_op (const struct bpf_map *map,size_t key_sz, -size_t value_sz,bool check_value_sz) +size_t value_sz,bool check_value_sz,__u64 flags) {if (!map_is_created (map))/* map is not yet created */ return -ENOENT ;
@@ -10698,6 +10698,20 @@ static int validate_map_op(const struct bpf_map *map, size_t key_sz,int num_cpu =libbpf_num_possible_cpus ();size_t elem_sz =roundup (map->def.value_size,8 ); +if (flags & (BPF_F_CPU |BPF_F_ALL_CPUS )) { +if ((flags &BPF_F_CPU ) && (flags &BPF_F_ALL_CPUS )) { +pr_warn ("map '%s': BPF_F_CPU and BPF_F_ALL_CPUS are mutually exclusive\n" , + map->name); +return -EINVAL ; + } +if (map->def.value_size != value_sz) { +pr_warn ("map '%s': unexpected value size %zu provided for either BPF_F_CPU or BPF_F_ALL_CPUS, expected %u\n" , + map->name, value_sz, map->def.value_size); +return -EINVAL ; + } +break ; + } +if (value_sz != num_cpu * elem_sz) {pr_warn ("map '%s': unexpected value size %zu provided for per-CPU map, expected %d * %zu = %zd\n" , map->name, value_sz, num_cpu, elem_sz, num_cpu * elem_sz);
@@ -10722,7 +10736,7 @@ int bpf_map__lookup_elem(const struct bpf_map *map, {int err; - err =validate_map_op (map, key_sz, value_sz, true); + err =validate_map_op (map, key_sz, value_sz, true, flags);if (err)return libbpf_err (err);
@@ -10735,7 +10749,7 @@ int bpf_map__update_elem(const struct bpf_map *map, {int err; - err =validate_map_op (map, key_sz, value_sz, true); + err =validate_map_op (map, key_sz, value_sz, true, flags);if (err)return libbpf_err (err);
@@ -10747,7 +10761,7 @@ int bpf_map__delete_elem(const struct bpf_map *map, {int err; - err =validate_map_op (map, key_sz,0 , false/* check_value_sz */ ); + err =validate_map_op (map, key_sz,0 , false/* check_value_sz */ , flags);if (err)return libbpf_err (err);
@@ -10760,7 +10774,7 @@ int bpf_map__lookup_and_delete_elem(const struct bpf_map *map, {int err; - err =validate_map_op (map, key_sz, value_sz, true); + err =validate_map_op (map, key_sz, value_sz, true, flags);if (err)return libbpf_err (err);
@@ -10772,7 +10786,7 @@ int bpf_map__get_next_key(const struct bpf_map *map, {int err; - err =validate_map_op (map, key_sz,0 , false/* check_value_sz */ ); + err =validate_map_op (map, key_sz,0 , false/* check_value_sz */ ,0 );if (err)return libbpf_err (err);
tools/lib/bpf/libbpf.h +8 -13
@@ -1196,12 +1196,13 @@ LIBBPF_API struct bpf_map *bpf_map__inner_map(struct bpf_map *map);* @param key_sz size in bytes of key data, needs to match BPF map definition's **key_size** * @param value pointer to memory in which looked up value will be stored * @param value_sz size in byte of value data memory; it has to match BPF map -* definition's **value_size**. For per-CPU BPF maps value size has to be -* a product of BPF map value size and number of possible CPUs in the system -* (could be fetched with **libbpf_num_possible_cpus()**). Note also that for -* per-CPU values value size has to be aligned up to closest 8 bytes for -* alignment reasons, so expected size is: `round_up(value_size, 8) -* * libbpf_num_possible_cpus()`. +* definition's **value_size**. For per-CPU BPF maps, value size can be +* `value_size` if either **BPF_F_CPU** or **BPF_F_ALL_CPUS** is specified +* in **flags**, otherwise a product of BPF map value size and number of +* possible CPUs in the system (could be fetched with +* **libbpf_num_possible_cpus()**). Note also that for per-CPU values value +* size has to be aligned up to closest 8 bytes, so expected size is: +* `round_up(value_size, 8) * libbpf_num_possible_cpus()`. * @flags extra flags passed to kernel for this operation * @return 0, on success; negative error, otherwise *
@@ -1219,13 +1220,7 @@ LIBBPF_API int bpf_map__lookup_elem(const struct bpf_map *map,* @param key pointer to memory containing bytes of the key * @param key_sz size in bytes of key data, needs to match BPF map definition's **key_size** * @param value pointer to memory containing bytes of the value -* @param value_sz size in byte of value data memory; it has to match BPF map -* definition's **value_size**. For per-CPU BPF maps value size has to be -* a product of BPF map value size and number of possible CPUs in the system -* (could be fetched with **libbpf_num_possible_cpus()**). Note also that for -* per-CPU values value size has to be aligned up to closest 8 bytes for -* alignment reasons, so expected size is: `round_up(value_size, 8) -* * libbpf_num_possible_cpus()`. +* @param value_sz refer to **bpf_map__lookup_elem**'s description.' * @flags extra flags passed to kernel for this operation * @return 0, on success; negative error, otherwise *
Signature · 1 line
2.51.0
PATCH v9 7/7 selftests/bpf: Add cases to test BPF_F_CPU and BPF_F_ALL_CPUS flags
In reply to Leon HwangAdd test coverage for the new BPF_F_CPU and BPF_F_ALL_CPUS flags support in percpu maps. The following APIs are exercised:
* bpf_map_update_batch()
* bpf_map_lookup_batch()
* bpf_map_update_elem()
* bpf_map__update_elem()
* bpf_map_lookup_elem_flags()
* bpf_map__lookup_elem()
--- .../selftests/bpf/prog_tests/percpu_alloc.c | 248 ++++++++++++++++++ .../selftests/bpf/progs/percpu_alloc_array.c | 32 +++ 2 files changed, 280 insertions(+)
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c +248 -0
@@ -1,5 +1,6 @@// SPDX-License-Identifier: GPL-2.0 #include <test_progs.h> +#include "cgroup_helpers.h" #include "percpu_alloc_array.skel.h" #include "percpu_alloc_cgrp_local_storage.skel.h" #include "percpu_alloc_fail.skel.h"
@@ -115,6 +116,245 @@ static void test_failure(void) {RUN_TESTS (percpu_alloc_fail); } +static void test_percpu_map_op_cpu_flag (struct bpf_map *map,void *keys,size_t key_sz, +u32 max_entries,bool test_batch) +{ +size_t value_sz =sizeof (u32), value_sz_cpus, value_sz_total; +u32 *values =NULL , *values_percpu =NULL ; +int i, j, cpu, map_fd, nr_cpus, err; +const u32 value =0xDEADC0DE ; +u32 count = max_entries, v; +u64 batch =0 , flags; +void *values_row; +LIBBPF_OPTS (bpf_map_batch_opts, batch_opts); + + nr_cpus =libbpf_num_possible_cpus (); +if (!ASSERT_GT (nr_cpus,0 ,"libbpf_num_possible_cpus" )) +return ; + + value_sz_cpus = value_sz * nr_cpus; + values =calloc (max_entries, value_sz_cpus); +if (!ASSERT_OK_PTR (values,"calloc values" )) +return ; + + values_percpu =calloc (max_entries,roundup (value_sz,8 ) * nr_cpus); +if (!ASSERT_OK_PTR (values_percpu,"calloc values_percpu" )) { +free (values); +return ; + } + + value_sz_total = value_sz_cpus * max_entries; +memset (values,0 , value_sz_total); + + map_fd =bpf_map__fd (map); + flags =BPF_F_CPU |BPF_F_ALL_CPUS ; + err =bpf_map_lookup_elem_flags (map_fd, keys, values, flags); +if (!ASSERT_ERR (err,"bpf_map_lookup_elem_flags err" )) + goto out; + + err =bpf_map_update_elem (map_fd, keys, values, flags); +if (!ASSERT_ERR (err,"bpf_map_update_elem err" )) + goto out; + + flags = (u64 )nr_cpus <<32 |BPF_F_CPU ; + err =bpf_map_update_elem (map_fd, keys, values, flags); +if (!ASSERT_EQ (err, -ERANGE ,"bpf_map_update_elem -ERANGE" )) + goto out; + + err =bpf_map__update_elem (map, keys, key_sz, values, value_sz, flags); +if (!ASSERT_EQ (err, -ERANGE ,"bpf_map__update_elem -ERANGE" )) + goto out; + + err =bpf_map_lookup_elem_flags (map_fd, keys, values, flags); +if (!ASSERT_EQ (err, -ERANGE ,"bpf_map_lookup_elem_flags -ERANGE" )) + goto out; + + err =bpf_map__lookup_elem (map, keys, key_sz, values, value_sz, flags); +if (!ASSERT_EQ (err, -ERANGE ,"bpf_map__lookup_elem -ERANGE" )) + goto out; + +for (cpu =0 ; cpu < nr_cpus; cpu++) { +/* clear value on all cpus */ + values[0 ] =0 ; + flags =BPF_F_ALL_CPUS ; +for (i =0 ; i < max_entries; i++) { + err =bpf_map__update_elem (map, keys + i * key_sz, key_sz, values, + value_sz, flags); +if (!ASSERT_OK (err,"bpf_map__update_elem all_cpus" )) + goto out; + } + +/* update value on specified cpu */ +for (i =0 ; i < max_entries; i++) { + values[0 ] = value; + flags = (u64 )cpu <<32 |BPF_F_CPU ; + err =bpf_map__update_elem (map, keys + i * key_sz, key_sz, values, + value_sz, flags); +if (!ASSERT_OK (err,"bpf_map__update_elem specified cpu" )) + goto out; + +/* lookup then check value on CPUs */ +for (j =0 ; j < nr_cpus; j++) { + flags = (u64 )j <<32 |BPF_F_CPU ; + err =bpf_map__lookup_elem (map, keys + i * key_sz, key_sz, values, + value_sz, flags); +if (!ASSERT_OK (err,"bpf_map__lookup_elem specified cpu" )) + goto out; +if (!ASSERT_EQ (values[0 ], j != cpu ?0 : value, +"bpf_map__lookup_elem value on specified cpu" )) + goto out; + } + } + } + +if (!test_batch) + goto out; + + batch_opts.elem_flags = (u64 )nr_cpus <<32 |BPF_F_CPU ; + err =bpf_map_update_batch (map_fd, keys, values, &max_entries, &batch_opts); +if (!ASSERT_EQ (err, -ERANGE ,"bpf_map_update_batch -ERANGE" )) + goto out; + +for (cpu =0 ; cpu < nr_cpus; cpu++) { +memset (values,0 , value_sz_total); + +/* clear values across all CPUs */ + batch_opts.elem_flags =BPF_F_ALL_CPUS ; + err =bpf_map_update_batch (map_fd, keys, values, &max_entries, &batch_opts); +if (!ASSERT_OK (err,"bpf_map_update_batch all_cpus" )) + goto out; + +/* update values on specified CPU */ +for (i =0 ; i < max_entries; i++) + values[i] = value; + + batch_opts.elem_flags = (u64 )cpu <<32 |BPF_F_CPU ; + err =bpf_map_update_batch (map_fd, keys, values, &max_entries, &batch_opts); +if (!ASSERT_OK (err,"bpf_map_update_batch specified cpu" )) + goto out; + +/* lookup values on specified CPU */ +memset (values,0 , max_entries * value_sz); + err =bpf_map_lookup_batch (map_fd,NULL , &batch, keys, values, &count, &batch_opts); +if (!ASSERT_TRUE (!err || err == -ENOENT ,"bpf_map_lookup_batch specified cpu" )) + goto out; + +for (i =0 ; i < max_entries; i++) +if (!ASSERT_EQ (values[i], value, +"bpf_map_lookup_batch value on specified cpu" )) + goto out; + +/* lookup values from all CPUs */ + batch_opts.elem_flags =0 ; +memset (values_percpu,0 ,roundup (value_sz,8 ) * nr_cpus * max_entries); + err =bpf_map_lookup_batch (map_fd,NULL , &batch, keys, values_percpu, &count, + &batch_opts); +if (!ASSERT_TRUE (!err || err == -ENOENT ,"bpf_map_lookup_batch all_cpus" )) + goto out; + +for (i =0 ; i < max_entries; i++) { + values_row = (void *) values_percpu + +roundup (value_sz,8 ) * i * nr_cpus; +for (j =0 ; j < nr_cpus; j++) { + v = *(u32 *) (values_row +roundup (value_sz,8 ) * j); +if (!ASSERT_EQ (v, j != cpu ?0 : value, +"bpf_map_lookup_batch value all_cpus" )) + goto out; + } + } + } + +out: +free (values_percpu); +free (values); +} + +static void test_percpu_map_cpu_flag (enum bpf_map_type map_type) +{ +struct percpu_alloc_array *skel; +size_t key_sz =sizeof (int ); +int *keys =NULL , i, err; +struct bpf_map *map; +u32 max_entries; + + skel =percpu_alloc_array__open (); +if (!ASSERT_OK_PTR (skel,"percpu_alloc_array__open" )) +return ; + + map = skel->maps.percpu; +bpf_map__set_type (map, map_type); + + err =percpu_alloc_array__load (skel); +if (!ASSERT_OK (err,"test_percpu_alloc__load" )) + goto out; + + max_entries =bpf_map__max_entries (map); + keys =calloc (max_entries, key_sz); +if (!ASSERT_OK_PTR (keys,"calloc keys" )) + goto out; + +for (i =0 ; i < max_entries; i++) + keys[i] = i; + +test_percpu_map_op_cpu_flag (map, keys, key_sz, max_entries, true); +out: +if (keys) +free (keys); +percpu_alloc_array__destroy (skel); +} + +static void test_percpu_array_cpu_flag (void ) +{ +test_percpu_map_cpu_flag (BPF_MAP_TYPE_PERCPU_ARRAY ); +} + +static void test_percpu_hash_cpu_flag (void ) +{ +test_percpu_map_cpu_flag (BPF_MAP_TYPE_PERCPU_HASH ); +} + +static void test_lru_percpu_hash_cpu_flag (void ) +{ +test_percpu_map_cpu_flag (BPF_MAP_TYPE_LRU_PERCPU_HASH ); +} + +static void test_percpu_cgroup_storage_cpu_flag (void ) +{ +struct bpf_cgroup_storage_key key; +struct percpu_alloc_array *skel; +int cgroup =-1 , prog_fd, err; +struct bpf_map *map; + + skel =percpu_alloc_array__open_and_load (); +if (!ASSERT_OK_PTR (skel,"percpu_alloc_array__open_and_load" )) +return ; + + cgroup =create_and_get_cgroup ("/cg_percpu" ); +if (!ASSERT_GE (cgroup,0 ,"create_and_get_cgroup" )) + goto out; + + err =join_cgroup ("/cg_percpu" ); +if (!ASSERT_OK (err,"join_cgroup" )) + goto out; + + prog_fd =bpf_program__fd (skel->progs.cgroup_egress); + err =bpf_prog_attach (prog_fd, cgroup,BPF_CGROUP_INET_EGRESS ,0 ); +if (!ASSERT_OK (err,"bpf_prog_attach" )) + goto out; + + map = skel->maps.percpu_cgroup_storage; + err =bpf_map_get_next_key (bpf_map__fd (map),NULL , &key); +if (!ASSERT_OK (err,"bpf_map_get_next_key" )) + goto out; + +test_percpu_map_op_cpu_flag (map, &key,sizeof (key),1 , false); +out: +bpf_prog_detach2 (-1 , cgroup,BPF_CGROUP_INET_EGRESS ); +close (cgroup); +cleanup_cgroup_environment (); +percpu_alloc_array__destroy (skel); +} +void test_percpu_alloc (void ) {if (test__start_subtest ("array" ))
@@ -125,4 +365,12 @@ void test_percpu_alloc(void)test_cgrp_local_storage ();if (test__start_subtest ("failure_tests" ))test_failure (); +if (test__start_subtest ("cpu_flag_percpu_array" )) +test_percpu_array_cpu_flag (); +if (test__start_subtest ("cpu_flag_percpu_hash" )) +test_percpu_hash_cpu_flag (); +if (test__start_subtest ("cpu_flag_lru_percpu_hash" )) +test_lru_percpu_hash_cpu_flag (); +if (test__start_subtest ("cpu_flag_percpu_cgroup_storage" )) +test_percpu_cgroup_storage_cpu_flag (); }
tools/testing/selftests/bpf/progs/percpu_alloc_array.c +32 -0
@@ -187,4 +187,36 @@ int BPF_PROG(test_array_map_10)return 0 ; } +struct { + __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); + __uint(max_entries, 2); + __type(key, int); + __type(value, u32); +} percpu SEC(".maps"); + +SEC("?fentry/bpf_fentry_test1") +int BPF_PROG (test_percpu_array ,int x) +{ +u64 value =0xDEADC0DE ; +int key =0 ; + +bpf_map_update_elem (&percpu, &key, &value,BPF_ANY ); +return 0 ; +} + +struct { + __uint(type, BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE); + __type(key, struct bpf_cgroup_storage_key); + __type(value, u32); +} percpu_cgroup_storage SEC(".maps"); + +SEC("cgroup_skb/egress") +int cgroup_egress (struct __sk_buff *skb) +{ +u32 *val =bpf_get_local_storage (&percpu_cgroup_storage,0 ); + + *val =1 ; +return 1 ; +} + char _license[] SEC("license") = "GPL";
Signature · 1 line
2.51.0
Re: [PATCH bpf-next v9 4/7] bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_hash and lru_percpu_hash maps
In reply to Leon HwangQuoted 70 lines · Leon Hwang
Introduce
BPF_F_ALL_CPUSflag support forpercpu_hashandlru_percpu_hashmaps to allow updating values for all CPUs with a single value for bothupdate_elemandupdate_batchAPIs.Introduce
BPF_F_CPUflag support forpercpu_hashandlru_percpu_hashmaps to allow:* update value for specified CPU for both
update_elemandupdate_batchAPIs.
* lookup value for specified CPU for bothlookup_elemandlookup_batchAPIs.The
BPF_F_CPUflag is passed via:*
map_flagsalong with embedded cpu info.
*elem_flagsalong with embedded cpu info.Signed-off-by Leon Hwang <[email protected]>--- include/linux/bpf.h | 4 ++- kernel/bpf/hashtab.c | 82 +++++++++++++++++++++++++++++--------------- kernel/bpf/syscall.c | 2 +- 3 files changed, 59 insertions(+), 29 deletions(-)include/linux/bpf.h +3
-1@@ -2717,7 +2717,7 @@ int map_set_for_each_callback_args(struct bpf_verifier_env *env,struct bpf_func_state *caller, struct bpf_func_state *callee); -int bpf_percpu_hash_copy (struct bpf_map *map,void *key,void *value); +int bpf_percpu_hash_copy (struct bpf_map *map,void *key,void *value,u64 flags);int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value,u64 flags);int bpf_percpu_hash_update (struct bpf_map *map,void *key,void *value,u64 flags);@@ -3772,6 +3772,8 @@ static inline bool bpf_map_supports_cpu_flags(enum bpf_map_type map_type) {switch (map_type) {case BPF_MAP_TYPE_PERCPU_ARRAY : +case BPF_MAP_TYPE_PERCPU_HASH : +case BPF_MAP_TYPE_LRU_PERCPU_HASH :return true;default :return false;kernel/bpf/hashtab.c +7
-3@@ -945,24 +945,32 @@ static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l) }static void pcpu_copy_value (struct bpf_htab *htab,void __percpu *pptr, -void *value,bool onallcpus) +void *value,bool onallcpus,u64 map_flags) {if (!onallcpus) {/* copy true value_size bytes */ copy_map_value (&htab->map,this_cpu_ptr (pptr), value); }else { -u32 size =round_up (htab->map.value_size,8 ); -int off =0 , cpu; +u32 size = (map_flags & (BPF_F_CPU |BPF_F_ALL_CPUS )) ? + htab->map.value_size :round_up (htab->map.value_size,8 ); +void *ptr; +int cpu; + +if (map_flags &BPF_F_CPU ) {
kernel/bpf/hashtab.c static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l) · line 967
Quoted 10 lines
kernel/bpf/hashtab.c +6
-2+ cpu = map_flags >>32 ; +memcpy (per_cpu_ptr (pptr, cpu), value, size); +return ; + }for_each_possible_cpu (cpu) { -copy_map_value_long (&htab->map,per_cpu_ptr (pptr, cpu), value + off); - off += size; + ptr = (map_flags &BPF_F_ALL_CPUS ) ? value : value + size * cpu; +memcpy (per_cpu_ptr (pptr, cpu), ptr, size);
ok, so you fixed the value_size problem and at the same time introduced blind memcpy() problem?.. Per-CPU maps are allowed to have some special fields (see BPF_REFCOUNT and BPF_KPTR_* checks in map_check_btf()), which have to be handled specially inside copy_map_value[_long](), we cannot just memcpy() blindly
all the other places use copy_map_value[_long](), why did you decide to switch to memcpy here?
pw-bot: cr
Quoted 4 lines
kernel/bpf/hashtab.c +0
-0} } }
[...]
Re: [PATCH bpf-next v9 5/7] bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_cgroup_storage maps
In reply to Leon HwangQuoted 69 lines · Leon Hwang
Introduce
BPF_F_ALL_CPUSflag support forpercpu_cgroup_storagemaps to allow updating values for all CPUs with a single value forupdate_elemAPI.Introduce
BPF_F_CPUflag support forpercpu_cgroup_storagemaps to allow:* update value for specified CPU for
update_elemAPI.
* lookup value for specified CPU forlookup_elemAPI.The
BPF_F_CPUflag is passed viamap_flagsalong with embedded cpu info.Signed-off-by Leon Hwang <[email protected]>--- include/linux/bpf-cgroup.h | 4 ++-- include/linux/bpf.h | 1 + kernel/bpf/local_storage.c | 27 ++++++++++++++++++++------- kernel/bpf/syscall.c | 2 +- 4 files changed, 24 insertions(+), 10 deletions(-)include/linux/bpf-cgroup.h +2
-2@@ -172,7 +172,7 @@ void bpf_cgroup_storage_link(struct bpf_cgroup_storage *storage,void bpf_cgroup_storage_unlink (struct bpf_cgroup_storage *storage);int bpf_cgroup_storage_assign (struct bpf_prog_aux *aux,struct bpf_map *map); -int bpf_percpu_cgroup_storage_copy (struct bpf_map *map,void *key,void *value); +int bpf_percpu_cgroup_storage_copy (struct bpf_map *map,void *key,void *value,u64 flags);int bpf_percpu_cgroup_storage_update (struct bpf_map *map,void *key,void *value,u64 flags);@@ -467,7 +467,7 @@ static inline struct bpf_cgroup_storage *bpf_cgroup_storage_alloc(static inline void bpf_cgroup_storage_free (struct bpf_cgroup_storage *storage) {}static inline int bpf_percpu_cgroup_storage_copy (struct bpf_map *map,void *key, -void *value) { +void *value,u64 flags) {return 0 ; } static inline int bpf_percpu_cgroup_storage_update(struct bpf_map *map,include/linux/bpf.h +1
-0@@ -3774,6 +3774,7 @@ static inline bool bpf_map_supports_cpu_flags(enum bpf_map_type map_type)case BPF_MAP_TYPE_PERCPU_ARRAY :case BPF_MAP_TYPE_PERCPU_HASH :case BPF_MAP_TYPE_LRU_PERCPU_HASH : +case BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE :return true;default :return false;kernel/bpf/local_storage.c +1
-1@@ -180,7 +180,7 @@ static long cgroup_storage_update_elem(struct bpf_map *map, void *key, }int bpf_percpu_cgroup_storage_copy (struct bpf_map *_map,void *key, -void *value) +void *value,u64 map_flags) {struct bpf_cgroup_storage_map *map =map_to_storage (_map);struct bpf_cgroup_storage *storage;
kernel/bpf/local_storage.c int bpf_percpu_cgroup_storage_copy(struct bpf_map *_map, void *key, · line 203
Quoted 6 lines
kernel/bpf/local_storage.c +3
-0@@ -198,12 +198,18 @@ int bpf_percpu_cgroup_storage_copy(struct bpf_map *_map, void *key,* access 'value_size' of them, so copying rounded areas * will not leak any kernel data */ +if (map_flags &BPF_F_CPU ) { + cpu = map_flags >>32 ; +memcpy (value,per_cpu_ptr (storage->percpu_buf, cpu), _map->value_size);
this is so far ok, because we don't seem to allow special fields for PERCPU_CGROUP_STORAGE, but it's best to switch this one to copy_map_value()
kernel/bpf/local_storage.c int bpf_percpu_cgroup_storage_copy(struct bpf_map *_map, void *key, · line 209
Quoted 6 lines
kernel/bpf/local_storage.c +2
-0+ goto unlock; + } size =round_up (_map->value_size,8 );for_each_possible_cpu (cpu) {bpf_long_memcpy (value + off,per_cpu_ptr (storage->percpu_buf, cpu), size);
and let's switch this to copy_map_value_long() to future-proof this:
copy_map_value[_long]() should work correctly with any type of map and will take care of all existing and future special fields
(but maybe have it as a separate patch with just that change to make it obvious)
Quoted 59 lines
kernel/bpf/local_storage.c +14
-6off += size; } +unlock:rcu_read_unlock ();return 0 ; }@@ -213,10 +219,11 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map *_map, void *key, {struct bpf_cgroup_storage_map *map =map_to_storage (_map);struct bpf_cgroup_storage *storage; -int cpu, off =0 ; +void *ptr;u32 size; +int cpu; -if (map_flags !=BPF_ANY && map_flags !=BPF_EXIST ) +if ((u32 )map_flags & ~(BPF_ANY |BPF_EXIST |BPF_F_CPU |BPF_F_ALL_CPUS ))return -EINVAL ;rcu_read_lock ();@@ -232,12 +239,18 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map *_map, void *key,* returned or zeros which were zero-filled by percpu_alloc, * so no kernel data leaks possible */ - size =round_up (_map->value_size,8 ); + size = (map_flags & (BPF_F_CPU |BPF_F_ALL_CPUS )) ? _map->value_size : +round_up (_map->value_size,8 ); +if (map_flags &BPF_F_CPU ) { + cpu = map_flags >>32 ; +memcpy (per_cpu_ptr (storage->percpu_buf, cpu), value, size); + goto unlock; + }for_each_possible_cpu (cpu) { -bpf_long_memcpy (per_cpu_ptr (storage->percpu_buf, cpu), - value + off, size); - off += size; + ptr = (map_flags &BPF_F_ALL_CPUS ) ? value : value + size * cpu; +memcpy (per_cpu_ptr (storage->percpu_buf, cpu), ptr, size); } +unlock:rcu_read_unlock ();return 0 ; }kernel/bpf/syscall.c +1
-2@@ -320,7 +320,7 @@ static int bpf_map_copy_value(struct bpf_map *map, void *key, void *value, }else if (map->map_type ==BPF_MAP_TYPE_PERCPU_ARRAY ) { err =bpf_percpu_array_copy (map, key, value, flags); }else if (map->map_type ==BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE ) { - err =bpf_percpu_cgroup_storage_copy (map, key, value); + err =bpf_percpu_cgroup_storage_copy (map, key, value, flags); }else if (map->map_type ==BPF_MAP_TYPE_STACK_TRACE ) { err =bpf_stackmap_extract (map, key, value, false); }else if (IS_FD_ARRAY (map) ||IS_FD_PROG_ARRAY (map)) { --2.51.0
Re: [PATCH bpf-next v9 4/7] bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_hash and lru_percpu_hash maps
In reply to Andrii NakryikoQuoted 21 lines · Andrii Nakryiko
Quoted 20 lines · Leon Hwang
Introduce
BPF_F_ALL_CPUSflag support forpercpu_hashandlru_percpu_hashmaps to allow updating values for all CPUs with a single value for bothupdate_elemandupdate_batchAPIs.Introduce
BPF_F_CPUflag support forpercpu_hashandlru_percpu_hashmaps to allow:* update value for specified CPU for both
update_elemandupdate_batchAPIs.
* lookup value for specified CPU for bothlookup_elemandlookup_batchAPIs.The
BPF_F_CPUflag is passed via:*
map_flagsalong with embedded cpu info.
*elem_flagsalong with embedded cpu info.Signed-off-by Leon Hwang <[email protected]>---
[...]
Quoted 7 lines
kernel/bpf/hashtab.c static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l) · line 967
Quoted 6 lines
for_each_possible_cpu(cpu) {kernel/bpf/hashtab.c +2
-2-copy_map_value_long (&htab->map,per_cpu_ptr (pptr, cpu), value + off); - off += size; + ptr = (map_flags &BPF_F_ALL_CPUS ) ? value : value + size * cpu; +memcpy (per_cpu_ptr (pptr, cpu), ptr, size);
Quoted 9 lines
ok, so you fixed the
value_sizeproblem and at the same time introduced blindmemcpy()problem?.. Per-CPU maps are allowed to have some special fields (seeBPF_REFCOUNTand BPF_KPTR_* checks inmap_check_btf()), which have to be handled specially inside copy_map_value[_long](), we cannot justmemcpy()blindlyall the other places use copy_map_value[_long](), why did you decide to switch to memcpy here?
You’re right — using memcpy() here is incorrect. I should be using copy_map_value() instead.
When comparing this path with bpf_percpu_array_update(), I noticed that bpf_obj_free_fields() is missing here. That was why I initially switched to memcpy().
To clarify:
1. If those special fields (like BPF_REFCOUNT or BPF_KPTR_*) are *not* supported here, memcpy() behaves the same as copy_map_value().
static inline void bpf_obj_memcpy(struct btf_record *rec,
void *dst, void *src, u32 size, bool long_memcpy)
{
u32 curr_off = 0;
int i;
if (IS_ERR_OR_NULL(rec)) {
if (long_memcpy)
bpf_long_memcpy(dst, src, round_up(size, 8));
else
memcpy(dst, src, size);
return;
}
...
}static inline void copy_map_value(struct bpf_map *map, void *dst, void *src)
{
bpf_obj_memcpy(map->record, dst, src, map->value_size, false);
}2. However, if those special fields *are* supported here, then missing bpf_obj_free_fields() seems like a real issue.
In that case, I’d like to send a separate patch set to add bpf_obj_free_fields() properly.
Does that sound reasonable?
Thanks,
Leon
[...]
Re: [PATCH bpf-next v9 5/7] bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_cgroup_storage maps
In reply to Andrii NakryikoQuoted 16 lines · Andrii Nakryiko
Quoted 15 lines · Leon Hwang
Introduce
BPF_F_ALL_CPUSflag support forpercpu_cgroup_storagemaps to allow updating values for all CPUs with a single value forupdate_elemAPI.Introduce
BPF_F_CPUflag support forpercpu_cgroup_storagemaps to allow:* update value for specified CPU for
update_elemAPI.
* lookup value for specified CPU forlookup_elemAPI.The
BPF_F_CPUflag is passed viamap_flagsalong with embedded cpu info.Signed-off-by Leon Hwang <[email protected]>---
[...]
Quoted 9 lines
Quoted 9 lines
int bpf_percpu_cgroup_storage_copy(struct
bpf_map*_map, void *key,kernel/bpf/local_storage.c +1
-1- void *value) + void *value, u64 map_flags) {struct bpf_cgroup_storage_map *map =map_to_storage (_map);struct bpf_cgroup_storage *storage;@@ -198,12 +198,18 @@ int bpf_percpu_cgroup_storage_copy(struct bpf_map *_map, void *key,* access 'value_size' of them, so copying rounded areas * will not leak any kernel data
Quoted 9 lines
kernel/bpf/local_storage.c int bpf_percpu_cgroup_storage_copy(struct bpf_map *_map, void *key, · line 203
Quoted 4 lines
kernel/bpf/local_storage.c +3
-0*/ +if (map_flags &BPF_F_CPU ) { + cpu = map_flags >>32 ; +memcpy (value,per_cpu_ptr (storage->percpu_buf, cpu), _map->value_size);this is so far ok, because we don't seem to allow special fields for
PERCPU_CGROUP_STORAGE, but it's best to switch this one tocopy_map_value()
Agreed to use copy_map_value() here.
Quoted 7 lines
kernel/bpf/local_storage.c int bpf_percpu_cgroup_storage_copy(struct bpf_map *_map, void *key, · line 209
Quoted 6 lines
kernel/bpf/local_storage.c +2
-0+ goto unlock; + } size =round_up (_map->value_size,8 );for_each_possible_cpu (cpu) {bpf_long_memcpy (value + off,per_cpu_ptr (storage->percpu_buf, cpu), size);
Quoted 6 lines
and let's switch this to
copy_map_value_long()to future-proof this:
copy_map_value[_long]() should work correctly with any type of map and will take care of all existing and future special fields(but maybe have it as a separate patch with just that change to make it obvious)
Ack.
Quoted 30 lines
Quoted 30 lines
kernel/bpf/local_storage.c +9
-3off += size; } +unlock:rcu_read_unlock ();return 0 ; }@@ -213,10 +219,11 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map *_map, void *key, {struct bpf_cgroup_storage_map *map =map_to_storage (_map);struct bpf_cgroup_storage *storage; -int cpu, off =0 ; +void *ptr;u32 size; +int cpu; -if (map_flags !=BPF_ANY && map_flags !=BPF_EXIST ) +if ((u32 )map_flags & ~(BPF_ANY |BPF_EXIST |BPF_F_CPU |BPF_F_ALL_CPUS ))return -EINVAL ;rcu_read_lock ();@@ -232,12 +239,18 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map *_map, void *key,* returned or zeros which were zero-filled by percpu_alloc, * so no kernel data leaks possible */ - size =round_up (_map->value_size,8 ); + size = (map_flags & (BPF_F_CPU |BPF_F_ALL_CPUS )) ? _map->value_size : +round_up (_map->value_size,8 ); +if (map_flags &BPF_F_CPU ) { + cpu = map_flags >>32 ; +memcpy (per_cpu_ptr (storage->percpu_buf, cpu), value, size);
Quoted 8 lines
kernel/bpf/local_storage.c int bpf_percpu_cgroup_storage_update(struct bpf_map *_map, void *key, · line 251
Quoted 8 lines
kernel/bpf/local_storage.c +4
-3+ goto unlock; + }for_each_possible_cpu (cpu) { -bpf_long_memcpy (per_cpu_ptr (storage->percpu_buf, cpu), - value + off, size); - off += size; + ptr = (map_flags &BPF_F_ALL_CPUS ) ? value : value + size * cpu; +memcpy (per_cpu_ptr (storage->percpu_buf, cpu), ptr, size);
Switch memcpy() to copy_map_value[_long](), too.
Thanks,
Leon
[...]
Re: [PATCH bpf-next v9 4/7] bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_hash and lru_percpu_hash maps
In reply to Leon HwangQuoted 81 lines · Leon Hwang
Quoted 21 lines · Andrii Nakryiko
Quoted 20 lines · Leon Hwang
Introduce
BPF_F_ALL_CPUSflag support forpercpu_hashandlru_percpu_hashmaps to allow updating values for all CPUs with a single value for bothupdate_elemandupdate_batchAPIs.Introduce
BPF_F_CPUflag support forpercpu_hashandlru_percpu_hashmaps to allow:* update value for specified CPU for both
update_elemandupdate_batchAPIs.
* lookup value for specified CPU for bothlookup_elemandlookup_batchAPIs.The
BPF_F_CPUflag is passed via:*
map_flagsalong with embedded cpu info.
*elem_flagsalong with embedded cpu info.Signed-off-by Leon Hwang <[email protected]>---
[...]
Quoted 7 lines
kernel/bpf/hashtab.c static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l) · line 967
Quoted 6 lines
for_each_possible_cpu(cpu) {kernel/bpf/hashtab.c +2
-2-copy_map_value_long (&htab->map,per_cpu_ptr (pptr, cpu), value + off); - off += size; + ptr = (map_flags &BPF_F_ALL_CPUS ) ? value : value + size * cpu; +memcpy (per_cpu_ptr (pptr, cpu), ptr, size);Quoted 9 lines
ok, so you fixed the
value_sizeproblem and at the same time introduced blindmemcpy()problem?.. Per-CPU maps are allowed to have some special fields (seeBPF_REFCOUNTand BPF_KPTR_* checks inmap_check_btf()), which have to be handled specially inside copy_map_value[_long](), we cannot justmemcpy()blindlyall the other places use copy_map_value[_long](), why did you decide to switch to memcpy here?
You’re right — using
memcpy()here is incorrect. I should be usingcopy_map_value()instead.When comparing this path with
bpf_percpu_array_update(), I noticed thatbpf_obj_free_fields()is missing here. That was why I initially switched tomemcpy().To clarify:
1. If those special fields (like
BPF_REFCOUNTor BPF_KPTR_*) are *not* supported here,memcpy()behaves the same ascopy_map_value().static inline void bpf_obj_memcpy(struct
btf_record*rec,
void *dst, void *src, u32 size, boollong_memcpy){ u32 curr_off = 0; int i; if (IS_ERR_OR_NULL(rec)) { if (long_memcpy) bpf_long_memcpy(dst, src, round_up(size, 8)); else memcpy(dst, src, size); return; } ... }static inline void
copy_map_value(struct bpf_map *map, void *dst, void *src){ bpf_obj_memcpy(map->record, dst, src, map->value_size, false); }
Quoted 5 lines
2. However, if those special fields *are* supported here, then missing
bpf_obj_free_fields()seems like a real issue.
In that case, I’d like to send a separate patch set to addbpf_obj_free_fields()properly.
Does that sound reasonable?
Add a test and we'll know for sure? But it looks like yes, you can use *some* special fields for per-CPU maps.
Quoted 5 lines
Thanks,
Leon[...]