bpf: Introduce BPF_F_CPU and BPF_F_ALL_CPUS flags for percpu maps
Subsystems (5)
v12 cover bpf: Introduce BPF_F_CPU and BPF_F_ALL_CPUS flags for percpu maps
This patch set introduces the BPF_F_CPU and BPF_F_ALL_CPUS flags for percpu maps, as the requirement of BPF_F_ALL_CPUS flag for percpu_array maps was discussed in the thread of
"[PATCH bpf-next v3 0/4] bpf: Introduce global percpu data"[1].
The goal of BPF_F_ALL_CPUS flag is to reduce data caching overhead in light skeletons by allowing a single value to be reused to update values across all CPUs. This avoids the M:N problem where M cached values are used to update a map on N CPUs kernel.
The BPF_F_CPU flag is accompanied by *flags*-embedded cpu info, which specifies the target CPU for the operation:
* For lookup operations: the flag field alongside cpu info enable querying a value on the specified CPU.
* For update operations: the flag field alongside cpu info enable updating value for specified CPU.
Links:
[1] https://lore.kernel.org/bpf/[email protected]/
Changes:
v11 -> v12:
* Dropped the v11 changes.
* Stabilized the lru_percpu_hash map test by keeping an extra spare entry, which can be used temporarily during updates to avoid unintended LRU evictions.
v10 -> v11:
* Support the combination of BPF_EXIST and BPF_F_CPU/BPF_F_ALL_CPUS for update operations.
* Fix unstable lru_percpu_hash map test using the combination of BPF_EXIST and BPF_F_CPU/BPF_F_ALL_CPUS to avoid LRU eviction (reported by Alexei).
v9 -> v10:
* Add tests to verify array and hash maps do not support BPF_F_CPU and BPF_F_ALL_CPUS flags.
* Address comment from Andrii:
* Copy map value using copy_map_value_long for percpu_cgroup_storage maps in a separate patch.
v8 -> v9:
* Change value type from u64 to u32 in selftests.
* Address comments from Andrii:
* Keep value_size unaligned and update everywhere for consistency when cpu flags are specified.
* Update value by getting pointer for percpu hash and percpu cgroup_storage maps.
v7 -> v8:
* Address comments from Andrii:
* Check BPF_F_LOCK when update percpu_array, percpu_hash and lru_percpu_hash maps.
* Refactor flags check in __htab_map_lookup_and_delete_batch().
* Keep value_size unaligned and copy value using copy_map_value() in __htab_map_lookup_and_delete_batch() when BPF_F_CPU is specified.
* Update warn message in libbpf's validate_map_op().
* Update comment of libbpf's bpf_map__lookup_elem().
v6 -> v7:
* Get correct value size for percpu_hash and lru_percpu_hash in update_batch API.
* Set 'count' as 'max_entries' in test cases for lookup_batch API.
* Address comment from Alexei:
* Move cpu flags check into bpf_map_check_op_flags().
v5 -> v6:
* Move bpf_map_check_op_flags() from 'bpf.h' to 'syscall.c'.
* Address comments from Alexei:
* Drop the refactoring code of data copying logic for percpu maps.
* Drop bpf_map_check_op_flags() wrappers.
v4 -> v5:
* Address comments from Andrii:
* Refactor data copying logic for all percpu maps.
* Drop this_cpu_ptr() micro-optimization.
* Drop cpu check in libbpf's validate_map_op().
* Enhance bpf_map_check_op_flags() using *allowed flags* instead of
'extra_flags_mask'.
v3 -> v4:
* Address comments from Andrii:
* Remove unnecessary map_type check in bpf_map_value_size().
* Reduce code churn.
* Remove unnecessary do_delete check in
__htab_map_lookup_and_delete_batch().
* Introduce bpf_percpu_copy_to_user() and bpf_percpu_copy_from_user().
* Rename check_map_flags() to bpf_map_check_op_flags() with
extra_flags_mask.
* Add human-readable pr_warn() explanations in validate_map_op().
* Use flags in bpf_map__delete_elem() and
bpf_map__lookup_and_delete_elem(). * Drop "for alignment reasons".
v3 link: https://lore.kernel.org/bpf/[email protected]/
v2 -> v3:
* Address comments from Alexei:
* Use BPF_F_ALL_CPUS instead of BPF_ALL_CPUS magic.
* Introduce these two cpu flags for all percpu maps.
* Address comments from Jiri:
* Reduce some unnecessary u32 cast.
* Refactor more generic map flags check function.
* A code style issue.
v2 link: https://lore.kernel.org/bpf/[email protected]/
v1 -> v2:
* Address comments from Andrii:
* Embed cpu info as high 32 bits of *flags* totally.
* Use ERANGE instead of E2BIG.
* Few format issues.
Leon Hwang (7):
bpf: Introduce BPF_F_CPU and BPF_F_ALL_CPUS flags
bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_array
maps
bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_hash
and lru_percpu_hash maps
bpf: Copy map value using copy_map_value_long for
percpu_cgroup_storage maps
bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for
percpu_cgroup_storage maps
libbpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu maps selftests/bpf: Add cases to test BPF_F_CPU and BPF_F_ALL_CPUS flags
include/linux/bpf-cgroup.h | 4 +- include/linux/bpf.h | 35 +- include/uapi/linux/bpf.h | 2 + kernel/bpf/arraymap.c | 29 +- kernel/bpf/hashtab.c | 94 +++-- kernel/bpf/local_storage.c | 27 +- kernel/bpf/syscall.c | 37 +- tools/include/uapi/linux/bpf.h | 2 + tools/lib/bpf/bpf.h | 8 + tools/lib/bpf/libbpf.c | 26 +- tools/lib/bpf/libbpf.h | 21 +- .../selftests/bpf/prog_tests/percpu_alloc.c | 328 ++++++++++++++++++ .../selftests/bpf/progs/percpu_alloc_array.c | 32 ++ 13 files changed, 560 insertions(+), 85 deletions(-)
--
2.51.2
PATCH v12 1/7 bpf: Introduce BPF_F_CPU and BPF_F_ALL_CPUS flags
Introduce BPF_F_CPU and BPF_F_ALL_CPUS flags and check them for following APIs:
* 'map_lookup_elem()'
* 'map_update_elem()'
* 'generic_map_lookup_batch()'
* 'generic_map_update_batch()'
And, get the correct value size for these APIs.
--- include/linux/bpf.h | 23 ++++++++++++++++++++++- include/uapi/linux/bpf.h | 2 ++ kernel/bpf/syscall.c | 31 +++++++++++++++++-------------- tools/include/uapi/linux/bpf.h | 2 ++ 4 files changed, 43 insertions(+), 15 deletions(-)
include/linux/bpf.h +22 -1
@@ -3829,14 +3829,35 @@ bpf_prog_update_insn_ptrs(struct bpf_prog *prog, u32 *offsets, void *image) }#endif +static inline bool bpf_map_supports_cpu_flags (enum bpf_map_type map_type) +{ +return false; +} +static inline int bpf_map_check_op_flags (struct bpf_map *map,u64 flags,u64 allowed_flags) { -if (flags & ~allowed_flags) +u32 cpu; + +if ((u32 )flags & ~allowed_flags)return -EINVAL ;if ((flags &BPF_F_LOCK ) && !btf_record_has_field (map->record,BPF_SPIN_LOCK ))return -EINVAL ; +if (!(flags &BPF_F_CPU ) && flags >>32 ) +return -EINVAL ; + +if (flags & (BPF_F_CPU |BPF_F_ALL_CPUS )) { +if (!bpf_map_supports_cpu_flags (map->map_type)) +return -EINVAL ; +if ((flags &BPF_F_CPU ) && (flags &BPF_F_ALL_CPUS )) +return -EINVAL ; + + cpu = flags >>32 ; +if ((flags &BPF_F_CPU ) && cpu >=num_possible_cpus ()) +return -ERANGE ; + } +return 0 ; }
include/uapi/linux/bpf.h +2 -0
@@ -1373,6 +1373,8 @@ enum {BPF_NOEXIST =1 ,/* create new element if it didn't exist */ BPF_EXIST =2 ,/* update existing element */ BPF_F_LOCK =4 ,/* spin_lock-ed map_lookup/map_update */ +BPF_F_CPU =8 ,/* cpu flag for percpu maps, upper 32-bit of flags is a cpu number */ + BPF_F_ALL_CPUS = 16, /* update value across all CPUs for percpu maps */ };/* flags for BPF_MAP_CREATE command */
kernel/bpf/syscall.c +17 -14
@@ -133,12 +133,14 @@ bool bpf_map_write_active(const struct bpf_map *map)return atomic64_read (&map->writecnt) !=0 ; } -static u32 bpf_map_value_size (const struct bpf_map *map) -{ -if (map->map_type ==BPF_MAP_TYPE_PERCPU_HASH || - map->map_type ==BPF_MAP_TYPE_LRU_PERCPU_HASH || - map->map_type ==BPF_MAP_TYPE_PERCPU_ARRAY || - map->map_type ==BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE ) +static u32 bpf_map_value_size (const struct bpf_map *map,u64 flags) +{ +if (flags & (BPF_F_CPU |BPF_F_ALL_CPUS )) +return map->value_size; +else if (map->map_type ==BPF_MAP_TYPE_PERCPU_HASH || + map->map_type ==BPF_MAP_TYPE_LRU_PERCPU_HASH || + map->map_type ==BPF_MAP_TYPE_PERCPU_ARRAY || + map->map_type ==BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE )return round_up (map->value_size,8 ) *num_possible_cpus ();else if (IS_FD_MAP (map))return sizeof (u32);
@@ -1732,7 +1734,7 @@ static int map_lookup_elem(union bpf_attr *attr)if (!(map_get_sys_perms (map, f) &FMODE_CAN_READ ))return -EPERM ; - err =bpf_map_check_op_flags (map, attr->flags,BPF_F_LOCK ); + err =bpf_map_check_op_flags (map, attr->flags,BPF_F_LOCK |BPF_F_CPU );if (err)return err;
@@ -1740,7 +1742,7 @@ static int map_lookup_elem(union bpf_attr *attr)if (IS_ERR (key))return PTR_ERR (key); - value_size =bpf_map_value_size (map); + value_size =bpf_map_value_size (map, attr->flags); err = -ENOMEM ; value =kvmalloc (value_size,GFP_USER | __GFP_NOWARN);
@@ -1807,7 +1809,7 @@ static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr) goto err_put; } - value_size =bpf_map_value_size (map); + value_size =bpf_map_value_size (map, attr->flags); value =kvmemdup_bpfptr (uvalue, value_size);if (IS_ERR (value)) { err =PTR_ERR (value);
@@ -2003,11 +2005,12 @@ int generic_map_update_batch(struct bpf_map *map, struct file *map_file,void *key, *value;int err =0 ; - err =bpf_map_check_op_flags (map, attr->batch.elem_flags,BPF_F_LOCK ); + err =bpf_map_check_op_flags (map, attr->batch.elem_flags, +BPF_F_LOCK |BPF_F_CPU |BPF_F_ALL_CPUS );if (err)return err; - value_size =bpf_map_value_size (map); + value_size =bpf_map_value_size (map, attr->batch.elem_flags); max_count = attr->batch.count;if (!max_count)
@@ -2062,11 +2065,11 @@ int generic_map_lookup_batch(struct bpf_map *map,u32 value_size, cp, max_count;int err; - err =bpf_map_check_op_flags (map, attr->batch.elem_flags,BPF_F_LOCK ); + err =bpf_map_check_op_flags (map, attr->batch.elem_flags,BPF_F_LOCK |BPF_F_CPU );if (err)return err; - value_size =bpf_map_value_size (map); + value_size =bpf_map_value_size (map, attr->batch.elem_flags); max_count = attr->batch.count;if (!max_count)
@@ -2188,7 +2191,7 @@ static int map_lookup_and_delete_elem(union bpf_attr *attr) goto err_put; } - value_size =bpf_map_value_size (map); + value_size =bpf_map_value_size (map,0 ); err = -ENOMEM ; value =kvmalloc (value_size,GFP_USER | __GFP_NOWARN);
tools/include/uapi/linux/bpf.h +2 -0
@@ -1373,6 +1373,8 @@ enum {BPF_NOEXIST =1 ,/* create new element if it didn't exist */ BPF_EXIST =2 ,/* update existing element */ BPF_F_LOCK =4 ,/* spin_lock-ed map_lookup/map_update */ +BPF_F_CPU =8 ,/* cpu flag for percpu maps, upper 32-bit of flags is a cpu number */ + BPF_F_ALL_CPUS = 16, /* update value across all CPUs for percpu maps */ };/* flags for BPF_MAP_CREATE command */
Signature · 1 line
2.51.2
PATCH v12 2/7 bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_array maps
In reply to Leon HwangIntroduce support for the BPF_F_ALL_CPUS flag in percpu_array maps to allow updating values for all CPUs with a single value for both update_elem and update_batch APIs.
Introduce support for the BPF_F_CPU flag in percpu_array maps to allow:
* update value for specified CPU for both update_elem and update_batch APIs.
* lookup value for specified CPU for both lookup_elem and lookup_batch APIs.
The BPF_F_CPU flag is passed via:
* map_flags of lookup_elem and update_elem APIs along with embedded cpu info.
* elem_flags of lookup_batch and update_batch APIs along with embedded cpu info.
--- include/linux/bpf.h | 9 +++++++-- kernel/bpf/arraymap.c | 29 +++++++++++++++++++++++------ kernel/bpf/syscall.c | 2 +- 3 files changed, 31 insertions(+), 9 deletions(-)
include/linux/bpf.h +7 -2
@@ -2762,7 +2762,7 @@ int map_set_for_each_callback_args(struct bpf_verifier_env *env, struct bpf_func_state *callee);int bpf_percpu_hash_copy (struct bpf_map *map,void *key,void *value); -int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value); +int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value,u64 flags);int bpf_percpu_hash_update (struct bpf_map *map,void *key,void *value,u64 flags); int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value,
@@ -3831,7 +3831,12 @@ bpf_prog_update_insn_ptrs(struct bpf_prog *prog, u32 *offsets, void *image)static inline bool bpf_map_supports_cpu_flags (enum bpf_map_type map_type) { -return false; +switch (map_type) { +case BPF_MAP_TYPE_PERCPU_ARRAY : +return true; +default : +return false; + } }static inline int bpf_map_check_op_flags (struct bpf_map *map,u64 flags,u64 allowed_flags)
kernel/bpf/arraymap.c +23 -6
@@ -307,7 +307,7 @@ static void *percpu_array_map_lookup_percpu_elem(struct bpf_map *map, void *key,return per_cpu_ptr (array->pptrs[index & array->index_mask], cpu); } -int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value) +int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value,u64 map_flags) { struct bpf_array *array = container_of(map, struct bpf_array, map);u32 index = *(u32 *)key;
@@ -325,11 +325,18 @@ int bpf_percpu_array_copy(struct bpf_map *map, void *key, void *value) size = array->elem_size;rcu_read_lock (); pptr = array->pptrs[index & array->index_mask]; +if (map_flags &BPF_F_CPU ) { + cpu = map_flags >>32 ; +copy_map_value (map, value,per_cpu_ptr (pptr, cpu)); +check_and_init_map_value (map, value); + goto unlock; + }for_each_possible_cpu (cpu) {copy_map_value_long (map, value + off,per_cpu_ptr (pptr, cpu));check_and_init_map_value (map, value + off); off += size; } +unlock:rcu_read_unlock ();return 0 ; }
@@ -398,10 +405,11 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value, struct bpf_array *array = container_of(map, struct bpf_array, map);u32 index = *(u32 *)key;void __percpu *pptr; -int cpu, off =0 ; +void *ptr, *val;u32 size; +int cpu; -if (unlikely (map_flags >BPF_EXIST )) +if (unlikely ((map_flags &BPF_F_LOCK ) || (u32 )map_flags >BPF_F_ALL_CPUS ))/* unknown flags */ return -EINVAL ;
@@ -422,11 +430,20 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value, size = array->elem_size;rcu_read_lock (); pptr = array->pptrs[index & array->index_mask]; +if (map_flags &BPF_F_CPU ) { + cpu = map_flags >>32 ; + ptr =per_cpu_ptr (pptr, cpu); +copy_map_value (map, ptr, value); +bpf_obj_free_fields (array->map.record, ptr); + goto unlock; + }for_each_possible_cpu (cpu) { -copy_map_value_long (map,per_cpu_ptr (pptr, cpu), value + off); -bpf_obj_free_fields (array->map.record,per_cpu_ptr (pptr, cpu)); - off += size; + ptr =per_cpu_ptr (pptr, cpu); + val = (map_flags &BPF_F_ALL_CPUS ) ? value : value + size * cpu; +copy_map_value (map, ptr, val); +bpf_obj_free_fields (array->map.record, ptr); } +unlock:rcu_read_unlock ();return 0 ; }
kernel/bpf/syscall.c +1 -1
@@ -318,7 +318,7 @@ static int bpf_map_copy_value(struct bpf_map *map, void *key, void *value, map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH) { err =bpf_percpu_hash_copy (map, key, value); } else if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) { - err =bpf_percpu_array_copy (map, key, value); + err =bpf_percpu_array_copy (map, key, value, flags); } else if (map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE) { err =bpf_percpu_cgroup_storage_copy (map, key, value); } else if (map->map_type == BPF_MAP_TYPE_STACK_TRACE) {
Signature · 1 line
2.51.2
PATCH v12 3/7 bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_hash and lru_percpu_hash maps
In reply to Leon HwangIntroduce BPF_F_ALL_CPUS flag support for percpu_hash and lru_percpu_hash maps to allow updating values for all CPUs with a single value for both update_elem and update_batch APIs.
Introduce BPF_F_CPU flag support for percpu_hash and lru_percpu_hash maps to allow:
* update value for specified CPU for both update_elem and update_batch APIs.
* lookup value for specified CPU for both lookup_elem and lookup_batch APIs.
The BPF_F_CPU flag is passed via:
* map_flags along with embedded cpu info.
* elem_flags along with embedded cpu info.
--- include/linux/bpf.h | 4 +- kernel/bpf/hashtab.c | 94 ++++++++++++++++++++++++++++++-------------- kernel/bpf/syscall.c | 2 +- 3 files changed, 68 insertions(+), 32 deletions(-)
include/linux/bpf.h +3 -1
@@ -2761,7 +2761,7 @@ int map_set_for_each_callback_args(struct bpf_verifier_env *env,struct bpf_func_state *caller, struct bpf_func_state *callee); -int bpf_percpu_hash_copy (struct bpf_map *map,void *key,void *value); +int bpf_percpu_hash_copy (struct bpf_map *map,void *key,void *value,u64 flags);int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value,u64 flags);int bpf_percpu_hash_update (struct bpf_map *map,void *key,void *value,u64 flags);
@@ -3833,6 +3833,8 @@ static inline bool bpf_map_supports_cpu_flags(enum bpf_map_type map_type) {switch (map_type) {case BPF_MAP_TYPE_PERCPU_ARRAY : +case BPF_MAP_TYPE_PERCPU_HASH : +case BPF_MAP_TYPE_LRU_PERCPU_HASH :return true;default :return false;
kernel/bpf/hashtab.c +64 -30
224 lines not on this page · raw message
kernel/bpf/syscall.c +1 -1
@@ -316,7 +316,7 @@ static int bpf_map_copy_value(struct bpf_map *map, void *key, void *value,bpf_disable_instrumentation ();if (map->map_type ==BPF_MAP_TYPE_PERCPU_HASH || map->map_type ==BPF_MAP_TYPE_LRU_PERCPU_HASH ) { - err =bpf_percpu_hash_copy (map, key, value); + err =bpf_percpu_hash_copy (map, key, value, flags); }else if (map->map_type ==BPF_MAP_TYPE_PERCPU_ARRAY ) { err =bpf_percpu_array_copy (map, key, value, flags); }else if (map->map_type ==BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE ) {
Signature · 1 line
2.51.2
PATCH v12 4/7 bpf: Copy map value using copy_map_value_long for percpu_cgroup_storage maps
In reply to Leon HwangCopy map value using 'copy_map_value_long()'. It's to keep consistent style with the way of other percpu maps.
No functional change intended.
--- kernel/bpf/local_storage.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
kernel/bpf/local_storage.c +2 -4
@@ -200,8 +200,7 @@ int bpf_percpu_cgroup_storage_copy(struct bpf_map *_map, void *key,*/ size =round_up (_map->value_size,8 );for_each_possible_cpu (cpu) { -bpf_long_memcpy (value + off, -per_cpu_ptr (storage->percpu_buf, cpu), size); +copy_map_value_long (_map, value + off,per_cpu_ptr (storage->percpu_buf, cpu)); off += size; }rcu_read_unlock ();
@@ -234,8 +233,7 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map *_map, void *key,*/ size =round_up (_map->value_size,8 );for_each_possible_cpu (cpu) { -bpf_long_memcpy (per_cpu_ptr (storage->percpu_buf, cpu), - value + off, size); +copy_map_value_long (_map,per_cpu_ptr (storage->percpu_buf, cpu), value + off); off += size; }rcu_read_unlock ();
Signature · 1 line
2.51.2
PATCH v12 5/7 bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_cgroup_storage maps
In reply to Leon HwangIntroduce BPF_F_ALL_CPUS flag support for percpu_cgroup_storage maps to allow updating values for all CPUs with a single value for update_elem API.
Introduce BPF_F_CPU flag support for percpu_cgroup_storage maps to allow:
* update value for specified CPU for update_elem API.
* lookup value for specified CPU for lookup_elem API.
The BPF_F_CPU flag is passed via map_flags along with embedded cpu info.
--- include/linux/bpf-cgroup.h | 4 ++-- include/linux/bpf.h | 1 + kernel/bpf/local_storage.c | 23 ++++++++++++++++++----- kernel/bpf/syscall.c | 2 +- 4 files changed, 22 insertions(+), 8 deletions(-)
include/linux/bpf-cgroup.h +2 -2
@@ -172,7 +172,7 @@ void bpf_cgroup_storage_link(struct bpf_cgroup_storage *storage,void bpf_cgroup_storage_unlink (struct bpf_cgroup_storage *storage);int bpf_cgroup_storage_assign (struct bpf_prog_aux *aux,struct bpf_map *map); -int bpf_percpu_cgroup_storage_copy (struct bpf_map *map,void *key,void *value); +int bpf_percpu_cgroup_storage_copy (struct bpf_map *map,void *key,void *value,u64 flags);int bpf_percpu_cgroup_storage_update (struct bpf_map *map,void *key,void *value,u64 flags);
@@ -467,7 +467,7 @@ static inline struct bpf_cgroup_storage *bpf_cgroup_storage_alloc(static inline void bpf_cgroup_storage_free (struct bpf_cgroup_storage *storage) {}static inline int bpf_percpu_cgroup_storage_copy (struct bpf_map *map,void *key, -void *value) { +void *value,u64 flags) {return 0 ; } static inline int bpf_percpu_cgroup_storage_update(struct bpf_map *map,
include/linux/bpf.h +1 -0
@@ -3835,6 +3835,7 @@ static inline bool bpf_map_supports_cpu_flags(enum bpf_map_type map_type)case BPF_MAP_TYPE_PERCPU_ARRAY :case BPF_MAP_TYPE_PERCPU_HASH :case BPF_MAP_TYPE_LRU_PERCPU_HASH : +case BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE :return true;default :return false;
kernel/bpf/local_storage.c +18 -5
@@ -180,7 +180,7 @@ static long cgroup_storage_update_elem(struct bpf_map *map, void *key, }int bpf_percpu_cgroup_storage_copy (struct bpf_map *_map,void *key, -void *value) +void *value,u64 map_flags) {struct bpf_cgroup_storage_map *map =map_to_storage (_map);struct bpf_cgroup_storage *storage;
@@ -198,11 +198,17 @@ int bpf_percpu_cgroup_storage_copy(struct bpf_map *_map, void *key,* access 'value_size' of them, so copying rounded areas * will not leak any kernel data */ +if (map_flags &BPF_F_CPU ) { + cpu = map_flags >>32 ; +copy_map_value (_map, value,per_cpu_ptr (storage->percpu_buf, cpu)); + goto unlock; + } size =round_up (_map->value_size,8 );for_each_possible_cpu (cpu) {copy_map_value_long (_map, value + off,per_cpu_ptr (storage->percpu_buf, cpu)); off += size; } +unlock:rcu_read_unlock ();return 0 ; }
@@ -212,10 +218,11 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map *_map, void *key, {struct bpf_cgroup_storage_map *map =map_to_storage (_map);struct bpf_cgroup_storage *storage; -int cpu, off =0 ; +void *val;u32 size; +int cpu; -if (map_flags !=BPF_ANY && map_flags !=BPF_EXIST ) +if ((u32 )map_flags & ~(BPF_ANY |BPF_EXIST |BPF_F_CPU |BPF_F_ALL_CPUS ))return -EINVAL ;rcu_read_lock ();
@@ -231,11 +238,17 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map *_map, void *key,* returned or zeros which were zero-filled by percpu_alloc, * so no kernel data leaks possible */ +if (map_flags &BPF_F_CPU ) { + cpu = map_flags >>32 ; +copy_map_value (_map,per_cpu_ptr (storage->percpu_buf, cpu), value); + goto unlock; + } size =round_up (_map->value_size,8 );for_each_possible_cpu (cpu) { -copy_map_value_long (_map,per_cpu_ptr (storage->percpu_buf, cpu), value + off); - off += size; + val = (map_flags &BPF_F_ALL_CPUS ) ? value : value + size * cpu; +copy_map_value (_map,per_cpu_ptr (storage->percpu_buf, cpu), val); } +unlock:rcu_read_unlock ();return 0 ; }
kernel/bpf/syscall.c +1 -1
@@ -320,7 +320,7 @@ static int bpf_map_copy_value(struct bpf_map *map, void *key, void *value, }else if (map->map_type ==BPF_MAP_TYPE_PERCPU_ARRAY ) { err =bpf_percpu_array_copy (map, key, value, flags); }else if (map->map_type ==BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE ) { - err =bpf_percpu_cgroup_storage_copy (map, key, value); + err =bpf_percpu_cgroup_storage_copy (map, key, value, flags); }else if (map->map_type ==BPF_MAP_TYPE_STACK_TRACE ) { err =bpf_stackmap_extract (map, key, value, false); }else if (IS_FD_ARRAY (map) ||IS_FD_PROG_ARRAY (map)) {
Signature · 1 line
2.51.2
PATCH v12 6/7 libbpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu maps
In reply to Leon HwangAdd libbpf support for the BPF_F_CPU flag for percpu maps by embedding the cpu info into the high 32 bits of:
1. **flags**: bpf_map_lookup_elem_flags(), bpf_map__lookup_elem(), bpf_map_update_elem() and bpf_map__update_elem()
2. **opts->elem_flags**: bpf_map_lookup_batch() and bpf_map_update_batch()
And the flag can be BPF_F_ALL_CPUS, but cannot be
'BPF_F_CPU | BPF_F_ALL_CPUS'.
Behavior:
* If the flag is BPF_F_ALL_CPUS, the update is applied across all CPUs.
* If the flag is BPF_F_CPU, it updates value only to the specified CPU.
* If the flag is BPF_F_CPU, lookup value only from the specified CPU.
* lookup does not support BPF_F_ALL_CPUS.
--- tools/lib/bpf/bpf.h | 8 ++++++++ tools/lib/bpf/libbpf.c | 26 ++++++++++++++++++++------ tools/lib/bpf/libbpf.h | 21 ++++++++------------- 3 files changed, 36 insertions(+), 19 deletions(-)
tools/lib/bpf/bpf.h +8 -0
@@ -289,6 +289,14 @@ LIBBPF_API int bpf_map_lookup_and_delete_batch(int fd, void *in_batch,* Update spin_lock-ed map elements. This must be * specified if the map value contains a spinlock. * +* **BPF_F_CPU** +* As for percpu maps, update value on the specified CPU. And the cpu +* info is embedded into the high 32 bits of **opts->elem_flags**. +* +* **BPF_F_ALL_CPUS** +* As for percpu maps, update value across all CPUs. This flag cannot +* be used with BPF_F_CPU at the same time. +* * @param fd BPF map file descriptor * @param keys pointer to an array of *count* keys * @param values pointer to an array of *count* values
tools/lib/bpf/libbpf.c +20 -6
@@ -10913,7 +10913,7 @@ bpf_object__find_map_fd_by_name(const struct bpf_object *obj, const char *name) }static int validate_map_op (const struct bpf_map *map,size_t key_sz, -size_t value_sz,bool check_value_sz) +size_t value_sz,bool check_value_sz,__u64 flags) {if (!map_is_created (map))/* map is not yet created */ return -ENOENT ;
@@ -10940,6 +10940,20 @@ static int validate_map_op(const struct bpf_map *map, size_t key_sz,int num_cpu =libbpf_num_possible_cpus ();size_t elem_sz =roundup (map->def.value_size,8 ); +if (flags & (BPF_F_CPU |BPF_F_ALL_CPUS )) { +if ((flags &BPF_F_CPU ) && (flags &BPF_F_ALL_CPUS )) { +pr_warn ("map '%s': BPF_F_CPU and BPF_F_ALL_CPUS are mutually exclusive\n" , + map->name); +return -EINVAL ; + } +if (map->def.value_size != value_sz) { +pr_warn ("map '%s': unexpected value size %zu provided for either BPF_F_CPU or BPF_F_ALL_CPUS, expected %u\n" , + map->name, value_sz, map->def.value_size); +return -EINVAL ; + } +break ; + } +if (value_sz != num_cpu * elem_sz) {pr_warn ("map '%s': unexpected value size %zu provided for per-CPU map, expected %d * %zu = %zd\n" , map->name, value_sz, num_cpu, elem_sz, num_cpu * elem_sz);
@@ -10964,7 +10978,7 @@ int bpf_map__lookup_elem(const struct bpf_map *map, {int err; - err =validate_map_op (map, key_sz, value_sz, true); + err =validate_map_op (map, key_sz, value_sz, true, flags);if (err)return libbpf_err (err);
@@ -10977,7 +10991,7 @@ int bpf_map__update_elem(const struct bpf_map *map, {int err; - err =validate_map_op (map, key_sz, value_sz, true); + err =validate_map_op (map, key_sz, value_sz, true, flags);if (err)return libbpf_err (err);
@@ -10989,7 +11003,7 @@ int bpf_map__delete_elem(const struct bpf_map *map, {int err; - err =validate_map_op (map, key_sz,0 , false/* check_value_sz */ ); + err =validate_map_op (map, key_sz,0 , false/* check_value_sz */ , flags);if (err)return libbpf_err (err);
@@ -11002,7 +11016,7 @@ int bpf_map__lookup_and_delete_elem(const struct bpf_map *map, {int err; - err =validate_map_op (map, key_sz, value_sz, true); + err =validate_map_op (map, key_sz, value_sz, true, flags);if (err)return libbpf_err (err);
@@ -11014,7 +11028,7 @@ int bpf_map__get_next_key(const struct bpf_map *map, {int err; - err =validate_map_op (map, key_sz,0 , false/* check_value_sz */ ); + err =validate_map_op (map, key_sz,0 , false/* check_value_sz */ ,0 );if (err)return libbpf_err (err);
tools/lib/bpf/libbpf.h +8 -13
@@ -1200,12 +1200,13 @@ LIBBPF_API struct bpf_map *bpf_map__inner_map(struct bpf_map *map);* @param key_sz size in bytes of key data, needs to match BPF map definition's **key_size** * @param value pointer to memory in which looked up value will be stored * @param value_sz size in byte of value data memory; it has to match BPF map -* definition's **value_size**. For per-CPU BPF maps value size has to be -* a product of BPF map value size and number of possible CPUs in the system -* (could be fetched with **libbpf_num_possible_cpus()**). Note also that for -* per-CPU values value size has to be aligned up to closest 8 bytes for -* alignment reasons, so expected size is: `round_up(value_size, 8) -* * libbpf_num_possible_cpus()`. +* definition's **value_size**. For per-CPU BPF maps, value size can be +* `value_size` if either **BPF_F_CPU** or **BPF_F_ALL_CPUS** is specified +* in **flags**, otherwise a product of BPF map value size and number of +* possible CPUs in the system (could be fetched with +* **libbpf_num_possible_cpus()**). Note also that for per-CPU values value +* size has to be aligned up to closest 8 bytes, so expected size is: +* `round_up(value_size, 8) * libbpf_num_possible_cpus()`. * @param flags extra flags passed to kernel for this operation * @return 0, on success; negative error, otherwise *
@@ -1223,13 +1224,7 @@ LIBBPF_API int bpf_map__lookup_elem(const struct bpf_map *map,* @param key pointer to memory containing bytes of the key * @param key_sz size in bytes of key data, needs to match BPF map definition's **key_size** * @param value pointer to memory containing bytes of the value -* @param value_sz size in byte of value data memory; it has to match BPF map -* definition's **value_size**. For per-CPU BPF maps value size has to be -* a product of BPF map value size and number of possible CPUs in the system -* (could be fetched with **libbpf_num_possible_cpus()**). Note also that for -* per-CPU values value size has to be aligned up to closest 8 bytes for -* alignment reasons, so expected size is: `round_up(value_size, 8) -* * libbpf_num_possible_cpus()`. +* @param value_sz refer to **bpf_map__lookup_elem**'s description.' * @param flags extra flags passed to kernel for this operation * @return 0, on success; negative error, otherwise *
Signature · 1 line
2.51.2
PATCH v12 7/7 selftests/bpf: Add cases to test BPF_F_CPU and BPF_F_ALL_CPUS flags
In reply to Leon HwangAdd test coverage for the new BPF_F_CPU and BPF_F_ALL_CPUS flags support in percpu maps. The following APIs are exercised:
* bpf_map_update_batch()
* bpf_map_lookup_batch()
* bpf_map_update_elem()
* bpf_map__update_elem()
* bpf_map_lookup_elem_flags()
* bpf_map__lookup_elem()
For lru_percpu_hash map, set max_entries to 'libbpf_num_possible_cpus() + 1' and only use the first 'libbpf_num_possible_cpus()' entries. This ensures a spare entry is always available in the LRU free list, avoiding eviction.
When updating an existing key in lru_percpu_hash map:
1. l_new = prealloc_lru_pop(); /* Borrow from free list */ 2. l_old = lookup_elem_raw(); /* Found, key exists */ 3. pcpu_copy_value(); /* In-place update */ 4. bpf_lru_push_free(); /* Return l_new to free list */
Also add negative tests to verify that non-percpu array and hash maps reject the BPF_F_CPU and BPF_F_ALL_CPUS flags.
--- .../selftests/bpf/prog_tests/percpu_alloc.c | 328 ++++++++++++++++++ .../selftests/bpf/progs/percpu_alloc_array.c | 32 ++ 2 files changed, 360 insertions(+)
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c +328 -0
@@ -1,5 +1,6 @@// SPDX-License-Identifier: GPL-2.0 #include <test_progs.h> +#include "cgroup_helpers.h" #include "percpu_alloc_array.skel.h" #include "percpu_alloc_cgrp_local_storage.skel.h" #include "percpu_alloc_fail.skel.h"
@@ -115,6 +116,321 @@ static void test_failure(void) {RUN_TESTS (percpu_alloc_fail); } +static void test_percpu_map_op_cpu_flag (struct bpf_map *map,void *keys,size_t key_sz,u32 entries, +int nr_cpus,bool test_batch) +{ +size_t value_sz =sizeof (u32), value_sz_cpus, value_sz_total; +u32 *values =NULL , *values_percpu =NULL ; +const u32 value =0xDEADC0DE ; +int i, j, cpu, map_fd, err; +u64 batch =0 , flags; +void *values_row; +u32 count, v; +LIBBPF_OPTS (bpf_map_batch_opts, batch_opts); + + value_sz_cpus = value_sz * nr_cpus; + values =calloc (entries, value_sz_cpus); +if (!ASSERT_OK_PTR (values,"calloc values" )) +return ; + + values_percpu =calloc (entries,roundup (value_sz,8 ) * nr_cpus); +if (!ASSERT_OK_PTR (values_percpu,"calloc values_percpu" )) { +free (values); +return ; + } + + value_sz_total = value_sz_cpus * entries; +memset (values,0 , value_sz_total); + + map_fd =bpf_map__fd (map); + flags =BPF_F_CPU |BPF_F_ALL_CPUS ; + err =bpf_map_lookup_elem_flags (map_fd, keys, values, flags); +if (!ASSERT_ERR (err,"bpf_map_lookup_elem_flags cpu|all_cpus" )) + goto out; + + err =bpf_map_update_elem (map_fd, keys, values, flags); +if (!ASSERT_ERR (err,"bpf_map_update_elem cpu|all_cpus" )) + goto out; + + flags =BPF_F_ALL_CPUS ; + err =bpf_map_lookup_elem_flags (map_fd, keys, values, flags); +if (!ASSERT_ERR (err,"bpf_map_lookup_elem_flags all_cpus" )) + goto out; + + flags =BPF_F_LOCK |BPF_F_CPU ; + err =bpf_map_lookup_elem_flags (map_fd, keys, values, flags); +if (!ASSERT_ERR (err,"bpf_map_lookup_elem_flags BPF_F_LOCK" )) + goto out; + + flags =BPF_F_LOCK |BPF_F_ALL_CPUS ; + err =bpf_map_update_elem (map_fd, keys, values, flags); +if (!ASSERT_ERR (err,"bpf_map_update_elem BPF_F_LOCK" )) + goto out; + + flags = (u64 )nr_cpus <<32 |BPF_F_CPU ; + err =bpf_map_update_elem (map_fd, keys, values, flags); +if (!ASSERT_EQ (err, -ERANGE ,"bpf_map_update_elem -ERANGE" )) + goto out; + + err =bpf_map__update_elem (map, keys, key_sz, values, value_sz, flags); +if (!ASSERT_EQ (err, -ERANGE ,"bpf_map__update_elem -ERANGE" )) + goto out; + + err =bpf_map_lookup_elem_flags (map_fd, keys, values, flags); +if (!ASSERT_EQ (err, -ERANGE ,"bpf_map_lookup_elem_flags -ERANGE" )) + goto out; + + err =bpf_map__lookup_elem (map, keys, key_sz, values, value_sz, flags); +if (!ASSERT_EQ (err, -ERANGE ,"bpf_map__lookup_elem -ERANGE" )) + goto out; + +for (cpu =0 ; cpu < nr_cpus; cpu++) { +/* clear value on all cpus */ + values[0 ] =0 ; + flags =BPF_F_ALL_CPUS ; +for (i =0 ; i < entries; i++) { + err =bpf_map__update_elem (map, keys + i * key_sz, key_sz, values, + value_sz, flags); +if (!ASSERT_OK (err,"bpf_map__update_elem all_cpus" )) + goto out; + } + +/* update value on specified cpu */ +for (i =0 ; i < entries; i++) { + values[0 ] = value; + flags = (u64 )cpu <<32 |BPF_F_CPU ; + err =bpf_map__update_elem (map, keys + i * key_sz, key_sz, values, + value_sz, flags); +if (!ASSERT_OK (err,"bpf_map__update_elem specified cpu" )) + goto out; + +/* lookup then check value on CPUs */ +for (j =0 ; j < nr_cpus; j++) { + flags = (u64 )j <<32 |BPF_F_CPU ; + err =bpf_map__lookup_elem (map, keys + i * key_sz, key_sz, values, + value_sz, flags); +if (!ASSERT_OK (err,"bpf_map__lookup_elem specified cpu" )) + goto out; +if (!ASSERT_EQ (values[0 ], j != cpu ?0 : value, +"bpf_map__lookup_elem value on specified cpu" )) + goto out; + } + } + } + +if (!test_batch) + goto out; + + count = entries; + batch_opts.elem_flags = (u64 )nr_cpus <<32 |BPF_F_CPU ; + err =bpf_map_update_batch (map_fd, keys, values, &count, &batch_opts); +if (!ASSERT_EQ (err, -ERANGE ,"bpf_map_update_batch -ERANGE" )) + goto out; + +for (cpu =0 ; cpu < nr_cpus; cpu++) { +memset (values,0 , value_sz_total); + +/* clear values across all CPUs */ + count = entries; + batch_opts.elem_flags =BPF_F_ALL_CPUS ; + err =bpf_map_update_batch (map_fd, keys, values, &count, &batch_opts); +if (!ASSERT_OK (err,"bpf_map_update_batch all_cpus" )) + goto out; + +/* update values on specified CPU */ +for (i =0 ; i < entries; i++) + values[i] = value; + + count = entries; + batch_opts.elem_flags = (u64 )cpu <<32 |BPF_F_CPU ; + err =bpf_map_update_batch (map_fd, keys, values, &count, &batch_opts); +if (!ASSERT_OK (err,"bpf_map_update_batch specified cpu" )) + goto out; + +/* lookup values on specified CPU */ + batch =0 ; + count = entries; +memset (values,0 , entries * value_sz); + err =bpf_map_lookup_batch (map_fd,NULL , &batch, keys, values, &count, &batch_opts); +if (!ASSERT_TRUE (!err || err == -ENOENT ,"bpf_map_lookup_batch specified cpu" )) + goto out; + +for (i =0 ; i < entries; i++) +if (!ASSERT_EQ (values[i], value, +"bpf_map_lookup_batch value on specified cpu" )) + goto out; + +/* lookup values from all CPUs */ + batch =0 ; + count = entries; + batch_opts.elem_flags =0 ; +memset (values_percpu,0 ,roundup (value_sz,8 ) * nr_cpus * entries); + err =bpf_map_lookup_batch (map_fd,NULL , &batch, keys, values_percpu, &count, + &batch_opts); +if (!ASSERT_TRUE (!err || err == -ENOENT ,"bpf_map_lookup_batch all_cpus" )) + goto out; + +for (i =0 ; i < entries; i++) { + values_row = (void *) values_percpu + +roundup (value_sz,8 ) * i * nr_cpus; +for (j =0 ; j < nr_cpus; j++) { + v = *(u32 *) (values_row +roundup (value_sz,8 ) * j); +if (!ASSERT_EQ (v, j != cpu ?0 : value, +"bpf_map_lookup_batch value all_cpus" )) + goto out; + } + } + } + +out: +free (values_percpu); +free (values); +} + + +static void test_percpu_map_cpu_flag (enum bpf_map_type map_type) +{ +struct percpu_alloc_array *skel; +size_t key_sz =sizeof (int ); +int *keys, nr_cpus, i, err; +struct bpf_map *map; +u32 max_entries; + + nr_cpus =libbpf_num_possible_cpus (); +if (!ASSERT_GT (nr_cpus,0 ,"libbpf_num_possible_cpus" )) +return ; + + max_entries = nr_cpus +1 ; + keys =calloc (max_entries, key_sz); +if (!ASSERT_OK_PTR (keys,"calloc keys" )) +return ; + +for (i =0 ; i < max_entries; i++) + keys[i] = i; + + skel =percpu_alloc_array__open (); +if (!ASSERT_OK_PTR (skel,"percpu_alloc_array__open" )) { +free (keys); +return ; + } + + map = skel->maps.percpu; +bpf_map__set_type (map, map_type); +bpf_map__set_max_entries (map, max_entries); + + err =percpu_alloc_array__load (skel); +if (!ASSERT_OK (err,"test_percpu_alloc__load" )) + goto out; + +test_percpu_map_op_cpu_flag (map, keys, key_sz, max_entries -1 , nr_cpus, true); +out: +percpu_alloc_array__destroy (skel); +free (keys); +} + +static void test_percpu_array_cpu_flag (void ) +{ +test_percpu_map_cpu_flag (BPF_MAP_TYPE_PERCPU_ARRAY ); +} + +static void test_percpu_hash_cpu_flag (void ) +{ +test_percpu_map_cpu_flag (BPF_MAP_TYPE_PERCPU_HASH ); +} + +static void test_lru_percpu_hash_cpu_flag (void ) +{ +test_percpu_map_cpu_flag (BPF_MAP_TYPE_LRU_PERCPU_HASH ); +} + +static void test_percpu_cgroup_storage_cpu_flag (void ) +{ +struct percpu_alloc_array *skel =NULL ; +struct bpf_cgroup_storage_key key; +int cgroup, prog_fd, nr_cpus, err; +struct bpf_map *map; + + nr_cpus =libbpf_num_possible_cpus (); +if (!ASSERT_GT (nr_cpus,0 ,"libbpf_num_possible_cpus" )) +return ; + + err =setup_cgroup_environment (); +if (!ASSERT_OK (err,"setup_cgroup_environment" )) +return ; + + cgroup =create_and_get_cgroup ("/cg_percpu" ); +if (!ASSERT_GE (cgroup,0 ,"create_and_get_cgroup" )) { +cleanup_cgroup_environment (); +return ; + } + + err =join_cgroup ("/cg_percpu" ); +if (!ASSERT_OK (err,"join_cgroup" )) + goto out; + + skel =percpu_alloc_array__open_and_load (); +if (!ASSERT_OK_PTR (skel,"percpu_alloc_array__open_and_load" )) + goto out; + + prog_fd =bpf_program__fd (skel->progs.cgroup_egress); + err =bpf_prog_attach (prog_fd, cgroup,BPF_CGROUP_INET_EGRESS ,0 ); +if (!ASSERT_OK (err,"bpf_prog_attach" )) + goto out; + + map = skel->maps.percpu_cgroup_storage; + err =bpf_map_get_next_key (bpf_map__fd (map),NULL , &key); +if (!ASSERT_OK (err,"bpf_map_get_next_key" )) + goto out; + +test_percpu_map_op_cpu_flag (map, &key,sizeof (key),1 , nr_cpus, false); +out: +bpf_prog_detach2 (-1 , cgroup,BPF_CGROUP_INET_EGRESS ); +close (cgroup); +cleanup_cgroup_environment (); +percpu_alloc_array__destroy (skel); +} + +static void test_map_op_cpu_flag (enum bpf_map_type map_type) +{ +u32 max_entries =1 , count = max_entries; +u64 flags, batch =0 , val =0 ; +int err, map_fd, key =0 ; +LIBBPF_OPTS (bpf_map_batch_opts, batch_opts); + + map_fd =bpf_map_create (map_type,"test_cpu_flag" ,sizeof (int ),sizeof (u64), max_entries, +NULL ); +if (!ASSERT_GE (map_fd,0 ,"bpf_map_create" )) +return ; + + flags =BPF_F_ALL_CPUS ; + err =bpf_map_update_elem (map_fd, &key, &val, flags); +ASSERT_ERR (err,"bpf_map_update_elem all_cpus" ); + + batch_opts.elem_flags =BPF_F_ALL_CPUS ; + err =bpf_map_update_batch (map_fd, &key, &val, &count, &batch_opts); +ASSERT_ERR (err,"bpf_map_update_batch all_cpus" ); + + flags =BPF_F_CPU ; + err =bpf_map_lookup_elem_flags (map_fd, &key, &val, flags); +ASSERT_ERR (err,"bpf_map_lookup_elem_flags cpu" ); + + batch_opts.elem_flags =BPF_F_CPU ; + err =bpf_map_lookup_batch (map_fd,NULL , &batch, &key, &val, &count, &batch_opts); +ASSERT_ERR (err,"bpf_map_lookup_batch cpu" ); + +close (map_fd); +} + +static void test_array_cpu_flag (void ) +{ +test_map_op_cpu_flag (BPF_MAP_TYPE_ARRAY ); +} + +static void test_hash_cpu_flag (void ) +{ +test_map_op_cpu_flag (BPF_MAP_TYPE_HASH ); +} +void test_percpu_alloc (void ) {if (test__start_subtest ("array" ))
@@ -125,4 +441,16 @@ void test_percpu_alloc(void)test_cgrp_local_storage ();if (test__start_subtest ("failure_tests" ))test_failure (); +if (test__start_subtest ("cpu_flag_percpu_array" )) +test_percpu_array_cpu_flag (); +if (test__start_subtest ("cpu_flag_percpu_hash" )) +test_percpu_hash_cpu_flag (); +if (test__start_subtest ("cpu_flag_lru_percpu_hash" )) +test_lru_percpu_hash_cpu_flag (); +if (test__start_subtest ("cpu_flag_percpu_cgroup_storage" )) +test_percpu_cgroup_storage_cpu_flag (); +if (test__start_subtest ("cpu_flag_array" )) +test_array_cpu_flag (); +if (test__start_subtest ("cpu_flag_hash" )) +test_hash_cpu_flag (); }
tools/testing/selftests/bpf/progs/percpu_alloc_array.c +32 -0
@@ -187,4 +187,36 @@ int BPF_PROG(test_array_map_10)return 0 ; } +struct { + __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); + __uint(max_entries, 2); + __type(key, int); + __type(value, u32); +} percpu SEC(".maps"); + +SEC("?fentry/bpf_fentry_test1") +int BPF_PROG (test_percpu_array ,int x) +{ +u64 value =0xDEADC0DE ; +int key =0 ; + +bpf_map_update_elem (&percpu, &key, &value,BPF_ANY ); +return 0 ; +} + +struct { + __uint(type, BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE); + __type(key, struct bpf_cgroup_storage_key); + __type(value, u32); +} percpu_cgroup_storage SEC(".maps"); + +SEC("cgroup_skb/egress") +int cgroup_egress (struct __sk_buff *skb) +{ +u32 *val =bpf_get_local_storage (&percpu_cgroup_storage,0 ); + + *val =1 ; +return 1 ; +} + char _license[] SEC("license") = "GPL";
Signature · 1 line
2.51.2
[email protected]
AI reviewRe: [PATCH bpf-next v12 3/7] bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_hash and lru_percpu_hash maps
In reply to Leon HwangCan htab_map_check_update_flags() correctly validate the new flags?
Quoted 4 lines
include/linux/bpf.h +0
-0
[ ... ]
Quoted 4 lines
kernel/bpf/hashtab.c +0
-0
[ ... ]
kernel/bpf/hashtab.c static long htab_lru_map_update_elem(struct bpf_map *map, void *key, void *value · line 1269
Quoted 13 lines
kernel/bpf/hashtab.c +9
-0@@ -1249,6 +1258,15 @@ static long htab_lru_map_update_elem(struct bpf_map *map, void *key, void *valuereturn ret; } +static int htab_map_check_update_flags (bool onallcpus,u64 map_flags) +{ +if (unlikely (!onallcpus && map_flags >BPF_EXIST )) +return -EINVAL ; +if (unlikely (onallcpus && ((map_flags &BPF_F_LOCK ) || (u32 )map_flags >BPF_F_ALL_CPUS ))) +return -EINVAL ; +return 0 ; +} +
Can htab_map_check_update_flags() correctly validate the new flags?
When onallcpus=false (the common case for normal map_update_elem operations via htab_percpu_map_update_elem()-> htab_map_update_elem_in_place()), the check at the first if statement compares the full 64-bit map_flags value against BPF_EXIST (2).
When BPF_F_CPU (8) is set, or when the CPU number is embedded in the upper 32 bits (e.g., 0x0000000100000008 for CPU 1), map_flags will be greater than BPF_EXIST, causing this function to return -EINVAL.
This appears inconsistent with the equivalent check in bpf_percpu_array_update() (commit 51f40f83b), which uses:
if (unlikely((map_flags & BPF_F_LOCK) || (u32)map_flags > BPF_F_ALL_CPUS))
The percpu_array check casts to u32 to examine only the lower 32 bits, allowing the CPU number in the upper 32 bits. The hash table check for onallcpus=false doesn't perform this cast, so it will reject valid flag combinations that include BPF_F_CPU.
Would this prevent the BPF_F_CPU functionality from working for normal percpu_hash update operations?
Quoted 13 lines
kernel/bpf/hashtab.c +3
-3@@ -1262,9 +1280,9 @@ static long htab_map_update_elem_in_place(struct bpf_map *map, void *key,u32 key_size, hash;int ret; -if (unlikely (map_flags >BPF_EXIST )) -/* unknown flags */ -return -EINVAL ; + ret =htab_map_check_update_flags (onallcpus, map_flags); +if (unlikely (ret)) +return ret;WARN_ON_ONCE (!bpf_rcu_lock_held ());
[ ... ]
kernel/bpf/hashtab.c static long __htab_lru_percpu_map_update_elem(struct bpf_map *map, void *key, · line 1347
Quoted 10 lines
kernel/bpf/hashtab.c +3
-3@@ -1324,9 +1342,9 @@ static long __htab_lru_percpu_map_update_elem(struct bpf_map *map, void *key,u32 key_size, hash;int ret; -if (unlikely (map_flags >BPF_EXIST )) -/* unknown flags */ -return -EINVAL ; + ret =htab_map_check_update_flags (onallcpus, map_flags); +if (unlikely (ret)) +return ret;
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/19708012130
[email protected]
AI reviewRe: [PATCH bpf-next v12 5/7] bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_cgroup_storage maps
In reply to Leon HwangShould check_and_init_map_value() be called here after copy_map_value()?
Quoted 4 lines
include/linux/bpf-cgroup.h +0
-0
[ ... ]
Quoted 4 lines
include/linux/bpf.h +0
-0
[ ... ]
Quoted 14 lines
kernel/bpf/local_storage.c +1
-1@@ -180,7 +180,7 @@ static long cgroup_storage_update_elem(struct bpf_map *map, void *key, }int bpf_percpu_cgroup_storage_copy (struct bpf_map *_map,void *key, -void *value) +void *value,u64 map_flags) {struct bpf_cgroup_storage_map *map =map_to_storage (_map);struct bpf_cgroup_storage *storage;
kernel/bpf/local_storage.c int bpf_percpu_cgroup_storage_copy(struct bpf_map *_map, void *key, · line 205
Quoted 8 lines
kernel/bpf/local_storage.c +5
-0@@ -198,11 +198,17 @@ int bpf_percpu_cgroup_storage_copy(struct bpf_map *_map, void *key,* access 'value_size' of them, so copying rounded areas * will not leak any kernel data */ +if (map_flags &BPF_F_CPU ) { + cpu = map_flags >>32 ; +copy_map_value (_map, value,per_cpu_ptr (storage->percpu_buf, cpu)); + goto unlock; + }
Should check_and_init_map_value() be called here after copy_map_value()?
The similar implementations in bpf_percpu_array_copy() and bpf_percpu_hash_copy() both call check_and_init_map_value() after copying a single CPU's value when BPF_F_CPU is set:
bpf_percpu_array_copy():
if (map_flags & BPF_F_CPU) {
cpu = map_flags >> 32;
copy_map_value(map, value, per_cpu_ptr(pptr, cpu));
check_and_init_map_value(map, value);
goto unlock;
}
bpf_percpu_hash_copy():
if (map_flags & BPF_F_CPU) {
cpu = map_flags >> 32;
copy_map_value(map, value, per_cpu_ptr(pptr, cpu));
check_and_init_map_value(map, value);
goto out;
}The check_and_init_map_value() call initializes special BTF fields (timers, kptrs, etc.) when copying map values to userspace. Without it, these fields might not be properly initialized if percpu_cgroup_storage map values contain such fields.
kernel/bpf/local_storage.c int bpf_percpu_cgroup_storage_copy(struct bpf_map *_map, void *key, · line 214
Quoted 9 lines
kernel/bpf/local_storage.c +1
-0size =round_up (_map->value_size,8 );for_each_possible_cpu (cpu) {copy_map_value_long (_map, value + off,per_cpu_ptr (storage->percpu_buf, cpu)); off += size; } +unlock:rcu_read_unlock ();return 0 ; }
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/19708012130
[email protected]
AI reviewRe: [PATCH bpf-next v12 2/7] bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_array maps
In reply to Leon HwangDoes changing this from bitwise-AND to equality break flag combination handling? The old code checked (map_flags & BPF_NOEXIST), which would trigger for any flag combination including BPF_NOEXIST.…
Quoted 4 lines
include/linux/bpf.h +0
-0
[ ... ]
Quoted 4 lines
kernel/bpf/arraymap.c +0
-0
[ ... ]
Quoted 15 lines
kernel/bpf/arraymap.c +3
-2@@ -398,10 +405,11 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value, struct bpf_array *array = container_of(map, struct bpf_array, map);u32 index = *(u32 *)key;void __percpu *pptr; -int cpu, off =0 ; +void *ptr, *val;u32 size; +int cpu; -if (unlikely (map_flags >BPF_EXIST )) +if (unlikely ((map_flags &BPF_F_LOCK ) || (u32 )map_flags >BPF_F_ALL_CPUS ))/* unknown flags */ return -EINVAL ;
kernel/bpf/arraymap.c int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value, · line 420
Quoted 5 lines
kernel/bpf/arraymap.c +1
-1@@ -409,7 +417,7 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value,/* all elements were pre-allocated, cannot insert a new one */ return -E2BIG ; -if (unlikely (map_flags &BPF_NOEXIST )) +if (unlikely (map_flags ==BPF_NOEXIST ))
^^
Does changing this from bitwise-AND to equality break flag combination handling? The old code checked (map_flags & BPF_NOEXIST), which would trigger for any flag combination including BPF_NOEXIST. The new code only checks (map_flags == BPF_NOEXIST), so if userspace passes (BPF_F_CPU | BPF_NOEXIST), the BPF_NOEXIST flag gets silently ignored instead of returning -EEXIST as before.
While all elements in percpu_array maps are pre-allocated and BPF_NOEXIST should always fail anyway, silently ignoring a flag the user explicitly set seems inconsistent with the principle of rejecting invalid flag combinations at validation time.
kernel/bpf/arraymap.c int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value, · line 423
Quoted 3 lines
kernel/bpf/arraymap.c +0
-0/* all elements already exist */ return -EEXIST ;
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/19708012130
Re: [PATCH bpf-next v12 2/7] bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_array maps
Quoted 28 lines · On 2025/11/26 23:11, [email protected] wrote:
Quoted 4 lines
include/linux/bpf.h +0
-0[ ... ]
Quoted 4 lines
kernel/bpf/arraymap.c +0
-0[ ... ]
Quoted 14 lines
kernel/bpf/arraymap.c +3
-2@@ -398,10 +405,11 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value, struct bpf_array *array = container_of(map, struct bpf_array, map);u32 index = *(u32 *)key;void __percpu *pptr; -int cpu, off =0 ; +void *ptr, *val;u32 size; +int cpu; -if (unlikely (map_flags >BPF_EXIST )) +if (unlikely ((map_flags &BPF_F_LOCK ) || (u32 )map_flags >BPF_F_ALL_CPUS ))/* unknown flags */ return -EINVAL ;
Quoted 8 lines
kernel/bpf/arraymap.c int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value, · line 420
Quoted 6 lines
kernel/bpf/arraymap.c +1
-1@@ -409,7 +417,7 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value,/* all elements were pre-allocated, cannot insert a new one */ return -E2BIG ; -if (unlikely (map_flags &BPF_NOEXIST )) +if (unlikely (map_flags ==BPF_NOEXIST ))^^
?
No such change in this version.
It seems that this change was inferred from v11 to v12 by AI itself.
Thanks,
Leon
Quoted 24 lines
Does changing this from bitwise-AND to equality break flag combination handling? The old code checked (
map_flags&BPF_NOEXIST), which would trigger for any flag combination includingBPF_NOEXIST. The new code only checks (map_flags==BPF_NOEXIST), so if userspace passes (BPF_F_CPU|BPF_NOEXIST), theBPF_NOEXISTflag gets silently ignored instead of returning-EEXISTas before.While all elements in
percpu_arraymaps are pre-allocated andBPF_NOEXISTshould always fail anyway, silently ignoring a flag the user explicitly set seems inconsistent with the principle of rejecting invalid flag combinations at validation time.kernel/bpf/arraymap.c int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value, · line 423
Quoted 3 lines
kernel/bpf/arraymap.c +0
-0/* all elements already exist */ return -EEXIST ;[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/19708012130
Re: [PATCH bpf-next v12 5/7] bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_cgroup_storage maps
In reply to [email protected]Quoted 30 lines · On 2025/11/26 23:11, [email protected] wrote:
Quoted 4 lines
include/linux/bpf-cgroup.h +0
-0[ ... ]
Quoted 4 lines
include/linux/bpf.h +0
-0[ ... ]
Quoted 16 lines
kernel/bpf/local_storage.c +1
-1@@ -180,7 +180,7 @@ static long cgroup_storage_update_elem(struct bpf_map *map, void *key, }int bpf_percpu_cgroup_storage_copy (struct bpf_map *_map,void *key, -void *value) +void *value,u64 map_flags) {struct bpf_cgroup_storage_map *map =map_to_storage (_map);struct bpf_cgroup_storage *storage;@@ -198,11 +198,17 @@ int bpf_percpu_cgroup_storage_copy(struct bpf_map *_map, void *key,* access 'value_size' of them, so copying rounded areas * will not leak any kernel data
Quoted 9 lines
kernel/bpf/local_storage.c int bpf_percpu_cgroup_storage_copy(struct bpf_map *_map, void *key, · line 205
Quoted 6 lines
kernel/bpf/local_storage.c +5
-0*/ +if (map_flags &BPF_F_CPU ) { + cpu = map_flags >>32 ; +copy_map_value (_map, value,per_cpu_ptr (storage->percpu_buf, cpu)); + goto unlock; + }Should
check_and_init_map_value()be called here aftercopy_map_value()?
It should not be called, because percpu_cgroup_storage maps do not support any special fields.
Thanks,
Leon
Quoted 42 lines
The similar implementations in
bpf_percpu_array_copy()andbpf_percpu_hash_copy()both callcheck_and_init_map_value()after copying a single CPU's value whenBPF_F_CPUis set:bpf_percpu_array_copy(): if (map_flags & BPF_F_CPU) { cpu = map_flags >> 32; copy_map_value(map, value, per_cpu_ptr(pptr, cpu)); check_and_init_map_value(map, value); goto unlock; } bpf_percpu_hash_copy(): if (map_flags & BPF_F_CPU) { cpu = map_flags >> 32; copy_map_value(map, value, per_cpu_ptr(pptr, cpu)); check_and_init_map_value(map, value); goto out; }The
check_and_init_map_value()call initializes special BTF fields
(timers, kptrs, etc.) when copying map values to userspace. Without it,> these fields might not be properly initialized ifpercpu_cgroup_storagemap values contain such fields.kernel/bpf/local_storage.c int bpf_percpu_cgroup_storage_copy(struct bpf_map *_map, void *key, · line 214
Quoted 9 lines
kernel/bpf/local_storage.c +1
-0size =round_up (_map->value_size,8 );for_each_possible_cpu (cpu) {copy_map_value_long (_map, value + off,per_cpu_ptr (storage->percpu_buf, cpu)); off += size; } +unlock:rcu_read_unlock ();return 0 ; }[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/19708012130
Re: [PATCH bpf-next v12 2/7] bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_array maps
In reply to Leon HwangQuoted 40 lines · Leon Hwang
Quoted 28 lines · On 2025/11/26 23:11, [email protected] wrote:
Quoted 4 lines
include/linux/bpf.h +0
-0[ ... ]
Quoted 4 lines
kernel/bpf/arraymap.c +0
-0[ ... ]
Quoted 14 lines
kernel/bpf/arraymap.c +3
-2@@ -398,10 +405,11 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value, struct bpf_array *array = container_of(map, struct bpf_array, map);u32 index = *(u32 *)key;void __percpu *pptr; -int cpu, off =0 ; +void *ptr, *val;u32 size; +int cpu; -if (unlikely (map_flags >BPF_EXIST )) +if (unlikely ((map_flags &BPF_F_LOCK ) || (u32 )map_flags >BPF_F_ALL_CPUS ))/* unknown flags */ return -EINVAL ;Quoted 8 lines
kernel/bpf/arraymap.c int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value, · line 420
Quoted 6 lines
kernel/bpf/arraymap.c +1
-1@@ -409,7 +417,7 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value,/* all elements were pre-allocated, cannot insert a new one */ return -E2BIG ; -if (unlikely (map_flags &BPF_NOEXIST )) +if (unlikely (map_flags ==BPF_NOEXIST ))^^
Quoted 5 lines
?
No such change in this version.
It seems that this change was inferred from v11 to v12 by AI itself
Thanks for flagging this, I'll try to find this section of the logs to see how the false positive checks failed to catch it.
-chris
Re: [PATCH bpf-next v12 3/7] bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_hash and lru_percpu_hash maps
In reply to [email protected]Quoted 44 lines · On 2025/11/26 23:11, [email protected] wrote:
Quoted 4 lines
include/linux/bpf.h +0
-0[ ... ]
Quoted 4 lines
kernel/bpf/hashtab.c +0
-0[ ... ]
kernel/bpf/hashtab.c static long htab_lru_map_update_elem(struct bpf_map *map, void *key, void *value · line 1269
Quoted 13 lines
kernel/bpf/hashtab.c +9
-0@@ -1249,6 +1258,15 @@ static long htab_lru_map_update_elem(struct bpf_map *map, void *key, void *valuereturn ret; } +static int htab_map_check_update_flags (bool onallcpus,u64 map_flags) +{ +if (unlikely (!onallcpus && map_flags >BPF_EXIST )) +return -EINVAL ; +if (unlikely (onallcpus && ((map_flags &BPF_F_LOCK ) || (u32 )map_flags >BPF_F_ALL_CPUS ))) +return -EINVAL ; +return 0 ; +} +Can
htab_map_check_update_flags()correctly validate the new flags?When onallcpus=false (the common case for normal
map_update_elemoperations via htab_percpu_map_update_elem()->htab_map_update_elem_in_place()), the check at the first if statement compares the full 64-bitmap_flagsvalue againstBPF_EXIST(2).When
BPF_F_CPU(8) is set, or when the CPU number is embedded in the upper 32 bits (e.g.,0x0000000100000008for CPU 1),map_flagswill be greater thanBPF_EXIST, causing this function to return-EINVAL.This appears inconsistent with the equivalent check in
bpf_percpu_array_update()(commit51f40f83b), which uses:if (unlikely((map_flags & BPF_F_LOCK) || (u32)map_flags > BPF_F_ALL_CPUS))
Quoted 8 lines
The
percpu_arraycheck casts to u32 to examine only the lower 32 bits, allowing the CPU number in the upper 32 bits. The hash table check for onallcpus=false doesn't perform this cast, so it will reject valid flag combinations that includeBPF_F_CPU.Would this prevent the
BPF_F_CPUfunctionality from working for normalpercpu_hashupdate operations?
For UAPI percpu_hash updates, this is fine because onallcpus is always true.
For BPF program–initiated percpu_hash updates, the behavior also remains unchanged since onallcpus is false, and the 'map_flags > BPF_EXIST' check behaves exactly as it did before.
Thanks,
Leon
Quoted 35 lines
Quoted 13 lines
kernel/bpf/hashtab.c +3
-3@@ -1262,9 +1280,9 @@ static long htab_map_update_elem_in_place(struct bpf_map *map, void *key,u32 key_size, hash;int ret; -if (unlikely (map_flags >BPF_EXIST )) -/* unknown flags */ -return -EINVAL ; + ret =htab_map_check_update_flags (onallcpus, map_flags); +if (unlikely (ret)) +return ret;WARN_ON_ONCE (!bpf_rcu_lock_held ());[ ... ]
kernel/bpf/hashtab.c static long __htab_lru_percpu_map_update_elem(struct bpf_map *map, void *key, · line 1347
Quoted 10 lines
kernel/bpf/hashtab.c +3
-3@@ -1324,9 +1342,9 @@ static long __htab_lru_percpu_map_update_elem(struct bpf_map *map, void *key,u32 key_size, hash;int ret; -if (unlikely (map_flags >BPF_EXIST )) -/* unknown flags */ -return -EINVAL ; + ret =htab_map_check_update_flags (onallcpus, map_flags); +if (unlikely (ret)) +return ret;[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/19708012130
Ihor Solodrai
reviewerRe: [PATCH bpf-next v12 2/7] bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_array maps
In reply to Chris MasonQuoted 45 lines · Chris Mason
Quoted 42 lines · Leon Hwang
Quoted 28 lines · On 2025/11/26 23:11, [email protected] wrote:
Quoted 4 lines
include/linux/bpf.h +0
-0[ ... ]
Quoted 4 lines
kernel/bpf/arraymap.c +0
-0[ ... ]
Quoted 14 lines
kernel/bpf/arraymap.c +3
-2@@ -398,10 +405,11 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value, struct bpf_array *array = container_of(map, struct bpf_array, map);u32 index = *(u32 *)key;void __percpu *pptr; -int cpu, off =0 ; +void *ptr, *val;u32 size; +int cpu; -if (unlikely (map_flags >BPF_EXIST )) +if (unlikely ((map_flags &BPF_F_LOCK ) || (u32 )map_flags >BPF_F_ALL_CPUS ))/* unknown flags */ return -EINVAL ;Quoted 8 lines
kernel/bpf/arraymap.c int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value, · line 420
Quoted 6 lines
kernel/bpf/arraymap.c +1
-1@@ -409,7 +417,7 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value,/* all elements were pre-allocated, cannot insert a new one */ return -E2BIG ; -if (unlikely (map_flags &BPF_NOEXIST )) +if (unlikely (map_flags ==BPF_NOEXIST ))^^?
Quoted 8 lines
Quoted 3 lines
No such change in this version.
It seems that this change was inferred from v11 to v12 by AI itself
Thanks for flagging this, I'll try to find this section of the logs to see how the false positive checks failed to catch it.
-chris
AI got confused here, this was not in the diff.
But it appears it got triggered, because there are these two code fragments nearby [1]:
if (unlikely(map_flags & BPF_NOEXIST)) /* all elements already exist */ return -EEXIST;
and
if (unlikely(map_flags == BPF_NOEXIST)) /* all elements already exist */ return -EEXIST;
Which is a good thing to notice even if this is intentional.
Anyone knows if it is?
https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/tree/kernel/bpf/arraymap.c#n356
Quoted 1 line
Quoted 22 lines · Leon Hwang
This patch set introduces the
BPF_F_CPUandBPF_F_ALL_CPUSflags for percpu maps, as the requirement ofBPF_F_ALL_CPUSflag forpercpu_arraymaps was discussed in the thread of
"[PATCH bpf-next v3 0/4] bpf: Introduce global percpu data"[1].The goal of
BPF_F_ALL_CPUSflag is to reduce data caching overhead in light skeletons by allowing a single value to be reused to update values across all CPUs. This avoids the M:N problem where M cached values are used to update a map on N CPUs kernel.The
BPF_F_CPUflag is accompanied by *flags*-embedded cpu info, which specifies the target CPU for the operation:* For lookup operations: the flag field alongside cpu info enable querying a value on the specified CPU.
* For update operations: the flag field alongside cpu info enable updating value for specified CPU.
Quoted 6 lines
Changes:
v11 -> v12:
* Dropped the v11 changes.
* Stabilized thelru_percpu_hashmap test by keeping an extra spare entry, which can be used temporarily during updates to avoid unintended LRU evictions.
Please rebase and resend.
This set got lost during holidays.