bpf: Introduce BPF_F_CPU flag for percpu_array map
Subsystems (5)
cover bpf: Introduce BPF_F_CPU flag for percpu_array map
This patch set introduces the BPF_F_CPU flag for percpu_array maps, as discussed in the thread of
"[PATCH bpf-next v3 0/4] bpf: Introduce global percpu data"[1].
The goal is to reduce data caching overhead in light skeletons by allowing a single value to be reused across all CPUs. This avoids the M:N problem where M cached values are used to update a map on N CPUs kernel.
The BPF_F_CPU flag is accompanied by a cpu field, which specifies the target CPUs for the operation:
* For lookup operations: the flag and cpu field enable querying a value on the specified CPU.
* For update operations:
* If cpu == (u32)~0, the provided value is copied to all CPUs.
* Otherwise, the value is copied to the specified CPU only.
Currently, this functionality is only supported for percpu_array maps.
Links:
[1] https://lore.kernel.org/bpf/[email protected]/
Changes:
RFC v2 -> v1:
* Address comments from Andrii:
* Use '&=' and '|='.
* Replace 'reuse_value' with simpler and less duplication code.
* Replace 'ASSERT_FALSE' with two 'ASSERT_OK_PTR's in self test.
RFC v1 -> RFC v2:
* Address comments from Andrii:
* Embed cpu to flags on kernel side.
* Change BPF_ALL_CPU macro to BPF_ALL_CPUS enum.
* Copy/update element within RCU protection.
* Update bpf_map_value_size() including BPF_F_CPU case.
* Use zero as default value to get cpu option.
* Update documents of APIs to be generic.
* Add size_t:0 to opts definitions.
* Update validate_map_op() including BPF_F_CPU case.
* Use LIBBPF_OPTS instead of DECLARE_LIBBPF_OPTS.
Leon Hwang (3):
bpf: Introduce BPF_F_CPU flag for percpu_array map bpf, libbpf: Support BPF_F_CPU for percpu_array map selftests/bpf: Add case to test BPF_F_CPU
include/linux/bpf.h | 3 +- include/uapi/linux/bpf.h | 7 + kernel/bpf/arraymap.c | 54 ++++-- kernel/bpf/syscall.c | 52 ++++-- tools/include/uapi/linux/bpf.h | 7 + tools/lib/bpf/bpf.c | 23 +++ tools/lib/bpf/bpf.h | 36 +++- tools/lib/bpf/libbpf.c | 56 +++++- tools/lib/bpf/libbpf.h | 53 +++++- tools/lib/bpf/libbpf.map | 5 + tools/lib/bpf/libbpf_common.h | 14 ++ .../selftests/bpf/prog_tests/percpu_alloc.c | 172 ++++++++++++++++++ .../selftests/bpf/progs/percpu_array_flag.c | 24 +++ 13 files changed, 459 insertions(+), 47 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/percpu_array_flag.c
--
2.50.1
PATCH 1/3 bpf: Introduce BPF_F_CPU flag for percpu_array map
This patch introduces support for the BPF_F_CPU flag in percpu_array maps to allow updating or looking up values for specified CPUs or for all CPUs with a single value.
This enhancement enables:
* Efficient update of all CPUs using a single value when cpu == (u32)~0.
* Targeted update or lookup for a specified CPU otherwise.
The flag is passed via:
* map_flags in bpf_percpu_array_update() along with embedded cpu field.
* elem_flags in generic_map_update_batch() along with separated cpu field.
--- include/linux/bpf.h | 3 +- include/uapi/linux/bpf.h | 7 +++++ kernel/bpf/arraymap.c | 54 ++++++++++++++++++++++++++-------- kernel/bpf/syscall.c | 52 ++++++++++++++++++++------------ tools/include/uapi/linux/bpf.h | 7 +++++ 5 files changed, 90 insertions(+), 33 deletions(-)
include/linux/bpf.h +2 -1
@@ -2671,7 +2671,8 @@ int map_set_for_each_callback_args(struct bpf_verifier_env *env, struct bpf_func_state *callee);int bpf_percpu_hash_copy (struct bpf_map *map,void *key,void *value); -int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value); +int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value, +u64 flags);int bpf_percpu_hash_update (struct bpf_map *map,void *key,void *value,u64 flags); int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value,
include/uapi/linux/bpf.h +7 -0
@@ -1372,6 +1372,12 @@ enum {BPF_NOEXIST =1 ,/* create new element if it didn't exist */ BPF_EXIST =2 ,/* update existing element */ BPF_F_LOCK =4 ,/* spin_lock-ed map_lookup/map_update */ + BPF_F_CPU = 8, /* map_update for percpu_array */ +}; + +enum { +/* indicate updating value across all CPUs for percpu maps. */ +BPF_ALL_CPUS = (__u32 )~0 , };/* flags for BPF_MAP_CREATE command */
@@ -1549,6 +1555,7 @@ union bpf_attr {__u32 map_fd;__u64 elem_flags;__u64 flags; +__u32 cpu; } batch;struct {/* anonymous struct used by BPF_PROG_LOAD command */
kernel/bpf/arraymap.c +41 -13
91 lines not on this page · raw message
kernel/bpf/syscall.c +33 -19
136 lines not on this page · raw message
tools/include/uapi/linux/bpf.h +7 -0
@@ -1372,6 +1372,12 @@ enum {BPF_NOEXIST =1 ,/* create new element if it didn't exist */ BPF_EXIST =2 ,/* update existing element */ BPF_F_LOCK =4 ,/* spin_lock-ed map_lookup/map_update */ + BPF_F_CPU = 8, /* map_update for percpu_array */ +}; + +enum { +/* indicate updating value across all CPUs for percpu maps. */ +BPF_ALL_CPUS = (__u32 )~0 , };/* flags for BPF_MAP_CREATE command */
@@ -1549,6 +1555,7 @@ union bpf_attr {__u32 map_fd;__u64 elem_flags;__u64 flags; +__u32 cpu; } batch;struct {/* anonymous struct used by BPF_PROG_LOAD command */
Signature · 1 line
2.50.1
PATCH 2/3 bpf, libbpf: Support BPF_F_CPU for percpu_array map
In reply to Leon HwangThis patch adds libbpf support for the BPF_F_CPU flag in percpu_array maps, introducing the following APIs:
1. bpf_map_update_elem_opts(): update with struct bpf_map_update_elem_opts
2. bpf_map_lookup_elem_opts(): lookup with struct bpf_map_lookup_elem_opts
3. bpf_map__update_elem_opts(): high-level wrapper with input validation
4. bpf_map__lookup_elem_opts(): high-level wrapper with input validation
Behavior:
* If opts->cpu == (u32)~0, the update is applied to all CPUs.
* Otherwise, it applies only to the specified CPU.
* Lookup APIs retrieve values from the target CPU when BPF_F_CPU is used.
--- tools/lib/bpf/bpf.c | 23 ++++++++++++++ tools/lib/bpf/bpf.h | 36 +++++++++++++++++++++- tools/lib/bpf/libbpf.c | 56 +++++++++++++++++++++++++++++++---- tools/lib/bpf/libbpf.h | 53 ++++++++++++++++++++++++++++----- tools/lib/bpf/libbpf.map | 5 ++++ tools/lib/bpf/libbpf_common.h | 14 +++++++++ 6 files changed, 173 insertions(+), 14 deletions(-)
tools/lib/bpf/bpf.c +23 -0
@@ -402,6 +402,17 @@ int bpf_map_update_elem(int fd, const void *key, const void *value,return libbpf_err_errno (ret); } +int bpf_map_update_elem_opts (int fd,const void *key,const void *value, +const struct bpf_map_update_elem_opts *opts) +{ +__u64 flags; +__u32 cpu; + + cpu =OPTS_GET (opts, cpu,0 ); + flags = ((__u64 ) cpu) <<32 |OPTS_GET (opts, flags,0 ); +return bpf_map_update_elem (fd, key, value, flags); +} +int bpf_map_lookup_elem (int fd,const void *key,void *value) { const size_t attr_sz = offsetofend(union bpf_attr, flags);
@@ -433,6 +444,17 @@ int bpf_map_lookup_elem_flags(int fd, const void *key, void *value, __u64 flags)return libbpf_err_errno (ret); } +int bpf_map_lookup_elem_opts (int fd,const void *key,void *value, +const struct bpf_map_lookup_elem_opts *opts) +{ +__u64 flags; +__u32 cpu; + + cpu =OPTS_GET (opts, cpu,0 ); + flags = ((__u64 ) cpu) <<32 |OPTS_GET (opts, flags,0 ); +return bpf_map_lookup_elem_flags (fd, key, value, flags); +} +int bpf_map_lookup_and_delete_elem (int fd,const void *key,void *value) { const size_t attr_sz = offsetofend(union bpf_attr, flags);
@@ -542,6 +564,7 @@ static int bpf_map_batch_common(int cmd, int fd, void *in_batch, attr.batch.count = *count; attr.batch.elem_flags =OPTS_GET (opts, elem_flags,0 ); attr.batch.flags =OPTS_GET (opts, flags,0 ); + attr.batch.cpu =OPTS_GET (opts, cpu,0 ); ret =sys_bpf (cmd, &attr, attr_sz); *count = attr.batch.count;
tools/lib/bpf/bpf.h +35 -1
@@ -163,12 +163,42 @@ LIBBPF_API int bpf_map_delete_elem_flags(int fd, const void *key, __u64 flags); LIBBPF_API int bpf_map_get_next_key(int fd, const void *key, void *next_key); LIBBPF_API int bpf_map_freeze(int fd); +/** +* @brief **bpf_map_update_elem_opts** allows for updating map with options. +* +* @param fd BPF map file descriptor +* @param key pointer to key +* @param value pointer to value +* @param opts options for configuring the way to update map +* @return 0, on success; negative error code, otherwise (errno is also set to +* the error code) +*/ +LIBBPF_API int bpf_map_update_elem_opts(int fd, const void *key, const void *value, +const struct bpf_map_update_elem_opts *opts); + +/** +* @brief **bpf_map_lookup_elem_opts** allows for looking up the value with +* options. +* +* @param fd BPF map file descriptor +* @param key pointer to key +* @param value pointer to value +* @param opts options for configuring the way to lookup map +* @return 0, on success; negative error code, otherwise (errno is also set to +* the error code) +*/ +LIBBPF_API int bpf_map_lookup_elem_opts(int fd, const void *key, void *value, +const struct bpf_map_lookup_elem_opts *opts); + +struct bpf_map_batch_opts {size_t sz;/* size of this struct for forward/backward compatibility */ __u64 elem_flags;__u64 flags; +__u32 cpu; + size_t:0; }; -#define bpf_map_batch_opts__last_field flags +#define bpf_map_batch_opts__last_field cpu/**
@@ -286,6 +316,10 @@ LIBBPF_API int bpf_map_lookup_and_delete_batch(int fd, void *in_batch,* Update spin_lock-ed map elements. This must be * specified if the map value contains a spinlock. * +* **BPF_F_CPU** +* As for percpu map, update value across all CPUs if **opts->cpu** is +* (__u32)~0, or on specified CPU otherwise. +* * @param fd BPF map file descriptor * @param keys pointer to an array of *count* keys * @param values pointer to an array of *count* values
tools/lib/bpf/libbpf.c +50 -6
109 lines not on this page · raw message
tools/lib/bpf/libbpf.h +46 -7
71 lines not on this page · raw message
tools/lib/bpf/libbpf.map +5 -0
@@ -448,4 +448,9 @@ LIBBPF_1.6.0 { } LIBBPF_1.5.0; LIBBPF_1.7.0 { + global: + bpf_map__lookup_elem_opts; + bpf_map__update_elem_opts; + bpf_map_lookup_elem_opts; + bpf_map_update_elem_opts; } LIBBPF_1.6.0;
tools/lib/bpf/libbpf_common.h +14 -0
@@ -89,4 +89,18 @@ memcpy(&NAME, &___##NAME, sizeof(NAME)); \ } while (0) +struct bpf_map_update_elem_opts { +size_t sz;/* size of this struct for forward/backward compatibility */ +__u64 flags; +__u32 cpu; + size_t:0; +}; + +struct bpf_map_lookup_elem_opts { +size_t sz;/* size of this struct for forward/backward compatibility */ +__u64 flags; +__u32 cpu; + size_t:0; +}; +#endif /* __LIBBPF_LIBBPF_COMMON_H */
Signature · 1 line
2.50.1
PATCH 3/3 selftests/bpf: Add case to test BPF_F_CPU
In reply to Leon HwangThis patch adds test coverage for the new BPF_F_CPU flag support in percpu_array maps. The following APIs are exercised:
* bpf_map_update_batch()
* bpf_map_lookup_batch()
* bpf_map_update_elem_opts()
* bpf_map__update_elem_opts()
* bpf_map_lookup_elem_opts()
* bpf_map__lookup_elem_opts()
cd tools/testing/selftests/bpf/
./test_progs -t percpu_alloc/cpu_flag_tests
253/13 percpu_alloc/cpu_flag_tests:OK
253 percpu_alloc:OK
Summary: 1/13 PASSED, 0 SKIPPED, 0 FAILED
--- .../selftests/bpf/prog_tests/percpu_alloc.c | 172 ++++++++++++++++++ .../selftests/bpf/progs/percpu_array_flag.c | 24 +++ 2 files changed, 196 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/percpu_array_flag.c
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c +172 -0
@@ -3,6 +3,7 @@#include "percpu_alloc_array.skel.h" #include "percpu_alloc_cgrp_local_storage.skel.h" #include "percpu_alloc_fail.skel.h" +#include "percpu_array_flag.skel.h" static void test_array (void ) {
@@ -115,6 +116,175 @@ static void test_failure(void) {RUN_TESTS (percpu_alloc_fail); } +static void test_cpu_flag (void ) +{ +int map_fd, *keys =NULL , value_size, cpu, i, j, nr_cpus, err; +size_t key_sz =sizeof (int ), value_sz =sizeof (u64); +struct percpu_array_flag *skel; +u64 batch =0 , *values =NULL ; +const u64 value =0xDEADC0DE ; +u32 count, max_entries; +struct bpf_map *map; +LIBBPF_OPTS (bpf_map_lookup_elem_opts, lookup_opts, + .flags = BPF_F_CPU, + .cpu = 0, + ); +LIBBPF_OPTS (bpf_map_update_elem_opts, update_opts, + .flags = BPF_F_CPU, + .cpu = 0, + ); +LIBBPF_OPTS (bpf_map_batch_opts, batch_opts, + .elem_flags = BPF_F_CPU, + .flags = 0, + ); + + nr_cpus =libbpf_num_possible_cpus (); +if (!ASSERT_GT (nr_cpus,0 ,"libbpf_num_possible_cpus" )) +return ; + + skel =percpu_array_flag__open_and_load (); +if (!ASSERT_OK_PTR (skel,"percpu_array_flag__open_and_load" )) +return ; + + map = skel->maps.percpu; + map_fd =bpf_map__fd (map); + max_entries =bpf_map__max_entries (map); + + value_size = value_sz * nr_cpus; + values =calloc (max_entries, value_size); +if (!ASSERT_OK_PTR (values,"calloc values" )) + goto out; + keys =calloc (max_entries, key_sz); +if (!ASSERT_OK_PTR (keys,"calloc keys" )) + goto out; + +for (i =0 ; i < max_entries; i++) + keys[i] = i; +memset (values,0 , max_entries * value_size); + + batch_opts.cpu = nr_cpus; + err =bpf_map_update_batch (map_fd, keys, values, &max_entries, &batch_opts); +if (!ASSERT_EQ (err, -E2BIG ,"bpf_map_update_batch -E2BIG" )) + goto out; + +for (cpu =0 ; cpu < nr_cpus; cpu++) { +memset (values,0 , max_entries * value_size); + +/* clear values on all CPUs */ + batch_opts.cpu =BPF_ALL_CPUS ; + batch_opts.elem_flags =BPF_F_CPU ; + err =bpf_map_update_batch (map_fd, keys, values, &max_entries, &batch_opts); +if (!ASSERT_OK (err,"bpf_map_update_batch all cpus" )) + goto out; + +/* update values on current CPU */ +for (i =0 ; i < max_entries; i++) + values[i] = value; + + batch_opts.cpu = cpu; + err =bpf_map_update_batch (map_fd, keys, values, &max_entries, &batch_opts); +if (!ASSERT_OK (err,"bpf_map_update_batch current cpu" )) + goto out; + +/* lookup values on current CPU */ + batch_opts.cpu = cpu; + batch_opts.elem_flags =BPF_F_CPU ; +memset (values,0 , max_entries * value_sz); + err =bpf_map_lookup_batch (map_fd,NULL , &batch, keys, values, &count, &batch_opts); +if (!ASSERT_TRUE (!err || err == -ENOENT ,"bpf_map_lookup_batch current cpu" )) + goto out; + +for (i =0 ; i < max_entries; i++) +if (!ASSERT_EQ (values[i], value,"value on current cpu" )) + goto out; + +/* lookup values on all CPUs */ + batch_opts.cpu =0 ; + batch_opts.elem_flags =0 ; +memset (values,0 , max_entries * value_size); + err =bpf_map_lookup_batch (map_fd,NULL , &batch, keys, values, &count, &batch_opts); +if (!ASSERT_TRUE (!err || err == -ENOENT ,"bpf_map_lookup_batch all cpus" )) + goto out; + +for (i =0 ; i < max_entries; i++) { +for (j =0 ; j < nr_cpus; j++) { +if (!ASSERT_EQ (values[i*nr_cpus + j], j != cpu ?0 : value, +"value on cpu" )) + goto out; + } + } + } + + update_opts.cpu = nr_cpus; + err =bpf_map_update_elem_opts (map_fd, keys, values, &update_opts); +if (!ASSERT_EQ (err, -E2BIG ,"bpf_map_update_elem_opts -E2BIG" )) + goto out; + + err =bpf_map__update_elem_opts (map, keys, key_sz, values, value_sz, + &update_opts); +if (!ASSERT_EQ (err, -E2BIG ,"bpf_map__update_elem_opts -E2BIG" )) + goto out; + + lookup_opts.cpu = nr_cpus; + err =bpf_map_lookup_elem_opts (map_fd, keys, values, &lookup_opts); +if (!ASSERT_EQ (err, -E2BIG ,"bpf_map_lookup_elem_opts -E2BIG" )) + goto out; + + err =bpf_map__lookup_elem_opts (map, keys, key_sz, values, value_sz, + &lookup_opts); +if (!ASSERT_EQ (err, -E2BIG ,"bpf_map__lookup_elem_opts -E2BIG" )) + goto out; + +/* clear value on all cpus */ + batch_opts.cpu =BPF_ALL_CPUS ; + batch_opts.elem_flags =BPF_F_CPU ; +memset (values,0 , max_entries * value_sz); + err =bpf_map_update_batch (map_fd, keys, values, &max_entries, &batch_opts); +if (!ASSERT_OK (err,"bpf_map_update_batch all cpus" )) + goto out; + +for (cpu =0 ; cpu < nr_cpus; cpu++) { +/* update value on current cpu */ + values[0 ] = value; + update_opts.cpu = cpu; +for (i =0 ; i < max_entries; i++) { + err =bpf_map__update_elem_opts (map, keys + i, + key_sz, values, + value_sz, &update_opts); +if (!ASSERT_OK (err,"bpf_map__update_elem_opts current cpu" )) + goto out; + +for (j =0 ; j < nr_cpus; j++) { +/* lookup then check value on CPUs */ + lookup_opts.cpu = j; + err =bpf_map__lookup_elem_opts (map, keys + i, + key_sz, values, + value_sz, + &lookup_opts); +if (!ASSERT_OK (err,"bpf_map__lookup_elem_opts current cpu" )) + goto out; +if (!ASSERT_EQ (values[0 ], j != cpu ?0 : value, +"bpf_map__lookup_elem_opts value on current cpu" )) + goto out; + } + } + +/* clear value on current cpu */ + values[0 ] =0 ; + err =bpf_map__update_elem_opts (map, keys, key_sz, values, + value_sz, &update_opts); +if (!ASSERT_OK (err,"bpf_map__update_elem_opts current cpu" )) + goto out; + } + +out: +if (keys) +free (keys); +if (values) +free (values); +percpu_array_flag__destroy (skel); +} +void test_percpu_alloc (void ) {if (test__start_subtest ("array" ))
@@ -125,4 +295,6 @@ void test_percpu_alloc(void)test_cgrp_local_storage ();if (test__start_subtest ("failure_tests" ))test_failure (); +if (test__start_subtest ("cpu_flag_tests" )) +test_cpu_flag (); }
tools/testing/selftests/bpf/progs/percpu_array_flag.c new file +24 -0
@@ -0,0 +1,24 @@ +// SPDX-License-Identifier: GPL-2.0 +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_tracing.h> + +struct { + __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); + __uint(max_entries, 2); + __type(key, int); + __type(value, u64); +} percpu SEC(".maps"); + +SEC("fentry/bpf_fentry_test1") +int BPF_PROG (test_percpu_array ,int x) +{ +u64 value =0xDEADC0DE ; +int key =0 ; + +bpf_map_update_elem (&percpu, &key, &value,BPF_ANY ); +return 0 ; +} + +char _license[] SEC("license") = "GPL"; +
Signature · 1 line
2.50.1
Quoted 56 lines · Leon Hwang
This patch introduces support for the
BPF_F_CPUflag inpercpu_arraymaps to allow updating or looking up values for specified CPUs or for all CPUs with a single value.This enhancement enables:
* Efficient update of all CPUs using a single value when cpu == (u32)~0.
* Targeted update or lookup for a specified CPU otherwise.The flag is passed via:
*
map_flagsinbpf_percpu_array_update()along with embedded cpu field.
*elem_flagsingeneric_map_update_batch()along with separated cpu field.Signed-off-by Leon Hwang <[email protected]>--- include/linux/bpf.h | 3 +- include/uapi/linux/bpf.h | 7 +++++ kernel/bpf/arraymap.c | 54 ++++++++++++++++++++++++++-------- kernel/bpf/syscall.c | 52 ++++++++++++++++++++------------ tools/include/uapi/linux/bpf.h | 7 +++++ 5 files changed, 90 insertions(+), 33 deletions(-)include/linux/bpf.h +2
-1@@ -2671,7 +2671,8 @@ int map_set_for_each_callback_args(struct bpf_verifier_env *env, struct bpf_func_state *callee);int bpf_percpu_hash_copy (struct bpf_map *map,void *key,void *value); -int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value); +int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value, +u64 flags);int bpf_percpu_hash_update (struct bpf_map *map,void *key,void *value,u64 flags); int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value,include/uapi/linux/bpf.h +6
-0@@ -1372,6 +1372,12 @@ enum {BPF_NOEXIST =1 ,/* create new element if it didn't exist */ BPF_EXIST =2 ,/* update existing element */ BPF_F_LOCK =4 ,/* spin_lock-ed map_lookup/map_update */ + BPF_F_CPU = 8, /* map_update for percpu_array */ +}; + +enum { +/* indicate updating value across all CPUs for percpu maps. */ +BPF_ALL_CPUS = (__u32 )~0 , };/* flags for BPF_MAP_CREATE command */
include/uapi/linux/bpf.h union bpf_attr { · line 1559
Quoted 5 lines
include/uapi/linux/bpf.h +1
-0@@ -1549,6 +1555,7 @@ union bpf_attr {__u32 map_fd;__u64 elem_flags;__u64 flags; +__u32 cpu; } batch;
So you use flags to pass cpu for singular lookup/delete operations, but separate cpu field for batch. We need to be consistent here. I think I initially suggested a separate cpu field, but given how much churn it's causing in API and usage, I guess I'm leaning towards just passing it through flags.
But if someone else has strong preferences, I can be convinced otherwise.
Either way, it has to be consistent between batched and non-batched API.
Other than that and minor formatting needs below, LGTM.
pw-bot: cr
Quoted 20 lines
include/uapi/linux/bpf.h +0
-0struct {/* anonymous struct used by BPF_PROG_LOAD command */ kernel/bpf/arraymap.c +3
-3@@ -295,17 +295,24 @@ static void *percpu_array_map_lookup_percpu_elem(struct bpf_map *map, void *key,return per_cpu_ptr (array->pptrs[index & array->index_mask], cpu); } -int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value) +int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value,u64 flags) { struct bpf_array *array = container_of(map, struct bpf_array, map);u32 index = *(u32 *)key;void __percpu *pptr; -int cpu, off =0 ; -u32 size; +u32 size, cpu; +int off =0 ;
kernel/bpf/arraymap.c static void *percpu_array_map_lookup_percpu_elem(struct bpf_map *map, void *key, · line 314
Quoted 10 lines
kernel/bpf/arraymap.c +6
-0if (unlikely (index >= array->map.max_entries))return -ENOENT ; + cpu = (u32)(flags >>32 ); + flags &= (u32 )~0 ; +if (unlikely (flags >BPF_F_CPU )) +return -EINVAL ; +if (unlikely ((flags &BPF_F_CPU ) && cpu >=num_possible_cpus ())) +return -E2BIG ;
nit: I'd probably do -ERANGE for this one
Quoted 4 lines
kernel/bpf/arraymap.c +1
-0+/* per_cpu areas are zero-filled and bpf programs can only * access 'value_size' of them, so copying rounded areas * will not leak any kernel data
[...]
Quoted 13 lines
kernel/bpf/arraymap.c +3
-2@@ -1941,19 +1945,25 @@ int generic_map_update_batch(struct bpf_map *map, struct file *map_file, {void __user *values =u64_to_user_ptr (attr->batch.values);void __user *keys =u64_to_user_ptr (attr->batch.keys); -u32 value_size, cp, max_count; +u32 value_size, cp, max_count, cpu = attr->batch.cpu; +u64 elem_flags = attr->batch.elem_flags;void *key, *value;int err =0 ; -if (attr->batch.elem_flags & ~BPF_F_LOCK ) +if (elem_flags & ~(BPF_F_LOCK |BPF_F_CPU ))return -EINVAL ;
kernel/bpf/arraymap.c int generic_map_update_batch(struct bpf_map *map, struct file *map_file, · line 1962
Quoted 10 lines
kernel/bpf/arraymap.c +3
-2-if ((attr->batch.elem_flags &BPF_F_LOCK ) && +if ((elem_flags &BPF_F_LOCK ) && !btf_record_has_field (map->record,BPF_SPIN_LOCK )) {return -EINVAL ; } - value_size =bpf_map_value_size (map); +if ((elem_flags &BPF_F_CPU ) && + map->map_type !=BPF_MAP_TYPE_PERCPU_ARRAY )
nit: keep on the single line
Quoted 7 lines
kernel/bpf/arraymap.c +4
-0+return -EINVAL ; + + value_size =bpf_map_value_size (map, elem_flags); + elem_flags |= ((u64 )cpu) <<32 ; max_count = attr->batch.count;if (!max_count)
[...]
kernel/bpf/arraymap.c
Quoted 9 lines
kernel/bpf/arraymap.c +4
-2-if ((attr->batch.elem_flags &BPF_F_LOCK ) && +if ((elem_flags &BPF_F_LOCK ) && !btf_record_has_field (map->record,BPF_SPIN_LOCK ))return -EINVAL ; - value_size =bpf_map_value_size (map); +if ((elem_flags &BPF_F_CPU ) && + map->map_type !=BPF_MAP_TYPE_PERCPU_ARRAY ) +return -EINVAL ;
same, formatting is off, but best to keep it single line
Quoted 6 lines
kernel/bpf/arraymap.c +3
-0+ + value_size =bpf_map_value_size (map, elem_flags); + elem_flags |= ((u64 )cpu) <<32 ; max_count = attr->batch.count;if (!max_count)
[...]
Re: [PATCH bpf-next 2/3] bpf, libbpf: Support BPF_F_CPU for percpu_array map
In reply to Leon HwangQuoted 9 lines · Leon Hwang
This patch adds libbpf support for the
BPF_F_CPUflag inpercpu_arraymaps, introducing the following APIs:1.
bpf_map_update_elem_opts(): update with structbpf_map_update_elem_opts
2.bpf_map_lookup_elem_opts(): lookup with structbpf_map_lookup_elem_opts
3.bpf_map__update_elem_opts(): high-level wrapper with input validation
4.bpf_map__lookup_elem_opts(): high-level wrapper with input validation
Quoted 8 lines
Behavior:
* If
opts->cpu== (u32)~0, the update is applied to all CPUs.
* Otherwise, it applies only to the specified CPU.
* Lookup APIs retrieve values from the target CPU whenBPF_F_CPUis used.Signed-off-by Leon Hwang <[email protected]>---
please use "libbpf: " prefix, not "bpf,libbpf:"
Then see my comment on flags vs separate field for cpu passing. If we go with just using flags, then I'd probably drop all the new libbpf APIs, because we already have bpf_map_lookup_elem_flags() and bpf_map_update_elem() (the latter accepts flags), so as far as low-level API we are good.
The comment describing the new BPF_F_CPU flag is good, so let's add it, but place it into bpf_map__lookup_elem() description (which, btw, also accepts flags, so no changes to API itself is necessary). Same for bpf_map__update_elem().
validate_map_op() logic will stay, but just will extract cpu from flags, right?
So overall less API churn, but same possibilities for users (plus we get better documentation, which is always nice).
Quoted 8 lines
tools/lib/bpf/bpf.c | 23 ++++++++++++++ tools/lib/bpf/bpf.h | 36 +++++++++++++++++++++- tools/lib/bpf/libbpf.c | 56 +++++++++++++++++++++++++++++++---- tools/lib/bpf/libbpf.h | 53 ++++++++++++++++++++++++++++----- tools/lib/bpf/libbpf.map | 5 ++++ tools/lib/bpf/libbpf_common.h | 14 +++++++++ 6 files changed, 173 insertions(+), 14 deletions(-)
[...]
Quoted 69 lines · Andrii Nakryiko
Quoted 56 lines · Leon Hwang
This patch introduces support for the
BPF_F_CPUflag inpercpu_arraymaps to allow updating or looking up values for specified CPUs or for all CPUs with a single value.This enhancement enables:
* Efficient update of all CPUs using a single value when cpu == (u32)~0.
* Targeted update or lookup for a specified CPU otherwise.The flag is passed via:
*
map_flagsinbpf_percpu_array_update()along with embedded cpu field.
*elem_flagsingeneric_map_update_batch()along with separated cpu field.Signed-off-by Leon Hwang <[email protected]>--- include/linux/bpf.h | 3 +- include/uapi/linux/bpf.h | 7 +++++ kernel/bpf/arraymap.c | 54 ++++++++++++++++++++++++++-------- kernel/bpf/syscall.c | 52 ++++++++++++++++++++------------ tools/include/uapi/linux/bpf.h | 7 +++++ 5 files changed, 90 insertions(+), 33 deletions(-)include/linux/bpf.h +2
-1@@ -2671,7 +2671,8 @@ int map_set_for_each_callback_args(struct bpf_verifier_env *env, struct bpf_func_state *callee);int bpf_percpu_hash_copy (struct bpf_map *map,void *key,void *value); -int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value); +int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value, +u64 flags);int bpf_percpu_hash_update (struct bpf_map *map,void *key,void *value,u64 flags); int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value,include/uapi/linux/bpf.h +6
-0@@ -1372,6 +1372,12 @@ enum {BPF_NOEXIST =1 ,/* create new element if it didn't exist */ BPF_EXIST =2 ,/* update existing element */ BPF_F_LOCK =4 ,/* spin_lock-ed map_lookup/map_update */ + BPF_F_CPU = 8, /* map_update for percpu_array */ +}; + +enum { +/* indicate updating value across all CPUs for percpu maps. */ +BPF_ALL_CPUS = (__u32 )~0 , };/* flags for BPF_MAP_CREATE command */ include/uapi/linux/bpf.h union bpf_attr { · line 1559
Quoted 5 lines
include/uapi/linux/bpf.h +1
-0@@ -1549,6 +1555,7 @@ union bpf_attr {__u32 map_fd;__u64 elem_flags;__u64 flags; +__u32 cpu; } batch;So you use flags to pass cpu for singular lookup/delete operations, but separate cpu field for batch. We need to be consistent here. I think I initially suggested a separate cpu field, but given how much churn it's causing in API and usage, I guess I'm leaning towards just passing it through flags.
Quoted 2 lines
But if someone else has strong preferences, I can be convinced otherwise.
Sure, they should be consistent.
Passing the CPU info through flags was actually my original idea, but I wasn’t sure how to convince you initially.
Let’s go ahead and pass the CPU through flags to minimize churn.
Quoted 30 lines
Either way, it has to be consistent between batched and non-batched API.
Other than that and minor formatting needs below, LGTM.
pw-bot: cr
Quoted 24 lines
include/uapi/linux/bpf.h +0
-0struct {/* anonymous struct used by BPF_PROG_LOAD command */ kernel/bpf/arraymap.c +3
-3@@ -295,17 +295,24 @@ static void *percpu_array_map_lookup_percpu_elem(struct bpf_map *map, void *key,return per_cpu_ptr (array->pptrs[index & array->index_mask], cpu); } -int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value) +int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value,u64 flags) { struct bpf_array *array = container_of(map, struct bpf_array, map);u32 index = *(u32 *)key;void __percpu *pptr; -int cpu, off =0 ; -u32 size; +u32 size, cpu; +int off =0 ;if (unlikely (index >= array->map.max_entries))return -ENOENT ;
Quoted 9 lines
kernel/bpf/arraymap.c static void *percpu_array_map_lookup_percpu_elem(struct bpf_map *map, void *key, · line 314
Quoted 6 lines
kernel/bpf/arraymap.c +6
-0+ cpu = (u32)(flags >>32 ); + flags &= (u32 )~0 ; +if (unlikely (flags >BPF_F_CPU )) +return -EINVAL ; +if (unlikely ((flags &BPF_F_CPU ) && cpu >=num_possible_cpus ())) +return -E2BIG ;nit: I'd probably do
-ERANGEfor this one
Ack.
Quoted 27 lines
Quoted 4 lines
kernel/bpf/arraymap.c +1
-0+/* per_cpu areas are zero-filled and bpf programs can only * access 'value_size' of them, so copying rounded areas * will not leak any kernel data [...]
Quoted 20 lines
kernel/bpf/arraymap.c +4
-3@@ -1941,19 +1945,25 @@ int generic_map_update_batch(struct bpf_map *map, struct file *map_file, {void __user *values =u64_to_user_ptr (attr->batch.values);void __user *keys =u64_to_user_ptr (attr->batch.keys); -u32 value_size, cp, max_count; +u32 value_size, cp, max_count, cpu = attr->batch.cpu; +u64 elem_flags = attr->batch.elem_flags;void *key, *value;int err =0 ; -if (attr->batch.elem_flags & ~BPF_F_LOCK ) +if (elem_flags & ~(BPF_F_LOCK |BPF_F_CPU ))return -EINVAL ; -if ((attr->batch.elem_flags &BPF_F_LOCK ) && +if ((elem_flags &BPF_F_LOCK ) && !btf_record_has_field (map->record,BPF_SPIN_LOCK )) {return -EINVAL ; }
Quoted 6 lines
kernel/bpf/arraymap.c int generic_map_update_batch(struct bpf_map *map, struct file *map_file, · line 1962
Quoted 3 lines
kernel/bpf/arraymap.c +2
-1- value_size =bpf_map_value_size (map); +if ((elem_flags &BPF_F_CPU ) && + map->map_type !=BPF_MAP_TYPE_PERCPU_ARRAY )nit: keep on the single line
Ack.
Btw, how many chars of one line do we allow? 100 chars?
Quoted 15 lines
Quoted 7 lines
kernel/bpf/arraymap.c +4
-0+return -EINVAL ; + + value_size =bpf_map_value_size (map, elem_flags); + elem_flags |= ((u64 )cpu) <<32 ; max_count = attr->batch.count;if (!max_count)[...]
Quoted 5 lines
kernel/bpf/arraymap.c +1
-1-if ((attr->batch.elem_flags &BPF_F_LOCK ) && +if ((elem_flags &BPF_F_LOCK ) && !btf_record_has_field (map->record,BPF_SPIN_LOCK ))return -EINVAL ;
Quoted 7 lines
kernel/bpf/arraymap.c
Quoted 4 lines
kernel/bpf/arraymap.c +3
-1- value_size =bpf_map_value_size (map); +if ((elem_flags &BPF_F_CPU ) && + map->map_type !=BPF_MAP_TYPE_PERCPU_ARRAY ) +return -EINVAL ;same, formatting is off, but best to keep it single line
Ack.
[...]
Thanks,
Leon
Eduard Zingerman
maintainerOn Tue, 2025-07-22 at 01:08 +0800, Leon Hwang wrote:
[...]
Quoted 27 lines
Quoted 20 lines
Quoted 20 lines
kernel/bpf/syscall.c +4
-3@@ -1941,19 +1945,25 @@ int generic_map_update_batch(struct bpf_map *map, struct file *map_file, {void __user *values =u64_to_user_ptr (attr->batch.values);void __user *keys =u64_to_user_ptr (attr->batch.keys); -u32 value_size, cp, max_count; +u32 value_size, cp, max_count, cpu = attr->batch.cpu; +u64 elem_flags = attr->batch.elem_flags;void *key, *value;int err =0 ; -if (attr->batch.elem_flags & ~BPF_F_LOCK ) +if (elem_flags & ~(BPF_F_LOCK |BPF_F_CPU ))return -EINVAL ; -if ((attr->batch.elem_flags &BPF_F_LOCK ) && +if ((elem_flags &BPF_F_LOCK ) && !btf_record_has_field (map->record,BPF_SPIN_LOCK )) {return -EINVAL ; }Quoted 6 lines
kernel/bpf/syscall.c int generic_map_update_batch(struct bpf_map *map, struct file *map_file, · line 1962
Quoted 3 lines
kernel/bpf/syscall.c +2
-1- value_size =bpf_map_value_size (map); +if ((elem_flags &BPF_F_CPU ) && + map->map_type !=BPF_MAP_TYPE_PERCPU_ARRAY )nit: keep on the single line
Quoted 3 lines
Ack.
Btw, how many chars of one line do we allow? 100 chars?
Yes, 100 chars per line (Andrii is on PTO).
Re: [PATCH bpf-next 2/3] bpf, libbpf: Support BPF_F_CPU for percpu_array map
In reply to Andrii NakryikoQuoted 12 lines · Andrii Nakryiko
Quoted 11 lines · Leon Hwang
This patch adds libbpf support for the
BPF_F_CPUflag inpercpu_arraymaps, introducing the following APIs:1.
bpf_map_update_elem_opts(): update with structbpf_map_update_elem_opts
2.bpf_map_lookup_elem_opts(): lookup with structbpf_map_lookup_elem_opts
3.bpf_map__update_elem_opts(): high-level wrapper with input validation
4.bpf_map__lookup_elem_opts(): high-level wrapper with input validationBehavior:
Quoted 9 lines
Quoted 6 lines
* If
opts->cpu== (u32)~0, the update is applied to all CPUs.
* Otherwise, it applies only to the specified CPU.
* Lookup APIs retrieve values from the target CPU whenBPF_F_CPUis used.Signed-off-by Leon Hwang <[email protected]>---
please use "libbpf: " prefix, not "bpf,libbpf:"
Ack.
Quoted 7 lines
Then see my comment on flags vs separate field for cpu passing. If we go with just using flags, then I'd probably drop all the new libbpf APIs, because we already have
bpf_map_lookup_elem_flags()andbpf_map_update_elem()(the latter accepts flags), so as far as low-level API we are good.
All the new libbpf APIs are unnecessary when pass the cpu through flags.
Quoted 5 lines
The comment describing the new
BPF_F_CPUflag is good, so let's add it, but place it intobpf_map__lookup_elem()description (which, btw, also accepts flags, so no changes to API itself is necessary). Same forbpf_map__update_elem().
No problem.
Quoted 2 lines
validate_map_op()logic will stay, but just will extract cpu from flags, right?
Yes, it is.
Quoted 3 lines
So overall less API churn, but same possibilities for users (plus we get better documentation, which is always nice).
When users want to use this feature, they should follow the libbpf API's documentation, no matter whether the cpu is separated or flags-embedded.
Thanks,
Leon
Quoted 10 lines
Quoted 8 lines
tools/lib/bpf/bpf.c | 23 ++++++++++++++ tools/lib/bpf/bpf.h | 36 +++++++++++++++++++++- tools/lib/bpf/libbpf.c | 56 +++++++++++++++++++++++++++++++---- tools/lib/bpf/libbpf.h | 53 ++++++++++++++++++++++++++++----- tools/lib/bpf/libbpf.map | 5 ++++ tools/lib/bpf/libbpf_common.h | 14 +++++++++ 6 files changed, 173 insertions(+), 14 deletions(-)[...]
Quoted 31 lines · Eduard Zingerman
On Tue, 2025-07-22 at 01:08 +0800, Leon Hwang wrote:
[...]
Quoted 27 lines
Quoted 26 lines
Quoted 13 lines
kernel/bpf/syscall.c +3
-2@@ -1941,19 +1945,25 @@ int generic_map_update_batch(struct bpf_map *map, struct file *map_file, {void __user *values =u64_to_user_ptr (attr->batch.values);void __user *keys =u64_to_user_ptr (attr->batch.keys); -u32 value_size, cp, max_count; +u32 value_size, cp, max_count, cpu = attr->batch.cpu; +u64 elem_flags = attr->batch.elem_flags;void *key, *value;int err =0 ; -if (attr->batch.elem_flags & ~BPF_F_LOCK ) +if (elem_flags & ~(BPF_F_LOCK |BPF_F_CPU ))return -EINVAL ;kernel/bpf/syscall.c int generic_map_update_batch(struct bpf_map *map, struct file *map_file, · line 1962
Quoted 10 lines
kernel/bpf/syscall.c +3
-2-if ((attr->batch.elem_flags &BPF_F_LOCK ) && +if ((elem_flags &BPF_F_LOCK ) && !btf_record_has_field (map->record,BPF_SPIN_LOCK )) {return -EINVAL ; } - value_size =bpf_map_value_size (map); +if ((elem_flags &BPF_F_CPU ) && + map->map_type !=BPF_MAP_TYPE_PERCPU_ARRAY )nit: keep on the single line
Quoted 5 lines
Quoted 3 lines
Ack.
Btw, how many chars of one line do we allow? 100 chars?
Yes, 100 chars per line (Andrii is on PTO).
Hi Eduard,
Thank you for your reply.
I’ll keep that in mind for the future.
Thanks,
Leon