Skip to content

bpf: Introduce BPF_F_CPU and BPF_F_ALL_CPUS flags for percpu maps

Leon Hwang bpf linux-kselftest lkml 13 versions – Applied

Status from this message: This series was applied to bpf/bpf-next.git (master)

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v13 Latest7 patches 0 repliesApplied 7/7
  2. v127 patches 6 repliesSuperseded
  3. v118 patches 3 repliesSuperseded
Earlier versions · 10
VersionPostedPatchesRepliesStatus
v108 patches2 repliesSuperseded
v97 patches5 repliesSuperseded
v87 patches5 repliesSuperseded
v77 patches21 repliesSuperseded
v67 patches2 repliesSuperseded
v59 patches4 repliesSuperseded
v47 patches10 repliesSuperseded
v36 patches10 repliesSuperseded
v23 patches6 repliesSuperseded
v13 patches6 repliesSuperseded

Changes in v13 from the cover letter

* No changes, rebased on latest tree.

Patches in v13 All 7 applied

#Subject and filesSizeRepliesApplied
1/7bpf: Introduce BPF_F_CPU and BPF_F_ALL_CPUS flagsinclude/linux/bpf.h include/uapi/linux/bpf.h kernel/bpf/syscall.c +1 more+43 -150 repliesApplied
2/7bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_array mapsinclude/linux/bpf.h kernel/bpf/arraymap.c kernel/bpf/syscall.c+31 -90 repliesApplied
3/7bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_hash and lru_percpu_hash mapsinclude/linux/bpf.h kernel/bpf/hashtab.c kernel/bpf/syscall.c+68 -320 repliesApplied
4/7bpf: Copy map value using copy_map_value_long for percpu_cgroup_storage mapskernel/bpf/local_storage.c+2 -40 repliesApplied
5/7bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_cgroup_storage mapsinclude/linux/bpf-cgroup.h include/linux/bpf.h kernel/bpf/local_storage.c +1 more+22 -80 repliesApplied
6/7libbpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu mapstools/lib/bpf/bpf.h tools/lib/bpf/libbpf.c tools/lib/bpf/libbpf.h+36 -190 repliesApplied
7/7selftests/bpf: Add cases to test BPF_F_CPU and BPF_F_ALL_CPUS flagstools/testing/selftests/bpf/prog_tests/percpu_alloc.c tools/testing/selftests/bpf/progs/percpu_alloc_array.c+360 -00 repliesApplied

Reviews trailers per patch

2 of 7 patches have a Reviewed-by or Acked-by.

PatchAndrii Nakryiko
1/7A-b
2/7
3/7
4/7
5/7
6/7A-b
7/7

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v13 thread

Cover letter [PATCH bpf-next v13 0/7] bpf: Introduce BPF_F_CPU and BPF_F_ALL_CPUS flags for percpu maps

This patch set introduces the BPF_F_CPU and BPF_F_ALL_CPUS flags for percpu maps, as the requirement of BPF_F_ALL_CPUS flag for percpu_array maps was discussed in the thread of
"[PATCH bpf-next v3 0/4] bpf: Introduce global percpu data"[1].

The goal of BPF_F_ALL_CPUS flag is to reduce data caching overhead in light skeletons by allowing a single value to be reused to update values across all CPUs. This avoids the M:N problem where M cached values are used to update a map on N CPUs kernel.

The BPF_F_CPU flag is accompanied by *flags*-embedded cpu info, which specifies the target CPU for the operation:

* For lookup operations: the flag field alongside cpu info enable querying a value on the specified CPU.
* For update operations: the flag field alongside cpu info enable updating value for specified CPU.

Links:
[1] https://lore.kernel.org/bpf/[email protected]/

Changes:
v12 -> v13:
* No changes, rebased on latest tree.

v11 -> v12:
* Dropped the v11 changes.
* Stabilized the lru_percpu_hash map test by keeping an extra spare entry, which can be used temporarily during updates to avoid unintended LRU evictions.

v10 -> v11:
* Support the combination of BPF_EXIST and BPF_F_CPU/BPF_F_ALL_CPUS for update operations.
* Fix unstable lru_percpu_hash map test using the combination of BPF_EXIST and BPF_F_CPU/BPF_F_ALL_CPUS to avoid LRU eviction (reported by Alexei).

v9 -> v10:
* Add tests to verify array and hash maps do not support BPF_F_CPU and BPF_F_ALL_CPUS flags.
* Address comment from Andrii:
* Copy map value using copy_map_value_long for percpu_cgroup_storage maps in a separate patch.

v8 -> v9:
* Change value type from u64 to u32 in selftests.
* Address comments from Andrii:
* Keep value_size unaligned and update everywhere for consistency when cpu flags are specified.
* Update value by getting pointer for percpu hash and percpu cgroup_storage maps.

v7 -> v8:
* Address comments from Andrii:
* Check BPF_F_LOCK when update percpu_array, percpu_hash and lru_percpu_hash maps.
* Refactor flags check in __htab_map_lookup_and_delete_batch().
* Keep value_size unaligned and copy value using copy_map_value() in __htab_map_lookup_and_delete_batch() when BPF_F_CPU is specified.
* Update warn message in libbpf's validate_map_op().
* Update comment of libbpf's bpf_map__lookup_elem().

v6 -> v7:
* Get correct value size for percpu_hash and lru_percpu_hash in update_batch API.
* Set 'count' as 'max_entries' in test cases for lookup_batch API.
* Address comment from Alexei:
* Move cpu flags check into bpf_map_check_op_flags().

v5 -> v6:
* Move bpf_map_check_op_flags() from 'bpf.h' to 'syscall.c'.
* Address comments from Alexei:
* Drop the refactoring code of data copying logic for percpu maps.
* Drop bpf_map_check_op_flags() wrappers.

v4 -> v5:
* Address comments from Andrii:
* Refactor data copying logic for all percpu maps.
* Drop this_cpu_ptr() micro-optimization.
* Drop cpu check in libbpf's validate_map_op().
* Enhance bpf_map_check_op_flags() using *allowed flags* instead of

    'extra_flags_mask'.

v3 -> v4:
* Address comments from Andrii:
* Remove unnecessary map_type check in bpf_map_value_size().
* Reduce code churn.
* Remove unnecessary do_delete check in

    __htab_map_lookup_and_delete_batch().
  * Introduce bpf_percpu_copy_to_user() and bpf_percpu_copy_from_user().
  * Rename check_map_flags() to bpf_map_check_op_flags() with
    extra_flags_mask.
  * Add human-readable pr_warn() explanations in validate_map_op().
  * Use flags in bpf_map__delete_elem() and
    bpf_map__lookup_and_delete_elem().

* Drop "for alignment reasons".
v3 link: https://lore.kernel.org/bpf/[email protected]/

v2 -> v3:
* Address comments from Alexei:
* Use BPF_F_ALL_CPUS instead of BPF_ALL_CPUS magic.
* Introduce these two cpu flags for all percpu maps.
* Address comments from Jiri:
* Reduce some unnecessary u32 cast.
* Refactor more generic map flags check function.
* A code style issue.
v2 link: https://lore.kernel.org/bpf/[email protected]/

v1 -> v2:
* Address comments from Andrii:
* Embed cpu info as high 32 bits of *flags* totally.
* Use ERANGE instead of E2BIG.
* Few format issues.

Leon Hwang (7):
bpf: Introduce BPF_F_CPU and BPF_F_ALL_CPUS flags
bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_array
maps
bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_hash
and lru_percpu_hash maps
bpf: Copy map value using copy_map_value_long for
percpu_cgroup_storage maps
bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for
percpu_cgroup_storage maps
libbpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu maps selftests/bpf: Add cases to test BPF_F_CPU and BPF_F_ALL_CPUS flags

 include/linux/bpf-cgroup.h                    |   4 +-
 include/linux/bpf.h                           |  35 +-
 include/uapi/linux/bpf.h                      |   2 +
 kernel/bpf/arraymap.c                         |  29 +-
 kernel/bpf/hashtab.c                          |  94 +++--
 kernel/bpf/local_storage.c                    |  27 +-
 kernel/bpf/syscall.c                          |  37 +-
 tools/include/uapi/linux/bpf.h                |   2 +
 tools/lib/bpf/bpf.h                           |   8 +
 tools/lib/bpf/libbpf.c                        |  26 +-
 tools/lib/bpf/libbpf.h                        |  21 +-
 .../selftests/bpf/prog_tests/percpu_alloc.c   | 328 ++++++++++++++++++
 .../selftests/bpf/progs/percpu_alloc_array.c  |  32 ++
 13 files changed, 560 insertions(+), 85 deletions(-)

--
2.52.0