Skip to content

bpf: Introduce global percpu data

Leon Hwang bpf linux-kselftest lkml sashiko-reviews 12 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v12 Latest10 patches 14 repliesIn review
  2. v1110 patches 7 repliesSuperseded
  3. v109 patches 22 repliesSuperseded
Earlier versions · 9
VersionPostedPatchesRepliesStatus
v99 patches3 repliesSuperseded
v89 patches13 repliesSuperseded
v711 patches24 repliesSuperseded
v612 patches8 repliesSuperseded
v59 patches8 repliesSuperseded
v48 patches19 repliesSuperseded
v34 patches19 repliesSuperseded
v24 patches11 repliesSuperseded
v14 patches0 repliesSuperseded

Changes in v12 from the cover letter

* Improve feature check in bpf_object__create_maps() in libbpf.
* Add percpu_array map support in bpf_map__set_value_size() in libbpf.
* Exercise bpf_map__set_value_size() in selftest.
* Drop dead warning in bpf_object__populate_internal_map() in libbpf. (Sashiko)
* v11: https://lore.kernel.org/bpf/[email protected]/

Patches in v12 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/10bpf: Drop duplicate blank lines in kernel/bpf/kernel/bpf/backtrack.c kernel/bpf/btf.c kernel/bpf/cfg.c +7 more+0 -301 reply
2/10bpf: Factor out check_map_mem_read helper in verifierkernel/bpf/verifier.c+43 -321 reply
3/10bpf: Introduce global percpu datakernel/bpf/arraymap.c kernel/bpf/const_fold.c kernel/bpf/fixups.c +1 more+84 -30 replies
4/10libbpf: Probe percpu data featuretools/lib/bpf/features.c tools/lib/bpf/libbpf_internal.h+37 -01 reply
5/10libbpf: Add support for global percpu datatools/lib/bpf/bpf_gen_internal.h tools/lib/bpf/gen_loader.c tools/lib/bpf/libbpf.c+72 -153 replies
6/10bpftool: Generate skeleton for global percpu datatools/bpf/bpftool/gen.c tools/lib/bpf/skel_internal.h+63 -144 replies
7/10selftests/bpf: Add tests to verify global percpu datatools/testing/selftests/bpf/Makefile tools/testing/selftests/bpf/prog_tests/global_data_init.c tools/testing/selftests/bpf/progs/test_global_percpu_data.c+230 -12 replies
8/10selftests/bpf: Test direct reading/writing read-only percpu_array maptools/testing/selftests/bpf/prog_tests/global_data_init.c+90 -00 replies
9/10selftests/bpf: Test verifier log for global percpu datatools/testing/selftests/bpf/prog_tests/global_data_init.c tools/testing/selftests/bpf/progs/test_global_percpu_data.c+29 -00 replies
10/10selftests/bpf: Verify bpf_iter for global percpu datatools/testing/selftests/bpf/prog_tests/global_data_init.c tools/testing/selftests/bpf/progs/test_global_percpu_data.c+77 -02 replies

Reviews trailers per patch

7 of 10 patches have a Reviewed-by or Acked-by.

PatchEmil TsalapatisEduard ZingermanAndrii NakryikoQuentin Monnet
1/10R-b
2/10A-b
3/10A-b
4/10R-b A-b
5/10
6/10A-b
7/10
8/10
9/10R-b
10/10R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 4 reviewers.

Read the v12 thread

Cover letter [PATCH bpf-next v12 00/10] bpf: Introduce global percpu data

This patch set introduces global percpu data, similar to commit 6316f78306c1 ("Merge branch 'support-global-data'"), to reduce restrictions in C for BPF programs.

With this enhancement, it becomes possible to define and use global percpu variables, like the DEFINE_PER_CPU() macro in the kernel include/linux/percpu-defs.h.

The section name for global peurcpu data is ".percpu". Even though, a one-byte percpu variable (e.g., char run SEC(".percpu") = 0;) can trigger a crash with Clang 17 [1], users are expected to use such small variables as global percpu data with newer Clang versions, which don't have the issue.

The idea stems from the bpfsnoop [2], which itself was inspired by retsnoop [3]. During testing of bpfsnoop on the v6.6 kernel, two LBR (Last Branch Record) entries were observed related to the bpf_get_smp_processor_id() helper.

Since commit 1ae6921009e5 ("bpf: inline bpf_get_smp_processor_id() helper"), the bpf_get_smp_processor_id() helper has been inlined on x86_64, reducing the overhead and consequently minimizing these two LBR records.

However, the introduction of global percpu data offers a more robust solution. By leveraging the percpu_array map and percpu instruction, global percpu data can be implemented intrinsically.

This feature also facilitates sharing percpu information between tail callers and callees or between freplace callers and callees through a shared global percpu variable. Previously, this was achieved using a 1-entry percpu_array map, which this patch set aims to improve upon.

Links:
[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://github.com/bpfsnoop/bpfsnoop
[3] https://github.com/anakryiko/retsnoop

Changes:
v11 -> v12:
* Improve feature check in bpf_object__create_maps() in libbpf.
* Add percpu_array map support in bpf_map__set_value_size() in libbpf.
* Exercise bpf_map__set_value_size() in selftest.
* Drop dead warning in bpf_object__populate_internal_map() in libbpf. (Sashiko)
* v11: https://lore.kernel.org/bpf/[email protected]/

v10 -> v11:
* Drop env->prog->jit_requested check when inlining insns for global percpu data.
* Do not autocreate percpu_array map when kernel does not have global percpu data support in libbpf.
* Check map->btf_value_type_id in bpftool's is_skel_data().
* Exercise bpf_map__lookup_elem() in selftest.
* Collect Reviewed-by tags from Emil, thanks.
* Drop all duplicate blank lines in kernel/bpf/*.c. (Emil)
* Factor out check_map_mem_read() helper. (Emil)
* Check bpf_jit_supports_percpu_insn() first in percpu_array_map_direct_value_addr/meta(). (Emil)
* Add comment for 'map->libbpf_type == LIBBPF_MAP_PERCPU' in libbpf's map_is_mmapable(). (Emil)
* Init update_flags as a const var in libbpf's bpf_object__populate_internal_map(). (Emil)
* Keep is_mmapable_map() beyond is_skel_data() in bpftool. (Emil)
* Add 'run' and 'cpu_id' in selftest. (Emil)
* Drop subskel test. Verify the generated subskel manually. (Emil)
* Add comment to the raw insns in selftest. (Emil)
* v10: https://lore.kernel.org/bpf/[email protected]/

v9 -> v10:
* Rebase latest bpf-next tree to resolve code conflict in verifier in patch #1.
* v9: https://lore.kernel.org/bpf/[email protected]/

v8 -> v9:
* Use real name for percpu data maps in libbpf in patch #4.
* Add long map name test in patch #6.
* Move parse_cpu_mask_file() to test_percpu_data_on_cpus() in test in patch #6.
* Validate map type in get_map_ident() for percpu data maps in patch #5.
* Update code comment in verifier in patch #2. (per Andrii)
* Pass 'type' to internal_map_name in libbpf in patch #4. (per Andrii)
* Factor out the helper is_skel_data() in bpftool in patch #5. (per Quentin and Andrii)
* v8: https://lore.kernel.org/bpf/[email protected]/

v7 -> v8:
* Send patch #1 and #2 separately that fix interpreter fallback issues. (Andrii)
* Use 'array->elem_size' to avoid 'range' local variable in percpu_array_map_direct_value_meta(). (Andrii)
* Keep original map name for percpu data's map in libbpf. (Andrii)
* Factor out helper bpf_map_is_skel_data() in bpftool. (Andrii)
* Update commit message of direct access read-only percpu_array map. (Andrii)
* Add test to verify that it is disallowed to directly write data of read-only percpu_array map. (Andrii)
* Drop unused 'num_cpus' in test. (bot+bpf-ci)
* Factor out helper test_percpu_data_on_cpus() in test. (bot+bpf-ci)
* v7: https://lore.kernel.org/bpf/[email protected]/

v6 -> v7:
* Use tgt_endian() in bpf_gen__map_update_elem() in patch #6. (Sashiko)
* Use sizeof(args) in verifier_snprintf test in patch #10. (Sashiko)
* Drop xlated test of v6. (Alexei)
* v6: https://lore.kernel.org/bpf/[email protected]/

v5 -> v6:
* Prevent running user addr_space_cast and addr_percpu insns in interpreter. (Sashiko)
* Cast __percpu pointer to u64 with (__force unsigned long). (lkp)
* Exclude BPF_MAP_TYPE_PERCPU_ARRAY in check_mem_access() before calling bpf_map_direct_read(), and add a test to verify it.
(Sashiko, bot+bpf-ci)
* Skip percpu data variables for subskeleton in bpftool. (Sashiko)
* Protect skel->percpu using mprotect(..., PROT_READ) in light skeleton. (Sashiko, bot+bpf-ci)
* Drop roundup() in tests. (Sashiko)
* Call test_global_percpu_data_verifier_log() without test__start_subtest(). (Sashiko)
* Cast insn->imm to __u64 with (__u32) in xlated test. (Sashiko)
* Check cnt using the new idx in xlated test. (Sashiko)
* v5: https://lore.kernel.org/bpf/[email protected]/

v4 -> v5:
* Add prog->jit_requested check to prevent running percpu data in interpreter in patch #1.
* Factor out verifier log tests using its own patch.
* Address comments from Alexei:
* Move map_type check from check_mem_access() to bpf_map_direct_read() in patch #2.
* Move BPF_MAP_TYPE_INSN_ARRAY map_type check from const_reg_xfer() to bpf_map_direct_read() in patch #2.
* Add a test to verify that the off of xlated ldimm64 insn matches the off encoded in the ELF ldimm64 insn.
* Drop patch #5 of v4.
* Address reviews from Sashiko:
* Update commit message of patch #6 to indicate that maps.percpu->mmaped has been marked as read-only in libbpf.
* Lookup elem on specified CPU using BPF_F_CPU in tests.
* Drop unnecessary err == -EOPNOTSUPP in test.
* Locate target field using its offset in the iter test.
* v4: https://lore.kernel.org/bpf/[email protected]/

v3 -> v4:
* Drop duplicate blank lines in verifier.
* Add percpu data feature probe in libbpf.
* Update percpu_array map using BPF_F_ALL_CPUS flag for lskel, if no cpu flag is set.
* Add two tests to verify verifier log.
* Add a test to verify mov64_percpu_reg instruction.
* Add a test to verify bpf_iter for percpu data map.
* Update percpu_array map using BPF_F_ALL_CPUS flag in libbpf (per Alexei and Andrii).
* Address comments from Andrii:
* Use .percpu as section identifier.
* Use bpf_jit_supports_percpu_insn() instead of CONFIG_SMP.
* Drop bpf_map__is_internal_percpu() API.
* Drop unnecessary __aligned(8) in libbpf, verified by selftest.
* Make mmap data read-only after loading prog.
v3: https://lore.kernel.org/bpf/[email protected]/

v2 -> v3:
* Use ".data..percpu" as PERCPU_DATA_SEC.
* Address comment from Alexei:
* Add u8, array of ints and struct { .. } vars to selftest.
v2: https://lore.kernel.org/bpf/[email protected]/

v1 -> v2:
* Address comments from Andrii:
* Use LIBBPF_MAP_PERCPU and SEC_PERCPU.
* Reuse mmaped of libbpf's struct bpf_map for .percpu map data.
* Set .percpu struct pointer to NULL after loading skeleton.
* Make sure value size of .percpu map is __aligned(8).
* Use raw_tp and opts.cpu to test global percpu variables on all CPUs.
* Address comments from Alexei:
* Test non-zero offset of global percpu variable.
* Test case about BPF_PSEUDO_MAP_IDX_VALUE.
v1: https://lore.kernel.org/bpf/[email protected]/

rfc -> v1:
* Address comments from Andrii:
* Keep one image of global percpu variable for all CPUs.
* Reject non-ARRAY map in bpf_map_direct_read(), check_reg_const_str(), and check_bpf_snprintf_call() in verifier.
* Split out libbpf changes from kernel-side changes.
* Use ".percpu" as PERCPU_DATA_SEC.
* Use enum libbpf_map_type to distinguish BSS, DATA, RODATA and PERCPU_DATA.
* Avoid using errno for checking err from libbpf_num_possible_cpus().
* Use "map '%s': " prefix for error message.
rfc: https://lore.kernel.org/bpf/[email protected]/

Leon Hwang (10):
bpf: Drop duplicate blank lines in kernel/bpf/
bpf: Factor out check_map_mem_read helper in verifier
bpf: Introduce global percpu data
libbpf: Probe percpu data feature
libbpf: Add support for global percpu data
bpftool: Generate skeleton for global percpu data selftests/bpf: Add tests to verify global percpu data selftests/bpf: Test direct reading/writing read-only percpu_array map selftests/bpf: Test verifier log for global percpu data selftests/bpf: Verify bpf_iter for global percpu data

 kernel/bpf/arraymap.c                         |  38 +-
 kernel/bpf/backtrack.c                        |   2 -
 kernel/bpf/btf.c                              |   1 -
 kernel/bpf/cfg.c                              |   1 -
 kernel/bpf/const_fold.c                       |   1 -
 kernel/bpf/fixups.c                           |  38 +-
 kernel/bpf/hashtab.c                          |   2 -
 kernel/bpf/helpers.c                          |   1 -
 kernel/bpf/liveness.c                         |   2 -
 kernel/bpf/queue_stack_maps.c                 |   1 -
 kernel/bpf/syscall.c                          |   5 -
 kernel/bpf/verifier.c                         | 100 +++---
 tools/bpf/bpftool/gen.c                       |  53 ++-
 tools/lib/bpf/bpf_gen_internal.h              |   3 +-
 tools/lib/bpf/features.c                      |  35 ++
 tools/lib/bpf/gen_loader.c                    |   3 +-
 tools/lib/bpf/libbpf.c                        |  81 ++++-
 tools/lib/bpf/libbpf_internal.h               |   2 +
 tools/lib/bpf/skel_internal.h                 |  24 +-
 tools/testing/selftests/bpf/Makefile          |   2 +-
 .../bpf/prog_tests/global_data_init.c         | 336 ++++++++++++++++++
 .../bpf/progs/test_global_percpu_data.c       |  89 +++++
 22 files changed, 725 insertions(+), 95 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/progs/test_global_percpu_data.c

--
2.55.0