bpf: Introduce global percpu data
Subsystems (6)
v7 cover bpf: Introduce global percpu data
This patch set introduces global percpu data, similar to commit 6316f78306c1 ("Merge branch 'support-global-data'"), to reduce restrictions in C for BPF programs.
With this enhancement, it becomes possible to define and use global percpu variables, like the DEFINE_PER_CPU() macro in the kernel include/linux/percpu-defs.h.
The section name for global peurcpu data is ".percpu". Even though, a one-byte percpu variable (e.g., char run SEC(".percpu") = 0;) can trigger a crash with Clang 17 [1], users are expected to use such small variables as global percpu data with newer Clang versions, which don't have the issue.
The idea stems from the bpfsnoop [2], which itself was inspired by retsnoop [3]. During testing of bpfsnoop on the v6.6 kernel, two LBR (Last Branch Record) entries were observed related to the bpf_get_smp_processor_id() helper.
Since commit 1ae6921009e5 ("bpf: inline bpf_get_smp_processor_id() helper"), the bpf_get_smp_processor_id() helper has been inlined on x86_64, reducing the overhead and consequently minimizing these two LBR records.
However, the introduction of global percpu data offers a more robust solution. By leveraging the percpu_array map and percpu instruction, global percpu data can be implemented intrinsically.
This feature also facilitates sharing percpu information between tail callers and callees or between freplace callers and callees through a shared global percpu variable. Previously, this was achieved using a 1-entry percpu_array map, which this patch set aims to improve upon.
Links:
[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://github.com/bpfsnoop/bpfsnoop
[3] https://github.com/anakryiko/retsnoop
Changes:
v6 -> v7:
* Use tgt_endian() in bpf_gen__map_update_elem() in patch #6. (Sashiko)
* Use sizeof(args) in verifier_snprintf test in patch #10. (Sashiko)
* Drop xlated test of v6. (Alexei)
* v6: https://lore.kernel.org/bpf/[email protected]/
v5 -> v6:
* Prevent running user addr_space_cast and addr_percpu insns in interpreter. (Sashiko)
* Cast __percpu pointer to u64 with (__force unsigned long). (lkp)
* Exclude BPF_MAP_TYPE_PERCPU_ARRAY in check_mem_access() before calling bpf_map_direct_read(), and add a test to verify it.
(Sashiko, bot+bpf-ci)
* Skip percpu data variables for subskeleton in bpftool. (Sashiko)
* Protect skel->percpu using mprotect(..., PROT_READ) in light skeleton. (Sashiko, bot+bpf-ci)
* Drop roundup() in tests. (Sashiko)
* Call test_global_percpu_data_verifier_log() without test__start_subtest(). (Sashiko)
* Cast insn->imm to __u64 with (__u32) in xlated test. (Sashiko)
* Check cnt using the new idx in xlated test. (Sashiko)
* v5: https://lore.kernel.org/bpf/[email protected]/
v4 -> v5:
* Add prog->jit_requested check to prevent running percpu data in interpreter in patch #1.
* Factor out verifier log tests using its own patch.
* Address comments from Alexei:
* Move map_type check from check_mem_access() to bpf_map_direct_read() in patch #2.
* Move BPF_MAP_TYPE_INSN_ARRAY map_type check from const_reg_xfer() to bpf_map_direct_read() in patch #2.
* Add a test to verify that the off of xlated ldimm64 insn matches the off encoded in the ELF ldimm64 insn.
* Drop patch #5 of v4.
* Address reviews from Sashiko:
* Update commit message of patch #6 to indicate that maps.percpu->mmaped has been marked as read-only in libbpf.
* Lookup elem on specified CPU using BPF_F_CPU in tests.
* Drop unnecessary err == -EOPNOTSUPP in test.
* Locate target field using its offset in the iter test.
* v4: https://lore.kernel.org/bpf/[email protected]/
v3 -> v4:
* Drop duplicate blank lines in verifier.
* Add percpu data feature probe in libbpf.
* Update percpu_array map using BPF_F_ALL_CPUS flag for lskel, if no cpu flag is set.
* Add two tests to verify verifier log.
* Add a test to verify mov64_percpu_reg instruction.
* Add a test to verify bpf_iter for percpu data map.
* Update percpu_array map using BPF_F_ALL_CPUS flag in libbpf (per Alexei and Andrii).
* Address comments from Andrii:
* Use .percpu as section identifier.
* Use bpf_jit_supports_percpu_insn() instead of CONFIG_SMP.
* Drop bpf_map__is_internal_percpu() API.
* Drop unnecessary __aligned(8) in libbpf, verified by selftest.
* Make mmap data read-only after loading prog.
v3: https://lore.kernel.org/bpf/[email protected]/
v2 -> v3:
* Use ".data..percpu" as PERCPU_DATA_SEC.
* Address comment from Alexei:
* Add u8, array of ints and struct { .. } vars to selftest.
v2: https://lore.kernel.org/bpf/[email protected]/
v1 -> v2:
* Address comments from Andrii:
* Use LIBBPF_MAP_PERCPU and SEC_PERCPU.
* Reuse mmaped of libbpf's struct bpf_map for .percpu map data.
* Set .percpu struct pointer to NULL after loading skeleton.
* Make sure value size of .percpu map is __aligned(8).
* Use raw_tp and opts.cpu to test global percpu variables on all CPUs.
* Address comments from Alexei:
* Test non-zero offset of global percpu variable.
* Test case about BPF_PSEUDO_MAP_IDX_VALUE.
v1: https://lore.kernel.org/bpf/[email protected]/
rfc -> v1:
* Address comments from Andrii:
* Keep one image of global percpu variable for all CPUs.
* Reject non-ARRAY map in bpf_map_direct_read(), check_reg_const_str(), and check_bpf_snprintf_call() in verifier.
* Split out libbpf changes from kernel-side changes.
* Use ".percpu" as PERCPU_DATA_SEC.
* Use enum libbpf_map_type to distinguish BSS, DATA, RODATA and PERCPU_DATA.
* Avoid using errno for checking err from libbpf_num_possible_cpus().
* Use "map '%s': " prefix for error message.
rfc: https://lore.kernel.org/bpf/[email protected]/
Leon Hwang (11):
bpf: Drop duplicate blank lines in verifier
bpf: Disallow interpreter fallback for user BPF_ADDR_SPACE_CAST insn
bpf: Disallow interpreter fallback for BPF_ADDR_PERCPU insn
bpf: Introduce global percpu data
libbpf: Probe percpu data feature
libbpf: Add support for global percpu data
bpftool: Generate skeleton for global percpu data selftests/bpf: Add tests to verify global percpu data selftests/bpf: Add test to verify accessing rdonly percpu_array selftests/bpf: Add tests to verify verifier log for global percpu data selftests/bpf: Add test to verify bpf_iter for global percpu data
include/linux/bpf.h | 6 +- kernel/bpf/arraymap.c | 39 ++- kernel/bpf/const_fold.c | 1 - kernel/bpf/core.c | 2 + kernel/bpf/fixups.c | 41 +++ kernel/bpf/verifier.c | 30 +- tools/bpf/bpftool/gen.c | 43 ++- tools/lib/bpf/bpf_gen_internal.h | 3 +- tools/lib/bpf/features.c | 35 +++ tools/lib/bpf/gen_loader.c | 3 +- tools/lib/bpf/libbpf.c | 68 ++++- tools/lib/bpf/libbpf_internal.h | 2 + tools/lib/bpf/skel_internal.h | 24 +- tools/testing/selftests/bpf/Makefile | 2 +- .../bpf/prog_tests/global_data_init.c | 286 ++++++++++++++++++ .../bpf/prog_tests/global_percpu_subskel.c | 37 +++ .../bpf/progs/test_global_percpu_data.c | 79 +++++ 17 files changed, 655 insertions(+), 46 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c create mode 100644 tools/testing/selftests/bpf/progs/test_global_percpu_data.c
Signature · 1 line
2.54.0
PATCH v7 1/11 bpf: Drop duplicate blank lines in verifier
There are many adjacent blank lines in the verifier that have accumulated over time.
Drop them for cleanup.
No functional changes intended.
--- kernel/bpf/verifier.c | 15 --------------- 1 file changed, 15 deletions(-)
kernel/bpf/verifier.c +0 -15
@@ -652,7 +652,6 @@ static void __mark_dynptr_reg(struct bpf_reg_state *reg,enum bpf_dynptr_type type, bool first_slot, int id, int parent_id); - static void mark_dynptr_stack_regs(struct bpf_verifier_env *env, struct bpf_reg_state *sreg1, struct bpf_reg_state *sreg2,
@@ -1689,7 +1688,6 @@ static bool same_callsites(struct bpf_verifier_state *a, struct bpf_verifier_stareturn true; } -void bpf_free_backedges (struct bpf_scc_visit *visit) {struct bpf_scc_backedge *backedge, *next;
@@ -2309,7 +2307,6 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env,return &elem->st; } -static int cmp_subprogs (const void *a,const void *b) { return ((struct bpf_subprog_info *)a)->start -
@@ -3331,7 +3328,6 @@ static bool is_spillable_regtype(enum bpf_reg_type type) } } -/* check if register is a constant scalar value */ static bool is_reg_const (struct bpf_reg_state *reg,bool subreg32) {
@@ -3993,7 +3989,6 @@ static int check_stack_read(struct bpf_verifier_env *env,return err; } -/* check_stack_write dispatches to check_stack_write_fixed_off or * check_stack_write_var_off. *
@@ -4788,7 +4783,6 @@ static int check_sock_access(struct bpf_verifier_env *env, int insn_idx, valid = false; } -if (valid) { env->insn_aux_data[insn_idx].ctx_field_size = info.ctx_field_size;
@@ -6608,7 +6602,6 @@ static int check_stack_range_initialized(if (err)return err; -if (tnum_is_const (reg->var_off)) { min_off = max_off = reg->var_off.value + off; }else {
@@ -7322,7 +7315,6 @@ static bool is_iter_new_kfunc(struct bpf_kfunc_call_arg_meta *meta)return meta->kfunc_flags &KF_ITER_NEW ; } -static bool is_iter_destroy_kfunc (struct bpf_kfunc_call_arg_meta *meta) {return meta->kfunc_flags &KF_ITER_DESTROY ;
@@ -11465,7 +11457,6 @@ static int process_irq_flag(struct bpf_verifier_env *env, struct bpf_reg_state *return 0 ; } -static int ref_set_non_owning (struct bpf_verifier_env *env,struct bpf_reg_state *reg) {struct btf_record *rec =reg_btf_record (reg);
@@ -16357,7 +16348,6 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn)return 0 ; } -static bool return_retval_range (struct bpf_verifier_env *env,struct bpf_retval_range *range) {enum bpf_prog_type prog_type =resolve_prog_type (env->prog);
@@ -18251,8 +18241,6 @@ static void release_insn_arrays(struct bpf_verifier_env *env)bpf_insn_array_release (env->insn_array_maps[i]); } - -/* The verifier does more data flow analysis than llvm and will not * explore branches that are dead at run time. Malicious programs can * have dead code too. Therefore replace all dead at-run-time code
@@ -18280,8 +18268,6 @@ static void sanitize_dead_code(struct bpf_verifier_env *env) } } - -static void free_states (struct bpf_verifier_env *env) {struct bpf_verifier_state_list *sl;
@@ -18563,7 +18549,6 @@ static int do_check_main(struct bpf_verifier_env *env)return ret; } -static void print_verification_stats (struct bpf_verifier_env *env) {/* Skip over hidden subprogs which are not verified. */
Signature · 1 line
2.54.0
PATCH v7 2/11 bpf: Disallow interpreter fallback for user BPF_ADDR_SPACE_CAST insn
In reply to Leon HwangSince interpreter is unable to handle the 'insn_is_cast_user()' insn, require JIT in __bpf_prog_select_runtime() when the prog has the insn.
--- include/linux/bpf.h | 1 + kernel/bpf/core.c | 1 + kernel/bpf/fixups.c | 2 ++ 3 files changed, 4 insertions(+)
include/linux/bpf.h +1 -0
@@ -1781,6 +1781,7 @@ struct bpf_prog_aux {bool changes_pkt_data;bool might_sleep;bool kprobe_write_ctx; +bool has_addr_space_cast_insn;struct {s32 keyring_serial;u8 keyring_type;
kernel/bpf/core.c +1 -0
@@ -2620,6 +2620,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct goto finalize;if (IS_ENABLED (CONFIG_BPF_JIT_ALWAYS_ON ) || + fp->aux->has_addr_space_cast_insn ||bpf_prog_has_kfunc_call (fp)) jit_needed = true;
kernel/bpf/fixups.c +2 -0
@@ -1523,6 +1523,8 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) insn->off =0 ; insn->imm =0 ; }/* cast from as(0) to as(1) should be handled by JIT */ +if (insn_is_cast_user (insn)) + prog->aux->has_addr_space_cast_insn = true; goto next_insn; }
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH bpf-next v7 02/11] bpf: Disallow interpreter fallback for user BPF_ADDR_SPACE_CAST insn
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
New issues:
- [Medium] Missing verifier log message for non-JITed programs rejected due to has_addr_space_cast_insn.
Pre-existing issues:
- [Critical] Incomplete fix allows BPF interpreter fallback for arena programs, leading to a kernel panic. --
commit 37ba328d18b1ffa46e00f11cf84efa654fd229ec
Author: Leon Hwang <[email protected]>
bpf: Disallow interpreter fallback for user BPF_ADDR_SPACE_CAST insn
Since interpreter is unable to handle the 'insn_is_cast_user()' insn, require JIT in __bpf_prog_select_runtime() when the prog has the insn.
kernel/bpf/core.c struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct · line 2625
Quoted 11 lines
kernel/bpf/core.c +1
-0@@ -2620,6 +2620,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct goto finalize;if (IS_ENABLED (CONFIG_BPF_JIT_ALWAYS_ON ) || + fp->aux->has_addr_space_cast_insn ||bpf_prog_has_kfunc_call (fp)) jit_needed = true;
[Severity: Medium]
If JIT is disabled or fails, returning -ENOTSUPP here will reject the program load silently without a verifier log message. Unlike bpf_prog_has_kfunc_call() which provides a helpful verbose() log in bpf_fixup_call_args():
if (has_kfunc_call) {
verbose(env, "calling kernel functions are not allowed in non-JITed programs\n");
return -EINVAL;
}Users encountering this for has_addr_space_cast_insn will not see an explanation for the EOPNOTSUPP error. Could we add a verifier log message explaining why JIT is required here?
kernel/bpf/fixups.c int bpf_do_misc_fixups(struct bpf_verifier_env *env) · line 1527
Quoted 10 lines
kernel/bpf/fixups.c +2
-0@@ -1523,6 +1523,8 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) insn->off =0 ; insn->imm =0 ; }/* cast from as(0) to as(1) should be handled by JIT */ +if (insn_is_cast_user (insn)) + prog->aux->has_addr_space_cast_insn = true;
[Severity: Critical]
This is a pre-existing issue, but the patch attempts to fix interpreter fallback for BPF_ADDR_SPACE_CAST and fails to address the underlying panic with arena programs.
If a program uses an arena and obtains a PTR_TO_ARENA without a cast_user instruction (for instance, using cast_kern), or if the arena map has BPF_F_NO_USER_CONV set, has_addr_space_cast_insn will not be set.
In the case of BPF_F_NO_USER_CONV, the instruction is rewritten to BPF_ALU | BPF_MOV | BPF_X right before the insn_is_cast_user() check.
Memory accesses using PTR_TO_ARENA are rewritten by bpf_do_misc_fixups() to use BPF_PROBE_MEM32 or BPF_PROBE_MEM32SX. If these arena programs fall back to the interpreter, wouldn't they hit the BUG_ON(1) in ___bpf_prog_run() because it lacks handlers for these opcodes?
kernel/bpf/core.c:___bpf_prog_run() {
...
default_label:
pr_warn("BPF interpreter: unknown opcode %02x (imm: 0x%x)\n",
insn->code, insn->imm);
BUG_ON(1);
...
}Should jit_needed be enforced in __bpf_prog_select_runtime() for all programs using fp->aux->arena instead?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Re: [PATCH bpf-next v7 02/11] bpf: Disallow interpreter fallback for user BPF_ADDR_SPACE_CAST insn
On 22/6/26 23:07, [email protected] wrote:
[...]
Quoted 18 lines
kernel/bpf/core.c struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct · line 2625
Quoted 11 lines
kernel/bpf/core.c +1
-0@@ -2620,6 +2620,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct goto finalize;if (IS_ENABLED (CONFIG_BPF_JIT_ALWAYS_ON ) || + fp->aux->has_addr_space_cast_insn ||bpf_prog_has_kfunc_call (fp)) jit_needed = true;[Severity: Medium]
If JIT is disabled or fails, returning-ENOTSUPPhere will reject the program load silently without a verifier log message. Unlikebpf_prog_has_kfunc_call()which provides a helpfulverbose()log inbpf_fixup_call_args():
Quoted 9 lines
if (has_kfunc_call) { verbose(env, "calling kernel functions are not allowed in non-JITed programs\n"); return -EINVAL; }Users encountering this for
has_addr_space_cast_insnwill not see an explanation for the EOPNOTSUPP error. Could we add a verifier log message explaining why JIT is required here?
No verifier log support here.
The verifier log has been finalized before calling __bpf_prog_select_runtime() in bpf_check().
Quoted 37 lines
kernel/bpf/fixups.c int bpf_do_misc_fixups(struct bpf_verifier_env *env) · line 1527
Quoted 10 lines
kernel/bpf/fixups.c +2
-0@@ -1523,6 +1523,8 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) insn->off =0 ; insn->imm =0 ; }/* cast from as(0) to as(1) should be handled by JIT */ +if (insn_is_cast_user (insn)) + prog->aux->has_addr_space_cast_insn = true;[Severity: Critical]
This is a pre-existing issue, but the patch attempts to fix interpreter fallback forBPF_ADDR_SPACE_CASTand fails to address the underlying panic with arena programs.If a program uses an arena and obtains a
PTR_TO_ARENAwithout acast_userinstruction (for instance, usingcast_kern), or if the arena map hasBPF_F_NO_USER_CONVset,has_addr_space_cast_insnwill not be set.In the case of
BPF_F_NO_USER_CONV, the instruction is rewritten toBPF_ALU|BPF_MOV|BPF_Xright before theinsn_is_cast_user()check.Memory accesses using
PTR_TO_ARENAare rewritten bybpf_do_misc_fixups()to useBPF_PROBE_MEM32orBPF_PROBE_MEM32SX. If these arena programs fall back to the interpreter, wouldn't they hit theBUG_ON(1)in___bpf_prog_run()because it lacks handlers for these opcodes?kernel/bpf/core.c:___bpf_prog_run() { ... default_label: pr_warn("BPF interpreter: unknown opcode %02x (imm: 0x%x)\n", insn->code, insn->imm); BUG_ON(1); ... }
Quoted 3 lines
Should
jit_neededbe enforced in__bpf_prog_select_runtime()for all programs usingfp->aux->arenainstead?
Sounds reasonable.
However, I think such similar issues should be fixed using separate
patches: all interpreter-unsupported insns should be marked as JIT-needed.
Like the way of "bpf: Fix unaligned interpreter panic on JIT fallback path" [1], we can verify whether there are the following insns in the prog by scanning prog->insnsi:
* user ADDR_SPACE_CAST
* PROBE_MEM32, PROBE_MEM32SX
* ADDR_PERCPU
* PROBE_ATOMIC
* indirect jump using insn_array map
I think these insns are not supported by interpreter.
Link:
[1]
https://lore.kernel.org/bpf/[email protected]/
Thanks,
Leon
Re: [PATCH bpf-next v7 02/11] bpf: Disallow interpreter fallback for user BPF_ADDR_SPACE_CAST insn
In reply to Leon HwangQuoted 4 lines · Leon Hwang
Since interpreter is unable to handle the '
insn_is_cast_user()' insn, require JIT in__bpf_prog_select_runtime()when the prog has the insn.
Quoted 8 lines
Fixes 142fd4d2dcf5 ("bpf: Add x86-64 JIT support for bpf_addr_space_cast instruction.")Signed-off-by Leon Hwang <[email protected]>--- include/linux/bpf.h | 1 + kernel/bpf/core.c | 1 + kernel/bpf/fixups.c | 2 ++ 3 files changed, 4 insertions(+)
apologies if this was discussed before, but this seems unrelated to the main patch set, why not send it as its own fix separately? same for the next patch, no?
Quoted 40 lines
include/linux/bpf.h +1
-0@@ -1781,6 +1781,7 @@ struct bpf_prog_aux {bool changes_pkt_data;bool might_sleep;bool kprobe_write_ctx; +bool has_addr_space_cast_insn;struct {s32 keyring_serial;u8 keyring_type;kernel/bpf/core.c +1
-0@@ -2620,6 +2620,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct goto finalize;if (IS_ENABLED (CONFIG_BPF_JIT_ALWAYS_ON ) || + fp->aux->has_addr_space_cast_insn ||bpf_prog_has_kfunc_call (fp)) jit_needed = true;kernel/bpf/fixups.c +2
-1@@ -1523,6 +1523,8 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) insn->off =0 ; insn->imm =0 ; }/* cast from as(0) to as(1) should be handled by JIT */ +if (insn_is_cast_user (insn)) + prog->aux->has_addr_space_cast_insn = true; goto next_insn; } --2.54.0
Re: [PATCH bpf-next v7 02/11] bpf: Disallow interpreter fallback for user BPF_ADDR_SPACE_CAST insn
Quoted 14 lines · Andrii Nakryiko
Quoted 4 lines · Leon Hwang
Since interpreter is unable to handle the '
insn_is_cast_user()' insn, require JIT in__bpf_prog_select_runtime()when the prog has the insn.Quoted 8 lines
Fixes 142fd4d2dcf5 ("bpf: Add x86-64 JIT support for bpf_addr_space_cast instruction.")Signed-off-by Leon Hwang <[email protected]>--- include/linux/bpf.h | 1 + kernel/bpf/core.c | 1 + kernel/bpf/fixups.c | 2 ++ 3 files changed, 4 insertions(+)
Quoted 4 lines
apologies if this was discussed before, but this seems unrelated to the main patch set, why not send it as its own fix separately? same for the next patch, no?
Will send separately to fix the issues of interpreter fallback.
Thanks,
Leon
PATCH v7 3/11 bpf: Disallow interpreter fallback for BPF_ADDR_PERCPU insn
In reply to Leon HwangSince interpreter is unable to handle the 'insn_is_mov_percpu_addr()' insn, require JIT in __bpf_prog_select_runtime() when the prog has the insn.
BTW, rename the helper bpf_map_supports_cpu_flags() to bpf_map_is_percpu_map().
--- include/linux/bpf.h | 5 +++-- kernel/bpf/core.c | 1 + kernel/bpf/fixups.c | 5 +++++ 3 files changed, 9 insertions(+), 2 deletions(-)
include/linux/bpf.h +3 -2
@@ -1782,6 +1782,7 @@ struct bpf_prog_aux {bool might_sleep;bool kprobe_write_ctx;bool has_addr_space_cast_insn; +bool has_addr_percpu_insn;struct {s32 keyring_serial;u8 keyring_type;
@@ -4170,7 +4171,7 @@ bpf_prog_update_insn_ptrs(struct bpf_prog *prog, u32 *offsets, void *image) }#endif -static inline bool bpf_map_supports_cpu_flags (enum bpf_map_type map_type) +static inline bool bpf_map_is_percpu_map (enum bpf_map_type map_type) {switch (map_type) {case BPF_MAP_TYPE_PERCPU_ARRAY :
@@ -4197,7 +4198,7 @@ static inline int bpf_map_check_op_flags(struct bpf_map *map, u64 flags, u64 allreturn -EINVAL ;if (flags & (BPF_F_CPU |BPF_F_ALL_CPUS )) { -if (!bpf_map_supports_cpu_flags (map->map_type)) +if (!bpf_map_is_percpu_map (map->map_type))return -EINVAL ;if ((flags &BPF_F_CPU ) && (flags &BPF_F_ALL_CPUS ))return -EINVAL ;
kernel/bpf/core.c +1 -0
@@ -2621,6 +2621,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, structif (IS_ENABLED (CONFIG_BPF_JIT_ALWAYS_ON ) || fp->aux->has_addr_space_cast_insn || + fp->aux->has_addr_percpu_insn ||bpf_prog_has_kfunc_call (fp)) jit_needed = true;
kernel/bpf/fixups.c +5 -0
@@ -2009,6 +2009,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)return -EFAULT ; } +if (bpf_map_is_percpu_map (map_ptr->map_type)) + prog->aux->has_addr_percpu_insn = true; + new_prog =bpf_patch_insn_data (env, i + delta, insn_buf, cnt);if (!new_prog)
@@ -2113,6 +2116,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)* way, it's fine to back out this inlining logic */ #ifdef CONFIG_SMP + prog->aux->has_addr_percpu_insn = true; insn_buf[0 ] =BPF_MOV64_IMM (BPF_REG_0 , (u32 )(unsigned long )&cpu_number); insn_buf[1 ] =BPF_MOV64_PERCPU_REG (BPF_REG_0 ,BPF_REG_0 ); insn_buf[2 ] =BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_0 ,0 );
@@ -2134,6 +2138,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)/* Implement bpf_get_current_task() and bpf_get_current_task_btf() inline. */ if ((insn->imm == BPF_FUNC_get_current_task || insn->imm == BPF_FUNC_get_current_task_btf) &&bpf_verifier_inlines_helper_call (env, insn->imm)) { + prog->aux->has_addr_percpu_insn = true; insn_buf[0 ] =BPF_MOV64_IMM (BPF_REG_0 , (u32 )(unsigned long )¤t_task); insn_buf[1 ] =BPF_MOV64_PERCPU_REG (BPF_REG_0 ,BPF_REG_0 ); insn_buf[2 ] =BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_0 ,0 );
Signature · 1 line
2.54.0
PATCH v7 4/11 bpf: Introduce global percpu data
In reply to Leon HwangIntroduce global percpu data, inspired by the commit 6316f78306c1 ("Merge branch 'support-global-data'"). It enables the definition of global percpu variables in BPF, similar to the include/linux/percpu-defs.h::DEFINE_PER_CPU() macro.
For example, in BPF, it is able to define a global percpu variable like:
int data SEC(".percpu");
With this patch, tools like retsnoop [1] and bpfsnoop [2] can simplify their BPF code for handling LBRs. The code can be updated from
static struct perf_branch_entry lbrs[1][MAX_LBR_ENTRIES] SEC(".data.lbrs");
to
static struct perf_branch_entry lbrs[MAX_LBR_ENTRIES] SEC(".percpu.lbrs");
This eliminates the need to retrieve the CPU ID using the bpf_get_smp_processor_id() helper.
Additionally, by reusing global percpu data map, sharing information between tail callers and callees or freplace callers and callees becomes simpler compared to reusing percpu_array maps.
Links:
[1] https://github.com/anakryiko/retsnoop
[2] https://github.com/bpfsnoop/bpfsnoop
--- kernel/bpf/arraymap.c | 39 +++++++++++++++++++++++++++++++++++++-- kernel/bpf/const_fold.c | 1 - kernel/bpf/fixups.c | 34 ++++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 15 +++++++++++++++ 4 files changed, 86 insertions(+), 3 deletions(-)
kernel/bpf/arraymap.c +37 -2
@@ -259,6 +259,38 @@ static void *percpu_array_map_lookup_elem(struct bpf_map *map, void *key)return this_cpu_ptr (array->pptrs[index & array->index_mask]); } +static int percpu_array_map_direct_value_addr (const struct bpf_map *map,u64 *imm,u32 off) +{ + struct bpf_array *array = container_of(map, struct bpf_array, map); + +if (map->max_entries !=1 ) +return -EOPNOTSUPP ; +if (off >= map->value_size) +return -EINVAL ; +if (!bpf_jit_supports_percpu_insn ()) +return -EOPNOTSUPP ; + + *imm = (u64 )(__force unsigned long ) array->pptrs[0 ]; +return 0 ; +} + +static int percpu_array_map_direct_value_meta (const struct bpf_map *map,u64 imm,u32 *off) +{ + struct bpf_array *array = container_of(map, struct bpf_array, map); +u64 base = (u64 )(__force unsigned long ) array->pptrs[0 ]; +u64 range = array->elem_size; + +if (map->max_entries !=1 ) +return -EOPNOTSUPP ; +if (imm < base || imm >= base + range) +return -ENOENT ; +if (!bpf_jit_supports_percpu_insn ()) +return -EOPNOTSUPP ; + + *off = imm - base; +return 0 ; +} +/* emit BPF instructions equivalent to C code of percpu_array_map_lookup_elem() */ static int percpu_array_map_gen_lookup (struct bpf_map *map,struct bpf_insn *insn_buf) {
@@ -551,9 +583,10 @@ static int array_map_check_btf(struct bpf_map *map, const struct btf_type *key_type, const struct btf_type *value_type) { - /* One exception for keyless BTF: .bss/.data/.rodata map */ + /* One exception for keyless BTF: .bss/.data/.rodata/.percpu map */ if (btf_type_is_void(key_type)) { - if (map->map_type != BPF_MAP_TYPE_ARRAY || + if ((map->map_type != BPF_MAP_TYPE_ARRAY && + map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) || map->max_entries != 1) return -EINVAL;
@@ -832,6 +865,8 @@ const struct bpf_map_ops percpu_array_map_ops = { .map_get_next_key = bpf_array_get_next_key, .map_lookup_elem = percpu_array_map_lookup_elem, .map_gen_lookup = percpu_array_map_gen_lookup, + .map_direct_value_addr = percpu_array_map_direct_value_addr, + .map_direct_value_meta = percpu_array_map_direct_value_meta, .map_update_elem = array_map_update_elem, .map_delete_elem = array_map_delete_elem, .map_lookup_percpu_elem = percpu_array_map_lookup_percpu_elem,
kernel/bpf/const_fold.c +0 -1
@@ -182,7 +182,6 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info *u64 val =0 ;if (!bpf_map_is_rdonly (map) || !map->ops->map_direct_value_addr || - map->map_type ==BPF_MAP_TYPE_INSN_ARRAY || off <0 || off + size > map->value_size ||bpf_map_direct_read (map, off, size, &val, is_ldsx)) { *dst = unknown;
kernel/bpf/fixups.c +34 -0
@@ -1821,6 +1821,40 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) goto next_insn; } +if (env->prog->jit_requested && +bpf_jit_supports_percpu_insn () && + insn->code == (BPF_LD |BPF_IMM |BPF_DW ) && + (insn->src_reg ==BPF_PSEUDO_MAP_VALUE || + insn->src_reg ==BPF_PSEUDO_MAP_IDX_VALUE )) { +struct bpf_map *map; + + aux = &env->insn_aux_data[i + delta]; + map = env->used_maps[aux->map_index]; +if (map->map_type !=BPF_MAP_TYPE_PERCPU_ARRAY ) + goto next_insn; + + prog->aux->has_addr_percpu_insn = true; + +/* +* Reuse the original ld_imm64 insn, and add one +* mov64_percpu_reg insn. +*/ + + insn_buf[0 ] = insn[1 ]; + insn_buf[1 ] =BPF_MOV64_PERCPU_REG (insn->dst_reg, insn->dst_reg); + cnt =2 ; + + i++; + new_prog =bpf_patch_insn_data (env, i + delta, insn_buf, cnt); +if (!new_prog) +return -ENOMEM ; + + delta += cnt -1 ; + env->prog = prog = new_prog; + insn = new_prog->insnsi + i + delta; + goto next_insn; + } +if (insn->code != (BPF_JMP |BPF_CALL )) goto next_insn;if (insn->src_reg ==BPF_PSEUDO_CALL )
kernel/bpf/verifier.c +15 -0
@@ -5562,6 +5562,8 @@ int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val,u64 addr;int err; +if (map->map_type ==BPF_MAP_TYPE_INSN_ARRAY || map->map_type ==BPF_MAP_TYPE_PERCPU_ARRAY ) +return -EINVAL ; err = map->ops->map_direct_value_addr (map, &addr, off);if (err)return err;
@@ -6113,6 +6115,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct bif (tnum_is_const (reg->var_off) &&bpf_map_is_rdonly (map) && map->ops->map_direct_value_addr && + map->map_type !=BPF_MAP_TYPE_PERCPU_ARRAY && map->map_type !=BPF_MAP_TYPE_INSN_ARRAY ) {int map_off = off + reg->var_off.value;u64 val =0 ;
@@ -8080,6 +8083,12 @@ static int check_arg_const_str(struct bpf_verifier_env *env,return -EACCES ; } +if (map->map_type ==BPF_MAP_TYPE_PERCPU_ARRAY ) { +verbose (env,"%s points to percpu_array map which cannot be used as const string\n" , +reg_arg_name (env, argno)); +return -EACCES ; + } +if (!bpf_map_is_rdonly (map)) {verbose (env,"%s does not point to a readonly map'\n" ,reg_arg_name (env, argno));return -EACCES ;
@@ -18166,6 +18175,12 @@ static int check_and_resolve_insns(struct bpf_verifier_env *env)return -EINVAL ; } +if (map->map_type ==BPF_MAP_TYPE_PERCPU_ARRAY && + !env->prog->jit_requested) { +verbose (env,"JIT is required to use global percpu data\n" ); +return -EOPNOTSUPP ; + } + err = map->ops->map_direct_value_addr (map, &addr, off);if (err) { verbose(env, "invalid access to map value pointer, value_size=%u off=%u\n",
Signature · 1 line
2.54.0
Quoted 60 lines · Leon Hwang
Introduce global percpu data, inspired by the commit
6316f78306c1("Merge branch 'support-global-data'"). It enables the definition of global percpu variables in BPF, similar to the include/linux/percpu-defs.h::DEFINE_PER_CPU() macro.For example, in BPF, it is able to define a global percpu variable like:
int data
SEC(".percpu");With this patch, tools like retsnoop [1] and bpfsnoop [2] can simplify their BPF code for handling LBRs. The code can be updated from
static struct
perf_branch_entrylbrs[1][MAX_LBR_ENTRIES]SEC(".data.lbrs");to
static struct
perf_branch_entrylbrs[MAX_LBR_ENTRIES]SEC(".percpu.lbrs");This eliminates the need to retrieve the CPU ID using the
bpf_get_smp_processor_id()helper.Additionally, by reusing global percpu data map, sharing information between tail callers and callees or freplace callers and callees becomes simpler compared to reusing
percpu_arraymaps.Links:
[1] https://github.com/anakryiko/retsnoop
[2] https://github.com/bpfsnoop/bpfsnoopSigned-off-by Leon Hwang <[email protected]>--- kernel/bpf/arraymap.c | 39 +++++++++++++++++++++++++++++++++++++-- kernel/bpf/const_fold.c | 1 - kernel/bpf/fixups.c | 34 ++++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 15 +++++++++++++++ 4 files changed, 86 insertions(+), 3 deletions(-)kernel/bpf/arraymap.c +14
-0@@ -259,6 +259,38 @@ static void *percpu_array_map_lookup_elem(struct bpf_map *map, void *key)return this_cpu_ptr (array->pptrs[index & array->index_mask]); } +static int percpu_array_map_direct_value_addr (const struct bpf_map *map,u64 *imm,u32 off) +{ + struct bpf_array *array = container_of(map, struct bpf_array, map); + +if (map->max_entries !=1 ) +return -EOPNOTSUPP ; +if (off >= map->value_size) +return -EINVAL ; +if (!bpf_jit_supports_percpu_insn ()) +return -EOPNOTSUPP ; + + *imm = (u64 )(__force unsigned long ) array->pptrs[0 ]; +return 0 ; +}
kernel/bpf/arraymap.c static void *percpu_array_map_lookup_elem(struct bpf_map *map, void *key) · line 285
Quoted 10 lines
kernel/bpf/arraymap.c +10
-0+ +static int percpu_array_map_direct_value_meta (const struct bpf_map *map,u64 imm,u32 *off) +{ + struct bpf_array *array = container_of(map, struct bpf_array, map); +u64 base = (u64 )(__force unsigned long ) array->pptrs[0 ]; +u64 range = array->elem_size; + +if (map->max_entries !=1 ) +return -EOPNOTSUPP ; +if (imm < base || imm >= base + range)
nit: not sure what's the value of range local variable, single place where you use array->elem_size
Quoted 38 lines
kernel/bpf/arraymap.c +13
-2+return -ENOENT ; +if (!bpf_jit_supports_percpu_insn ()) +return -EOPNOTSUPP ; + + *off = imm - base; +return 0 ; +} +/* emit BPF instructions equivalent to C code of percpu_array_map_lookup_elem() */ static int percpu_array_map_gen_lookup (struct bpf_map *map,struct bpf_insn *insn_buf) {@@ -551,9 +583,10 @@ static int array_map_check_btf(struct bpf_map *map, const struct btf_type *key_type, const struct btf_type *value_type) { - /* One exception for keyless BTF: .bss/.data/.rodata map */ + /* One exception for keyless BTF: .bss/.data/.rodata/.percpu map */ if (btf_type_is_void(key_type)) { - if (map->map_type != BPF_MAP_TYPE_ARRAY || + if ((map->map_type != BPF_MAP_TYPE_ARRAY && + map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) || map->max_entries != 1) return -EINVAL;@@ -832,6 +865,8 @@ const struct bpf_map_ops percpu_array_map_ops = { .map_get_next_key = bpf_array_get_next_key, .map_lookup_elem = percpu_array_map_lookup_elem, .map_gen_lookup = percpu_array_map_gen_lookup, + .map_direct_value_addr = percpu_array_map_direct_value_addr, + .map_direct_value_meta = percpu_array_map_direct_value_meta, .map_update_elem = array_map_update_elem, .map_delete_elem = array_map_delete_elem, .map_lookup_percpu_elem = percpu_array_map_lookup_percpu_elem,kernel/bpf/const_fold.c +0
-0
kernel/bpf/const_fold.c static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info * · line 185
Quoted 4 lines
kernel/bpf/const_fold.c +0
-1@@ -182,7 +182,6 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info *u64 val =0 ; if (!bpf_map_is_rdonly(map) || !map->ops->map_direct_value_addr || - map->map_type == BPF_MAP_TYPE_INSN_ARRAY ||
can you please elaborate on why this change is needed?
Quoted 3 lines
kernel/bpf/const_fold.c +0
-0off <0 || off + size > map->value_size || bpf_map_direct_read(map, off, size, &val, is_ldsx)) { *dst = unknown;
[...]
Quoted 1 line · Andrii Nakryiko
On Mon, Jun 22, 2026 at 7:37 AM Leon Hwang <[email protected]> wrote:
[...]
Quoted 5 lines
Quoted 5 lines
kernel/bpf/arraymap.c +5
-0+ +static int percpu_array_map_direct_value_meta (const struct bpf_map *map,u64 imm,u32 *off) +{ + struct bpf_array *array = container_of(map, struct bpf_array, map); +u64 base = (u64 )(__force unsigned long ) array->pptrs[0 ];
Quoted 9 lines
kernel/bpf/arraymap.c static void *percpu_array_map_lookup_elem(struct bpf_map *map, void *key) · line 285
Quoted 5 lines
kernel/bpf/arraymap.c +5
-0+u64 range = array->elem_size; + +if (map->max_entries !=1 ) +return -EOPNOTSUPP ; +if (imm < base || imm >= base + range)nit: not sure what's the value of range local variable, single place where you use
array->elem_size
Hmm, this was copied from array_map_direct_value_meta().
Will drop "range", and use "array->elem_size" here.
Quoted 8 lines
Quoted 8 lines
kernel/bpf/arraymap.c +8
-0+return -ENOENT ; +if (!bpf_jit_supports_percpu_insn ()) +return -EOPNOTSUPP ; + + *off = imm - base; +return 0 ; +} +
[...]
Quoted 7 lines
Quoted 7 lines
kernel/bpf/const_fold.c +0
-0@@ -182,7 +182,6 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info *u64 val =0 ;
Quoted 5 lines
kernel/bpf/const_fold.c static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info * · line 185
Quoted 2 lines
kernel/bpf/const_fold.c +0
-1if (!bpf_map_is_rdonly(map) || !map->ops->map_direct_value_addr || - map->map_type == BPF_MAP_TYPE_INSN_ARRAY ||can you please elaborate on why this change is needed?
This was suggested by Alexei in v4:
https://lore.kernel.org/bpf/CAADnVQJ0o38xXn6E5ub7kWyoZPiaevr9mwDNxv0GbxyguVTKUg@mail.gmail.com/
Thanks,
Leon
Quoted 5 lines
Quoted 3 lines
kernel/bpf/const_fold.c +0
-0off <0 || off + size > map->value_size || bpf_map_direct_read(map, off, size, &val, is_ldsx)) { *dst = unknown;[...]
PATCH v7 5/11 libbpf: Probe percpu data feature
In reply to Leon Hwanglibbpf needs a reliable way to distinguish kernels that can support global percpu data from those that cannot.
Add a dedicated feature probe, so libbpf can make capability decisions early and fail predictably when global percpu data is unavailable.
--- tools/lib/bpf/features.c | 35 +++++++++++++++++++++++++++++++++ tools/lib/bpf/libbpf_internal.h | 2 ++ 2 files changed, 37 insertions(+)
tools/lib/bpf/features.c +35 -0
@@ -620,6 +620,38 @@ static int probe_bpf_syscall_common_attrs(int token_fd)return probe_sys_bpf_ext (); } +static int probe_kern_percpu_data (int token_fd) +{ +struct bpf_insn insns[] = { +BPF_LD_MAP_VALUE (BPF_REG_1 ,0 ,0 ), +BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_1 ,0 ), +BPF_EXIT_INSN (), + }; +LIBBPF_OPTS (bpf_map_create_opts, map_opts, + .token_fd = token_fd, + .map_flags = token_fd ? BPF_F_TOKEN_FD : 0, + ); +LIBBPF_OPTS (bpf_prog_load_opts, prog_opts, + .token_fd = token_fd, + .prog_flags = token_fd ? BPF_F_TOKEN_FD : 0, + ); +int ret, map, insn_cnt =ARRAY_SIZE (insns); + + map =bpf_map_create (BPF_MAP_TYPE_PERCPU_ARRAY ,"libbpf_percpu" ,sizeof (int ),8 ,1 , + &map_opts); +if (map <0 ) { +pr_warn ("Error in %s(): %s. Couldn't create simple percpu_array map.\n" , + __func__,errstr (map)); +return map; + } + + insns[0 ].imm = map; + + ret =bpf_prog_load (BPF_PROG_TYPE_SOCKET_FILTER ,NULL ,"GPL" , insns, insn_cnt, &prog_opts); +close (map); +return probe_fd (ret); +} +typedef int (*feature_probe_fn )(int /* token_fd */ );static struct kern_feature_cache feature_cache;
@@ -707,6 +739,9 @@ static struct kern_feature_desc { [FEAT_BPF_SYSCALL_COMMON_ATTRS ] = {"BPF syscall common attributes support" , probe_bpf_syscall_common_attrs, }, + [FEAT_PERCPU_DATA ] = { +"kernel supports percpu data" , probe_kern_percpu_data, + }, };bool feat_supported (struct kern_feature_cache *cache,enum kern_feature_id feat_id)
tools/lib/bpf/libbpf_internal.h +2 -0
@@ -401,6 +401,8 @@ enum kern_feature_id {FEAT_BTF_LAYOUT ,/* Kernel supports BPF syscall common attributes */ FEAT_BPF_SYSCALL_COMMON_ATTRS , +/* Kernel supports percpu data */ +FEAT_PERCPU_DATA , __FEAT_CNT, };
Signature · 1 line
2.54.0
Re: [PATCH bpf-next v7 05/11] libbpf: Probe percpu data feature
Quoted 4 lines · Leon Hwang
libbpf needs a reliable way to distinguish kernels that can support global percpu data from those that cannot.
Quoted 9 lines
Add a dedicated feature probe, so libbpf can make capability decisions early and fail predictably when global percpu data is unavailable.
Signed-off-by Leon Hwang <[email protected]>--- tools/lib/bpf/features.c | 35 +++++++++++++++++++++++++++++++++ tools/lib/bpf/libbpf_internal.h | 2 ++ 2 files changed, 37 insertions(+)
lgtm
Quoted 69 lines
tools/lib/bpf/features.c +35
-0@@ -620,6 +620,38 @@ static int probe_bpf_syscall_common_attrs(int token_fd)return probe_sys_bpf_ext (); } +static int probe_kern_percpu_data (int token_fd) +{ +struct bpf_insn insns[] = { +BPF_LD_MAP_VALUE (BPF_REG_1 ,0 ,0 ), +BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_1 ,0 ), +BPF_EXIT_INSN (), + }; +LIBBPF_OPTS (bpf_map_create_opts, map_opts, + .token_fd = token_fd, + .map_flags = token_fd ? BPF_F_TOKEN_FD : 0, + ); +LIBBPF_OPTS (bpf_prog_load_opts, prog_opts, + .token_fd = token_fd, + .prog_flags = token_fd ? BPF_F_TOKEN_FD : 0, + ); +int ret, map, insn_cnt =ARRAY_SIZE (insns); + + map =bpf_map_create (BPF_MAP_TYPE_PERCPU_ARRAY ,"libbpf_percpu" ,sizeof (int ),8 ,1 , + &map_opts); +if (map <0 ) { +pr_warn ("Error in %s(): %s. Couldn't create simple percpu_array map.\n" , + __func__,errstr (map)); +return map; + } + + insns[0 ].imm = map; + + ret =bpf_prog_load (BPF_PROG_TYPE_SOCKET_FILTER ,NULL ,"GPL" , insns, insn_cnt, &prog_opts); +close (map); +return probe_fd (ret); +} +typedef int (*feature_probe_fn )(int /* token_fd */ );static struct kern_feature_cache feature_cache;@@ -707,6 +739,9 @@ static struct kern_feature_desc { [FEAT_BPF_SYSCALL_COMMON_ATTRS ] = {"BPF syscall common attributes support" , probe_bpf_syscall_common_attrs, }, + [FEAT_PERCPU_DATA ] = { +"kernel supports percpu data" , probe_kern_percpu_data, + }, };bool feat_supported (struct kern_feature_cache *cache,enum kern_feature_id feat_id)tools/lib/bpf/libbpf_internal.h +2
-1@@ -401,6 +401,8 @@ enum kern_feature_id {FEAT_BTF_LAYOUT ,/* Kernel supports BPF syscall common attributes */ FEAT_BPF_SYSCALL_COMMON_ATTRS , +/* Kernel supports percpu data */ +FEAT_PERCPU_DATA , __FEAT_CNT, }; --2.54.0
Re: [PATCH bpf-next v7 05/11] libbpf: Probe percpu data feature
Quoted 15 lines · Andrii Nakryiko
Quoted 4 lines · Leon Hwang
libbpf needs a reliable way to distinguish kernels that can support global percpu data from those that cannot.
Quoted 9 lines
Add a dedicated feature probe, so libbpf can make capability decisions early and fail predictably when global percpu data is unavailable.
Signed-off-by Leon Hwang <[email protected]>--- tools/lib/bpf/features.c | 35 +++++++++++++++++++++++++++++++++ tools/lib/bpf/libbpf_internal.h | 2 ++ 2 files changed, 37 insertions(+)
Quoted 4 lines
lgtm
Acked-by Andrii Nakryiko <[email protected]>
Thanks for your review and Acked-by.
Leon
PATCH v7 6/11 libbpf: Add support for global percpu data
In reply to Leon HwangAdd support for global percpu data in libbpf by adding a new ".percpu" section, similar to ".data". It enables efficient handling of percpu global variables in bpf programs.
When generating loader for lightweight skeleton, update the percpu_array map used for global percpu data using BPF_F_ALL_CPUS, in order to update values across all CPUs using one value slot.
Unlike global data, the mmaped data for global percpu data will be marked as read-only after populating the percpu_array map. Thereafter, users can read those initialized percpu data after loading prog. If they want to update the percpu data after loading prog, they have to update the percpu_array map using key=0 instead.
--- tools/lib/bpf/bpf_gen_internal.h | 3 +- tools/lib/bpf/gen_loader.c | 3 +- tools/lib/bpf/libbpf.c | 68 ++++++++++++++++++++++++++------ 3 files changed, 60 insertions(+), 14 deletions(-)
tools/lib/bpf/bpf_gen_internal.h +2 -1
@@ -66,7 +66,8 @@ void bpf_gen__prog_load(struct bpf_gen *gen, enum bpf_prog_type prog_type, const char *prog_name, const char *license, struct bpf_insn *insns, size_t insn_cnt, struct bpf_prog_load_opts *load_attr, int prog_idx); -void bpf_gen__map_update_elem (struct bpf_gen *gen,int map_idx,void *value,__u32 value_size); +void bpf_gen__map_update_elem (struct bpf_gen *gen,int map_idx,void *value,__u32 value_size, +__u64 flags);void bpf_gen__map_freeze (struct bpf_gen *gen,int map_idx);void bpf_gen__record_attach_target (struct bpf_gen *gen,const char *name,enum bpf_attach_type type); void bpf_gen__record_extern(struct bpf_gen *gen, const char *name, bool is_weak,
tools/lib/bpf/gen_loader.c +2 -1
@@ -1190,7 +1190,7 @@ void bpf_gen__prog_load(struct bpf_gen *gen, }void bpf_gen__map_update_elem (struct bpf_gen *gen,int map_idx,void *pvalue, -__u32 value_size) +__u32 value_size,__u64 flags) { int attr_size = offsetofend(union bpf_attr, flags);int map_update_attr, value, key;
@@ -1198,6 +1198,7 @@ void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue,int zero =0 ;memset (&attr,0 , attr_size); + attr.flags =tgt_endian (flags); value =add_data (gen, pvalue, value_size); key =add_data (gen, &zero,sizeof (zero));
tools/lib/bpf/libbpf.c +56 -12
179 lines not on this page · raw message
Signature · 1 line
2.54.0
Re: [PATCH bpf-next v7 06/11] libbpf: Add support for global percpu data
Quoted 22 lines · Leon Hwang
Add support for global percpu data in libbpf by adding a new ".percpu" section, similar to ".data". It enables efficient handling of percpu global variables in bpf programs.
When generating loader for lightweight skeleton, update the
percpu_arraymap used for global percpu data usingBPF_F_ALL_CPUS, in order to update values across all CPUs using one value slot.Unlike global data, the mmaped data for global percpu data will be marked as read-only after populating the
percpu_arraymap. Thereafter, users can read those initialized percpu data after loading prog. If they want to update the percpu data after loading prog, they have to update thepercpu_arraymap using key=0 instead.Signed-off-by Leon Hwang <[email protected]>--- tools/lib/bpf/bpf_gen_internal.h | 3 +- tools/lib/bpf/gen_loader.c | 3 +- tools/lib/bpf/libbpf.c | 68 ++++++++++++++++++++++++++------ 3 files changed, 60 insertions(+), 14 deletions(-)
[...]
tools/lib/bpf/libbpf.c bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map) · line 5396
Quoted 11 lines
tools/lib/bpf/libbpf.c +7
-0@@ -5353,6 +5387,13 @@ bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map)return err; } map->mmaped = mmaped; + }else if (is_percpu) { +if (mprotect (map->mmaped, mmap_sz,PROT_READ )) { + err = -errno; +pr_warn ("map '%s': failed to mprotect() contents: %s\n" , +bpf_map__name (map),errstr (err)); +return err; + }
hm... do we need to do this? what happens for LIBBPF_MAP_KCONFIG, do we just null out the initial image? should we just do that here? Basically what I am asking is how important it is to access initial per-CPU image after load? What's the realistic use case for that?
Quoted 15 lines
tools/lib/bpf/libbpf.c +3
-5}else if (map->mmaped) {munmap (map->mmaped, mmap_sz); map->mmaped =NULL ;@@ -10806,16 +10847,19 @@ int bpf_map__fd(const struct bpf_map *map)static bool map_uses_real_name (const struct bpf_map *map) { -/* Since libbpf started to support custom .data.* and .rodata.* maps, -* their user-visible name differs from kernel-visible name. Users see -* such map's corresponding ELF section name as a map name. -* This check distinguishes .data/.rodata from .data.* and .rodata.* -* maps to know which name has to be returned to the user. +/* +* Since libbpf started to support custom .data.*, .rodata.* and +* .percpu.* maps, their user-visible name differs from
tools/lib/bpf/libbpf.c int bpf_map__fd(const struct bpf_map *map) · line 10862
Quoted 10 lines
tools/lib/bpf/libbpf.c +5
-0+* kernel-visible name. Users see such map's corresponding ELF section +* name as a map name. This check distinguishes plain .data/.rodata/.percpu +* from .data.*, .rodata.* and .percpu.* to choose which name to return. */ if (map->libbpf_type ==LIBBPF_MAP_DATA &&strcmp (map->real_name,DATA_SEC ) !=0 )return true;if (map->libbpf_type ==LIBBPF_MAP_RODATA &&strcmp (map->real_name,RODATA_SEC ) !=0 )return true; +if (map->libbpf_type ==LIBBPF_MAP_PERCPU &&strcmp (map->real_name,PERCPU_SEC ) !=0 ) +return true;
um... this is extra logic for DATA/RODATA is supposed to be backwards compatible legacy stuff. We shouldn't need this for PERCPU_SEC. It should actually be called just ".percpu", not "<object_name>.percpu".
Quoted 7 lines
tools/lib/bpf/libbpf.c +0
-1return false; } --2.54.0
Re: [PATCH bpf-next v7 06/11] libbpf: Add support for global percpu data
Quoted 1 line · Andrii Nakryiko
On Mon, Jun 22, 2026 at 7:37 AM Leon Hwang <[email protected]> wrote:
[...]
Quoted 8 lines
Quoted 8 lines
tools/lib/bpf/libbpf.c +4
-0@@ -5353,6 +5387,13 @@ bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map)return err; } map->mmaped = mmaped; + }else if (is_percpu) { +if (mprotect (map->mmaped, mmap_sz,PROT_READ )) { + err = -errno; + pr_warn("map '%s': failed to mprotect() contents: %s\n",
Quoted 9 lines
tools/lib/bpf/libbpf.c bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map) · line 5396
Quoted 3 lines
tools/lib/bpf/libbpf.c +3
-0+ bpf_map__name(map), errstr(err)); +return err; + }hm... do we need to do this? what happens for
LIBBPF_MAP_KCONFIG, do we just null out the initial image? should we just do that here? Basically what I am asking is how important it is to access initial per-CPU image after load? What's the realistic use case for that?
This was suggested by you in v3:
https://lore.kernel.org/bpf/CAEf4BzY9KeVeo2+6Ht1v3rL6UdwNxABZCSK1OZ_sD8qhpYZaeQ@mail.gmail.com/
Quoted 21 lines
Quoted 21 lines
tools/lib/bpf/libbpf.c +6
-5}else if (map->mmaped) {munmap (map->mmaped, mmap_sz); map->mmaped =NULL ;@@ -10806,16 +10847,19 @@ int bpf_map__fd(const struct bpf_map *map)static bool map_uses_real_name (const struct bpf_map *map) { -/* Since libbpf started to support custom .data.* and .rodata.* maps, -* their user-visible name differs from kernel-visible name. Users see -* such map's corresponding ELF section name as a map name. -* This check distinguishes .data/.rodata from .data.* and .rodata.* -* maps to know which name has to be returned to the user. +/* +* Since libbpf started to support custom .data.*, .rodata.* and +* .percpu.* maps, their user-visible name differs from +* kernel-visible name. Users see such map's corresponding ELF section +* name as a map name. This check distinguishes plain .data/.rodata/.percpu +* from .data.*, .rodata.* and .percpu.* to choose which name to return. */ if (map->libbpf_type ==LIBBPF_MAP_DATA &&strcmp (map->real_name,DATA_SEC ) !=0 )return true;
Quoted 9 lines
tools/lib/bpf/libbpf.c int bpf_map__fd(const struct bpf_map *map) · line 10862
Quoted 4 lines
tools/lib/bpf/libbpf.c +2
-0if (map->libbpf_type ==LIBBPF_MAP_RODATA &&strcmp (map->real_name,RODATA_SEC ) !=0 )return true; +if (map->libbpf_type ==LIBBPF_MAP_PERCPU &&strcmp (map->real_name,PERCPU_SEC ) !=0 ) +return true;um... this is extra logic for DATA/RODATA is supposed to be backwards compatible legacy stuff. We shouldn't need this for
PERCPU_SEC. It should actually be called just ".percpu", not "<object_name>.percpu".
Understand the backwards compatibility after reading the comment in internal_map_name().
Will drop this hunk, and set "map->name" as ".percpu" in bpf_object__init_internal_map().
Thanks,
Leon
Re: [PATCH bpf-next v7 06/11] libbpf: Add support for global percpu data
Quoted 22 lines · Leon Hwang
Quoted 1 line · Andrii Nakryiko
On Mon, Jun 22, 2026 at 7:37 AM Leon Hwang <[email protected]> wrote:
[...]
Quoted 8 lines
Quoted 8 lines
tools/lib/bpf/libbpf.c +4
-0@@ -5353,6 +5387,13 @@ bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map)return err; } map->mmaped = mmaped; + }else if (is_percpu) { +if (mprotect (map->mmaped, mmap_sz,PROT_READ )) { + err = -errno; + pr_warn("map '%s': failed to mprotect() contents: %s\n",Quoted 9 lines
tools/lib/bpf/libbpf.c bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map) · line 5396
Quoted 3 lines
tools/lib/bpf/libbpf.c +3
-0+ bpf_map__name(map), errstr(err)); +return err; + }hm... do we need to do this? what happens for
LIBBPF_MAP_KCONFIG, do we just null out the initial image? should we just do that here? Basically what I am asking is how important it is to access initial per-CPU image after load? What's the realistic use case for that?
Quoted 3 lines
This was suggested by you in v3:
https://lore.kernel.org/bpf/CAEf4BzY9KeVeo2+6Ht1v3rL6UdwNxABZCSK1OZ_sD8qhpYZaeQ@mail.gmail.com/
ah, the dangling pointer in skeleton that needs clearing, I forgot already :) ok, I don't mind mprotect(), it just was a new case that no other map followed, so I was curious if we can avoid deviations. But that brings back the KCONFIG map question, can you please check what's happening for it? Maybe we should do the same mprotect instead of dangling pointer (if we have dangling pointer, of course).
Quoted 39 lines
Quoted 21 lines
Quoted 21 lines
tools/lib/bpf/libbpf.c +6
-5}else if (map->mmaped) {munmap (map->mmaped, mmap_sz); map->mmaped =NULL ;@@ -10806,16 +10847,19 @@ int bpf_map__fd(const struct bpf_map *map)static bool map_uses_real_name (const struct bpf_map *map) { -/* Since libbpf started to support custom .data.* and .rodata.* maps, -* their user-visible name differs from kernel-visible name. Users see -* such map's corresponding ELF section name as a map name. -* This check distinguishes .data/.rodata from .data.* and .rodata.* -* maps to know which name has to be returned to the user. +/* +* Since libbpf started to support custom .data.*, .rodata.* and +* .percpu.* maps, their user-visible name differs from +* kernel-visible name. Users see such map's corresponding ELF section +* name as a map name. This check distinguishes plain .data/.rodata/.percpu +* from .data.*, .rodata.* and .percpu.* to choose which name to return. */ if (map->libbpf_type ==LIBBPF_MAP_DATA &&strcmp (map->real_name,DATA_SEC ) !=0 )return true;Quoted 9 lines
tools/lib/bpf/libbpf.c int bpf_map__fd(const struct bpf_map *map) · line 10862
Quoted 4 lines
tools/lib/bpf/libbpf.c +2
-0if (map->libbpf_type ==LIBBPF_MAP_RODATA &&strcmp (map->real_name,RODATA_SEC ) !=0 )return true; +if (map->libbpf_type ==LIBBPF_MAP_PERCPU &&strcmp (map->real_name,PERCPU_SEC ) !=0 ) +return true;um... this is extra logic for DATA/RODATA is supposed to be backwards compatible legacy stuff. We shouldn't need this for
PERCPU_SEC. It should actually be called just ".percpu", not "<object_name>.percpu".Understand the backwards compatibility after reading the comment in
internal_map_name().Will drop this hunk, and set "
map->name" as ".percpu" inbpf_object__init_internal_map().Thanks,
Leon
Re: [PATCH bpf-next v7 06/11] libbpf: Add support for global percpu data
Quoted 6 lines · Andrii Nakryiko
Quoted 4 lines · Leon Hwang
This was suggested by you in v3:
https://lore.kernel.org/bpf/CAEf4BzY9KeVeo2+6Ht1v3rL6UdwNxABZCSK1OZ_sD8qhpYZaeQ@mail.gmail.com/
Quoted 7 lines
ah, the dangling pointer in skeleton that needs clearing, I forgot already :) ok, I don't mind
mprotect(), it just was a new case that no other map followed, so I was curious if we can avoid deviations. But that brings back the KCONFIG map question, can you please check what's happening for it? Maybe we should do the same mprotect instead of dangling pointer (if we have dangling pointer, of course).
Looking at prog_tests/skeleton.c, skel->kconfig is NULL before loading, then becomes a read-only mmaped pointer after loading.
So, skel->kconfig is not a dangling pointer after loading.
It looks good to mprotect() skel->percpu.
Thanks,
Leon
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
} else if (map->mmaped) { munmap(map->mmaped, mmap_sz); map->mmaped = NULL;(quoted diff) +0
-0@@ -10806,16 +10847,19 @@ int bpf_map__fd(const struct bpf_map *map)
PATCH v7 7/11 bpftool: Generate skeleton for global percpu data
In reply to Leon HwangEnhance bpftool to generate skeletons that properly handle global percpu variables. The generated skeleton now includes a dedicated structure for percpu data, allowing users to initialize and access percpu variables more efficiently.
For global percpu variables, the skeleton now includes a nested structure, e.g.:
struct test_global_percpu_data {
struct bpf_object_skeleton *skeleton;
struct bpf_object *obj;
struct {
struct bpf_map *percpu;
} maps;
// ...
struct test_global_percpu_data__percpu {
int data;
char run;
struct {
char set;
int i;
int nums[7];
} struct_data;
int nums[7];
} *percpu;
// ...
}; * The "struct test_global_percpu_data__percpu *percpu" points to
initialized data, which is actually "maps.percpu->mmaped".
* Before loading the skeleton, updating the
"struct test_global_percpu_data__percpu *percpu" modifies the initial
value of the corresponding global percpu variables.
* After loading the skeleton, "maps.percpu->mmaped" has been marked as
read-only in libbpf. If users want to update the global percpu
variables, they have to update the "maps.percpu" map instead.
* For lightweight skeleton, "lskel->percpu" will be protected by
"mprotect(p, sz, PROT_READ)".* For subskeleton, those variables of global percpu data will be skipped.
--- tools/bpf/bpftool/gen.c | 43 +++++++++++++++++++++++++++-------- tools/lib/bpf/skel_internal.h | 24 +++++++++++++++++-- 2 files changed, 56 insertions(+), 11 deletions(-)
tools/bpf/bpftool/gen.c +34 -9
@@ -92,7 +92,7 @@ static void get_header_guard(char *guard, const char *obj_name, const char *suffstatic bool get_map_ident (const struct bpf_map *map,char *buf,size_t buf_sz) { -static const char *sfxs[] = {".data" ,".rodata" ,".bss" ,".kconfig" }; +static const char *sfxs[] = {".data" ,".rodata" ,".bss" ,".percpu" ,".kconfig" };const char *name =bpf_map__name (map);int i, n;
@@ -117,7 +117,7 @@ static bool get_map_ident(const struct bpf_map *map, char *buf, size_t buf_sz)static bool get_datasec_ident (const char *sec_name,char *buf,size_t buf_sz) { -static const char *pfxs[] = {".data" ,".rodata" ,".bss" ,".kconfig" }; +static const char *pfxs[] = {".data" ,".rodata" ,".bss" ,".percpu" ,".kconfig" };int i, n;/* recognize hard coded LLVM section name */
@@ -254,6 +254,11 @@ static const struct btf_type *find_type_for_map(struct btf *btf, const char *mapreturn NULL ; } +static bool bpf_map_is_percpu_data (const struct bpf_map *map) +{ +return bpf_map__is_internal (map) &&bpf_map__type (map) ==BPF_MAP_TYPE_PERCPU_ARRAY ; +} +static bool is_mmapable_map (const struct bpf_map *map,char *buf,size_t sz) {size_t tmp_sz;
@@ -263,13 +268,12 @@ static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)return true; } -if (!bpf_map__is_internal (map) || !(bpf_map__map_flags (map) &BPF_F_MMAPABLE )) -return false; - -if (!get_map_ident (map, buf, sz)) -return false; +if (bpf_map__is_internal (map) && + ((bpf_map__map_flags (map) &BPF_F_MMAPABLE ) ||bpf_map_is_percpu_data (map)) && +get_map_ident (map, buf, sz)) +return true; -return true; +return false; }static int codegen_datasecs (struct bpf_object *obj,const char *obj_name)
@@ -343,6 +347,9 @@ static int codegen_subskel_datasecs(struct bpf_object *obj, const char *obj_nameif (!is_mmapable_map (map, map_ident,sizeof (map_ident)))continue ; +if (bpf_map_is_percpu_data (map)) +continue ; + sec =find_type_for_map (btf, map_ident);if (!sec)continue ;
@@ -669,7 +676,7 @@ static void codegen_destroy(struct bpf_object *obj, const char *obj_name)if (!get_map_ident (map, ident,sizeof (ident)))continue ;if (bpf_map__is_internal (map) && - (bpf_map__map_flags (map) &BPF_F_MMAPABLE )) + ((bpf_map__map_flags (map) &BPF_F_MMAPABLE ) ||bpf_map_is_percpu_data (map)))printf ("\tskel_free_map_data(skel->%1$s, skel->maps.%1$s.initial_value, %2$zu);\n" , ident,bpf_map_mmap_sz (map)); codegen("\
@@ -850,6 +857,20 @@ static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *hif (!is_mmapable_map (map, ident,sizeof (ident)))continue ; +if (bpf_map_is_percpu_data (map)) { +codegen ("\ +\n\ +err = skel_protect_map_data(skel->%1$s, &skel->maps.%1$s.initial_value, %2$zd);\n\ +if (err) \n\ +return err; \n\ +#ifdef __KERNEL__ \n\ +skel->%1$s = NULL; \n\ +#endif \n\ +" , + ident,bpf_map_mmap_sz (map)); +continue ; + } +if (bpf_map__map_flags (map) &BPF_F_RDONLY_PROG ) mmap_flags ="PROT_READ" ;else
@@ -1740,6 +1761,8 @@ static int do_subskeleton(int argc, char **argv)if (!is_mmapable_map (map, ident,sizeof (ident)))continue ; +if (bpf_map_is_percpu_data (map)) +continue ; map_type_id =bpf_map__btf_value_type_id (map);if (map_type_id <=0 ) {
@@ -1863,6 +1886,8 @@ static int do_subskeleton(int argc, char **argv)bpf_object__for_each_map (map, obj) {if (!is_mmapable_map (map, ident,sizeof (ident)))continue ; +if (bpf_map_is_percpu_data (map)) +continue ; map_type_id =bpf_map__btf_value_type_id (map);if (map_type_id <=0 )
tools/lib/bpf/skel_internal.h +22 -2
@@ -135,8 +135,10 @@ static inline void skel_free_map_data(void *p, __u64 addr, size_t sz) {if (addr != ~0ULL )kvfree (p); -/* When addr == ~0ULL the 'p' points to -* ((struct bpf_array *)map)->value. See skel_finalize_map_data. +/* +* When addr == ~0ULL the init buffer has already been released. +* For skel_finalize_map_data(), 'p' points to +* ((struct bpf_array *)map)->value. */ }
@@ -174,6 +176,15 @@ static inline void *skel_finalize_map_data(__u64 *init_val, size_t mmap_sz, int return addr; } +static inline int skel_protect_map_data (void *p,__u64 *init_val,size_t sz) +{ + (void )sz; + +kvfree (p); + *init_val = ~0ULL ; +return 0 ; +} +#else static inline void *skel_alloc (size_t size)
@@ -212,6 +223,15 @@ static inline void *skel_finalize_map_data(__u64 *init_val, size_t mmap_sz, intreturn NULL ; return addr; } + +static inline int skel_protect_map_data (void *p,__u64 *init_val,size_t sz) +{ + (void )init_val; + +if (mprotect (p, sz,PROT_READ )) +return -errno; +return 0 ; +}#endif static inline int skel_closenz (int fd)
Signature · 1 line
2.54.0
Re: [PATCH bpf-next v7 07/11] bpftool: Generate skeleton for global percpu data
Quoted 50 lines · Leon Hwang
Enhance bpftool to generate skeletons that properly handle global percpu variables. The generated skeleton now includes a dedicated structure for percpu data, allowing users to initialize and access percpu variables more efficiently.
For global percpu variables, the skeleton now includes a nested structure, e.g.:
struct test_global_percpu_data { struct bpf_object_skeleton *skeleton; struct bpf_object *obj; struct { struct bpf_map *percpu; } maps; // ... struct test_global_percpu_data__percpu { int data; char run; struct { char set; int i; int nums[7]; } struct_data; int nums[7]; } *percpu; // ... };* The "struct
test_global_percpu_data__percpu*percpu" points toinitialized data, which is actually "maps.percpu->mmaped". * Before loading the skeleton, updating the "struct test_global_percpu_data__percpu *percpu" modifies the initial value of the corresponding global percpu variables. * After loading the skeleton, "maps.percpu->mmaped" has been marked as read-only in libbpf. If users want to update the global percpu variables, they have to update the "maps.percpu" map instead. * For lightweight skeleton, "lskel->percpu" will be protected by "mprotect(p, sz, PROT_READ)".* For subskeleton, those variables of global percpu data will be skipped.
Assisted-by Codex:gpt-5.5-xhighSigned-off-by Leon Hwang <[email protected]>--- tools/bpf/bpftool/gen.c | 43 +++++++++++++++++++++++++++-------- tools/lib/bpf/skel_internal.h | 24 +++++++++++++++++-- 2 files changed, 56 insertions(+), 11 deletions(-)
[...]
Quoted 4 lines
tools/bpf/bpftool/gen.c +0
-0@@ -263,13 +268,12 @@ static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)return true; }
tools/bpf/bpftool/gen.c static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz) · line 275
Quoted 10 lines
tools/bpf/bpftool/gen.c +4
-5-if (!bpf_map__is_internal (map) || !(bpf_map__map_flags (map) &BPF_F_MMAPABLE )) -return false; - -if (!get_map_ident (map, buf, sz)) -return false; +if (bpf_map__is_internal (map) && + ((bpf_map__map_flags (map) &BPF_F_MMAPABLE ) ||bpf_map_is_percpu_data (map)) && +get_map_ident (map, buf, sz)) +return true;
just add if (bpf_map_is_percpu_data(map) return true;? maybe also move get_map_ident check a bit earlier. I think that will be a bit cleaner, this condition is quite hard to follow
Quoted 16 lines
tools/bpf/bpftool/gen.c +4
-1-return true; +return false; }static int codegen_datasecs (struct bpf_object *obj,const char *obj_name)@@ -343,6 +347,9 @@ static int codegen_subskel_datasecs(struct bpf_object *obj, const char *obj_nameif (!is_mmapable_map (map, map_ident,sizeof (map_ident)))continue ; +if (bpf_map_is_percpu_data (map)) +continue ; + sec =find_type_for_map (btf, map_ident);if (!sec)continue ;
tools/bpf/bpftool/gen.c static void codegen_destroy(struct bpf_object *obj, const char *obj_name) · line 679
Quoted 5 lines
tools/bpf/bpftool/gen.c +1
-1@@ -669,7 +676,7 @@ static void codegen_destroy(struct bpf_object *obj, const char *obj_name)if (!get_map_ident (map, ident,sizeof (ident)))continue ;if (bpf_map__is_internal (map) && - (bpf_map__map_flags (map) &BPF_F_MMAPABLE )) + ((bpf_map__map_flags (map) &BPF_F_MMAPABLE ) ||bpf_map_is_percpu_data (map)))
also we can add a helper to check if it's an internal map with a dedicated data section in the skeleton (too lazy to think of a good name right now.. ;)
Quoted 3 lines
tools/bpf/bpftool/gen.c +0
-0printf ("\tskel_free_map_data(skel->%1$s, skel->maps.%1$s.initial_value, %2$zu);\n" , ident,bpf_map_mmap_sz (map)); codegen("\
[...]
Re: [PATCH bpf-next v7 07/11] bpftool: Generate skeleton for global percpu data
Quoted 1 line · Andrii Nakryiko
On Mon, Jun 22, 2026 at 7:37 AM Leon Hwang <[email protected]> wrote:
[...]
Quoted 4 lines
Quoted 4 lines
tools/bpf/bpftool/gen.c +0
-0@@ -263,13 +268,12 @@ static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)return true; }
Quoted 15 lines
tools/bpf/bpftool/gen.c static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz) · line 275
Quoted 10 lines
tools/bpf/bpftool/gen.c +4
-5-if (!bpf_map__is_internal (map) || !(bpf_map__map_flags (map) &BPF_F_MMAPABLE )) -return false; - -if (!get_map_ident (map, buf, sz)) -return false; +if (bpf_map__is_internal (map) && + ((bpf_map__map_flags (map) &BPF_F_MMAPABLE ) ||bpf_map_is_percpu_data (map)) && +get_map_ident (map, buf, sz)) +return true;just add
if (bpf_map_is_percpu_data(map) return true;? maybe also moveget_map_identcheck a bit earlier. I think that will be a bit cleaner, this condition is quite hard to follow
Makes sense.
With adding the helper bpf_map_is_skel_data(), will this change looks more readable?
+static bool bpf_map_is_skel_data(const struct bpf_map *map) +{
+ return bpf_map__is_internal(map) &&
+ ((bpf_map__map_flags(map) & BPF_F_MMAPABLE) ||
+ bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY);
+}
+
static bool is_mmapable_map(const struct bpf_map *map, char *buf,size_t sz)
{
size_t tmp_sz;
@@ -263,7 +274,7 @@ static bool is_mmapable_map(const struct bpf_map
*map, char *buf, size_t sz)
return true;
}- if (!bpf_map__is_internal(map) || !(bpf_map__map_flags(map) &BPF_F_MMAPABLE))
+ if (!bpf_map_is_skel_data(map))
return false;
if (!get_map_ident(map, buf, sz))Quoted 9 lines
Quoted 9 lines
tools/bpf/bpftool/gen.c +1
-1-return true; +return false; }static int codegen_datasecs (struct bpf_object *obj,const char *obj_name)@@ -343,6 +347,9 @@ static int codegen_subskel_datasecs(struct bpf_object *obj, const char *obj_nameif (!is_mmapable_map (map, map_ident,sizeof (map_ident)))continue ;
Quoted 17 lines
tools/bpf/bpftool/gen.c static void codegen_destroy(struct bpf_object *obj, const char *obj_name) · line 679
Quoted 12 lines
tools/bpf/bpftool/gen.c +4
-1+if (bpf_map_is_percpu_data (map)) +continue ; + sec =find_type_for_map (btf, map_ident);if (!sec)continue ;@@ -669,7 +676,7 @@ static void codegen_destroy(struct bpf_object *obj, const char *obj_name)if (!get_map_ident (map, ident,sizeof (ident)))continue ;if (bpf_map__is_internal (map) && - (bpf_map__map_flags (map) &BPF_F_MMAPABLE )) + ((bpf_map__map_flags (map) &BPF_F_MMAPABLE ) ||bpf_map_is_percpu_data (map)))also we can add a helper to check if it's an internal map with a dedicated data section in the skeleton (too lazy to think of a good name right now.. ;)
How about bpf_map_is_skel_data()?
Then, this 'if' will be updated as 'if (bpf_map_is_skel_data(map))'.
Thanks,
Leon
Quoted 5 lines
Quoted 3 lines
tools/bpf/bpftool/gen.c +0
-0printf ("\tskel_free_map_data(skel->%1$s, skel->maps.%1$s.initial_value, %2$zu);\n" , ident,bpf_map_mmap_sz (map)); codegen("\[...]
Re: [PATCH bpf-next v7 07/11] bpftool: Generate skeleton for global percpu data
Quoted 30 lines · Leon Hwang
Quoted 1 line · Andrii Nakryiko
On Mon, Jun 22, 2026 at 7:37 AM Leon Hwang <[email protected]> wrote:
[...]
Quoted 4 lines
Quoted 4 lines
tools/bpf/bpftool/gen.c +0
-0@@ -263,13 +268,12 @@ static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)return true; }Quoted 15 lines
tools/bpf/bpftool/gen.c static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz) · line 275
Quoted 10 lines
tools/bpf/bpftool/gen.c +4
-5-if (!bpf_map__is_internal (map) || !(bpf_map__map_flags (map) &BPF_F_MMAPABLE )) -return false; - -if (!get_map_ident (map, buf, sz)) -return false; +if (bpf_map__is_internal (map) && + ((bpf_map__map_flags (map) &BPF_F_MMAPABLE ) ||bpf_map_is_percpu_data (map)) && +get_map_ident (map, buf, sz)) +return true;just add
if (bpf_map_is_percpu_data(map) return true;? maybe also moveget_map_identcheck a bit earlier. I think that will be a bit cleaner, this condition is quite hard to followMakes sense.
With adding the helper
bpf_map_is_skel_data(), will this change looks more readable?
Review
Quoted 5 lines
(quoted diff) +5
-0+static bool bpf_map_is_skel_data(const struct bpf_map *map) +{ + return bpf_map__is_internal(map) && + ((bpf_map__map_flags(map) & BPF_F_MMAPABLE) || + bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY);
if (!bpf_map__is_internal(map))
return false;
return (bpf_map__map_flags(map) & BPF_F_MMAPABLE) || bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY;
or just split that last return into two if (...) return true);
both work for me
Quoted 46 lines
tools/bpf/bpftool/gen.c +2
-0+} + static bool is_mmapable_map(const struct bpf_map *map, char *buf,
size_tsz){ size_t tmp_sz;(quoted diff) +1
-1@@ -263,7 +274,7 @@ static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz) return true; } - if (!bpf_map__is_internal(map) || !(bpf_map__map_flags(map) & BPF_F_MMAPABLE)) + if (!bpf_map_is_skel_data(map)) return false;if (!get_map_ident(map, buf, sz))Quoted 9 lines
Quoted 9 lines
tools/bpf/bpftool/gen.c +1
-1-return true; +return false; }static int codegen_datasecs (struct bpf_object *obj,const char *obj_name)@@ -343,6 +347,9 @@ static int codegen_subskel_datasecs(struct bpf_object *obj, const char *obj_nameif (!is_mmapable_map (map, map_ident,sizeof (map_ident)))continue ;Quoted 17 lines
tools/bpf/bpftool/gen.c static void codegen_destroy(struct bpf_object *obj, const char *obj_name) · line 679
Quoted 12 lines
tools/bpf/bpftool/gen.c +4
-1+if (bpf_map_is_percpu_data (map)) +continue ; + sec =find_type_for_map (btf, map_ident);if (!sec)continue ;@@ -669,7 +676,7 @@ static void codegen_destroy(struct bpf_object *obj, const char *obj_name)if (!get_map_ident (map, ident,sizeof (ident)))continue ;if (bpf_map__is_internal (map) && - (bpf_map__map_flags (map) &BPF_F_MMAPABLE )) + ((bpf_map__map_flags (map) &BPF_F_MMAPABLE ) ||bpf_map_is_percpu_data (map)))also we can add a helper to check if it's an internal map with a dedicated data section in the skeleton (too lazy to think of a good name right now.. ;)
Quoted 3 lines
How about
bpf_map_is_skel_data()?Then, this 'if' will be updated as 'if (
bpf_map_is_skel_data(map))'.
yeah, I like the name
Quoted 10 lines
Thanks,
LeonQuoted 5 lines
Quoted 3 lines
tools/bpf/bpftool/gen.c +0
-0printf ("\tskel_free_map_data(skel->%1$s, skel->maps.%1$s.initial_value, %2$zu);\n" , ident,bpf_map_mmap_sz (map)); codegen("\[...]
Re: [PATCH bpf-next v7 07/11] bpftool: Generate skeleton for global percpu data
Quoted 15 lines · Andrii Nakryiko
Review
Quoted 12 lines · Leon Hwang
Makes sense.
With adding the helper
bpf_map_is_skel_data(), will this change looks more readable?(quoted diff) +5
-0+static bool bpf_map_is_skel_data(const struct bpf_map *map) +{ + return bpf_map__is_internal(map) && + ((bpf_map__map_flags(map) & BPF_F_MMAPABLE) || + bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY);if (!bpf_map__is_internal(map))
Quoted 7 lines
return false;return (
bpf_map__map_flags(map)&BPF_F_MMAPABLE) ||bpf_map__type(map)==BPF_MAP_TYPE_PERCPU_ARRAY;or just split that last return into two if (...) return true);
Will follow your suggestion, and use two if (...) return true.
Thanks,
Leon
Quoted 3 lines
both work for me
Quoted 1 line
[...]
PATCH v7 8/11 selftests/bpf: Add tests to verify global percpu data
In reply to Leon HwangIf the arch, like s390x, does not support percpu insn, these cases won't test global percpu data by checking FEAT_PERCPU_DATA support.
The following APIs have been tested for global percpu data:
1. bpf_map__set_initial_value()
2. bpf_map__initial_value()
3. generated percpu struct pointer pointing to internal map's mmaped data
4. bpf_map__lookup_elem() for global percpu data map
At the same time, the case is also tested with 'bpftool gen skeleton -L'.
Add a test to verify that the live vars of subskel won't include the vars for global percpu data.
--- tools/testing/selftests/bpf/Makefile | 2 +- .../bpf/prog_tests/global_data_init.c | 184 ++++++++++++++++++ .../bpf/prog_tests/global_percpu_subskel.c | 37 ++++ .../bpf/progs/test_global_percpu_data.c | 31 +++ 4 files changed, 253 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c create mode 100644 tools/testing/selftests/bpf/progs/test_global_percpu_data.c
tools/testing/selftests/bpf/Makefile +1 -1
@@ -533,7 +533,7 @@ LSKELS_SIGNED := fentry_test.c fexit_test.c atomics.c # Generate both light skeleton and libbpf skeleton for these LSKELS_EXTRA := test_ksyms_module.c test_ksyms_weak.c kfunc_call_test.c \ - kfunc_call_test_subprog.c + kfunc_call_test_subprog.c test_global_percpu_data.c SKEL_BLACKLIST += $$(LSKELS) $$(LSKELS_SIGNED) test_static_linked.skel.h-deps := test_static_linked1.bpf.o test_static_linked2.bpf.o
tools/testing/selftests/bpf/prog_tests/global_data_init.c +184 -0
192 lines not on this page · raw message
tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c new file +37 -0
@@ -0,0 +1,37 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <test_progs.h> +#include "test_global_percpu_data.subskel.h" + +void test_global_percpu_data_subskel (void ) +{ +struct test_global_percpu_data *subskel =NULL ; +struct bpf_object *obj; +int i; + + obj =bpf_object__open_file ("./test_global_percpu_data.bpf.o" ,NULL ); +if (!ASSERT_OK_PTR (obj,"bpf_object__open_file" )) +return ; + + subskel =test_global_percpu_data__open (obj); +if (!ASSERT_OK_PTR (subskel,"test_global_percpu_data__open" )) + goto out; + +if (!ASSERT_OK_PTR (subskel->subskel,"subskel" )) + goto out; +if (!ASSERT_OK_PTR (subskel->maps.percpu,"maps.percpu" )) + goto out; +ASSERT_EQ (bpf_map__type (subskel->maps.percpu),BPF_MAP_TYPE_PERCPU_ARRAY , +"percpu_map_type" ); +ASSERT_GT (subskel->subskel->var_cnt,0 ,"var_cnt" ); + +for (i =0 ; i < subskel->subskel->var_cnt; i++) { +const struct bpf_var_skeleton *var; + + var = (void *) subskel->subskel->vars + i * subskel->subskel->var_skel_sz; +ASSERT_NEQ (var->map, &subskel->maps.percpu,"var" ); + } + +out: +test_global_percpu_data__destroy (subskel); +bpf_object__close (obj); +}
tools/testing/selftests/bpf/progs/test_global_percpu_data.c new file +31 -0
@@ -0,0 +1,31 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +int data SEC(".percpu") = -1; +int nums[7] SEC(".percpu"); +char run SEC(".percpu") = 0; +struct { +char set; +int i; +int nums[7 ]; +} struct_data SEC(".percpu") = { + .set = 0, + .i =-1 , +}; + +SEC("raw_tp/task_rename") +__auxiliary +int update_percpu_data(void *ctx) +{ + struct_data.nums[6 ] =0xc0de ; + struct_data.set =1 ; + struct_data.i =1 ; + nums[6 ] =0xc0de ; + data =1 ; + run =1 ; +return 0 ; +} + +char _license[] SEC("license") = "GPL";
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH bpf-next v7 08/11] selftests/bpf: Add tests to verify global percpu data
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The subskel test incorrectly asserts that `var_cnt > 0`, causing the test to reliably fail. --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The subskel test incorrectly asserts that var_cnt > 0, causing the test to reliably fail. --
commit 55ee2a40d2c9fa6afdbccf30892e0d5b8f148c44
Author: Leon Hwang <[email protected]>
selftests/bpf: Add tests to verify global percpu data
This commit adds tests to verify global percpu data using libbpf APIs. It also introduces a subskeleton test to ensure that the live variables of a subskel do not include variables for global percpu data.
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c new file +0
-0
[ ... ]
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c new file +5
-0+void test_global_percpu_data_subskel (void ) +{ +struct test_global_percpu_data *subskel =NULL ; +struct bpf_object *obj; +int i;
[ ... ]
tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c line 25
Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c new file +7
-0+if (!ASSERT_OK_PTR (subskel->subskel,"subskel" )) + goto out; +if (!ASSERT_OK_PTR (subskel->maps.percpu,"maps.percpu" )) + goto out; +ASSERT_EQ (bpf_map__type (subskel->maps.percpu),BPF_MAP_TYPE_PERCPU_ARRAY , +"percpu_map_type" ); +ASSERT_GT (subskel->subskel->var_cnt,0 ,"var_cnt" );
[Severity: Medium]
Will this assertion cause the test to unconditionally fail?
Looking at test_global_percpu_data.c, all defined variables are placed in the .percpu section. Since bpftool explicitly skips .percpu maps when generating subskeletons, there appear to be no non-percpu maps with BTF variables present.
If that is the case, subskel->subskel->var_cnt will be exactly 0, causing this check to evaluate to ASSERT_GT(0, 0) and reliably fail the test.
Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c new file +3
-0+ +for (i =0 ; i < subskel->subskel->var_cnt; i++) { +const struct bpf_var_skeleton *var;
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
Re: [PATCH bpf-next v7 08/11] selftests/bpf: Add tests to verify global percpu data
On 22/6/26 22:57, [email protected] wrote:
[...]
Quoted 12 lines
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c new file +0
-0[ ... ]
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c new file +5
-0+void test_global_percpu_data_subskel (void ) +{ +struct test_global_percpu_data *subskel =NULL ; +struct bpf_object *obj; +int i;[ ... ]
Quoted 11 lines
tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c line 25
Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c new file +7
-0+if (!ASSERT_OK_PTR (subskel->subskel,"subskel" )) + goto out; +if (!ASSERT_OK_PTR (subskel->maps.percpu,"maps.percpu" )) + goto out; +ASSERT_EQ (bpf_map__type (subskel->maps.percpu),BPF_MAP_TYPE_PERCPU_ARRAY , +"percpu_map_type" ); +ASSERT_GT (subskel->subskel->var_cnt,0 ,"var_cnt" );[Severity: Medium]
Will this assertion cause the test to unconditionally fail?
Luckily, no.
Quoted 8 lines
Looking at
test_global_percpu_data.c, all defined variables are placed in the .percpu section. Since bpftool explicitly skips .percpu maps when generating subskeletons, there appear to be no non-percpu maps with BTF variables present.If that is the case,
subskel->subskel->var_cntwill be exactly 0, causing this check to evaluate toASSERT_GT(0, 0)and reliably fail the test.
Looking at test_global_percpu_data.c, there are 5 non-percpu global variables, which were added in patch #11.
volatile const __u32 num_cpus = 0;
volatile const int offsetof_num;
volatile const int elem_sz;__u32 percpu_data_sum = 0;
bool run_iter = false;
So, subskel->subskel->var_cnt will be 5.
In the generated test_global_percpu_data.subskel.h:
/* vars */
s->var_cnt = 5;Thanks,
Leon
Quoted 5 lines
Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c new file +3
-0+ +for (i =0 ; i < subskel->subskel->var_cnt; i++) { +const struct bpf_var_skeleton *var;[ ... ]
[email protected]
AI reviewRe: [PATCH bpf-next v7 08/11] selftests/bpf: Add tests to verify global percpu data
In reply to Leon HwangThis isn't a bug, but num_cpus is only read by the ASSERT_GT guard and is never used afterwards; the per-CPU loop is driven entirely by num_online from parse_cpu_mask_file(). Is the…
Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +0
-0
[ ... ]
Quoted 15 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +14
-0@@ -60,3 +63,184 @@ void test_global_data_init(void) + +static void test_global_percpu_data_init (void ) +{ +struct test_global_percpu_data__percpu init_value = {}; +struct test_global_percpu_data__percpu *init_data; +int key, prog_fd, err, num_cpus, num_online, i; +struct test_global_percpu_data *skel =NULL ; +__u64 args[2 ] = {0x1234ULL ,0x5678ULL }; +size_t init_data_sz; +struct bpf_map *map; +bool *online; + LIBBPF_OPTS(bpf_test_run_opts, topts, + .ctx_in = args, + .ctx_size_in = sizeof(args),
tools/testing/selftests/bpf/prog_tests/global_data_init.c void test_global_data_init(void) · line 86
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +10
-0+ .flags = BPF_F_TEST_RUN_ON_CPU, + ); + + num_cpus =libbpf_num_possible_cpus (); +if (!ASSERT_GT (num_cpus,0 ,"libbpf_num_possible_cpus" )) +return ; + + err =parse_cpu_mask_file ("/sys/devices/system/cpu/online" , &online, &num_online); +if (!ASSERT_OK (err,"parse_cpu_mask_file" )) +return ;
This isn't a bug, but num_cpus is only read by the ASSERT_GT guard and is never used afterwards; the per-CPU loop is driven entirely by num_online from parse_cpu_mask_file(). Is the libbpf_num_possible_cpus() check needed here, or is num_online sufficient? The same pattern appears in test_global_percpu_data_lskel().
[ ... ]
Quoted 2 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +2
-0+static void test_global_percpu_data_lskel (void ) +{
[ ... ]
tools/testing/selftests/bpf/prog_tests/global_data_init.c void test_global_data_init(void) · line 228
Quoted 12 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +12
-0+ flags = ((__u64 ) i <<32 ) |BPF_F_CPU ; + err =bpf_map_lookup_elem_flags (map_fd, &key, &data, flags); +if (!ASSERT_OK (err,"bpf_map_lookup_elem_flags" )) + goto out; + +ASSERT_EQ (data.data,1 ,"data.data" ); +ASSERT_TRUE (data.run,"data.run" ); +ASSERT_EQ (data.nums[6 ],0xc0de ,"data.nums[6]" ); +ASSERT_EQ (data.struct_data.i,1 ,"struct_data.i" ); +ASSERT_TRUE (data.struct_data.set,"struct_data.set" ); +ASSERT_EQ (data.struct_data.nums[6 ],0xc0de ,"struct_data.nums[6]" ); + }
This isn't a bug, but this post-run assertion block duplicates the one in test_global_percpu_data_init() almost verbatim; the two loops share the same online-CPU iteration, the same bpf_prog_test_run_opts setup, and the same six data.* checks, differing only in the lookup call (bpf_map_lookup_elem_flags() vs bpf_map__lookup_elem()). Could the six data.* checks be factored into a shared helper taking the looked-up value?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/27961789929
Re: [PATCH bpf-next v7 08/11] selftests/bpf: Add tests to verify global percpu data
Quoted 30 lines · On 22/6/26 23:24, [email protected] wrote:
Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +0
-0[ ... ]
Quoted 23 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +22
-0@@ -60,3 +63,184 @@ void test_global_data_init(void) + +static void test_global_percpu_data_init (void ) +{ +struct test_global_percpu_data__percpu init_value = {}; +struct test_global_percpu_data__percpu *init_data; +int key, prog_fd, err, num_cpus, num_online, i; +struct test_global_percpu_data *skel =NULL ; +__u64 args[2 ] = {0x1234ULL ,0x5678ULL }; +size_t init_data_sz; +struct bpf_map *map; +bool *online; +LIBBPF_OPTS (bpf_test_run_opts, topts, + .ctx_in = args, + .ctx_size_in = sizeof(args), + .flags = BPF_F_TEST_RUN_ON_CPU, + ); + + num_cpus =libbpf_num_possible_cpus (); +if (!ASSERT_GT (num_cpus,0 ,"libbpf_num_possible_cpus" )) +return ; + + err =parse_cpu_mask_file ("/sys/devices/system/cpu/online" , &online, &num_online);
Quoted 9 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c void test_global_data_init(void) · line 86
Quoted 2 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +2
-0+if (!ASSERT_OK (err,"parse_cpu_mask_file" )) +return ;This isn't a bug, but
num_cpusis only read by theASSERT_GTguard and is never used afterwards; the per-CPU loop is driven entirely bynum_onlinefromparse_cpu_mask_file(). Is thelibbpf_num_possible_cpus()check needed here, or isnum_onlinesufficient? The same pattern appears intest_global_percpu_data_lskel().
Will drop the unused "num_cpus".
Quoted 18 lines
[ ... ]
Quoted 2 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +2
-0+static void test_global_percpu_data_lskel (void ) +{[ ... ]
Quoted 11 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +11
-0+ flags = ((__u64 ) i <<32 ) |BPF_F_CPU ; + err =bpf_map_lookup_elem_flags (map_fd, &key, &data, flags); +if (!ASSERT_OK (err,"bpf_map_lookup_elem_flags" )) + goto out; + +ASSERT_EQ (data.data,1 ,"data.data" ); +ASSERT_TRUE (data.run,"data.run" ); +ASSERT_EQ (data.nums[6 ],0xc0de ,"data.nums[6]" ); +ASSERT_EQ (data.struct_data.i,1 ,"struct_data.i" ); +ASSERT_TRUE (data.struct_data.set,"struct_data.set" ); +ASSERT_EQ (data.struct_data.nums[6 ],0xc0de ,"struct_data.nums[6]" );
Quoted 9 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c
Quoted 1 line
tools/testing/selftests/bpf/prog_tests/global_data_init.c +1
-0+ }This isn't a bug, but this post-run assertion block duplicates the one in
test_global_percpu_data_init()almost verbatim; the two loops share the same online-CPU iteration, the samebpf_prog_test_run_optssetup, and the same six data.* checks, differing only in the lookup call (bpf_map_lookup_elem_flags()vsbpf_map__lookup_elem()). Could the six data.* checks be factored into a shared helper taking the looked-up value?
Good catch.
Will factor out a helper for the whole 'for' block.
Thanks,
Leon
PATCH v7 9/11 selftests/bpf: Add test to verify accessing rdonly percpu_array
In reply to Leon HwangSince percpu_array supports direct-read, it should not break accessing rdonly percpu_array.
Add a test to verify that adding '.map_direct_value_addr' to percpu_array won't break the case.
--- .../bpf/prog_tests/global_data_init.c | 44 +++++++++++++++++++ 1 file changed, 44 insertions(+)
tools/testing/selftests/bpf/prog_tests/global_data_init.c +44 -0
@@ -232,6 +232,48 @@ static void test_global_percpu_data_lskel(void)free (online); } +static void test_global_percpu_data_rdonly_direct_read (void ) +{ +LIBBPF_OPTS (bpf_map_create_opts, map_opts, + .map_flags = BPF_F_RDONLY_PROG, + ); +struct bpf_insn insns[] = { +BPF_ST_MEM (BPF_W ,BPF_REG_10 ,-8 ,0 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-8 ), +BPF_LD_MAP_FD (BPF_REG_1 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , BPF_FUNC_map_lookup_elem), +BPF_JMP_IMM (BPF_JEQ ,BPF_REG_0 ,0 ,1 ), +BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_0 ,0 ), +BPF_EXIT_INSN (), + }; +int key =0 , map_fd, prog_fd =-1 , err; +__u64 value =0 ; + + map_fd =bpf_map_create (BPF_MAP_TYPE_PERCPU_ARRAY ,"percpu_ro_map" ,sizeof (int ), +sizeof (__u64),1 , &map_opts); +if (!ASSERT_GE (map_fd,0 ,"bpf_map_create" )) +return ; + + err =bpf_map_update_elem (map_fd, &key, &value,BPF_F_ALL_CPUS ); +if (!ASSERT_OK (err,"bpf_map_update_elem" )) + goto out; + + err =bpf_map_freeze (map_fd); +if (!ASSERT_OK (err,"bpf_map_freeze" )) + goto out; + + insns[3 ].imm = map_fd; + prog_fd =bpf_prog_load (BPF_PROG_TYPE_SOCKET_FILTER ,"percpu_ro_prog" ,"GPL" , insns, +ARRAY_SIZE (insns),NULL ); +ASSERT_GE (prog_fd,0 ,"bpf_prog_load" ); + +out: +if (prog_fd >=0 ) +close (prog_fd); +close (map_fd); +} +void test_global_percpu_data (void ) {if (!feat_supported (NULL ,FEAT_PERCPU_DATA )) {
@@ -243,4 +285,6 @@ void test_global_percpu_data(void)test_global_percpu_data_init ();if (test__start_subtest ("lskel" ))test_global_percpu_data_lskel (); +if (test__start_subtest ("rdonly_direct_read" )) +test_global_percpu_data_rdonly_direct_read (); }
Signature · 1 line
2.54.0
Re: [PATCH bpf-next v7 09/11] selftests/bpf: Add test to verify accessing rdonly percpu_array
Quoted 2 lines · Leon Hwang
Since
percpu_arraysupports direct-read, it should not break accessing
what does "it" refer to here?
Quoted 4 lines
rdonly
percpu_array.Add a test to verify that adding '.map_direct_value_addr' to
percpu_arraywon't break the case.
what case?
I think you are testing (read-only) direct access to read-only per-CPU array, is that right? Just write that out? Though, really, negative test (we *cannot* access read-only per-cpu map for write operation) would be more trustworthy for this...
Quoted 70 lines
Assisted-by Codex:gpt-5.5-xhighSigned-off-by Leon Hwang <[email protected]>--- .../bpf/prog_tests/global_data_init.c | 44 +++++++++++++++++++ 1 file changed, 44 insertions(+)tools/testing/selftests/bpf/prog_tests/global_data_init.c +44
-1@@ -232,6 +232,48 @@ static void test_global_percpu_data_lskel(void)free (online); } +static void test_global_percpu_data_rdonly_direct_read (void ) +{ +LIBBPF_OPTS (bpf_map_create_opts, map_opts, + .map_flags = BPF_F_RDONLY_PROG, + ); +struct bpf_insn insns[] = { +BPF_ST_MEM (BPF_W ,BPF_REG_10 ,-8 ,0 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-8 ), +BPF_LD_MAP_FD (BPF_REG_1 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , BPF_FUNC_map_lookup_elem), +BPF_JMP_IMM (BPF_JEQ ,BPF_REG_0 ,0 ,1 ), +BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_0 ,0 ), +BPF_EXIT_INSN (), + }; +int key =0 , map_fd, prog_fd =-1 , err; +__u64 value =0 ; + + map_fd =bpf_map_create (BPF_MAP_TYPE_PERCPU_ARRAY ,"percpu_ro_map" ,sizeof (int ), +sizeof (__u64),1 , &map_opts); +if (!ASSERT_GE (map_fd,0 ,"bpf_map_create" )) +return ; + + err =bpf_map_update_elem (map_fd, &key, &value,BPF_F_ALL_CPUS ); +if (!ASSERT_OK (err,"bpf_map_update_elem" )) + goto out; + + err =bpf_map_freeze (map_fd); +if (!ASSERT_OK (err,"bpf_map_freeze" )) + goto out; + + insns[3 ].imm = map_fd; + prog_fd =bpf_prog_load (BPF_PROG_TYPE_SOCKET_FILTER ,"percpu_ro_prog" ,"GPL" , insns, +ARRAY_SIZE (insns),NULL ); +ASSERT_GE (prog_fd,0 ,"bpf_prog_load" ); + +out: +if (prog_fd >=0 ) +close (prog_fd); +close (map_fd); +} +void test_global_percpu_data (void ) {if (!feat_supported (NULL ,FEAT_PERCPU_DATA )) {@@ -243,4 +285,6 @@ void test_global_percpu_data(void)test_global_percpu_data_init ();if (test__start_subtest ("lskel" ))test_global_percpu_data_lskel (); +if (test__start_subtest ("rdonly_direct_read" )) +test_global_percpu_data_rdonly_direct_read (); } --2.54.0
Re: [PATCH bpf-next v7 09/11] selftests/bpf: Add test to verify accessing rdonly percpu_array
Quoted 8 lines · Andrii Nakryiko
Quoted 2 lines · Leon Hwang
Since
percpu_arraysupports direct-read, it should not break accessingwhat does "it" refer to here?
Quoted 2 lines
rdonly
percpu_array.
Quoted 7 lines
Quoted 2 lines
Add a test to verify that adding '.map_direct_value_addr' to
percpu_arraywon't break the case.what case?
I think you are testing (read-only) direct access to read-only per-CPU array, is that right? Just write that out? Though, really, negative
Right, will update the commit msg.
It was to test the change of patch #4:
@@ -6113,6 +6115,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct b
if (tnum_is_const(reg->var_off) &&
bpf_map_is_rdonly(map) && map->ops->map_direct_value_addr &&
+ map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY &&
map->map_type != BPF_MAP_TYPE_INSN_ARRAY) {
int map_off = off + reg->var_off.value; u64 val = 0;
Quoted 3 lines
test (we *cannot* access read-only per-cpu map for write operation) would be more trustworthy for this...
Makes sense.
I think it is worth testing both positive and negative cases:
1. (read-only) direct access the data of read-only percpu data's percpu_array map to test the above change.
2. direct writing the data of read-only percpu data's percpu_array map is disallowed.
Thanks,
Leon
Quoted 2 lines
Quoted 2 lines
[...]
Re: [PATCH bpf-next v7 09/11] selftests/bpf: Add test to verify accessing rdonly percpu_array
Quoted 36 lines · Leon Hwang
Quoted 8 lines · Andrii Nakryiko
Quoted 2 lines · Leon Hwang
Since
percpu_arraysupports direct-read, it should not break accessingwhat does "it" refer to here?
Quoted 2 lines
rdonly
percpu_array.Quoted 7 lines
Quoted 2 lines
Add a test to verify that adding '.map_direct_value_addr' to
percpu_arraywon't break the case.what case?
I think you are testing (read-only) direct access to read-only per-CPU array, is that right? Just write that out? Though, really, negative
Right, will update the commit msg.
It was to test the change of patch #4:
(quoted diff) +0
-0@@ -6113,6 +6115,7 @@ static int check_mem_access(struct
bpf_verifier_env*env, intinsn_idx, struct bif (tnum_is_const(reg->var_off) && bpf_map_is_rdonly(map) && map->ops->map_direct_value_addr && + map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY && map->map_type != BPF_MAP_TYPE_INSN_ARRAY) { int map_off = off + reg->var_off.value; u64 val = 0;Quoted 3 lines
test (we *cannot* access read-only per-cpu map for write operation) would be more trustworthy for this...
Quoted 9 lines
Makes sense.
I think it is worth testing both positive and negative cases:
1. (read-only) direct access the data of read-only percpu data's
percpu_arraymap to test the above change.
2. direct writing the data of read-only percpu data'spercpu_arraymap is disallowed.
both is even better, but negative test is testing a condition that is hard to miss because you won't try to do that in practice )
Quoted 5 lines
Thanks,
LeonQuoted 2 lines
Quoted 2 lines
[...]
Re: [PATCH bpf-next v7 09/11] selftests/bpf: Add test to verify accessing rdonly percpu_array
Quoted 1 line · Andrii Nakryiko
On Tue, Jun 23, 2026 at 9:16 PM Leon Hwang <[email protected]> wrote:
[...]
Quoted 5 lines
Quoted 5 lines
Makes sense.
I think it is worth testing both positive and negative cases:
Quoted 9 lines
Quoted 5 lines
1. (read-only) direct access the data of read-only percpu data's
percpu_arraymap to test the above change.
2. direct writing the data of read-only percpu data'spercpu_arraymap is disallowed.both is even better, but negative test is testing a condition that is hard to miss because you won't try to do that in practice )
Got it.
Will implement both of them.
Thanks,
Leon
PATCH v7 10/11 selftests/bpf: Add tests to verify verifier log for global percpu data
In reply to Leon HwangAdd two tests to verify the verifier log
"R%d points to percpu_array map which cannot be used as const string\n".
--- .../bpf/prog_tests/global_data_init.c | 6 +++++ .../bpf/progs/test_global_percpu_data.c | 23 +++++++++++++++++++ 2 files changed, 29 insertions(+)
tools/testing/selftests/bpf/prog_tests/global_data_init.c +6 -0
@@ -274,6 +274,11 @@ static void test_global_percpu_data_rdonly_direct_read(void)close (map_fd); } +static void test_global_percpu_data_verifier_log (void ) +{ +RUN_TESTS (test_global_percpu_data); +} +void test_global_percpu_data (void ) {if (!feat_supported (NULL ,FEAT_PERCPU_DATA )) {
@@ -287,4 +292,5 @@ void test_global_percpu_data(void)test_global_percpu_data_lskel ();if (test__start_subtest ("rdonly_direct_read" ))test_global_percpu_data_rdonly_direct_read (); +test_global_percpu_data_verifier_log (); }
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +23 -0
@@ -28,4 +28,27 @@ int update_percpu_data(void *ctx)return 0 ; } +static const char fmt[] SEC(".percpu.fmt") = "data %d\n"; + +SEC("?kprobe") +__failure __msg("R{{[0-9]+}} points to percpu_array map which cannot be used as const string") +int verifier_strncmp(void *ctx) +{ +return bpf_strncmp ("test" ,5 , fmt); +} + +SEC("?kprobe") +__failure __msg("R{{[0-9]+}} points to percpu_array map which cannot be used as const string") +int verifier_snprintf(void *ctx) +{ +u64 args[] = { data }; +char buf[128 ]; +int len; + + len =bpf_snprintf (buf,sizeof (buf), fmt, args,sizeof (args)); +if (len >0 ) +bpf_printk ("snprintf: %s\n" , buf); +return 0 ; +} + char _license[] SEC("license") = "GPL";
Signature · 1 line
2.54.0
PATCH v7 11/11 selftests/bpf: Add test to verify bpf_iter for global percpu data
In reply to Leon HwangAdd a test to verify that it is OK to iter the percpu_array map used for global percpu data.
--- .../bpf/prog_tests/global_data_init.c | 52 +++++++++++++++++++ .../bpf/progs/test_global_percpu_data.c | 25 +++++++++ 2 files changed, 77 insertions(+)
tools/testing/selftests/bpf/prog_tests/global_data_init.c +52 -0
@@ -279,6 +279,56 @@ static void test_global_percpu_data_verifier_log(void)RUN_TESTS (test_global_percpu_data); } +static void test_global_percpu_data_iter (void ) +{ +DECLARE_LIBBPF_OPTS (bpf_iter_attach_opts, opts); +struct test_global_percpu_data *skel; +union bpf_iter_link_info linfo = {}; +struct bpf_link *link =NULL ; +int fd, num_cpus, len, err; +char buf[16 ]; + + num_cpus =libbpf_num_possible_cpus (); +if (!ASSERT_GT (num_cpus,0 ,"libbpf_num_possible_cpus" )) +return ; + + skel =test_global_percpu_data__open (); +if (!ASSERT_OK_PTR (skel,"test_global_percpu_data__open" )) +return ; + + skel->rodata->num_cpus = num_cpus; + skel->rodata->offsetof_num = offsetof(struct test_global_percpu_data__percpu , struct_data); + skel->rodata->offsetof_num +=sizeof (skel->percpu->struct_data) -sizeof (int ); + skel->rodata->elem_sz =roundup (sizeof (struct test_global_percpu_data__percpu ),8 ); + skel->percpu->struct_data.nums[6 ] =0xc0de ; + + err =test_global_percpu_data__load (skel); +if (!ASSERT_OK (err,"test_global_percpu_data__load" )) + goto out; + + linfo.map.map_fd =bpf_map__fd (skel->maps.percpu); + opts.link_info = &linfo; + opts.link_info_len =sizeof (linfo); + link =bpf_program__attach_iter (skel->progs.dump_percpu_data, &opts); +if (!ASSERT_OK_PTR (link,"bpf_program__attach_iter" )) + goto out; + + fd =bpf_iter_create (bpf_link__fd (link)); +if (!ASSERT_GE (fd,0 ,"bpf_iter_create" )) + goto out; + +while ((len =read (fd, buf,sizeof (buf))) >0 ) +do { }while (0 ); +ASSERT_EQ (len,0 ,"read iter" ); +ASSERT_TRUE (skel->bss->run_iter,"run_iter" ); +ASSERT_EQ (skel->bss->percpu_data_sum,0xc0de * num_cpus,"percpu_data_sum" ); + +close (fd); +out: +bpf_link__destroy (link); +test_global_percpu_data__destroy (skel); +} +void test_global_percpu_data (void ) {if (!feat_supported (NULL ,FEAT_PERCPU_DATA )) {
@@ -293,4 +343,6 @@ void test_global_percpu_data(void)if (test__start_subtest ("rdonly_direct_read" ))test_global_percpu_data_rdonly_direct_read ();test_global_percpu_data_verifier_log (); +if (test__start_subtest ("iter" )) +test_global_percpu_data_iter (); }
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +25 -0
@@ -51,4 +51,29 @@ int verifier_snprintf(void *ctx)return 0 ; } +volatile const __u32 num_cpus =0 ; +volatile const int offsetof_num; +volatile const int elem_sz; +__u32 percpu_data_sum =0 ; +bool run_iter = false; + +SEC("iter/bpf_map_elem") +__auxiliary +int dump_percpu_data (struct bpf_iter__bpf_map_elem *ctx) +{ +void *pptr = ctx->value; +int i; + +if (!pptr) +return 0 ; + + run_iter = true; + +for (i =0 ; i < num_cpus; i++) { + percpu_data_sum += *(int *) (pptr + offsetof_num); + pptr += elem_sz; + } +return 0 ; +} + char _license[] SEC("license") = "GPL";
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH bpf-next v7 11/11] selftests/bpf: Add test to verify bpf_iter for global percpu data
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Uninitialized kernel heap memory leak in per-CPU maps via bpf_iter and BPF_MAP_LOOKUP_ELEM due to padding bytes not being zeroed. --
commit a629f2d1587e23ca18c8365ab951d5623f170340
Author: Leon Hwang <[email protected]>
selftests/bpf: Add test to verify bpf_iter for global percpu data
This patch adds a selftest to verify that a BPF iterator can safely iterate over the percpu_array map used for global per-CPU data. It introduces test_global_percpu_data_iter() in global_data_init.c and an iterator program dump_percpu_data() in test_global_percpu_data.c.
Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +0
-0
[ ... ]
Quoted 15 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +11
-0@@ -279,6 +279,56 @@ static void test_global_percpu_data_verifier_log(void)RUN_TESTS (test_global_percpu_data); } +static void test_global_percpu_data_iter (void ) +{ +DECLARE_LIBBPF_OPTS (bpf_iter_attach_opts, opts); +struct test_global_percpu_data *skel; +union bpf_iter_link_info linfo = {}; +struct bpf_link *link =NULL ; +int fd, num_cpus, len, err; +char buf[16 ]; + + num_cpus =libbpf_num_possible_cpus (); +if (!ASSERT_GT (num_cpus,0 ,"libbpf_num_possible_cpus" ))
tools/testing/selftests/bpf/prog_tests/global_data_init.c static void test_global_percpu_data_verifier_log(void) · line 302
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +10
-0+return ; + + skel =test_global_percpu_data__open (); +if (!ASSERT_OK_PTR (skel,"test_global_percpu_data__open" )) +return ; + + skel->rodata->num_cpus = num_cpus; + skel->rodata->offsetof_num = offsetof(struct test_global_percpu_data__percpu , struct_data); + skel->rodata->offsetof_num +=sizeof (skel->percpu->struct_data) -sizeof (int ); + skel->rodata->elem_sz =roundup (sizeof (struct test_global_percpu_data__percpu ),8 );
[Severity: High]
This is a pre-existing issue, but does iterating over per-CPU maps expose uninitialized kernel heap memory?
When working with per-CPU maps, temporary buffers are allocated using kmalloc without the __GFP_ZERO flag in functions like bpf_iter_init_array_map in kernel/bpf/arraymap.c:
kernel/bpf/arraymap.c:bpf_iter_init_array_map() {
...
value_buf = kmalloc(buf_size, GFP_USER | __GFP_NOWARN);
...
}This is also done in kernel/bpf/hashtab.c:bpf_iter_init_hash_map().
If the map contains a BTF record, bpf_obj_memcpy in include/linux/bpf.h explicitly stops at map->value_size instead of filling the entire rounded-up size:
include/linux/bpf.h:bpf_obj_memcpy() {
...
memcpy(dst + curr_off, src + curr_off, size - curr_off);
...
}This fails to overwrite the padding bytes up to round_up(map->value_size, 8). Because the BPF verifier allows the iterator program to read the entire round_up(map->value_size, 8) * num_possible_cpus() bytes, the uninitialized padding from kmalloc is exposed to the BPF program.
Similarly, in kernel/bpf/syscall.c:map_lookup_elem(), kvmalloc is used without zeroing:
kernel/bpf/syscall.c:map_lookup_elem() {
...
value = kvmalloc(value_size, GFP_USER | __GFP_NOWARN);
...
}This means sys_bpf BPF_MAP_LOOKUP_ELEM also copies this uninitialized padding back to user space via copy_to_user, causing a direct kernel heap info leak.
Could this be addressed by zeroing these allocations or ensuring the padding is cleared when copying map values?
Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +3
-0+ skel->percpu->struct_data.nums[6 ] =0xc0de ; + + err =test_global_percpu_data__load (skel);
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
Re: [PATCH bpf-next v7 11/11] selftests/bpf: Add test to verify bpf_iter for global percpu data
On 22/6/26 23:08, [email protected] wrote:
[...]
Quoted 71 lines
Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +0
-0[ ... ]
Quoted 15 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +11
-0@@ -279,6 +279,56 @@ static void test_global_percpu_data_verifier_log(void)RUN_TESTS (test_global_percpu_data); } +static void test_global_percpu_data_iter (void ) +{ +DECLARE_LIBBPF_OPTS (bpf_iter_attach_opts, opts); +struct test_global_percpu_data *skel; +union bpf_iter_link_info linfo = {}; +struct bpf_link *link =NULL ; +int fd, num_cpus, len, err; +char buf[16 ]; + + num_cpus =libbpf_num_possible_cpus (); +if (!ASSERT_GT (num_cpus,0 ,"libbpf_num_possible_cpus" ))tools/testing/selftests/bpf/prog_tests/global_data_init.c static void test_global_percpu_data_verifier_log(void) · line 302
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +10
-0+return ; + + skel =test_global_percpu_data__open (); +if (!ASSERT_OK_PTR (skel,"test_global_percpu_data__open" )) +return ; + + skel->rodata->num_cpus = num_cpus; + skel->rodata->offsetof_num = offsetof(struct test_global_percpu_data__percpu , struct_data); + skel->rodata->offsetof_num +=sizeof (skel->percpu->struct_data) -sizeof (int ); + skel->rodata->elem_sz =roundup (sizeof (struct test_global_percpu_data__percpu ),8 );[Severity: High]
This is a pre-existing issue, but does iterating over per-CPU maps expose uninitialized kernel heap memory?When working with per-CPU maps, temporary buffers are allocated using kmalloc without the
__GFP_ZEROflag in functions likebpf_iter_init_array_mapinkernel/bpf/arraymap.c:kernel/bpf/arraymap.c:bpf_iter_init_array_map() { ... value_buf = kmalloc(buf_size, GFP_USER | __GFP_NOWARN); ... }This is also done in kernel/bpf/hashtab.c:bpf_iter_init_hash_map().
If the map contains a BTF record,
bpf_obj_memcpyininclude/linux/bpf.hexplicitly stops atmap->value_sizeinstead of filling the entire rounded-up size:include/linux/bpf.h:bpf_obj_memcpy() { ... memcpy(dst + curr_off, src + curr_off, size - curr_off); ... }This fails to overwrite the padding bytes up to
round_up(map->value_size, 8). Because the BPF verifier allows the iterator program to read the entireround_up(map->value_size, 8)*num_possible_cpus()bytes, the uninitialized padding from kmalloc is exposed to the BPF program.Similarly, in kernel/bpf/syscall.c:map_lookup_elem(), kvmalloc is used without zeroing:
kernel/bpf/syscall.c:map_lookup_elem() { ... value = kvmalloc(value_size, GFP_USER | __GFP_NOWARN); ... }
Quoted 6 lines
This means
sys_bpfBPF_MAP_LOOKUP_ELEMalso copies this uninitialized padding back to user space viacopy_to_user, causing a direct kernel heap info leak.Could this be addressed by zeroing these allocations or ensuring the padding is cleared when copying map values?
Will verify the issue by separate selftests.
If true, will post separate patch to fix it.
Thanks,
Leon
Quoted 5 lines
Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +3
-0+ skel->percpu->struct_data.nums[6 ] =0xc0de ; + + err =test_global_percpu_data__load (skel);[ ... ]
Re: [PATCH bpf-next v7 11/11] selftests/bpf: Add test to verify bpf_iter for global percpu data
Quoted 60 lines · Leon Hwang
On 22/6/26 23:08, [email protected] wrote:
[...]Quoted 58 lines
Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +0
-0[ ... ]
Quoted 15 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +11
-0@@ -279,6 +279,56 @@ static void test_global_percpu_data_verifier_log(void)RUN_TESTS (test_global_percpu_data); } +static void test_global_percpu_data_iter (void ) +{ +DECLARE_LIBBPF_OPTS (bpf_iter_attach_opts, opts); +struct test_global_percpu_data *skel; +union bpf_iter_link_info linfo = {}; +struct bpf_link *link =NULL ; +int fd, num_cpus, len, err; +char buf[16 ]; + + num_cpus =libbpf_num_possible_cpus (); +if (!ASSERT_GT (num_cpus,0 ,"libbpf_num_possible_cpus" ))tools/testing/selftests/bpf/prog_tests/global_data_init.c static void test_global_percpu_data_verifier_log(void) · line 302
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +10
-0+return ; + + skel =test_global_percpu_data__open (); +if (!ASSERT_OK_PTR (skel,"test_global_percpu_data__open" )) +return ; + + skel->rodata->num_cpus = num_cpus; + skel->rodata->offsetof_num = offsetof(struct test_global_percpu_data__percpu , struct_data); + skel->rodata->offsetof_num +=sizeof (skel->percpu->struct_data) -sizeof (int ); + skel->rodata->elem_sz =roundup (sizeof (struct test_global_percpu_data__percpu ),8 );[Severity: High]
This is a pre-existing issue, but does iterating over per-CPU maps expose uninitialized kernel heap memory?When working with per-CPU maps, temporary buffers are allocated using kmalloc without the
__GFP_ZEROflag in functions likebpf_iter_init_array_mapinkernel/bpf/arraymap.c:kernel/bpf/arraymap.c:bpf_iter_init_array_map() { ... value_buf = kmalloc(buf_size, GFP_USER | __GFP_NOWARN); ... }This is also done in kernel/bpf/hashtab.c:bpf_iter_init_hash_map().
If the map contains a BTF record,
bpf_obj_memcpyininclude/linux/bpf.hexplicitly stops atmap->value_sizeinstead of filling the entire rounded-up size:include/linux/bpf.h:bpf_obj_memcpy() { ... memcpy(dst + curr_off, src + curr_off, size - curr_off); ... }This fails to overwrite the padding bytes up to
round_up(map->value_size, 8).
True.
The padding bytes of non-zeroed k/kvmalloc() could be exposed to user space.
Verified the issue by a LOOKUP_ELEM test:
https://github.com/Asphaltt/bpf/commits/bpf/avoid-uninit-mem/v1/.
test_map_uninit_mem_exposure:FAIL:zeroed tail bytes unexpected memory mismatch
actual:
2B 2B 2B 2B
expected:
00 00 00 00Will post the fix later.
Thanks,
Leon
Quoted 33 lines
Quoted 13 lines
Because the BPF verifier allows the iterator program to read the entire
round_up(map->value_size, 8)*num_possible_cpus()bytes, the uninitialized padding from kmalloc is exposed to the BPF program.Similarly, in kernel/bpf/syscall.c:map_lookup_elem(), kvmalloc is used without zeroing:
kernel/bpf/syscall.c:map_lookup_elem() { ... value = kvmalloc(value_size, GFP_USER | __GFP_NOWARN); ... }Quoted 6 lines
This means
sys_bpfBPF_MAP_LOOKUP_ELEMalso copies this uninitialized padding back to user space viacopy_to_user, causing a direct kernel heap info leak.Could this be addressed by zeroing these allocations or ensuring the padding is cleared when copying map values?
Will verify the issue by separate selftests.
If true, will post separate patch to fix it.
Thanks,
LeonQuoted 5 lines
Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +3
-0+ skel->percpu->struct_data.nums[6 ] =0xc0de ; + + err =test_global_percpu_data__load (skel);[ ... ]
[email protected]
AI reviewRe: [PATCH bpf-next v7 11/11] selftests/bpf: Add test to verify bpf_iter for global percpu data
In reply to Leon HwangThis isn't a bug, but could the empty drain body here be a bare ';' rather than 'do { } while (0);'? The other bpf_iter tests in this directory (prog_tests/bpf_iter.c) drain the iterator fd with a…
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +0
-0@@ -279,6 +279,56 @@ static void test_global_percpu_data_verifier_log(void)
[ ... ]
tools/testing/selftests/bpf/prog_tests/global_data_init.c static void test_global_percpu_data_verifier_log(void) · line 322
Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +7
-0+ fd =bpf_iter_create (bpf_link__fd (link)); +if (!ASSERT_GE (fd,0 ,"bpf_iter_create" )) + goto out; + +while ((len =read (fd, buf,sizeof (buf))) >0 ) +do { }while (0 ); +ASSERT_EQ (len,0 ,"read iter" );
This isn't a bug, but could the empty drain body here be a bare ';' rather than 'do { } while (0);'? The other bpf_iter tests in this directory (prog_tests/bpf_iter.c) drain the iterator fd with a bare ';', so a plain ';' would read more like the neighbouring code.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/27961789929
Re: [PATCH bpf-next v7 11/11] selftests/bpf: Add test to verify bpf_iter for global percpu data
Quoted 8 lines · On 22/6/26 23:24, [email protected] wrote:
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +0
-0@@ -279,6 +279,56 @@ static void test_global_percpu_data_verifier_log(void)[ ... ]
Quoted 14 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c static void test_global_percpu_data_verifier_log(void) · line 322
Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +7
-0+ fd =bpf_iter_create (bpf_link__fd (link)); +if (!ASSERT_GE (fd,0 ,"bpf_iter_create" )) + goto out; + +while ((len =read (fd, buf,sizeof (buf))) >0 ) +do { }while (0 ); +ASSERT_EQ (len,0 ,"read iter" );This isn't a bug, but could the empty drain body here be a bare ';' rather than 'do { } while (0);'? The other
bpf_itertests in this directory (prog_tests/bpf_iter.c) drain the iterator fd with a bare ';', so a plain ';' would read more like the neighbouring code.
I think 'do { } while (0);' is more readable than a dangling ';', and avoids -Wempty-body warning btw.
Thanks,
Leon