bpf: Introduce global percpu data
Subsystems (6)
v4 cover bpf: Introduce global percpu data
This patch set introduces global percpu data, similar to commit 6316f78306c1 ("Merge branch 'support-global-data'"), to reduce restrictions in C for BPF programs.
With this enhancement, it becomes possible to define and use global percpu variables, like the DEFINE_PER_CPU() macro in the kernel include/linux/percpu-defs.h.
The section name for global peurcpu data is ".percpu". Even though, a one-byte percpu variable (e.g., char run SEC(".percpu") = 0;) can trigger a crash with Clang 17 [1], users are expected to use such small variables as global percpu data with newer Clang versions, which don't have the issue.
The idea stems from the bpfsnoop [2], which itself was inspired by retsnoop [3]. During testing of bpfsnoop on the v6.6 kernel, two LBR (Last Branch Record) entries were observed related to the bpf_get_smp_processor_id() helper.
Since commit 1ae6921009e5 ("bpf: inline bpf_get_smp_processor_id() helper"), the bpf_get_smp_processor_id() helper has been inlined on x86_64, reducing the overhead and consequently minimizing these two LBR records.
However, the introduction of global percpu data offers a more robust solution. By leveraging the percpu_array map and percpu instruction, global percpu data can be implemented intrinsically.
This feature also facilitates sharing percpu information between tail callers and callees or between freplace callers and callees through a shared global percpu variable. Previously, this was achieved using a 1-entry percpu_array map, which this patch set aims to improve upon.
Links:
[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://github.com/bpfsnoop/bpfsnoop
[3] https://github.com/anakryiko/retsnoop
Changes:
v3 -> v4:
* Drop duplicate blank lines in verifier.
* Add percpu data feature probe in libbpf.
* Update percpu_array map using BPF_F_ALL_CPUS flag for lskel, if no cpu flag is set.
* Add two tests to verify verifier log.
* Add a test to verify mov64_percpu_reg instruction.
* Add a test to verify bpf_iter for percpu data map.
* Update percpu_array map using BPF_F_ALL_CPUS flag in libbpf (per Alexei and Andrii).
* Address comments from Andrii:
* Use .percpu as section identifier.
* Use bpf_jit_supports_percpu_insn() instead of CONFIG_SMP.
* Drop bpf_map__is_internal_percpu() API.
* Drop unnecessary __aligned(8) in libbpf, verified by selftest.
* Make mmap data read-only after loading prog.
v3: https://lore.kernel.org/bpf/[email protected]/
v2 -> v3:
* Use ".data..percpu" as PERCPU_DATA_SEC.
* Address comment from Alexei:
* Add u8, array of ints and struct { .. } vars to selftest.
v2: https://lore.kernel.org/bpf/[email protected]/
v1 -> v2:
* Address comments from Andrii:
* Use LIBBPF_MAP_PERCPU and SEC_PERCPU.
* Reuse mmaped of libbpf's struct bpf_map for .percpu map data.
* Set .percpu struct pointer to NULL after loading skeleton.
* Make sure value size of .percpu map is __aligned(8).
* Use raw_tp and opts.cpu to test global percpu variables on all CPUs.
* Address comments from Alexei:
* Test non-zero offset of global percpu variable.
* Test case about BPF_PSEUDO_MAP_IDX_VALUE.
v1: https://lore.kernel.org/bpf/[email protected]/
rfc -> v1:
* Address comments from Andrii:
* Keep one image of global percpu variable for all CPUs.
* Reject non-ARRAY map in bpf_map_direct_read(), check_reg_const_str(), and check_bpf_snprintf_call() in verifier.
* Split out libbpf changes from kernel-side changes.
* Use ".percpu" as PERCPU_DATA_SEC.
* Use enum libbpf_map_type to distinguish BSS, DATA, RODATA and PERCPU_DATA.
* Avoid using errno for checking err from libbpf_num_possible_cpus().
* Use "map '%s': " prefix for error message.
rfc: https://lore.kernel.org/bpf/[email protected]/
Leon Hwang (8):
bpf: Drop duplicate blank lines in verifier
bpf: Introduce global percpu data
libbpf: Probe percpu data feature
libbpf: Add support for global percpu data
bpf: Update per-CPU maps using BPF_F_ALL_CPUS flag
bpftool: Generate skeleton for global percpu data selftests/bpf: Add tests to verify global percpu data selftests/bpf: Add a test to verify bpf_iter for global percpu data
kernel/bpf/arraymap.c | 41 ++- kernel/bpf/fixups.c | 31 ++ kernel/bpf/syscall.c | 15 + kernel/bpf/verifier.c | 27 +- tools/bpf/bpftool/gen.c | 24 +- tools/lib/bpf/bpf_gen_internal.h | 3 +- tools/lib/bpf/features.c | 35 +++ tools/lib/bpf/gen_loader.c | 3 +- tools/lib/bpf/libbpf.c | 67 ++++- tools/lib/bpf/libbpf_internal.h | 2 + tools/testing/selftests/bpf/Makefile | 2 +- .../bpf/prog_tests/global_data_init.c | 276 ++++++++++++++++++ .../bpf/progs/test_global_percpu_data.c | 102 +++++++ 13 files changed, 582 insertions(+), 46 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/test_global_percpu_data.c
--
2.53.0
PATCH v4 1/8 bpf: Drop duplicate blank lines in verifier
There are many adjacent blank lines in the verifier that have accumulated over time.
Drop them for cleanup.
No functional changes intended.
--- kernel/bpf/verifier.c | 18 ------------------ 1 file changed, 18 deletions(-)
kernel/bpf/verifier.c +0 -18
@@ -513,7 +513,6 @@ static bool helper_multiple_ref_obj_use(enum bpf_func_id func_id,return ref_obj_uses >1 ; } -static bool is_spi_bounds_valid (struct bpf_func_state *state,int spi,int nr_slots) {int allocated_slots = state->allocated_stack /BPF_REG_SIZE ;
@@ -619,7 +618,6 @@ static void __mark_dynptr_reg(struct bpf_reg_state *reg,enum bpf_dynptr_type type, bool first_slot, int dynptr_id); - static void mark_dynptr_stack_regs(struct bpf_verifier_env *env, struct bpf_reg_state *sreg1, struct bpf_reg_state *sreg2,
@@ -1655,7 +1653,6 @@ static bool same_callsites(struct bpf_verifier_state *a, struct bpf_verifier_stareturn true; } -void bpf_free_backedges (struct bpf_scc_visit *visit) {struct bpf_scc_backedge *backedge, *next;
@@ -2625,7 +2622,6 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env,return &elem->st; } -static int cmp_subprogs (const void *a,const void *b) { return ((struct bpf_subprog_info *)a)->start -
@@ -3679,7 +3675,6 @@ static bool is_spillable_regtype(enum bpf_reg_type type) } } -/* check if register is a constant scalar value */ static bool is_reg_const (struct bpf_reg_state *reg,bool subreg32) {
@@ -4347,7 +4342,6 @@ static int check_stack_read(struct bpf_verifier_env *env,return err; } -/* check_stack_write dispatches to check_stack_write_fixed_off or * check_stack_write_var_off. *
@@ -5033,7 +5027,6 @@ static int check_sock_access(struct bpf_verifier_env *env, int insn_idx, valid = false; } -if (valid) { env->insn_aux_data[insn_idx].ctx_field_size = info.ctx_field_size;
@@ -6836,7 +6829,6 @@ static int check_stack_range_initialized(if (err)return err; -if (tnum_is_const (reg->var_off)) { min_off = max_off = reg->var_off.value + off; }else {
@@ -7541,7 +7533,6 @@ static bool is_iter_new_kfunc(struct bpf_kfunc_call_arg_meta *meta)return meta->kfunc_flags &KF_ITER_NEW ; } -static bool is_iter_destroy_kfunc (struct bpf_kfunc_call_arg_meta *meta) {return meta->kfunc_flags &KF_ITER_DESTROY ;
@@ -10459,7 +10450,6 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insnif (!reg)return -EFAULT ; -if (meta.dynptr_id) {verifier_bug (env,"meta.dynptr_id already set" );return -EFAULT ;
@@ -10798,7 +10788,6 @@ static bool is_kfunc_release(struct bpf_kfunc_call_arg_meta *meta)return meta->kfunc_flags &KF_RELEASE ; } -static bool is_kfunc_destructive (struct bpf_kfunc_call_arg_meta *meta) {return meta->kfunc_flags &KF_DESTRUCTIVE ;
@@ -11571,7 +11560,6 @@ static int process_irq_flag(struct bpf_verifier_env *env, int regno,return 0 ; } -static int ref_set_non_owning (struct bpf_verifier_env *env,struct bpf_reg_state *reg) {struct btf_record *rec =reg_btf_record (reg);
@@ -16715,7 +16703,6 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn)return 0 ; } -static bool return_retval_range (struct bpf_verifier_env *env,struct bpf_retval_range *range) {enum bpf_prog_type prog_type =resolve_prog_type (env->prog);
@@ -18587,8 +18574,6 @@ static void release_insn_arrays(struct bpf_verifier_env *env)bpf_insn_array_release (env->insn_array_maps[i]); } - -/* The verifier does more data flow analysis than llvm and will not * explore branches that are dead at run time. Malicious programs can * have dead code too. Therefore replace all dead at-run-time code
@@ -18616,8 +18601,6 @@ static void sanitize_dead_code(struct bpf_verifier_env *env) } } - -static void free_states (struct bpf_verifier_env *env) {struct bpf_verifier_state_list *sl;
@@ -18880,7 +18863,6 @@ static int do_check_main(struct bpf_verifier_env *env)return ret; } -static void print_verification_stats (struct bpf_verifier_env *env) {int i;
Signature · 1 line
2.53.0
PATCH v4 2/8 bpf: Introduce global percpu data
In reply to Leon HwangIntroduce global percpu data, inspired by the commit 6316f78306c1 ("Merge branch 'support-global-data'"). It enables the definition of global percpu variables in BPF, similar to the include/linux/percpu-defs.h::DEFINE_PER_CPU() macro.
For example, in BPF, it is able to define a global percpu variable like:
int data SEC(".percpu");
With this patch, tools like retsnoop [1] and bpfsnoop [2] can simplify their BPF code for handling LBRs. The code can be updated from
static struct perf_branch_entry lbrs[1][MAX_LBR_ENTRIES] SEC(".data.lbrs");
to
static struct perf_branch_entry lbrs[MAX_LBR_ENTRIES] SEC(".percpu.lbrs");
This eliminates the need to retrieve the CPU ID using the bpf_get_smp_processor_id() helper.
Additionally, by reusing global percpu data map, sharing information between tail callers and callees or freplace callers and callees becomes simpler compared to reusing percpu_array maps.
Links:
[1] https://github.com/anakryiko/retsnoop
[2] https://github.com/bpfsnoop/bpfsnoop
--- kernel/bpf/arraymap.c | 41 +++++++++++++++++++++++++++++++++++++++-- kernel/bpf/fixups.c | 31 +++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 9 ++++++++- 3 files changed, 78 insertions(+), 3 deletions(-)
kernel/bpf/arraymap.c +39 -2
@@ -261,6 +261,40 @@ static void *percpu_array_map_lookup_elem(struct bpf_map *map, void *key)return this_cpu_ptr (array->pptrs[index & array->index_mask]); } +static int percpu_array_map_direct_value_addr (const struct bpf_map *map, +u64 *imm,u32 off) +{ + struct bpf_array *array = container_of(map, struct bpf_array, map); + +if (map->max_entries !=1 ) +return -EOPNOTSUPP ; +if (off >= map->value_size) +return -EINVAL ; +if (!bpf_jit_supports_percpu_insn ()) +return -EOPNOTSUPP ; + + *imm = (u64 ) array->pptrs[0 ]; +return 0 ; +} + +static int percpu_array_map_direct_value_meta (const struct bpf_map *map, +u64 imm,u32 *off) +{ + struct bpf_array *array = container_of(map, struct bpf_array, map); +u64 base = (u64 ) array->pptrs[0 ]; +u64 range = array->elem_size; + +if (map->max_entries !=1 ) +return -EOPNOTSUPP ; +if (imm < base || imm >= base + range) +return -ENOENT ; +if (!bpf_jit_supports_percpu_insn ()) +return -EOPNOTSUPP ; + + *off = imm - base; +return 0 ; +} +/* emit BPF instructions equivalent to C code of percpu_array_map_lookup_elem() */ static int percpu_array_map_gen_lookup (struct bpf_map *map,struct bpf_insn *insn_buf) {
@@ -553,9 +587,10 @@ static int array_map_check_btf(struct bpf_map *map, const struct btf_type *key_type, const struct btf_type *value_type) { - /* One exception for keyless BTF: .bss/.data/.rodata map */ + /* One exception for keyless BTF: .bss/.data/.rodata/.percpu map */ if (btf_type_is_void(key_type)) { - if (map->map_type != BPF_MAP_TYPE_ARRAY || + if ((map->map_type != BPF_MAP_TYPE_ARRAY && + map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) || map->max_entries != 1) return -EINVAL;
@@ -834,6 +869,8 @@ const struct bpf_map_ops percpu_array_map_ops = { .map_get_next_key = bpf_array_get_next_key, .map_lookup_elem = percpu_array_map_lookup_elem, .map_gen_lookup = percpu_array_map_gen_lookup, + .map_direct_value_addr = percpu_array_map_direct_value_addr, + .map_direct_value_meta = percpu_array_map_direct_value_meta, .map_update_elem = array_map_update_elem, .map_delete_elem = array_map_delete_elem, .map_lookup_percpu_elem = percpu_array_map_lookup_percpu_elem,
kernel/bpf/fixups.c +31 -0
@@ -1682,6 +1682,37 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) goto next_insn; } +if (bpf_jit_supports_percpu_insn () && + insn->code == (BPF_LD |BPF_IMM |BPF_DW ) && + (insn->src_reg ==BPF_PSEUDO_MAP_VALUE || + insn->src_reg ==BPF_PSEUDO_MAP_IDX_VALUE )) { +struct bpf_map *map; + + aux = &env->insn_aux_data[i + delta]; + map = env->used_maps[aux->map_index]; +if (map->map_type !=BPF_MAP_TYPE_PERCPU_ARRAY ) + goto next_insn; + +/* +* Reuse the original ld_imm64 insn, and add one +* mov64_percpu_reg insn. +*/ + + insn_buf[0 ] = insn[1 ]; + insn_buf[1 ] =BPF_MOV64_PERCPU_REG (insn->dst_reg, insn->dst_reg); + cnt =2 ; + + i++; + new_prog =bpf_patch_insn_data (env, i + delta, insn_buf, cnt); +if (!new_prog) +return -ENOMEM ; + + delta += cnt -1 ; + env->prog = prog = new_prog; + insn = new_prog->insnsi + i + delta; + goto next_insn; + } +if (insn->code != (BPF_JMP |BPF_CALL )) goto next_insn;if (insn->src_reg ==BPF_PSEUDO_CALL )
kernel/bpf/verifier.c +8 -1
@@ -6362,7 +6362,8 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regnif (tnum_is_const (reg->var_off) &&bpf_map_is_rdonly (map) && map->ops->map_direct_value_addr && - map->map_type !=BPF_MAP_TYPE_INSN_ARRAY ) { + map->map_type !=BPF_MAP_TYPE_INSN_ARRAY && + map->map_type !=BPF_MAP_TYPE_PERCPU_ARRAY ) {int map_off = off + reg->var_off.value;u64 val =0 ;
@@ -8358,6 +8359,12 @@ static int check_reg_const_str(struct bpf_verifier_env *env,return -EACCES ; } +if (map->map_type ==BPF_MAP_TYPE_PERCPU_ARRAY ) { +verbose (env,"R%d points to percpu_array map which cannot be used as const string\n" , + regno); +return -EACCES ; + } +if (!bpf_map_is_rdonly (map)) {verbose (env,"R%d does not point to a readonly map'\n" , regno);return -EACCES ;
Signature · 1 line
2.53.0
PATCH v4 3/8 libbpf: Probe percpu data feature
In reply to Leon Hwanglibbpf needs a reliable way to distinguish kernels that can support global percpu data from those that cannot.
Add a dedicated feature probe, so libbpf can make capability decisions early and fail predictably when global percpu data is unavailable.
--- tools/lib/bpf/features.c | 35 +++++++++++++++++++++++++++++++++ tools/lib/bpf/libbpf_internal.h | 2 ++ 2 files changed, 37 insertions(+)
tools/lib/bpf/features.c +35 -0
@@ -615,6 +615,38 @@ static int probe_kern_btf_layout(int token_fd) (char *)layout, token_fd)); } +static int probe_kern_percpu_data (int token_fd) +{ +struct bpf_insn insns[] = { +BPF_LD_MAP_VALUE (BPF_REG_1 ,0 ,0 ), +BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_1 ,0 ), +BPF_EXIT_INSN (), + }; +LIBBPF_OPTS (bpf_map_create_opts, map_opts, + .token_fd = token_fd, + .map_flags = token_fd ? BPF_F_TOKEN_FD : 0, + ); +LIBBPF_OPTS (bpf_prog_load_opts, prog_opts, + .token_fd = token_fd, + .prog_flags = token_fd ? BPF_F_TOKEN_FD : 0, + ); +int ret, map, insn_cnt =ARRAY_SIZE (insns); + + map =bpf_map_create (BPF_MAP_TYPE_PERCPU_ARRAY ,"libbpf_percpu" ,sizeof (int ),8 ,1 , + &map_opts); +if (map <0 ) { +pr_warn ("Error in %s(): %s. Couldn't create simple percpu_array map.\n" , + __func__,errstr (map)); +return map; + } + + insns[0 ].imm = map; + + ret =bpf_prog_load (BPF_PROG_TYPE_SOCKET_FILTER ,NULL ,"GPL" , insns, insn_cnt, &prog_opts); +close (map); +return probe_fd (ret); +} +typedef int (*feature_probe_fn )(int /* token_fd */ );static struct kern_feature_cache feature_cache;
@@ -699,6 +731,9 @@ static struct kern_feature_desc { [FEAT_BTF_LAYOUT ] = {"kernel supports BTF layout" , probe_kern_btf_layout, }, + [FEAT_PERCPU_DATA ] = { +"kernel supports percpu data" , probe_kern_percpu_data, + }, };bool feat_supported (struct kern_feature_cache *cache,enum kern_feature_id feat_id)
tools/lib/bpf/libbpf_internal.h +2 -0
@@ -398,6 +398,8 @@ enum kern_feature_id {FEAT_UPROBE_SYSCALL ,/* Kernel supports BTF layout information */ FEAT_BTF_LAYOUT , +/* Kernel supports percpu data */ +FEAT_PERCPU_DATA , __FEAT_CNT, };
Signature · 1 line
2.53.0
PATCH v4 4/8 libbpf: Add support for global percpu data
In reply to Leon HwangAdd support for global percpu data in libbpf by adding a new ".percpu" section, similar to ".data". It enables efficient handling of percpu global variables in bpf programs.
When generating loader for lightweight skeleton, update the percpu_array map used for global percpu data using BPF_F_ALL_CPUS, in order to update values across all CPUs using one value slot.
Unlike global data, the mmaped data for global percpu data will be marked as read-only after populating the percpu_array map. Thereafter, users can read those initialized percpu data after loading prog. If they want to update the percpu data after loading prog, they have to update the percpu_array map using key=0 instead.
--- tools/lib/bpf/bpf_gen_internal.h | 3 +- tools/lib/bpf/gen_loader.c | 3 +- tools/lib/bpf/libbpf.c | 67 ++++++++++++++++++++++++++------ 3 files changed, 59 insertions(+), 14 deletions(-)
tools/lib/bpf/bpf_gen_internal.h +2 -1
@@ -66,7 +66,8 @@ void bpf_gen__prog_load(struct bpf_gen *gen, enum bpf_prog_type prog_type, const char *prog_name, const char *license, struct bpf_insn *insns, size_t insn_cnt, struct bpf_prog_load_opts *load_attr, int prog_idx); -void bpf_gen__map_update_elem (struct bpf_gen *gen,int map_idx,void *value,__u32 value_size); +void bpf_gen__map_update_elem (struct bpf_gen *gen,int map_idx,void *value,__u32 value_size, +__u64 flags);void bpf_gen__map_freeze (struct bpf_gen *gen,int map_idx);void bpf_gen__record_attach_target (struct bpf_gen *gen,const char *name,enum bpf_attach_type type); void bpf_gen__record_extern(struct bpf_gen *gen, const char *name, bool is_weak,
tools/lib/bpf/gen_loader.c +2 -1
@@ -1158,7 +1158,7 @@ void bpf_gen__prog_load(struct bpf_gen *gen, }void bpf_gen__map_update_elem (struct bpf_gen *gen,int map_idx,void *pvalue, -__u32 value_size) +__u32 value_size,__u64 flags) { int attr_size = offsetofend(union bpf_attr, flags);int map_update_attr, value, key;
@@ -1166,6 +1166,7 @@ void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue,int zero =0 ;memset (&attr,0 , attr_size); + attr.flags = flags; value =add_data (gen, pvalue, value_size); key =add_data (gen, &zero,sizeof (zero));
tools/lib/bpf/libbpf.c +55 -12
178 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH v4 5/8 bpf: Update per-CPU maps using BPF_F_ALL_CPUS flag
In reply to Leon HwangWhen updating per-CPU maps via the lightweight skeleton loader, use a single value slot across all CPUs. This avoids two potential issues when updating on an M-CPU kernel with N cached slots (N < M), especially when N is much smaller than M:
1) The update may trigger a page fault when copying data from the last slot, as the read may go beyond the allocated buffer.
2) The update may copy unexpected data from slots [N, M-1].
--- kernel/bpf/syscall.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
kernel/bpf/syscall.c +15 -0
@@ -1785,6 +1785,21 @@ static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr) goto err_put; } +/* +* When updating per-CPU maps via the lightweight skeleton +* loader, use a single value slot across all CPUs. This avoids +* two potential issues when updating on an M-CPU kernel with +* N cached slots (N < M), especially when N is much smaller +* than M: +* 1) The update may trigger a page fault when copying data from +* the last slot, as the read may go beyond the allocated +* buffer. +* 2) The update may copy unexpected data from slots [N, M-1]. +*/ +if (bpfptr_is_kernel (uattr) &&bpf_map_supports_cpu_flags (map->map_type) && + !(attr->flags & (BPF_F_CPU |BPF_F_ALL_CPUS ))) + attr->flags |=BPF_F_ALL_CPUS ; + err =bpf_map_check_op_flags (map, attr->flags, ~0 );if (err) goto err_put;
Signature · 1 line
2.53.0
PATCH v4 6/8 bpftool: Generate skeleton for global percpu data
In reply to Leon HwangEnhance bpftool to generate skeletons that properly handle global percpu variables. The generated skeleton now includes a dedicated structure for percpu data, allowing users to initialize and access percpu variables more efficiently.
For global percpu variables, the skeleton now includes a nested structure, e.g.:
struct test_global_percpu_data {
struct bpf_object_skeleton *skeleton;
struct bpf_object *obj;
struct {
struct bpf_map *percpu;
} maps;
// ...
struct test_global_percpu_data__percpu {
int data;
char run;
struct {
char set;
int i;
int nums[7];
} struct_data;
int nums[7];
} *percpu;
// ...
}; * The "struct test_global_percpu_data__percpu *percpu" points to
initialized data, which is actually "maps.percpu->mmaped".
* Before loading the skeleton, updating the
"struct test_global_percpu_data__percpu *percpu" modifies the initial value of the corresponding global percpu variables.
* After loading the skeleton, "maps.percpu->mmaped" has been marked as read-only. If users want to update the global percpu variables, they have to update the "maps.percpu" map instead.
--- tools/bpf/bpftool/gen.c | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-)
tools/bpf/bpftool/gen.c +14 -10
@@ -92,7 +92,7 @@ static void get_header_guard(char *guard, const char *obj_name, const char *suffstatic bool get_map_ident (const struct bpf_map *map,char *buf,size_t buf_sz) { -static const char *sfxs[] = {".data" ,".rodata" ,".bss" ,".kconfig" }; +static const char *sfxs[] = {".data" ,".rodata" ,".bss" ,".percpu" ,".kconfig" };const char *name =bpf_map__name (map);int i, n;
@@ -117,7 +117,7 @@ static bool get_map_ident(const struct bpf_map *map, char *buf, size_t buf_sz)static bool get_datasec_ident (const char *sec_name,char *buf,size_t buf_sz) { -static const char *pfxs[] = {".data" ,".rodata" ,".bss" ,".kconfig" }; +static const char *pfxs[] = {".data" ,".rodata" ,".bss" ,".percpu" ,".kconfig" };int i, n;/* recognize hard coded LLVM section name */
@@ -254,6 +254,11 @@ static const struct btf_type *find_type_for_map(struct btf *btf, const char *mapreturn NULL ; } +static bool bpf_map_is_percpu_data (const struct bpf_map *map) +{ +return bpf_map__is_internal (map) &&bpf_map__type (map) ==BPF_MAP_TYPE_PERCPU_ARRAY ; +} +static bool is_mmapable_map (const struct bpf_map *map,char *buf,size_t sz) {size_t tmp_sz;
@@ -263,13 +268,12 @@ static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)return true; } -if (!bpf_map__is_internal (map) || !(bpf_map__map_flags (map) &BPF_F_MMAPABLE )) -return false; - -if (!get_map_ident (map, buf, sz)) -return false; +if (bpf_map__is_internal (map) && + ((bpf_map__map_flags (map) &BPF_F_MMAPABLE ) ||bpf_map_is_percpu_data (map)) && +get_map_ident (map, buf, sz)) +return true; -return true; +return false; }static int codegen_datasecs (struct bpf_object *obj,const char *obj_name)
@@ -669,7 +673,7 @@ static void codegen_destroy(struct bpf_object *obj, const char *obj_name)if (!get_map_ident (map, ident,sizeof (ident)))continue ;if (bpf_map__is_internal (map) && - (bpf_map__map_flags (map) &BPF_F_MMAPABLE )) + ((bpf_map__map_flags (map) &BPF_F_MMAPABLE ) ||bpf_map_is_percpu_data (map)))printf ("\tskel_free_map_data(skel->%1$s, skel->maps.%1$s.initial_value, %2$zu);\n" , ident,bpf_map_mmap_sz (map)); codegen("\
@@ -847,7 +851,7 @@ static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *hbpf_object__for_each_map (map, obj) {const char *mmap_flags; -if (!is_mmapable_map (map, ident,sizeof (ident))) +if (!is_mmapable_map (map, ident,sizeof (ident)) ||bpf_map_is_percpu_data (map))continue ;if (bpf_map__map_flags (map) &BPF_F_RDONLY_PROG )
Signature · 1 line
2.53.0
PATCH v4 7/8 selftests/bpf: Add tests to verify global percpu data
In reply to Leon HwangIf the arch, like s390x, does not support percpu insn, these cases won't test global percpu data by checking -EOPNOTSUPP after loading prog.
The following APIs have been tested for global percpu data:
1. bpf_map__set_initial_value()
2. bpf_map__initial_value()
3. generated percpu struct pointer pointing to internal map's mmaped data
4. bpf_map__lookup_elem() for global percpu data map
At the same time, the case is also tested with 'bpftool gen skeleton -L'.
The verifier log is also tested.
--- tools/testing/selftests/bpf/Makefile | 2 +- .../bpf/prog_tests/global_data_init.c | 223 ++++++++++++++++++ .../bpf/progs/test_global_percpu_data.c | 66 ++++++ 3 files changed, 290 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/progs/test_global_percpu_data.c
tools/testing/selftests/bpf/Makefile +1 -1
@@ -504,7 +504,7 @@ LSKELS_SIGNED := fentry_test.c fexit_test.c atomics.c # Generate both light skeleton and libbpf skeleton for these LSKELS_EXTRA := test_ksyms_module.c test_ksyms_weak.c kfunc_call_test.c \ - kfunc_call_test_subprog.c + kfunc_call_test_subprog.c test_global_percpu_data.c SKEL_BLACKLIST += $$(LSKELS) $$(LSKELS_SIGNED) test_static_linked.skel.h-deps := test_static_linked1.bpf.o test_static_linked2.bpf.o
tools/testing/selftests/bpf/prog_tests/global_data_init.c +223 -0
231 lines not on this page · raw message
tools/testing/selftests/bpf/progs/test_global_percpu_data.c new file +66 -0
66 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH v4 8/8 selftests/bpf: Add a test to verify bpf_iter for global percpu data
In reply to Leon HwangAdd a test to verify that it is OK to iter the percpu_array map used for global percpu data.
--- .../bpf/prog_tests/global_data_init.c | 53 +++++++++++++++++++ .../bpf/progs/test_global_percpu_data.c | 36 +++++++++++++ 2 files changed, 89 insertions(+)
tools/testing/selftests/bpf/prog_tests/global_data_init.c +53 -0
@@ -269,6 +269,57 @@ static void test_global_percpu_data_verifier_failure(void)RUN_TESTS (test_global_percpu_data); } +static void test_global_percpu_data_iter (void ) +{ +DECLARE_LIBBPF_OPTS (bpf_iter_attach_opts, opts); +struct test_global_percpu_data *skel; +union bpf_iter_link_info linfo = {}; +struct bpf_link *link =NULL ; +int fd, num_cpus, len, err; +char buf[16 ]; + + num_cpus =libbpf_num_possible_cpus (); +if (!ASSERT_GT (num_cpus,0 ,"libbpf_num_possible_cpus" )) +return ; + + skel =test_global_percpu_data__open (); +if (!ASSERT_OK_PTR (skel,"test_global_percpu_data__open" )) +return ; + + skel->rodata->num_cpus = num_cpus; + skel->percpu->struct_data.nums[6 ] =0xc0de ; + + err =test_global_percpu_data__load (skel); +if (err == -EOPNOTSUPP ) { +test__skip (); + goto out; + } +if (!ASSERT_OK (err,"test_global_percpu_data__load" )) + goto out; + + linfo.map.map_fd =bpf_map__fd (skel->maps.percpu); + opts.link_info = &linfo; + opts.link_info_len =sizeof (linfo); + link =bpf_program__attach_iter (skel->progs.dump_percpu_data, &opts); +if (!ASSERT_OK_PTR (link,"bpf_program__attach_iter" )) + goto out; + + fd =bpf_iter_create (bpf_link__fd (link)); +if (!ASSERT_GE (fd,0 ,"bpf_iter_create" )) + goto out; + +while ((len =read (fd, buf,sizeof (buf))) >0 ) +do { }while (0 ); +ASSERT_EQ (len,0 ,"read iter" ); +ASSERT_TRUE (skel->bss->run_iter,"run_iter" ); +ASSERT_EQ (skel->bss->percpu_data_sum,0xc0de * num_cpus,"percpu_data_sum" ); + +close (fd); +out: +bpf_link__destroy (link); +test_global_percpu_data__destroy (skel); +} +void test_global_percpu_data (void ) {if (!feat_supported (NULL ,FEAT_PERCPU_DATA )) {
@@ -282,4 +333,6 @@ void test_global_percpu_data(void)test_global_percpu_data_lskel ();if (test__start_subtest ("verifier_failure" ))test_global_percpu_data_verifier_failure (); +if (test__start_subtest ("iter" )) +test_global_percpu_data_iter (); }
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +36 -0
@@ -63,4 +63,40 @@ int verifier_percpu_read(void *ctx)return c =='d' ; } +volatile const __u32 num_cpus =0 ; +__u32 percpu_data_sum =0 ; +bool run_iter = false; + +SEC("iter/bpf_map_elem") +__auxiliary +int dump_percpu_data (struct bpf_iter__bpf_map_elem *ctx) +{ +struct { +int data; +char run; +struct { +char set; +int i; +int nums[7 ]; + } struct_data; +int nums[7 ]; + } *pptr = ctx->value; +__u32 step; +int i; + +if (!pptr) +return 0 ; + + run_iter = true; + +/* percpu array element size is aligned to 8 */ + step = (sizeof (*pptr) +7 ) & ~7 ; + +for (i =0 ; i < num_cpus; i++) { + percpu_data_sum += pptr->struct_data.nums[6 ]; + pptr = (void *)pptr + step; + } +return 0 ; +} + char _license[] SEC("license") = "GPL";
Signature · 1 line
2.53.0
[email protected]
AI reviewThis correctly prevents constant folding of percpu array values in check_mem_access(). However, should the same exclusion also be added to const_reg_xfer() in kernel/bpf/const_fold.c?
kernel/bpf/verifier.c static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · line 6366
Quoted 11 lines
kernel/bpf/verifier.c +2
-1@@ -6362,7 +6362,8 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regnif (tnum_is_const (reg->var_off) &&bpf_map_is_rdonly (map) && map->ops->map_direct_value_addr && - map->map_type !=BPF_MAP_TYPE_INSN_ARRAY ) { + map->map_type !=BPF_MAP_TYPE_INSN_ARRAY && + map->map_type !=BPF_MAP_TYPE_PERCPU_ARRAY ) {
This correctly prevents constant folding of percpu array values in check_mem_access(). However, should the same exclusion also be added to const_reg_xfer() in kernel/bpf/const_fold.c?
That function has a parallel constant folding path via bpf_map_direct_read() that only excludes BPF_MAP_TYPE_INSN_ARRAY:
kernel/bpf/const_fold.c:const_reg_xfer() {
...
case BPF_LDX:
...
if (!bpf_map_is_rdonly(map) ||
!map->ops->map_direct_value_addr ||
map->map_type == BPF_MAP_TYPE_INSN_ARRAY ||
off < 0 || off + size > map->value_size ||
bpf_map_direct_read(map, off, size, &val, is_ldsx)) {
*dst = unknown;
break;
}
...
}Since percpu_array_map_ops now has map_direct_value_addr set, bpf_map_direct_read() would dereference the raw percpu base pointer directly, rather than the actual per-cpu address. For a frozen read-only percpu array, the constant-folded value could differ from what the program reads at runtime after the BPF_MOV64_PERCPU_REG fixup converts the address to the correct per-cpu copy.
In the v3 series, bpf_map_direct_read() itself had a guard (map->map_type != BPF_MAP_TYPE_ARRAY), which protected all callers. The v4 moved this to caller-side checks but appears to have missed const_reg_xfer().
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24401974087
Quoted 41 lines · On 2026/4/14 22:10, [email protected] wrote:
kernel/bpf/verifier.c static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · line 6366
Quoted 11 lines
kernel/bpf/verifier.c +2
-1@@ -6362,7 +6362,8 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regnif (tnum_is_const (reg->var_off) &&bpf_map_is_rdonly (map) && map->ops->map_direct_value_addr && - map->map_type !=BPF_MAP_TYPE_INSN_ARRAY ) { + map->map_type !=BPF_MAP_TYPE_INSN_ARRAY && + map->map_type !=BPF_MAP_TYPE_PERCPU_ARRAY ) {This correctly prevents constant folding of percpu array values in
check_mem_access(). However, should the same exclusion also be added toconst_reg_xfer()inkernel/bpf/const_fold.c?That function has a parallel constant folding path via
bpf_map_direct_read()that only excludesBPF_MAP_TYPE_INSN_ARRAY:kernel/bpf/const_fold.c:const_reg_xfer() { ... case BPF_LDX: ... if (!bpf_map_is_rdonly(map) || !map->ops->map_direct_value_addr || map->map_type == BPF_MAP_TYPE_INSN_ARRAY || off < 0 || off + size > map->value_size || bpf_map_direct_read(map, off, size, &val, is_ldsx)) { *dst = unknown; break; } ... }Since
percpu_array_map_opsnow hasmap_direct_value_addrset,bpf_map_direct_read()would dereference the raw percpu base pointer directly, rather than the actual per-cpu address. For a frozen read-only percpu array, the constant-folded value could differ from what the program reads at runtime after theBPF_MOV64_PERCPU_REGfixup converts the address to the correct per-cpu copy.
Quoted 6 lines
In the v3 series,
bpf_map_direct_read()itself had a guard (map->map_type!=BPF_MAP_TYPE_ARRAY), which protected all callers. The v4 moved this to caller-side checks but appears to have missedconst_reg_xfer().
Correct.
Will add a guard in bpf_map_direct_read() in the next revision:
if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) return -EINVAL;
Thanks,
Leon
[email protected]
AI reviewRe: [PATCH bpf-next v4 5/8] bpf: Update per-CPU maps using BPF_F_ALL_CPUS flag
In reply to Leon HwangSashiko AI review found 1 potential issue(s): - [High] The patch unconditionally forces the BPF_F_ALL_CPUS flag for any kernel-originated per-CPU map update that omits CPU flags, breaking the…
Sashiko AI review found 1 potential issue(s):
- [High] The patch unconditionally forces the BPF_F_ALL_CPUS flag for any kernel-originated per-CPU map update that omits CPU flags, breaking the bpf_sys_bpf() helper API for BPF_PROG_TYPE_SYSCALL programs. --
commit 357a4fab1d0fc8586bb52bd3aab10e26fd39a2a6
Author: Leon Hwang <[email protected]>
bpf: Update per-CPU maps using BPF_F_ALL_CPUS flag
This commit modifies map_update_elem() to automatically apply the BPF_F_ALL_CPUS flag for kernel-originated updates to per-CPU maps when no CPU flags are specified. This is intended to fix potential out-of-bounds reads when updating maps via the lightweight skeleton loader on systems where the number of cached slots is less than the number of CPUs.
Quoted 12 lines
kernel/bpf/syscall.c +4
-0@@ -1785,6 +1785,21 @@ static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr) goto err_put; } +/* +* When updating per-CPU maps via the lightweight skeleton +* loader, use a single value slot across all CPUs. This avoids +* two potential issues when updating on an M-CPU kernel with
kernel/bpf/syscall.c static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr) · line 1801
Quoted 10 lines
kernel/bpf/syscall.c +10
-0+* N cached slots (N < M), especially when N is much smaller +* than M: +* 1) The update may trigger a page fault when copying data from +* the last slot, as the read may go beyond the allocated +* buffer. +* 2) The update may copy unexpected data from slots [N, M-1]. +*/ +if (bpfptr_is_kernel (uattr) &&bpf_map_supports_cpu_flags (map->map_type) && + !(attr->flags & (BPF_F_CPU |BPF_F_ALL_CPUS ))) + attr->flags |=BPF_F_ALL_CPUS ;
Does this unintentionally break the bpf_sys_bpf() helper API for BPF_PROG_TYPE_SYSCALL programs?
Because bpf_sys_bpf() also wraps the pointer with KERNEL_BPFPTR, bpfptr_is_kernel(uattr) will evaluate to true for any eBPF program calling it.
Prior to this patch, a BPF_PROG_TYPE_SYSCALL program could legitimately update a per-CPU map with distinct values for each CPU by providing an array of num_possible_cpus() elements and omitting any CPU flags, which mirrors standard user-space API semantics.
With this change, it appears the kernel silently overrides attr->flags by setting BPF_F_ALL_CPUS. This truncates the internal value_size calculation down to a single element (map->value_size), reads only the first element of the BPF program's provided buffer, and incorrectly replicates it across all CPUs.
Should this workaround for the skeleton loader be constrained to bpftool gen skeleton instead of mutating the core syscall handler?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
[email protected]
AI reviewRe: [PATCH bpf-next v4 6/8] bpftool: Generate skeleton for global percpu data
In reply to Leon HwangSashiko AI review found 1 potential issue(s): - [High] Missing read-only memory protection for `.percpu` initial value buffer in generated light skeletons. --
Sashiko AI review found 1 potential issue(s):
- [High] Missing read-only memory protection for .percpu initial value buffer in generated light skeletons. --
commit e61b5ee6e5a7cee758b644de0f6b4aaea40f020c
Author: Leon Hwang <[email protected]>
bpftool: Generate skeleton for global percpu data
Enhance bpftool to generate skeletons that properly handle global percpu variables. The skeleton includes a nested structure for percpu data, which is marked as read-only after loading so users must update the map directly.
Quoted 3 lines
tools/bpf/bpftool/gen.c +0
-0
[ ... ]
tools/bpf/bpftool/gen.c static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *h · line 857
Quoted 9 lines
tools/bpf/bpftool/gen.c +1
-1@@ -847,7 +851,7 @@ static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *hbpf_object__for_each_map (map, obj) {const char *mmap_flags; -if (!is_mmapable_map (map, ident,sizeof (ident))) +if (!is_mmapable_map (map, ident,sizeof (ident)) ||bpf_map_is_percpu_data (map))continue ;if (bpf_map__map_flags (map) &BPF_F_RDONLY_PROG )
The commit message notes that after loading the skeleton, maps.percpu->mmaped has been marked as read-only.
By skipping the .percpu map entirely here during light skeleton generation in gen_trace(), does this leave the user-space buffer for .percpu writable?
If a user modifies skel->percpu after the light skeleton is loaded, will it silently succeed without actually updating the kernel per-cpu map? Does the generated light skeleton need an explicit mprotect(..., PROT_READ) for the .percpu initial value buffer to enforce the read-only constraint?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
[email protected]
AI reviewRe: [PATCH bpf-next v4 7/8] selftests/bpf: Add tests to verify global percpu data
In reply to Leon HwangSashiko AI review found 1 potential issue(s): - [Medium] The tests use the raw CPU ID to index into the densely packed `percpu_data` array, which will cause an out-of-bounds read and test failures…
Sashiko AI review found 1 potential issue(s):
- [Medium] The tests use the raw CPU ID to index into the densely packed percpu_data array, which will cause an out-of-bounds read and test failures on systems with sparse CPU masks. --
commit a7c2685b539ebc816cdb05decab1d7a8a4e1328e
Author: Leon Hwang <[email protected]>
selftests/bpf: Add tests to verify global percpu data
This commit adds new selftests to verify the behavior of global percpu data by checking map initial values, struct pointers, and looking up elements from global percpu data maps.
Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +0
-0
[ ... ]
Quoted 2 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +2
-0+static void test_global_percpu_data_init (void ) +{
[ ... ]
tools/testing/selftests/bpf/prog_tests/global_data_init.c void test_global_data_init(void) · line 177
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +10
-0+ err =bpf_map__lookup_elem (map, &key,sizeof (key), percpu_data, + elem_sz * num_cpus,0 ); +if (!ASSERT_OK (err,"bpf_map__lookup_elem" )) + goto out; + +for (i =0 ; i < num_online; i++) { +if (!online[i]) +continue ; + + data = (void *)percpu_data + elem_sz * i;
Will this logic cause an out-of-bounds read on systems with sparse CPU masks?
When reading a per-CPU map using bpf_map__lookup_elem(), the kernel packs the per-CPU values densely into the output buffer by iterating over the possible CPU mask.
The test allocates num_cpus (the total count of possible CPUs) elements for the output buffer, but then iterates over the raw CPU IDs using i and uses i as an index into the densely packed array.
On systems with sparse CPU masks (for example, where CPUs 0 and 100 are possible/online), num_cpus will be 2. When the loop reaches i = 100, it appears it will attempt to access percpu_data + elem_sz * 100, reading past the bounds of the allocated memory.
Could we maintain a separate index counter that increments only for possible CPUs to match the kernel's dense packing behavior?
[ ... ]
Quoted 2 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +2
-0+static void test_global_percpu_data_lskel (void ) +{
[ ... ]
tools/testing/selftests/bpf/prog_tests/global_data_init.c void test_global_data_init(void) · line 252
Quoted 9 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +9
-0+ err =bpf_map_lookup_elem (map_fd, &key, percpu_data); +if (!ASSERT_OK (err,"bpf_map_lookup_elem" )) + goto out; + +for (i =0 ; i < num_online; i++) { +if (!online[i]) +continue ; + + data = (void *)percpu_data + elem_sz * i;
Does this loop have the same bounds checking issue as mentioned above?
If i is a raw CPU ID on a system with a sparse CPU mask, this calculation could exceed the bounds of the percpu_data buffer.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
[email protected]
AI reviewRe: [PATCH bpf-next v4 8/8] selftests/bpf: Add a test to verify bpf_iter for global percpu data
In reply to Leon HwangSashiko AI review found 3 potential issue(s): - [Medium] The `dump_percpu_data` program is autoloaded by default, breaking existing subtests on older kernels. - [Medium] Checking `err ==…
Sashiko AI review found 3 potential issue(s):
- [Medium] The dump_percpu_data program is autoloaded by default, breaking existing subtests on older kernels.
- [Medium] Checking err == -EOPNOTSUPP after skeleton load incorrectly handles unsupported kernels.
- [Medium] Casting the global datasec to an anonymous C struct relies on fragile compiler layout assumptions. --
commit 1e00e3e2b318918e67a2c7d2e251132e1b985484
Author: Leon Hwang <[email protected]>
selftests/bpf: Add a test to verify bpf_iter for global percpu data
Add a test to verify that it is OK to iter the percpu_array map used for global percpu data.
Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +0
-0
[ ... ]
Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +2
-0@@ -269,6 +269,57 @@ static void test_global_percpu_data_verifier_failure(void) +static void test_global_percpu_data_iter (void ) +{
[ ... ]
tools/testing/selftests/bpf/prog_tests/global_data_init.c static void test_global_percpu_data_verifier_failure(void) · line 296
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +5
-0+ err =test_global_percpu_data__load (skel); +if (err == -EOPNOTSUPP ) { +test__skip (); + goto out; + }
Will this correctly skip on older kernels? When the kernel verifier rejects an unsupported program type or feature during bpf_prog_load(), it typically returns -EINVAL (or -EACCES), not -EOPNOTSUPP.
Libbpf passes this error code back to the caller, so this condition might evaluate to false, causing the test to fail on the subsequent ASSERT_OK(err) check instead of gracefully skipping.
[ ... ]
Quoted 4 lines
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +0
-0
[ ... ]
tools/testing/selftests/bpf/progs/test_global_percpu_data.c int verifier_percpu_read(void *ctx) · line 72
Quoted 3 lines
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +3
-0+SEC("iter/bpf_map_elem") +__auxiliary +int dump_percpu_data (struct bpf_iter__bpf_map_elem *ctx)
Will this break existing subtests on older kernels? The bpf program dump_percpu_data is defined with SEC("iter/bpf_map_elem") without the ? prefix, meaning libbpf will attempt to autoload it whenever test_global_percpu_data__load() is called.
Since the existing init and lskel subtests also load the full skeleton, they might inadvertently attempt to load this new iterator program. Should this use SEC("?iter/bpf_map_elem") to prevent autoloading by default, and only be explicitly autoloaded in the iter subtest?
tools/testing/selftests/bpf/progs/test_global_percpu_data.c int verifier_percpu_read(void *ctx) · line 83
Quoted 11 lines
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +11
-0+{ +struct { +int data; +char run; +struct { +char set; +int i; +int nums[7 ]; + } struct_data; +int nums[7 ]; + } *pptr = ctx->value;
Is it safe to cast the global datasec to an anonymous C struct like this?
In the BPF source, these are declared as independent global variables. The layout of these variables in the .percpu ELF section is determined entirely by the compiler. While the current compiler version might emit the uninitialized nums array after the initialized variables (matching the struct layout), this relies on undocumented compiler behavior.
If the compiler alters the layout, or if variables are added or initialized differently in the future, the field offsets will mismatch. This could cause the step calculation to diverge from the map's actual value_size, leading to out-of-bounds reads.
Would it be safer to group these variables into a single explicit struct in the BPF code to guarantee their memory layout?
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
Quoted 48 lines · Leon Hwang
Quoted 41 lines · On 2026/4/14 22:10, [email protected] wrote:
kernel/bpf/verifier.c static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · line 6366
Quoted 11 lines
kernel/bpf/verifier.c +2
-1@@ -6362,7 +6362,8 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regnif (tnum_is_const (reg->var_off) &&bpf_map_is_rdonly (map) && map->ops->map_direct_value_addr && - map->map_type !=BPF_MAP_TYPE_INSN_ARRAY ) { + map->map_type !=BPF_MAP_TYPE_INSN_ARRAY && + map->map_type !=BPF_MAP_TYPE_PERCPU_ARRAY ) {This correctly prevents constant folding of percpu array values in
check_mem_access(). However, should the same exclusion also be added toconst_reg_xfer()inkernel/bpf/const_fold.c?That function has a parallel constant folding path via
bpf_map_direct_read()that only excludesBPF_MAP_TYPE_INSN_ARRAY:kernel/bpf/const_fold.c:const_reg_xfer() { ... case BPF_LDX: ... if (!bpf_map_is_rdonly(map) || !map->ops->map_direct_value_addr || map->map_type == BPF_MAP_TYPE_INSN_ARRAY || off < 0 || off + size > map->value_size || bpf_map_direct_read(map, off, size, &val, is_ldsx)) { *dst = unknown; break; } ... }Since
percpu_array_map_opsnow hasmap_direct_value_addrset,bpf_map_direct_read()would dereference the raw percpu base pointer directly, rather than the actual per-cpu address. For a frozen read-only percpu array, the constant-folded value could differ from what the program reads at runtime after theBPF_MOV64_PERCPU_REGfixup converts the address to the correct per-cpu copy.Quoted 6 lines
In the v3 series,
bpf_map_direct_read()itself had a guard (map->map_type!=BPF_MAP_TYPE_ARRAY), which protected all callers. The v4 moved this to caller-side checks but appears to have missedconst_reg_xfer().
Quoted 6 lines
Correct.
Will add a guard in
bpf_map_direct_read()in the next revision:if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) return -EINVAL;
hold on.
map->ops->map_direct_value_addr &&
- map->map_type != BPF_MAP_TYPE_INSN_ARRAY) {
+ map->map_type != BPF_MAP_TYPE_INSN_ARRAY &&
+ map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) {map_direct_value_addr() is set, but then immediately disallowed ? Where else it's used?
Even if value_addr is working, then map_direct_value_meta() looks broken.
Re: [PATCH bpf-next v4 5/8] bpf: Update per-CPU maps using BPF_F_ALL_CPUS flag
In reply to Leon HwangQuoted 26 lines · Leon Hwang
When updating per-CPU maps via the lightweight skeleton loader, use a single value slot across all CPUs. This avoids two potential issues when updating on an M-CPU kernel with N cached slots (N < M), especially when N is much smaller than M:
1) The update may trigger a page fault when copying data from the last slot, as the read may go beyond the allocated buffer.
2) The update may copy unexpected data from slots [N, M-1].Signed-off-by Leon Hwang <[email protected]>--- kernel/bpf/syscall.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+)kernel/bpf/syscall.c +4
-0@@ -1785,6 +1785,21 @@ static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr) goto err_put; } +/* +* When updating per-CPU maps via the lightweight skeleton +* loader, use a single value slot across all CPUs. This avoids +* two potential issues when updating on an M-CPU kernel with
kernel/bpf/syscall.c static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr) · line 1801
Quoted 10 lines
kernel/bpf/syscall.c +10
-0+* N cached slots (N < M), especially when N is much smaller +* than M: +* 1) The update may trigger a page fault when copying data from +* the last slot, as the read may go beyond the allocated +* buffer. +* 2) The update may copy unexpected data from slots [N, M-1]. +*/ +if (bpfptr_is_kernel (uattr) &&bpf_map_supports_cpu_flags (map->map_type) && + !(attr->flags & (BPF_F_CPU |BPF_F_ALL_CPUS ))) + attr->flags |=BPF_F_ALL_CPUS ;
This looks like a hack. It's not addressing the actual bug. If there is a bug submit it separately with fixes tag.
Quoted 2 lines · Alexei Starovoitov
Quoted 1 line · Leon Hwang
On 2026/4/14 22:10, [email protected] wrote:
[...]
Quoted 19 lines
Quoted 6 lines
Quoted 6 lines
In the v3 series,
bpf_map_direct_read()itself had a guard (map->map_type!=BPF_MAP_TYPE_ARRAY), which protected all callers. The v4 moved this to caller-side checks but appears to have missedconst_reg_xfer().Quoted 6 lines
Correct.
Will add a guard in
bpf_map_direct_read()in the next revision:if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) return -EINVAL;hold on.
kernel/bpf/verifier.c +2
-1map->ops->map_direct_value_addr && - map->map_type != BPF_MAP_TYPE_INSN_ARRAY) { + map->map_type != BPF_MAP_TYPE_INSN_ARRAY && + map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) {
Quoted 5 lines
map_direct_value_addr()is set, but then immediately disallowed ? Where else it's used?Even if
value_addris working, thenmap_direct_value_meta()looks broken.
Ah, let me dive deeper.
Thanks,
Leon
Re: [PATCH bpf-next v4 5/8] bpf: Update per-CPU maps using BPF_F_ALL_CPUS flag
In reply to Alexei StarovoitovQuoted 32 lines · Alexei Starovoitov
Quoted 31 lines · Leon Hwang
When updating per-CPU maps via the lightweight skeleton loader, use a single value slot across all CPUs. This avoids two potential issues when updating on an M-CPU kernel with N cached slots (N < M), especially when N is much smaller than M:
1) The update may trigger a page fault when copying data from the last slot, as the read may go beyond the allocated buffer.
2) The update may copy unexpected data from slots [N, M-1].Signed-off-by Leon Hwang <[email protected]>--- kernel/bpf/syscall.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+)kernel/bpf/syscall.c +9
-0@@ -1785,6 +1785,21 @@ static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr) goto err_put; } +/* +* When updating per-CPU maps via the lightweight skeleton +* loader, use a single value slot across all CPUs. This avoids +* two potential issues when updating on an M-CPU kernel with +* N cached slots (N < M), especially when N is much smaller +* than M: +* 1) The update may trigger a page fault when copying data from +* the last slot, as the read may go beyond the allocated +* buffer.
Quoted 8 lines
kernel/bpf/syscall.c static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr) · line 1801
Quoted 5 lines
kernel/bpf/syscall.c +5
-0+* 2) The update may copy unexpected data from slots [N, M-1]. +*/ +if (bpfptr_is_kernel (uattr) &&bpf_map_supports_cpu_flags (map->map_type) && + !(attr->flags & (BPF_F_CPU |BPF_F_ALL_CPUS ))) + attr->flags |=BPF_F_ALL_CPUS ;This looks like a hack. It's not addressing the actual bug. If there is a bug submit it separately with fixes tag.
Sure, will verify whether it is a bug. If it is, will fix it with separate patch.
Thanks,
Leon
Re: [PATCH bpf-next v4 5/8] bpf: Update per-CPU maps using BPF_F_ALL_CPUS flag
In reply to [email protected]Quoted 1 line · On 15/4/26 05:02, [email protected] wrote:
Sashiko AI review found 1 potential issue(s):
[...]
Quoted 26 lines
Quoted 4 lines
kernel/bpf/syscall.c +4
-0+/* +* When updating per-CPU maps via the lightweight skeleton +* loader, use a single value slot across all CPUs. This avoids +* two potential issues when updating on an M-CPU kernel with kernel/bpf/syscall.c static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr) · line 1801
Quoted 10 lines
kernel/bpf/syscall.c +10
-0+* N cached slots (N < M), especially when N is much smaller +* than M: +* 1) The update may trigger a page fault when copying data from +* the last slot, as the read may go beyond the allocated +* buffer. +* 2) The update may copy unexpected data from slots [N, M-1]. +*/ +if (bpfptr_is_kernel (uattr) &&bpf_map_supports_cpu_flags (map->map_type) && + !(attr->flags & (BPF_F_CPU |BPF_F_ALL_CPUS ))) + attr->flags |=BPF_F_ALL_CPUS ;Does this unintentionally break the
bpf_sys_bpf()helper API forBPF_PROG_TYPE_SYSCALLprograms?Because
bpf_sys_bpf()also wraps the pointer withKERNEL_BPFPTR,bpfptr_is_kernel(uattr)will evaluate to true for any eBPF program calling it.Prior to this patch, a
BPF_PROG_TYPE_SYSCALLprogram could legitimately update a per-CPU map with distinct values for each CPU by providing an array ofnum_possible_cpus()elements and omitting any CPU flags, which mirrors standard user-space API semantics.
Quoted 9 lines
With this change, it appears the kernel silently overrides
attr->flagsby settingBPF_F_ALL_CPUS. This truncates the internalvalue_sizecalculation down to a single element (map->value_size), reads only the first element of the BPF program's provided buffer, and incorrectly replicates it across all CPUs.Should this workaround for the skeleton loader be constrained to bpftool gen skeleton instead of mutating the core syscall handler?
It was my intent. Even though BPF_PROG_TYPE_SYSCALL progs could be broken, it should be fixed when verifying the oob issue is true for SYSCALL progs.
This patch will be dropped from this series. And, a separate patch will be posted if the oob issue is true.
Thanks,
Leon
Re: [PATCH bpf-next v4 6/8] bpftool: Generate skeleton for global percpu data
In reply to [email protected]Quoted 32 lines · On 15/4/26 05:26, [email protected] wrote:
Sashiko AI review found 1 potential issue(s):
- [High] Missing read-only memory protection for.percpuinitial value buffer in generated light skeletons. --commit
e61b5ee6e5a7cee758b644de0f6b4aaea40f020c
Author: Leon Hwang <[email protected]>bpftool: Generate skeleton for global percpu data
Enhance bpftool to generate skeletons that properly handle global percpu variables. The skeleton includes a nested structure for percpu data, which is marked as read-only after loading so users must update the map directly.
Quoted 3 lines
tools/bpf/bpftool/gen.c +0
-0[ ... ]
tools/bpf/bpftool/gen.c static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *h · line 857
Quoted 9 lines
tools/bpf/bpftool/gen.c +1
-1@@ -847,7 +851,7 @@ static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *hbpf_object__for_each_map (map, obj) {const char *mmap_flags; -if (!is_mmapable_map (map, ident,sizeof (ident))) +if (!is_mmapable_map (map, ident,sizeof (ident)) ||bpf_map_is_percpu_data (map))continue ;if (bpf_map__map_flags (map) &BPF_F_RDONLY_PROG )The commit message notes that after loading the skeleton,
maps.percpu->mmapedhas been marked as read-only.
Quoted 8 lines
By skipping the .percpu map entirely here during light skeleton generation in
gen_trace(), does this leave the user-space buffer for .percpu writable?If a user modifies
skel->percpuafter the light skeleton is loaded, will it silently succeed without actually updating the kernel per-cpu map? Does the generated light skeleton need an explicitmprotect(..., PROT_READ)for the .percpu initial value buffer to enforce the read-only constraint?
The mprotect(..., PROT_READ) was added in patch #4.
I think it'd better to move the change to this patch. Then, readers won't miss the mprotect().
Thanks,
Leon
Re: [PATCH bpf-next v4 7/8] selftests/bpf: Add tests to verify global percpu data
In reply to [email protected]Quoted 72 lines · On 15/4/26 05:45, [email protected] wrote:
Sashiko AI review found 1 potential issue(s):
- [Medium] The tests use the raw CPU ID to index into the densely packedpercpu_dataarray, which will cause an out-of-bounds read and test failures on systems with sparse CPU masks. --commit
a7c2685b539ebc816cdb05decab1d7a8a4e1328e
Author: Leon Hwang <[email protected]>selftests/bpf: Add tests to verify global percpu data
This commit adds new selftests to verify the behavior of global percpu data by checking map initial values, struct pointers, and looking up elements from global percpu data maps.
Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +0
-0[ ... ]
Quoted 2 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +2
-0+static void test_global_percpu_data_init (void ) +{[ ... ]
tools/testing/selftests/bpf/prog_tests/global_data_init.c void test_global_data_init(void) · line 177
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +10
-0+ err =bpf_map__lookup_elem (map, &key,sizeof (key), percpu_data, + elem_sz * num_cpus,0 ); +if (!ASSERT_OK (err,"bpf_map__lookup_elem" )) + goto out; + +for (i =0 ; i < num_online; i++) { +if (!online[i]) +continue ; + + data = (void *)percpu_data + elem_sz * i;Will this logic cause an out-of-bounds read on systems with sparse CPU masks?
When reading a per-CPU map using
bpf_map__lookup_elem(), the kernel packs the per-CPU values densely into the output buffer by iterating over the possible CPU mask.The test allocates
num_cpus(the total count of possible CPUs) elements for the output buffer, but then iterates over the raw CPU IDs using i and uses i as an index into the densely packed array.On systems with sparse CPU masks (for example, where CPUs 0 and 100 are possible/online),
num_cpuswill be 2. When the loop reaches i = 100, it appears it will attempt to accesspercpu_data+elem_sz* 100, reading past the bounds of the allocated memory.Could we maintain a separate index counter that increments only for possible CPUs to match the kernel's dense packing behavior?
[ ... ]
Quoted 2 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +2
-0+static void test_global_percpu_data_lskel (void ) +{[ ... ]
tools/testing/selftests/bpf/prog_tests/global_data_init.c void test_global_data_init(void) · line 252
Quoted 9 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +9
-0+ err =bpf_map_lookup_elem (map_fd, &key, percpu_data); +if (!ASSERT_OK (err,"bpf_map_lookup_elem" )) + goto out; + +for (i =0 ; i < num_online; i++) { +if (!online[i]) +continue ; + + data = (void *)percpu_data + elem_sz * i;Does this loop have the same bounds checking issue as mentioned above?
Quoted 3 lines
If i is a raw CPU ID on a system with a sparse CPU mask, this calculation could exceed the bounds of the
percpu_databuffer.
Yep, agree with the review.
Instead of lookup values across all possible CPUs, it should lookup value on specified CPU using BPF_F_CPU, because the test only runs on online CPUs.
Thanks,
Leon
Re: [PATCH bpf-next v4 8/8] selftests/bpf: Add a test to verify bpf_iter for global percpu data
In reply to [email protected]Quoted 37 lines · On 15/4/26 06:08, [email protected] wrote:
Sashiko AI review found 3 potential issue(s):
- [Medium] Thedump_percpu_dataprogram is autoloaded by default, breaking existing subtests on older kernels.
- [Medium] Checkingerr == -EOPNOTSUPPafter skeleton load incorrectly handles unsupported kernels.
- [Medium] Casting the global datasec to an anonymous C struct relies on fragile compiler layout assumptions. --commit
1e00e3e2b318918e67a2c7d2e251132e1b985484
Author: Leon Hwang <[email protected]>selftests/bpf: Add a test to verify
bpf_iterfor global percpu dataAdd a test to verify that it is OK to iter the
percpu_arraymap used for global percpu data.Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +0
-0[ ... ]
Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +2
-0@@ -269,6 +269,57 @@ static void test_global_percpu_data_verifier_failure(void) +static void test_global_percpu_data_iter (void ) +{[ ... ]
tools/testing/selftests/bpf/prog_tests/global_data_init.c static void test_global_percpu_data_verifier_failure(void) · line 296
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +5
-0+ err =test_global_percpu_data__load (skel); +if (err == -EOPNOTSUPP ) { +test__skip (); + goto out; + }Will this correctly skip on older kernels? When the kernel verifier rejects an unsupported program type or feature during
bpf_prog_load(), it typically returns-EINVAL(or-EACCES), not-EOPNOTSUPP.
Quoted 4 lines
Libbpf passes this error code back to the caller, so this condition might evaluate to false, causing the test to fail on the subsequent
ASSERT_OK(err)check instead of gracefully skipping.
This err check should be dropped, because this test won't run when the kernel does not support global percpu data.
See previous patch:
+void test_global_percpu_data(void)
+{
+ if (!feat_supported(NULL, FEAT_PERCPU_DATA)) {
+ test__skip();
+ return;
+ }
Quoted 18 lines
[ ... ]
Quoted 4 lines
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +0
-0[ ... ]
tools/testing/selftests/bpf/progs/test_global_percpu_data.c int verifier_percpu_read(void *ctx) · line 72
Quoted 3 lines
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +3
-0+SEC("iter/bpf_map_elem") +__auxiliary +int dump_percpu_data (struct bpf_iter__bpf_map_elem *ctx)Will this break existing subtests on older kernels? The bpf program
dump_percpu_datais defined withSEC("iter/bpf_map_elem")without the ? prefix, meaning libbpf will attempt to autoload it whenevertest_global_percpu_data__load()is called.
Quoted 5 lines
Since the existing init and lskel subtests also load the full skeleton, they might inadvertently attempt to load this new iterator program. Should this use
SEC("?iter/bpf_map_elem")to prevent autoloading by default, and only be explicitly autoloaded in the iter subtest?
Since older kernels do not have global percpu data support, this test won't run on them. See aforementioned explanation.
Quoted 26 lines
tools/testing/selftests/bpf/progs/test_global_percpu_data.c int verifier_percpu_read(void *ctx) · line 83
Quoted 11 lines
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +11
-0+{ +struct { +int data; +char run; +struct { +char set; +int i; +int nums[7 ]; + } struct_data; +int nums[7 ]; + } *pptr = ctx->value;Is it safe to cast the global datasec to an anonymous C struct like this?
In the BPF source, these are declared as independent global variables. The layout of these variables in the .percpu ELF section is determined entirely by the compiler. While the current compiler version might emit the uninitialized nums array after the initialized variables (matching the struct layout), this relies on undocumented compiler behavior.
If the compiler alters the layout, or if variables are added or initialized differently in the future, the field offsets will mismatch. This could cause the step calculation to diverge from the map's actual
value_size, leading to out-of-bounds reads.
Quoted 5 lines
Would it be safer to group these variables into a single explicit struct in the BPF code to guarantee their memory layout?
[ ... ]
Correct, I should not use such an anonymous struct, even if it matches the generated skeleton's embedded struct.
A better approach is to locate the int by offset, which will match its offset in the generated skeleton's embedded struct.
Thanks,
Leon
Quoted 23 lines · Leon Hwang
Quoted 2 lines · Alexei Starovoitov
Quoted 1 line · Leon Hwang
On 2026/4/14 22:10, [email protected] wrote:
[...]
Quoted 19 lines
Quoted 6 lines
Quoted 6 lines
In the v3 series,
bpf_map_direct_read()itself had a guard (map->map_type!=BPF_MAP_TYPE_ARRAY), which protected all callers. The v4 moved this to caller-side checks but appears to have missedconst_reg_xfer().Quoted 6 lines
Correct.
Will add a guard in
bpf_map_direct_read()in the next revision:if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) return -EINVAL;hold on.
kernel/bpf/verifier.c +2
-1map->ops->map_direct_value_addr && - map->map_type != BPF_MAP_TYPE_INSN_ARRAY) { + map->map_type != BPF_MAP_TYPE_INSN_ARRAY && + map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) {
Quoted 5 lines
Quoted 5 lines
map_direct_value_addr()is set, but then immediately disallowed ? Where else it's used?Even if
value_addris working, thenmap_direct_value_meta()looks broken.
IIUC, map_direct_value_meta() is only used for dumping xlated insns. If no available map_direct_value_addr(), map_direct_value_meta() won't be called.
Quoted 3 lines
Ah, let me dive deeper.
As for the above changes, let me explain them using diff snippet.
@@ -5808,6 +5808,8 @@ int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val,
u64 addr;
int err;+ if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY)
+ return -EINVAL;
err = map->ops->map_direct_value_addr(map, &addr, off);
if (err)
return err;It is to guard percpu_array map against const_reg_xfer(). Instead of updating const_reg_xfer(), better to update bpf_map_direct_read(). WDYT?
@@ -6362,7 +6364,8 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn
if (tnum_is_const(reg->var_off) &&
bpf_map_is_rdonly(map) &&
map->ops->map_direct_value_addr &&
- map->map_type != BPF_MAP_TYPE_INSN_ARRAY) {
+ map->map_type != BPF_MAP_TYPE_INSN_ARRAY &&
+ map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) {Since directly reading data from global percpu data is different from the way of global data, such reading data for global percpu data must perform the 'this_cpu_off' to read the data on current CPU.
At the moment of load time, the reading cannot read data like the way of global data, also cannot cause verifier failure.
I'll add a test against bpf_map_is_rdonly()+percpu_array.
int map_off = off + reg->var_off.value;
u64 val = 0;
err = bpf_map_direct_read(map, map_off, size,
&val, is_ldsx);
if (err)
return err;
regs[value_regno].type = SCALAR_VALUE;
__mark_reg_known(®s[value_regno], val);Thanks,
Leon
Re: [PATCH bpf-next v4 5/8] bpf: Update per-CPU maps using BPF_F_ALL_CPUS flag
In reply to Leon HwangQuoted 42 lines · Leon Hwang
Quoted 32 lines · Alexei Starovoitov
Quoted 31 lines · Leon Hwang
When updating per-CPU maps via the lightweight skeleton loader, use a single value slot across all CPUs. This avoids two potential issues when updating on an M-CPU kernel with N cached slots (N < M), especially when N is much smaller than M:
1) The update may trigger a page fault when copying data from the last slot, as the read may go beyond the allocated buffer.
2) The update may copy unexpected data from slots [N, M-1].Signed-off-by Leon Hwang <[email protected]>--- kernel/bpf/syscall.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+)kernel/bpf/syscall.c +9
-0@@ -1785,6 +1785,21 @@ static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr) goto err_put; } +/* +* When updating per-CPU maps via the lightweight skeleton +* loader, use a single value slot across all CPUs. This avoids +* two potential issues when updating on an M-CPU kernel with +* N cached slots (N < M), especially when N is much smaller +* than M: +* 1) The update may trigger a page fault when copying data from +* the last slot, as the read may go beyond the allocated +* buffer. Quoted 8 lines
kernel/bpf/syscall.c static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr) · line 1801
Quoted 5 lines
kernel/bpf/syscall.c +5
-0+* 2) The update may copy unexpected data from slots [N, M-1]. +*/ +if (bpfptr_is_kernel (uattr) &&bpf_map_supports_cpu_flags (map->map_type) && + !(attr->flags & (BPF_F_CPU |BPF_F_ALL_CPUS ))) + attr->flags |=BPF_F_ALL_CPUS ;This looks like a hack. It's not addressing the actual bug. If there is a bug submit it separately with fixes tag.
Quoted 3 lines
Sure, will verify whether it is a bug. If it is, will fix it with separate patch.
By implementing two selftests [1] against syscall progs and lightweight skeleton, it is not a real issue.
The assuming oob reading might occur when updating percpu_array map using a small value buffer for syscall progs. However, the oob reading won't make kernel panic, because copy kernel memory using copy_from_kernel_nofault(). However, as for lskel, the oob reading doesn't occur, even set value size as 32000 and update percpu_array map using an int.
The assuming issue #2 that copies unexpected data is true for both syscall progs and lskel. However, I think it's users' responsibility to correctly update percpu_array map using big enough value data.
In conclusion, I'll drop this patch from this series, and won't send separate patch because it is not a bug.
[1] https://github.com/Asphaltt/bpf/commits/bpf/lskel-oob/v1/
Thanks,
Leon
Quoted 28 lines · Leon Hwang
Quoted 26 lines · Leon Hwang
Quoted 2 lines · Alexei Starovoitov
Quoted 1 line · Leon Hwang
On 2026/4/14 22:10, [email protected] wrote:
[...]
Quoted 22 lines
Quoted 6 lines
Quoted 6 lines
In the v3 series,
bpf_map_direct_read()itself had a guard (map->map_type!=BPF_MAP_TYPE_ARRAY), which protected all callers. The v4 moved this to caller-side checks but appears to have missedconst_reg_xfer().Quoted 6 lines
Correct.
Will add a guard in
bpf_map_direct_read()in the next revision:if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) return -EINVAL;hold on.
map->ops->map_direct_value_addr &&kernel/bpf/verifier.c +2
-1- map->map_type != BPF_MAP_TYPE_INSN_ARRAY) { + map->map_type != BPF_MAP_TYPE_INSN_ARRAY && + map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) {
map_direct_value_addr()is set, but then immediately disallowed ? Where else it's used?
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Even if
value_addris working, thenmap_direct_value_meta()looks broken.IIUC,
map_direct_value_meta()is only used for dumping xlated insns. If no availablemap_direct_value_addr(),map_direct_value_meta()won't be called.
yes, but then xlated insn are bogus, no?
Quoted 17 lines
Quoted 3 lines
Ah, let me dive deeper.
As for the above changes, let me explain them using diff snippet.
(quoted diff) +2
-0@@ -5808,6 +5808,8 @@ int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val, u64 addr; int err; + if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) + return -EINVAL; err = map->ops->map_direct_value_addr(map, &addr, off); if (err)return err;
Quoted 2 lines
It is to guard
percpu_arraymap againstconst_reg_xfer(). Instead of updatingconst_reg_xfer(), better to updatebpf_map_direct_read(). WDYT?
yeah and move map_type != BPF_MAP_TYPE_INSN_ARRAY check into bpf_map_direct_read() as well.
To cleanup const_reg_xfer() a bit.
Quoted 29 lines · Alexei Starovoitov
Quoted 28 lines · Leon Hwang
Quoted 26 lines · Leon Hwang
Quoted 2 lines · Alexei Starovoitov
Quoted 1 line · Leon Hwang
On 2026/4/14 22:10, [email protected] wrote:
[...]
Quoted 22 lines
Quoted 6 lines
Quoted 6 lines
In the v3 series,
bpf_map_direct_read()itself had a guard (map->map_type!=BPF_MAP_TYPE_ARRAY), which protected all callers. The v4 moved this to caller-side checks but appears to have missedconst_reg_xfer().Quoted 6 lines
Correct.
Will add a guard in
bpf_map_direct_read()in the next revision:if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) return -EINVAL;hold on.
map->ops->map_direct_value_addr &&(quoted diff) +2
-1- map->map_type != BPF_MAP_TYPE_INSN_ARRAY) { + map->map_type != BPF_MAP_TYPE_INSN_ARRAY && + map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) {
map_direct_value_addr()is set, but then immediately disallowed ? Where else it's used?
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Even if
value_addris working, thenmap_direct_value_meta()looks broken.IIUC,
map_direct_value_meta()is only used for dumping xlated insns. If no availablemap_direct_value_addr(),map_direct_value_meta()won't be called.yes, but then xlated insn are bogus, no?
No. They look well.
./test_progs -t global_percpu_data -v
XLATED:
=============
0: r1 = 0x900000007
2: r1 = &(void __percpu *)(r1)
3: r1 = *(u8 *)(r1 +0)
4: w0 = 1
5: if w1 == 0x64 goto pc+1
6: w0 = 0
7: exit
=============
#152/5 global_percpu_data/verifier_percpu_read:OK
I also verified the xlated insns via bpftool.
static SEC(".data") __u32 cnt = 0; /* .data vs .percpu */
SEC("xdp")
int xdp_prog(struct xdp_md *ctx)
{
cnt++;
__u32 cpu = bpf_get_smp_processor_id();
bpf_printk("cpu: %u, cnt: %u\n", cpu, cnt);
return XDP_PASS;
}When the section of cnt is ".data", bpftool d x n xdp_prog:
int xdp_prog(struct xdp_md * ctx):
; cnt++;
0: (18) r6 = map[id:23][0]+0
2: (61) r1 = *(u32 *)(r6 +0)
; cnt++;
3: (07) r1 += 1
; cnt++;
4: (63) *(u32 *)(r6 +0) = r1
; __u32 cpu = bpf_get_smp_processor_id();
5: (b7) r0 = -1280774092
6: (bf) r0 = &(void __percpu *)(r0)
7: (61) r0 = *(u32 *)(r0 +0)
; bpf_printk("cpu: %u, cnt: %u\n", cpu, cnt);
8: (61) r4 = *(u32 *)(r6 +0)
9: (18) r1 = map[id:24][0]+0
11: (b7) r2 = 18
12: (bf) r3 = r0
13: (85) call bpf_trace_printk#-129408
; return XDP_PASS;
14: (b7) r0 = 2
15: (95) exit
When the section of cnt is ".percpu", bpftool d x n xdp_prog:
int xdp_prog(struct xdp_md * ctx):
; cnt++;
0: (18) r6 = map[id:28][0]+0
2: (bf) r6 = &(void __percpu *)(r6)
3: (61) r1 = *(u32 *)(r6 +0)
; cnt++;
4: (07) r1 += 1
; cnt++;
5: (63) *(u32 *)(r6 +0) = r1
; __u32 cpu = bpf_get_smp_processor_id();
6: (b7) r0 = -1280774092
7: (bf) r0 = &(void __percpu *)(r0)
8: (61) r0 = *(u32 *)(r0 +0)
; bpf_printk("cpu: %u, cnt: %u\n", cpu, cnt);
9: (61) r4 = *(u32 *)(r6 +0)
10: (18) r1 = map[id:30][0]+0
12: (b7) r2 = 18
13: (bf) r3 = r0
14: (85) call bpf_trace_printk#-129408
; return XDP_PASS;
15: (b7) r0 = 2
16: (95) exit
The difference between these xlated insns is "r6 = &(void __percpu *)(r6)". This insn is for ".percpu", not for ".data".
Quoted 17 lines
Quoted 17 lines
Quoted 3 lines
Ah, let me dive deeper.
As for the above changes, let me explain them using diff snippet.
(quoted diff) +2
-0@@ -5808,6 +5808,8 @@ int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val, u64 addr; int err; + if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) + return -EINVAL; err = map->ops->map_direct_value_addr(map, &addr, off); if (err)return err;
Quoted 6 lines
Quoted 2 lines
It is to guard
percpu_arraymap againstconst_reg_xfer(). Instead of updatingconst_reg_xfer(), better to updatebpf_map_direct_read(). WDYT?yeah and move
map_type!=BPF_MAP_TYPE_INSN_ARRAYcheck intobpf_map_direct_read()as well.
To cleanupconst_reg_xfer()a bit.
Before sending the next revision, just confirm the change:
1. Move "map->map_type == BPF_MAP_TYPE_INSN_ARRAY" from const_reg_xfer() to bpf_map_direct_read().
2. Keep "map->map_type != BPF_MAP_TYPE_INSN_ARRAY" in check_mem_access(), because we should not propagate the error from bpf_map_direct_read() for insn_array and percpu_array.
Thanks,
Leon
---
kernel/bpf/const_fold.c +0 -1
@@ -174,7 +181,6 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info *u64 val =0 ;if (!bpf_map_is_rdonly (map) || !map->ops->map_direct_value_addr || - map->map_type ==BPF_MAP_TYPE_INSN_ARRAY || off <0 || off + size > map->value_size ||bpf_map_direct_read (map, off, size, &val, is_ldsx)) { *dst = unknown;
kernel/bpf/verifier.c +4 -1
@@ -5816,6 +5816,8 @@ int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val,u64 addr;int err; +if (map->map_type ==BPF_MAP_TYPE_INSN_ARRAY || map->map_type ==BPF_MAP_TYPE_PERCPU_ARRAY ) +return -EINVAL ; err = map->ops->map_direct_value_addr (map, &addr, off);if (err)return err;
@@ -6370,7 +6372,8 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regnif (tnum_is_const (reg->var_off) &&bpf_map_is_rdonly (map) && map->ops->map_direct_value_addr && - map->map_type !=BPF_MAP_TYPE_INSN_ARRAY ) { + map->map_type !=BPF_MAP_TYPE_INSN_ARRAY && + map->map_type !=BPF_MAP_TYPE_PERCPU_ARRAY ) {int map_off = off + reg->var_off.value;u64 val =0 ;
Quoted 5 lines · Leon Hwang
int
xdp_prog(struct xdp_md * ctx):; cnt++; 0: (18) r6 = map[id:28][0]+0 2: (bf) r6 = &(void __percpu *)(r6)
well. that insn was inserted by the verifier and it shows up in xlated. That was expected.
The point about 'bogus xlated' was about offset translation. map_direct_value_meta() should recover proper insns[i + 1].imm = off; In your example it's zero, so not an interesting test.
Quoted 62 lines
3: (61) r1 = *(u32 *)(r6 +0) ; cnt++; 4: (07) r1 += 1 ; cnt++; 5: (63) *(u32 *)(r6 +0) = r1 ; __u32 cpu = bpf_get_smp_processor_id(); 6: (b7) r0 = -1280774092 7: (bf) r0 = &(void __percpu *)(r0) 8: (61) r0 = *(u32 *)(r0 +0) ; bpf_printk("cpu: %u, cnt: %u\n", cpu, cnt); 9: (61) r4 = *(u32 *)(r6 +0)10: (18) r1 = map[id:30][0]+0
12: (b7) r2 = 18
13: (bf) r3 = r0
14: (85) call bpf_trace_printk#-129408
; returnXDP_PASS;
15: (b7) r0 = 2
16: (95) exitThe difference between these xlated insns is "r6 = &(void
__percpu*)(r6)". This insn is for ".percpu", not for ".data".Quoted 17 lines
Quoted 17 lines
Quoted 3 lines
Ah, let me dive deeper.
As for the above changes, let me explain them using diff snippet.
kernel/bpf/verifier.c +2
-0@@ -5808,6 +5808,8 @@ int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val, u64 addr;int err; +if (map->map_type ==BPF_MAP_TYPE_PERCPU_ARRAY ) +return -EINVAL ; err = map->ops->map_direct_value_addr (map, &addr, off);if (err)return err;Quoted 6 lines
Quoted 2 lines
It is to guard
percpu_arraymap againstconst_reg_xfer(). Instead of updatingconst_reg_xfer(), better to updatebpf_map_direct_read(). WDYT?yeah and move
map_type!=BPF_MAP_TYPE_INSN_ARRAYcheck intobpf_map_direct_read()as well.
To cleanupconst_reg_xfer()a bit.Before sending the next revision, just confirm the change:
1. Move "
map->map_type==BPF_MAP_TYPE_INSN_ARRAY" fromconst_reg_xfer()tobpf_map_direct_read().
2. Keep "map->map_type!=BPF_MAP_TYPE_INSN_ARRAY" incheck_mem_access(), because we should not propagate the error frombpf_map_direct_read()forinsn_arrayandpercpu_array.Thanks,
Leon---
kernel/bpf/const_fold.c static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info * · line 177
Quoted 4 lines
kernel/bpf/const_fold.c +0
-1@@ -174,7 +181,6 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info *u64 val =0 ; if (!bpf_map_is_rdonly(map) || !map->ops->map_direct_value_addr || - map->map_type == BPF_MAP_TYPE_INSN_ARRAY ||
yes
kernel/bpf/verifier.c int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val, · line 5820
Quoted 12 lines
kernel/bpf/const_fold.c +0
-0off <0 || off + size > map->value_size || bpf_map_direct_read(map, off, size, &val, is_ldsx)) { *dst = unknown;kernel/bpf/verifier.c +2
-0@@ -5816,6 +5816,8 @@ int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val,u64 addr;int err; +if (map->map_type ==BPF_MAP_TYPE_INSN_ARRAY || map->map_type ==BPF_MAP_TYPE_PERCPU_ARRAY ) +return -EINVAL ;
yes
kernel/bpf/verifier.c static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · line 6376
Quoted 10 lines
kernel/bpf/verifier.c +2
-1err = map->ops->map_direct_value_addr (map, &addr, off);if (err)return err;@@ -6370,7 +6372,8 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regnif (tnum_is_const (reg->var_off) &&bpf_map_is_rdonly (map) && map->ops->map_direct_value_addr && - map->map_type !=BPF_MAP_TYPE_INSN_ARRAY ) { + map->map_type !=BPF_MAP_TYPE_INSN_ARRAY && + map->map_type !=BPF_MAP_TYPE_PERCPU_ARRAY ) {
why add BPF_MAP_TYPE_PERCPU_ARRAY here?
Quoted 2 lines
kernel/bpf/verifier.c +0
-0int map_off = off + reg->var_off.value;u64 val =0 ;
Quoted 7 lines · Alexei Starovoitov
Quoted 5 lines · Leon Hwang
int
xdp_prog(struct xdp_md * ctx):
; cnt++;
0: (18) r6 = map[id:28][0]+0
2: (bf) r6 = &(void__percpu*)(r6)
Quoted 6 lines
well. that insn was inserted by the verifier and it shows up in xlated. That was expected.
The point about 'bogus xlated' was about offset translation.map_direct_value_meta()should recover proper insns[i + 1].imm = off; In your example it's zero, so not an interesting test.
The original insns[i + 1] wasn't broken. See the key change of this patch.
+ /*
+ * Reuse the original ld_imm64 insn, and add one
+ * mov64_percpu_reg insn.
+ */
+
+ insn_buf[0] = insn[1];
+ insn_buf[1] = BPF_MOV64_PERCPU_REG(insn->dst_reg, insn->dst_reg);
+ cnt = 2;
+
+ i++;
+ new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, cnt);
+ if (!new_prog)
+ return -ENOMEM;
So, it didn't touch the original ld_imm64 insn. That said, no bogus xlated about offset translation.
Quoted 1 line
[...]
Quoted 10 lines
kernel/bpf/verifier.c static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · line 6376
Quoted 7 lines
(quoted diff) +0
-0@@ -6370,7 +6372,8 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regnif (
tnum_is_const(reg->var_off)&&bpf_map_is_rdonly(map)&&map->ops->map_direct_value_addr&&kernel/bpf/verifier.c +2
-1- map->map_type != BPF_MAP_TYPE_INSN_ARRAY) { + map->map_type != BPF_MAP_TYPE_INSN_ARRAY && + map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) {why add
BPF_MAP_TYPE_PERCPU_ARRAYhere?
After looking into bpf_map_is_rdonly(), yes, no need to add check for percpu_array here.
Thanks,
Leon
Quoted 2 lines
Quoted 2 lines
int
map_off= off +reg->var_off.value;
u64 val = 0;
Quoted 16 lines · Leon Hwang
Quoted 7 lines · Alexei Starovoitov
Quoted 5 lines · Leon Hwang
int
xdp_prog(struct xdp_md * ctx):
; cnt++;
0: (18) r6 = map[id:28][0]+0
2: (bf) r6 = &(void__percpu*)(r6)Quoted 6 lines
well. that insn was inserted by the verifier and it shows up in xlated. That was expected.
The point about 'bogus xlated' was about offset translation.map_direct_value_meta()should recover proper insns[i + 1].imm = off; In your example it's zero, so not an interesting test.
Quoted 2 lines
The original insns[i + 1] wasn't broken. See the key change of this patch.
You're missing the point insns[i + 1].imm = off is how off returned from map_direct_value_meta() is used in updating _this_ ldimm64 insn.
Quoted 17 lines · Alexei Starovoitov
Quoted 16 lines · Leon Hwang
Quoted 13 lines · Alexei Starovoitov
Quoted 5 lines · Leon Hwang
int
xdp_prog(struct xdp_md * ctx):
; cnt++;
0: (18) r6 = map[id:28][0]+0
2: (bf) r6 = &(void__percpu*)(r6)well. that insn was inserted by the verifier and it shows up in xlated. That was expected.
The point about 'bogus xlated' was about offset translation.map_direct_value_meta()should recover proper insns[i + 1].imm = off; In your example it's zero, so not an interesting test.
Quoted 6 lines
Quoted 2 lines
The original insns[i + 1] wasn't broken. See the key change of this patch.
You're missing the point insns[i + 1].imm = off is how off returned from
map_direct_value_meta()is used in updating _this_ ldimm64 insn.
After looking into bpf_insn_prepare_dump(), agreed with "not an interesting test".
Then, updated the selftest with non-zero offset. And updated disasm_helpers.c::disasm_insn() to dump such ldimm64 insn as map info by the way of bpftool/xlated_dumper.c::print_imm().
+static volatile const char fmt2[] SEC(".percpu.fmt") = "data %d\n"; +
+SEC("?kprobe")
+__success
+__xlated("r{{[0-9]+}} = map[id:{{[0-9]+}}][0]+9") +__xlated("r{{[0-9]+}} = &(void __percpu *)(r{{[0-9]+}})") +__xlated("r{{[0-9]+}} = *(u8 *)(r{{[0-9]+}} +4)")
+int verifier_percpu_read(void *ctx)
+{
+ char c = fmt2[4];
+
+ return c == ' ';
+}
It would verify the following xlated insns.
XLATED:
=============
0: r1 = map[id:7][0]+9
2: r1 = &(void __percpu *)(r1)
3: r1 = *(u8 *)(r1 +4)
4: w0 = 1
5: if w1 == 0x20 goto pc+1
6: w0 = 0
7: exit
The selftest looks better now.
Thanks,
Leon
Quoted 33 lines · Leon Hwang
Quoted 17 lines · Alexei Starovoitov
Quoted 16 lines · Leon Hwang
Quoted 13 lines · Alexei Starovoitov
Quoted 5 lines · Leon Hwang
int
xdp_prog(struct xdp_md * ctx):
; cnt++;
0: (18) r6 = map[id:28][0]+0
2: (bf) r6 = &(void__percpu*)(r6)well. that insn was inserted by the verifier and it shows up in xlated. That was expected.
The point about 'bogus xlated' was about offset translation.map_direct_value_meta()should recover proper insns[i + 1].imm = off; In your example it's zero, so not an interesting test.Quoted 6 lines
Quoted 2 lines
The original insns[i + 1] wasn't broken. See the key change of this patch.
You're missing the point insns[i + 1].imm = off is how off returned from
map_direct_value_meta()is used in updating _this_ ldimm64 insn.After looking into
bpf_insn_prepare_dump(), agreed with "not an interesting test".Then, updated the selftest with non-zero offset. And updated
disasm_helpers.c::disasm_insn()to dump such ldimm64 insn as map info by the way of bpftool/xlated_dumper.c::print_imm().
Review
Quoted 7 lines
(quoted diff) +7
-0+static volatile const char fmt2[] SEC(".percpu.fmt") = "data %d\n"; + +SEC("?kprobe") +__success +__xlated("r{{[0-9]+}} = map[id:{{[0-9]+}}][0]+9") +__xlated("r{{[0-9]+}} = &(void __percpu *)(r{{[0-9]+}})") +__xlated("r{{[0-9]+}} = *(u8 *)(r{{[0-9]+}} +4)")
This is still pointless test.
Quoted 7 lines
(quoted diff) +6
-0+int verifier_percpu_read(void *ctx) +{ + char c = fmt2[4]; + + return c == ' '; +}
Quoted 5 lines
It would verify the following xlated insns.
XLATED:
=============
0: r1 = map[id:7][0]+9
what is 9 ? was it there in the incoming insn?
The whole point that xlated needs to translate it back. Start from scratch and think what map_direct_value_meta() suppose to do.
Quoted 1 line · Alexei Starovoitov
On Tue, Apr 21, 2026 at 7:24 AM Leon Hwang <[email protected]> wrote:
[...]
Quoted 6 lines
Quoted 3 lines
XLATED:
=============
0: r1 = map[id:7][0]+9what is 9 ? was it there in the incoming insn?
Right, the literal 9 is a mystery here.
Quoted 3 lines
The whole point that xlated needs to translate it back. Start from scratch and think what
map_direct_value_meta()suppose to do.
Thanks for pointing this out.
map_direct_value_addr() encodes the offset as part of the ldimm64 immediate (imm = base + off), and map_direct_value_meta() is supposed to recover the same off from it.
So, the test should verify that the off of xlated ldimm64 insn matches the off encoded in the ELF ldimm64 insn.
Thanks,
Leon