Skip to contentSkip to conversation

bpf: Introduce global percpu data

PATCH Superseded Leon Hwang 4 participants 33 messages Apr 14 – Apr 22

Subsystems (6)
Kselftest BPF BPF selftests BPF core +2 more subsystems

v4 of 12 Superseded Latest is v12 →

lore
Share & tools
lore

Leon Hwang

author
#0 lore raw

v4 cover bpf: Introduce global percpu data

This patch set introduces global percpu data, similar to commit 6316f78306c1 ("Merge branch 'support-global-data'"), to reduce restrictions in C for BPF programs.

With this enhancement, it becomes possible to define and use global percpu variables, like the DEFINE_PER_CPU() macro in the kernel include/linux/percpu-defs.h.

The section name for global peurcpu data is ".percpu". Even though, a one-byte percpu variable (e.g., char run SEC(".percpu") = 0;) can trigger a crash with Clang 17 [1], users are expected to use such small variables as global percpu data with newer Clang versions, which don't have the issue.

The idea stems from the bpfsnoop [2], which itself was inspired by retsnoop [3]. During testing of bpfsnoop on the v6.6 kernel, two LBR (Last Branch Record) entries were observed related to the bpf_get_smp_processor_id() helper.

Since commit 1ae6921009e5 ("bpf: inline bpf_get_smp_processor_id() helper"), the bpf_get_smp_processor_id() helper has been inlined on x86_64, reducing the overhead and consequently minimizing these two LBR records.

However, the introduction of global percpu data offers a more robust solution. By leveraging the percpu_array map and percpu instruction, global percpu data can be implemented intrinsically.

This feature also facilitates sharing percpu information between tail callers and callees or between freplace callers and callees through a shared global percpu variable. Previously, this was achieved using a 1-entry percpu_array map, which this patch set aims to improve upon.

Links:
[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://github.com/bpfsnoop/bpfsnoop
[3] https://github.com/anakryiko/retsnoop

Changes:
v3 -> v4:
* Drop duplicate blank lines in verifier.
* Add percpu data feature probe in libbpf.
* Update percpu_array map using BPF_F_ALL_CPUS flag for lskel, if no cpu flag is set.
* Add two tests to verify verifier log.
* Add a test to verify mov64_percpu_reg instruction.
* Add a test to verify bpf_iter for percpu data map.
* Update percpu_array map using BPF_F_ALL_CPUS flag in libbpf (per Alexei and Andrii).
* Address comments from Andrii:
* Use .percpu as section identifier.
* Use bpf_jit_supports_percpu_insn() instead of CONFIG_SMP.
* Drop bpf_map__is_internal_percpu() API.
* Drop unnecessary __aligned(8) in libbpf, verified by selftest.
* Make mmap data read-only after loading prog.
v3: https://lore.kernel.org/bpf/[email protected]/

v2 -> v3:
* Use ".data..percpu" as PERCPU_DATA_SEC.
* Address comment from Alexei:
* Add u8, array of ints and struct { .. } vars to selftest.
v2: https://lore.kernel.org/bpf/[email protected]/

v1 -> v2:
* Address comments from Andrii:
* Use LIBBPF_MAP_PERCPU and SEC_PERCPU.
* Reuse mmaped of libbpf's struct bpf_map for .percpu map data.
* Set .percpu struct pointer to NULL after loading skeleton.
* Make sure value size of .percpu map is __aligned(8).
* Use raw_tp and opts.cpu to test global percpu variables on all CPUs.
* Address comments from Alexei:
* Test non-zero offset of global percpu variable.
* Test case about BPF_PSEUDO_MAP_IDX_VALUE.
v1: https://lore.kernel.org/bpf/[email protected]/

rfc -> v1:
* Address comments from Andrii:
* Keep one image of global percpu variable for all CPUs.
* Reject non-ARRAY map in bpf_map_direct_read(), check_reg_const_str(), and check_bpf_snprintf_call() in verifier.
* Split out libbpf changes from kernel-side changes.
* Use ".percpu" as PERCPU_DATA_SEC.
* Use enum libbpf_map_type to distinguish BSS, DATA, RODATA and PERCPU_DATA.
* Avoid using errno for checking err from libbpf_num_possible_cpus().
* Use "map '%s': " prefix for error message.
rfc: https://lore.kernel.org/bpf/[email protected]/

Leon Hwang (8):
bpf: Drop duplicate blank lines in verifier
bpf: Introduce global percpu data
libbpf: Probe percpu data feature
libbpf: Add support for global percpu data
bpf: Update per-CPU maps using BPF_F_ALL_CPUS flag
bpftool: Generate skeleton for global percpu data selftests/bpf: Add tests to verify global percpu data selftests/bpf: Add a test to verify bpf_iter for global percpu data

 kernel/bpf/arraymap.c                         |  41 ++-
 kernel/bpf/fixups.c                           |  31 ++
 kernel/bpf/syscall.c                          |  15 +
 kernel/bpf/verifier.c                         |  27 +-
 tools/bpf/bpftool/gen.c                       |  24 +-
 tools/lib/bpf/bpf_gen_internal.h              |   3 +-
 tools/lib/bpf/features.c                      |  35 +++
 tools/lib/bpf/gen_loader.c                    |   3 +-
 tools/lib/bpf/libbpf.c                        |  67 ++++-
 tools/lib/bpf/libbpf_internal.h               |   2 +
 tools/testing/selftests/bpf/Makefile          |   2 +-
 .../bpf/prog_tests/global_data_init.c         | 276 ++++++++++++++++++
 .../bpf/progs/test_global_percpu_data.c       | 102 +++++++
 13 files changed, 582 insertions(+), 46 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/progs/test_global_percpu_data.c

--
2.53.0

author
#1 lore raw

PATCH v4 1/8 bpf: Drop duplicate blank lines in verifier

There are many adjacent blank lines in the verifier that have accumulated over time.

Drop them for cleanup.

No functional changes intended.

---
 kernel/bpf/verifier.c | 18 ------------------
 1 file changed, 18 deletions(-)
kernel/bpf/verifier.c +0 -18
index 9e4980128151..1757c5720503 100644
@@ -513,7 +513,6 @@ static bool helper_multiple_ref_obj_use(enum bpf_func_id func_id,
 	return ref_obj_uses > 1;
 }
 
-
 static bool is_spi_bounds_valid(struct bpf_func_state *state, int spi, int nr_slots)
 {
        int allocated_slots = state->allocated_stack / BPF_REG_SIZE;
@@ -619,7 +618,6 @@ static void __mark_dynptr_reg(struct bpf_reg_state *reg,
 			      enum bpf_dynptr_type type,
 			      bool first_slot, int dynptr_id);
 
-
 static void mark_dynptr_stack_regs(struct bpf_verifier_env *env,
 				   struct bpf_reg_state *sreg1,
 				   struct bpf_reg_state *sreg2,
@@ -1655,7 +1653,6 @@ static bool same_callsites(struct bpf_verifier_state *a, struct bpf_verifier_sta
 	return true;
 }
 
-
 void bpf_free_backedges(struct bpf_scc_visit *visit)
 {
 	struct bpf_scc_backedge *backedge, *next;
@@ -2625,7 +2622,6 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env,
 	return &elem->st;
 }
 
-
 static int cmp_subprogs(const void *a, const void *b)
 {
 	return ((struct bpf_subprog_info *)a)->start -
@@ -3679,7 +3675,6 @@ static bool is_spillable_regtype(enum bpf_reg_type type)
 	}
 }
 
-
 /* check if register is a constant scalar value */
 static bool is_reg_const(struct bpf_reg_state *reg, bool subreg32)
 {
@@ -4347,7 +4342,6 @@ static int check_stack_read(struct bpf_verifier_env *env,
 	return err;
 }
 
-
 /* check_stack_write dispatches to check_stack_write_fixed_off or
  * check_stack_write_var_off.
  *
@@ -5033,7 +5027,6 @@ static int check_sock_access(struct bpf_verifier_env *env, int insn_idx,
 		valid = false;
 	}
 
-
 	if (valid) {
 		env->insn_aux_data[insn_idx].ctx_field_size =
 			info.ctx_field_size;
@@ -6836,7 +6829,6 @@ static int check_stack_range_initialized(
 	if (err)
 		return err;
 
-
 	if (tnum_is_const(reg->var_off)) {
 		min_off = max_off = reg->var_off.value + off;
 	} else {
@@ -7541,7 +7533,6 @@ static bool is_iter_new_kfunc(struct bpf_kfunc_call_arg_meta *meta)
 	return meta->kfunc_flags & KF_ITER_NEW;
 }
 
-
 static bool is_iter_destroy_kfunc(struct bpf_kfunc_call_arg_meta *meta)
 {
 	return meta->kfunc_flags & KF_ITER_DESTROY;
@@ -10459,7 +10450,6 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
 		if (!reg)
 			return -EFAULT;
 
-
 		if (meta.dynptr_id) {
 			verifier_bug(env, "meta.dynptr_id already set");
 			return -EFAULT;
@@ -10798,7 +10788,6 @@ static bool is_kfunc_release(struct bpf_kfunc_call_arg_meta *meta)
 	return meta->kfunc_flags & KF_RELEASE;
 }
 
-
 static bool is_kfunc_destructive(struct bpf_kfunc_call_arg_meta *meta)
 {
 	return meta->kfunc_flags & KF_DESTRUCTIVE;
@@ -11571,7 +11560,6 @@ static int process_irq_flag(struct bpf_verifier_env *env, int regno,
 	return 0;
 }
 
-
 static int ref_set_non_owning(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
 {
 	struct btf_record *rec = reg_btf_record(reg);
@@ -16715,7 +16703,6 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn)
 	return 0;
 }
 
-
 static bool return_retval_range(struct bpf_verifier_env *env, struct bpf_retval_range *range)
 {
 	enum bpf_prog_type prog_type = resolve_prog_type(env->prog);
@@ -18587,8 +18574,6 @@ static void release_insn_arrays(struct bpf_verifier_env *env)
 		bpf_insn_array_release(env->insn_array_maps[i]);
 }
 
-
-
 /* The verifier does more data flow analysis than llvm and will not
  * explore branches that are dead at run time. Malicious programs can
  * have dead code too. Therefore replace all dead at-run-time code
@@ -18616,8 +18601,6 @@ static void sanitize_dead_code(struct bpf_verifier_env *env)
 	}
 }
 
-
-
 static void free_states(struct bpf_verifier_env *env)
 {
 	struct bpf_verifier_state_list *sl;
@@ -18880,7 +18863,6 @@ static int do_check_main(struct bpf_verifier_env *env)
 	return ret;
 }
 
-
 static void print_verification_stats(struct bpf_verifier_env *env)
 {
 	int i;
Signature · 1 line
2.53.0
author
#2 lore raw

PATCH v4 2/8 bpf: Introduce global percpu data

In reply to Leon Hwang

Introduce global percpu data, inspired by the commit 6316f78306c1 ("Merge branch 'support-global-data'"). It enables the definition of global percpu variables in BPF, similar to the include/linux/percpu-defs.h::DEFINE_PER_CPU() macro.

For example, in BPF, it is able to define a global percpu variable like:

int data SEC(".percpu");

With this patch, tools like retsnoop [1] and bpfsnoop [2] can simplify their BPF code for handling LBRs. The code can be updated from

static struct perf_branch_entry lbrs[1][MAX_LBR_ENTRIES] SEC(".data.lbrs");

to

static struct perf_branch_entry lbrs[MAX_LBR_ENTRIES] SEC(".percpu.lbrs");

This eliminates the need to retrieve the CPU ID using the bpf_get_smp_processor_id() helper.

Additionally, by reusing global percpu data map, sharing information between tail callers and callees or freplace callers and callees becomes simpler compared to reusing percpu_array maps.

Links:
[1] https://github.com/anakryiko/retsnoop
[2] https://github.com/bpfsnoop/bpfsnoop

---
 kernel/bpf/arraymap.c | 41 +++++++++++++++++++++++++++++++++++++++--
 kernel/bpf/fixups.c   | 31 +++++++++++++++++++++++++++++++
 kernel/bpf/verifier.c |  9 ++++++++-
 3 files changed, 78 insertions(+), 3 deletions(-)
kernel/bpf/arraymap.c +39 -2
index 5e25e0353509..b85ac59d808c 100644
@@ -261,6 +261,40 @@ static void *percpu_array_map_lookup_elem(struct bpf_map *map, void *key)
 	return this_cpu_ptr(array->pptrs[index & array->index_mask]);
 }
 
+static int percpu_array_map_direct_value_addr(const struct bpf_map *map,
+					      u64 *imm, u32 off)
+{
+	struct bpf_array *array = container_of(map, struct bpf_array, map);
+
+	if (map->max_entries != 1)
+		return -EOPNOTSUPP;
+	if (off >= map->value_size)
+		return -EINVAL;
+	if (!bpf_jit_supports_percpu_insn())
+		return -EOPNOTSUPP;
+
+	*imm = (u64) array->pptrs[0];
+	return 0;
+}
+
+static int percpu_array_map_direct_value_meta(const struct bpf_map *map,
+					      u64 imm, u32 *off)
+{
+	struct bpf_array *array = container_of(map, struct bpf_array, map);
+	u64 base = (u64) array->pptrs[0];
+	u64 range = array->elem_size;
+
+	if (map->max_entries != 1)
+		return -EOPNOTSUPP;
+	if (imm < base || imm >= base + range)
+		return -ENOENT;
+	if (!bpf_jit_supports_percpu_insn())
+		return -EOPNOTSUPP;
+
+	*off = imm - base;
+	return 0;
+}
+
 /* emit BPF instructions equivalent to C code of percpu_array_map_lookup_elem() */
 static int percpu_array_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf)
 {
@@ -553,9 +587,10 @@ static int array_map_check_btf(struct bpf_map *map,
 			       const struct btf_type *key_type,
 			       const struct btf_type *value_type)
 {
-	/* One exception for keyless BTF: .bss/.data/.rodata map */
+	/* One exception for keyless BTF: .bss/.data/.rodata/.percpu map */
 	if (btf_type_is_void(key_type)) {
-		if (map->map_type != BPF_MAP_TYPE_ARRAY ||
+		if ((map->map_type != BPF_MAP_TYPE_ARRAY &&
+		     map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) ||
 		    map->max_entries != 1)
 			return -EINVAL;
 
@@ -834,6 +869,8 @@ const struct bpf_map_ops percpu_array_map_ops = {
 	.map_get_next_key = bpf_array_get_next_key,
 	.map_lookup_elem = percpu_array_map_lookup_elem,
 	.map_gen_lookup = percpu_array_map_gen_lookup,
+	.map_direct_value_addr = percpu_array_map_direct_value_addr,
+	.map_direct_value_meta = percpu_array_map_direct_value_meta,
 	.map_update_elem = array_map_update_elem,
 	.map_delete_elem = array_map_delete_elem,
 	.map_lookup_percpu_elem = percpu_array_map_lookup_percpu_elem,
kernel/bpf/fixups.c +31 -0
index 67c9b28767e1..5f16fd4e0461 100644
@@ -1682,6 +1682,37 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
 			goto next_insn;
 		}
 
+		if (bpf_jit_supports_percpu_insn() &&
+		    insn->code == (BPF_LD | BPF_IMM | BPF_DW) &&
+		    (insn->src_reg == BPF_PSEUDO_MAP_VALUE ||
+		     insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE)) {
+			struct bpf_map *map;
+
+			aux = &env->insn_aux_data[i + delta];
+			map = env->used_maps[aux->map_index];
+			if (map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY)
+				goto next_insn;
+
+			/*
+			 * Reuse the original ld_imm64 insn, and add one
+			 * mov64_percpu_reg insn.
+			 */
+
+			insn_buf[0] = insn[1];
+			insn_buf[1] = BPF_MOV64_PERCPU_REG(insn->dst_reg, insn->dst_reg);
+			cnt = 2;
+
+			i++;
+			new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, cnt);
+			if (!new_prog)
+				return -ENOMEM;
+
+			delta    += cnt - 1;
+			env->prog = prog = new_prog;
+			insn      = new_prog->insnsi + i + delta;
+			goto next_insn;
+		}
+
 		if (insn->code != (BPF_JMP | BPF_CALL))
 			goto next_insn;
 		if (insn->src_reg == BPF_PSEUDO_CALL)
kernel/bpf/verifier.c +8 -1
index 1757c5720503..8ec24d63a2c4 100644
@@ -6362,7 +6362,8 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn
 			if (tnum_is_const(reg->var_off) &&
 			    bpf_map_is_rdonly(map) &&
 			    map->ops->map_direct_value_addr &&
-			    map->map_type != BPF_MAP_TYPE_INSN_ARRAY) {
+			    map->map_type != BPF_MAP_TYPE_INSN_ARRAY &&
+			    map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) {
 				int map_off = off + reg->var_off.value;
 				u64 val = 0;
 
@@ -8358,6 +8359,12 @@ static int check_reg_const_str(struct bpf_verifier_env *env,
 		return -EACCES;
 	}
 
+	if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) {
+		verbose(env, "R%d points to percpu_array map which cannot be used as const string\n",
+			regno);
+		return -EACCES;
+	}
+
 	if (!bpf_map_is_rdonly(map)) {
 		verbose(env, "R%d does not point to a readonly map'\n", regno);
 		return -EACCES;
Signature · 1 line
2.53.0
author
#16 lore raw

PATCH v4 3/8 libbpf: Probe percpu data feature

In reply to Leon Hwang

libbpf needs a reliable way to distinguish kernels that can support global percpu data from those that cannot.

Add a dedicated feature probe, so libbpf can make capability decisions early and fail predictably when global percpu data is unavailable.

---
 tools/lib/bpf/features.c        | 35 +++++++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf_internal.h |  2 ++
 2 files changed, 37 insertions(+)
tools/lib/bpf/features.c +35 -0
index 4f19a0d79b0c..4ffdae42ffd7 100644
@@ -615,6 +615,38 @@ static int probe_kern_btf_layout(int token_fd)
 						 (char *)layout, token_fd));
 }
 
+static int probe_kern_percpu_data(int token_fd)
+{
+	struct bpf_insn insns[] = {
+		BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0),
+		BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0),
+		BPF_EXIT_INSN(),
+	};
+	LIBBPF_OPTS(bpf_map_create_opts, map_opts,
+		.token_fd = token_fd,
+		.map_flags = token_fd ? BPF_F_TOKEN_FD : 0,
+	);
+	LIBBPF_OPTS(bpf_prog_load_opts, prog_opts,
+		.token_fd = token_fd,
+		.prog_flags = token_fd ? BPF_F_TOKEN_FD : 0,
+	);
+	int ret, map, insn_cnt = ARRAY_SIZE(insns);
+
+	map = bpf_map_create(BPF_MAP_TYPE_PERCPU_ARRAY, "libbpf_percpu", sizeof(int), 8, 1,
+			     &map_opts);
+	if (map < 0) {
+		pr_warn("Error in %s(): %s. Couldn't create simple percpu_array map.\n",
+			__func__, errstr(map));
+		return map;
+	}
+
+	insns[0].imm = map;
+
+	ret = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, NULL, "GPL", insns, insn_cnt, &prog_opts);
+	close(map);
+	return probe_fd(ret);
+}
+
 typedef int (*feature_probe_fn)(int /* token_fd */);
 
 static struct kern_feature_cache feature_cache;
@@ -699,6 +731,9 @@ static struct kern_feature_desc {
 	[FEAT_BTF_LAYOUT] = {
 		"kernel supports BTF layout", probe_kern_btf_layout,
 	},
+	[FEAT_PERCPU_DATA] = {
+		"kernel supports percpu data", probe_kern_percpu_data,
+	},
 };
 
 bool feat_supported(struct kern_feature_cache *cache, enum kern_feature_id feat_id)
tools/lib/bpf/libbpf_internal.h +2 -0
index 3781c45b46d3..f62da241a53f 100644
@@ -398,6 +398,8 @@ enum kern_feature_id {
 	FEAT_UPROBE_SYSCALL,
 	/* Kernel supports BTF layout information */
 	FEAT_BTF_LAYOUT,
+	/* Kernel supports percpu data */
+	FEAT_PERCPU_DATA,
 	__FEAT_CNT,
 };
 
Signature · 1 line
2.53.0
author
#17 lore raw

PATCH v4 4/8 libbpf: Add support for global percpu data

In reply to Leon Hwang

Add support for global percpu data in libbpf by adding a new ".percpu" section, similar to ".data". It enables efficient handling of percpu global variables in bpf programs.

When generating loader for lightweight skeleton, update the percpu_array map used for global percpu data using BPF_F_ALL_CPUS, in order to update values across all CPUs using one value slot.

Unlike global data, the mmaped data for global percpu data will be marked as read-only after populating the percpu_array map. Thereafter, users can read those initialized percpu data after loading prog. If they want to update the percpu data after loading prog, they have to update the percpu_array map using key=0 instead.

---
 tools/lib/bpf/bpf_gen_internal.h |  3 +-
 tools/lib/bpf/gen_loader.c       |  3 +-
 tools/lib/bpf/libbpf.c           | 67 ++++++++++++++++++++++++++------
 3 files changed, 59 insertions(+), 14 deletions(-)
tools/lib/bpf/bpf_gen_internal.h +2 -1
index 49af4260b8e6..5ea8383805d3 100644
@@ -66,7 +66,8 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 			enum bpf_prog_type prog_type, const char *prog_name,
 			const char *license, struct bpf_insn *insns, size_t insn_cnt,
 			struct bpf_prog_load_opts *load_attr, int prog_idx);
-void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *value, __u32 value_size);
+void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *value, __u32 value_size,
+			      __u64 flags);
 void bpf_gen__map_freeze(struct bpf_gen *gen, int map_idx);
 void bpf_gen__record_attach_target(struct bpf_gen *gen, const char *name, enum bpf_attach_type type);
 void bpf_gen__record_extern(struct bpf_gen *gen, const char *name, bool is_weak,
tools/lib/bpf/gen_loader.c +2 -1
index cd5c2543f54d..3374f2e01ef2 100644
@@ -1158,7 +1158,7 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 }
 
 void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue,
-			      __u32 value_size)
+			      __u32 value_size, __u64 flags)
 {
 	int attr_size = offsetofend(union bpf_attr, flags);
 	int map_update_attr, value, key;
@@ -1166,6 +1166,7 @@ void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue,
 	int zero = 0;
 
 	memset(&attr, 0, attr_size);
+	attr.flags = flags;
 
 	value = add_data(gen, pvalue, value_size);
 	key = add_data(gen, &zero, sizeof(zero));
tools/lib/bpf/libbpf.c +55 -12

178 lines not on this page · raw message

Signature · 1 line
2.53.0
author
#18 lore raw

PATCH v4 5/8 bpf: Update per-CPU maps using BPF_F_ALL_CPUS flag

In reply to Leon Hwang

When updating per-CPU maps via the lightweight skeleton loader, use a single value slot across all CPUs. This avoids two potential issues when updating on an M-CPU kernel with N cached slots (N < M), especially when N is much smaller than M:

1) The update may trigger a page fault when copying data from the last slot, as the read may go beyond the allocated buffer.
2) The update may copy unexpected data from slots [N, M-1].

---
 kernel/bpf/syscall.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
kernel/bpf/syscall.c +15 -0
index b73b25c63073..f0f3785ef57d 100644
@@ -1785,6 +1785,21 @@ static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr)
 		goto err_put;
 	}
 
+	/*
+	 * When updating per-CPU maps via the lightweight skeleton
+	 * loader, use a single value slot across all CPUs. This avoids
+	 * two potential issues when updating on an M-CPU kernel with
+	 * N cached slots (N < M), especially when N is much smaller
+	 * than M:
+	 * 1) The update may trigger a page fault when copying data from
+	 *    the last slot, as the read may go beyond the allocated
+	 *    buffer.
+	 * 2) The update may copy unexpected data from slots [N, M-1].
+	 */
+	if (bpfptr_is_kernel(uattr) && bpf_map_supports_cpu_flags(map->map_type) &&
+	    !(attr->flags & (BPF_F_CPU | BPF_F_ALL_CPUS)))
+		attr->flags |= BPF_F_ALL_CPUS;
+
 	err = bpf_map_check_op_flags(map, attr->flags, ~0);
 	if (err)
 		goto err_put;
Signature · 1 line
2.53.0
author
#24 lore raw

PATCH v4 6/8 bpftool: Generate skeleton for global percpu data

In reply to Leon Hwang

Enhance bpftool to generate skeletons that properly handle global percpu variables. The generated skeleton now includes a dedicated structure for percpu data, allowing users to initialize and access percpu variables more efficiently.

For global percpu variables, the skeleton now includes a nested structure, e.g.:

struct test_global_percpu_data {
	struct bpf_object_skeleton *skeleton;
	struct bpf_object *obj;
	struct {
		struct bpf_map *percpu;
	} maps;
	// ...
	struct test_global_percpu_data__percpu {
		int data;
		char run;
		struct {
			char set;
			int i;
			int nums[7];
		} struct_data;
		int nums[7];
	} *percpu;

	// ...
};

* The "struct test_global_percpu_data__percpu *percpu" points to

    initialized data, which is actually "maps.percpu->mmaped".

* Before loading the skeleton, updating the
"struct test_global_percpu_data__percpu *percpu" modifies the initial value of the corresponding global percpu variables.
* After loading the skeleton, "maps.percpu->mmaped" has been marked as read-only. If users want to update the global percpu variables, they have to update the "maps.percpu" map instead.

---
 tools/bpf/bpftool/gen.c | 24 ++++++++++++++----------
 1 file changed, 14 insertions(+), 10 deletions(-)
tools/bpf/bpftool/gen.c +14 -10
index 2f9e10752e28..ec63eb871443 100644
@@ -92,7 +92,7 @@ static void get_header_guard(char *guard, const char *obj_name, const char *suff
 
 static bool get_map_ident(const struct bpf_map *map, char *buf, size_t buf_sz)
 {
-	static const char *sfxs[] = { ".data", ".rodata", ".bss", ".kconfig" };
+	static const char *sfxs[] = { ".data", ".rodata", ".bss", ".percpu", ".kconfig" };
 	const char *name = bpf_map__name(map);
 	int i, n;
 
@@ -117,7 +117,7 @@ static bool get_map_ident(const struct bpf_map *map, char *buf, size_t buf_sz)
 
 static bool get_datasec_ident(const char *sec_name, char *buf, size_t buf_sz)
 {
-	static const char *pfxs[] = { ".data", ".rodata", ".bss", ".kconfig" };
+	static const char *pfxs[] = { ".data", ".rodata", ".bss", ".percpu", ".kconfig" };
 	int i, n;
 
 	/* recognize hard coded LLVM section name */
@@ -254,6 +254,11 @@ static const struct btf_type *find_type_for_map(struct btf *btf, const char *map
 	return NULL;
 }
 
+static bool bpf_map_is_percpu_data(const struct bpf_map *map)
+{
+	return bpf_map__is_internal(map) && bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY;
+}
+
 static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)
 {
 	size_t tmp_sz;
@@ -263,13 +268,12 @@ static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)
 		return true;
 	}
 
-	if (!bpf_map__is_internal(map) || !(bpf_map__map_flags(map) & BPF_F_MMAPABLE))
-		return false;
-
-	if (!get_map_ident(map, buf, sz))
-		return false;
+	if (bpf_map__is_internal(map) &&
+	    ((bpf_map__map_flags(map) & BPF_F_MMAPABLE) || bpf_map_is_percpu_data(map)) &&
+	    get_map_ident(map, buf, sz))
+		return true;
 
-	return true;
+	return false;
 }
 
 static int codegen_datasecs(struct bpf_object *obj, const char *obj_name)
@@ -669,7 +673,7 @@ static void codegen_destroy(struct bpf_object *obj, const char *obj_name)
 		if (!get_map_ident(map, ident, sizeof(ident)))
 			continue;
 		if (bpf_map__is_internal(map) &&
-		    (bpf_map__map_flags(map) & BPF_F_MMAPABLE))
+		    ((bpf_map__map_flags(map) & BPF_F_MMAPABLE) || bpf_map_is_percpu_data(map)))
 			printf("\tskel_free_map_data(skel->%1$s, skel->maps.%1$s.initial_value, %2$zu);\n",
 			       ident, bpf_map_mmap_sz(map));
 		codegen("\
@@ -847,7 +851,7 @@ static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *h
 	bpf_object__for_each_map(map, obj) {
 		const char *mmap_flags;
 
-		if (!is_mmapable_map(map, ident, sizeof(ident)))
+		if (!is_mmapable_map(map, ident, sizeof(ident)) || bpf_map_is_percpu_data(map))
 			continue;
 
 		if (bpf_map__map_flags(map) & BPF_F_RDONLY_PROG)
Signature · 1 line
2.53.0
author
#27 lore raw

PATCH v4 7/8 selftests/bpf: Add tests to verify global percpu data

In reply to Leon Hwang

If the arch, like s390x, does not support percpu insn, these cases won't test global percpu data by checking -EOPNOTSUPP after loading prog.

The following APIs have been tested for global percpu data:

1. bpf_map__set_initial_value()
2. bpf_map__initial_value()
3. generated percpu struct pointer pointing to internal map's mmaped data
4. bpf_map__lookup_elem() for global percpu data map

At the same time, the case is also tested with 'bpftool gen skeleton -L'.

The verifier log is also tested.

---
 tools/testing/selftests/bpf/Makefile          |   2 +-
 .../bpf/prog_tests/global_data_init.c         | 223 ++++++++++++++++++
 .../bpf/progs/test_global_percpu_data.c       |  66 ++++++
 3 files changed, 290 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/bpf/progs/test_global_percpu_data.c
tools/testing/selftests/bpf/Makefile +1 -1
index f75c4f52c028..39ae3583983b 100644
@@ -504,7 +504,7 @@ LSKELS_SIGNED := fentry_test.c fexit_test.c atomics.c
 
 # Generate both light skeleton and libbpf skeleton for these
 LSKELS_EXTRA := test_ksyms_module.c test_ksyms_weak.c kfunc_call_test.c \
-	kfunc_call_test_subprog.c
+	kfunc_call_test_subprog.c test_global_percpu_data.c
 SKEL_BLACKLIST += $$(LSKELS) $$(LSKELS_SIGNED)
 
 test_static_linked.skel.h-deps := test_static_linked1.bpf.o test_static_linked2.bpf.o
tools/testing/selftests/bpf/prog_tests/global_data_init.c +223 -0

231 lines not on this page · raw message

tools/testing/selftests/bpf/progs/test_global_percpu_data.c new file +66 -0

66 lines not on this page · raw message

Signature · 1 line
2.53.0
author
#30 lore raw

PATCH v4 8/8 selftests/bpf: Add a test to verify bpf_iter for global percpu data

In reply to Leon Hwang

Add a test to verify that it is OK to iter the percpu_array map used for global percpu data.

---
 .../bpf/prog_tests/global_data_init.c         | 53 +++++++++++++++++++
 .../bpf/progs/test_global_percpu_data.c       | 36 +++++++++++++
 2 files changed, 89 insertions(+)
tools/testing/selftests/bpf/prog_tests/global_data_init.c +53 -0
index 0b0384b6515e..9ff09ff1981c 100644
@@ -269,6 +269,57 @@ static void test_global_percpu_data_verifier_failure(void)
 	RUN_TESTS(test_global_percpu_data);
 }
 
+static void test_global_percpu_data_iter(void)
+{
+	DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);
+	struct test_global_percpu_data *skel;
+	union bpf_iter_link_info linfo = {};
+	struct bpf_link *link = NULL;
+	int fd, num_cpus, len, err;
+	char buf[16];
+
+	num_cpus = libbpf_num_possible_cpus();
+	if (!ASSERT_GT(num_cpus, 0, "libbpf_num_possible_cpus"))
+		return;
+
+	skel = test_global_percpu_data__open();
+	if (!ASSERT_OK_PTR(skel, "test_global_percpu_data__open"))
+		return;
+
+	skel->rodata->num_cpus = num_cpus;
+	skel->percpu->struct_data.nums[6] = 0xc0de;
+
+	err = test_global_percpu_data__load(skel);
+	if (err == -EOPNOTSUPP) {
+		test__skip();
+		goto out;
+	}
+	if (!ASSERT_OK(err, "test_global_percpu_data__load"))
+		goto out;
+
+	linfo.map.map_fd = bpf_map__fd(skel->maps.percpu);
+	opts.link_info = &linfo;
+	opts.link_info_len = sizeof(linfo);
+	link = bpf_program__attach_iter(skel->progs.dump_percpu_data, &opts);
+	if (!ASSERT_OK_PTR(link, "bpf_program__attach_iter"))
+		goto out;
+
+	fd = bpf_iter_create(bpf_link__fd(link));
+	if (!ASSERT_GE(fd, 0, "bpf_iter_create"))
+		goto out;
+
+	while ((len = read(fd, buf, sizeof(buf))) > 0)
+		do { } while (0);
+	ASSERT_EQ(len, 0, "read iter");
+	ASSERT_TRUE(skel->bss->run_iter, "run_iter");
+	ASSERT_EQ(skel->bss->percpu_data_sum, 0xc0de * num_cpus, "percpu_data_sum");
+
+	close(fd);
+out:
+	bpf_link__destroy(link);
+	test_global_percpu_data__destroy(skel);
+}
+
 void test_global_percpu_data(void)
 {
 	if (!feat_supported(NULL, FEAT_PERCPU_DATA)) {
@@ -282,4 +333,6 @@ void test_global_percpu_data(void)
 		test_global_percpu_data_lskel();
 	if (test__start_subtest("verifier_failure"))
 		test_global_percpu_data_verifier_failure();
+	if (test__start_subtest("iter"))
+		test_global_percpu_data_iter();
 }
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +36 -0
index 947721c21f30..0ddb820509eb 100644
@@ -63,4 +63,40 @@ int verifier_percpu_read(void *ctx)
 	return c == 'd';
 }
 
+volatile const __u32 num_cpus = 0;
+__u32 percpu_data_sum = 0;
+bool run_iter = false;
+
+SEC("iter/bpf_map_elem")
+__auxiliary
+int dump_percpu_data(struct bpf_iter__bpf_map_elem *ctx)
+{
+	struct {
+		int data;
+		char run;
+		struct {
+			char set;
+			int i;
+			int nums[7];
+		} struct_data;
+		int nums[7];
+	} *pptr = ctx->value;
+	__u32 step;
+	int i;
+
+	if (!pptr)
+		return 0;
+
+	run_iter = true;
+
+	/* percpu array element size is aligned to 8 */
+	step = (sizeof(*pptr) + 7) & ~7;
+
+	for (i = 0; i < num_cpus; i++) {
+		percpu_data_sum += pptr->struct_data.nums[6];
+		pptr = (void *)pptr + step;
+	}
+	return 0;
+}
+
 char _license[] SEC("license") = "GPL";
Signature · 1 line
2.53.0
#3 lore raw
In reply to Leon Hwang
This correctly prevents constant folding of percpu array values in check_mem_access(). However, should the same exclusion also be added to const_reg_xfer() in kernel/bpf/const_fold.c?

kernel/bpf/verifier.c static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · line 6366

Quoted 11 lines
kernel/bpf/verifier.c +2 -1
index 1757c5720503..8ec24d63a2c4 100644
@@ -6362,7 +6362,8 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn
 			if (tnum_is_const(reg->var_off) &&
 			    bpf_map_is_rdonly(map) &&
 			    map->ops->map_direct_value_addr &&
-			    map->map_type != BPF_MAP_TYPE_INSN_ARRAY) {
+			    map->map_type != BPF_MAP_TYPE_INSN_ARRAY &&
+			    map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) {

This correctly prevents constant folding of percpu array values in check_mem_access(). However, should the same exclusion also be added to const_reg_xfer() in kernel/bpf/const_fold.c?

That function has a parallel constant folding path via bpf_map_direct_read() that only excludes BPF_MAP_TYPE_INSN_ARRAY:

kernel/bpf/const_fold.c:const_reg_xfer() {
    ...
    case BPF_LDX:
        ...
        if (!bpf_map_is_rdonly(map) ||
            !map->ops->map_direct_value_addr ||
            map->map_type == BPF_MAP_TYPE_INSN_ARRAY ||
            off < 0 || off + size > map->value_size ||
            bpf_map_direct_read(map, off, size, &val, is_ldsx)) {
                *dst = unknown;
                break;
        }
    ...
}

Since percpu_array_map_ops now has map_direct_value_addr set, bpf_map_direct_read() would dereference the raw percpu base pointer directly, rather than the actual per-cpu address. For a frozen read-only percpu array, the constant-folded value could differ from what the program reads at runtime after the BPF_MOV64_PERCPU_REG fixup converts the address to the correct per-cpu copy.

In the v3 series, bpf_map_direct_read() itself had a guard (map->map_type != BPF_MAP_TYPE_ARRAY), which protected all callers. The v4 moved this to caller-side checks but appears to have missed const_reg_xfer().

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24401974087

author
#4 lore raw
Quoted 41 lines · On 2026/4/14 22:10, [email protected] wrote:

kernel/bpf/verifier.c static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · line 6366

Quoted 11 lines
kernel/bpf/verifier.c +2 -1
index 1757c5720503..8ec24d63a2c4 100644
@@ -6362,7 +6362,8 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn
 			if (tnum_is_const(reg->var_off) &&
 			    bpf_map_is_rdonly(map) &&
 			    map->ops->map_direct_value_addr &&
-			    map->map_type != BPF_MAP_TYPE_INSN_ARRAY) {
+			    map->map_type != BPF_MAP_TYPE_INSN_ARRAY &&
+			    map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) {

This correctly prevents constant folding of percpu array values in check_mem_access(). However, should the same exclusion also be added to const_reg_xfer() in kernel/bpf/const_fold.c?

That function has a parallel constant folding path via bpf_map_direct_read() that only excludes BPF_MAP_TYPE_INSN_ARRAY:

kernel/bpf/const_fold.c:const_reg_xfer() {
    ...
    case BPF_LDX:
        ...
        if (!bpf_map_is_rdonly(map) ||
            !map->ops->map_direct_value_addr ||
            map->map_type == BPF_MAP_TYPE_INSN_ARRAY ||
            off < 0 || off + size > map->value_size ||
            bpf_map_direct_read(map, off, size, &val, is_ldsx)) {
                *dst = unknown;
                break;
        }
    ...
}

Since percpu_array_map_ops now has map_direct_value_addr set, bpf_map_direct_read() would dereference the raw percpu base pointer directly, rather than the actual per-cpu address. For a frozen read-only percpu array, the constant-folded value could differ from what the program reads at runtime after the BPF_MOV64_PERCPU_REG fixup converts the address to the correct per-cpu copy.

Quoted 6 lines

In the v3 series, bpf_map_direct_read() itself had a guard (map->map_type != BPF_MAP_TYPE_ARRAY), which protected all callers. The v4 moved this to caller-side checks but appears to have missed const_reg_xfer().

Correct.

Will add a guard in bpf_map_direct_read() in the next revision:

	if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY)
		return -EINVAL;

Thanks,
Leon

#19 lore raw

Re: [PATCH bpf-next v4 5/8] bpf: Update per-CPU maps using BPF_F_ALL_CPUS flag

In reply to Leon Hwang
Sashiko AI review found 1 potential issue(s): - [High] The patch unconditionally forces the BPF_F_ALL_CPUS flag for any kernel-originated per-CPU map update that omits CPU flags, breaking the…

Sashiko AI review found 1 potential issue(s):
- [High] The patch unconditionally forces the BPF_F_ALL_CPUS flag for any kernel-originated per-CPU map update that omits CPU flags, breaking the bpf_sys_bpf() helper API for BPF_PROG_TYPE_SYSCALL programs. --

commit 357a4fab1d0fc8586bb52bd3aab10e26fd39a2a6
Author: Leon Hwang <[email protected]>

bpf: Update per-CPU maps using BPF_F_ALL_CPUS flag

This commit modifies map_update_elem() to automatically apply the BPF_F_ALL_CPUS flag for kernel-originated updates to per-CPU maps when no CPU flags are specified. This is intended to fix potential out-of-bounds reads when updating maps via the lightweight skeleton loader on systems where the number of cached slots is less than the number of CPUs.

Quoted 12 lines
kernel/bpf/syscall.c +4 -0
index b73b25c630734..f0f3785ef57d0 100644
@@ -1785,6 +1785,21 @@ static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr)
 		goto err_put;
 	}
 
+	/*
+	 * When updating per-CPU maps via the lightweight skeleton
+	 * loader, use a single value slot across all CPUs. This avoids
+	 * two potential issues when updating on an M-CPU kernel with

kernel/bpf/syscall.c static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr) · line 1801

Quoted 10 lines
kernel/bpf/syscall.c +10 -0
+	 * N cached slots (N < M), especially when N is much smaller
+	 * than M:
+	 * 1) The update may trigger a page fault when copying data from
+	 *    the last slot, as the read may go beyond the allocated
+	 *    buffer.
+	 * 2) The update may copy unexpected data from slots [N, M-1].
+	 */
+	if (bpfptr_is_kernel(uattr) && bpf_map_supports_cpu_flags(map->map_type) &&
+	    !(attr->flags & (BPF_F_CPU | BPF_F_ALL_CPUS)))
+		attr->flags |= BPF_F_ALL_CPUS;

Does this unintentionally break the bpf_sys_bpf() helper API for BPF_PROG_TYPE_SYSCALL programs?

Because bpf_sys_bpf() also wraps the pointer with KERNEL_BPFPTR, bpfptr_is_kernel(uattr) will evaluate to true for any eBPF program calling it.

Prior to this patch, a BPF_PROG_TYPE_SYSCALL program could legitimately update a per-CPU map with distinct values for each CPU by providing an array of num_possible_cpus() elements and omitting any CPU flags, which mirrors standard user-space API semantics.

With this change, it appears the kernel silently overrides attr->flags by setting BPF_F_ALL_CPUS. This truncates the internal value_size calculation down to a single element (map->value_size), reads only the first element of the BPF program's provided buffer, and incorrectly replicates it across all CPUs.

Should this workaround for the skeleton loader be constrained to bpftool gen skeleton instead of mutating the core syscall handler?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
#25 lore raw

Re: [PATCH bpf-next v4 6/8] bpftool: Generate skeleton for global percpu data

In reply to Leon Hwang
Sashiko AI review found 1 potential issue(s): - [High] Missing read-only memory protection for `.percpu` initial value buffer in generated light skeletons. --

Sashiko AI review found 1 potential issue(s):
- [High] Missing read-only memory protection for .percpu initial value buffer in generated light skeletons. --

commit e61b5ee6e5a7cee758b644de0f6b4aaea40f020c
Author: Leon Hwang <[email protected]>

bpftool: Generate skeleton for global percpu data

Enhance bpftool to generate skeletons that properly handle global percpu variables. The skeleton includes a nested structure for percpu data, which is marked as read-only after loading so users must update the map directly.

Quoted 3 lines
tools/bpf/bpftool/gen.c +0 -0

[ ... ]

tools/bpf/bpftool/gen.c static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *h · line 857

Quoted 9 lines
tools/bpf/bpftool/gen.c +1 -1
@@ -847,7 +851,7 @@ static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *h
 	bpf_object__for_each_map(map, obj) {
 		const char *mmap_flags;
 
-		if (!is_mmapable_map(map, ident, sizeof(ident)))
+		if (!is_mmapable_map(map, ident, sizeof(ident)) || bpf_map_is_percpu_data(map))
 			continue;
 
 		if (bpf_map__map_flags(map) & BPF_F_RDONLY_PROG)

The commit message notes that after loading the skeleton, maps.percpu->mmaped has been marked as read-only.

By skipping the .percpu map entirely here during light skeleton generation in gen_trace(), does this leave the user-space buffer for .percpu writable?

If a user modifies skel->percpu after the light skeleton is loaded, will it silently succeed without actually updating the kernel per-cpu map? Does the generated light skeleton need an explicit mprotect(..., PROT_READ) for the .percpu initial value buffer to enforce the read-only constraint?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
#28 lore raw

Re: [PATCH bpf-next v4 7/8] selftests/bpf: Add tests to verify global percpu data

In reply to Leon Hwang
Sashiko AI review found 1 potential issue(s): - [Medium] The tests use the raw CPU ID to index into the densely packed `percpu_data` array, which will cause an out-of-bounds read and test failures…

Sashiko AI review found 1 potential issue(s):
- [Medium] The tests use the raw CPU ID to index into the densely packed percpu_data array, which will cause an out-of-bounds read and test failures on systems with sparse CPU masks. --

commit a7c2685b539ebc816cdb05decab1d7a8a4e1328e
Author: Leon Hwang <[email protected]>

selftests/bpf: Add tests to verify global percpu data

This commit adds new selftests to verify the behavior of global percpu data by checking map initial values, struct pointers, and looking up elements from global percpu data maps.

Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +0 -0

[ ... ]

Quoted 2 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +2 -0
+static void test_global_percpu_data_init(void)
+{

[ ... ]

tools/testing/selftests/bpf/prog_tests/global_data_init.c void test_global_data_init(void) · line 177

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +10 -0
+	err = bpf_map__lookup_elem(map, &key, sizeof(key), percpu_data,
+				   elem_sz * num_cpus, 0);
+	if (!ASSERT_OK(err, "bpf_map__lookup_elem"))
+		goto out;
+
+	for (i = 0; i < num_online; i++) {
+		if (!online[i])
+			continue;
+
+		data = (void *)percpu_data + elem_sz * i;

Will this logic cause an out-of-bounds read on systems with sparse CPU masks?

When reading a per-CPU map using bpf_map__lookup_elem(), the kernel packs the per-CPU values densely into the output buffer by iterating over the possible CPU mask.

The test allocates num_cpus (the total count of possible CPUs) elements for the output buffer, but then iterates over the raw CPU IDs using i and uses i as an index into the densely packed array.

On systems with sparse CPU masks (for example, where CPUs 0 and 100 are possible/online), num_cpus will be 2. When the loop reaches i = 100, it appears it will attempt to access percpu_data + elem_sz * 100, reading past the bounds of the allocated memory.

Could we maintain a separate index counter that increments only for possible CPUs to match the kernel's dense packing behavior?

[ ... ]

Quoted 2 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +2 -0
+static void test_global_percpu_data_lskel(void)
+{

[ ... ]

tools/testing/selftests/bpf/prog_tests/global_data_init.c void test_global_data_init(void) · line 252

Quoted 9 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +9 -0
+	err = bpf_map_lookup_elem(map_fd, &key, percpu_data);
+	if (!ASSERT_OK(err, "bpf_map_lookup_elem"))
+		goto out;
+
+	for (i = 0; i < num_online; i++) {
+		if (!online[i])
+			continue;
+
+		data = (void *)percpu_data + elem_sz * i;

Does this loop have the same bounds checking issue as mentioned above?

If i is a raw CPU ID on a system with a sparse CPU mask, this calculation could exceed the bounds of the percpu_data buffer.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
#31 lore raw

Re: [PATCH bpf-next v4 8/8] selftests/bpf: Add a test to verify bpf_iter for global percpu data

In reply to Leon Hwang
Sashiko AI review found 3 potential issue(s): - [Medium] The `dump_percpu_data` program is autoloaded by default, breaking existing subtests on older kernels. - [Medium] Checking `err ==…

Sashiko AI review found 3 potential issue(s):
- [Medium] The dump_percpu_data program is autoloaded by default, breaking existing subtests on older kernels.
- [Medium] Checking err == -EOPNOTSUPP after skeleton load incorrectly handles unsupported kernels.
- [Medium] Casting the global datasec to an anonymous C struct relies on fragile compiler layout assumptions. --

commit 1e00e3e2b318918e67a2c7d2e251132e1b985484
Author: Leon Hwang <[email protected]>

selftests/bpf: Add a test to verify bpf_iter for global percpu data

Add a test to verify that it is OK to iter the percpu_array map used for global percpu data.

Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +0 -0
index 0b0384b6515ed..9ff09ff1981c4 100644

[ ... ]

Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +2 -0
@@ -269,6 +269,57 @@ static void test_global_percpu_data_verifier_failure(void)
+static void test_global_percpu_data_iter(void)
+{

[ ... ]

tools/testing/selftests/bpf/prog_tests/global_data_init.c static void test_global_percpu_data_verifier_failure(void) · line 296

Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +5 -0
+	err = test_global_percpu_data__load(skel);
+	if (err == -EOPNOTSUPP) {
+		test__skip();
+		goto out;
+	}

Will this correctly skip on older kernels? When the kernel verifier rejects an unsupported program type or feature during bpf_prog_load(), it typically returns -EINVAL (or -EACCES), not -EOPNOTSUPP.

Libbpf passes this error code back to the caller, so this condition might evaluate to false, causing the test to fail on the subsequent ASSERT_OK(err) check instead of gracefully skipping.

[ ... ]

Quoted 4 lines
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +0 -0
index 947721c21f30c..0ddb820509ebf 100644

[ ... ]

tools/testing/selftests/bpf/progs/test_global_percpu_data.c int verifier_percpu_read(void *ctx) · line 72

Quoted 3 lines
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +3 -0
+SEC("iter/bpf_map_elem")
+__auxiliary
+int dump_percpu_data(struct bpf_iter__bpf_map_elem *ctx)

Will this break existing subtests on older kernels? The bpf program dump_percpu_data is defined with SEC("iter/bpf_map_elem") without the ? prefix, meaning libbpf will attempt to autoload it whenever test_global_percpu_data__load() is called.

Since the existing init and lskel subtests also load the full skeleton, they might inadvertently attempt to load this new iterator program. Should this use SEC("?iter/bpf_map_elem") to prevent autoloading by default, and only be explicitly autoloaded in the iter subtest?

tools/testing/selftests/bpf/progs/test_global_percpu_data.c int verifier_percpu_read(void *ctx) · line 83

Quoted 11 lines
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +11 -0
+{
+	struct {
+		int data;
+		char run;
+		struct {
+			char set;
+			int i;
+			int nums[7];
+		} struct_data;
+		int nums[7];
+	} *pptr = ctx->value;

Is it safe to cast the global datasec to an anonymous C struct like this?

In the BPF source, these are declared as independent global variables. The layout of these variables in the .percpu ELF section is determined entirely by the compiler. While the current compiler version might emit the uninitialized nums array after the initialized variables (matching the struct layout), this relies on undocumented compiler behavior.

If the compiler alters the layout, or if variables are added or initialized differently in the future, the field offsets will mismatch. This could cause the step calculation to diverge from the map's actual value_size, leading to out-of-bounds reads.

Would it be safer to group these variables into a single explicit struct in the BPF code to guarantee their memory layout?

[ ... ]

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
#5 lore raw
In reply to Leon Hwang
Quoted 48 lines · Leon Hwang
Quoted 41 lines · On 2026/4/14 22:10, [email protected] wrote:

kernel/bpf/verifier.c static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · line 6366

Quoted 11 lines
kernel/bpf/verifier.c +2 -1
index 1757c5720503..8ec24d63a2c4 100644
@@ -6362,7 +6362,8 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn
 			if (tnum_is_const(reg->var_off) &&
 			    bpf_map_is_rdonly(map) &&
 			    map->ops->map_direct_value_addr &&
-			    map->map_type != BPF_MAP_TYPE_INSN_ARRAY) {
+			    map->map_type != BPF_MAP_TYPE_INSN_ARRAY &&
+			    map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) {

This correctly prevents constant folding of percpu array values in check_mem_access(). However, should the same exclusion also be added to const_reg_xfer() in kernel/bpf/const_fold.c?

That function has a parallel constant folding path via bpf_map_direct_read() that only excludes BPF_MAP_TYPE_INSN_ARRAY:

kernel/bpf/const_fold.c:const_reg_xfer() {
    ...
    case BPF_LDX:
        ...
        if (!bpf_map_is_rdonly(map) ||
            !map->ops->map_direct_value_addr ||
            map->map_type == BPF_MAP_TYPE_INSN_ARRAY ||
            off < 0 || off + size > map->value_size ||
            bpf_map_direct_read(map, off, size, &val, is_ldsx)) {
                *dst = unknown;
                break;
        }
    ...
}

Since percpu_array_map_ops now has map_direct_value_addr set, bpf_map_direct_read() would dereference the raw percpu base pointer directly, rather than the actual per-cpu address. For a frozen read-only percpu array, the constant-folded value could differ from what the program reads at runtime after the BPF_MOV64_PERCPU_REG fixup converts the address to the correct per-cpu copy.

Quoted 6 lines

In the v3 series, bpf_map_direct_read() itself had a guard (map->map_type != BPF_MAP_TYPE_ARRAY), which protected all callers. The v4 moved this to caller-side checks but appears to have missed const_reg_xfer().

Quoted 6 lines

Correct.

Will add a guard in bpf_map_direct_read() in the next revision:

	if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY)
		return -EINVAL;

hold on.
map->ops->map_direct_value_addr &&

-			    map->map_type != BPF_MAP_TYPE_INSN_ARRAY) {
+			    map->map_type != BPF_MAP_TYPE_INSN_ARRAY &&
+			    map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) {

map_direct_value_addr() is set, but then immediately disallowed ? Where else it's used?

Even if value_addr is working, then map_direct_value_meta() looks broken.

#21 lore raw

Re: [PATCH bpf-next v4 5/8] bpf: Update per-CPU maps using BPF_F_ALL_CPUS flag

In reply to Leon Hwang
Quoted 26 lines · Leon Hwang

When updating per-CPU maps via the lightweight skeleton loader, use a single value slot across all CPUs. This avoids two potential issues when updating on an M-CPU kernel with N cached slots (N < M), especially when N is much smaller than M:

1) The update may trigger a page fault when copying data from the last slot, as the read may go beyond the allocated buffer.
2) The update may copy unexpected data from slots [N, M-1].

---
 kernel/bpf/syscall.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
kernel/bpf/syscall.c +4 -0
index b73b25c63073..f0f3785ef57d 100644
@@ -1785,6 +1785,21 @@ static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr)
 		goto err_put;
 	}
 
+	/*
+	 * When updating per-CPU maps via the lightweight skeleton
+	 * loader, use a single value slot across all CPUs. This avoids
+	 * two potential issues when updating on an M-CPU kernel with

kernel/bpf/syscall.c static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr) · line 1801

Quoted 10 lines
kernel/bpf/syscall.c +10 -0
+	 * N cached slots (N < M), especially when N is much smaller
+	 * than M:
+	 * 1) The update may trigger a page fault when copying data from
+	 *    the last slot, as the read may go beyond the allocated
+	 *    buffer.
+	 * 2) The update may copy unexpected data from slots [N, M-1].
+	 */
+	if (bpfptr_is_kernel(uattr) && bpf_map_supports_cpu_flags(map->map_type) &&
+	    !(attr->flags & (BPF_F_CPU | BPF_F_ALL_CPUS)))
+		attr->flags |= BPF_F_ALL_CPUS;

This looks like a hack. It's not addressing the actual bug. If there is a bug submit it separately with fixes tag.

author
#6 lore raw
In reply to Alexei Starovoitov
Quoted 2 lines · Alexei Starovoitov
Quoted 1 line · Leon Hwang

On 2026/4/14 22:10, [email protected] wrote:

[...]

Quoted 19 lines
Quoted 6 lines
Quoted 6 lines

In the v3 series, bpf_map_direct_read() itself had a guard (map->map_type != BPF_MAP_TYPE_ARRAY), which protected all callers. The v4 moved this to caller-side checks but appears to have missed const_reg_xfer().

Quoted 6 lines

Correct.

Will add a guard in bpf_map_direct_read() in the next revision:

	if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY)
		return -EINVAL;

hold on.

kernel/bpf/verifier.c +2 -1
 			    map->ops->map_direct_value_addr &&
-			    map->map_type != BPF_MAP_TYPE_INSN_ARRAY) {
+			    map->map_type != BPF_MAP_TYPE_INSN_ARRAY &&
+			    map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) {
Quoted 5 lines

map_direct_value_addr() is set, but then immediately disallowed ? Where else it's used?

Even if value_addr is working, then map_direct_value_meta() looks broken.

Ah, let me dive deeper.

Thanks,
Leon

author
#22 lore raw

Re: [PATCH bpf-next v4 5/8] bpf: Update per-CPU maps using BPF_F_ALL_CPUS flag

In reply to Alexei Starovoitov
Quoted 32 lines · Alexei Starovoitov
Quoted 31 lines · Leon Hwang

When updating per-CPU maps via the lightweight skeleton loader, use a single value slot across all CPUs. This avoids two potential issues when updating on an M-CPU kernel with N cached slots (N < M), especially when N is much smaller than M:

1) The update may trigger a page fault when copying data from the last slot, as the read may go beyond the allocated buffer.
2) The update may copy unexpected data from slots [N, M-1].

---
 kernel/bpf/syscall.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
kernel/bpf/syscall.c +9 -0
index b73b25c63073..f0f3785ef57d 100644
@@ -1785,6 +1785,21 @@ static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr)
 		goto err_put;
 	}
 
+	/*
+	 * When updating per-CPU maps via the lightweight skeleton
+	 * loader, use a single value slot across all CPUs. This avoids
+	 * two potential issues when updating on an M-CPU kernel with
+	 * N cached slots (N < M), especially when N is much smaller
+	 * than M:
+	 * 1) The update may trigger a page fault when copying data from
+	 *    the last slot, as the read may go beyond the allocated
+	 *    buffer.
Quoted 8 lines

kernel/bpf/syscall.c static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr) · line 1801

Quoted 5 lines
kernel/bpf/syscall.c +5 -0
+	 * 2) The update may copy unexpected data from slots [N, M-1].
+	 */
+	if (bpfptr_is_kernel(uattr) && bpf_map_supports_cpu_flags(map->map_type) &&
+	    !(attr->flags & (BPF_F_CPU | BPF_F_ALL_CPUS)))
+		attr->flags |= BPF_F_ALL_CPUS;

This looks like a hack. It's not addressing the actual bug. If there is a bug submit it separately with fixes tag.

Sure, will verify whether it is a bug. If it is, will fix it with separate patch.

Thanks,
Leon

author
#20 lore raw

Re: [PATCH bpf-next v4 5/8] bpf: Update per-CPU maps using BPF_F_ALL_CPUS flag

In reply to [email protected]
Quoted 1 line · On 15/4/26 05:02, [email protected] wrote:

Sashiko AI review found 1 potential issue(s):

[...]

Quoted 26 lines
Quoted 4 lines
kernel/bpf/syscall.c +4 -0
+	/*
+	 * When updating per-CPU maps via the lightweight skeleton
+	 * loader, use a single value slot across all CPUs. This avoids
+	 * two potential issues when updating on an M-CPU kernel with

kernel/bpf/syscall.c static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr) · line 1801

Quoted 10 lines
kernel/bpf/syscall.c +10 -0
+	 * N cached slots (N < M), especially when N is much smaller
+	 * than M:
+	 * 1) The update may trigger a page fault when copying data from
+	 *    the last slot, as the read may go beyond the allocated
+	 *    buffer.
+	 * 2) The update may copy unexpected data from slots [N, M-1].
+	 */
+	if (bpfptr_is_kernel(uattr) && bpf_map_supports_cpu_flags(map->map_type) &&
+	    !(attr->flags & (BPF_F_CPU | BPF_F_ALL_CPUS)))
+		attr->flags |= BPF_F_ALL_CPUS;

Does this unintentionally break the bpf_sys_bpf() helper API for BPF_PROG_TYPE_SYSCALL programs?

Because bpf_sys_bpf() also wraps the pointer with KERNEL_BPFPTR, bpfptr_is_kernel(uattr) will evaluate to true for any eBPF program calling it.

Prior to this patch, a BPF_PROG_TYPE_SYSCALL program could legitimately update a per-CPU map with distinct values for each CPU by providing an array of num_possible_cpus() elements and omitting any CPU flags, which mirrors standard user-space API semantics.

Quoted 9 lines

With this change, it appears the kernel silently overrides attr->flags by setting BPF_F_ALL_CPUS. This truncates the internal value_size calculation down to a single element (map->value_size), reads only the first element of the BPF program's provided buffer, and incorrectly replicates it across all CPUs.

Should this workaround for the skeleton loader be constrained to bpftool gen skeleton instead of mutating the core syscall handler?

It was my intent. Even though BPF_PROG_TYPE_SYSCALL progs could be broken, it should be fixed when verifying the oob issue is true for SYSCALL progs.

This patch will be dropped from this series. And, a separate patch will be posted if the oob issue is true.

Thanks,
Leon

author
#26 lore raw

Re: [PATCH bpf-next v4 6/8] bpftool: Generate skeleton for global percpu data

In reply to [email protected]
Quoted 32 lines · On 15/4/26 05:26, [email protected] wrote:

Sashiko AI review found 1 potential issue(s):
- [High] Missing read-only memory protection for .percpu initial value buffer in generated light skeletons. --

commit e61b5ee6e5a7cee758b644de0f6b4aaea40f020c
Author: Leon Hwang <[email protected]>

bpftool: Generate skeleton for global percpu data

Enhance bpftool to generate skeletons that properly handle global percpu variables. The skeleton includes a nested structure for percpu data, which is marked as read-only after loading so users must update the map directly.

Quoted 3 lines
tools/bpf/bpftool/gen.c +0 -0

[ ... ]

tools/bpf/bpftool/gen.c static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *h · line 857

Quoted 9 lines
tools/bpf/bpftool/gen.c +1 -1
@@ -847,7 +851,7 @@ static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *h
 	bpf_object__for_each_map(map, obj) {
 		const char *mmap_flags;
 
-		if (!is_mmapable_map(map, ident, sizeof(ident)))
+		if (!is_mmapable_map(map, ident, sizeof(ident)) || bpf_map_is_percpu_data(map))
 			continue;
 
 		if (bpf_map__map_flags(map) & BPF_F_RDONLY_PROG)

The commit message notes that after loading the skeleton, maps.percpu->mmaped has been marked as read-only.

Quoted 8 lines

By skipping the .percpu map entirely here during light skeleton generation in gen_trace(), does this leave the user-space buffer for .percpu writable?

If a user modifies skel->percpu after the light skeleton is loaded, will it silently succeed without actually updating the kernel per-cpu map? Does the generated light skeleton need an explicit mprotect(..., PROT_READ) for the .percpu initial value buffer to enforce the read-only constraint?

The mprotect(..., PROT_READ) was added in patch #4.

I think it'd better to move the change to this patch. Then, readers won't miss the mprotect().

Thanks,
Leon

author
#29 lore raw

Re: [PATCH bpf-next v4 7/8] selftests/bpf: Add tests to verify global percpu data

In reply to [email protected]
Quoted 72 lines · On 15/4/26 05:45, [email protected] wrote:

Sashiko AI review found 1 potential issue(s):
- [Medium] The tests use the raw CPU ID to index into the densely packed percpu_data array, which will cause an out-of-bounds read and test failures on systems with sparse CPU masks. --

commit a7c2685b539ebc816cdb05decab1d7a8a4e1328e
Author: Leon Hwang <[email protected]>

selftests/bpf: Add tests to verify global percpu data

This commit adds new selftests to verify the behavior of global percpu data by checking map initial values, struct pointers, and looking up elements from global percpu data maps.

Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +0 -0

[ ... ]

Quoted 2 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +2 -0
+static void test_global_percpu_data_init(void)
+{

[ ... ]

tools/testing/selftests/bpf/prog_tests/global_data_init.c void test_global_data_init(void) · line 177

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +10 -0
+	err = bpf_map__lookup_elem(map, &key, sizeof(key), percpu_data,
+				   elem_sz * num_cpus, 0);
+	if (!ASSERT_OK(err, "bpf_map__lookup_elem"))
+		goto out;
+
+	for (i = 0; i < num_online; i++) {
+		if (!online[i])
+			continue;
+
+		data = (void *)percpu_data + elem_sz * i;

Will this logic cause an out-of-bounds read on systems with sparse CPU masks?

When reading a per-CPU map using bpf_map__lookup_elem(), the kernel packs the per-CPU values densely into the output buffer by iterating over the possible CPU mask.

The test allocates num_cpus (the total count of possible CPUs) elements for the output buffer, but then iterates over the raw CPU IDs using i and uses i as an index into the densely packed array.

On systems with sparse CPU masks (for example, where CPUs 0 and 100 are possible/online), num_cpus will be 2. When the loop reaches i = 100, it appears it will attempt to access percpu_data + elem_sz * 100, reading past the bounds of the allocated memory.

Could we maintain a separate index counter that increments only for possible CPUs to match the kernel's dense packing behavior?

[ ... ]

Quoted 2 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +2 -0
+static void test_global_percpu_data_lskel(void)
+{

[ ... ]

tools/testing/selftests/bpf/prog_tests/global_data_init.c void test_global_data_init(void) · line 252

Quoted 9 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +9 -0
+	err = bpf_map_lookup_elem(map_fd, &key, percpu_data);
+	if (!ASSERT_OK(err, "bpf_map_lookup_elem"))
+		goto out;
+
+	for (i = 0; i < num_online; i++) {
+		if (!online[i])
+			continue;
+
+		data = (void *)percpu_data + elem_sz * i;

Does this loop have the same bounds checking issue as mentioned above?

Quoted 3 lines

If i is a raw CPU ID on a system with a sparse CPU mask, this calculation could exceed the bounds of the percpu_data buffer.

Yep, agree with the review.

Instead of lookup values across all possible CPUs, it should lookup value on specified CPU using BPF_F_CPU, because the test only runs on online CPUs.

Thanks,
Leon

author
#32 lore raw

Re: [PATCH bpf-next v4 8/8] selftests/bpf: Add a test to verify bpf_iter for global percpu data

In reply to [email protected]
Quoted 37 lines · On 15/4/26 06:08, [email protected] wrote:

Sashiko AI review found 3 potential issue(s):
- [Medium] The dump_percpu_data program is autoloaded by default, breaking existing subtests on older kernels.
- [Medium] Checking err == -EOPNOTSUPP after skeleton load incorrectly handles unsupported kernels.
- [Medium] Casting the global datasec to an anonymous C struct relies on fragile compiler layout assumptions. --

commit 1e00e3e2b318918e67a2c7d2e251132e1b985484
Author: Leon Hwang <[email protected]>

selftests/bpf: Add a test to verify bpf_iter for global percpu data

Add a test to verify that it is OK to iter the percpu_array map used for global percpu data.

Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +0 -0
index 0b0384b6515ed..9ff09ff1981c4 100644

[ ... ]

Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +2 -0
@@ -269,6 +269,57 @@ static void test_global_percpu_data_verifier_failure(void)
+static void test_global_percpu_data_iter(void)
+{

[ ... ]

tools/testing/selftests/bpf/prog_tests/global_data_init.c static void test_global_percpu_data_verifier_failure(void) · line 296

Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +5 -0
+	err = test_global_percpu_data__load(skel);
+	if (err == -EOPNOTSUPP) {
+		test__skip();
+		goto out;
+	}

Will this correctly skip on older kernels? When the kernel verifier rejects an unsupported program type or feature during bpf_prog_load(), it typically returns -EINVAL (or -EACCES), not -EOPNOTSUPP.

Quoted 4 lines

Libbpf passes this error code back to the caller, so this condition might evaluate to false, causing the test to fail on the subsequent ASSERT_OK(err) check instead of gracefully skipping.

This err check should be dropped, because this test won't run when the kernel does not support global percpu data.

See previous patch:

+void test_global_percpu_data(void)
+{
+ if (!feat_supported(NULL, FEAT_PERCPU_DATA)) {
+ test__skip();
+ return;
+ }

Quoted 18 lines

[ ... ]

Quoted 4 lines
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +0 -0
index 947721c21f30c..0ddb820509ebf 100644

[ ... ]

tools/testing/selftests/bpf/progs/test_global_percpu_data.c int verifier_percpu_read(void *ctx) · line 72

Quoted 3 lines
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +3 -0
+SEC("iter/bpf_map_elem")
+__auxiliary
+int dump_percpu_data(struct bpf_iter__bpf_map_elem *ctx)

Will this break existing subtests on older kernels? The bpf program dump_percpu_data is defined with SEC("iter/bpf_map_elem") without the ? prefix, meaning libbpf will attempt to autoload it whenever test_global_percpu_data__load() is called.

Quoted 5 lines

Since the existing init and lskel subtests also load the full skeleton, they might inadvertently attempt to load this new iterator program. Should this use SEC("?iter/bpf_map_elem") to prevent autoloading by default, and only be explicitly autoloaded in the iter subtest?

Since older kernels do not have global percpu data support, this test won't run on them. See aforementioned explanation.

Quoted 26 lines

tools/testing/selftests/bpf/progs/test_global_percpu_data.c int verifier_percpu_read(void *ctx) · line 83

Quoted 11 lines
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +11 -0
+{
+	struct {
+		int data;
+		char run;
+		struct {
+			char set;
+			int i;
+			int nums[7];
+		} struct_data;
+		int nums[7];
+	} *pptr = ctx->value;

Is it safe to cast the global datasec to an anonymous C struct like this?

In the BPF source, these are declared as independent global variables. The layout of these variables in the .percpu ELF section is determined entirely by the compiler. While the current compiler version might emit the uninitialized nums array after the initialized variables (matching the struct layout), this relies on undocumented compiler behavior.

If the compiler alters the layout, or if variables are added or initialized differently in the future, the field offsets will mismatch. This could cause the step calculation to diverge from the map's actual value_size, leading to out-of-bounds reads.

Quoted 5 lines

Would it be safer to group these variables into a single explicit struct in the BPF code to guarantee their memory layout?

[ ... ]

Correct, I should not use such an anonymous struct, even if it matches the generated skeleton's embedded struct.

A better approach is to locate the int by offset, which will match its offset in the generated skeleton's embedded struct.

Thanks,
Leon

author
#7 lore raw
In reply to Leon Hwang
Quoted 23 lines · Leon Hwang
Quoted 2 lines · Alexei Starovoitov
Quoted 1 line · Leon Hwang

On 2026/4/14 22:10, [email protected] wrote:

[...]

Quoted 19 lines
Quoted 6 lines
Quoted 6 lines

In the v3 series, bpf_map_direct_read() itself had a guard (map->map_type != BPF_MAP_TYPE_ARRAY), which protected all callers. The v4 moved this to caller-side checks but appears to have missed const_reg_xfer().

Quoted 6 lines

Correct.

Will add a guard in bpf_map_direct_read() in the next revision:

	if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY)
		return -EINVAL;

hold on.

kernel/bpf/verifier.c +2 -1
 			    map->ops->map_direct_value_addr &&
-			    map->map_type != BPF_MAP_TYPE_INSN_ARRAY) {
+			    map->map_type != BPF_MAP_TYPE_INSN_ARRAY &&
+			    map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) {
Quoted 5 lines
Quoted 5 lines

map_direct_value_addr() is set, but then immediately disallowed ? Where else it's used?

Even if value_addr is working, then map_direct_value_meta() looks broken.

IIUC, map_direct_value_meta() is only used for dumping xlated insns. If no available map_direct_value_addr(), map_direct_value_meta() won't be called.

Quoted 3 lines

Ah, let me dive deeper.

As for the above changes, let me explain them using diff snippet.

@@ -5808,6 +5808,8 @@ int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val,

        u64 addr;
        int err;

+ if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY)

+               return -EINVAL;
        err = map->ops->map_direct_value_addr(map, &addr, off);
        if (err)
                return err;

It is to guard percpu_array map against const_reg_xfer(). Instead of updating const_reg_xfer(), better to update bpf_map_direct_read(). WDYT?

@@ -6362,7 +6364,8 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn

                        if (tnum_is_const(reg->var_off) &&
                            bpf_map_is_rdonly(map) &&
                            map->ops->map_direct_value_addr &&
-                           map->map_type != BPF_MAP_TYPE_INSN_ARRAY) {
+                           map->map_type != BPF_MAP_TYPE_INSN_ARRAY &&
+                           map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) {

Since directly reading data from global percpu data is different from the way of global data, such reading data for global percpu data must perform the 'this_cpu_off' to read the data on current CPU.

At the moment of load time, the reading cannot read data like the way of global data, also cannot cause verifier failure.

I'll add a test against bpf_map_is_rdonly()+percpu_array.

                                int map_off = off + reg->var_off.value;
                                u64 val = 0;

				err = bpf_map_direct_read(map, map_off, size,
							  &val, is_ldsx);
				if (err)
					return err;

				regs[value_regno].type = SCALAR_VALUE;
				__mark_reg_known(&regs[value_regno], val);

Thanks,
Leon

author
#23 lore raw

Re: [PATCH bpf-next v4 5/8] bpf: Update per-CPU maps using BPF_F_ALL_CPUS flag

In reply to Leon Hwang
Quoted 42 lines · Leon Hwang
Quoted 32 lines · Alexei Starovoitov
Quoted 31 lines · Leon Hwang

When updating per-CPU maps via the lightweight skeleton loader, use a single value slot across all CPUs. This avoids two potential issues when updating on an M-CPU kernel with N cached slots (N < M), especially when N is much smaller than M:

1) The update may trigger a page fault when copying data from the last slot, as the read may go beyond the allocated buffer.
2) The update may copy unexpected data from slots [N, M-1].

---
 kernel/bpf/syscall.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
kernel/bpf/syscall.c +9 -0
index b73b25c63073..f0f3785ef57d 100644
@@ -1785,6 +1785,21 @@ static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr)
 		goto err_put;
 	}
 
+	/*
+	 * When updating per-CPU maps via the lightweight skeleton
+	 * loader, use a single value slot across all CPUs. This avoids
+	 * two potential issues when updating on an M-CPU kernel with
+	 * N cached slots (N < M), especially when N is much smaller
+	 * than M:
+	 * 1) The update may trigger a page fault when copying data from
+	 *    the last slot, as the read may go beyond the allocated
+	 *    buffer.
Quoted 8 lines

kernel/bpf/syscall.c static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr) · line 1801

Quoted 5 lines
kernel/bpf/syscall.c +5 -0
+	 * 2) The update may copy unexpected data from slots [N, M-1].
+	 */
+	if (bpfptr_is_kernel(uattr) && bpf_map_supports_cpu_flags(map->map_type) &&
+	    !(attr->flags & (BPF_F_CPU | BPF_F_ALL_CPUS)))
+		attr->flags |= BPF_F_ALL_CPUS;

This looks like a hack. It's not addressing the actual bug. If there is a bug submit it separately with fixes tag.

Quoted 3 lines

Sure, will verify whether it is a bug. If it is, will fix it with separate patch.

By implementing two selftests [1] against syscall progs and lightweight skeleton, it is not a real issue.

The assuming oob reading might occur when updating percpu_array map using a small value buffer for syscall progs. However, the oob reading won't make kernel panic, because copy kernel memory using copy_from_kernel_nofault(). However, as for lskel, the oob reading doesn't occur, even set value size as 32000 and update percpu_array map using an int.

The assuming issue #2 that copies unexpected data is true for both syscall progs and lskel. However, I think it's users' responsibility to correctly update percpu_array map using big enough value data.

In conclusion, I'll drop this patch from this series, and won't send separate patch because it is not a bug.

[1] https://github.com/Asphaltt/bpf/commits/bpf/lskel-oob/v1/

Thanks,
Leon

#8 lore raw
In reply to Leon Hwang
Quoted 28 lines · Leon Hwang
Quoted 26 lines · Leon Hwang
Quoted 2 lines · Alexei Starovoitov
Quoted 1 line · Leon Hwang

On 2026/4/14 22:10, [email protected] wrote:

[...]

Quoted 22 lines
Quoted 6 lines
Quoted 6 lines

In the v3 series, bpf_map_direct_read() itself had a guard (map->map_type != BPF_MAP_TYPE_ARRAY), which protected all callers. The v4 moved this to caller-side checks but appears to have missed const_reg_xfer().

Quoted 6 lines

Correct.

Will add a guard in bpf_map_direct_read() in the next revision:

    if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY)
            return -EINVAL;

hold on.

                         map->ops->map_direct_value_addr &&
kernel/bpf/verifier.c +2 -1
-                        map->map_type != BPF_MAP_TYPE_INSN_ARRAY) {
+                        map->map_type != BPF_MAP_TYPE_INSN_ARRAY &&
+                        map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) {

map_direct_value_addr() is set, but then immediately disallowed ? Where else it's used?

Quoted 6 lines
Quoted 2 lines
Quoted 2 lines

Even if value_addr is working, then map_direct_value_meta() looks broken.

IIUC, map_direct_value_meta() is only used for dumping xlated insns. If no available map_direct_value_addr(), map_direct_value_meta() won't be called.

yes, but then xlated insn are bogus, no?

Quoted 17 lines
Quoted 3 lines

Ah, let me dive deeper.

As for the above changes, let me explain them using diff snippet.

(quoted diff) +2 -0
@@ -5808,6 +5808,8 @@ int bpf_map_direct_read(struct bpf_map *map, int
 off, int size, u64 *val,
        u64 addr;
        int err;
 
+       if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY)
+               return -EINVAL;
        err = map->ops->map_direct_value_addr(map, &addr, off);
        if (err)
                return err;
Quoted 2 lines

It is to guard percpu_array map against const_reg_xfer(). Instead of updating const_reg_xfer(), better to update bpf_map_direct_read(). WDYT?

yeah and move map_type != BPF_MAP_TYPE_INSN_ARRAY check into bpf_map_direct_read() as well.
To cleanup const_reg_xfer() a bit.

author
#9 lore raw
Quoted 29 lines · Alexei Starovoitov
Quoted 28 lines · Leon Hwang
Quoted 26 lines · Leon Hwang
Quoted 2 lines · Alexei Starovoitov
Quoted 1 line · Leon Hwang

On 2026/4/14 22:10, [email protected] wrote:

[...]

Quoted 22 lines
Quoted 6 lines
Quoted 6 lines

In the v3 series, bpf_map_direct_read() itself had a guard (map->map_type != BPF_MAP_TYPE_ARRAY), which protected all callers. The v4 moved this to caller-side checks but appears to have missed const_reg_xfer().

Quoted 6 lines

Correct.

Will add a guard in bpf_map_direct_read() in the next revision:

    if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY)
            return -EINVAL;

hold on.

                         map->ops->map_direct_value_addr &&
(quoted diff) +2 -1
-                        map->map_type != BPF_MAP_TYPE_INSN_ARRAY) {
+                        map->map_type != BPF_MAP_TYPE_INSN_ARRAY &&
+                        map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) {

map_direct_value_addr() is set, but then immediately disallowed ? Where else it's used?

Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines

Even if value_addr is working, then map_direct_value_meta() looks broken.

IIUC, map_direct_value_meta() is only used for dumping xlated insns. If no available map_direct_value_addr(), map_direct_value_meta() won't be called.

yes, but then xlated insn are bogus, no?

No. They look well.

./test_progs -t global_percpu_data -v

XLATED:
=============
0: r1 = 0x900000007
2: r1 = &(void __percpu *)(r1)
3: r1 = *(u8 *)(r1 +0)
4: w0 = 1
5: if w1 == 0x64 goto pc+1
6: w0 = 0
7: exit
=============
#152/5 global_percpu_data/verifier_percpu_read:OK

I also verified the xlated insns via bpftool.

static SEC(".data") __u32 cnt = 0; /* .data vs .percpu */

SEC("xdp")
int xdp_prog(struct xdp_md *ctx)

{
    cnt++;

    __u32 cpu = bpf_get_smp_processor_id();
    bpf_printk("cpu: %u, cnt: %u\n", cpu, cnt);

    return XDP_PASS;
}

When the section of cnt is ".data", bpftool d x n xdp_prog:

int xdp_prog(struct xdp_md * ctx):
; cnt++;
0: (18) r6 = map[id:23][0]+0
2: (61) r1 = *(u32 *)(r6 +0)
; cnt++;
3: (07) r1 += 1
; cnt++;
4: (63) *(u32 *)(r6 +0) = r1
; __u32 cpu = bpf_get_smp_processor_id();
5: (b7) r0 = -1280774092
6: (bf) r0 = &(void __percpu *)(r0)
7: (61) r0 = *(u32 *)(r0 +0)
; bpf_printk("cpu: %u, cnt: %u\n", cpu, cnt);
8: (61) r4 = *(u32 *)(r6 +0)
9: (18) r1 = map[id:24][0]+0
11: (b7) r2 = 18
12: (bf) r3 = r0
13: (85) call bpf_trace_printk#-129408
; return XDP_PASS;
14: (b7) r0 = 2
15: (95) exit

When the section of cnt is ".percpu", bpftool d x n xdp_prog:

int xdp_prog(struct xdp_md * ctx):
; cnt++;
0: (18) r6 = map[id:28][0]+0
2: (bf) r6 = &(void __percpu *)(r6)
3: (61) r1 = *(u32 *)(r6 +0)
; cnt++;
4: (07) r1 += 1
; cnt++;
5: (63) *(u32 *)(r6 +0) = r1
; __u32 cpu = bpf_get_smp_processor_id();
6: (b7) r0 = -1280774092
7: (bf) r0 = &(void __percpu *)(r0)
8: (61) r0 = *(u32 *)(r0 +0)
; bpf_printk("cpu: %u, cnt: %u\n", cpu, cnt);
9: (61) r4 = *(u32 *)(r6 +0)
10: (18) r1 = map[id:30][0]+0
12: (b7) r2 = 18
13: (bf) r3 = r0
14: (85) call bpf_trace_printk#-129408
; return XDP_PASS;
15: (b7) r0 = 2
16: (95) exit

The difference between these xlated insns is "r6 = &(void __percpu *)(r6)". This insn is for ".percpu", not for ".data".

Quoted 17 lines
Quoted 17 lines
Quoted 3 lines

Ah, let me dive deeper.

As for the above changes, let me explain them using diff snippet.

(quoted diff) +2 -0
@@ -5808,6 +5808,8 @@ int bpf_map_direct_read(struct bpf_map *map, int
 off, int size, u64 *val,
        u64 addr;
        int err;
 
+       if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY)
+               return -EINVAL;
        err = map->ops->map_direct_value_addr(map, &addr, off);
        if (err)
                return err;
Quoted 6 lines
Quoted 2 lines

It is to guard percpu_array map against const_reg_xfer(). Instead of updating const_reg_xfer(), better to update bpf_map_direct_read(). WDYT?

yeah and move map_type != BPF_MAP_TYPE_INSN_ARRAY check into bpf_map_direct_read() as well.
To cleanup const_reg_xfer() a bit.

Before sending the next revision, just confirm the change:

1. Move "map->map_type == BPF_MAP_TYPE_INSN_ARRAY" from const_reg_xfer() to bpf_map_direct_read().
2. Keep "map->map_type != BPF_MAP_TYPE_INSN_ARRAY" in check_mem_access(), because we should not propagate the error from bpf_map_direct_read() for insn_array and percpu_array.

Thanks,
Leon

---

kernel/bpf/const_fold.c +0 -1
@@ -174,7 +181,6 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info *
                u64 val = 0;
 
                if (!bpf_map_is_rdonly(map) || !map->ops->map_direct_value_addr ||
-                   map->map_type == BPF_MAP_TYPE_INSN_ARRAY ||
                    off < 0 || off + size > map->value_size ||
                    bpf_map_direct_read(map, off, size, &val, is_ldsx)) {
                        *dst = unknown;
kernel/bpf/verifier.c +4 -1
@@ -5816,6 +5816,8 @@ int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val,
        u64 addr;
        int err;
 
+       if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY || map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY)
+               return -EINVAL;
        err = map->ops->map_direct_value_addr(map, &addr, off);
        if (err)
                return err;
@@ -6370,7 +6372,8 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn
                        if (tnum_is_const(reg->var_off) &&
                            bpf_map_is_rdonly(map) &&
                            map->ops->map_direct_value_addr &&
-                           map->map_type != BPF_MAP_TYPE_INSN_ARRAY) {
+                           map->map_type != BPF_MAP_TYPE_INSN_ARRAY &&
+                           map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) {
                                int map_off = off + reg->var_off.value;
                                u64 val = 0;
#10 lore raw
Quoted 5 lines · Leon Hwang

int xdp_prog(struct xdp_md * ctx):

; cnt++;
   0: (18) r6 = map[id:28][0]+0
   2: (bf) r6 = &(void __percpu *)(r6)

well. that insn was inserted by the verifier and it shows up in xlated. That was expected.
The point about 'bogus xlated' was about offset translation. map_direct_value_meta() should recover proper insns[i + 1].imm = off; In your example it's zero, so not an interesting test.

Quoted 62 lines
   3: (61) r1 = *(u32 *)(r6 +0)
; cnt++;
   4: (07) r1 += 1
; cnt++;
   5: (63) *(u32 *)(r6 +0) = r1
; __u32 cpu = bpf_get_smp_processor_id();
   6: (b7) r0 = -1280774092
   7: (bf) r0 = &(void __percpu *)(r0)
   8: (61) r0 = *(u32 *)(r0 +0)
; bpf_printk("cpu: %u, cnt: %u\n", cpu, cnt);
   9: (61) r4 = *(u32 *)(r6 +0)

10: (18) r1 = map[id:30][0]+0
12: (b7) r2 = 18
13: (bf) r3 = r0
14: (85) call bpf_trace_printk#-129408
; return XDP_PASS;
15: (b7) r0 = 2
16: (95) exit

The difference between these xlated insns is "r6 = &(void __percpu *)(r6)". This insn is for ".percpu", not for ".data".

Quoted 17 lines
Quoted 17 lines
Quoted 3 lines

Ah, let me dive deeper.

As for the above changes, let me explain them using diff snippet.

kernel/bpf/verifier.c +2 -0
@@ -5808,6 +5808,8 @@ int bpf_map_direct_read(struct bpf_map *map, int
 off, int size, u64 *val,
        u64 addr;
        int err;
 
+       if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY)
+               return -EINVAL;
        err = map->ops->map_direct_value_addr(map, &addr, off);
        if (err)
                return err;
Quoted 6 lines
Quoted 2 lines

It is to guard percpu_array map against const_reg_xfer(). Instead of updating const_reg_xfer(), better to update bpf_map_direct_read(). WDYT?

yeah and move map_type != BPF_MAP_TYPE_INSN_ARRAY check into bpf_map_direct_read() as well.
To cleanup const_reg_xfer() a bit.

Before sending the next revision, just confirm the change:

1. Move "map->map_type == BPF_MAP_TYPE_INSN_ARRAY" from const_reg_xfer() to bpf_map_direct_read().
2. Keep "map->map_type != BPF_MAP_TYPE_INSN_ARRAY" in check_mem_access(), because we should not propagate the error from bpf_map_direct_read() for insn_array and percpu_array.

Thanks,
Leon

---

kernel/bpf/const_fold.c static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info * · line 177

Quoted 4 lines
kernel/bpf/const_fold.c +0 -1
@@ -174,7 +181,6 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info *
                u64 val = 0;
 
                if (!bpf_map_is_rdonly(map) || !map->ops->map_direct_value_addr ||
-                   map->map_type == BPF_MAP_TYPE_INSN_ARRAY ||

yes

kernel/bpf/verifier.c int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val, · line 5820

Quoted 12 lines
kernel/bpf/const_fold.c +0 -0
                    off < 0 || off + size > map->value_size ||
                    bpf_map_direct_read(map, off, size, &val, is_ldsx)) {
                        *dst = unknown;
kernel/bpf/verifier.c +2 -0
@@ -5816,6 +5816,8 @@ int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val,
        u64 addr;
        int err;
 
+       if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY || map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY)
+               return -EINVAL;

yes

kernel/bpf/verifier.c static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · line 6376

Quoted 10 lines
kernel/bpf/verifier.c +2 -1
        err = map->ops->map_direct_value_addr(map, &addr, off);
        if (err)
                return err;
@@ -6370,7 +6372,8 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn
                        if (tnum_is_const(reg->var_off) &&
                            bpf_map_is_rdonly(map) &&
                            map->ops->map_direct_value_addr &&
-                           map->map_type != BPF_MAP_TYPE_INSN_ARRAY) {
+                           map->map_type != BPF_MAP_TYPE_INSN_ARRAY &&
+                           map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) {

why add BPF_MAP_TYPE_PERCPU_ARRAY here?

Quoted 2 lines
kernel/bpf/verifier.c +0 -0
                                int map_off = off + reg->var_off.value;
                                u64 val = 0;
author
#11 lore raw
Quoted 7 lines · Alexei Starovoitov
Quoted 5 lines · Leon Hwang

int xdp_prog(struct xdp_md * ctx):
; cnt++;
0: (18) r6 = map[id:28][0]+0
2: (bf) r6 = &(void __percpu *)(r6)

Quoted 6 lines

well. that insn was inserted by the verifier and it shows up in xlated. That was expected.
The point about 'bogus xlated' was about offset translation. map_direct_value_meta() should recover proper insns[i + 1].imm = off; In your example it's zero, so not an interesting test.

The original insns[i + 1] wasn't broken. See the key change of this patch.

+ /*
+ * Reuse the original ld_imm64 insn, and add one
+ * mov64_percpu_reg insn.
+ */
+
+ insn_buf[0] = insn[1];
+ insn_buf[1] = BPF_MOV64_PERCPU_REG(insn->dst_reg, insn->dst_reg);
+ cnt = 2;
+
+ i++;
+ new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, cnt);
+ if (!new_prog)
+ return -ENOMEM;

So, it didn't touch the original ld_imm64 insn. That said, no bogus xlated about offset translation.

Quoted 1 line

[...]

Quoted 10 lines

kernel/bpf/verifier.c static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · line 6376

Quoted 7 lines
(quoted diff) +0 -0
@@ -6370,7 +6372,8 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn

if (tnum_is_const(reg->var_off) &&
bpf_map_is_rdonly(map) &&
map->ops->map_direct_value_addr &&

kernel/bpf/verifier.c +2 -1
-                           map->map_type != BPF_MAP_TYPE_INSN_ARRAY) {
+                           map->map_type != BPF_MAP_TYPE_INSN_ARRAY &&
+                           map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) {

why add BPF_MAP_TYPE_PERCPU_ARRAY here?

After looking into bpf_map_is_rdonly(), yes, no need to add check for percpu_array here.

Thanks,
Leon

Quoted 2 lines
Quoted 2 lines

int map_off = off + reg->var_off.value;
u64 val = 0;

#12 lore raw
Quoted 16 lines · Leon Hwang
Quoted 7 lines · Alexei Starovoitov
Quoted 5 lines · Leon Hwang

int xdp_prog(struct xdp_md * ctx):
; cnt++;
0: (18) r6 = map[id:28][0]+0
2: (bf) r6 = &(void __percpu *)(r6)

Quoted 6 lines

well. that insn was inserted by the verifier and it shows up in xlated. That was expected.
The point about 'bogus xlated' was about offset translation. map_direct_value_meta() should recover proper insns[i + 1].imm = off; In your example it's zero, so not an interesting test.

Quoted 2 lines

The original insns[i + 1] wasn't broken. See the key change of this patch.

You're missing the point insns[i + 1].imm = off is how off returned from map_direct_value_meta() is used in updating _this_ ldimm64 insn.

author
#13 lore raw
Quoted 17 lines · Alexei Starovoitov
Quoted 16 lines · Leon Hwang
Quoted 13 lines · Alexei Starovoitov
Quoted 5 lines · Leon Hwang

int xdp_prog(struct xdp_md * ctx):
; cnt++;
0: (18) r6 = map[id:28][0]+0
2: (bf) r6 = &(void __percpu *)(r6)

well. that insn was inserted by the verifier and it shows up in xlated. That was expected.
The point about 'bogus xlated' was about offset translation. map_direct_value_meta() should recover proper insns[i + 1].imm = off; In your example it's zero, so not an interesting test.

Quoted 6 lines
Quoted 2 lines

The original insns[i + 1] wasn't broken. See the key change of this patch.

You're missing the point insns[i + 1].imm = off is how off returned from map_direct_value_meta() is used in updating _this_ ldimm64 insn.

After looking into bpf_insn_prepare_dump(), agreed with "not an interesting test".

Then, updated the selftest with non-zero offset. And updated disasm_helpers.c::disasm_insn() to dump such ldimm64 insn as map info by the way of bpftool/xlated_dumper.c::print_imm().

+static volatile const char fmt2[] SEC(".percpu.fmt") = "data %d\n"; +
+SEC("?kprobe")
+__success
+__xlated("r{{[0-9]+}} = map[id:{{[0-9]+}}][0]+9") +__xlated("r{{[0-9]+}} = &(void __percpu *)(r{{[0-9]+}})") +__xlated("r{{[0-9]+}} = *(u8 *)(r{{[0-9]+}} +4)")
+int verifier_percpu_read(void *ctx)
+{
+ char c = fmt2[4];
+
+ return c == ' ';
+}

It would verify the following xlated insns.

XLATED:
=============
0: r1 = map[id:7][0]+9
2: r1 = &(void __percpu *)(r1)
3: r1 = *(u8 *)(r1 +4)
4: w0 = 1
5: if w1 == 0x20 goto pc+1
6: w0 = 0
7: exit

The selftest looks better now.

Thanks,
Leon

#14 lore raw
Quoted 33 lines · Leon Hwang
Quoted 17 lines · Alexei Starovoitov
Quoted 16 lines · Leon Hwang
Quoted 13 lines · Alexei Starovoitov
Quoted 5 lines · Leon Hwang

int xdp_prog(struct xdp_md * ctx):
; cnt++;
0: (18) r6 = map[id:28][0]+0
2: (bf) r6 = &(void __percpu *)(r6)

well. that insn was inserted by the verifier and it shows up in xlated. That was expected.
The point about 'bogus xlated' was about offset translation. map_direct_value_meta() should recover proper insns[i + 1].imm = off; In your example it's zero, so not an interesting test.

Quoted 6 lines
Quoted 2 lines

The original insns[i + 1] wasn't broken. See the key change of this patch.

You're missing the point insns[i + 1].imm = off is how off returned from map_direct_value_meta() is used in updating _this_ ldimm64 insn.

After looking into bpf_insn_prepare_dump(), agreed with "not an interesting test".

Then, updated the selftest with non-zero offset. And updated disasm_helpers.c::disasm_insn() to dump such ldimm64 insn as map info by the way of bpftool/xlated_dumper.c::print_imm().

Review

Quoted 7 lines
(quoted diff) +7 -0
+static volatile const char fmt2[] SEC(".percpu.fmt") = "data %d\n";
+
+SEC("?kprobe")
+__success
+__xlated("r{{[0-9]+}} = map[id:{{[0-9]+}}][0]+9")
+__xlated("r{{[0-9]+}} = &(void __percpu *)(r{{[0-9]+}})")
+__xlated("r{{[0-9]+}} = *(u8 *)(r{{[0-9]+}} +4)")

This is still pointless test.

Quoted 7 lines
(quoted diff) +6 -0
+int verifier_percpu_read(void *ctx)
+{
+       char c = fmt2[4];
+
+       return c == ' ';
+}
Quoted 5 lines

It would verify the following xlated insns.

XLATED:
=============
0: r1 = map[id:7][0]+9

what is 9 ? was it there in the incoming insn?

The whole point that xlated needs to translate it back. Start from scratch and think what map_direct_value_meta() suppose to do.

author
#15 lore raw
Quoted 1 line · Alexei Starovoitov

On Tue, Apr 21, 2026 at 7:24 AM Leon Hwang <[email protected]> wrote:

[...]

Quoted 6 lines
Quoted 3 lines

XLATED:
=============
0: r1 = map[id:7][0]+9

what is 9 ? was it there in the incoming insn?

Right, the literal 9 is a mystery here.

Quoted 3 lines

The whole point that xlated needs to translate it back. Start from scratch and think what map_direct_value_meta() suppose to do.

Thanks for pointing this out.

map_direct_value_addr() encodes the offset as part of the ldimm64 immediate (imm = base + off), and map_direct_value_meta() is supposed to recover the same off from it.

So, the test should verify that the off of xlated ldimm64 insn matches the off encoded in the ELF ldimm64 insn.

Thanks,
Leon