Skip to contentSkip to conversation

bpf: Introduce global percpu data

PATCH Superseded Leon Hwang 5 participants 35 messages Jul 15 – Aug 4

Subsystems (6)
Kselftest BPF BPF selftests BPF core +2 more subsystems

v10 of 12 Superseded Latest is v12 →

lore
Share & tools
lore

Leon Hwang

author
#0 lore raw

v10 cover bpf: Introduce global percpu data

This patch set introduces global percpu data, similar to commit 6316f78306c1 ("Merge branch 'support-global-data'"), to reduce restrictions in C for BPF programs.

With this enhancement, it becomes possible to define and use global percpu variables, like the DEFINE_PER_CPU() macro in the kernel include/linux/percpu-defs.h.

The section name for global peurcpu data is ".percpu". Even though, a one-byte percpu variable (e.g., char run SEC(".percpu") = 0;) can trigger a crash with Clang 17 [1], users are expected to use such small variables as global percpu data with newer Clang versions, which don't have the issue.

The idea stems from the bpfsnoop [2], which itself was inspired by retsnoop [3]. During testing of bpfsnoop on the v6.6 kernel, two LBR (Last Branch Record) entries were observed related to the bpf_get_smp_processor_id() helper.

Since commit 1ae6921009e5 ("bpf: inline bpf_get_smp_processor_id() helper"), the bpf_get_smp_processor_id() helper has been inlined on x86_64, reducing the overhead and consequently minimizing these two LBR records.

However, the introduction of global percpu data offers a more robust solution. By leveraging the percpu_array map and percpu instruction, global percpu data can be implemented intrinsically.

This feature also facilitates sharing percpu information between tail callers and callees or between freplace callers and callees through a shared global percpu variable. Previously, this was achieved using a 1-entry percpu_array map, which this patch set aims to improve upon.

Links:
[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://github.com/bpfsnoop/bpfsnoop
[3] https://github.com/anakryiko/retsnoop

Changes:
v9 -> v10:
* Rebase latest bpf-next tree to resolve code conflict in verifier in patch #1.
* v9: https://lore.kernel.org/bpf/[email protected]/

v8 -> v9:
* Use real name for percpu data maps in libbpf in patch #4.
* Add long map name test in patch #6.
* Move parse_cpu_mask_file() to test_percpu_data_on_cpus() in test in patch #6.
* Validate map type in get_map_ident() for percpu data maps in patch #5.
* Update code comment in verifier in patch #2. (per Andrii)
* Pass 'type' to internal_map_name in libbpf in patch #4. (per Andrii)
* Factor out the helper is_skel_data() in bpftool in patch #5. (per Quentin and Andrii)
* v8: https://lore.kernel.org/bpf/[email protected]/

v7 -> v8:
* Send patch #1 and #2 separately that fix interpreter fallback issues. (Andrii)
* Use 'array->elem_size' to avoid 'range' local variable in percpu_array_map_direct_value_meta(). (Andrii)
* Keep original map name for percpu data's map in libbpf. (Andrii)
* Factor out helper bpf_map_is_skel_data() in bpftool. (Andrii)
* Update commit message of direct access read-only percpu_array map. (Andrii)
* Add test to verify that it is disallowed to directly write data of read-only percpu_array map. (Andrii)
* Drop unused 'num_cpus' in test. (bot+bpf-ci)
* Factor out helper test_percpu_data_on_cpus() in test. (bot+bpf-ci)
* v7: https://lore.kernel.org/bpf/[email protected]/

v6 -> v7:
* Use tgt_endian() in bpf_gen__map_update_elem() in patch #6. (Sashiko)
* Use sizeof(args) in verifier_snprintf test in patch #10. (Sashiko)
* Drop xlated test of v6. (Alexei)
* v6: https://lore.kernel.org/bpf/[email protected]/

v5 -> v6:
* Prevent running user addr_space_cast and addr_percpu insns in interpreter. (Sashiko)
* Cast __percpu pointer to u64 with (__force unsigned long). (lkp)
* Exclude BPF_MAP_TYPE_PERCPU_ARRAY in check_mem_access() before calling bpf_map_direct_read(), and add a test to verify it.
(Sashiko, bot+bpf-ci)
* Skip percpu data variables for subskeleton in bpftool. (Sashiko)
* Protect skel->percpu using mprotect(..., PROT_READ) in light skeleton. (Sashiko, bot+bpf-ci)
* Drop roundup() in tests. (Sashiko)
* Call test_global_percpu_data_verifier_log() without test__start_subtest(). (Sashiko)
* Cast insn->imm to __u64 with (__u32) in xlated test. (Sashiko)
* Check cnt using the new idx in xlated test. (Sashiko)
* v5: https://lore.kernel.org/bpf/[email protected]/

v4 -> v5:
* Add prog->jit_requested check to prevent running percpu data in interpreter in patch #1.
* Factor out verifier log tests using its own patch.
* Address comments from Alexei:
* Move map_type check from check_mem_access() to bpf_map_direct_read() in patch #2.
* Move BPF_MAP_TYPE_INSN_ARRAY map_type check from const_reg_xfer() to bpf_map_direct_read() in patch #2.
* Add a test to verify that the off of xlated ldimm64 insn matches the off encoded in the ELF ldimm64 insn.
* Drop patch #5 of v4.
* Address reviews from Sashiko:
* Update commit message of patch #6 to indicate that maps.percpu->mmaped has been marked as read-only in libbpf.
* Lookup elem on specified CPU using BPF_F_CPU in tests.
* Drop unnecessary err == -EOPNOTSUPP in test.
* Locate target field using its offset in the iter test.
* v4: https://lore.kernel.org/bpf/[email protected]/

v3 -> v4:
* Drop duplicate blank lines in verifier.
* Add percpu data feature probe in libbpf.
* Update percpu_array map using BPF_F_ALL_CPUS flag for lskel, if no cpu flag is set.
* Add two tests to verify verifier log.
* Add a test to verify mov64_percpu_reg instruction.
* Add a test to verify bpf_iter for percpu data map.
* Update percpu_array map using BPF_F_ALL_CPUS flag in libbpf (per Alexei and Andrii).
* Address comments from Andrii:
* Use .percpu as section identifier.
* Use bpf_jit_supports_percpu_insn() instead of CONFIG_SMP.
* Drop bpf_map__is_internal_percpu() API.
* Drop unnecessary __aligned(8) in libbpf, verified by selftest.
* Make mmap data read-only after loading prog.
v3: https://lore.kernel.org/bpf/[email protected]/

v2 -> v3:
* Use ".data..percpu" as PERCPU_DATA_SEC.
* Address comment from Alexei:
* Add u8, array of ints and struct { .. } vars to selftest.
v2: https://lore.kernel.org/bpf/[email protected]/

v1 -> v2:
* Address comments from Andrii:
* Use LIBBPF_MAP_PERCPU and SEC_PERCPU.
* Reuse mmaped of libbpf's struct bpf_map for .percpu map data.
* Set .percpu struct pointer to NULL after loading skeleton.
* Make sure value size of .percpu map is __aligned(8).
* Use raw_tp and opts.cpu to test global percpu variables on all CPUs.
* Address comments from Alexei:
* Test non-zero offset of global percpu variable.
* Test case about BPF_PSEUDO_MAP_IDX_VALUE.
v1: https://lore.kernel.org/bpf/[email protected]/

rfc -> v1:
* Address comments from Andrii:
* Keep one image of global percpu variable for all CPUs.
* Reject non-ARRAY map in bpf_map_direct_read(), check_reg_const_str(), and check_bpf_snprintf_call() in verifier.
* Split out libbpf changes from kernel-side changes.
* Use ".percpu" as PERCPU_DATA_SEC.
* Use enum libbpf_map_type to distinguish BSS, DATA, RODATA and PERCPU_DATA.
* Avoid using errno for checking err from libbpf_num_possible_cpus().
* Use "map '%s': " prefix for error message.
rfc: https://lore.kernel.org/bpf/[email protected]/

Leon Hwang (9):
bpf: Drop duplicate blank lines in verifier
bpf: Introduce global percpu data
libbpf: Probe percpu data feature
libbpf: Add support for global percpu data
bpftool: Generate skeleton for global percpu data selftests/bpf: Add tests to verify global percpu data selftests/bpf: Test direct reading/writing read-only percpu_array map selftests/bpf: Test verifier log for global percpu data selftests/bpf: Verify bpf_iter for global percpu data

 kernel/bpf/arraymap.c                         |  38 ++-
 kernel/bpf/const_fold.c                       |   1 -
 kernel/bpf/fixups.c                           |  38 +++
 kernel/bpf/verifier.c                         |  29 +-
 tools/bpf/bpftool/gen.c                       |  76 ++++-
 tools/lib/bpf/bpf_gen_internal.h              |   3 +-
 tools/lib/bpf/features.c                      |  35 +++
 tools/lib/bpf/gen_loader.c                    |   3 +-
 tools/lib/bpf/libbpf.c                        |  72 ++++-
 tools/lib/bpf/libbpf_internal.h               |   2 +
 tools/lib/bpf/skel_internal.h                 |  24 +-
 tools/testing/selftests/bpf/Makefile          |   2 +-
 .../bpf/prog_tests/global_data_init.c         | 292 ++++++++++++++++++
 .../bpf/prog_tests/global_percpu_subskel.c    |  37 +++
 .../bpf/progs/test_global_percpu_data.c       |  81 +++++
 15 files changed, 683 insertions(+), 50 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_global_percpu_data.c
Signature · 1 line
2.55.0
author
#1 lore raw

PATCH v10 1/9 bpf: Drop duplicate blank lines in verifier

There are many adjacent blank lines in the verifier that have accumulated over time.

Drop them for cleanup.

No functional changes intended.

---
 kernel/bpf/verifier.c | 14 --------------
 1 file changed, 14 deletions(-)
kernel/bpf/verifier.c +0 -14
index de816063ae63..e1244a721194 100644
@@ -634,7 +634,6 @@ static void __mark_dynptr_reg(struct bpf_reg_state *reg,
 			      enum bpf_dynptr_type type,
 			      bool first_slot, int id, int parent_id);
 
-
 static void mark_dynptr_stack_regs(struct bpf_verifier_env *env,
 				   struct bpf_reg_state *sreg1,
 				   struct bpf_reg_state *sreg2,
@@ -1671,7 +1670,6 @@ static bool same_callsites(struct bpf_verifier_state *a, struct bpf_verifier_sta
 	return true;
 }
 
-
 void bpf_free_backedges(struct bpf_scc_visit *visit)
 {
 	struct bpf_scc_backedge *backedge, *next;
@@ -2291,7 +2289,6 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env,
 	return &elem->st;
 }
 
-
 static int cmp_subprogs(const void *a, const void *b)
 {
 	return ((struct bpf_subprog_info *)a)->start -
@@ -4044,7 +4041,6 @@ static int check_stack_read(struct bpf_verifier_env *env,
 	return err;
 }
 
-
 /* check_stack_write dispatches to check_stack_write_fixed_off or
  * check_stack_write_var_off.
  *
@@ -4840,7 +4836,6 @@ static int check_sock_access(struct bpf_verifier_env *env, int insn_idx,
 		valid = false;
 	}
 
-
 	if (valid) {
 		env->insn_aux_data[insn_idx].ctx_field_size =
 			info.ctx_field_size;
@@ -6683,7 +6678,6 @@ static int check_stack_range_initialized(
 	if (err)
 		return err;
 
-
 	if (tnum_is_const(reg->var_off)) {
 		min_off = max_off = reg->var_off.value + off;
 	} else {
@@ -7404,7 +7398,6 @@ static bool is_iter_new_kfunc(struct bpf_call_arg_meta *meta)
 	return meta->kfunc_flags & KF_ITER_NEW;
 }
 
-
 static bool is_iter_destroy_kfunc(struct bpf_call_arg_meta *meta)
 {
 	return meta->kfunc_flags & KF_ITER_DESTROY;
@@ -11539,7 +11532,6 @@ static int process_irq_flag(struct bpf_verifier_env *env, struct bpf_reg_state *
 	return 0;
 }
 
-
 static int ref_set_non_owning(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
 {
 	struct btf_record *rec = reg_btf_record(reg);
@@ -16421,7 +16413,6 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn)
 	return 0;
 }
 
-
 static bool return_retval_range(struct bpf_verifier_env *env, struct bpf_retval_range *range)
 {
 	enum bpf_prog_type prog_type = resolve_prog_type(env->prog);
@@ -18363,8 +18354,6 @@ static void release_insn_arrays(struct bpf_verifier_env *env)
 		bpf_insn_array_release(env->insn_array_maps[i]);
 }
 
-
-
 /* The verifier does more data flow analysis than llvm and will not
  * explore branches that are dead at run time. Malicious programs can
  * have dead code too. Therefore replace all dead at-run-time code
@@ -18392,8 +18381,6 @@ static void sanitize_dead_code(struct bpf_verifier_env *env)
 	}
 }
 
-
-
 static void free_states(struct bpf_verifier_env *env)
 {
 	struct bpf_verifier_state_list *sl;
@@ -18675,7 +18662,6 @@ static int do_check_main(struct bpf_verifier_env *env)
 	return ret;
 }
 
-
 static void print_verification_stats(struct bpf_verifier_env *env)
 {
 	/* Skip over hidden subprogs which are not verified. */
Signature · 1 line
2.55.0
author
#5 lore raw

PATCH v10 2/9 bpf: Introduce global percpu data

In reply to Leon Hwang

Introduce global percpu data, inspired by the commit 6316f78306c1 ("Merge branch 'support-global-data'"). It enables the definition of global percpu variables in BPF, similar to the include/linux/percpu-defs.h::DEFINE_PER_CPU() macro.

For example, in BPF, it is able to define a global percpu variable like:

int data SEC(".percpu");

With this patch, tools like retsnoop [1] and bpfsnoop [2] can simplify their BPF code for handling LBRs. The code can be updated from

static struct perf_branch_entry lbrs[1][MAX_LBR_ENTRIES] SEC(".data.lbrs");

to

static struct perf_branch_entry lbrs[MAX_LBR_ENTRIES] SEC(".percpu.lbrs");

This eliminates the need to retrieve the CPU ID using the bpf_get_smp_processor_id() helper.

Additionally, by reusing global percpu data map, sharing information between tail callers and callees or freplace callers and callees becomes simpler compared to reusing percpu_array maps.

Links:
[1] https://github.com/anakryiko/retsnoop
[2] https://github.com/bpfsnoop/bpfsnoop

---
 kernel/bpf/arraymap.c   | 38 ++++++++++++++++++++++++++++++++++++--
 kernel/bpf/const_fold.c |  1 -
 kernel/bpf/fixups.c     | 38 ++++++++++++++++++++++++++++++++++++++
 kernel/bpf/verifier.c   | 15 +++++++++++++++
 4 files changed, 89 insertions(+), 3 deletions(-)
kernel/bpf/arraymap.c +36 -2
index 248b4818178c..c4e9430941e5 100644
@@ -259,6 +259,37 @@ static void *percpu_array_map_lookup_elem(struct bpf_map *map, void *key)
 	return this_cpu_ptr(array->pptrs[index & array->index_mask]);
 }
 
+static int percpu_array_map_direct_value_addr(const struct bpf_map *map, u64 *imm, u32 off)
+{
+	struct bpf_array *array = container_of(map, struct bpf_array, map);
+
+	if (map->max_entries != 1)
+		return -EOPNOTSUPP;
+	if (off >= map->value_size)
+		return -EINVAL;
+	if (!bpf_jit_supports_percpu_insn())
+		return -EOPNOTSUPP;
+
+	*imm = (u64)(__force unsigned long) array->pptrs[0];
+	return 0;
+}
+
+static int percpu_array_map_direct_value_meta(const struct bpf_map *map, u64 imm, u32 *off)
+{
+	struct bpf_array *array = container_of(map, struct bpf_array, map);
+	u64 base = (u64)(__force unsigned long) array->pptrs[0];
+
+	if (map->max_entries != 1)
+		return -EOPNOTSUPP;
+	if (imm < base || imm >= base + array->elem_size)
+		return -ENOENT;
+	if (!bpf_jit_supports_percpu_insn())
+		return -EOPNOTSUPP;
+
+	*off = imm - base;
+	return 0;
+}
+
 /* emit BPF instructions equivalent to C code of percpu_array_map_lookup_elem() */
 static int percpu_array_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf)
 {
@@ -551,9 +582,10 @@ static int array_map_check_btf(struct bpf_map *map,
 			       const struct btf_type *key_type,
 			       const struct btf_type *value_type)
 {
-	/* One exception for keyless BTF: .bss/.data/.rodata map */
+	/* One exception for keyless BTF: .bss/.data/.rodata/.percpu map */
 	if (btf_type_is_void(key_type)) {
-		if (map->map_type != BPF_MAP_TYPE_ARRAY ||
+		if ((map->map_type != BPF_MAP_TYPE_ARRAY &&
+		     map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) ||
 		    map->max_entries != 1)
 			return -EINVAL;
 
@@ -832,6 +864,8 @@ const struct bpf_map_ops percpu_array_map_ops = {
 	.map_get_next_key = bpf_array_get_next_key,
 	.map_lookup_elem = percpu_array_map_lookup_elem,
 	.map_gen_lookup = percpu_array_map_gen_lookup,
+	.map_direct_value_addr = percpu_array_map_direct_value_addr,
+	.map_direct_value_meta = percpu_array_map_direct_value_meta,
 	.map_update_elem = array_map_update_elem,
 	.map_delete_elem = array_map_delete_elem,
 	.map_lookup_percpu_elem = percpu_array_map_lookup_percpu_elem,
kernel/bpf/const_fold.c +0 -1
index b2a19acadb91..5787246bef30 100644
@@ -182,7 +182,6 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info *
 		u64 val = 0;
 
 		if (!bpf_map_is_rdonly(map) || !map->ops->map_direct_value_addr ||
-		    map->map_type == BPF_MAP_TYPE_INSN_ARRAY ||
 		    off < 0 || off + size > map->value_size ||
 		    bpf_map_direct_read(map, off, size, &val, is_ldsx)) {
 			*dst = unknown;
kernel/bpf/fixups.c +38 -0
index d3be972714b2..02642bba73e5 100644
@@ -1818,6 +1818,44 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
 			goto next_insn;
 		}
 
+		if (env->prog->jit_requested &&
+		    bpf_jit_supports_percpu_insn() &&
+		    insn->code == (BPF_LD | BPF_IMM | BPF_DW) &&
+		    (insn->src_reg == BPF_PSEUDO_MAP_VALUE ||
+		     insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE)) {
+			struct bpf_map *map;
+
+			aux = &env->insn_aux_data[i + delta];
+			map = env->used_maps[aux->map_index];
+			if (map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY)
+				goto next_insn;
+
+			prog->jit_required = true;
+
+			/*
+			 * We are *skipping* first half of ld_imm64 insn
+			 * with 'i++;', patching over second half of it
+			 * with that same half + mov64_percpu_reg insn.
+			 * All because bpf_patch_insn_data() can only
+			 * replace one 8-byte insn, which does not work
+			 * well for ld_imm64 insn.
+			 */
+
+			insn_buf[0] = insn[1];
+			insn_buf[1] = BPF_MOV64_PERCPU_REG(insn->dst_reg, insn->dst_reg);
+			cnt = 2;
+
+			i++;
+			new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, cnt);
+			if (!new_prog)
+				return -ENOMEM;
+
+			delta    += cnt - 1;
+			env->prog = prog = new_prog;
+			insn      = new_prog->insnsi + i + delta;
+			goto next_insn;
+		}
+
 		if (insn->code != (BPF_JMP | BPF_CALL))
 			goto next_insn;
 		if (insn->src_reg == BPF_PSEUDO_CALL)
kernel/bpf/verifier.c +15 -0
index e1244a721194..ae442ea217c4 100644
@@ -5615,6 +5615,8 @@ int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val,
 	u64 addr;
 	int err;
 
+	if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY || map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY)
+		return -EINVAL;
 	err = map->ops->map_direct_value_addr(map, &addr, off);
 	if (err)
 		return err;
@@ -6174,6 +6176,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct b
 			if (tnum_is_const(reg->var_off) &&
 			    bpf_map_is_rdonly(map) &&
 			    map->ops->map_direct_value_addr &&
+			    map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY &&
 			    map->map_type != BPF_MAP_TYPE_INSN_ARRAY) {
 				int map_off = off + reg->var_off.value;
 				u64 val = 0;
@@ -8176,6 +8179,12 @@ static int check_arg_const_str(struct bpf_verifier_env *env,
 		return -EACCES;
 	}
 
+	if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) {
+		verbose(env, "%s points to percpu_array map which cannot be used as const string\n",
+			reg_arg_name(env, argno));
+		return -EACCES;
+	}
+
 	if (!bpf_map_is_rdonly(map)) {
 		verbose(env, "%s does not point to a readonly map'\n", reg_arg_name(env, argno));
 		return -EACCES;
@@ -18279,6 +18288,12 @@ static int check_and_resolve_insns(struct bpf_verifier_env *env)
 					return -EINVAL;
 				}
 
+				if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY &&
+				    !env->prog->jit_requested) {
+					verbose(env, "JIT is required to use global percpu data\n");
+					return -EOPNOTSUPP;
+				}
+
 				err = map->ops->map_direct_value_addr(map, &addr, off);
 				if (err) {
 					verbose(env, "invalid access to map value pointer, value_size=%u off=%u\n",
Signature · 1 line
2.55.0
author
#9 lore raw

PATCH v10 3/9 libbpf: Probe percpu data feature

In reply to Leon Hwang

libbpf needs a reliable way to distinguish kernels that can support global percpu data from those that cannot.

Add a dedicated feature probe, so libbpf can make capability decisions early and fail predictably when global percpu data is unavailable.

---
 tools/lib/bpf/features.c        | 35 +++++++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf_internal.h |  2 ++
 2 files changed, 37 insertions(+)
tools/lib/bpf/features.c +35 -0
index b7e388f99d0b..ef9581c11303 100644
@@ -620,6 +620,38 @@ static int probe_bpf_syscall_common_attrs(int token_fd)
 	return probe_sys_bpf_ext();
 }
 
+static int probe_kern_percpu_data(int token_fd)
+{
+	struct bpf_insn insns[] = {
+		BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0),
+		BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0),
+		BPF_EXIT_INSN(),
+	};
+	LIBBPF_OPTS(bpf_map_create_opts, map_opts,
+		.token_fd = token_fd,
+		.map_flags = token_fd ? BPF_F_TOKEN_FD : 0,
+	);
+	LIBBPF_OPTS(bpf_prog_load_opts, prog_opts,
+		.token_fd = token_fd,
+		.prog_flags = token_fd ? BPF_F_TOKEN_FD : 0,
+	);
+	int ret, map, insn_cnt = ARRAY_SIZE(insns);
+
+	map = bpf_map_create(BPF_MAP_TYPE_PERCPU_ARRAY, "libbpf_percpu", sizeof(int), 8, 1,
+			     &map_opts);
+	if (map < 0) {
+		pr_warn("Error in %s(): %s. Couldn't create simple percpu_array map.\n",
+			__func__, errstr(map));
+		return map;
+	}
+
+	insns[0].imm = map;
+
+	ret = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, NULL, "GPL", insns, insn_cnt, &prog_opts);
+	close(map);
+	return probe_fd(ret);
+}
+
 typedef int (*feature_probe_fn)(int /* token_fd */);
 
 static struct kern_feature_cache feature_cache;
@@ -707,6 +739,9 @@ static struct kern_feature_desc {
 	[FEAT_BPF_SYSCALL_COMMON_ATTRS] = {
 		"BPF syscall common attributes support", probe_bpf_syscall_common_attrs,
 	},
+	[FEAT_PERCPU_DATA] = {
+		"kernel supports percpu data", probe_kern_percpu_data,
+	},
 };
 
 bool feat_supported(struct kern_feature_cache *cache, enum kern_feature_id feat_id)
tools/lib/bpf/libbpf_internal.h +2 -0
index d5b7db703b3f..b786f1a35567 100644
@@ -401,6 +401,8 @@ enum kern_feature_id {
 	FEAT_BTF_LAYOUT,
 	/* Kernel supports BPF syscall common attributes */
 	FEAT_BPF_SYSCALL_COMMON_ATTRS,
+	/* Kernel supports percpu data */
+	FEAT_PERCPU_DATA,
 	__FEAT_CNT,
 };
 
Signature · 1 line
2.55.0
author
#11 lore raw

PATCH v10 4/9 libbpf: Add support for global percpu data

In reply to Leon Hwang

Add support for global percpu data in libbpf by adding a new ".percpu" section, similar to ".data". It enables efficient handling of percpu global variables in bpf programs.

When generating loader for lightweight skeleton, update the percpu_array map used for global percpu data using BPF_F_ALL_CPUS, in order to update values across all CPUs using one value slot.

Unlike global data, the mmaped data for global percpu data will be marked as read-only after populating the percpu_array map. Thereafter, users can read those initialized percpu data after loading prog. If they want to update the percpu data after loading prog, they have to update the percpu_array map using key=0 instead.

---
 tools/lib/bpf/bpf_gen_internal.h |  3 +-
 tools/lib/bpf/gen_loader.c       |  3 +-
 tools/lib/bpf/libbpf.c           | 72 +++++++++++++++++++++++++++-----
 3 files changed, 65 insertions(+), 13 deletions(-)
tools/lib/bpf/bpf_gen_internal.h +2 -1
index 042569187752..6c5ad6c55e8a 100644
@@ -65,7 +65,8 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 			enum bpf_prog_type prog_type, const char *prog_name,
 			const char *license, struct bpf_insn *insns, size_t insn_cnt,
 			struct bpf_prog_load_opts *load_attr, int prog_idx);
-void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *value, __u32 value_size);
+void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *value, __u32 value_size,
+			      __u64 flags);
 void bpf_gen__map_freeze(struct bpf_gen *gen, int map_idx);
 void bpf_gen__record_attach_target(struct bpf_gen *gen, const char *name, enum bpf_attach_type type);
 void bpf_gen__record_extern(struct bpf_gen *gen, const char *name, bool is_weak,
tools/lib/bpf/gen_loader.c +2 -1
index 6e3dd5242761..af3a04f161ac 100644
@@ -1128,7 +1128,7 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 }
 
 void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue,
-			      __u32 value_size)
+			      __u32 value_size, __u64 flags)
 {
 	int attr_size = offsetofend(union bpf_attr, flags);
 	int map_update_attr, value, key;
@@ -1136,6 +1136,7 @@ void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue,
 	int zero = 0;
 
 	memset(&attr, 0, attr_size);
+	attr.flags = tgt_endian(flags);
 
 	value = add_data(gen, pvalue, value_size);
 	key = add_data(gen, &zero, sizeof(zero));
tools/lib/bpf/libbpf.c +61 -11

201 lines not on this page · raw message

Signature · 1 line
2.55.0
author
#18 lore raw

PATCH v10 5/9 bpftool: Generate skeleton for global percpu data

In reply to Leon Hwang

Enhance bpftool to generate skeletons that properly handle global percpu variables. The generated skeleton now includes a dedicated structure for percpu data, allowing users to initialize and access percpu variables more efficiently.

For global percpu variables, the skeleton now includes a nested structure, e.g.:

struct test_global_percpu_data {
	struct bpf_object_skeleton *skeleton;
	struct bpf_object *obj;
	struct {
		struct bpf_map *percpu;
	} maps;
	// ...
	struct test_global_percpu_data__percpu {
		int data;
		char run;
		struct {
			char set;
			int i;
			int nums[7];
		} struct_data;
		int nums[7];
	} *percpu;

	// ...
};

* The "struct test_global_percpu_data__percpu *percpu" points to

    initialized data, which is actually "maps.percpu->mmaped".
  * Before loading the skeleton, updating the
    "struct test_global_percpu_data__percpu *percpu" modifies the initial
    value of the corresponding global percpu variables.
  * After loading the skeleton, "maps.percpu->mmaped" has been marked as
    read-only in libbpf. If users want to update the global percpu
    variables, they have to update the "maps.percpu" map instead.
  * For lightweight skeleton, "lskel->percpu" will be protected by
    "mprotect(p, sz, PROT_READ)".

* For subskeleton, those variables of global percpu data will be skipped.

Assisted-by Codex:gpt-5.5-xhigh
---
 tools/bpf/bpftool/gen.c       | 76 +++++++++++++++++++++++++++--------
 tools/lib/bpf/skel_internal.h | 24 ++++++++++-
 2 files changed, 81 insertions(+), 19 deletions(-)
tools/bpf/bpftool/gen.c +59 -17
index a01d06d22d1a..b60db1108149 100644
@@ -101,6 +101,12 @@ static bool get_map_ident(const struct bpf_map *map, char *buf, size_t buf_sz)
 		return true;
 	}
 
+	if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY) {
+		snprintf(buf, buf_sz, "%s", name + 1);
+		sanitize_identifier(buf);
+		return true;
+	}
+
 	for  (i = 0, n = ARRAY_SIZE(sfxs); i < n; i++) {
 		const char *sfx = sfxs[i], *p;
 
@@ -117,7 +123,7 @@ static bool get_map_ident(const struct bpf_map *map, char *buf, size_t buf_sz)
 
 static bool get_datasec_ident(const char *sec_name, char *buf, size_t buf_sz)
 {
-	static const char *pfxs[] = { ".data", ".rodata", ".bss", ".kconfig" };
+	static const char *pfxs[] = { ".data", ".rodata", ".bss", ".percpu", ".kconfig" };
 	int i, n;
 
 	/* recognize hard coded LLVM section name */
@@ -254,7 +260,7 @@ static const struct btf_type *find_type_for_map(struct btf *btf, const char *map
 	return NULL;
 }
 
-static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)
+static bool is_skel_data(const struct bpf_map *map, char *buf, size_t sz)
 {
 	size_t tmp_sz;
 
@@ -263,13 +269,19 @@ static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)
 		return true;
 	}
 
-	if (!bpf_map__is_internal(map) || !(bpf_map__map_flags(map) & BPF_F_MMAPABLE))
+	if (!bpf_map__is_internal(map))
 		return false;
 
 	if (!get_map_ident(map, buf, sz))
 		return false;
 
-	return true;
+	if (bpf_map__map_flags(map) & BPF_F_MMAPABLE)
+		return true;
+
+	if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY)
+		return true;
+
+	return false;
 }
 
 static int codegen_datasecs(struct bpf_object *obj, const char *obj_name)
@@ -286,8 +298,11 @@ static int codegen_datasecs(struct bpf_object *obj, const char *obj_name)
 		return -errno;
 
 	bpf_object__for_each_map(map, obj) {
-		/* only generate definitions for memory-mapped internal maps */
-		if (!is_mmapable_map(map, map_ident, sizeof(map_ident)))
+		/*
+		 * Only generate definitions for internal maps that have
+		 * mmapped data.
+		 */
+		if (!is_skel_data(map, map_ident, sizeof(map_ident)))
 			continue;
 
 		sec = find_type_for_map(btf, map_ident);
@@ -339,8 +354,14 @@ static int codegen_subskel_datasecs(struct bpf_object *obj, const char *obj_name
 		return -errno;
 
 	bpf_object__for_each_map(map, obj) {
-		/* only generate definitions for memory-mapped internal maps */
-		if (!is_mmapable_map(map, map_ident, sizeof(map_ident)))
+		/*
+		 * Only generate definitions for internal maps that have
+		 * mmapped data.
+		 */
+		if (!is_skel_data(map, map_ident, sizeof(map_ident)))
+			continue;
+
+		if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY)
 			continue;
 
 		sec = find_type_for_map(btf, map_ident);
@@ -493,7 +514,10 @@ static size_t bpf_map_mmap_sz(const struct bpf_map *map)
 	return map_sz;
 }
 
-/* Emit type size asserts for all top-level fields in memory-mapped internal maps. */
+/*
+ * Emit type size asserts for all top-level fields in internal maps that
+ * have mmaped data.
+ */
 static void codegen_asserts(struct bpf_object *obj, const char *obj_name)
 {
 	struct btf *btf = bpf_object__btf(obj);
@@ -517,7 +541,7 @@ static void codegen_asserts(struct bpf_object *obj, const char *obj_name)
 		", obj_name);
 
 	bpf_object__for_each_map(map, obj) {
-		if (!is_mmapable_map(map, map_ident, sizeof(map_ident)))
+		if (!is_skel_data(map, map_ident, sizeof(map_ident)))
 			continue;
 
 		sec = find_type_for_map(btf, map_ident);
@@ -669,7 +693,8 @@ static void codegen_destroy(struct bpf_object *obj, const char *obj_name)
 		if (!get_map_ident(map, ident, sizeof(ident)))
 			continue;
 		if (bpf_map__is_internal(map) &&
-		    (bpf_map__map_flags(map) & BPF_F_MMAPABLE))
+		    ((bpf_map__map_flags(map) & BPF_F_MMAPABLE) ||
+		     bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY))
 			printf("\tskel_free_map_data(skel->%1$s, skel->maps.%1$s.initial_value, %2$zu);\n",
 			       ident, bpf_map_mmap_sz(map));
 		codegen("\
@@ -741,7 +766,7 @@ static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *h
 		const void *mmap_data = NULL;
 		size_t mmap_size = 0;
 
-		if (!is_mmapable_map(map, ident, sizeof(ident)))
+		if (!is_skel_data(map, ident, sizeof(ident)))
 			continue;
 
 		codegen("\
@@ -849,8 +874,22 @@ static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *h
 	bpf_object__for_each_map(map, obj) {
 		const char *mmap_flags;
 
-		if (!is_mmapable_map(map, ident, sizeof(ident)))
+		if (!is_skel_data(map, ident, sizeof(ident)))
+			continue;
+
+		if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY) {
+			codegen("\
+		\n\
+			err = skel_protect_map_data(skel->%1$s, &skel->maps.%1$s.initial_value, %2$zd);\n\
+			if (err)					    \n\
+				return err;				    \n\
+		#ifdef __KERNEL__					    \n\
+			skel->%1$s = NULL;				    \n\
+		#endif							    \n\
+			",
+			ident, bpf_map_mmap_sz(map));
 			continue;
+		}
 
 		if (bpf_map__map_flags(map) & BPF_F_RDONLY_PROG)
 			mmap_flags = "PROT_READ";
@@ -955,8 +994,7 @@ codegen_maps_skeleton(struct bpf_object *obj, size_t map_cnt, bool mmaped, bool
 				map->map = &obj->maps.%s;	    \n\
 			",
 			i, bpf_map__name(map), ident);
-		/* memory-mapped internal maps */
-		if (mmaped && is_mmapable_map(map, ident, sizeof(ident))) {
+		if (mmaped && is_skel_data(map, ident, sizeof(ident))) {
 			printf("\tmap->mmaped = (void **)&obj->%s;\n", ident);
 		}
 
@@ -1740,7 +1778,9 @@ static int do_subskeleton(int argc, char **argv)
 		/* Also count all maps that have a name */
 		map_cnt++;
 
-		if (!is_mmapable_map(map, ident, sizeof(ident)))
+		if (!is_skel_data(map, ident, sizeof(ident)))
+			continue;
+		if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY)
 			continue;
 
 		map_type_id = bpf_map__btf_value_type_id(map);
@@ -1863,7 +1903,9 @@ static int do_subskeleton(int argc, char **argv)
 
 	/* walk through each symbol and emit the runtime representation */
 	bpf_object__for_each_map(map, obj) {
-		if (!is_mmapable_map(map, ident, sizeof(ident)))
+		if (!is_skel_data(map, ident, sizeof(ident)))
+			continue;
+		if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY)
 			continue;
 
 		map_type_id = bpf_map__btf_value_type_id(map);
tools/lib/bpf/skel_internal.h +22 -2
index 53fee53d36d5..1f3f332dffbe 100644
@@ -131,8 +131,10 @@ static inline void skel_free_map_data(void *p, __u64 addr, size_t sz)
 {
 	if (addr != ~0ULL)
 		kvfree(p);
-	/* When addr == ~0ULL the 'p' points to
-	 * ((struct bpf_array *)map)->value. See skel_finalize_map_data.
+	/*
+	 * When addr == ~0ULL the init buffer has already been released.
+	 * For skel_finalize_map_data(), 'p' points to
+	 * ((struct bpf_array *)map)->value.
 	 */
 }
 
@@ -170,6 +172,15 @@ static inline void *skel_finalize_map_data(__u64 *init_val, size_t mmap_sz, int
 	return addr;
 }
 
+static inline int skel_protect_map_data(void *p, __u64 *init_val, size_t sz)
+{
+	(void)sz;
+
+	kvfree(p);
+	*init_val = ~0ULL;
+	return 0;
+}
+
 #else
 
 static inline void *skel_alloc(size_t size)
@@ -208,6 +219,15 @@ static inline void *skel_finalize_map_data(__u64 *init_val, size_t mmap_sz, int
 		return NULL;
 	return addr;
 }
+
+static inline int skel_protect_map_data(void *p, __u64 *init_val, size_t sz)
+{
+	(void)init_val;
+
+	if (mprotect(p, sz, PROT_READ))
+		return -errno;
+	return 0;
+}
 #endif
 
 static inline int skel_closenz(int fd)
Signature · 1 line
2.55.0
author
#21 lore raw

PATCH v10 6/9 selftests/bpf: Add tests to verify global percpu data

In reply to Leon Hwang

If the arch, like s390x, does not support percpu insn, these cases won't test global percpu data by checking FEAT_PERCPU_DATA support.

The following APIs have been tested for global percpu data:

1. bpf_map__set_initial_value()
2. bpf_map__initial_value()
3. generated percpu struct pointer pointing to internal map's mmaped data
4. bpf_map__lookup_elem() for global percpu data map

At the same time, the case is also tested with 'bpftool gen skeleton -L'.

Add a test to verify that the live vars of subskel won't include the vars for global percpu data.

Assisted-by Codex:gpt-5.5-xhigh
---
 tools/testing/selftests/bpf/Makefile          |   2 +-
 .../bpf/prog_tests/global_data_init.c         | 151 ++++++++++++++++++
 .../bpf/prog_tests/global_percpu_subskel.c    |  37 +++++
 .../bpf/progs/test_global_percpu_data.c       |  33 ++++
 4 files changed, 222 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_global_percpu_data.c
tools/testing/selftests/bpf/Makefile +1 -1
index b642ee489ea6..c37ed9e7b97c 100644
@@ -533,7 +533,7 @@ LSKELS_SIGNED := fentry_test.c fexit_test.c atomics.c
 
 # Generate both light skeleton and libbpf skeleton for these
 LSKELS_EXTRA := test_ksyms_module.c test_ksyms_weak.c kfunc_call_test.c \
-	kfunc_call_test_subprog.c
+	kfunc_call_test_subprog.c test_global_percpu_data.c
 SKEL_BLACKLIST += $$(LSKELS) $$(LSKELS_SIGNED)
 
 test_static_linked.skel.h-deps := test_static_linked1.bpf.o test_static_linked2.bpf.o
tools/testing/selftests/bpf/prog_tests/global_data_init.c +151 -0

159 lines not on this page · raw message

tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c new file +37 -0
new file mode 100644
index 000000000000..8aebd533d86b
@@ -0,0 +1,37 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include "test_global_percpu_data.subskel.h"
+
+void test_global_percpu_data_subskel(void)
+{
+	struct test_global_percpu_data *subskel = NULL;
+	struct bpf_object *obj;
+	int i;
+
+	obj = bpf_object__open_file("./test_global_percpu_data.bpf.o", NULL);
+	if (!ASSERT_OK_PTR(obj, "bpf_object__open_file"))
+		return;
+
+	subskel = test_global_percpu_data__open(obj);
+	if (!ASSERT_OK_PTR(subskel, "test_global_percpu_data__open"))
+		goto out;
+
+	if (!ASSERT_OK_PTR(subskel->subskel, "subskel"))
+		goto out;
+	if (!ASSERT_OK_PTR(subskel->maps.percpu, "maps.percpu"))
+		goto out;
+	ASSERT_EQ(bpf_map__type(subskel->maps.percpu), BPF_MAP_TYPE_PERCPU_ARRAY,
+		  "percpu_map_type");
+	ASSERT_GT(subskel->subskel->var_cnt, 0, "var_cnt");
+
+	for (i = 0; i < subskel->subskel->var_cnt; i++) {
+		const struct bpf_var_skeleton *var;
+
+		var = (void *) subskel->subskel->vars + i * subskel->subskel->var_skel_sz;
+		ASSERT_NEQ(var->map, &subskel->maps.percpu, "var");
+	}
+
+out:
+	test_global_percpu_data__destroy(subskel);
+	bpf_object__close(obj);
+}
tools/testing/selftests/bpf/progs/test_global_percpu_data.c new file +33 -0
new file mode 100644
index 000000000000..54380dfb11a5
@@ -0,0 +1,33 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+int unused SEC(".percpu.looooooooong");
+int data2 SEC(".percpu.data");
+int data SEC(".percpu") = -1;
+int nums[7] SEC(".percpu");
+char run SEC(".percpu") = 0;
+struct {
+	char set;
+	int i;
+	int nums[7];
+} struct_data SEC(".percpu") = {
+	.set = 0,
+	.i = -1,
+};
+
+SEC("raw_tp/task_rename")
+__auxiliary
+int update_percpu_data(void *ctx)
+{
+	struct_data.nums[6] = 0xc0de;
+	struct_data.set = 1;
+	struct_data.i = 1;
+	nums[6] = 0xc0de;
+	data = 1;
+	run = 1;
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
Signature · 1 line
2.55.0
author
#28 lore raw

PATCH v10 7/9 selftests/bpf: Test direct reading/writing read-only percpu_array map

In reply to Leon Hwang

Verify these two cases:

1. Direct reading the data of read-only percpu data's percpu_array map is allowed.
2. Direct writing the data of read-only percpu data's percpu_array map is disallowed.

Assisted-by Codex:gpt-5.5-xhigh
---
 .../bpf/prog_tests/global_data_init.c         | 83 +++++++++++++++++++
 1 file changed, 83 insertions(+)
tools/testing/selftests/bpf/prog_tests/global_data_init.c +83 -0
index f7a56e3fdbce..d4405646847a 100644
@@ -199,6 +199,85 @@ static void test_global_percpu_data_lskel(void)
 	test_global_percpu_data_lskel__destroy(lskel);
 }
 
+static int create_rdonly_percpu_array(void)
+{
+	LIBBPF_OPTS(bpf_map_create_opts, map_opts,
+		    .map_flags = BPF_F_RDONLY_PROG,
+	);
+	int key = 0, map_fd, err;
+	__u64 value = 0;
+
+	map_fd = bpf_map_create(BPF_MAP_TYPE_PERCPU_ARRAY, "percpu_ro_map", sizeof(int),
+				sizeof(__u64), 1, &map_opts);
+	if (!ASSERT_GE(map_fd, 0, "bpf_map_create"))
+		return -1;
+
+	err = bpf_map_update_elem(map_fd, &key, &value, BPF_F_ALL_CPUS);
+	if (!ASSERT_OK(err, "bpf_map_update_elem"))
+		goto out;
+
+	err = bpf_map_freeze(map_fd);
+	if (!ASSERT_OK(err, "bpf_map_freeze"))
+		goto out;
+
+	return map_fd;
+
+out:
+	close(map_fd);
+	return -1;
+}
+
+static void test_global_percpu_data_rdonly_direct_read(void)
+{
+	struct bpf_insn insns[] = {
+		BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0),
+		BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0),
+		BPF_EXIT_INSN(),
+	};
+	int map_fd, prog_fd;
+
+	map_fd = create_rdonly_percpu_array();
+	if (map_fd < 0)
+		return;
+
+	insns[0].imm = map_fd;
+	prog_fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, "percpu_ro_prog", "GPL", insns,
+				ARRAY_SIZE(insns), NULL);
+	if (ASSERT_GE(prog_fd, 0, "bpf_prog_load"))
+		close(prog_fd);
+	close(map_fd);
+}
+
+static void test_global_percpu_data_rdonly_direct_write(void)
+{
+	LIBBPF_OPTS(bpf_prog_load_opts, prog_opts);
+	struct bpf_insn insns[] = {
+		BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0),
+		BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0),
+		BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0),
+		BPF_EXIT_INSN(),
+	};
+	char log_buf[256] = {};
+	int map_fd, prog_fd;
+
+	prog_opts.log_buf = log_buf;
+	prog_opts.log_size = sizeof(log_buf);
+	prog_opts.log_level = 1;
+
+	map_fd = create_rdonly_percpu_array();
+	if (map_fd < 0)
+		return;
+
+	insns[0].imm = map_fd;
+	prog_fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, "percpu_ro_prog", "GPL", insns,
+				ARRAY_SIZE(insns), &prog_opts);
+	if (!ASSERT_LT(prog_fd, 0, "bpf_prog_load"))
+		close(prog_fd);
+	else
+		ASSERT_HAS_SUBSTR(log_buf, "write into map forbidden", "verifier log");
+	close(map_fd);
+}
+
 void test_global_percpu_data(void)
 {
 	if (!feat_supported(NULL, FEAT_PERCPU_DATA)) {
@@ -210,4 +289,8 @@ void test_global_percpu_data(void)
 		test_global_percpu_data_init();
 	if (test__start_subtest("lskel"))
 		test_global_percpu_data_lskel();
+	if (test__start_subtest("rdonly_direct_read"))
+		test_global_percpu_data_rdonly_direct_read();
+	if (test__start_subtest("rdonly_direct_write"))
+		test_global_percpu_data_rdonly_direct_write();
 }
Signature · 1 line
2.55.0
author
#31 lore raw

PATCH v10 8/9 selftests/bpf: Test verifier log for global percpu data

In reply to Leon Hwang

Add two tests to verify the verifier log
"R%d points to percpu_array map which cannot be used as const string\n".

---
 .../bpf/prog_tests/global_data_init.c         |  6 +++++
 .../bpf/progs/test_global_percpu_data.c       | 23 +++++++++++++++++++
 2 files changed, 29 insertions(+)
tools/testing/selftests/bpf/prog_tests/global_data_init.c +6 -0
index d4405646847a..598b7dde6581 100644
@@ -278,6 +278,11 @@ static void test_global_percpu_data_rdonly_direct_write(void)
 	close(map_fd);
 }
 
+static void test_global_percpu_data_verifier_log(void)
+{
+	RUN_TESTS(test_global_percpu_data);
+}
+
 void test_global_percpu_data(void)
 {
 	if (!feat_supported(NULL, FEAT_PERCPU_DATA)) {
@@ -293,4 +298,5 @@ void test_global_percpu_data(void)
 		test_global_percpu_data_rdonly_direct_read();
 	if (test__start_subtest("rdonly_direct_write"))
 		test_global_percpu_data_rdonly_direct_write();
+	test_global_percpu_data_verifier_log();
 }
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +23 -0
index 54380dfb11a5..b852a0b07508 100644
@@ -30,4 +30,27 @@ int update_percpu_data(void *ctx)
 	return 0;
 }
 
+static const char fmt[] SEC(".percpu.fmt") = "data %d\n";
+
+SEC("?kprobe")
+__failure __msg("R{{[0-9]+}} points to percpu_array map which cannot be used as const string")
+int verifier_strncmp(void *ctx)
+{
+	return bpf_strncmp("test", 5, fmt);
+}
+
+SEC("?kprobe")
+__failure __msg("R{{[0-9]+}} points to percpu_array map which cannot be used as const string")
+int verifier_snprintf(void *ctx)
+{
+	u64 args[] = { data };
+	char buf[128];
+	int len;
+
+	len = bpf_snprintf(buf, sizeof(buf), fmt, args, sizeof(args));
+	if (len > 0)
+		bpf_printk("snprintf: %s\n", buf);
+	return 0;
+}
+
 char _license[] SEC("license") = "GPL";
Signature · 1 line
2.55.0
author
#33 lore raw

PATCH v10 9/9 selftests/bpf: Verify bpf_iter for global percpu data

In reply to Leon Hwang

Add a test to verify that it is OK to iter the percpu_array map used for global percpu data.

---
 .../bpf/prog_tests/global_data_init.c         | 52 +++++++++++++++++++
 .../bpf/progs/test_global_percpu_data.c       | 25 +++++++++
 2 files changed, 77 insertions(+)
tools/testing/selftests/bpf/prog_tests/global_data_init.c +52 -0
index 598b7dde6581..e5a40ea36ee7 100644
@@ -283,6 +283,56 @@ static void test_global_percpu_data_verifier_log(void)
 	RUN_TESTS(test_global_percpu_data);
 }
 
+static void test_global_percpu_data_iter(void)
+{
+	DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);
+	struct test_global_percpu_data *skel;
+	union bpf_iter_link_info linfo = {};
+	struct bpf_link *link = NULL;
+	int fd, num_cpus, len, err;
+	char buf[16];
+
+	num_cpus = libbpf_num_possible_cpus();
+	if (!ASSERT_GT(num_cpus, 0, "libbpf_num_possible_cpus"))
+		return;
+
+	skel = test_global_percpu_data__open();
+	if (!ASSERT_OK_PTR(skel, "test_global_percpu_data__open"))
+		return;
+
+	skel->rodata->num_cpus = num_cpus;
+	skel->rodata->offsetof_num = offsetof(struct test_global_percpu_data__percpu, struct_data);
+	skel->rodata->offsetof_num += sizeof(skel->percpu->struct_data) - sizeof(int);
+	skel->rodata->elem_sz = roundup(sizeof(struct test_global_percpu_data__percpu), 8);
+	skel->percpu->struct_data.nums[6] = 0xc0de;
+
+	err = test_global_percpu_data__load(skel);
+	if (!ASSERT_OK(err, "test_global_percpu_data__load"))
+		goto out;
+
+	linfo.map.map_fd = bpf_map__fd(skel->maps.percpu);
+	opts.link_info = &linfo;
+	opts.link_info_len = sizeof(linfo);
+	link = bpf_program__attach_iter(skel->progs.dump_percpu_data, &opts);
+	if (!ASSERT_OK_PTR(link, "bpf_program__attach_iter"))
+		goto out;
+
+	fd = bpf_iter_create(bpf_link__fd(link));
+	if (!ASSERT_GE(fd, 0, "bpf_iter_create"))
+		goto out;
+
+	while ((len = read(fd, buf, sizeof(buf))) > 0)
+		do { } while (0);
+	ASSERT_EQ(len, 0, "read iter");
+	ASSERT_TRUE(skel->bss->run_iter, "run_iter");
+	ASSERT_EQ(skel->bss->percpu_data_sum, 0xc0de * num_cpus, "percpu_data_sum");
+
+	close(fd);
+out:
+	bpf_link__destroy(link);
+	test_global_percpu_data__destroy(skel);
+}
+
 void test_global_percpu_data(void)
 {
 	if (!feat_supported(NULL, FEAT_PERCPU_DATA)) {
@@ -299,4 +349,6 @@ void test_global_percpu_data(void)
 	if (test__start_subtest("rdonly_direct_write"))
 		test_global_percpu_data_rdonly_direct_write();
 	test_global_percpu_data_verifier_log();
+	if (test__start_subtest("iter"))
+		test_global_percpu_data_iter();
 }
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +25 -0
index b852a0b07508..8b1f625017d1 100644
@@ -53,4 +53,29 @@ int verifier_snprintf(void *ctx)
 	return 0;
 }
 
+volatile const __u32 num_cpus = 0;
+volatile const int offsetof_num;
+volatile const int elem_sz;
+__u32 percpu_data_sum = 0;
+bool run_iter = false;
+
+SEC("iter/bpf_map_elem")
+__auxiliary
+int dump_percpu_data(struct bpf_iter__bpf_map_elem *ctx)
+{
+	void *pptr = ctx->value;
+	int i;
+
+	if (!pptr)
+		return 0;
+
+	run_iter = true;
+
+	for (i = 0; i < num_cpus; i++) {
+		percpu_data_sum += *(int *) (pptr + offsetof_num);
+		pptr += elem_sz;
+	}
+	return 0;
+}
+
 char _license[] SEC("license") = "GPL";
Signature · 1 line
2.55.0
#22 lore raw

Re: [PATCH bpf-next v10 6/9] selftests/bpf: Add tests to verify global percpu data

In reply to Leon Hwang
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The `test_global_percpu_data_subskel` selftest fails unconditionally because it asserts `var_cnt…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The test_global_percpu_data_subskel selftest fails unconditionally because it asserts var_cnt > 0 on a subskeleton that is generated with zero variables. --

--- Patch [6]: [PATCH bpf-next v10 6/9] selftests/bpf: Add tests to verify global percpu data --- commit c5a32c0630bc8d9b3aedfc264c75289a1f8e6d98
Author: Leon Hwang <[email protected]>

selftests/bpf: Add tests to verify global percpu data

This commit adds selftests to verify operations on global percpu data in BPF programs. It tests various libbpf APIs and skeleton behaviors for percpu maps. It also adds a test to ensure live variables of a subskeleton do not include global percpu data.

Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c new file +0 -0
new file mode 100644
index 0000000000000..8aebd533d86b5
@@ -0,0 +1,37 @@

[ ... ]

tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c line 25

Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c new file +5 -0
+	if (!ASSERT_OK_PTR(subskel->maps.percpu, "maps.percpu"))
+		goto out;
+	ASSERT_EQ(bpf_map__type(subskel->maps.percpu), BPF_MAP_TYPE_PERCPU_ARRAY,
+		  "percpu_map_type");
+	ASSERT_GT(subskel->subskel->var_cnt, 0, "var_cnt");

[Severity: Medium]
Since test_global_percpu_data.c only contains variables in .percpu* sections, and the subskeleton generator skips percpu array variables, will var_cnt always be exactly 0 here, causing this test to unconditionally fail?

Would it make sense to either add a standard global variable to test_global_percpu_data.c (e.g. in .bss or .data), or adjust this assertion?

Quoted 29 lines
tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c new file +12 -0
+
+	for (i = 0; i < subskel->subskel->var_cnt; i++) {
+		const struct bpf_var_skeleton *var;
+
+		var = (void *) subskel->subskel->vars + i * subskel->subskel->var_skel_sz;
+		ASSERT_NEQ(var->map, &subskel->maps.percpu, "var");
+	}
+
+out:
+	test_global_percpu_data__destroy(subskel);
+	bpf_object__close(obj);
+}
tools/testing/selftests/bpf/progs/test_global_percpu_data.c new file +10 -0
new file mode 100644
index 0000000000000..54380dfb11a5a
@@ -0,0 +1,33 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+int unused SEC(".percpu.looooooooong");
+int data2 SEC(".percpu.data");
+int data SEC(".percpu") = -1;
+int nums[7] SEC(".percpu");
+char run SEC(".percpu") = 0;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
#12 lore raw

Re: [PATCH bpf-next v10 4/9] libbpf: Add support for global percpu data

In reply to Leon Hwang
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] BPF skeleton generation completely omits the `.percpu` map, breaking the promised ergonomic…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] BPF skeleton generation completely omits the .percpu map, breaking the promised ergonomic access. --

--- Patch [4]: [PATCH bpf-next v10 4/9] libbpf: Add support for global percpu data ---
(Note: The format of this review is altered to use a free-form summary rather than inline patch quoting due to automated recitation restrictions.)

commit c26518c258a32645d11a26c0d0ce50242231d6a6
Author: Leon Hwang <[email protected]>
libbpf: Add support for global percpu data

This commit adds support for a new ".percpu" section in libbpf, enabling efficient handling of global percpu variables. It handles the section by creating a percpu_array map and populating it at load time.

[Severity: Medium]
The commit message mentions that users can read initialized percpu data after loading the program. Does the current implementation fully support BPF skeleton generation for this new section?

In tools/lib/bpf/libbpf.c:map_is_mmapable(), the new code explicitly returns false when map->libbpf_type == LIBBPF_MAP_PERCPU.

Because PERCPU_ARRAY maps cannot be memory mapped, this prevents the skeleton generator from emitting the typed datasec pointer (such as skel->percpu).

Additionally, does bpftool need a concurrent update to fully support this? Looking at get_map_ident() in tools/bpf/bpftool/gen.c, the sfxs[] array only contains ".data", ".rodata", ".bss", and ".kconfig".

Without adding ".percpu" to the bpftool suffix list, it appears the skeleton generator will entirely omit the map from the generated header. If neither skel->percpu nor skel->maps.percpu is emitted, will users be forced to fall back to untyped manual lookups like bpf_object__find_map_by_name() instead of having the intended ergonomic skeleton access?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
#24 lore raw

Re: [PATCH bpf-next v10 6/9] selftests/bpf: Add tests to verify global percpu data

In reply to Leon Hwang
This isn't a bug, but item 4 lists bpf_map__lookup_elem(), while the added test reads the map through the fd-based bpf_map_lookup_elem_flags():
Quoted 12 lines

selftests/bpf: Add tests to verify global percpu data

If the arch, like s390x, does not support percpu insn, these cases won't test global percpu data by checking FEAT_PERCPU_DATA support.

The following APIs have been tested for global percpu data:

1. bpf_map__set_initial_value()
2. bpf_map__initial_value()
3. generated percpu struct pointer pointing to internal map's mmaped data
4. bpf_map__lookup_elem() for global percpu data map

Quoted 7 lines

At the same time, the case is also tested with 'bpftool gen skeleton -L'.

Add a test to verify that the live vars of subskel won't include the vars for global percpu data.

Assisted-by Codex:gpt-5.5-xhigh

This isn't a bug, but item 4 lists bpf_map__lookup_elem(), while the added test reads the map through the fd-based bpf_map_lookup_elem_flags():

    err = bpf_map_lookup_elem_flags(map_fd, &key, &data, flags);

Should the changelog name the API that is actually exercised?

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/29428917849

author
#13 lore raw

Re: [PATCH bpf-next v10 4/9] libbpf: Add support for global percpu data

In reply to [email protected]
Quoted 15 lines · On 16/7/26 00:00, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] BPF skeleton generation completely omits the .percpu map, breaking the promised ergonomic access. --

--- Patch [4]: [PATCH bpf-next v10 4/9] libbpf: Add support for global percpu data ---
(Note: The format of this review is altered to use a free-form summary rather than inline patch quoting due to automated recitation restrictions.)

commit c26518c258a32645d11a26c0d0ce50242231d6a6
Author: Leon Hwang <[email protected]>
libbpf: Add support for global percpu data

This commit adds support for a new ".percpu" section in libbpf, enabling efficient handling of global percpu variables. It handles the section by creating a percpu_array map and populating it at load time.

Quoted 7 lines

[Severity: Medium]
The commit message mentions that users can read initialized percpu data after loading the program. Does the current implementation fully support BPF skeleton generation for this new section?

In tools/lib/bpf/libbpf.c:map_is_mmapable(), the new code explicitly returns false when map->libbpf_type == LIBBPF_MAP_PERCPU.

map_is_mmapable() returns false for the internal percpu_array map to avoid 'def->map_flags |= BPF_F_MMAPABLE'. But, it does not prevent allocating mmaped memory for the internal percpu_array map.

In patch #6, the generated skeleton with this new section has been tested well.

Quoted 9 lines

Because PERCPU_ARRAY maps cannot be memory mapped, this prevents the skeleton generator from emitting the typed datasec pointer (such as skel->percpu).

Additionally, does bpftool need a concurrent update to fully support this? Looking at get_map_ident() in tools/bpf/bpftool/gen.c, the sfxs[] array only contains ".data", ".rodata", ".bss", and ".kconfig".

Quoted 6 lines

Without adding ".percpu" to the bpftool suffix list, it appears the skeleton generator will entirely omit the map from the generated header. If neither skel->percpu nor skel->maps.percpu is emitted, will users be forced to fall back to untyped manual lookups like bpf_object__find_map_by_name() instead of having the intended ergonomic skeleton access?

Hmm, Sashiko missed the map type check in get_map_ident() in patch #5.

Adding ".percpu" to the sfxs[] array is not robust, because ".percpu.data" could be translated to "data" identifier instead of "percpu_data".

With the type check, ".percpu.data" will be translated to "percpu_data" correctly.

Thanks,
Leon

author
#23 lore raw

Re: [PATCH bpf-next v10 6/9] selftests/bpf: Add tests to verify global percpu data

In reply to [email protected]
Quoted 35 lines · On 15/7/26 23:55, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The test_global_percpu_data_subskel selftest fails unconditionally because it asserts var_cnt > 0 on a subskeleton that is generated with zero variables. --

--- Patch [6]: [PATCH bpf-next v10 6/9] selftests/bpf: Add tests to verify global percpu data --- commit c5a32c0630bc8d9b3aedfc264c75289a1f8e6d98
Author: Leon Hwang <[email protected]>

selftests/bpf: Add tests to verify global percpu data

This commit adds selftests to verify operations on global percpu data in BPF programs. It tests various libbpf APIs and skeleton behaviors for percpu maps. It also adds a test to ensure live variables of a subskeleton do not include global percpu data.

Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c new file +0 -0
new file mode 100644
index 0000000000000..8aebd533d86b5
@@ -0,0 +1,37 @@

[ ... ]

tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c line 25

Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c new file +5 -0
+	if (!ASSERT_OK_PTR(subskel->maps.percpu, "maps.percpu"))
+		goto out;
+	ASSERT_EQ(bpf_map__type(subskel->maps.percpu), BPF_MAP_TYPE_PERCPU_ARRAY,
+		  "percpu_map_type");
+	ASSERT_GT(subskel->subskel->var_cnt, 0, "var_cnt");

[Severity: Medium]
Since test_global_percpu_data.c only contains variables in .percpu* sections, and the subskeleton generator skips percpu array variables, will var_cnt always be exactly 0 here, causing this test to unconditionally fail?

Quoted 2 lines

Would it make sense to either add a standard global variable to test_global_percpu_data.c (e.g. in .bss or .data), or adjust this assertion?

My bad to miss a new 'int data3 SEC(".data.percpu")' in test_global_percpu_data.c in this revision, which would mute this review.

Thanks,
Leon

Quoted 1 line

[...]

author
#25 lore raw

Re: [PATCH bpf-next v10 6/9] selftests/bpf: Add tests to verify global percpu data

In reply to [email protected]
Quoted 20 lines · On 16/7/26 00:11, [email protected] wrote:
Quoted 12 lines

selftests/bpf: Add tests to verify global percpu data

If the arch, like s390x, does not support percpu insn, these cases won't test global percpu data by checking FEAT_PERCPU_DATA support.

The following APIs have been tested for global percpu data:

1. bpf_map__set_initial_value()
2. bpf_map__initial_value()
3. generated percpu struct pointer pointing to internal map's mmaped data
4. bpf_map__lookup_elem() for global percpu data map

Quoted 7 lines

At the same time, the case is also tested with 'bpftool gen skeleton -L'.

Add a test to verify that the live vars of subskel won't include the vars for global percpu data.

Assisted-by Codex:gpt-5.5-xhigh
Quoted 6 lines

This isn't a bug, but item 4 lists bpf_map__lookup_elem(), while the added test reads the map through the fd-based bpf_map_lookup_elem_flags():

    err = bpf_map_lookup_elem_flags(map_fd, &key, &data, flags);

Should the changelog name the API that is actually exercised?

Good catch.

Actually, there's no bpf_map__lookup_elem() callsite in this test.

Will update the test to exercise both bpf_map__lookup_elem() and bpf_map_lookup_elem_flags(). And update the commit message meanwhile.

Thanks,
Leon

Quoted 1 line

[...]

reviewer Reviewed-by on v10 1/9
#2 lore raw

Re: [PATCH bpf-next v10 1/9] bpf: Drop duplicate blank lines in verifier

In reply to Leon Hwang
Quoted 6 lines · Leon Hwang

There are many adjacent blank lines in the verifier that have accumulated over time.

Drop them for cleanup.

No functional changes intended.

What about the files we factored out of the verifier? E.g., backtrack.c also has the same issue.

Quoted 108 lines
---
 kernel/bpf/verifier.c | 14 --------------
 1 file changed, 14 deletions(-)
kernel/bpf/verifier.c +0 -14
index de816063ae63..e1244a721194 100644
@@ -634,7 +634,6 @@ static void __mark_dynptr_reg(struct bpf_reg_state *reg,
 			      enum bpf_dynptr_type type,
 			      bool first_slot, int id, int parent_id);
 
-
 static void mark_dynptr_stack_regs(struct bpf_verifier_env *env,
 				   struct bpf_reg_state *sreg1,
 				   struct bpf_reg_state *sreg2,
@@ -1671,7 +1670,6 @@ static bool same_callsites(struct bpf_verifier_state *a, struct bpf_verifier_sta
 	return true;
 }
 
-
 void bpf_free_backedges(struct bpf_scc_visit *visit)
 {
 	struct bpf_scc_backedge *backedge, *next;
@@ -2291,7 +2289,6 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env,
 	return &elem->st;
 }
 
-
 static int cmp_subprogs(const void *a, const void *b)
 {
 	return ((struct bpf_subprog_info *)a)->start -
@@ -4044,7 +4041,6 @@ static int check_stack_read(struct bpf_verifier_env *env,
 	return err;
 }
 
-
 /* check_stack_write dispatches to check_stack_write_fixed_off or
  * check_stack_write_var_off.
  *
@@ -4840,7 +4836,6 @@ static int check_sock_access(struct bpf_verifier_env *env, int insn_idx,
 		valid = false;
 	}
 
-
 	if (valid) {
 		env->insn_aux_data[insn_idx].ctx_field_size =
 			info.ctx_field_size;
@@ -6683,7 +6678,6 @@ static int check_stack_range_initialized(
 	if (err)
 		return err;
 
-
 	if (tnum_is_const(reg->var_off)) {
 		min_off = max_off = reg->var_off.value + off;
 	} else {
@@ -7404,7 +7398,6 @@ static bool is_iter_new_kfunc(struct bpf_call_arg_meta *meta)
 	return meta->kfunc_flags & KF_ITER_NEW;
 }
 
-
 static bool is_iter_destroy_kfunc(struct bpf_call_arg_meta *meta)
 {
 	return meta->kfunc_flags & KF_ITER_DESTROY;
@@ -11539,7 +11532,6 @@ static int process_irq_flag(struct bpf_verifier_env *env, struct bpf_reg_state *
 	return 0;
 }
 
-
 static int ref_set_non_owning(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
 {
 	struct btf_record *rec = reg_btf_record(reg);
@@ -16421,7 +16413,6 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn)
 	return 0;
 }
 
-
 static bool return_retval_range(struct bpf_verifier_env *env, struct bpf_retval_range *range)
 {
 	enum bpf_prog_type prog_type = resolve_prog_type(env->prog);
@@ -18363,8 +18354,6 @@ static void release_insn_arrays(struct bpf_verifier_env *env)
 		bpf_insn_array_release(env->insn_array_maps[i]);
 }
 
-
-
 /* The verifier does more data flow analysis than llvm and will not
  * explore branches that are dead at run time. Malicious programs can
  * have dead code too. Therefore replace all dead at-run-time code
@@ -18392,8 +18381,6 @@ static void sanitize_dead_code(struct bpf_verifier_env *env)
 	}
 }
 
-
-
 static void free_states(struct bpf_verifier_env *env)
 {
 	struct bpf_verifier_state_list *sl;
@@ -18675,7 +18662,6 @@ static int do_check_main(struct bpf_verifier_env *env)
 	return ret;
 }
 
-
 static void print_verification_stats(struct bpf_verifier_env *env)
 {
 	/* Skip over hidden subprogs which are not verified. */
author
#3 lore raw

Re: [PATCH bpf-next v10 1/9] bpf: Drop duplicate blank lines in verifier

Quoted 10 lines · Emil Tsalapatis
Quoted 6 lines · Leon Hwang

There are many adjacent blank lines in the verifier that have accumulated over time.

Drop them for cleanup.

No functional changes intended.

Thanks for your review.

Quoted 3 lines

What about the files we factored out of the verifier? E.g., backtrack.c also has the same issue.

Probably, I can post a patch to clean all adjacent blank lines in kernel/bpf/ when posting another series.

Thanks,
Leon

Quoted 1 line

[...]

#6 lore raw
In reply to Leon Hwang
Quoted 45 lines · Leon Hwang

Introduce global percpu data, inspired by the commit 6316f78306c1 ("Merge branch 'support-global-data'"). It enables the definition of global percpu variables in BPF, similar to the include/linux/percpu-defs.h::DEFINE_PER_CPU() macro.

For example, in BPF, it is able to define a global percpu variable like:

int data SEC(".percpu");

With this patch, tools like retsnoop [1] and bpfsnoop [2] can simplify their BPF code for handling LBRs. The code can be updated from

static struct perf_branch_entry lbrs[1][MAX_LBR_ENTRIES] SEC(".data.lbrs");

to

static struct perf_branch_entry lbrs[MAX_LBR_ENTRIES] SEC(".percpu.lbrs");

This eliminates the need to retrieve the CPU ID using the bpf_get_smp_processor_id() helper.

Additionally, by reusing global percpu data map, sharing information between tail callers and callees or freplace callers and callees becomes simpler compared to reusing percpu_array maps.

Links:
[1] https://github.com/anakryiko/retsnoop
[2] https://github.com/bpfsnoop/bpfsnoop

---
 kernel/bpf/arraymap.c   | 38 ++++++++++++++++++++++++++++++++++++--
 kernel/bpf/const_fold.c |  1 -
 kernel/bpf/fixups.c     | 38 ++++++++++++++++++++++++++++++++++++++
 kernel/bpf/verifier.c   | 15 +++++++++++++++
 4 files changed, 89 insertions(+), 3 deletions(-)
kernel/bpf/arraymap.c +0 -0
index 248b4818178c..c4e9430941e5 100644
@@ -259,6 +259,37 @@ static void *percpu_array_map_lookup_elem(struct bpf_map *map, void *key)
 	return this_cpu_ptr(array->pptrs[index & array->index_mask]);
 }
 

kernel/bpf/arraymap.c static void *percpu_array_map_lookup_elem(struct bpf_map *map, void *key) · line 271

Quoted 10 lines
kernel/bpf/arraymap.c +10 -0
+static int percpu_array_map_direct_value_addr(const struct bpf_map *map, u64 *imm, u32 off)
+{
+	struct bpf_array *array = container_of(map, struct bpf_array, map);
+
+	if (map->max_entries != 1)
+		return -EOPNOTSUPP;
+	if (off >= map->value_size)
+		return -EINVAL;
+	if (!bpf_jit_supports_percpu_insn())
+		return -EOPNOTSUPP;

Nit: This should be first, if the JIT doesn't support the instruction all else is moot. Same below.

Quoted 121 lines
kernel/bpf/arraymap.c +26 -2
+
+	*imm = (u64)(__force unsigned long) array->pptrs[0];
+	return 0;
+}
+
+static int percpu_array_map_direct_value_meta(const struct bpf_map *map, u64 imm, u32 *off)
+{
+	struct bpf_array *array = container_of(map, struct bpf_array, map);
+	u64 base = (u64)(__force unsigned long) array->pptrs[0];
+
+	if (map->max_entries != 1)
+		return -EOPNOTSUPP;
+	if (imm < base || imm >= base + array->elem_size)
+		return -ENOENT;
+	if (!bpf_jit_supports_percpu_insn())
+		return -EOPNOTSUPP;
+
+	*off = imm - base;
+	return 0;
+}
+
 /* emit BPF instructions equivalent to C code of percpu_array_map_lookup_elem() */
 static int percpu_array_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf)
 {
@@ -551,9 +582,10 @@ static int array_map_check_btf(struct bpf_map *map,
 			       const struct btf_type *key_type,
 			       const struct btf_type *value_type)
 {
-	/* One exception for keyless BTF: .bss/.data/.rodata map */
+	/* One exception for keyless BTF: .bss/.data/.rodata/.percpu map */
 	if (btf_type_is_void(key_type)) {
-		if (map->map_type != BPF_MAP_TYPE_ARRAY ||
+		if ((map->map_type != BPF_MAP_TYPE_ARRAY &&
+		     map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) ||
 		    map->max_entries != 1)
 			return -EINVAL;
 
@@ -832,6 +864,8 @@ const struct bpf_map_ops percpu_array_map_ops = {
 	.map_get_next_key = bpf_array_get_next_key,
 	.map_lookup_elem = percpu_array_map_lookup_elem,
 	.map_gen_lookup = percpu_array_map_gen_lookup,
+	.map_direct_value_addr = percpu_array_map_direct_value_addr,
+	.map_direct_value_meta = percpu_array_map_direct_value_meta,
 	.map_update_elem = array_map_update_elem,
 	.map_delete_elem = array_map_delete_elem,
 	.map_lookup_percpu_elem = percpu_array_map_lookup_percpu_elem,
kernel/bpf/const_fold.c +0 -1
index b2a19acadb91..5787246bef30 100644
@@ -182,7 +182,6 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info *
 		u64 val = 0;
 
 		if (!bpf_map_is_rdonly(map) || !map->ops->map_direct_value_addr ||
-		    map->map_type == BPF_MAP_TYPE_INSN_ARRAY ||
 		    off < 0 || off + size > map->value_size ||
 		    bpf_map_direct_read(map, off, size, &val, is_ldsx)) {
 			*dst = unknown;
kernel/bpf/fixups.c +38 -0
index d3be972714b2..02642bba73e5 100644
@@ -1818,6 +1818,44 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
 			goto next_insn;
 		}
 
+		if (env->prog->jit_requested &&
+		    bpf_jit_supports_percpu_insn() &&
+		    insn->code == (BPF_LD | BPF_IMM | BPF_DW) &&
+		    (insn->src_reg == BPF_PSEUDO_MAP_VALUE ||
+		     insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE)) {
+			struct bpf_map *map;
+
+			aux = &env->insn_aux_data[i + delta];
+			map = env->used_maps[aux->map_index];
+			if (map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY)
+				goto next_insn;
+
+			prog->jit_required = true;
+
+			/*
+			 * We are *skipping* first half of ld_imm64 insn
+			 * with 'i++;', patching over second half of it
+			 * with that same half + mov64_percpu_reg insn.
+			 * All because bpf_patch_insn_data() can only
+			 * replace one 8-byte insn, which does not work
+			 * well for ld_imm64 insn.
+			 */
+
+			insn_buf[0] = insn[1];
+			insn_buf[1] = BPF_MOV64_PERCPU_REG(insn->dst_reg, insn->dst_reg);
+			cnt = 2;
+
+			i++;
+			new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, cnt);
+			if (!new_prog)
+				return -ENOMEM;
+
+			delta    += cnt - 1;
+			env->prog = prog = new_prog;
+			insn      = new_prog->insnsi + i + delta;
+			goto next_insn;
+		}
+
 		if (insn->code != (BPF_JMP | BPF_CALL))
 			goto next_insn;
 		if (insn->src_reg == BPF_PSEUDO_CALL)
kernel/bpf/verifier.c +2 -0
index e1244a721194..ae442ea217c4 100644
@@ -5615,6 +5615,8 @@ int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val,
 	u64 addr;
 	int err;
 
+	if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY || map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY)
+		return -EINVAL;
 	err = map->ops->map_direct_value_addr(map, &addr, off);
 	if (err)
 		return err;

kernel/bpf/verifier.c static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct b · line 6180

Quoted 5 lines
kernel/bpf/verifier.c +1 -0
@@ -6174,6 +6176,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct b
 			if (tnum_is_const(reg->var_off) &&
 			    bpf_map_is_rdonly(map) &&
 			    map->ops->map_direct_value_addr &&
+			    map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY &&
 			    map->map_type != BPF_MAP_TYPE_INSN_ARRAY) {

An issue I have with the patch is that it takes the special-casing we already have for INSN_ARRAY and makes it worse. The instruction arrays implement direct_value_addr, but it's not actually valid to call direct_value_addr from a bunch of places where it's called, so we have to special case check the map type to prevent it. Case in point the special casing in check_arg_const_str in the thunk below.

Imo we should consider making a different map method different from direct_value_addr that INSN_ARRAY and PERCPU_ARRAY implement. That way we remove most of the map type checks in those paths. I'm not exactly sure how that would look like, but I could take a look at it tomorrow.

Quoted 16 lines
kernel/bpf/verifier.c +6 -0
 				int map_off = off + reg->var_off.value;
 				u64 val = 0;
@@ -8176,6 +8179,12 @@ static int check_arg_const_str(struct bpf_verifier_env *env,
 		return -EACCES;
 	}
 
+	if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) {
+		verbose(env, "%s points to percpu_array map which cannot be used as const string\n",
+			reg_arg_name(env, argno));
+		return -EACCES;
+	}
+
 	if (!bpf_map_is_rdonly(map)) {
 		verbose(env, "%s does not point to a readonly map'\n", reg_arg_name(env, argno));
 		return -EACCES;

kernel/bpf/verifier.c static int check_and_resolve_insns(struct bpf_verifier_env *env) · line 18296

Quoted 9 lines
kernel/bpf/verifier.c +6 -0
@@ -18279,6 +18288,12 @@ static int check_and_resolve_insns(struct bpf_verifier_env *env)
 					return -EINVAL;
 				}
 
+				if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY &&
+				    !env->prog->jit_requested) {
+					verbose(env, "JIT is required to use global percpu data\n");
+					return -EOPNOTSUPP;
+				}
+

Another place where we add map-specific checks where there are none.

Quoted 3 lines
kernel/bpf/verifier.c +0 -0
 				err = map->ops->map_direct_value_addr(map, &addr, off);
 				if (err) {
 					verbose(env, "invalid access to map value pointer, value_size=%u off=%u\n",
#14 lore raw

Re: [PATCH bpf-next v10 4/9] libbpf: Add support for global percpu data

In reply to Leon Hwang
Quoted 94 lines · Leon Hwang

Add support for global percpu data in libbpf by adding a new ".percpu" section, similar to ".data". It enables efficient handling of percpu global variables in bpf programs.

When generating loader for lightweight skeleton, update the percpu_array map used for global percpu data using BPF_F_ALL_CPUS, in order to update values across all CPUs using one value slot.

Unlike global data, the mmaped data for global percpu data will be marked as read-only after populating the percpu_array map. Thereafter, users can read those initialized percpu data after loading prog. If they want to update the percpu data after loading prog, they have to update the percpu_array map using key=0 instead.

---
 tools/lib/bpf/bpf_gen_internal.h |  3 +-
 tools/lib/bpf/gen_loader.c       |  3 +-
 tools/lib/bpf/libbpf.c           | 72 +++++++++++++++++++++++++++-----
 3 files changed, 65 insertions(+), 13 deletions(-)
tools/lib/bpf/bpf_gen_internal.h +2 -1
index 042569187752..6c5ad6c55e8a 100644
@@ -65,7 +65,8 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 			enum bpf_prog_type prog_type, const char *prog_name,
 			const char *license, struct bpf_insn *insns, size_t insn_cnt,
 			struct bpf_prog_load_opts *load_attr, int prog_idx);
-void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *value, __u32 value_size);
+void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *value, __u32 value_size,
+			      __u64 flags);
 void bpf_gen__map_freeze(struct bpf_gen *gen, int map_idx);
 void bpf_gen__record_attach_target(struct bpf_gen *gen, const char *name, enum bpf_attach_type type);
 void bpf_gen__record_extern(struct bpf_gen *gen, const char *name, bool is_weak,
tools/lib/bpf/gen_loader.c +2 -1
index 6e3dd5242761..af3a04f161ac 100644
@@ -1128,7 +1128,7 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 }
 
 void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue,
-			      __u32 value_size)
+			      __u32 value_size, __u64 flags)
 {
 	int attr_size = offsetofend(union bpf_attr, flags);
 	int map_update_attr, value, key;
@@ -1136,6 +1136,7 @@ void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue,
 	int zero = 0;
 
 	memset(&attr, 0, attr_size);
+	attr.flags = tgt_endian(flags);
 
 	value = add_data(gen, pvalue, value_size);
 	key = add_data(gen, &zero, sizeof(zero));
tools/lib/bpf/libbpf.c +5 -0
index f88b7c8de304..e4104d18521b 100644
@@ -541,6 +541,7 @@ struct bpf_struct_ops {
 };
 
 #define DATA_SEC ".data"
+#define PERCPU_SEC ".percpu"
 #define BSS_SEC ".bss"
 #define RODATA_SEC ".rodata"
 #define KCONFIG_SEC ".kconfig"
@@ -555,6 +556,7 @@ enum libbpf_map_type {
 	LIBBPF_MAP_BSS,
 	LIBBPF_MAP_RODATA,
 	LIBBPF_MAP_KCONFIG,
+	LIBBPF_MAP_PERCPU,
 };
 
 struct bpf_map_def {
@@ -666,6 +668,7 @@ enum sec_type {
 	SEC_DATA,
 	SEC_RODATA,
 	SEC_ST_OPS,
+	SEC_PERCPU,
 };
 
 struct elf_sec_desc {
@@ -1839,6 +1842,8 @@ static size_t bpf_map_mmap_sz(const struct bpf_map *map)
 	switch (map->def.type) {
 	case BPF_MAP_TYPE_ARRAY:
 		return array_map_mmap_sz(map->def.value_size, map->def.max_entries);
+	case BPF_MAP_TYPE_PERCPU_ARRAY:
+		return map->def.value_size;
 	case BPF_MAP_TYPE_ARENA:
 		return page_sz * map->def.max_entries;
 	default:

tools/lib/bpf/libbpf.c static int bpf_map_mmap_resize(struct bpf_map *map, size_t old_sz, size_t new_sz · line 1875

Quoted 6 lines
tools/lib/bpf/libbpf.c +2 -1
@@ -1866,7 +1871,8 @@ static int bpf_map_mmap_resize(struct bpf_map *map, size_t old_sz, size_t new_sz
 	return 0;
 }
 
-static char *internal_map_name(struct bpf_object *obj, const char *real_name)
+static char *internal_map_name(struct bpf_object *obj, const char *real_name,
+			       enum libbpf_map_type type)

We can avoid passing the type here by testing against ".percpu" since that's how we are deriving the map type in the first place. But more importantly:

Quoted 4 lines
tools/lib/bpf/libbpf.c +0 -0
 {
 	char map_name[BPF_OBJ_NAME_LEN], *p;
 	int pfx_len, sfx_len = max((size_t)7, strlen(real_name));

tools/lib/bpf/libbpf.c static char *internal_map_name(struct bpf_object *obj, const char *real_name) · line 1920

Quoted 10 lines
tools/lib/bpf/libbpf.c +5 -2
@@ -1907,8 +1913,11 @@ static char *internal_map_name(struct bpf_object *obj, const char *real_name)
 	if (sfx_len >= BPF_OBJ_NAME_LEN)
 		sfx_len = BPF_OBJ_NAME_LEN - 1;
 
-	/* if there are two or more dots in map name, it's a custom dot map */
-	if (strchr(real_name + 1, '.') != NULL)
+	/*
+	 * Don't prefix the bpf_object name if this is a custom dot map
+	 * (containing two or more dots) or a percpu data map.
+	 */
+	if (strchr(real_name + 1, '.') != NULL || type == LIBBPF_MAP_PERCPU)

Is there a reason we don't use the exact same logic as the other internal maps here? I understand that a lot of the conventions around the naming are there for legacy reasons, but it seems like we're singling out the .percpu section for highly nonbvious reasons. Imo we should consider doing the same prefixing for a bare ".percpu" section that we do for the other internal ones. At the very least, there needs to be some explanation as to why .percpu gets special treatment.

@Andrii Wdyt?

tools/lib/bpf/libbpf.c static bool map_is_mmapable(struct bpf_object *obj, struct bpf_map *map) · line 1951

Quoted 10 lines
tools/lib/bpf/libbpf.c +1 -1
 		pfx_len = 0;
 	else
 		pfx_len = min((size_t)BPF_OBJ_NAME_LEN - sfx_len - 1, strlen(obj->name));
@@ -1938,7 +1947,7 @@ static bool map_is_mmapable(struct bpf_object *obj, struct bpf_map *map)
 	struct btf_var_secinfo *vsi;
 	int i, n;
 
-	if (!map->btf_value_type_id)
+	if (!map->btf_value_type_id || map->libbpf_type == LIBBPF_MAP_PERCPU)
 		return false;

Nit: These are two separate checks rolled into one, and each one checks a different thing. THe type check against MAP_PERCPU merits a comment as
well: It's the only internal section that is not really mappable because there's no way to represent it as userspace state.

Quoted 93 lines
tools/lib/bpf/libbpf.c +22 -5
 	t = btf__type_by_id(obj->btf, map->btf_value_type_id);
@@ -1962,6 +1971,7 @@ static int
 bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
 			      const char *real_name, int sec_idx, void *data, size_t data_sz)
 {
+	bool is_percpu = type == LIBBPF_MAP_PERCPU;
 	struct bpf_map_def *def;
 	struct bpf_map *map;
 	size_t mmap_sz;
@@ -1975,7 +1985,7 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
 	map->sec_idx = sec_idx;
 	map->sec_offset = 0;
 	map->real_name = strdup(real_name);
-	map->name = internal_map_name(obj, real_name);
+	map->name = internal_map_name(obj, real_name, type);
 	if (!map->real_name || !map->name) {
 		zfree(&map->real_name);
 		zfree(&map->name);
@@ -1983,7 +1993,7 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
 	}
 
 	def = &map->def;
-	def->type = BPF_MAP_TYPE_ARRAY;
+	def->type = is_percpu ? BPF_MAP_TYPE_PERCPU_ARRAY : BPF_MAP_TYPE_ARRAY;
 	def->key_size = sizeof(int);
 	def->value_size = data_sz;
 	def->max_entries = 1;
@@ -1996,8 +2006,9 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
 	if (map_is_mmapable(obj, map))
 		def->map_flags |= BPF_F_MMAPABLE;
 
-	pr_debug("map '%s' (global data): at sec_idx %d, offset %zu, flags %x.\n",
-		 map->name, map->sec_idx, map->sec_offset, def->map_flags);
+	pr_debug("map '%s' (global %sdata): at sec_idx %d, offset %zu, flags %x.\n",
+		 map->name, is_percpu ? "percpu " : "", map->sec_idx,
+		 map->sec_offset, def->map_flags);
 
 	mmap_sz = bpf_map_mmap_sz(map);
 	map->mmaped = mmap(NULL, mmap_sz, PROT_READ | PROT_WRITE,
@@ -2057,6 +2068,13 @@ static int bpf_object__init_global_data_maps(struct bpf_object *obj)
 							    NULL,
 							    sec_desc->data->d_size);
 			break;
+		case SEC_PERCPU:
+			sec_name = elf_sec_name(obj, elf_sec_by_idx(obj, sec_idx));
+			err = bpf_object__init_internal_map(obj, LIBBPF_MAP_PERCPU,
+							    sec_name, sec_idx,
+							    sec_desc->data->d_buf,
+							    sec_desc->data->d_size);
+			break;
 		default:
 			/* skip */
 			break;
@@ -4016,6 +4034,11 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
 				sec_desc->sec_type = SEC_RODATA;
 				sec_desc->shdr = sh;
 				sec_desc->data = data;
+			} else if (strcmp(name, PERCPU_SEC) == 0 ||
+				   str_has_pfx(name, PERCPU_SEC ".")) {
+				sec_desc->sec_type = SEC_PERCPU;
+				sec_desc->shdr = sh;
+				sec_desc->data = data;
 			} else if (strcmp(name, STRUCT_OPS_SEC) == 0 ||
 				   strcmp(name, STRUCT_OPS_LINK_SEC) == 0 ||
 				   strcmp(name, "?" STRUCT_OPS_SEC) == 0 ||
@@ -4544,6 +4567,7 @@ static bool bpf_object__shndx_is_data(const struct bpf_object *obj,
 	case SEC_BSS:
 	case SEC_DATA:
 	case SEC_RODATA:
+	case SEC_PERCPU:
 		return true;
 	default:
 		return false;
@@ -4569,6 +4593,8 @@ bpf_object__section_to_libbpf_map_type(const struct bpf_object *obj, int shndx)
 		return LIBBPF_MAP_DATA;
 	case SEC_RODATA:
 		return LIBBPF_MAP_RODATA;
+	case SEC_PERCPU:
+		return LIBBPF_MAP_PERCPU;
 	default:
 		return LIBBPF_MAP_UNSPEC;
 	}
@@ -4944,7 +4970,7 @@ static int map_fill_btf_type_info(struct bpf_object *obj, struct bpf_map *map)
 
 	/*
 	 * LLVM annotates global data differently in BTF, that is,
-	 * only as '.data', '.bss' or '.rodata'.
+	 * only as '.data', '.bss', '.percpu' or '.rodata'.
 	 */
 	if (!bpf_map__is_internal(map))
 		return -ENOENT;

tools/lib/bpf/libbpf.c static int · line 5322

Quoted 4 lines
tools/lib/bpf/libbpf.c +1 -0
@@ -5293,18 +5319,30 @@ static int
 bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map)
 {
 	enum libbpf_map_type map_type = map->libbpf_type;
+	bool is_percpu = map_type == LIBBPF_MAP_PERCPU;

Nit: If we do

	__u64 update_flags = is_percpu ? BPF_F_ALL_CPUS : 0;

we can declare the variable as const and ...

tools/lib/bpf/libbpf.c static int · line 5335

Quoted 13 lines
tools/lib/bpf/libbpf.c +10 -0
+	__u64 update_flags = 0;
 	int err, zero = 0;
 	size_t mmap_sz;
 
+	if (is_percpu) {
+		if (!obj->gen_loader && !kernel_supports(obj, FEAT_PERCPU_DATA)) {
+			pr_warn("map '%s': kernel does not support percpu data.\n",
+				bpf_map__name(map));
+			return -EOPNOTSUPP;
+		}
+
+		update_flags = BPF_F_ALL_CPUS;
+	}

... we can collapse the above into a single nested level:

	if (is_percpu && !obj->gen_loader && !kernel_supports(obj, FEAT_PERCPU_DATA)) {
		...
	}
Quoted 33 lines
tools/lib/bpf/libbpf.c +10 -2
+
 	if (obj->gen_loader) {
 		bpf_gen__map_update_elem(obj->gen_loader, map - obj->maps,
-					 map->mmaped, map->def.value_size);
+					 map->mmaped, map->def.value_size, update_flags);
 		if (map_type == LIBBPF_MAP_RODATA || map_type == LIBBPF_MAP_KCONFIG)
 			bpf_gen__map_freeze(obj->gen_loader, map - obj->maps);
 		return 0;
 	}
 
-	err = bpf_map_update_elem(map->fd, &zero, map->mmaped, 0);
+	err = bpf_map_update_elem(map->fd, &zero, map->mmaped, update_flags);
 	if (err) {
 		err = -errno;
 		pr_warn("map '%s': failed to set initial contents: %s\n",
@@ -5349,6 +5387,13 @@ bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map)
 			return err;
 		}
 		map->mmaped = mmaped;
+	} else if (is_percpu) {
+		if (mprotect(map->mmaped, mmap_sz, PROT_READ)) {
+			err = -errno;
+			pr_warn("map '%s': failed to mprotect() contents: %s\n",
+				bpf_map__name(map), errstr(err));
+			return err;
+		}
 	} else if (map->mmaped) {
 		munmap(map->mmaped, mmap_sz);
 		map->mmaped = NULL;
@@ -10807,11 +10852,16 @@ static bool map_uses_real_name(const struct bpf_map *map)
 	 * such map's corresponding ELF section name as a map name.
 	 * This check distinguishes .data/.rodata from .data.* and .rodata.*
 	 * maps to know which name has to be returned to the user.

tools/lib/bpf/libbpf.c static bool map_uses_real_name(const struct bpf_map *map) · line 10864

Quoted 10 lines
tools/lib/bpf/libbpf.c +5 -0
+	 * Map name of the custom .percpu.* maps might be truncated to
+	 * BPF_OBJ_NAME_LEN-1 chars in internal_map_name(). Hence, percpu data
+	 * maps must use real name for their user-visible name.
 	 */
 	if (map->libbpf_type == LIBBPF_MAP_DATA && strcmp(map->real_name, DATA_SEC) != 0)
 		return true;
 	if (map->libbpf_type == LIBBPF_MAP_RODATA && strcmp(map->real_name, RODATA_SEC) != 0)
 		return true;
+	if (map->libbpf_type == LIBBPF_MAP_PERCPU)
+		return true;

Same comment as above here wrt uniformity. This is the part that requires us to check agianst the map type in __init_internal_map().

Quoted 3 lines
tools/lib/bpf/libbpf.c +0 -0
 	return false;
 }
 
#19 lore raw

Re: [PATCH bpf-next v10 5/9] bpftool: Generate skeleton for global percpu data

In reply to Leon Hwang
Quoted 77 lines · Leon Hwang

Enhance bpftool to generate skeletons that properly handle global percpu variables. The generated skeleton now includes a dedicated structure for percpu data, allowing users to initialize and access percpu variables more efficiently.

For global percpu variables, the skeleton now includes a nested structure, e.g.:

struct test_global_percpu_data {
	struct bpf_object_skeleton *skeleton;
	struct bpf_object *obj;
	struct {
		struct bpf_map *percpu;
	} maps;
	// ...
	struct test_global_percpu_data__percpu {
		int data;
		char run;
		struct {
			char set;
			int i;
			int nums[7];
		} struct_data;
		int nums[7];
	} *percpu;

	// ...
};

* The "struct test_global_percpu_data__percpu *percpu" points to

    initialized data, which is actually "maps.percpu->mmaped".
  * Before loading the skeleton, updating the
    "struct test_global_percpu_data__percpu *percpu" modifies the initial
    value of the corresponding global percpu variables.
  * After loading the skeleton, "maps.percpu->mmaped" has been marked as
    read-only in libbpf. If users want to update the global percpu
    variables, they have to update the "maps.percpu" map instead.
  * For lightweight skeleton, "lskel->percpu" will be protected by
    "mprotect(p, sz, PROT_READ)".

* For subskeleton, those variables of global percpu data will be skipped.

Assisted-by Codex:gpt-5.5-xhigh
---
 tools/bpf/bpftool/gen.c       | 76 +++++++++++++++++++++++++++--------
 tools/lib/bpf/skel_internal.h | 24 ++++++++++-
 2 files changed, 81 insertions(+), 19 deletions(-)
tools/bpf/bpftool/gen.c +7 -1
index a01d06d22d1a..b60db1108149 100644
@@ -101,6 +101,12 @@ static bool get_map_ident(const struct bpf_map *map, char *buf, size_t buf_sz)
 		return true;
 	}
 
+	if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY) {
+		snprintf(buf, buf_sz, "%s", name + 1);
+		sanitize_identifier(buf);
+		return true;
+	}
+
 	for  (i = 0, n = ARRAY_SIZE(sfxs); i < n; i++) {
 		const char *sfx = sfxs[i], *p;
 
@@ -117,7 +123,7 @@ static bool get_map_ident(const struct bpf_map *map, char *buf, size_t buf_sz)
 
 static bool get_datasec_ident(const char *sec_name, char *buf, size_t buf_sz)
 {
-	static const char *pfxs[] = { ".data", ".rodata", ".bss", ".kconfig" };
+	static const char *pfxs[] = { ".data", ".rodata", ".bss", ".percpu", ".kconfig" };
 	int i, n;
 
 	/* recognize hard coded LLVM section name */

tools/bpf/bpftool/gen.c static const struct btf_type *find_type_for_map(struct btf *btf, const char *map · line 263

Quoted 5 lines
tools/bpf/bpftool/gen.c +1 -1
@@ -254,7 +260,7 @@ static const struct btf_type *find_type_for_map(struct btf *btf, const char *map
 	return NULL;
 }
 
-static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)
+static bool is_skel_data(const struct bpf_map *map, char *buf, size_t sz)

Here we change the function to take PERCPU_ARRAY into account, but then immediately turn around and add checks of the form is_skel_data && type != BPF_MAP_TYPE_PERCPU_ARRAY. If we keep both the existing mmapable_map and add a separate is_skel_data map (possibly using is_mappable_map), we can remove those additional checks.

Quoted 44 lines
tools/bpf/bpftool/gen.c +13 -5
 {
 	size_t tmp_sz;
 
@@ -263,13 +269,19 @@ static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)
 		return true;
 	}
 
-	if (!bpf_map__is_internal(map) || !(bpf_map__map_flags(map) & BPF_F_MMAPABLE))
+	if (!bpf_map__is_internal(map))
 		return false;
 
 	if (!get_map_ident(map, buf, sz))
 		return false;
 
-	return true;
+	if (bpf_map__map_flags(map) & BPF_F_MMAPABLE)
+		return true;
+
+	if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY)
+		return true;
+
+	return false;
 }
 
 static int codegen_datasecs(struct bpf_object *obj, const char *obj_name)
@@ -286,8 +298,11 @@ static int codegen_datasecs(struct bpf_object *obj, const char *obj_name)
 		return -errno;
 
 	bpf_object__for_each_map(map, obj) {
-		/* only generate definitions for memory-mapped internal maps */
-		if (!is_mmapable_map(map, map_ident, sizeof(map_ident)))
+		/*
+		 * Only generate definitions for internal maps that have
+		 * mmapped data.
+		 */
+		if (!is_skel_data(map, map_ident, sizeof(map_ident)))
 			continue;
 
 		sec = find_type_for_map(btf, map_ident);
@@ -339,8 +354,14 @@ static int codegen_subskel_datasecs(struct bpf_object *obj, const char *obj_name
 		return -errno;
 
 	bpf_object__for_each_map(map, obj) {
-		/* only generate definitions for memory-mapped internal maps */

tools/bpf/bpftool/gen.c static int codegen_subskel_datasecs(struct bpf_object *obj, const char *obj_name · line 365

Quoted 10 lines
tools/bpf/bpftool/gen.c +8 -1
-		if (!is_mmapable_map(map, map_ident, sizeof(map_ident)))
+		/*
+		 * Only generate definitions for internal maps that have
+		 * mmapped data.
+		 */
+		if (!is_skel_data(map, map_ident, sizeof(map_ident)))
+			continue;
+
+		if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY)
 			continue;

Case in point, the old is_mmapable_map would work fine here.

tools/bpf/bpftool/gen.c static size_t bpf_map_mmap_sz(const struct bpf_map *map) · line 519

Quoted 10 lines
tools/bpf/bpftool/gen.c +3 -1
 		sec = find_type_for_map(btf, map_ident);
@@ -493,7 +514,10 @@ static size_t bpf_map_mmap_sz(const struct bpf_map *map)
 	return map_sz;
 }
 
-/* Emit type size asserts for all top-level fields in memory-mapped internal maps. */
+/*
+ * Emit type size asserts for all top-level fields in internal maps that
+ * have mmaped data.

I do not think the new comment makes it any clearer, the difference is "memory-mapped internal maps" and "maps that are memory-mapped initially but actually inaccessible after loading because there's no way to represent them as a mapping of the skeleton" AFAICT which is not easy to infer currently.

Quoted 14 lines
tools/bpf/bpftool/gen.c +2 -1
+ */
 static void codegen_asserts(struct bpf_object *obj, const char *obj_name)
 {
 	struct btf *btf = bpf_object__btf(obj);
@@ -517,7 +541,7 @@ static void codegen_asserts(struct bpf_object *obj, const char *obj_name)
 		", obj_name);
 
 	bpf_object__for_each_map(map, obj) {
-		if (!is_mmapable_map(map, map_ident, sizeof(map_ident)))
+		if (!is_skel_data(map, map_ident, sizeof(map_ident)))
 			continue;
 
 		sec = find_type_for_map(btf, map_ident);

tools/bpf/bpftool/gen.c static void codegen_destroy(struct bpf_object *obj, const char *obj_name) · line 697

Quoted 6 lines
tools/bpf/bpftool/gen.c +2 -1
@@ -669,7 +693,8 @@ static void codegen_destroy(struct bpf_object *obj, const char *obj_name)
 		if (!get_map_ident(map, ident, sizeof(ident)))
 			continue;
 		if (bpf_map__is_internal(map) &&
-		    (bpf_map__map_flags(map) & BPF_F_MMAPABLE))
+		    ((bpf_map__map_flags(map) & BPF_F_MMAPABLE) ||
+		     bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY))

I think this gives us a hint on how we could structure is_skel_data above, we now have a special calss that is "mappable maps + PERCPU_ARRAY whose initial representation in userspace is mappable, but the loaded map isn't".

Quoted 47 lines
tools/bpf/bpftool/gen.c +17 -4
 			printf("\tskel_free_map_data(skel->%1$s, skel->maps.%1$s.initial_value, %2$zu);\n",
 			       ident, bpf_map_mmap_sz(map));
 		codegen("\
@@ -741,7 +766,7 @@ static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *h
 		const void *mmap_data = NULL;
 		size_t mmap_size = 0;
 
-		if (!is_mmapable_map(map, ident, sizeof(ident)))
+		if (!is_skel_data(map, ident, sizeof(ident)))
 			continue;
 
 		codegen("\
@@ -849,8 +874,22 @@ static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *h
 	bpf_object__for_each_map(map, obj) {
 		const char *mmap_flags;
 
-		if (!is_mmapable_map(map, ident, sizeof(ident)))
+		if (!is_skel_data(map, ident, sizeof(ident)))
+			continue;
+
+		if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY) {
+			codegen("\
+		\n\
+			err = skel_protect_map_data(skel->%1$s, &skel->maps.%1$s.initial_value, %2$zd);\n\
+			if (err)					    \n\
+				return err;				    \n\
+		#ifdef __KERNEL__					    \n\
+			skel->%1$s = NULL;				    \n\
+		#endif							    \n\
+			",
+			ident, bpf_map_mmap_sz(map));
 			continue;
+		}
 
 		if (bpf_map__map_flags(map) & BPF_F_RDONLY_PROG)
 			mmap_flags = "PROT_READ";
@@ -955,8 +994,7 @@ codegen_maps_skeleton(struct bpf_object *obj, size_t map_cnt, bool mmaped, bool
 				map->map = &obj->maps.%s;	    \n\
 			",
 			i, bpf_map__name(map), ident);
-		/* memory-mapped internal maps */
-		if (mmaped && is_mmapable_map(map, ident, sizeof(ident))) {
+		if (mmaped && is_skel_data(map, ident, sizeof(ident))) {
 			printf("\tmap->mmaped = (void **)&obj->%s;\n", ident);
 		}
 

tools/bpf/bpftool/gen.c static int do_subskeleton(int argc, char **argv) · line 1784

Quoted 8 lines
tools/bpf/bpftool/gen.c +3 -1
@@ -1740,7 +1778,9 @@ static int do_subskeleton(int argc, char **argv)
 		/* Also count all maps that have a name */
 		map_cnt++;
 
-		if (!is_mmapable_map(map, ident, sizeof(ident)))
+		if (!is_skel_data(map, ident, sizeof(ident)))
+			continue;
+		if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY)
 			continue;

Another instance of what I was talking about.

Quoted 62 lines
tools/bpf/bpftool/gen.c +3 -1
 		map_type_id = bpf_map__btf_value_type_id(map);
@@ -1863,7 +1903,9 @@ static int do_subskeleton(int argc, char **argv)
 
 	/* walk through each symbol and emit the runtime representation */
 	bpf_object__for_each_map(map, obj) {
-		if (!is_mmapable_map(map, ident, sizeof(ident)))
+		if (!is_skel_data(map, ident, sizeof(ident)))
+			continue;
+		if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY)
 			continue;
 
 		map_type_id = bpf_map__btf_value_type_id(map);
tools/lib/bpf/skel_internal.h +22 -2
index 53fee53d36d5..1f3f332dffbe 100644
@@ -131,8 +131,10 @@ static inline void skel_free_map_data(void *p, __u64 addr, size_t sz)
 {
 	if (addr != ~0ULL)
 		kvfree(p);
-	/* When addr == ~0ULL the 'p' points to
-	 * ((struct bpf_array *)map)->value. See skel_finalize_map_data.
+	/*
+	 * When addr == ~0ULL the init buffer has already been released.
+	 * For skel_finalize_map_data(), 'p' points to
+	 * ((struct bpf_array *)map)->value.
 	 */
 }
 
@@ -170,6 +172,15 @@ static inline void *skel_finalize_map_data(__u64 *init_val, size_t mmap_sz, int
 	return addr;
 }
 
+static inline int skel_protect_map_data(void *p, __u64 *init_val, size_t sz)
+{
+	(void)sz;
+
+	kvfree(p);
+	*init_val = ~0ULL;
+	return 0;
+}
+
 #else
 
 static inline void *skel_alloc(size_t size)
@@ -208,6 +219,15 @@ static inline void *skel_finalize_map_data(__u64 *init_val, size_t mmap_sz, int
 		return NULL;
 	return addr;
 }
+
+static inline int skel_protect_map_data(void *p, __u64 *init_val, size_t sz)
+{
+	(void)init_val;
+
+	if (mprotect(p, sz, PROT_READ))
+		return -errno;
+	return 0;
+}
 #endif
 
 static inline int skel_closenz(int fd)
reviewer Reviewed-by on v10 3/9
#10 lore raw

Re: [PATCH bpf-next v10 3/9] libbpf: Probe percpu data feature

In reply to Leon Hwang
Quoted 8 lines · Leon Hwang

libbpf needs a reliable way to distinguish kernels that can support global percpu data from those that cannot.

Add a dedicated feature probe, so libbpf can make capability decisions early and fail predictably when global percpu data is unavailable.

Quoted 71 lines
---
 tools/lib/bpf/features.c        | 35 +++++++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf_internal.h |  2 ++
 2 files changed, 37 insertions(+)
tools/lib/bpf/features.c +35 -0
index b7e388f99d0b..ef9581c11303 100644
@@ -620,6 +620,38 @@ static int probe_bpf_syscall_common_attrs(int token_fd)
 	return probe_sys_bpf_ext();
 }
 
+static int probe_kern_percpu_data(int token_fd)
+{
+	struct bpf_insn insns[] = {
+		BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0),
+		BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0),
+		BPF_EXIT_INSN(),
+	};
+	LIBBPF_OPTS(bpf_map_create_opts, map_opts,
+		.token_fd = token_fd,
+		.map_flags = token_fd ? BPF_F_TOKEN_FD : 0,
+	);
+	LIBBPF_OPTS(bpf_prog_load_opts, prog_opts,
+		.token_fd = token_fd,
+		.prog_flags = token_fd ? BPF_F_TOKEN_FD : 0,
+	);
+	int ret, map, insn_cnt = ARRAY_SIZE(insns);
+
+	map = bpf_map_create(BPF_MAP_TYPE_PERCPU_ARRAY, "libbpf_percpu", sizeof(int), 8, 1,
+			     &map_opts);
+	if (map < 0) {
+		pr_warn("Error in %s(): %s. Couldn't create simple percpu_array map.\n",
+			__func__, errstr(map));
+		return map;
+	}
+
+	insns[0].imm = map;
+
+	ret = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, NULL, "GPL", insns, insn_cnt, &prog_opts);
+	close(map);
+	return probe_fd(ret);
+}
+
 typedef int (*feature_probe_fn)(int /* token_fd */);
 
 static struct kern_feature_cache feature_cache;
@@ -707,6 +739,9 @@ static struct kern_feature_desc {
 	[FEAT_BPF_SYSCALL_COMMON_ATTRS] = {
 		"BPF syscall common attributes support", probe_bpf_syscall_common_attrs,
 	},
+	[FEAT_PERCPU_DATA] = {
+		"kernel supports percpu data", probe_kern_percpu_data,
+	},
 };
 
 bool feat_supported(struct kern_feature_cache *cache, enum kern_feature_id feat_id)
tools/lib/bpf/libbpf_internal.h +2 -0
index d5b7db703b3f..b786f1a35567 100644
@@ -401,6 +401,8 @@ enum kern_feature_id {
 	FEAT_BTF_LAYOUT,
 	/* Kernel supports BPF syscall common attributes */
 	FEAT_BPF_SYSCALL_COMMON_ATTRS,
+	/* Kernel supports percpu data */
+	FEAT_PERCPU_DATA,
 	__FEAT_CNT,
 };
 
#26 lore raw

Re: [PATCH bpf-next v10 6/9] selftests/bpf: Add tests to verify global percpu data

In reply to Leon Hwang
Quoted 87 lines · Leon Hwang

If the arch, like s390x, does not support percpu insn, these cases won't test global percpu data by checking FEAT_PERCPU_DATA support.

The following APIs have been tested for global percpu data:

1. bpf_map__set_initial_value()
2. bpf_map__initial_value()
3. generated percpu struct pointer pointing to internal map's mmaped data
4. bpf_map__lookup_elem() for global percpu data map

At the same time, the case is also tested with 'bpftool gen skeleton -L'.

Add a test to verify that the live vars of subskel won't include the vars for global percpu data.

Assisted-by Codex:gpt-5.5-xhigh
---
 tools/testing/selftests/bpf/Makefile          |   2 +-
 .../bpf/prog_tests/global_data_init.c         | 151 ++++++++++++++++++
 .../bpf/prog_tests/global_percpu_subskel.c    |  37 +++++
 .../bpf/progs/test_global_percpu_data.c       |  33 ++++
 4 files changed, 222 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_global_percpu_data.c
tools/testing/selftests/bpf/Makefile +1 -1
index b642ee489ea6..c37ed9e7b97c 100644
@@ -533,7 +533,7 @@ LSKELS_SIGNED := fentry_test.c fexit_test.c atomics.c
 
 # Generate both light skeleton and libbpf skeleton for these
 LSKELS_EXTRA := test_ksyms_module.c test_ksyms_weak.c kfunc_call_test.c \
-	kfunc_call_test_subprog.c
+	kfunc_call_test_subprog.c test_global_percpu_data.c
 SKEL_BLACKLIST += $$(LSKELS) $$(LSKELS_SIGNED)
 
 test_static_linked.skel.h-deps := test_static_linked1.bpf.o test_static_linked2.bpf.o
tools/testing/selftests/bpf/prog_tests/global_data_init.c +34 -0
index 8466332d7406..f7a56e3fdbce 100644
@@ -1,5 +1,8 @@
 // SPDX-License-Identifier: GPL-2.0
 #include <test_progs.h>
+#include "bpf/libbpf_internal.h"
+#include "test_global_percpu_data.skel.h"
+#include "test_global_percpu_data.lskel.h"
 
 void test_global_data_init(void)
 {
@@ -60,3 +63,151 @@ void test_global_data_init(void)
 	free(newval);
 	bpf_object__close(obj);
 }
+
+static void test_percpu_data_on_cpus(int map_fd, int prog_fd)
+{
+	__u64 args[2] = {0x1234ULL, 0x5678ULL};
+	LIBBPF_OPTS(bpf_test_run_opts, topts,
+		    .ctx_in = args,
+		    .ctx_size_in = sizeof(args),
+		    .flags = BPF_F_TEST_RUN_ON_CPU,
+	);
+	int i, err, key = 0, num_online;
+	bool *online;
+
+	err = parse_cpu_mask_file("/sys/devices/system/cpu/online", &online, &num_online);
+	if (!ASSERT_OK(err, "parse_cpu_mask_file"))
+		return;
+
+	/* run on every online-CPU */
+	for (i = 0; i < num_online; i++) {
+		struct test_global_percpu_data__percpu data = {};
+		__u64 flags;
+
+		if (!online[i])
+			continue;
+
+		topts.cpu = i;
+		topts.retval = -1;
+		err = bpf_prog_test_run_opts(prog_fd, &topts);
+		ASSERT_OK(err, "bpf_prog_test_run_opts");
+		ASSERT_EQ(topts.retval, 0, "bpf_prog_test_run_opts retval");
+
+		flags = ((__u64) i << 32) | BPF_F_CPU;

tools/testing/selftests/bpf/prog_tests/global_data_init.c void test_global_data_init(void) · line 106

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +10 -0
+		err = bpf_map_lookup_elem_flags(map_fd, &key, &data, flags);
+		if (!ASSERT_OK(err, "bpf_map_lookup_elem_flags"))
+			break;
+
+		ASSERT_EQ(data.data, 1, "data.data");
+		ASSERT_TRUE(data.run, "data.run");
+		ASSERT_EQ(data.nums[6], 0xc0de, "data.nums[6]");
+		ASSERT_EQ(data.struct_data.i, 1, "struct_data.i");
+		ASSERT_TRUE(data.struct_data.set, "struct_data.set");
+		ASSERT_EQ(data.struct_data.nums[6], 0xc0de, "struct_data.nums[6]");

Can we add a pre-run assert to ensure that the per-cpu data has not already been modified by another run? Can we also add some cpuid specific assignment to ensure the runs are done on the proper CPU?

Quoted 127 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +107 -0
+	}
+
+	free(online);
+}
+
+static void test_global_percpu_data_init(void)
+{
+	struct test_global_percpu_data__percpu init_value = {};
+	struct test_global_percpu_data__percpu *init_data;
+	struct test_global_percpu_data *skel = NULL;
+	size_t init_data_sz;
+	struct bpf_map *map;
+	int prog_fd, err;
+
+	skel = test_global_percpu_data__open();
+	if (!ASSERT_OK_PTR(skel, "test_global_percpu_data__open"))
+		goto out;
+	if (!ASSERT_OK_PTR(skel->percpu, "skel->percpu"))
+		goto out;
+	if (!ASSERT_OK_PTR(skel->percpu_data, "skel->percpu_data"))
+		goto out;
+	if (!ASSERT_OK_PTR(skel->percpu_looooooooong, "skel->percpu_looooooooong"))
+		goto out;
+
+	ASSERT_STREQ(bpf_map__name(skel->maps.percpu_data), ".percpu.data", "data map name");
+	ASSERT_STREQ(bpf_map__name(skel->maps.percpu_looooooooong), ".percpu.looooooooong",
+		     "long map name");
+	ASSERT_STREQ(bpf_map__name(skel->maps.percpu), ".percpu", "map name");
+	ASSERT_EQ(skel->percpu->data, -1, "skel->percpu->data");
+	ASSERT_FALSE(skel->percpu->run, "skel->percpu->run");
+	ASSERT_EQ(skel->percpu->nums[6], 0, "skel->percpu->nums[6]");
+	ASSERT_EQ(skel->percpu->struct_data.i, -1, "struct_data.i");
+	ASSERT_FALSE(skel->percpu->struct_data.set, "struct_data.set");
+	ASSERT_EQ(skel->percpu->struct_data.nums[6], 0, "struct_data.nums[6]");
+
+	map = skel->maps.percpu;
+	if (!ASSERT_EQ(bpf_map__type(map), BPF_MAP_TYPE_PERCPU_ARRAY, "bpf_map__type"))
+		goto out;
+
+	init_value.data = 2;
+	init_value.nums[6] = -1;
+	init_value.struct_data.i = 2;
+	init_value.struct_data.nums[6] = -1;
+	err = bpf_map__set_initial_value(map, &init_value, sizeof(init_value));
+	if (!ASSERT_OK(err, "bpf_map__set_initial_value"))
+		goto out;
+
+	init_data = bpf_map__initial_value(map, &init_data_sz);
+	if (!ASSERT_OK_PTR(init_data, "bpf_map__initial_value"))
+		goto out;
+
+	ASSERT_EQ(init_data->data, init_value.data, "init_value data");
+	ASSERT_EQ(init_data->run, init_value.run, "init_value run");
+	ASSERT_EQ(init_data->struct_data.i, init_value.struct_data.i, "init_value struct_data.i");
+	ASSERT_EQ(init_data->struct_data.nums[6], init_value.struct_data.nums[6],
+		  "init_value struct_data.nums[6]");
+	ASSERT_EQ(init_data_sz, sizeof(init_value), "init_value size");
+	ASSERT_EQ((void *) init_data, (void *) skel->percpu, "skel->percpu eq init_data");
+	ASSERT_EQ(skel->percpu->data, init_value.data, "skel->percpu->data");
+	ASSERT_EQ(skel->percpu->run, init_value.run, "skel->percpu->run");
+	ASSERT_EQ(skel->percpu->struct_data.i, init_value.struct_data.i,
+		  "skel->percpu->struct_data.i");
+	ASSERT_EQ(skel->percpu->struct_data.nums[6], init_value.struct_data.nums[6],
+		  "skel->percpu->struct_data.nums[6]");
+
+	err = test_global_percpu_data__load(skel);
+	if (!ASSERT_OK(err, "test_global_percpu_data__load"))
+		goto out;
+
+	ASSERT_OK_PTR(skel->percpu, "skel->percpu");
+
+	prog_fd = bpf_program__fd(skel->progs.update_percpu_data);
+	test_percpu_data_on_cpus(bpf_map__fd(map), prog_fd);
+
+out:
+	test_global_percpu_data__destroy(skel);
+}
+
+static void test_global_percpu_data_lskel(void)
+{
+	struct test_global_percpu_data_lskel *lskel = NULL;
+	int prog_fd, map_fd;
+
+	lskel = test_global_percpu_data_lskel__open_and_load();
+	if (!ASSERT_OK_PTR(lskel, "test_global_percpu_data_lskel__open_and_load"))
+		goto out;
+
+	map_fd = lskel->maps.percpu.map_fd;
+	prog_fd = lskel->progs.update_percpu_data.prog_fd;
+	test_percpu_data_on_cpus(map_fd, prog_fd);
+
+out:
+	test_global_percpu_data_lskel__destroy(lskel);
+}
+
+void test_global_percpu_data(void)
+{
+	if (!feat_supported(NULL, FEAT_PERCPU_DATA)) {
+		test__skip();
+		return;
+	}
+
+	if (test__start_subtest("init"))
+		test_global_percpu_data_init();
+	if (test__start_subtest("lskel"))
+		test_global_percpu_data_lskel();
+}
tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c new file +14 -0
new file mode 100644
index 000000000000..8aebd533d86b
@@ -0,0 +1,37 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include "test_global_percpu_data.subskel.h"
+
+void test_global_percpu_data_subskel(void)
+{
+	struct test_global_percpu_data *subskel = NULL;
+	struct bpf_object *obj;
+	int i;
+
+	obj = bpf_object__open_file("./test_global_percpu_data.bpf.o", NULL);
+	if (!ASSERT_OK_PTR(obj, "bpf_object__open_file"))
+		return;
+

tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c line 24

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c new file +10 -0
+	subskel = test_global_percpu_data__open(obj);
+	if (!ASSERT_OK_PTR(subskel, "test_global_percpu_data__open"))
+		goto out;
+
+	if (!ASSERT_OK_PTR(subskel->subskel, "subskel"))
+		goto out;
+	if (!ASSERT_OK_PTR(subskel->maps.percpu, "maps.percpu"))
+		goto out;
+	ASSERT_EQ(bpf_map__type(subskel->maps.percpu), BPF_MAP_TYPE_PERCPU_ARRAY,
+		  "percpu_map_type");

Not sure why these assertions would be necessary, wouldn't the test crash or quickly fail if they didn't hold?

tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c line 32

Quoted 8 lines
tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c new file +8 -0
+	ASSERT_GT(subskel->subskel->var_cnt, 0, "var_cnt");
+
+	for (i = 0; i < subskel->subskel->var_cnt; i++) {
+		const struct bpf_var_skeleton *var;
+
+		var = (void *) subskel->subskel->vars + i * subskel->subskel->var_skel_sz;
+		ASSERT_NEQ(var->map, &subskel->maps.percpu, "var");
+	}

If we turn the substest to SYSCALL instead of TRACEPOINT we can read and report the return value as the test happens and keep them self-contained.

Quoted 11 lines
tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c new file +5 -0
+
+out:
+	test_global_percpu_data__destroy(subskel);
+	bpf_object__close(obj);
+}
tools/testing/selftests/bpf/progs/test_global_percpu_data.c new file +0 -0
new file mode 100644
index 000000000000..54380dfb11a5

tools/testing/selftests/bpf/progs/test_global_percpu_data.c line 10

Quoted 10 lines
tools/testing/selftests/bpf/progs/test_global_percpu_data.c new file +10 -0
@@ -0,0 +1,33 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+int unused SEC(".percpu.looooooooong");
+int data2 SEC(".percpu.data");
+int data SEC(".percpu") = -1;
+int nums[7] SEC(".percpu");
+char run SEC(".percpu") = 0;

Can we add a comment that those are used to test the names in the userspace portion of the test? As they stand they seem unused.

tools/testing/selftests/bpf/progs/test_global_percpu_data.c line 20

Quoted 10 lines
tools/testing/selftests/bpf/progs/test_global_percpu_data.c new file +10 -0
+struct {
+	char set;
+	int i;
+	int nums[7];
+} struct_data SEC(".percpu") = {
+	.set = 0,
+	.i = -1,
+};
+
+SEC("raw_tp/task_rename")

Is this actually installed as a tracepoint? I think we run it as a program. Is putting it in raw_tp doing anything in any way?

Quoted 13 lines
tools/testing/selftests/bpf/progs/test_global_percpu_data.c new file +13 -0
+__auxiliary
+int update_percpu_data(void *ctx)
+{
+	struct_data.nums[6] = 0xc0de;
+	struct_data.set = 1;
+	struct_data.i = 1;
+	nums[6] = 0xc0de;
+	data = 1;
+	run = 1;
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
#29 lore raw

Re: [PATCH bpf-next v10 7/9] selftests/bpf: Test direct reading/writing read-only percpu_array map

In reply to Leon Hwang
Quoted 46 lines · Leon Hwang

Verify these two cases:

1. Direct reading the data of read-only percpu data's percpu_array map is allowed.
2. Direct writing the data of read-only percpu data's percpu_array map is disallowed.

Assisted-by Codex:gpt-5.5-xhigh
---
 .../bpf/prog_tests/global_data_init.c         | 83 +++++++++++++++++++
 1 file changed, 83 insertions(+)
tools/testing/selftests/bpf/prog_tests/global_data_init.c +25 -0
index f7a56e3fdbce..d4405646847a 100644
@@ -199,6 +199,85 @@ static void test_global_percpu_data_lskel(void)
 	test_global_percpu_data_lskel__destroy(lskel);
 }
 
+static int create_rdonly_percpu_array(void)
+{
+	LIBBPF_OPTS(bpf_map_create_opts, map_opts,
+		    .map_flags = BPF_F_RDONLY_PROG,
+	);
+	int key = 0, map_fd, err;
+	__u64 value = 0;
+
+	map_fd = bpf_map_create(BPF_MAP_TYPE_PERCPU_ARRAY, "percpu_ro_map", sizeof(int),
+				sizeof(__u64), 1, &map_opts);
+	if (!ASSERT_GE(map_fd, 0, "bpf_map_create"))
+		return -1;
+
+	err = bpf_map_update_elem(map_fd, &key, &value, BPF_F_ALL_CPUS);
+	if (!ASSERT_OK(err, "bpf_map_update_elem"))
+		goto out;
+
+	err = bpf_map_freeze(map_fd);
+	if (!ASSERT_OK(err, "bpf_map_freeze"))
+		goto out;
+
+	return map_fd;
+
+out:
+	close(map_fd);

tools/testing/selftests/bpf/prog_tests/global_data_init.c static void test_global_percpu_data_lskel(void) · line 236

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +10 -0
+	return -1;
+}
+
+static void test_global_percpu_data_rdonly_direct_read(void)
+{
+	struct bpf_insn insns[] = {
+		BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0),
+		BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0),
+		BPF_EXIT_INSN(),
+	};

Why hardcode this as inline assembly? Can we do a syscall example in the progs/? That way there will also be a concrete example of how we're using the feature in practice.

Quoted 56 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +48 -0
+	int map_fd, prog_fd;
+
+	map_fd = create_rdonly_percpu_array();
+	if (map_fd < 0)
+		return;
+
+	insns[0].imm = map_fd;
+	prog_fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, "percpu_ro_prog", "GPL", insns,
+				ARRAY_SIZE(insns), NULL);
+	if (ASSERT_GE(prog_fd, 0, "bpf_prog_load"))
+		close(prog_fd);
+	close(map_fd);
+}
+
+static void test_global_percpu_data_rdonly_direct_write(void)
+{
+	LIBBPF_OPTS(bpf_prog_load_opts, prog_opts);
+	struct bpf_insn insns[] = {
+		BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0),
+		BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0),
+		BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0),
+		BPF_EXIT_INSN(),
+	};
+	char log_buf[256] = {};
+	int map_fd, prog_fd;
+
+	prog_opts.log_buf = log_buf;
+	prog_opts.log_size = sizeof(log_buf);
+	prog_opts.log_level = 1;
+
+	map_fd = create_rdonly_percpu_array();
+	if (map_fd < 0)
+		return;
+
+	insns[0].imm = map_fd;
+	prog_fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, "percpu_ro_prog", "GPL", insns,
+				ARRAY_SIZE(insns), &prog_opts);
+	if (!ASSERT_LT(prog_fd, 0, "bpf_prog_load"))
+		close(prog_fd);
+	else
+		ASSERT_HAS_SUBSTR(log_buf, "write into map forbidden", "verifier log");
+	close(map_fd);
+}
+
 void test_global_percpu_data(void)
 {
 	if (!feat_supported(NULL, FEAT_PERCPU_DATA)) {
@@ -210,4 +289,8 @@ void test_global_percpu_data(void)
 		test_global_percpu_data_init();
 	if (test__start_subtest("lskel"))
 		test_global_percpu_data_lskel();
+	if (test__start_subtest("rdonly_direct_read"))
+		test_global_percpu_data_rdonly_direct_read();
+	if (test__start_subtest("rdonly_direct_write"))
+		test_global_percpu_data_rdonly_direct_write();
 }
reviewer Reviewed-by on v10 8/9
#32 lore raw

Re: [PATCH bpf-next v10 8/9] selftests/bpf: Test verifier log for global percpu data

In reply to Leon Hwang
Quoted 4 lines · Leon Hwang

Add two tests to verify the verifier log
"R%d points to percpu_array map which cannot be used as const string\n".

Quoted 59 lines
---
 .../bpf/prog_tests/global_data_init.c         |  6 +++++
 .../bpf/progs/test_global_percpu_data.c       | 23 +++++++++++++++++++
 2 files changed, 29 insertions(+)
tools/testing/selftests/bpf/prog_tests/global_data_init.c +6 -0
index d4405646847a..598b7dde6581 100644
@@ -278,6 +278,11 @@ static void test_global_percpu_data_rdonly_direct_write(void)
 	close(map_fd);
 }
 
+static void test_global_percpu_data_verifier_log(void)
+{
+	RUN_TESTS(test_global_percpu_data);
+}
+
 void test_global_percpu_data(void)
 {
 	if (!feat_supported(NULL, FEAT_PERCPU_DATA)) {
@@ -293,4 +298,5 @@ void test_global_percpu_data(void)
 		test_global_percpu_data_rdonly_direct_read();
 	if (test__start_subtest("rdonly_direct_write"))
 		test_global_percpu_data_rdonly_direct_write();
+	test_global_percpu_data_verifier_log();
 }
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +23 -0
index 54380dfb11a5..b852a0b07508 100644
@@ -30,4 +30,27 @@ int update_percpu_data(void *ctx)
 	return 0;
 }
 
+static const char fmt[] SEC(".percpu.fmt") = "data %d\n";
+
+SEC("?kprobe")
+__failure __msg("R{{[0-9]+}} points to percpu_array map which cannot be used as const string")
+int verifier_strncmp(void *ctx)
+{
+	return bpf_strncmp("test", 5, fmt);
+}
+
+SEC("?kprobe")
+__failure __msg("R{{[0-9]+}} points to percpu_array map which cannot be used as const string")
+int verifier_snprintf(void *ctx)
+{
+	u64 args[] = { data };
+	char buf[128];
+	int len;
+
+	len = bpf_snprintf(buf, sizeof(buf), fmt, args, sizeof(args));
+	if (len > 0)
+		bpf_printk("snprintf: %s\n", buf);
+	return 0;
+}
+
 char _license[] SEC("license") = "GPL";
reviewer Reviewed-by on v10 9/9
#34 lore raw

Re: [PATCH bpf-next v10 9/9] selftests/bpf: Verify bpf_iter for global percpu data

In reply to Leon Hwang
Quoted 4 lines · Leon Hwang

Add a test to verify that it is OK to iter the percpu_array map used for global percpu data.

Quoted 107 lines
---
 .../bpf/prog_tests/global_data_init.c         | 52 +++++++++++++++++++
 .../bpf/progs/test_global_percpu_data.c       | 25 +++++++++
 2 files changed, 77 insertions(+)
tools/testing/selftests/bpf/prog_tests/global_data_init.c +52 -0
index 598b7dde6581..e5a40ea36ee7 100644
@@ -283,6 +283,56 @@ static void test_global_percpu_data_verifier_log(void)
 	RUN_TESTS(test_global_percpu_data);
 }
 
+static void test_global_percpu_data_iter(void)
+{
+	DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);
+	struct test_global_percpu_data *skel;
+	union bpf_iter_link_info linfo = {};
+	struct bpf_link *link = NULL;
+	int fd, num_cpus, len, err;
+	char buf[16];
+
+	num_cpus = libbpf_num_possible_cpus();
+	if (!ASSERT_GT(num_cpus, 0, "libbpf_num_possible_cpus"))
+		return;
+
+	skel = test_global_percpu_data__open();
+	if (!ASSERT_OK_PTR(skel, "test_global_percpu_data__open"))
+		return;
+
+	skel->rodata->num_cpus = num_cpus;
+	skel->rodata->offsetof_num = offsetof(struct test_global_percpu_data__percpu, struct_data);
+	skel->rodata->offsetof_num += sizeof(skel->percpu->struct_data) - sizeof(int);
+	skel->rodata->elem_sz = roundup(sizeof(struct test_global_percpu_data__percpu), 8);
+	skel->percpu->struct_data.nums[6] = 0xc0de;
+
+	err = test_global_percpu_data__load(skel);
+	if (!ASSERT_OK(err, "test_global_percpu_data__load"))
+		goto out;
+
+	linfo.map.map_fd = bpf_map__fd(skel->maps.percpu);
+	opts.link_info = &linfo;
+	opts.link_info_len = sizeof(linfo);
+	link = bpf_program__attach_iter(skel->progs.dump_percpu_data, &opts);
+	if (!ASSERT_OK_PTR(link, "bpf_program__attach_iter"))
+		goto out;
+
+	fd = bpf_iter_create(bpf_link__fd(link));
+	if (!ASSERT_GE(fd, 0, "bpf_iter_create"))
+		goto out;
+
+	while ((len = read(fd, buf, sizeof(buf))) > 0)
+		do { } while (0);
+	ASSERT_EQ(len, 0, "read iter");
+	ASSERT_TRUE(skel->bss->run_iter, "run_iter");
+	ASSERT_EQ(skel->bss->percpu_data_sum, 0xc0de * num_cpus, "percpu_data_sum");
+
+	close(fd);
+out:
+	bpf_link__destroy(link);
+	test_global_percpu_data__destroy(skel);
+}
+
 void test_global_percpu_data(void)
 {
 	if (!feat_supported(NULL, FEAT_PERCPU_DATA)) {
@@ -299,4 +349,6 @@ void test_global_percpu_data(void)
 	if (test__start_subtest("rdonly_direct_write"))
 		test_global_percpu_data_rdonly_direct_write();
 	test_global_percpu_data_verifier_log();
+	if (test__start_subtest("iter"))
+		test_global_percpu_data_iter();
 }
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +25 -0
index b852a0b07508..8b1f625017d1 100644
@@ -53,4 +53,29 @@ int verifier_snprintf(void *ctx)
 	return 0;
 }
 
+volatile const __u32 num_cpus = 0;
+volatile const int offsetof_num;
+volatile const int elem_sz;
+__u32 percpu_data_sum = 0;
+bool run_iter = false;
+
+SEC("iter/bpf_map_elem")
+__auxiliary
+int dump_percpu_data(struct bpf_iter__bpf_map_elem *ctx)
+{
+	void *pptr = ctx->value;
+	int i;
+
+	if (!pptr)
+		return 0;
+
+	run_iter = true;
+
+	for (i = 0; i < num_cpus; i++) {
+		percpu_data_sum += *(int *) (pptr + offsetof_num);
+		pptr += elem_sz;
+	}
+	return 0;
+}
+
 char _license[] SEC("license") = "GPL";
author
#7 lore raw
In reply to Emil Tsalapatis
Quoted 1 line · Emil Tsalapatis

On Wed Jul 15, 2026 at 11:32 AM EDT, Leon Hwang wrote:

[...]

Quoted 14 lines

kernel/bpf/arraymap.c static void *percpu_array_map_lookup_elem(struct bpf_map *map, void *key) · line 271

Quoted 11 lines
kernel/bpf/arraymap.c +10 -0
+static int percpu_array_map_direct_value_addr(const struct bpf_map *map, u64 *imm, u32 off)
+{
+	struct bpf_array *array = container_of(map, struct bpf_array, map);
+
+	if (map->max_entries != 1)
+		return -EOPNOTSUPP;
+	if (off >= map->value_size)
+		return -EINVAL;
+	if (!bpf_jit_supports_percpu_insn())
+		return -EOPNOTSUPP;

Nit: This should be first, if the JIT doesn't support the instruction all else is moot. Same below.

Ack.

Quoted 22 lines
Quoted 22 lines
kernel/bpf/arraymap.c +21 -0
+
+	*imm = (u64)(__force unsigned long) array->pptrs[0];
+	return 0;
+}
+
+static int percpu_array_map_direct_value_meta(const struct bpf_map *map, u64 imm, u32 *off)
+{
+	struct bpf_array *array = container_of(map, struct bpf_array, map);
+	u64 base = (u64)(__force unsigned long) array->pptrs[0];
+
+	if (map->max_entries != 1)
+		return -EOPNOTSUPP;
+	if (imm < base || imm >= base + array->elem_size)
+		return -ENOENT;
+	if (!bpf_jit_supports_percpu_insn())
+		return -EOPNOTSUPP;
+
+	*off = imm - base;
+	return 0;
+}
+

[...]

Quoted 27 lines
Quoted 14 lines
kernel/bpf/verifier.c +2 -0
index e1244a721194..ae442ea217c4 100644
@@ -5615,6 +5615,8 @@ int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val,
 	u64 addr;
 	int err;
 
+	if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY || map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY)
+		return -EINVAL;
 	err = map->ops->map_direct_value_addr(map, &addr, off);
 	if (err)
 		return err;

kernel/bpf/verifier.c static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct b · line 6180

Quoted 5 lines
kernel/bpf/verifier.c +1 -0
@@ -6174,6 +6176,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct b
 			if (tnum_is_const(reg->var_off) &&
 			    bpf_map_is_rdonly(map) &&
 			    map->ops->map_direct_value_addr &&
+			    map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY &&
 			    map->map_type != BPF_MAP_TYPE_INSN_ARRAY) {

An issue I have with the patch is that it takes the special-casing we already have for INSN_ARRAY and makes it worse. The instruction arrays implement direct_value_addr, but it's not actually valid to call direct_value_addr from a bunch of places where it's called, so we have to special case check the map type to prevent it. Case in point the special casing in check_arg_const_str in the thunk below.

Quoted 4 lines

Imo we should consider making a different map method different from direct_value_addr that INSN_ARRAY and PERCPU_ARRAY implement. That way we remove most of the map type checks in those paths. I'm not exactly sure how that would look like, but I could take a look at it tomorrow.

Seems reasonable to me. I don't have good candidates for the new map method name.

Instead of making a new map method, how about factoring out a helper for the 'if (t == BPF_READ && value_regno >= 0) {}' block? It will simplify these map_type checks in the helper.

Quoted 20 lines
Quoted 20 lines
kernel/bpf/verifier.c +6 -0
 				int map_off = off + reg->var_off.value;
 				u64 val = 0;
@@ -8176,6 +8179,12 @@ static int check_arg_const_str(struct bpf_verifier_env *env,
 		return -EACCES;
 	}
 
+	if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) {
+		verbose(env, "%s points to percpu_array map which cannot be used as const string\n",
+			reg_arg_name(env, argno));
+		return -EACCES;
+	}
+
 	if (!bpf_map_is_rdonly(map)) {
 		verbose(env, "%s does not point to a readonly map'\n", reg_arg_name(env, argno));
 		return -EACCES;
@@ -18279,6 +18288,12 @@ static int check_and_resolve_insns(struct bpf_verifier_env *env)
 					return -EINVAL;
 				}
 
Quoted 8 lines

kernel/bpf/verifier.c static int check_and_resolve_insns(struct bpf_verifier_env *env) · line 18296

Quoted 6 lines
kernel/bpf/verifier.c +6 -0
+				if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY &&
+				    !env->prog->jit_requested) {
+					verbose(env, "JIT is required to use global percpu data\n");
+					return -EOPNOTSUPP;
+				}
+

Another place where we add map-specific checks where there are none.

I think this check is unnecessary, because 'prog->jit_required = true' has been set for global percpu data.

Will drop this if.

Thanks,
Leon

Quoted 5 lines
Quoted 4 lines
kernel/bpf/verifier.c +0 -0
 				err = map->ops->map_direct_value_addr(map, &addr, off);
 				if (err) {
 					verbose(env, "invalid access to map value pointer, value_size=%u off=%u\n",
author
#15 lore raw

Re: [PATCH bpf-next v10 4/9] libbpf: Add support for global percpu data

In reply to Emil Tsalapatis
Quoted 1 line · Emil Tsalapatis

On Wed Jul 15, 2026 at 11:32 AM EDT, Leon Hwang wrote:

[...]

Quoted 30 lines
Quoted 12 lines

struct elf_sec_desc {

tools/lib/bpf/libbpf.c +2 -0
@@ -1839,6 +1842,8 @@ static size_t bpf_map_mmap_sz(const struct bpf_map *map)
 	switch (map->def.type) {
 	case BPF_MAP_TYPE_ARRAY:
 		return array_map_mmap_sz(map->def.value_size, map->def.max_entries);
+	case BPF_MAP_TYPE_PERCPU_ARRAY:
+		return map->def.value_size;
 	case BPF_MAP_TYPE_ARENA:
 		return page_sz * map->def.max_entries;
 	default:

tools/lib/bpf/libbpf.c static int bpf_map_mmap_resize(struct bpf_map *map, size_t old_sz, size_t new_sz · line 1875

Quoted 6 lines
tools/lib/bpf/libbpf.c +2 -1
@@ -1866,7 +1871,8 @@ static int bpf_map_mmap_resize(struct bpf_map *map, size_t old_sz, size_t new_sz
 	return 0;
 }
 
-static char *internal_map_name(struct bpf_object *obj, const char *real_name)
+static char *internal_map_name(struct bpf_object *obj, const char *real_name,
+			       enum libbpf_map_type type)

We can avoid passing the type here by testing against ".percpu" since that's how we are deriving the map type in the first place. But more importantly:

Quoted 7 lines
tools/lib/bpf/libbpf.c +0 -0
 {
 	char map_name[BPF_OBJ_NAME_LEN], *p;
 	int pfx_len, sfx_len = max((size_t)7, strlen(real_name));
@@ -1907,8 +1913,11 @@ static char *internal_map_name(struct bpf_object *obj, const char *real_name)
 	if (sfx_len >= BPF_OBJ_NAME_LEN)
 		sfx_len = BPF_OBJ_NAME_LEN - 1;
 
Quoted 15 lines

tools/lib/bpf/libbpf.c static char *internal_map_name(struct bpf_object *obj, const char *real_name) · line 1920

Quoted 7 lines
tools/lib/bpf/libbpf.c +5 -2
-	/* if there are two or more dots in map name, it's a custom dot map */
-	if (strchr(real_name + 1, '.') != NULL)
+	/*
+	 * Don't prefix the bpf_object name if this is a custom dot map
+	 * (containing two or more dots) or a percpu data map.
+	 */
+	if (strchr(real_name + 1, '.') != NULL || type == LIBBPF_MAP_PERCPU)

Is there a reason we don't use the exact same logic as the other internal maps here? I understand that a lot of the conventions around the naming are there for legacy reasons, but it seems like we're singling out the .percpu section for highly nonbvious reasons. Imo we should consider doing the same prefixing for a bare ".percpu" section that we do for the other internal ones. At the very least, there needs to be some explanation as to why .percpu gets special treatment.

I prefer passing 'type'. Excluding _PERCPU here is to avoid the legacy naming convention for new internal maps.

Quoted 10 lines

@Andrii Wdyt?

Quoted 7 lines
tools/lib/bpf/libbpf.c +0 -0
 		pfx_len = 0;
 	else
 		pfx_len = min((size_t)BPF_OBJ_NAME_LEN - sfx_len - 1, strlen(obj->name));
@@ -1938,7 +1947,7 @@ static bool map_is_mmapable(struct bpf_object *obj, struct bpf_map *map)
 	struct btf_var_secinfo *vsi;
 	int i, n;
 
Quoted 8 lines

tools/lib/bpf/libbpf.c static bool map_is_mmapable(struct bpf_object *obj, struct bpf_map *map) · line 1951

Quoted 3 lines
tools/lib/bpf/libbpf.c +1 -1
-	if (!map->btf_value_type_id)
+	if (!map->btf_value_type_id || map->libbpf_type == LIBBPF_MAP_PERCPU)
 		return false;

Nit: These are two separate checks rolled into one, and each one checks a different thing. THe type check against MAP_PERCPU merits a comment as
well: It's the only internal section that is not really mappable because there's no way to represent it as userspace state.

Ack.

Will add a new iff for libbpf_type check with a comment.

Quoted 11 lines
Quoted 11 lines
tools/lib/bpf/libbpf.c +1 -0
 	t = btf__type_by_id(obj->btf, map->btf_value_type_id);
@@ -1962,6 +1971,7 @@ static int
 bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
 			      const char *real_name, int sec_idx, void *data, size_t data_sz)
 {
+	bool is_percpu = type == LIBBPF_MAP_PERCPU;
 	struct bpf_map_def *def;
 	struct bpf_map *map;
 	size_t mmap_sz;

[...]

Quoted 36 lines
Quoted 10 lines
tools/lib/bpf/libbpf.c +1 -1
@@ -4944,7 +4970,7 @@ static int map_fill_btf_type_info(struct bpf_object *obj, struct bpf_map *map)
 
 	/*
 	 * LLVM annotates global data differently in BTF, that is,
-	 * only as '.data', '.bss' or '.rodata'.
+	 * only as '.data', '.bss', '.percpu' or '.rodata'.
 	 */
 	if (!bpf_map__is_internal(map))
 		return -ENOENT;

tools/lib/bpf/libbpf.c static int · line 5322

Quoted 4 lines
tools/lib/bpf/libbpf.c +1 -0
@@ -5293,18 +5319,30 @@ static int
 bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map)
 {
 	enum libbpf_map_type map_type = map->libbpf_type;
+	bool is_percpu = map_type == LIBBPF_MAP_PERCPU;

Nit: If we do

	__u64 update_flags = is_percpu ? BPF_F_ALL_CPUS : 0;

we can declare the variable as const and ...

tools/lib/bpf/libbpf.c static int · line 5335

Quoted 13 lines
tools/lib/bpf/libbpf.c +10 -0
+	__u64 update_flags = 0;
 	int err, zero = 0;
 	size_t mmap_sz;
 
+	if (is_percpu) {
+		if (!obj->gen_loader && !kernel_supports(obj, FEAT_PERCPU_DATA)) {
+			pr_warn("map '%s': kernel does not support percpu data.\n",
+				bpf_map__name(map));
+			return -EOPNOTSUPP;
+		}
+
+		update_flags = BPF_F_ALL_CPUS;
+	}

... we can collapse the above into a single nested level:

Quoted 3 lines
	if (is_percpu && !obj->gen_loader && !kernel_supports(obj, FEAT_PERCPU_DATA)) {
		...
	}

Good point.

Quoted 39 lines
Quoted 39 lines
tools/lib/bpf/libbpf.c +13 -2
+
 	if (obj->gen_loader) {
 		bpf_gen__map_update_elem(obj->gen_loader, map - obj->maps,
-					 map->mmaped, map->def.value_size);
+					 map->mmaped, map->def.value_size, update_flags);
 		if (map_type == LIBBPF_MAP_RODATA || map_type == LIBBPF_MAP_KCONFIG)
 			bpf_gen__map_freeze(obj->gen_loader, map - obj->maps);
 		return 0;
 	}
 
-	err = bpf_map_update_elem(map->fd, &zero, map->mmaped, 0);
+	err = bpf_map_update_elem(map->fd, &zero, map->mmaped, update_flags);
 	if (err) {
 		err = -errno;
 		pr_warn("map '%s': failed to set initial contents: %s\n",
@@ -5349,6 +5387,13 @@ bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map)
 			return err;
 		}
 		map->mmaped = mmaped;
+	} else if (is_percpu) {
+		if (mprotect(map->mmaped, mmap_sz, PROT_READ)) {
+			err = -errno;
+			pr_warn("map '%s': failed to mprotect() contents: %s\n",
+				bpf_map__name(map), errstr(err));
+			return err;
+		}
 	} else if (map->mmaped) {
 		munmap(map->mmaped, mmap_sz);
 		map->mmaped = NULL;
@@ -10807,11 +10852,16 @@ static bool map_uses_real_name(const struct bpf_map *map)
 	 * such map's corresponding ELF section name as a map name.
 	 * This check distinguishes .data/.rodata from .data.* and .rodata.*
 	 * maps to know which name has to be returned to the user.
+	 * Map name of the custom .percpu.* maps might be truncated to
+	 * BPF_OBJ_NAME_LEN-1 chars in internal_map_name(). Hence, percpu data
+	 * maps must use real name for their user-visible name.
 	 */
 	if (map->libbpf_type == LIBBPF_MAP_DATA && strcmp(map->real_name, DATA_SEC) != 0)
Quoted 8 lines

tools/lib/bpf/libbpf.c static bool map_uses_real_name(const struct bpf_map *map) · line 10864

Quoted 5 lines
tools/lib/bpf/libbpf.c +2 -0
 		return true;
 	if (map->libbpf_type == LIBBPF_MAP_RODATA && strcmp(map->real_name, RODATA_SEC) != 0)
 		return true;
+	if (map->libbpf_type == LIBBPF_MAP_PERCPU)
+		return true;

Same comment as above here wrt uniformity. This is the part that requires us to check agianst the map type in __init_internal_map().

Let us wait for Andrii's comment.

Thanks,
Leon

Quoted 5 lines
Quoted 4 lines
tools/lib/bpf/libbpf.c +0 -0
 	return false;
 }
 
author
#20 lore raw

Re: [PATCH bpf-next v10 5/9] bpftool: Generate skeleton for global percpu data

In reply to Emil Tsalapatis
Quoted 1 line · Emil Tsalapatis

On Wed Jul 15, 2026 at 11:32 AM EDT, Leon Hwang wrote:

[...]

Quoted 12 lines
Quoted 12 lines

static bool get_datasec_ident(const char *sec_name, char *buf, size_t buf_sz) {

tools/bpf/bpftool/gen.c +1 -1
-	static const char *pfxs[] = { ".data", ".rodata", ".bss", ".kconfig" };
+	static const char *pfxs[] = { ".data", ".rodata", ".bss", ".percpu", ".kconfig" };
 	int i, n;
 
 	/* recognize hard coded LLVM section name */
@@ -254,7 +260,7 @@ static const struct btf_type *find_type_for_map(struct btf *btf, const char *map
 	return NULL;
 }
 
Quoted 8 lines

tools/bpf/bpftool/gen.c static const struct btf_type *find_type_for_map(struct btf *btf, const char *map · line 263

Quoted 2 lines
tools/bpf/bpftool/gen.c +1 -1
-static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)
+static bool is_skel_data(const struct bpf_map *map, char *buf, size_t sz)

Here we change the function to take PERCPU_ARRAY into account, but then immediately turn around and add checks of the form is_skel_data && type != BPF_MAP_TYPE_PERCPU_ARRAY. If we keep both the existing mmapable_map and add a separate is_skel_data map (possibly using is_mappable_map), we can remove those additional checks.

Agreed.

I'd like to keep this new is_skel_data(), then update the is_mmapable_map() to:

static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)

{
	return is_skel_data(map, buf, sz) && bpf_map__type(map) !=

BPF_MAP_TYPE_PERCPU_ARRAY;
}

Yep, it is not a good choice to use '!=' here. But it can simplify the code.

Quoted 43 lines
Quoted 43 lines
tools/bpf/bpftool/gen.c +13 -4
 {
 	size_t tmp_sz;
 
@@ -263,13 +269,19 @@ static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)
 		return true;
 	}
 
-	if (!bpf_map__is_internal(map) || !(bpf_map__map_flags(map) & BPF_F_MMAPABLE))
+	if (!bpf_map__is_internal(map))
 		return false;
 
 	if (!get_map_ident(map, buf, sz))
 		return false;
 
-	return true;
+	if (bpf_map__map_flags(map) & BPF_F_MMAPABLE)
+		return true;
+
+	if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY)
+		return true;
+
+	return false;
 }
 
 static int codegen_datasecs(struct bpf_object *obj, const char *obj_name)
@@ -286,8 +298,11 @@ static int codegen_datasecs(struct bpf_object *obj, const char *obj_name)
 		return -errno;
 
 	bpf_object__for_each_map(map, obj) {
-		/* only generate definitions for memory-mapped internal maps */
-		if (!is_mmapable_map(map, map_ident, sizeof(map_ident)))
+		/*
+		 * Only generate definitions for internal maps that have
+		 * mmapped data.
+		 */
+		if (!is_skel_data(map, map_ident, sizeof(map_ident)))
 			continue;
 
 		sec = find_type_for_map(btf, map_ident);
@@ -339,8 +354,14 @@ static int codegen_subskel_datasecs(struct bpf_object *obj, const char *obj_name
 		return -errno;
 
Quoted 14 lines

tools/bpf/bpftool/gen.c static int codegen_subskel_datasecs(struct bpf_object *obj, const char *obj_name · line 365

Quoted 12 lines
tools/bpf/bpftool/gen.c +8 -2
 	bpf_object__for_each_map(map, obj) {
-		/* only generate definitions for memory-mapped internal maps */
-		if (!is_mmapable_map(map, map_ident, sizeof(map_ident)))
+		/*
+		 * Only generate definitions for internal maps that have
+		 * mmapped data.
+		 */
+		if (!is_skel_data(map, map_ident, sizeof(map_ident)))
+			continue;
+
+		if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY)
 			continue;

Case in point, the old is_mmapable_map would work fine here.

Yes.

Quoted 7 lines
Quoted 7 lines
tools/bpf/bpftool/gen.c +0 -0
 		sec = find_type_for_map(btf, map_ident);
@@ -493,7 +514,10 @@ static size_t bpf_map_mmap_sz(const struct bpf_map *map)
 	return map_sz;
 }
 
Quoted 10 lines

tools/bpf/bpftool/gen.c static size_t bpf_map_mmap_sz(const struct bpf_map *map) · line 519

Quoted 4 lines
tools/bpf/bpftool/gen.c +3 -1
-/* Emit type size asserts for all top-level fields in memory-mapped internal maps. */
+/*
+ * Emit type size asserts for all top-level fields in internal maps that
+ * have mmaped data.

I do not think the new comment makes it any clearer, the difference is "memory-mapped internal maps" and "maps that are memory-mapped initially but actually inaccessible after loading because there's no way to represent them as a mapping of the skeleton" AFAICT which is not easy to infer currently.

Hmm, it is hard to explain the new case that the mmaped data is visible in user-space, but is invisible in kernel-space.

Will drop this change.

Quoted 13 lines
Quoted 13 lines
tools/bpf/bpftool/gen.c +2 -1
+ */
 static void codegen_asserts(struct bpf_object *obj, const char *obj_name)
 {
 	struct btf *btf = bpf_object__btf(obj);
@@ -517,7 +541,7 @@ static void codegen_asserts(struct bpf_object *obj, const char *obj_name)
 		", obj_name);
 
 	bpf_object__for_each_map(map, obj) {
-		if (!is_mmapable_map(map, map_ident, sizeof(map_ident)))
+		if (!is_skel_data(map, map_ident, sizeof(map_ident)))
 			continue;
 
Quoted 13 lines

tools/bpf/bpftool/gen.c static void codegen_destroy(struct bpf_object *obj, const char *obj_name) · line 697

Quoted 8 lines
tools/bpf/bpftool/gen.c +2 -1
 		sec = find_type_for_map(btf, map_ident);
@@ -669,7 +693,8 @@ static void codegen_destroy(struct bpf_object *obj, const char *obj_name)
 		if (!get_map_ident(map, ident, sizeof(ident)))
 			continue;
 		if (bpf_map__is_internal(map) &&
-		    (bpf_map__map_flags(map) & BPF_F_MMAPABLE))
+		    ((bpf_map__map_flags(map) & BPF_F_MMAPABLE) ||
+		     bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY))

I think this gives us a hint on how we could structure is_skel_data above, we now have a special calss that is "mappable maps + PERCPU_ARRAY whose initial representation in userspace is mappable, but the loaded map isn't".

Will use is_skel_data() here.

Thanks,
Leon

Quoted 122 lines
Quoted 48 lines
tools/bpf/bpftool/gen.c +17 -4
 			printf("\tskel_free_map_data(skel->%1$s, skel->maps.%1$s.initial_value, %2$zu);\n",
 			       ident, bpf_map_mmap_sz(map));
 		codegen("\
@@ -741,7 +766,7 @@ static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *h
 		const void *mmap_data = NULL;
 		size_t mmap_size = 0;
 
-		if (!is_mmapable_map(map, ident, sizeof(ident)))
+		if (!is_skel_data(map, ident, sizeof(ident)))
 			continue;
 
 		codegen("\
@@ -849,8 +874,22 @@ static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *h
 	bpf_object__for_each_map(map, obj) {
 		const char *mmap_flags;
 
-		if (!is_mmapable_map(map, ident, sizeof(ident)))
+		if (!is_skel_data(map, ident, sizeof(ident)))
+			continue;
+
+		if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY) {
+			codegen("\
+		\n\
+			err = skel_protect_map_data(skel->%1$s, &skel->maps.%1$s.initial_value, %2$zd);\n\
+			if (err)					    \n\
+				return err;				    \n\
+		#ifdef __KERNEL__					    \n\
+			skel->%1$s = NULL;				    \n\
+		#endif							    \n\
+			",
+			ident, bpf_map_mmap_sz(map));
 			continue;
+		}
 
 		if (bpf_map__map_flags(map) & BPF_F_RDONLY_PROG)
 			mmap_flags = "PROT_READ";
@@ -955,8 +994,7 @@ codegen_maps_skeleton(struct bpf_object *obj, size_t map_cnt, bool mmaped, bool
 				map->map = &obj->maps.%s;	    \n\
 			",
 			i, bpf_map__name(map), ident);
-		/* memory-mapped internal maps */
-		if (mmaped && is_mmapable_map(map, ident, sizeof(ident))) {
+		if (mmaped && is_skel_data(map, ident, sizeof(ident))) {
 			printf("\tmap->mmaped = (void **)&obj->%s;\n", ident);
 		}
 

tools/bpf/bpftool/gen.c static int do_subskeleton(int argc, char **argv) · line 1784

Quoted 8 lines
tools/bpf/bpftool/gen.c +3 -1
@@ -1740,7 +1778,9 @@ static int do_subskeleton(int argc, char **argv)
 		/* Also count all maps that have a name */
 		map_cnt++;
 
-		if (!is_mmapable_map(map, ident, sizeof(ident)))
+		if (!is_skel_data(map, ident, sizeof(ident)))
+			continue;
+		if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY)
 			continue;

Another instance of what I was talking about.

Quoted 62 lines
tools/bpf/bpftool/gen.c +3 -1
 		map_type_id = bpf_map__btf_value_type_id(map);
@@ -1863,7 +1903,9 @@ static int do_subskeleton(int argc, char **argv)
 
 	/* walk through each symbol and emit the runtime representation */
 	bpf_object__for_each_map(map, obj) {
-		if (!is_mmapable_map(map, ident, sizeof(ident)))
+		if (!is_skel_data(map, ident, sizeof(ident)))
+			continue;
+		if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY)
 			continue;
 
 		map_type_id = bpf_map__btf_value_type_id(map);
tools/lib/bpf/skel_internal.h +22 -2
index 53fee53d36d5..1f3f332dffbe 100644
@@ -131,8 +131,10 @@ static inline void skel_free_map_data(void *p, __u64 addr, size_t sz)
 {
 	if (addr != ~0ULL)
 		kvfree(p);
-	/* When addr == ~0ULL the 'p' points to
-	 * ((struct bpf_array *)map)->value. See skel_finalize_map_data.
+	/*
+	 * When addr == ~0ULL the init buffer has already been released.
+	 * For skel_finalize_map_data(), 'p' points to
+	 * ((struct bpf_array *)map)->value.
 	 */
 }
 
@@ -170,6 +172,15 @@ static inline void *skel_finalize_map_data(__u64 *init_val, size_t mmap_sz, int
 	return addr;
 }
 
+static inline int skel_protect_map_data(void *p, __u64 *init_val, size_t sz)
+{
+	(void)sz;
+
+	kvfree(p);
+	*init_val = ~0ULL;
+	return 0;
+}
+
 #else
 
 static inline void *skel_alloc(size_t size)
@@ -208,6 +219,15 @@ static inline void *skel_finalize_map_data(__u64 *init_val, size_t mmap_sz, int
 		return NULL;
 	return addr;
 }
+
+static inline int skel_protect_map_data(void *p, __u64 *init_val, size_t sz)
+{
+	(void)init_val;
+
+	if (mprotect(p, sz, PROT_READ))
+		return -errno;
+	return 0;
+}
 #endif
 
 static inline int skel_closenz(int fd)
author
#27 lore raw

Re: [PATCH bpf-next v10 6/9] selftests/bpf: Add tests to verify global percpu data

In reply to Emil Tsalapatis
Quoted 1 line · Emil Tsalapatis

On Wed Jul 15, 2026 at 11:32 AM EDT, Leon Hwang wrote:

[...]

Quoted 37 lines
Quoted 37 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +37 -0
+
+static void test_percpu_data_on_cpus(int map_fd, int prog_fd)
+{
+	__u64 args[2] = {0x1234ULL, 0x5678ULL};
+	LIBBPF_OPTS(bpf_test_run_opts, topts,
+		    .ctx_in = args,
+		    .ctx_size_in = sizeof(args),
+		    .flags = BPF_F_TEST_RUN_ON_CPU,
+	);
+	int i, err, key = 0, num_online;
+	bool *online;
+
+	err = parse_cpu_mask_file("/sys/devices/system/cpu/online", &online, &num_online);
+	if (!ASSERT_OK(err, "parse_cpu_mask_file"))
+		return;
+
+	/* run on every online-CPU */
+	for (i = 0; i < num_online; i++) {
+		struct test_global_percpu_data__percpu data = {};
+		__u64 flags;
+
+		if (!online[i])
+			continue;
+
+		topts.cpu = i;
+		topts.retval = -1;
+		err = bpf_prog_test_run_opts(prog_fd, &topts);
+		ASSERT_OK(err, "bpf_prog_test_run_opts");
+		ASSERT_EQ(topts.retval, 0, "bpf_prog_test_run_opts retval");
+
+		flags = ((__u64) i << 32) | BPF_F_CPU;
+		err = bpf_map_lookup_elem_flags(map_fd, &key, &data, flags);
+		if (!ASSERT_OK(err, "bpf_map_lookup_elem_flags"))
+			break;
+
+		ASSERT_EQ(data.data, 1, "data.data");
+		ASSERT_TRUE(data.run, "data.run");
Quoted 8 lines

tools/testing/selftests/bpf/prog_tests/global_data_init.c void test_global_data_init(void) · line 106

Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +4 -0
+		ASSERT_EQ(data.nums[6], 0xc0de, "data.nums[6]");
+		ASSERT_EQ(data.struct_data.i, 1, "struct_data.i");
+		ASSERT_TRUE(data.struct_data.set, "struct_data.set");
+		ASSERT_EQ(data.struct_data.nums[6], 0xc0de, "struct_data.nums[6]");

Can we add a pre-run assert to ensure that the per-cpu data has not already been modified by another run? Can we also add some cpuid specific assignment to ensure the runs are done on the proper CPU?

Ack.

Quoted 6 lines
Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +5 -0
+	}
+
+	free(online);
+}
+

[...]

Quoted 34 lines
Quoted 20 lines
tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c new file +14 -0
new file mode 100644
index 000000000000..8aebd533d86b
@@ -0,0 +1,37 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include "test_global_percpu_data.subskel.h"
+
+void test_global_percpu_data_subskel(void)
+{
+	struct test_global_percpu_data *subskel = NULL;
+	struct bpf_object *obj;
+	int i;
+
+	obj = bpf_object__open_file("./test_global_percpu_data.bpf.o", NULL);
+	if (!ASSERT_OK_PTR(obj, "bpf_object__open_file"))
+		return;
+

tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c line 24

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c new file +10 -0
+	subskel = test_global_percpu_data__open(obj);
+	if (!ASSERT_OK_PTR(subskel, "test_global_percpu_data__open"))
+		goto out;
+
+	if (!ASSERT_OK_PTR(subskel->subskel, "subskel"))
+		goto out;
+	if (!ASSERT_OK_PTR(subskel->maps.percpu, "maps.percpu"))
+		goto out;
+	ASSERT_EQ(bpf_map__type(subskel->maps.percpu), BPF_MAP_TYPE_PERCPU_ARRAY,
+		  "percpu_map_type");

Not sure why these assertions would be necessary, wouldn't the test crash or quickly fail if they didn't hold?

Quoted 12 lines

tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c line 32

Quoted 8 lines
tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c new file +8 -0
+	ASSERT_GT(subskel->subskel->var_cnt, 0, "var_cnt");
+
+	for (i = 0; i < subskel->subskel->var_cnt; i++) {
+		const struct bpf_var_skeleton *var;
+
+		var = (void *) subskel->subskel->vars + i * subskel->subskel->var_skel_sz;
+		ASSERT_NEQ(var->map, &subskel->maps.percpu, "var");
+	}

If we turn the substest to SYSCALL instead of TRACEPOINT we can read and report the return value as the test happens and keep them self-contained.

It is to verify that the bpftool-generated subskeleton should not contain the global percpu variables.

I think I should test it manually by checking the generated .subskel.h.

Will drop this subskel test.

Quoted 17 lines
Quoted 17 lines
tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c new file +5 -0
+
+out:
+	test_global_percpu_data__destroy(subskel);
+	bpf_object__close(obj);
+}
tools/testing/selftests/bpf/progs/test_global_percpu_data.c new file +5 -0
new file mode 100644
index 000000000000..54380dfb11a5
@@ -0,0 +1,33 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
Quoted 8 lines

tools/testing/selftests/bpf/progs/test_global_percpu_data.c line 10

Quoted 5 lines
tools/testing/selftests/bpf/progs/test_global_percpu_data.c new file +5 -0
+int unused SEC(".percpu.looooooooong");
+int data2 SEC(".percpu.data");
+int data SEC(".percpu") = -1;
+int nums[7] SEC(".percpu");
+char run SEC(".percpu") = 0;

Can we add a comment that those are used to test the names in the userspace portion of the test? As they stand they seem unused.

Ack.

Quoted 14 lines

tools/testing/selftests/bpf/progs/test_global_percpu_data.c line 20

Quoted 11 lines
tools/testing/selftests/bpf/progs/test_global_percpu_data.c new file +10 -0
+struct {
+	char set;
+	int i;
+	int nums[7];
+} struct_data SEC(".percpu") = {
+	.set = 0,
+	.i = -1,
+};
+
+SEC("raw_tp/task_rename")

Is this actually installed as a tracepoint? I think we run it as a program. Is putting it in raw_tp doing anything in any way?

Will drop '/task_rename'.

Thanks,
Leon

Quoted 15 lines
Quoted 14 lines
tools/testing/selftests/bpf/progs/test_global_percpu_data.c new file +13 -0
+__auxiliary
+int update_percpu_data(void *ctx)
+{
+	struct_data.nums[6] = 0xc0de;
+	struct_data.set = 1;
+	struct_data.i = 1;
+	nums[6] = 0xc0de;
+	data = 1;
+	run = 1;
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
author
#30 lore raw

Re: [PATCH bpf-next v10 7/9] selftests/bpf: Test direct reading/writing read-only percpu_array map

In reply to Emil Tsalapatis
Quoted 53 lines · Emil Tsalapatis
Quoted 52 lines · Leon Hwang

Verify these two cases:

1. Direct reading the data of read-only percpu data's percpu_array map is allowed.
2. Direct writing the data of read-only percpu data's percpu_array map is disallowed.

Assisted-by Codex:gpt-5.5-xhigh
---
 .../bpf/prog_tests/global_data_init.c         | 83 +++++++++++++++++++
 1 file changed, 83 insertions(+)
tools/testing/selftests/bpf/prog_tests/global_data_init.c +31 -0
index f7a56e3fdbce..d4405646847a 100644
@@ -199,6 +199,85 @@ static void test_global_percpu_data_lskel(void)
 	test_global_percpu_data_lskel__destroy(lskel);
 }
 
+static int create_rdonly_percpu_array(void)
+{
+	LIBBPF_OPTS(bpf_map_create_opts, map_opts,
+		    .map_flags = BPF_F_RDONLY_PROG,
+	);
+	int key = 0, map_fd, err;
+	__u64 value = 0;
+
+	map_fd = bpf_map_create(BPF_MAP_TYPE_PERCPU_ARRAY, "percpu_ro_map", sizeof(int),
+				sizeof(__u64), 1, &map_opts);
+	if (!ASSERT_GE(map_fd, 0, "bpf_map_create"))
+		return -1;
+
+	err = bpf_map_update_elem(map_fd, &key, &value, BPF_F_ALL_CPUS);
+	if (!ASSERT_OK(err, "bpf_map_update_elem"))
+		goto out;
+
+	err = bpf_map_freeze(map_fd);
+	if (!ASSERT_OK(err, "bpf_map_freeze"))
+		goto out;
+
+	return map_fd;
+
+out:
+	close(map_fd);
+	return -1;
+}
+
+static void test_global_percpu_data_rdonly_direct_read(void)
+{
+	struct bpf_insn insns[] = {
Quoted 8 lines

tools/testing/selftests/bpf/prog_tests/global_data_init.c static void test_global_percpu_data_lskel(void) · line 236

Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +4 -0
+		BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0),
+		BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0),
+		BPF_EXIT_INSN(),
+	};

Why hardcode this as inline assembly? Can we do a syscall example in the progs/? That way there will also be a concrete example of how we're using the feature in practice.

We are unable to implement it with a syscall example, because libbpf does not support creating a rdonly internal map for global percpu data. We have no '.ropercpu' section support in libbpf.

Will add comment to explain the reason of hardcoded insns.

Thanks,
Leon

Quoted 1 line

[...]

#8 lore raw
In reply to Leon Hwang
Quoted 87 lines · Leon Hwang
Quoted 1 line · Emil Tsalapatis

On Wed Jul 15, 2026 at 11:32 AM EDT, Leon Hwang wrote:

[...]

Quoted 14 lines

kernel/bpf/arraymap.c static void *percpu_array_map_lookup_elem(struct bpf_map *map, void *key) · line 271

Quoted 11 lines
kernel/bpf/arraymap.c +10 -0
+static int percpu_array_map_direct_value_addr(const struct bpf_map *map, u64 *imm, u32 off)
+{
+	struct bpf_array *array = container_of(map, struct bpf_array, map);
+
+	if (map->max_entries != 1)
+		return -EOPNOTSUPP;
+	if (off >= map->value_size)
+		return -EINVAL;
+	if (!bpf_jit_supports_percpu_insn())
+		return -EOPNOTSUPP;

Nit: This should be first, if the JIT doesn't support the instruction all else is moot. Same below.

Ack.

Quoted 22 lines
Quoted 22 lines
kernel/bpf/arraymap.c +21 -0
+
+	*imm = (u64)(__force unsigned long) array->pptrs[0];
+	return 0;
+}
+
+static int percpu_array_map_direct_value_meta(const struct bpf_map *map, u64 imm, u32 *off)
+{
+	struct bpf_array *array = container_of(map, struct bpf_array, map);
+	u64 base = (u64)(__force unsigned long) array->pptrs[0];
+
+	if (map->max_entries != 1)
+		return -EOPNOTSUPP;
+	if (imm < base || imm >= base + array->elem_size)
+		return -ENOENT;
+	if (!bpf_jit_supports_percpu_insn())
+		return -EOPNOTSUPP;
+
+	*off = imm - base;
+	return 0;
+}
+

[...]

Quoted 27 lines
Quoted 14 lines
kernel/bpf/verifier.c +2 -0
index e1244a721194..ae442ea217c4 100644
@@ -5615,6 +5615,8 @@ int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val,
 	u64 addr;
 	int err;
 
+	if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY || map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY)
+		return -EINVAL;
 	err = map->ops->map_direct_value_addr(map, &addr, off);
 	if (err)
 		return err;

kernel/bpf/verifier.c static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct b · line 6180

Quoted 5 lines
kernel/bpf/verifier.c +1 -0
@@ -6174,6 +6176,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct b
 			if (tnum_is_const(reg->var_off) &&
 			    bpf_map_is_rdonly(map) &&
 			    map->ops->map_direct_value_addr &&
+			    map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY &&
 			    map->map_type != BPF_MAP_TYPE_INSN_ARRAY) {

An issue I have with the patch is that it takes the special-casing we already have for INSN_ARRAY and makes it worse. The instruction arrays implement direct_value_addr, but it's not actually valid to call direct_value_addr from a bunch of places where it's called, so we have to special case check the map type to prevent it. Case in point the special casing in check_arg_const_str in the thunk below.

Quoted 4 lines

Imo we should consider making a different map method different from direct_value_addr that INSN_ARRAY and PERCPU_ARRAY implement. That way we remove most of the map type checks in those paths. I'm not exactly sure how that would look like, but I could take a look at it tomorrow.

Seems reasonable to me. I don't have good candidates for the new map method name.

Instead of making a new map method, how about factoring out a helper for the 'if (t == BPF_READ && value_regno >= 0) {}' block? It will simplify these map_type checks in the helper.

[...]

Quoted 30 lines
Quoted 20 lines
Quoted 20 lines
kernel/bpf/verifier.c +6 -0
 				int map_off = off + reg->var_off.value;
 				u64 val = 0;
@@ -8176,6 +8179,12 @@ static int check_arg_const_str(struct bpf_verifier_env *env,
 		return -EACCES;
 	}
 
+	if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) {
+		verbose(env, "%s points to percpu_array map which cannot be used as const string\n",
+			reg_arg_name(env, argno));
+		return -EACCES;
+	}
+
 	if (!bpf_map_is_rdonly(map)) {
 		verbose(env, "%s does not point to a readonly map'\n", reg_arg_name(env, argno));
 		return -EACCES;
@@ -18279,6 +18288,12 @@ static int check_and_resolve_insns(struct bpf_verifier_env *env)
 					return -EINVAL;
 				}
 
Quoted 8 lines

kernel/bpf/verifier.c static int check_and_resolve_insns(struct bpf_verifier_env *env) · line 18296

Quoted 6 lines
kernel/bpf/verifier.c +6 -0
+				if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY &&
+				    !env->prog->jit_requested) {
+					verbose(env, "JIT is required to use global percpu data\n");
+					return -EOPNOTSUPP;
+				}
+

Another place where we add map-specific checks where there are none.

Quoted 4 lines

I think this check is unnecessary, because 'prog->jit_required = true' has been set for global percpu data.

Will drop this if.

I think you're right, in which case we can drop it. In turn this means wrapping the check above in a helper adds enough structure to the logic. Adding the new method would still work valid but it'd be more borderline in terms of simplifying the code because we''ll have abstracted most of the type checks away.

Quoted 9 lines

Thanks,
Leon

Quoted 5 lines
Quoted 4 lines
kernel/bpf/verifier.c +0 -0
 				err = map->ops->map_direct_value_addr(map, &addr, off);
 				if (err) {
 					verbose(env, "invalid access to map value pointer, value_size=%u off=%u\n",
author
#4 lore raw

Re: [PATCH bpf-next v10 1/9] bpf: Drop duplicate blank lines in verifier

In reply to Leon Hwang
Quoted 15 lines · Leon Hwang
Quoted 10 lines · Emil Tsalapatis
Quoted 6 lines · Leon Hwang

There are many adjacent blank lines in the verifier that have accumulated over time.

Drop them for cleanup.

No functional changes intended.

Thanks for your review.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

What about the files we factored out of the verifier? E.g., backtrack.c also has the same issue.

Probably, I can post a patch to clean all adjacent blank lines in kernel/bpf/ when posting another series.

Will drop all adjacent blank lines in kernel/bpf/ in the next revision, as I'm planning to post the next revision.

Thanks,
Leon

author
#16 lore raw

Re: [PATCH bpf-next v10 4/9] libbpf: Add support for global percpu data

In reply to Leon Hwang
Quoted 183 lines · Leon Hwang
Quoted 1 line · Emil Tsalapatis

On Wed Jul 15, 2026 at 11:32 AM EDT, Leon Hwang wrote:

[...]

Quoted 30 lines
Quoted 12 lines

struct elf_sec_desc {

tools/lib/bpf/libbpf.c +2 -0
@@ -1839,6 +1842,8 @@ static size_t bpf_map_mmap_sz(const struct bpf_map *map)
 	switch (map->def.type) {
 	case BPF_MAP_TYPE_ARRAY:
 		return array_map_mmap_sz(map->def.value_size, map->def.max_entries);
+	case BPF_MAP_TYPE_PERCPU_ARRAY:
+		return map->def.value_size;
 	case BPF_MAP_TYPE_ARENA:
 		return page_sz * map->def.max_entries;
 	default:

tools/lib/bpf/libbpf.c static int bpf_map_mmap_resize(struct bpf_map *map, size_t old_sz, size_t new_sz · line 1875

Quoted 6 lines
tools/lib/bpf/libbpf.c +2 -1
@@ -1866,7 +1871,8 @@ static int bpf_map_mmap_resize(struct bpf_map *map, size_t old_sz, size_t new_sz
 	return 0;
 }
 
-static char *internal_map_name(struct bpf_object *obj, const char *real_name)
+static char *internal_map_name(struct bpf_object *obj, const char *real_name,
+			       enum libbpf_map_type type)

We can avoid passing the type here by testing against ".percpu" since that's how we are deriving the map type in the first place. But more importantly:

Quoted 7 lines
tools/lib/bpf/libbpf.c +0 -0
 {
 	char map_name[BPF_OBJ_NAME_LEN], *p;
 	int pfx_len, sfx_len = max((size_t)7, strlen(real_name));
@@ -1907,8 +1913,11 @@ static char *internal_map_name(struct bpf_object *obj, const char *real_name)
 	if (sfx_len >= BPF_OBJ_NAME_LEN)
 		sfx_len = BPF_OBJ_NAME_LEN - 1;
 
Quoted 15 lines

tools/lib/bpf/libbpf.c static char *internal_map_name(struct bpf_object *obj, const char *real_name) · line 1920

Quoted 7 lines
tools/lib/bpf/libbpf.c +5 -2
-	/* if there are two or more dots in map name, it's a custom dot map */
-	if (strchr(real_name + 1, '.') != NULL)
+	/*
+	 * Don't prefix the bpf_object name if this is a custom dot map
+	 * (containing two or more dots) or a percpu data map.
+	 */
+	if (strchr(real_name + 1, '.') != NULL || type == LIBBPF_MAP_PERCPU)

Is there a reason we don't use the exact same logic as the other internal maps here? I understand that a lot of the conventions around the naming are there for legacy reasons, but it seems like we're singling out the .percpu section for highly nonbvious reasons. Imo we should consider doing the same prefixing for a bare ".percpu" section that we do for the other internal ones. At the very least, there needs to be some explanation as to why .percpu gets special treatment.

I prefer passing 'type'. Excluding _PERCPU here is to avoid the legacy naming convention for new internal maps.

Quoted 10 lines

@Andrii Wdyt?

Quoted 7 lines
tools/lib/bpf/libbpf.c +0 -0
 		pfx_len = 0;
 	else
 		pfx_len = min((size_t)BPF_OBJ_NAME_LEN - sfx_len - 1, strlen(obj->name));
@@ -1938,7 +1947,7 @@ static bool map_is_mmapable(struct bpf_object *obj, struct bpf_map *map)
 	struct btf_var_secinfo *vsi;
 	int i, n;
 
Quoted 8 lines

tools/lib/bpf/libbpf.c static bool map_is_mmapable(struct bpf_object *obj, struct bpf_map *map) · line 1951

Quoted 3 lines
tools/lib/bpf/libbpf.c +1 -1
-	if (!map->btf_value_type_id)
+	if (!map->btf_value_type_id || map->libbpf_type == LIBBPF_MAP_PERCPU)
 		return false;

Nit: These are two separate checks rolled into one, and each one checks a different thing. THe type check against MAP_PERCPU merits a comment as
well: It's the only internal section that is not really mappable because there's no way to represent it as userspace state.

Ack.

Will add a new iff for libbpf_type check with a comment.

Quoted 11 lines
Quoted 11 lines
tools/lib/bpf/libbpf.c +1 -0
 	t = btf__type_by_id(obj->btf, map->btf_value_type_id);
@@ -1962,6 +1971,7 @@ static int
 bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
 			      const char *real_name, int sec_idx, void *data, size_t data_sz)
 {
+	bool is_percpu = type == LIBBPF_MAP_PERCPU;
 	struct bpf_map_def *def;
 	struct bpf_map *map;
 	size_t mmap_sz;

[...]

Quoted 36 lines
Quoted 10 lines
tools/lib/bpf/libbpf.c +1 -1
@@ -4944,7 +4970,7 @@ static int map_fill_btf_type_info(struct bpf_object *obj, struct bpf_map *map)
 
 	/*
 	 * LLVM annotates global data differently in BTF, that is,
-	 * only as '.data', '.bss' or '.rodata'.
+	 * only as '.data', '.bss', '.percpu' or '.rodata'.
 	 */
 	if (!bpf_map__is_internal(map))
 		return -ENOENT;

tools/lib/bpf/libbpf.c static int · line 5322

Quoted 4 lines
tools/lib/bpf/libbpf.c +1 -0
@@ -5293,18 +5319,30 @@ static int
 bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map)
 {
 	enum libbpf_map_type map_type = map->libbpf_type;
+	bool is_percpu = map_type == LIBBPF_MAP_PERCPU;

Nit: If we do

	__u64 update_flags = is_percpu ? BPF_F_ALL_CPUS : 0;

we can declare the variable as const and ...

tools/lib/bpf/libbpf.c static int · line 5335

Quoted 13 lines
tools/lib/bpf/libbpf.c +10 -0
+	__u64 update_flags = 0;
 	int err, zero = 0;
 	size_t mmap_sz;
 
+	if (is_percpu) {
+		if (!obj->gen_loader && !kernel_supports(obj, FEAT_PERCPU_DATA)) {
+			pr_warn("map '%s': kernel does not support percpu data.\n",
+				bpf_map__name(map));
+			return -EOPNOTSUPP;
+		}
+
+		update_flags = BPF_F_ALL_CPUS;
+	}

... we can collapse the above into a single nested level:

Quoted 3 lines
	if (is_percpu && !obj->gen_loader && !kernel_supports(obj, FEAT_PERCPU_DATA)) {
		...
	}

Good point.

Quoted 39 lines
Quoted 39 lines
tools/lib/bpf/libbpf.c +13 -2
+
 	if (obj->gen_loader) {
 		bpf_gen__map_update_elem(obj->gen_loader, map - obj->maps,
-					 map->mmaped, map->def.value_size);
+					 map->mmaped, map->def.value_size, update_flags);
 		if (map_type == LIBBPF_MAP_RODATA || map_type == LIBBPF_MAP_KCONFIG)
 			bpf_gen__map_freeze(obj->gen_loader, map - obj->maps);
 		return 0;
 	}
 
-	err = bpf_map_update_elem(map->fd, &zero, map->mmaped, 0);
+	err = bpf_map_update_elem(map->fd, &zero, map->mmaped, update_flags);
 	if (err) {
 		err = -errno;
 		pr_warn("map '%s': failed to set initial contents: %s\n",
@@ -5349,6 +5387,13 @@ bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map)
 			return err;
 		}
 		map->mmaped = mmaped;
+	} else if (is_percpu) {
+		if (mprotect(map->mmaped, mmap_sz, PROT_READ)) {
+			err = -errno;
+			pr_warn("map '%s': failed to mprotect() contents: %s\n",
+				bpf_map__name(map), errstr(err));
+			return err;
+		}
 	} else if (map->mmaped) {
 		munmap(map->mmaped, mmap_sz);
 		map->mmaped = NULL;
@@ -10807,11 +10852,16 @@ static bool map_uses_real_name(const struct bpf_map *map)
 	 * such map's corresponding ELF section name as a map name.
 	 * This check distinguishes .data/.rodata from .data.* and .rodata.*
 	 * maps to know which name has to be returned to the user.
+	 * Map name of the custom .percpu.* maps might be truncated to
+	 * BPF_OBJ_NAME_LEN-1 chars in internal_map_name(). Hence, percpu data
+	 * maps must use real name for their user-visible name.
 	 */
 	if (map->libbpf_type == LIBBPF_MAP_DATA && strcmp(map->real_name, DATA_SEC) != 0)
Quoted 8 lines

tools/lib/bpf/libbpf.c static bool map_uses_real_name(const struct bpf_map *map) · line 10864

Quoted 5 lines
tools/lib/bpf/libbpf.c +2 -0
 		return true;
 	if (map->libbpf_type == LIBBPF_MAP_RODATA && strcmp(map->real_name, RODATA_SEC) != 0)
 		return true;
+	if (map->libbpf_type == LIBBPF_MAP_PERCPU)
+		return true;

Same comment as above here wrt uniformity. This is the part that requires us to check agianst the map type in __init_internal_map().

Quoted 2 lines

Let us wait for Andrii's comment.

Hi Andrii,

Any comment?

Thanks,
Leon

#17 lore raw

Re: [PATCH bpf-next v10 4/9] libbpf: Add support for global percpu data

In reply to Leon Hwang
Quoted 21 lines · Leon Hwang
Quoted 1 line · Emil Tsalapatis

On Wed Jul 15, 2026 at 11:32 AM EDT, Leon Hwang wrote:

[...]

Quoted 15 lines
Quoted 15 lines

struct elf_sec_desc {

tools/lib/bpf/libbpf.c +2 -0
@@ -1839,6 +1842,8 @@ static size_t bpf_map_mmap_sz(const struct bpf_map *map)
     switch (map->def.type) {
     case BPF_MAP_TYPE_ARRAY:
             return array_map_mmap_sz(map->def.value_size, map->def.max_entries);
+    case BPF_MAP_TYPE_PERCPU_ARRAY:
+            return map->def.value_size;
     case BPF_MAP_TYPE_ARENA:
             return page_sz * map->def.max_entries;
     default:
@@ -1866,7 +1871,8 @@ static int bpf_map_mmap_resize(struct bpf_map *map, size_t old_sz, size_t new_sz
     return 0;
 }
 
Quoted 8 lines
Quoted 8 lines

tools/lib/bpf/libbpf.c static int bpf_map_mmap_resize(struct bpf_map *map, size_t old_sz, size_t new_sz · line 1875

Quoted 3 lines
tools/lib/bpf/libbpf.c +2 -1
-static char *internal_map_name(struct bpf_object *obj, const char *real_name)
+static char *internal_map_name(struct bpf_object *obj, const char *real_name,
+                           enum libbpf_map_type type)

We can avoid passing the type here by testing against ".percpu" since that's how we are deriving the map type in the first place. But more importantly:

I think type is cleaner, because it's not just .percpu, but also .percpu.whateveryouwant, so having type is cleaner, IMO.

Quoted 24 lines
Quoted 7 lines
Quoted 7 lines
tools/lib/bpf/libbpf.c +0 -0
 {
     char map_name[BPF_OBJ_NAME_LEN], *p;
     int pfx_len, sfx_len = max((size_t)7, strlen(real_name));
@@ -1907,8 +1913,11 @@ static char *internal_map_name(struct bpf_object *obj, const char *real_name)
     if (sfx_len >= BPF_OBJ_NAME_LEN)
             sfx_len = BPF_OBJ_NAME_LEN - 1;
 
Quoted 15 lines

tools/lib/bpf/libbpf.c static char *internal_map_name(struct bpf_object *obj, const char *real_name) · line 1920

Quoted 7 lines
tools/lib/bpf/libbpf.c +5 -2
-    /* if there are two or more dots in map name, it's a custom dot map */
-    if (strchr(real_name + 1, '.') != NULL)
+    /*
+     * Don't prefix the bpf_object name if this is a custom dot map
+     * (containing two or more dots) or a percpu data map.
+     */
+    if (strchr(real_name + 1, '.') != NULL || type == LIBBPF_MAP_PERCPU)

Is there a reason we don't use the exact same logic as the other internal maps here? I understand that a lot of the conventions around the naming are there for legacy reasons, but it seems like we're singling out the .percpu section for highly nonbvious reasons. Imo we should consider doing the same prefixing for a bare ".percpu" section that we do for the other internal ones. At the very least, there needs to be some explanation as to why .percpu gets special treatment.

Quoted 2 lines

I prefer passing 'type'. Excluding _PERCPU here is to avoid the legacy naming convention for new internal maps.

+1

and it's not that .percpu gets special treatment, it's all the legacy maps that have special treatment

Quoted 21 lines
Quoted 11 lines

@Andrii Wdyt?

Quoted 7 lines
tools/lib/bpf/libbpf.c +0 -0
             pfx_len = 0;
     else
             pfx_len = min((size_t)BPF_OBJ_NAME_LEN - sfx_len - 1, strlen(obj->name));
@@ -1938,7 +1947,7 @@ static bool map_is_mmapable(struct bpf_object *obj, struct bpf_map *map)
     struct btf_var_secinfo *vsi;
     int i, n;
 
Quoted 8 lines

tools/lib/bpf/libbpf.c static bool map_is_mmapable(struct bpf_object *obj, struct bpf_map *map) · line 1951

Quoted 3 lines
tools/lib/bpf/libbpf.c +1 -1
-    if (!map->btf_value_type_id)
+    if (!map->btf_value_type_id || map->libbpf_type == LIBBPF_MAP_PERCPU)
             return false;

Nit: These are two separate checks rolled into one, and each one checks a different thing. THe type check against MAP_PERCPU merits a comment as
well: It's the only internal section that is not really mappable because there's no way to represent it as userspace state.

Quoted 3 lines

Ack.

Will add a new iff for libbpf_type check with a comment.

yep, but don't overdo comments, it's not that hard to understand why per-cpu map is not mmapable

Quoted 105 lines
Quoted 12 lines
Quoted 12 lines
tools/lib/bpf/libbpf.c +1 -0
     t = btf__type_by_id(obj->btf, map->btf_value_type_id);
@@ -1962,6 +1971,7 @@ static int
 bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
                           const char *real_name, int sec_idx, void *data, size_t data_sz)
 {
+    bool is_percpu = type == LIBBPF_MAP_PERCPU;
     struct bpf_map_def *def;
     struct bpf_map *map;
     size_t mmap_sz;

[...]

Quoted 34 lines
Quoted 10 lines
tools/lib/bpf/libbpf.c +1 -1
@@ -4944,7 +4970,7 @@ static int map_fill_btf_type_info(struct bpf_object *obj, struct bpf_map *map)
 
     /*
      * LLVM annotates global data differently in BTF, that is,
-     * only as '.data', '.bss' or '.rodata'.
+     * only as '.data', '.bss', '.percpu' or '.rodata'.
      */
     if (!bpf_map__is_internal(map))
             return -ENOENT;

tools/lib/bpf/libbpf.c static int · line 5322

Quoted 4 lines
tools/lib/bpf/libbpf.c +1 -0
@@ -5293,18 +5319,30 @@ static int
 bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map)
 {
     enum libbpf_map_type map_type = map->libbpf_type;
+    bool is_percpu = map_type == LIBBPF_MAP_PERCPU;

Nit: If we do

      __u64 update_flags = is_percpu ? BPF_F_ALL_CPUS : 0;

we can declare the variable as const and ...

tools/lib/bpf/libbpf.c static int · line 5335

Quoted 13 lines
tools/lib/bpf/libbpf.c +10 -0
+    __u64 update_flags = 0;
     int err, zero = 0;
     size_t mmap_sz;
 
+    if (is_percpu) {
+            if (!obj->gen_loader && !kernel_supports(obj, FEAT_PERCPU_DATA)) {
+                    pr_warn("map '%s': kernel does not support percpu data.\n",
+                            bpf_map__name(map));
+                    return -EOPNOTSUPP;
+            }
+
+            update_flags = BPF_F_ALL_CPUS;
+    }
Quoted 5 lines

... we can collapse the above into a single nested level:

      if (is_percpu && !obj->gen_loader && !kernel_supports(obj, FEAT_PERCPU_DATA)) {
              ...
      }

Good point.

Quoted 39 lines
Quoted 39 lines
tools/lib/bpf/libbpf.c +13 -2
+
     if (obj->gen_loader) {
             bpf_gen__map_update_elem(obj->gen_loader, map - obj->maps,
-                                     map->mmaped, map->def.value_size);
+                                     map->mmaped, map->def.value_size, update_flags);
             if (map_type == LIBBPF_MAP_RODATA || map_type == LIBBPF_MAP_KCONFIG)
                     bpf_gen__map_freeze(obj->gen_loader, map - obj->maps);
             return 0;
     }
 
-    err = bpf_map_update_elem(map->fd, &zero, map->mmaped, 0);
+    err = bpf_map_update_elem(map->fd, &zero, map->mmaped, update_flags);
     if (err) {
             err = -errno;
             pr_warn("map '%s': failed to set initial contents: %s\n",
@@ -5349,6 +5387,13 @@ bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map)
                     return err;
             }
             map->mmaped = mmaped;
+    } else if (is_percpu) {
+            if (mprotect(map->mmaped, mmap_sz, PROT_READ)) {
+                    err = -errno;
+                    pr_warn("map '%s': failed to mprotect() contents: %s\n",
+                            bpf_map__name(map), errstr(err));
+                    return err;
+            }
     } else if (map->mmaped) {
             munmap(map->mmaped, mmap_sz);
             map->mmaped = NULL;
@@ -10807,11 +10852,16 @@ static bool map_uses_real_name(const struct bpf_map *map)
      * such map's corresponding ELF section name as a map name.
      * This check distinguishes .data/.rodata from .data.* and .rodata.*
      * maps to know which name has to be returned to the user.
+     * Map name of the custom .percpu.* maps might be truncated to
+     * BPF_OBJ_NAME_LEN-1 chars in internal_map_name(). Hence, percpu data
+     * maps must use real name for their user-visible name.
      */
     if (map->libbpf_type == LIBBPF_MAP_DATA && strcmp(map->real_name, DATA_SEC) != 0)
Quoted 8 lines

tools/lib/bpf/libbpf.c static bool map_uses_real_name(const struct bpf_map *map) · line 10864

Quoted 5 lines
tools/lib/bpf/libbpf.c +2 -0
             return true;
     if (map->libbpf_type == LIBBPF_MAP_RODATA && strcmp(map->real_name, RODATA_SEC) != 0)
             return true;
+    if (map->libbpf_type == LIBBPF_MAP_PERCPU)
+            return true;

Same comment as above here wrt uniformity. This is the part that requires us to check agianst the map type in __init_internal_map().

Quoted 2 lines

Let us wait for Andrii's comment.

I think it's fine and basically inevitable

Quoted 9 lines

Thanks,
Leon

Quoted 5 lines
Quoted 4 lines
tools/lib/bpf/libbpf.c +0 -0
     return false;
 }