Skip to contentSkip to conversation

bpf: Introduce global percpu data

PATCH Superseded Leon Hwang 3 participants 23 messages Jun 15 – Jun 20

Subsystems (6)
Kselftest BPF BPF selftests BPF core +2 more subsystems

v6 of 12 Superseded Latest is v12 →

lore
Share & tools
lore

Leon Hwang

author
#0 lore raw

v6 cover bpf: Introduce global percpu data

This patch set introduces global percpu data, similar to commit 6316f78306c1 ("Merge branch 'support-global-data'"), to reduce restrictions in C for BPF programs.

With this enhancement, it becomes possible to define and use global percpu variables, like the DEFINE_PER_CPU() macro in the kernel include/linux/percpu-defs.h.

The section name for global peurcpu data is ".percpu". Even though, a one-byte percpu variable (e.g., char run SEC(".percpu") = 0;) can trigger a crash with Clang 17 [1], users are expected to use such small variables as global percpu data with newer Clang versions, which don't have the issue.

The idea stems from the bpfsnoop [2], which itself was inspired by retsnoop [3]. During testing of bpfsnoop on the v6.6 kernel, two LBR (Last Branch Record) entries were observed related to the bpf_get_smp_processor_id() helper.

Since commit 1ae6921009e5 ("bpf: inline bpf_get_smp_processor_id() helper"), the bpf_get_smp_processor_id() helper has been inlined on x86_64, reducing the overhead and consequently minimizing these two LBR records.

However, the introduction of global percpu data offers a more robust solution. By leveraging the percpu_array map and percpu instruction, global percpu data can be implemented intrinsically.

This feature also facilitates sharing percpu information between tail callers and callees or between freplace callers and callees through a shared global percpu variable. Previously, this was achieved using a 1-entry percpu_array map, which this patch set aims to improve upon.

Links:
[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://github.com/bpfsnoop/bpfsnoop
[3] https://github.com/anakryiko/retsnoop

Changes:
v5 -> v6:
* Prevent running user addr_space_cast and addr_percpu insns in interpreter. (sashiko-bot)
* Cast __percpu pointer to u64 with (__force unsigned long). (lkp)
* Exclude BPF_MAP_TYPE_PERCPU_ARRAY in check_mem_access() before calling bpf_map_direct_read(), and add a test to verify it. (sashiko-bot, bot+bpf-ci)
* Skip percpu data variables for subskeleton in bpftool. (sashiko-bot)
* Protect skel->percpu using mprotect(..., PROT_READ) in light skeleton. (sashiko-bot, bot+bpf-ci)
* Drop roundup() in tests. (sashiko-bot)
* Call test_global_percpu_data_verifier_log() without test__start_subtest(). (sashiko-bot)
* Cast insn->imm to __u64 with (__u32) in xlated test. (sashiko-bot)
* Check cnt using the new idx in xlated test. (sashiko-bot)
* v5: https://lore.kernel.org/bpf/[email protected]/

v4 -> v5:
* Add prog->jit_requested check to prevent running percpu data in interpreter in patch #1.
* Factor out verifier log tests using its own patch.
* Address comments from Alexei:
* Move map_type check from check_mem_access() to bpf_map_direct_read() in patch #2.
* Move BPF_MAP_TYPE_INSN_ARRAY map_type check from const_reg_xfer() to bpf_map_direct_read() in patch #2.
* Add a test to verify that the off of xlated ldimm64 insn matches the off encoded in the ELF ldimm64 insn.
* Drop patch #5 of v4.
* Address reviews from sashiko-bot:
* Update commit message of patch #6 to indicate that maps.percpu->mmaped has been marked as read-only in libbpf.
* Lookup elem on specified CPU using BPF_F_CPU in tests.
* Drop unnecessary err == -EOPNOTSUPP in test.
* Locate target field using its offset in the iter test.
* v4: https://lore.kernel.org/bpf/[email protected]/

v3 -> v4:
* Drop duplicate blank lines in verifier.
* Add percpu data feature probe in libbpf.
* Update percpu_array map using BPF_F_ALL_CPUS flag for lskel, if no cpu flag is set.
* Add two tests to verify verifier log.
* Add a test to verify mov64_percpu_reg instruction.
* Add a test to verify bpf_iter for percpu data map.
* Update percpu_array map using BPF_F_ALL_CPUS flag in libbpf (per Alexei and Andrii).
* Address comments from Andrii:
* Use .percpu as section identifier.
* Use bpf_jit_supports_percpu_insn() instead of CONFIG_SMP.
* Drop bpf_map__is_internal_percpu() API.
* Drop unnecessary __aligned(8) in libbpf, verified by selftest.
* Make mmap data read-only after loading prog.
v3: https://lore.kernel.org/bpf/[email protected]/

v2 -> v3:
* Use ".data..percpu" as PERCPU_DATA_SEC.
* Address comment from Alexei:
* Add u8, array of ints and struct { .. } vars to selftest.
v2: https://lore.kernel.org/bpf/[email protected]/

v1 -> v2:
* Address comments from Andrii:
* Use LIBBPF_MAP_PERCPU and SEC_PERCPU.
* Reuse mmaped of libbpf's struct bpf_map for .percpu map data.
* Set .percpu struct pointer to NULL after loading skeleton.
* Make sure value size of .percpu map is __aligned(8).
* Use raw_tp and opts.cpu to test global percpu variables on all CPUs.
* Address comments from Alexei:
* Test non-zero offset of global percpu variable.
* Test case about BPF_PSEUDO_MAP_IDX_VALUE.
v1: https://lore.kernel.org/bpf/[email protected]/

rfc -> v1:
* Address comments from Andrii:
* Keep one image of global percpu variable for all CPUs.
* Reject non-ARRAY map in bpf_map_direct_read(), check_reg_const_str(), and check_bpf_snprintf_call() in verifier.
* Split out libbpf changes from kernel-side changes.
* Use ".percpu" as PERCPU_DATA_SEC.
* Use enum libbpf_map_type to distinguish BSS, DATA, RODATA and PERCPU_DATA.
* Avoid using errno for checking err from libbpf_num_possible_cpus().
* Use "map '%s': " prefix for error message.
rfc: https://lore.kernel.org/bpf/[email protected]/

Leon Hwang (12):
bpf: Drop duplicate blank lines in verifier
bpf: Disallow interpreter fallback for user BPF_ADDR_SPACE_CAST insn
bpf: Disallow interpreter fallback for BPF_ADDR_PERCPU insn
bpf: Introduce global percpu data
libbpf: Probe percpu data feature
libbpf: Add support for global percpu data
bpftool: Generate skeleton for global percpu data selftests/bpf: Add tests to verify global percpu data selftests/bpf: Add test to verify accessing rdonly percpu_array selftests/bpf: Add tests to verify verifier log for global percpu data selftests/bpf: Add test to verify xlated insns for global percpu data selftests/bpf: Add test to verify bpf_iter for global percpu data

 include/linux/bpf.h                           |   6 +-
 kernel/bpf/arraymap.c                         |  39 +-
 kernel/bpf/const_fold.c                       |   1 -
 kernel/bpf/core.c                             |   2 +
 kernel/bpf/fixups.c                           |  41 ++
 kernel/bpf/verifier.c                         |  30 +-
 tools/bpf/bpftool/gen.c                       |  43 +-
 tools/lib/bpf/bpf_gen_internal.h              |   3 +-
 tools/lib/bpf/features.c                      |  35 ++
 tools/lib/bpf/gen_loader.c                    |   3 +-
 tools/lib/bpf/libbpf.c                        |  68 +++-
 tools/lib/bpf/libbpf_internal.h               |   2 +
 tools/lib/bpf/skel_internal.h                 |  24 +-
 tools/testing/selftests/bpf/Makefile          |   2 +-
 .../bpf/prog_tests/global_data_init.c         | 373 ++++++++++++++++++
 .../bpf/prog_tests/global_percpu_subskel.c    |  37 ++
 .../bpf/progs/test_global_percpu_data.c       |  90 +++++
 17 files changed, 753 insertions(+), 46 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_global_percpu_data.c
Signature · 1 line
2.54.0
author
#1 lore raw

PATCH v6 1/12 bpf: Drop duplicate blank lines in verifier

There are many adjacent blank lines in the verifier that have accumulated over time.

Drop them for cleanup.

No functional changes intended.

---
 kernel/bpf/verifier.c | 15 ---------------
 1 file changed, 15 deletions(-)
kernel/bpf/verifier.c +0 -15
index 2abc79dbf281..721f883bf640 100644
@@ -652,7 +652,6 @@ static void __mark_dynptr_reg(struct bpf_reg_state *reg,
 			      enum bpf_dynptr_type type,
 			      bool first_slot, int id, int parent_id);
 
-
 static void mark_dynptr_stack_regs(struct bpf_verifier_env *env,
 				   struct bpf_reg_state *sreg1,
 				   struct bpf_reg_state *sreg2,
@@ -1689,7 +1688,6 @@ static bool same_callsites(struct bpf_verifier_state *a, struct bpf_verifier_sta
 	return true;
 }
 
-
 void bpf_free_backedges(struct bpf_scc_visit *visit)
 {
 	struct bpf_scc_backedge *backedge, *next;
@@ -2309,7 +2307,6 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env,
 	return &elem->st;
 }
 
-
 static int cmp_subprogs(const void *a, const void *b)
 {
 	return ((struct bpf_subprog_info *)a)->start -
@@ -3331,7 +3328,6 @@ static bool is_spillable_regtype(enum bpf_reg_type type)
 	}
 }
 
-
 /* check if register is a constant scalar value */
 static bool is_reg_const(struct bpf_reg_state *reg, bool subreg32)
 {
@@ -3993,7 +3989,6 @@ static int check_stack_read(struct bpf_verifier_env *env,
 	return err;
 }
 
-
 /* check_stack_write dispatches to check_stack_write_fixed_off or
  * check_stack_write_var_off.
  *
@@ -4788,7 +4783,6 @@ static int check_sock_access(struct bpf_verifier_env *env, int insn_idx,
 		valid = false;
 	}
 
-
 	if (valid) {
 		env->insn_aux_data[insn_idx].ctx_field_size =
 			info.ctx_field_size;
@@ -6608,7 +6602,6 @@ static int check_stack_range_initialized(
 	if (err)
 		return err;
 
-
 	if (tnum_is_const(reg->var_off)) {
 		min_off = max_off = reg->var_off.value + off;
 	} else {
@@ -7322,7 +7315,6 @@ static bool is_iter_new_kfunc(struct bpf_kfunc_call_arg_meta *meta)
 	return meta->kfunc_flags & KF_ITER_NEW;
 }
 
-
 static bool is_iter_destroy_kfunc(struct bpf_kfunc_call_arg_meta *meta)
 {
 	return meta->kfunc_flags & KF_ITER_DESTROY;
@@ -11465,7 +11457,6 @@ static int process_irq_flag(struct bpf_verifier_env *env, struct bpf_reg_state *
 	return 0;
 }
 
-
 static int ref_set_non_owning(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
 {
 	struct btf_record *rec = reg_btf_record(reg);
@@ -16357,7 +16348,6 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn)
 	return 0;
 }
 
-
 static bool return_retval_range(struct bpf_verifier_env *env, struct bpf_retval_range *range)
 {
 	enum bpf_prog_type prog_type = resolve_prog_type(env->prog);
@@ -18251,8 +18241,6 @@ static void release_insn_arrays(struct bpf_verifier_env *env)
 		bpf_insn_array_release(env->insn_array_maps[i]);
 }
 
-
-
 /* The verifier does more data flow analysis than llvm and will not
  * explore branches that are dead at run time. Malicious programs can
  * have dead code too. Therefore replace all dead at-run-time code
@@ -18280,8 +18268,6 @@ static void sanitize_dead_code(struct bpf_verifier_env *env)
 	}
 }
 
-
-
 static void free_states(struct bpf_verifier_env *env)
 {
 	struct bpf_verifier_state_list *sl;
@@ -18563,7 +18549,6 @@ static int do_check_main(struct bpf_verifier_env *env)
 	return ret;
 }
 
-
 static void print_verification_stats(struct bpf_verifier_env *env)
 {
 	/* Skip over hidden subprogs which are not verified. */
Signature · 1 line
2.54.0
author
#2 lore raw

PATCH v6 2/12 bpf: Disallow interpreter fallback for user BPF_ADDR_SPACE_CAST insn

In reply to Leon Hwang

Since interpreter is unable to handle the 'insn_is_cast_user()' insn, require JIT in __bpf_prog_select_runtime() when the prog has the insn.

Fixes 142fd4d2dcf5 ("bpf: Add x86-64 JIT support for bpf_addr_space_cast instruction.")
---
 include/linux/bpf.h | 1 +
 kernel/bpf/core.c   | 1 +
 kernel/bpf/fixups.c | 2 ++
 3 files changed, 4 insertions(+)
include/linux/bpf.h +1 -0
index 56f5da2b437f..5f48a6ab8a1a 100644
@@ -1780,6 +1780,7 @@ struct bpf_prog_aux {
 	bool changes_pkt_data;
 	bool might_sleep;
 	bool kprobe_write_ctx;
+	bool has_addr_space_cast_insn;
 	struct {
 		s32 keyring_serial;
 		u8 keyring_type;
kernel/bpf/core.c +1 -0
index 649cce41e13f..49398b5bd172 100644
@@ -2620,6 +2620,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct
 		goto finalize;
 
 	if (IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON) ||
+	    fp->aux->has_addr_space_cast_insn ||
 	    bpf_prog_has_kfunc_call(fp))
 		jit_needed = true;
 
kernel/bpf/fixups.c +2 -0
index 3cf2cc6e3ab6..2d5958774b61 100644
@@ -1523,6 +1523,8 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
 				insn->off = 0;
 				insn->imm = 0;
 			} /* cast from as(0) to as(1) should be handled by JIT */
+			if (insn_is_cast_user(insn))
+				prog->aux->has_addr_space_cast_insn = true;
 			goto next_insn;
 		}
 
Signature · 1 line
2.54.0
author
#3 lore raw

PATCH v6 3/12 bpf: Disallow interpreter fallback for BPF_ADDR_PERCPU insn

In reply to Leon Hwang

Since interpreter is unable to handle the 'insn_is_mov_percpu_addr()' insn, require JIT in __bpf_prog_select_runtime() when the prog has the insn.

BTW, rename the helper bpf_map_supports_cpu_flags() to bpf_map_is_percpu_map().

Fixes 7bdbf7446305 ("bpf: add special internal-only MOV instruction to resolve per-CPU addrs")
---
 include/linux/bpf.h | 5 +++--
 kernel/bpf/core.c   | 1 +
 kernel/bpf/fixups.c | 5 +++++
 3 files changed, 9 insertions(+), 2 deletions(-)
include/linux/bpf.h +3 -2
index 5f48a6ab8a1a..24ab17a25046 100644
@@ -1781,6 +1781,7 @@ struct bpf_prog_aux {
 	bool might_sleep;
 	bool kprobe_write_ctx;
 	bool has_addr_space_cast_insn;
+	bool has_addr_percpu_insn;
 	struct {
 		s32 keyring_serial;
 		u8 keyring_type;
@@ -4167,7 +4168,7 @@ bpf_prog_update_insn_ptrs(struct bpf_prog *prog, u32 *offsets, void *image)
 }
 #endif
 
-static inline bool bpf_map_supports_cpu_flags(enum bpf_map_type map_type)
+static inline bool bpf_map_is_percpu_map(enum bpf_map_type map_type)
 {
 	switch (map_type) {
 	case BPF_MAP_TYPE_PERCPU_ARRAY:
@@ -4194,7 +4195,7 @@ static inline int bpf_map_check_op_flags(struct bpf_map *map, u64 flags, u64 all
 		return -EINVAL;
 
 	if (flags & (BPF_F_CPU | BPF_F_ALL_CPUS)) {
-		if (!bpf_map_supports_cpu_flags(map->map_type))
+		if (!bpf_map_is_percpu_map(map->map_type))
 			return -EINVAL;
 		if ((flags & BPF_F_CPU) && (flags & BPF_F_ALL_CPUS))
 			return -EINVAL;
kernel/bpf/core.c +1 -0
index 49398b5bd172..69203d58e0ad 100644
@@ -2621,6 +2621,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct
 
 	if (IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON) ||
 	    fp->aux->has_addr_space_cast_insn ||
+	    fp->aux->has_addr_percpu_insn ||
 	    bpf_prog_has_kfunc_call(fp))
 		jit_needed = true;
 
kernel/bpf/fixups.c +5 -0
index 2d5958774b61..9552ddcf6936 100644
@@ -2009,6 +2009,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
 					return -EFAULT;
 				}
 
+				if (bpf_map_is_percpu_map(map_ptr->map_type))
+					prog->aux->has_addr_percpu_insn = true;
+
 				new_prog = bpf_patch_insn_data(env, i + delta,
 							       insn_buf, cnt);
 				if (!new_prog)
@@ -2113,6 +2116,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
 			 * way, it's fine to back out this inlining logic
 			 */
 #ifdef CONFIG_SMP
+			prog->aux->has_addr_percpu_insn = true;
 			insn_buf[0] = BPF_MOV64_IMM(BPF_REG_0, (u32)(unsigned long)&cpu_number);
 			insn_buf[1] = BPF_MOV64_PERCPU_REG(BPF_REG_0, BPF_REG_0);
 			insn_buf[2] = BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, 0);
@@ -2134,6 +2138,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
 		/* Implement bpf_get_current_task() and bpf_get_current_task_btf() inline. */
 		if ((insn->imm == BPF_FUNC_get_current_task || insn->imm == BPF_FUNC_get_current_task_btf) &&
 		    bpf_verifier_inlines_helper_call(env, insn->imm)) {
+			prog->aux->has_addr_percpu_insn = true;
 			insn_buf[0] = BPF_MOV64_IMM(BPF_REG_0, (u32)(unsigned long)&current_task);
 			insn_buf[1] = BPF_MOV64_PERCPU_REG(BPF_REG_0, BPF_REG_0);
 			insn_buf[2] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0);
Signature · 1 line
2.54.0
author
#4 lore raw

PATCH v6 4/12 bpf: Introduce global percpu data

In reply to Leon Hwang

Introduce global percpu data, inspired by the commit 6316f78306c1 ("Merge branch 'support-global-data'"). It enables the definition of global percpu variables in BPF, similar to the include/linux/percpu-defs.h::DEFINE_PER_CPU() macro.

For example, in BPF, it is able to define a global percpu variable like:

int data SEC(".percpu");

With this patch, tools like retsnoop [1] and bpfsnoop [2] can simplify their BPF code for handling LBRs. The code can be updated from

static struct perf_branch_entry lbrs[1][MAX_LBR_ENTRIES] SEC(".data.lbrs");

to

static struct perf_branch_entry lbrs[MAX_LBR_ENTRIES] SEC(".percpu.lbrs");

This eliminates the need to retrieve the CPU ID using the bpf_get_smp_processor_id() helper.

Additionally, by reusing global percpu data map, sharing information between tail callers and callees or freplace callers and callees becomes simpler compared to reusing percpu_array maps.

Links:
[1] https://github.com/anakryiko/retsnoop
[2] https://github.com/bpfsnoop/bpfsnoop

---
 kernel/bpf/arraymap.c   | 39 +++++++++++++++++++++++++++++++++++++--
 kernel/bpf/const_fold.c |  1 -
 kernel/bpf/fixups.c     | 34 ++++++++++++++++++++++++++++++++++
 kernel/bpf/verifier.c   | 15 +++++++++++++++
 4 files changed, 86 insertions(+), 3 deletions(-)
kernel/bpf/arraymap.c +37 -2
index 248b4818178c..876aa2604f41 100644
@@ -259,6 +259,38 @@ static void *percpu_array_map_lookup_elem(struct bpf_map *map, void *key)
 	return this_cpu_ptr(array->pptrs[index & array->index_mask]);
 }
 
+static int percpu_array_map_direct_value_addr(const struct bpf_map *map, u64 *imm, u32 off)
+{
+	struct bpf_array *array = container_of(map, struct bpf_array, map);
+
+	if (map->max_entries != 1)
+		return -EOPNOTSUPP;
+	if (off >= map->value_size)
+		return -EINVAL;
+	if (!bpf_jit_supports_percpu_insn())
+		return -EOPNOTSUPP;
+
+	*imm = (u64)(__force unsigned long) array->pptrs[0];
+	return 0;
+}
+
+static int percpu_array_map_direct_value_meta(const struct bpf_map *map, u64 imm, u32 *off)
+{
+	struct bpf_array *array = container_of(map, struct bpf_array, map);
+	u64 base = (u64)(__force unsigned long) array->pptrs[0];
+	u64 range = array->elem_size;
+
+	if (map->max_entries != 1)
+		return -EOPNOTSUPP;
+	if (imm < base || imm >= base + range)
+		return -ENOENT;
+	if (!bpf_jit_supports_percpu_insn())
+		return -EOPNOTSUPP;
+
+	*off = imm - base;
+	return 0;
+}
+
 /* emit BPF instructions equivalent to C code of percpu_array_map_lookup_elem() */
 static int percpu_array_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf)
 {
@@ -551,9 +583,10 @@ static int array_map_check_btf(struct bpf_map *map,
 			       const struct btf_type *key_type,
 			       const struct btf_type *value_type)
 {
-	/* One exception for keyless BTF: .bss/.data/.rodata map */
+	/* One exception for keyless BTF: .bss/.data/.rodata/.percpu map */
 	if (btf_type_is_void(key_type)) {
-		if (map->map_type != BPF_MAP_TYPE_ARRAY ||
+		if ((map->map_type != BPF_MAP_TYPE_ARRAY &&
+		     map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) ||
 		    map->max_entries != 1)
 			return -EINVAL;
 
@@ -832,6 +865,8 @@ const struct bpf_map_ops percpu_array_map_ops = {
 	.map_get_next_key = bpf_array_get_next_key,
 	.map_lookup_elem = percpu_array_map_lookup_elem,
 	.map_gen_lookup = percpu_array_map_gen_lookup,
+	.map_direct_value_addr = percpu_array_map_direct_value_addr,
+	.map_direct_value_meta = percpu_array_map_direct_value_meta,
 	.map_update_elem = array_map_update_elem,
 	.map_delete_elem = array_map_delete_elem,
 	.map_lookup_percpu_elem = percpu_array_map_lookup_percpu_elem,
kernel/bpf/const_fold.c +0 -1
index b2a19acadb91..5787246bef30 100644
@@ -182,7 +182,6 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info *
 		u64 val = 0;
 
 		if (!bpf_map_is_rdonly(map) || !map->ops->map_direct_value_addr ||
-		    map->map_type == BPF_MAP_TYPE_INSN_ARRAY ||
 		    off < 0 || off + size > map->value_size ||
 		    bpf_map_direct_read(map, off, size, &val, is_ldsx)) {
 			*dst = unknown;
kernel/bpf/fixups.c +34 -0
index 9552ddcf6936..802d49e4a643 100644
@@ -1821,6 +1821,40 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
 			goto next_insn;
 		}
 
+		if (env->prog->jit_requested &&
+		    bpf_jit_supports_percpu_insn() &&
+		    insn->code == (BPF_LD | BPF_IMM | BPF_DW) &&
+		    (insn->src_reg == BPF_PSEUDO_MAP_VALUE ||
+		     insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE)) {
+			struct bpf_map *map;
+
+			aux = &env->insn_aux_data[i + delta];
+			map = env->used_maps[aux->map_index];
+			if (map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY)
+				goto next_insn;
+
+			prog->aux->has_addr_percpu_insn = true;
+
+			/*
+			 * Reuse the original ld_imm64 insn, and add one
+			 * mov64_percpu_reg insn.
+			 */
+
+			insn_buf[0] = insn[1];
+			insn_buf[1] = BPF_MOV64_PERCPU_REG(insn->dst_reg, insn->dst_reg);
+			cnt = 2;
+
+			i++;
+			new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, cnt);
+			if (!new_prog)
+				return -ENOMEM;
+
+			delta    += cnt - 1;
+			env->prog = prog = new_prog;
+			insn      = new_prog->insnsi + i + delta;
+			goto next_insn;
+		}
+
 		if (insn->code != (BPF_JMP | BPF_CALL))
 			goto next_insn;
 		if (insn->src_reg == BPF_PSEUDO_CALL)
kernel/bpf/verifier.c +15 -0
index 721f883bf640..9a2ae6a10e90 100644
@@ -5562,6 +5562,8 @@ int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val,
 	u64 addr;
 	int err;
 
+	if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY || map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY)
+		return -EINVAL;
 	err = map->ops->map_direct_value_addr(map, &addr, off);
 	if (err)
 		return err;
@@ -6113,6 +6115,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct b
 			if (tnum_is_const(reg->var_off) &&
 			    bpf_map_is_rdonly(map) &&
 			    map->ops->map_direct_value_addr &&
+			    map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY &&
 			    map->map_type != BPF_MAP_TYPE_INSN_ARRAY) {
 				int map_off = off + reg->var_off.value;
 				u64 val = 0;
@@ -8080,6 +8083,12 @@ static int check_arg_const_str(struct bpf_verifier_env *env,
 		return -EACCES;
 	}
 
+	if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) {
+		verbose(env, "%s points to percpu_array map which cannot be used as const string\n",
+			reg_arg_name(env, argno));
+		return -EACCES;
+	}
+
 	if (!bpf_map_is_rdonly(map)) {
 		verbose(env, "%s does not point to a readonly map'\n", reg_arg_name(env, argno));
 		return -EACCES;
@@ -18166,6 +18175,12 @@ static int check_and_resolve_insns(struct bpf_verifier_env *env)
 					return -EINVAL;
 				}
 
+				if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY &&
+				    !env->prog->jit_requested) {
+					verbose(env, "JIT is required to use global percpu data\n");
+					return -EOPNOTSUPP;
+				}
+
 				err = map->ops->map_direct_value_addr(map, &addr, off);
 				if (err) {
 					verbose(env, "invalid access to map value pointer, value_size=%u off=%u\n",
Signature · 1 line
2.54.0
author
#5 lore raw

PATCH v6 5/12 libbpf: Probe percpu data feature

In reply to Leon Hwang

libbpf needs a reliable way to distinguish kernels that can support global percpu data from those that cannot.

Add a dedicated feature probe, so libbpf can make capability decisions early and fail predictably when global percpu data is unavailable.

---
 tools/lib/bpf/features.c        | 35 +++++++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf_internal.h |  2 ++
 2 files changed, 37 insertions(+)
tools/lib/bpf/features.c +35 -0
index b7e388f99d0b..ef9581c11303 100644
@@ -620,6 +620,38 @@ static int probe_bpf_syscall_common_attrs(int token_fd)
 	return probe_sys_bpf_ext();
 }
 
+static int probe_kern_percpu_data(int token_fd)
+{
+	struct bpf_insn insns[] = {
+		BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0),
+		BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0),
+		BPF_EXIT_INSN(),
+	};
+	LIBBPF_OPTS(bpf_map_create_opts, map_opts,
+		.token_fd = token_fd,
+		.map_flags = token_fd ? BPF_F_TOKEN_FD : 0,
+	);
+	LIBBPF_OPTS(bpf_prog_load_opts, prog_opts,
+		.token_fd = token_fd,
+		.prog_flags = token_fd ? BPF_F_TOKEN_FD : 0,
+	);
+	int ret, map, insn_cnt = ARRAY_SIZE(insns);
+
+	map = bpf_map_create(BPF_MAP_TYPE_PERCPU_ARRAY, "libbpf_percpu", sizeof(int), 8, 1,
+			     &map_opts);
+	if (map < 0) {
+		pr_warn("Error in %s(): %s. Couldn't create simple percpu_array map.\n",
+			__func__, errstr(map));
+		return map;
+	}
+
+	insns[0].imm = map;
+
+	ret = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, NULL, "GPL", insns, insn_cnt, &prog_opts);
+	close(map);
+	return probe_fd(ret);
+}
+
 typedef int (*feature_probe_fn)(int /* token_fd */);
 
 static struct kern_feature_cache feature_cache;
@@ -707,6 +739,9 @@ static struct kern_feature_desc {
 	[FEAT_BPF_SYSCALL_COMMON_ATTRS] = {
 		"BPF syscall common attributes support", probe_bpf_syscall_common_attrs,
 	},
+	[FEAT_PERCPU_DATA] = {
+		"kernel supports percpu data", probe_kern_percpu_data,
+	},
 };
 
 bool feat_supported(struct kern_feature_cache *cache, enum kern_feature_id feat_id)
tools/lib/bpf/libbpf_internal.h +2 -0
index 04cd303fb5a8..47ae39125f68 100644
@@ -401,6 +401,8 @@ enum kern_feature_id {
 	FEAT_BTF_LAYOUT,
 	/* Kernel supports BPF syscall common attributes */
 	FEAT_BPF_SYSCALL_COMMON_ATTRS,
+	/* Kernel supports percpu data */
+	FEAT_PERCPU_DATA,
 	__FEAT_CNT,
 };
 
Signature · 1 line
2.54.0
author
#6 lore raw

PATCH v6 6/12 libbpf: Add support for global percpu data

In reply to Leon Hwang

Add support for global percpu data in libbpf by adding a new ".percpu" section, similar to ".data". It enables efficient handling of percpu global variables in bpf programs.

When generating loader for lightweight skeleton, update the percpu_array map used for global percpu data using BPF_F_ALL_CPUS, in order to update values across all CPUs using one value slot.

Unlike global data, the mmaped data for global percpu data will be marked as read-only after populating the percpu_array map. Thereafter, users can read those initialized percpu data after loading prog. If they want to update the percpu data after loading prog, they have to update the percpu_array map using key=0 instead.

---
 tools/lib/bpf/bpf_gen_internal.h |  3 +-
 tools/lib/bpf/gen_loader.c       |  3 +-
 tools/lib/bpf/libbpf.c           | 68 ++++++++++++++++++++++++++------
 3 files changed, 60 insertions(+), 14 deletions(-)
tools/lib/bpf/bpf_gen_internal.h +2 -1
index 49af4260b8e6..5ea8383805d3 100644
@@ -66,7 +66,8 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 			enum bpf_prog_type prog_type, const char *prog_name,
 			const char *license, struct bpf_insn *insns, size_t insn_cnt,
 			struct bpf_prog_load_opts *load_attr, int prog_idx);
-void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *value, __u32 value_size);
+void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *value, __u32 value_size,
+			      __u64 flags);
 void bpf_gen__map_freeze(struct bpf_gen *gen, int map_idx);
 void bpf_gen__record_attach_target(struct bpf_gen *gen, const char *name, enum bpf_attach_type type);
 void bpf_gen__record_extern(struct bpf_gen *gen, const char *name, bool is_weak,
tools/lib/bpf/gen_loader.c +2 -1
index d79695f01c87..7ceb096f18ec 100644
@@ -1190,7 +1190,7 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 }
 
 void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue,
-			      __u32 value_size)
+			      __u32 value_size, __u64 flags)
 {
 	int attr_size = offsetofend(union bpf_attr, flags);
 	int map_update_attr, value, key;
@@ -1198,6 +1198,7 @@ void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue,
 	int zero = 0;
 
 	memset(&attr, 0, attr_size);
+	attr.flags = flags;
 
 	value = add_data(gen, pvalue, value_size);
 	key = add_data(gen, &zero, sizeof(zero));
tools/lib/bpf/libbpf.c +56 -12

179 lines not on this page · raw message

Signature · 1 line
2.54.0
author
#9 lore raw

PATCH v6 7/12 bpftool: Generate skeleton for global percpu data

In reply to Leon Hwang

Enhance bpftool to generate skeletons that properly handle global percpu variables. The generated skeleton now includes a dedicated structure for percpu data, allowing users to initialize and access percpu variables more efficiently.

For global percpu variables, the skeleton now includes a nested structure, e.g.:

struct test_global_percpu_data {
	struct bpf_object_skeleton *skeleton;
	struct bpf_object *obj;
	struct {
		struct bpf_map *percpu;
	} maps;
	// ...
	struct test_global_percpu_data__percpu {
		int data;
		char run;
		struct {
			char set;
			int i;
			int nums[7];
		} struct_data;
		int nums[7];
	} *percpu;

	// ...
};

* The "struct test_global_percpu_data__percpu *percpu" points to

    initialized data, which is actually "maps.percpu->mmaped".
  * Before loading the skeleton, updating the
    "struct test_global_percpu_data__percpu *percpu" modifies the initial
    value of the corresponding global percpu variables.
  * After loading the skeleton, "maps.percpu->mmaped" has been marked as
    read-only in libbpf. If users want to update the global percpu
    variables, they have to update the "maps.percpu" map instead.
  * For lightweight skeleton, "lskel->percpu" will be protected by
    "mprotect(p, sz, PROT_READ)".

* For subskeleton, those variables of global percpu data will be skipped.

Assisted-by Codex:gpt-5.5-xhigh
---
 tools/bpf/bpftool/gen.c       | 43 +++++++++++++++++++++++++++--------
 tools/lib/bpf/skel_internal.h | 24 +++++++++++++++++--
 2 files changed, 56 insertions(+), 11 deletions(-)
tools/bpf/bpftool/gen.c +34 -9
index 6ae7262ebe0c..2200ab004d2b 100644
@@ -92,7 +92,7 @@ static void get_header_guard(char *guard, const char *obj_name, const char *suff
 
 static bool get_map_ident(const struct bpf_map *map, char *buf, size_t buf_sz)
 {
-	static const char *sfxs[] = { ".data", ".rodata", ".bss", ".kconfig" };
+	static const char *sfxs[] = { ".data", ".rodata", ".bss", ".percpu", ".kconfig" };
 	const char *name = bpf_map__name(map);
 	int i, n;
 
@@ -117,7 +117,7 @@ static bool get_map_ident(const struct bpf_map *map, char *buf, size_t buf_sz)
 
 static bool get_datasec_ident(const char *sec_name, char *buf, size_t buf_sz)
 {
-	static const char *pfxs[] = { ".data", ".rodata", ".bss", ".kconfig" };
+	static const char *pfxs[] = { ".data", ".rodata", ".bss", ".percpu", ".kconfig" };
 	int i, n;
 
 	/* recognize hard coded LLVM section name */
@@ -254,6 +254,11 @@ static const struct btf_type *find_type_for_map(struct btf *btf, const char *map
 	return NULL;
 }
 
+static bool bpf_map_is_percpu_data(const struct bpf_map *map)
+{
+	return bpf_map__is_internal(map) && bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY;
+}
+
 static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)
 {
 	size_t tmp_sz;
@@ -263,13 +268,12 @@ static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)
 		return true;
 	}
 
-	if (!bpf_map__is_internal(map) || !(bpf_map__map_flags(map) & BPF_F_MMAPABLE))
-		return false;
-
-	if (!get_map_ident(map, buf, sz))
-		return false;
+	if (bpf_map__is_internal(map) &&
+	    ((bpf_map__map_flags(map) & BPF_F_MMAPABLE) || bpf_map_is_percpu_data(map)) &&
+	    get_map_ident(map, buf, sz))
+		return true;
 
-	return true;
+	return false;
 }
 
 static int codegen_datasecs(struct bpf_object *obj, const char *obj_name)
@@ -343,6 +347,9 @@ static int codegen_subskel_datasecs(struct bpf_object *obj, const char *obj_name
 		if (!is_mmapable_map(map, map_ident, sizeof(map_ident)))
 			continue;
 
+		if (bpf_map_is_percpu_data(map))
+			continue;
+
 		sec = find_type_for_map(btf, map_ident);
 		if (!sec)
 			continue;
@@ -669,7 +676,7 @@ static void codegen_destroy(struct bpf_object *obj, const char *obj_name)
 		if (!get_map_ident(map, ident, sizeof(ident)))
 			continue;
 		if (bpf_map__is_internal(map) &&
-		    (bpf_map__map_flags(map) & BPF_F_MMAPABLE))
+		    ((bpf_map__map_flags(map) & BPF_F_MMAPABLE) || bpf_map_is_percpu_data(map)))
 			printf("\tskel_free_map_data(skel->%1$s, skel->maps.%1$s.initial_value, %2$zu);\n",
 			       ident, bpf_map_mmap_sz(map));
 		codegen("\
@@ -850,6 +857,20 @@ static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *h
 		if (!is_mmapable_map(map, ident, sizeof(ident)))
 			continue;
 
+		if (bpf_map_is_percpu_data(map)) {
+			codegen("\
+		\n\
+			err = skel_protect_map_data(skel->%1$s, &skel->maps.%1$s.initial_value, %2$zd);\n\
+			if (err)					    \n\
+				return err;				    \n\
+		#ifdef __KERNEL__					    \n\
+			skel->%1$s = NULL;				    \n\
+		#endif							    \n\
+			",
+			ident, bpf_map_mmap_sz(map));
+			continue;
+		}
+
 		if (bpf_map__map_flags(map) & BPF_F_RDONLY_PROG)
 			mmap_flags = "PROT_READ";
 		else
@@ -1740,6 +1761,8 @@ static int do_subskeleton(int argc, char **argv)
 
 		if (!is_mmapable_map(map, ident, sizeof(ident)))
 			continue;
+		if (bpf_map_is_percpu_data(map))
+			continue;
 
 		map_type_id = bpf_map__btf_value_type_id(map);
 		if (map_type_id <= 0) {
@@ -1863,6 +1886,8 @@ static int do_subskeleton(int argc, char **argv)
 	bpf_object__for_each_map(map, obj) {
 		if (!is_mmapable_map(map, ident, sizeof(ident)))
 			continue;
+		if (bpf_map_is_percpu_data(map))
+			continue;
 
 		map_type_id = bpf_map__btf_value_type_id(map);
 		if (map_type_id <= 0)
tools/lib/bpf/skel_internal.h +22 -2
index 74503d358bc8..485b0cd17017 100644
@@ -135,8 +135,10 @@ static inline void skel_free_map_data(void *p, __u64 addr, size_t sz)
 {
 	if (addr != ~0ULL)
 		kvfree(p);
-	/* When addr == ~0ULL the 'p' points to
-	 * ((struct bpf_array *)map)->value. See skel_finalize_map_data.
+	/*
+	 * When addr == ~0ULL the init buffer has already been released.
+	 * For skel_finalize_map_data(), 'p' points to
+	 * ((struct bpf_array *)map)->value.
 	 */
 }
 
@@ -174,6 +176,15 @@ static inline void *skel_finalize_map_data(__u64 *init_val, size_t mmap_sz, int
 	return addr;
 }
 
+static inline int skel_protect_map_data(void *p, __u64 *init_val, size_t sz)
+{
+	(void)sz;
+
+	kvfree(p);
+	*init_val = ~0ULL;
+	return 0;
+}
+
 #else
 
 static inline void *skel_alloc(size_t size)
@@ -212,6 +223,15 @@ static inline void *skel_finalize_map_data(__u64 *init_val, size_t mmap_sz, int
 		return NULL;
 	return addr;
 }
+
+static inline int skel_protect_map_data(void *p, __u64 *init_val, size_t sz)
+{
+	(void)init_val;
+
+	if (mprotect(p, sz, PROT_READ))
+		return -errno;
+	return 0;
+}
 #endif
 
 static inline int skel_closenz(int fd)
Signature · 1 line
2.54.0
author
#10 lore raw

PATCH v6 8/12 selftests/bpf: Add tests to verify global percpu data

In reply to Leon Hwang

If the arch, like s390x, does not support percpu insn, these cases won't test global percpu data by checking FEAT_PERCPU_DATA support.

The following APIs have been tested for global percpu data:

1. bpf_map__set_initial_value()
2. bpf_map__initial_value()
3. generated percpu struct pointer pointing to internal map's mmaped data
4. bpf_map__lookup_elem() for global percpu data map

At the same time, the case is also tested with 'bpftool gen skeleton -L'.

Add a test to verify that the live vars of subskel won't include the vars for global percpu data.

Assisted-by Codex:gpt-5.5-xhigh
---
 tools/testing/selftests/bpf/Makefile          |   2 +-
 .../bpf/prog_tests/global_data_init.c         | 184 ++++++++++++++++++
 .../bpf/prog_tests/global_percpu_subskel.c    |  37 ++++
 .../bpf/progs/test_global_percpu_data.c       |  31 +++
 4 files changed, 253 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_global_percpu_data.c
tools/testing/selftests/bpf/Makefile +1 -1
index b642ee489ea6..c37ed9e7b97c 100644
@@ -533,7 +533,7 @@ LSKELS_SIGNED := fentry_test.c fexit_test.c atomics.c
 
 # Generate both light skeleton and libbpf skeleton for these
 LSKELS_EXTRA := test_ksyms_module.c test_ksyms_weak.c kfunc_call_test.c \
-	kfunc_call_test_subprog.c
+	kfunc_call_test_subprog.c test_global_percpu_data.c
 SKEL_BLACKLIST += $$(LSKELS) $$(LSKELS_SIGNED)
 
 test_static_linked.skel.h-deps := test_static_linked1.bpf.o test_static_linked2.bpf.o
tools/testing/selftests/bpf/prog_tests/global_data_init.c +184 -0

192 lines not on this page · raw message

tools/testing/selftests/bpf/prog_tests/global_percpu_subskel.c new file +37 -0
new file mode 100644
index 000000000000..8aebd533d86b
@@ -0,0 +1,37 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include "test_global_percpu_data.subskel.h"
+
+void test_global_percpu_data_subskel(void)
+{
+	struct test_global_percpu_data *subskel = NULL;
+	struct bpf_object *obj;
+	int i;
+
+	obj = bpf_object__open_file("./test_global_percpu_data.bpf.o", NULL);
+	if (!ASSERT_OK_PTR(obj, "bpf_object__open_file"))
+		return;
+
+	subskel = test_global_percpu_data__open(obj);
+	if (!ASSERT_OK_PTR(subskel, "test_global_percpu_data__open"))
+		goto out;
+
+	if (!ASSERT_OK_PTR(subskel->subskel, "subskel"))
+		goto out;
+	if (!ASSERT_OK_PTR(subskel->maps.percpu, "maps.percpu"))
+		goto out;
+	ASSERT_EQ(bpf_map__type(subskel->maps.percpu), BPF_MAP_TYPE_PERCPU_ARRAY,
+		  "percpu_map_type");
+	ASSERT_GT(subskel->subskel->var_cnt, 0, "var_cnt");
+
+	for (i = 0; i < subskel->subskel->var_cnt; i++) {
+		const struct bpf_var_skeleton *var;
+
+		var = (void *) subskel->subskel->vars + i * subskel->subskel->var_skel_sz;
+		ASSERT_NEQ(var->map, &subskel->maps.percpu, "var");
+	}
+
+out:
+	test_global_percpu_data__destroy(subskel);
+	bpf_object__close(obj);
+}
tools/testing/selftests/bpf/progs/test_global_percpu_data.c new file +31 -0
new file mode 100644
index 000000000000..ba92ffb0ca49
@@ -0,0 +1,31 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+int data SEC(".percpu") = -1;
+int nums[7] SEC(".percpu");
+char run SEC(".percpu") = 0;
+struct {
+	char set;
+	int i;
+	int nums[7];
+} struct_data SEC(".percpu") = {
+	.set = 0,
+	.i = -1,
+};
+
+SEC("raw_tp/task_rename")
+__auxiliary
+int update_percpu_data(void *ctx)
+{
+	struct_data.nums[6] = 0xc0de;
+	struct_data.set = 1;
+	struct_data.i = 1;
+	nums[6] = 0xc0de;
+	data = 1;
+	run = 1;
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
Signature · 1 line
2.54.0
author
#11 lore raw

PATCH v6 9/12 selftests/bpf: Add test to verify accessing rdonly percpu_array

In reply to Leon Hwang

Since percpu_array supports direct-read, it should not break accessing rdonly percpu_array.

Add a test to verify that adding '.map_direct_value_addr' to percpu_array won't break the case.

Assisted-by Codex:gpt-5.5-xhigh
---
 .../bpf/prog_tests/global_data_init.c         | 44 +++++++++++++++++++
 1 file changed, 44 insertions(+)
tools/testing/selftests/bpf/prog_tests/global_data_init.c +44 -0
index ea7e4e3d91cf..f59d83919058 100644
@@ -232,6 +232,48 @@ static void test_global_percpu_data_lskel(void)
 	free(online);
 }
 
+static void test_global_percpu_data_rdonly_direct_read(void)
+{
+	LIBBPF_OPTS(bpf_map_create_opts, map_opts,
+		    .map_flags = BPF_F_RDONLY_PROG,
+	);
+	struct bpf_insn insns[] = {
+		BPF_ST_MEM(BPF_W, BPF_REG_10, -8, 0),
+		BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+		BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+		BPF_LD_MAP_FD(BPF_REG_1, 0),
+		BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
+		BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1),
+		BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0),
+		BPF_EXIT_INSN(),
+	};
+	int key = 0, map_fd, prog_fd = -1, err;
+	__u64 value = 0;
+
+	map_fd = bpf_map_create(BPF_MAP_TYPE_PERCPU_ARRAY, "percpu_ro_map", sizeof(int),
+				sizeof(__u64), 1, &map_opts);
+	if (!ASSERT_GE(map_fd, 0, "bpf_map_create"))
+		return;
+
+	err = bpf_map_update_elem(map_fd, &key, &value, BPF_F_ALL_CPUS);
+	if (!ASSERT_OK(err, "bpf_map_update_elem"))
+		goto out;
+
+	err = bpf_map_freeze(map_fd);
+	if (!ASSERT_OK(err, "bpf_map_freeze"))
+		goto out;
+
+	insns[3].imm = map_fd;
+	prog_fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, "percpu_ro_prog", "GPL", insns,
+				ARRAY_SIZE(insns), NULL);
+	ASSERT_GE(prog_fd, 0, "bpf_prog_load");
+
+out:
+	if (prog_fd >= 0)
+		close(prog_fd);
+	close(map_fd);
+}
+
 void test_global_percpu_data(void)
 {
 	if (!feat_supported(NULL, FEAT_PERCPU_DATA)) {
@@ -243,4 +285,6 @@ void test_global_percpu_data(void)
 		test_global_percpu_data_init();
 	if (test__start_subtest("lskel"))
 		test_global_percpu_data_lskel();
+	if (test__start_subtest("rdonly_direct_read"))
+		test_global_percpu_data_rdonly_direct_read();
 }
Signature · 1 line
2.54.0
author
#12 lore raw

PATCH v6 10/12 selftests/bpf: Add tests to verify verifier log for global percpu data

In reply to Leon Hwang

Add two tests to verify the verifier log
"R%d points to percpu_array map which cannot be used as const string\n".

---
 .../bpf/prog_tests/global_data_init.c         |  6 +++++
 .../bpf/progs/test_global_percpu_data.c       | 23 +++++++++++++++++++
 2 files changed, 29 insertions(+)
tools/testing/selftests/bpf/prog_tests/global_data_init.c +6 -0
index f59d83919058..e4c290492ff8 100644
@@ -274,6 +274,11 @@ static void test_global_percpu_data_rdonly_direct_read(void)
 	close(map_fd);
 }
 
+static void test_global_percpu_data_verifier_log(void)
+{
+	RUN_TESTS(test_global_percpu_data);
+}
+
 void test_global_percpu_data(void)
 {
 	if (!feat_supported(NULL, FEAT_PERCPU_DATA)) {
@@ -287,4 +292,5 @@ void test_global_percpu_data(void)
 		test_global_percpu_data_lskel();
 	if (test__start_subtest("rdonly_direct_read"))
 		test_global_percpu_data_rdonly_direct_read();
+	test_global_percpu_data_verifier_log();
 }
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +23 -0
index ba92ffb0ca49..2222ad3a49bd 100644
@@ -28,4 +28,27 @@ int update_percpu_data(void *ctx)
 	return 0;
 }
 
+static const char fmt[] SEC(".percpu.fmt") = "data %d\n";
+
+SEC("?kprobe")
+__failure __msg("R{{[0-9]+}} points to percpu_array map which cannot be used as const string")
+int verifier_strncmp(void *ctx)
+{
+	return bpf_strncmp("test", 5, fmt);
+}
+
+SEC("?kprobe")
+__failure __msg("R{{[0-9]+}} points to percpu_array map which cannot be used as const string")
+int verifier_snprintf(void *ctx)
+{
+	u64 args[] = { data };
+	char buf[128];
+	int len;
+
+	len = bpf_snprintf(buf, sizeof(buf), fmt, args, 1);
+	if (len > 0)
+		bpf_printk("snprintf: %s\n", buf);
+	return 0;
+}
+
 char _license[] SEC("license") = "GPL";
Signature · 1 line
2.54.0
author
#15 lore raw

PATCH v6 11/12 selftests/bpf: Add test to verify xlated insns for global percpu data

In reply to Leon Hwang

Add a test to verify global percpu data related xlated insns:

1. ld_imm64: compare xlated one with the one in ELF object file.
2. mov64_percpu_reg: it is added by verifier.

---
 .../bpf/prog_tests/global_data_init.c         | 87 +++++++++++++++++++
 .../bpf/progs/test_global_percpu_data.c       | 11 +++
 2 files changed, 98 insertions(+)
tools/testing/selftests/bpf/prog_tests/global_data_init.c +87 -0
index e4c290492ff8..fcc2c4ec8644 100644
@@ -279,6 +279,91 @@ static void test_global_percpu_data_verifier_log(void)
 	RUN_TESTS(test_global_percpu_data);
 }
 
+static int find_ld_imm64(const struct bpf_insn *insns, size_t insn_cnt, struct bpf_insn *ld_imm64)
+{
+	size_t i;
+
+	for (i = 0; i < insn_cnt; i++) {
+		if (insns[i].code == (BPF_LD | BPF_IMM | BPF_DW)) {
+			ld_imm64[0] = insns[i];
+			ld_imm64[1] = insns[i + 1];
+			return i;
+		}
+	}
+
+	return -ENOENT;
+}
+
+/*
+ * Special (internal-only) form of mov, used to resolve per-CPU addrs:
+ * dst_reg = src_reg + <percpu_base_off>
+ * BPF_ADDR_PERCPU is used as a special insn->off value.
+ */
+#define BPF_ADDR_PERCPU	(-1)
+
+#define BPF_MOV64_PERCPU_REG(DST, SRC)				\
+	((struct bpf_insn) {					\
+		.code  = BPF_ALU64 | BPF_MOV | BPF_X,		\
+		.dst_reg = DST,					\
+		.src_reg = SRC,					\
+		.off   = BPF_ADDR_PERCPU,			\
+		.imm   = 0 })
+
+static __u64 ld_imm64_to_u64(const struct bpf_insn *insn)
+{
+	return ((__u64)(__u32) insn[1].imm << 32) | (__u32) insn[0].imm;
+}
+
+static void test_global_percpu_data_xlated(void)
+{
+	struct bpf_insn ld_imm64_raw[2], ld_imm64_xlated[2], mov64_percpu_reg, *insns = NULL;
+	size_t insn_sz = sizeof(struct bpf_insn);
+	struct test_global_percpu_data *skel;
+	struct bpf_program *prog;
+	int idx, err;
+	__u32 cnt;
+
+	skel = test_global_percpu_data__open();
+	if (!ASSERT_OK_PTR(skel, "test_global_percpu_data__open"))
+		return;
+
+	prog = skel->progs.verifier_percpu_read;
+	idx = find_ld_imm64(bpf_program__insns(prog), bpf_program__insn_cnt(prog), ld_imm64_raw);
+	if (!ASSERT_GE(idx, 0, "find_ld_imm64 raw"))
+		goto out;
+
+	err = test_global_percpu_data__load(skel);
+	if (!ASSERT_OK(err, "test_global_percpu_data__load"))
+		goto out;
+
+	err = get_xlated_program(bpf_program__fd(prog), &insns, &cnt);
+	if (!ASSERT_OK(err, "get_xlated_program"))
+		goto out;
+
+	idx = find_ld_imm64(insns, cnt, ld_imm64_xlated);
+	if (!ASSERT_GE(idx, 0, "find_ld_imm64 xlated"))
+		goto out;
+
+	if (!ASSERT_GT(cnt, idx + 2, "xlated insn count"))
+		goto out;
+
+	ASSERT_EQ(ld_imm64_xlated[0].code, ld_imm64_raw[0].code, "ld_imm64 opcode");
+	ASSERT_TRUE(ld_imm64_xlated[0].dst_reg == ld_imm64_raw[0].dst_reg, "ld_imm64 dst_reg");
+	/*
+	 * The xlated instruction has the map ID in imm and the offset
+	 * in the next instruction's imm. The raw instruction just has
+	 * the offset in its imm.
+	 */
+	ASSERT_EQ(ld_imm64_xlated[1].imm, ld_imm64_to_u64(ld_imm64_raw), "ld_imm64 off");
+
+	mov64_percpu_reg = BPF_MOV64_PERCPU_REG(ld_imm64_raw[0].dst_reg, ld_imm64_raw[0].dst_reg);
+	ASSERT_MEMEQ(&insns[idx + 2], &mov64_percpu_reg, insn_sz, "mov64_percpu_reg");
+
+out:
+	test_global_percpu_data__destroy(skel);
+	free(insns);
+}
+
 void test_global_percpu_data(void)
 {
 	if (!feat_supported(NULL, FEAT_PERCPU_DATA)) {
@@ -293,4 +378,6 @@ void test_global_percpu_data(void)
 	if (test__start_subtest("rdonly_direct_read"))
 		test_global_percpu_data_rdonly_direct_read();
 	test_global_percpu_data_verifier_log();
+	if (test__start_subtest("xlated"))
+		test_global_percpu_data_xlated();
 }
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +11 -0
index 2222ad3a49bd..9a1b1a314c2f 100644
@@ -51,4 +51,15 @@ int verifier_snprintf(void *ctx)
 	return 0;
 }
 
+static volatile const char fmt2[] SEC(".percpu.fmt") = "data %d\n";
+
+SEC("kprobe")
+__auxiliary
+int verifier_percpu_read(void *ctx)
+{
+	char c = fmt2[4];
+
+	return c == ' ';
+}
+
 char _license[] SEC("license") = "GPL";
Signature · 1 line
2.54.0
author
#22 lore raw

PATCH v6 12/12 selftests/bpf: Add test to verify bpf_iter for global percpu data

In reply to Leon Hwang

Add a test to verify that it is OK to iter the percpu_array map used for global percpu data.

---
 .../bpf/prog_tests/global_data_init.c         | 52 +++++++++++++++++++
 .../bpf/progs/test_global_percpu_data.c       | 25 +++++++++
 2 files changed, 77 insertions(+)
tools/testing/selftests/bpf/prog_tests/global_data_init.c +52 -0
index fcc2c4ec8644..635084d057c2 100644
@@ -364,6 +364,56 @@ static void test_global_percpu_data_xlated(void)
 	free(insns);
 }
 
+static void test_global_percpu_data_iter(void)
+{
+	DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);
+	struct test_global_percpu_data *skel;
+	union bpf_iter_link_info linfo = {};
+	struct bpf_link *link = NULL;
+	int fd, num_cpus, len, err;
+	char buf[16];
+
+	num_cpus = libbpf_num_possible_cpus();
+	if (!ASSERT_GT(num_cpus, 0, "libbpf_num_possible_cpus"))
+		return;
+
+	skel = test_global_percpu_data__open();
+	if (!ASSERT_OK_PTR(skel, "test_global_percpu_data__open"))
+		return;
+
+	skel->rodata->num_cpus = num_cpus;
+	skel->rodata->offsetof_num = offsetof(struct test_global_percpu_data__percpu, struct_data);
+	skel->rodata->offsetof_num += sizeof(skel->percpu->struct_data) - sizeof(int);
+	skel->rodata->elem_sz = roundup(sizeof(struct test_global_percpu_data__percpu), 8);
+	skel->percpu->struct_data.nums[6] = 0xc0de;
+
+	err = test_global_percpu_data__load(skel);
+	if (!ASSERT_OK(err, "test_global_percpu_data__load"))
+		goto out;
+
+	linfo.map.map_fd = bpf_map__fd(skel->maps.percpu);
+	opts.link_info = &linfo;
+	opts.link_info_len = sizeof(linfo);
+	link = bpf_program__attach_iter(skel->progs.dump_percpu_data, &opts);
+	if (!ASSERT_OK_PTR(link, "bpf_program__attach_iter"))
+		goto out;
+
+	fd = bpf_iter_create(bpf_link__fd(link));
+	if (!ASSERT_GE(fd, 0, "bpf_iter_create"))
+		goto out;
+
+	while ((len = read(fd, buf, sizeof(buf))) > 0)
+		do { } while (0);
+	ASSERT_EQ(len, 0, "read iter");
+	ASSERT_TRUE(skel->bss->run_iter, "run_iter");
+	ASSERT_EQ(skel->bss->percpu_data_sum, 0xc0de * num_cpus, "percpu_data_sum");
+
+	close(fd);
+out:
+	bpf_link__destroy(link);
+	test_global_percpu_data__destroy(skel);
+}
+
 void test_global_percpu_data(void)
 {
 	if (!feat_supported(NULL, FEAT_PERCPU_DATA)) {
@@ -380,4 +430,6 @@ void test_global_percpu_data(void)
 	test_global_percpu_data_verifier_log();
 	if (test__start_subtest("xlated"))
 		test_global_percpu_data_xlated();
+	if (test__start_subtest("iter"))
+		test_global_percpu_data_iter();
 }
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +25 -0
index 9a1b1a314c2f..037c8a05be4b 100644
@@ -62,4 +62,29 @@ int verifier_percpu_read(void *ctx)
 	return c == ' ';
 }
 
+volatile const __u32 num_cpus = 0;
+volatile const int offsetof_num;
+volatile const int elem_sz;
+__u32 percpu_data_sum = 0;
+bool run_iter = false;
+
+SEC("iter/bpf_map_elem")
+__auxiliary
+int dump_percpu_data(struct bpf_iter__bpf_map_elem *ctx)
+{
+	void *pptr = ctx->value;
+	int i;
+
+	if (!pptr)
+		return 0;
+
+	run_iter = true;
+
+	for (i = 0; i < num_cpus; i++) {
+		percpu_data_sum += *(int *) (pptr + offsetof_num);
+		pptr += elem_sz;
+	}
+	return 0;
+}
+
 char _license[] SEC("license") = "GPL";
Signature · 1 line
2.54.0
#13 lore raw

Re: [PATCH bpf-next v6 10/12] selftests/bpf: Add tests to verify verifier log for global percpu data

In reply to Leon Hwang
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] Incorrect `data_len` argument passed to `bpf_snprintf` makes the negative test brittle. --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Incorrect data_len argument passed to bpf_snprintf makes the negative test brittle. --

commit d5b0b79b30e924d1fec11028be9289d47ccc5221
Author: Leon Hwang <[email protected]>

selftests/bpf: Add tests to verify verifier log for global percpu data

This commit adds new eBPF programs to verify the verifier log correctly rejects bpf_strncmp and bpf_snprintf when a percpu array map is used as a const string. It also updates the prog test to execute these new checks.

Quoted 4 lines
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +0 -0
index ba92ffb0ca499..2222ad3a49bd7 100644

[ ... ]

tools/testing/selftests/bpf/progs/test_global_percpu_data.c int update_percpu_data(void *ctx) · line 48

Quoted 9 lines
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +9 -0
+SEC("?kprobe")
+__failure __msg("R{{[0-9]+}} points to percpu_array map which cannot be used as const string")
+int verifier_snprintf(void *ctx)
+{
+	u64 args[] = { data };
+	char buf[128];
+	int len;
+
+	len = bpf_snprintf(buf, sizeof(buf), fmt, args, 1);

[Severity: Low]
Should this be passing sizeof(args) instead of 1?

The bpf_snprintf helper requires data_len to be the size of the data array in bytes. Passing 1 makes the test brittle if the verifier's argument evaluation order changes in the future, as it would fail on the data_len argument before checking the fmt argument.

Quoted 4 lines
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +4 -0
+	if (len > 0)
+		bpf_printk("snprintf: %s\n", buf);
+	return 0;
+}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
#7 lore raw

Re: [PATCH bpf-next v6 06/12] libbpf: Add support for global percpu data

In reply to Leon Hwang
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Missing endianness conversion for `flags` in `bpf_gen__map_update_elem` breaks cross-compiled BPF…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Missing endianness conversion for flags in bpf_gen__map_update_elem breaks cross-compiled BPF skeleton loaders. --

commit fc64dc7d301e1cf5be2bffba25f597b23162b717
Author: Leon Hwang <[email protected]>

libbpf: Add support for global percpu data

This commit adds support for global percpu data in libbpf by introducing a new ".percpu" section. It updates the gen_loader to populate the percpu_array map using the BPF_F_ALL_CPUS flag.

Quoted 4 lines
tools/lib/bpf/gen_loader.c +0 -0
index d79695f01c87f..7ceb096f18ecb 100644

[ ... ]

tools/lib/bpf/gen_loader.c void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue, · line 1201

Quoted 5 lines
tools/lib/bpf/gen_loader.c +1 -0
@@ -1198,6 +1198,7 @@ void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue,
 	int zero = 0;
 
 	memset(&attr, 0, attr_size);
+	attr.flags = flags;

[Severity: High]
Does this assignment need to use tgt_endian() to support cross-compilation?

When libbpf generates a skeleton loader for a target architecture with a different endianness than the host, fields populated into the union bpf_attr blob usually need to be converted to the target byte order.

For global percpu data maps, this flag will be BPF_F_ALL_CPUS (0x10). If a little-endian host generates a skeleton for a big-endian target, a direct assignment of the 64-bit value could result in it being interpreted as 0x1000000000000000 by the target kernel, which would cause the sys_bpf call to fail with -EINVAL.

Quoted 3 lines
tools/lib/bpf/gen_loader.c +0 -0
 	value = add_data(gen, pvalue, value_size);
 	key = add_data(gen, &zero, sizeof(zero));
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
#16 lore raw

Re: [PATCH bpf-next v6 11/12] selftests/bpf: Add test to verify xlated insns for global percpu data

In reply to Leon Hwang
Quoted 88 lines · Leon Hwang

Add a test to verify global percpu data related xlated insns:

1. ld_imm64: compare xlated one with the one in ELF object file.
2. mov64_percpu_reg: it is added by verifier.

---
 .../bpf/prog_tests/global_data_init.c         | 87 +++++++++++++++++++
 .../bpf/progs/test_global_percpu_data.c       | 11 +++
 2 files changed, 98 insertions(+)
tools/testing/selftests/bpf/prog_tests/global_data_init.c +69 -0
index e4c290492ff8..fcc2c4ec8644 100644
@@ -279,6 +279,91 @@ static void test_global_percpu_data_verifier_log(void)
 	RUN_TESTS(test_global_percpu_data);
 }
 
+static int find_ld_imm64(const struct bpf_insn *insns, size_t insn_cnt, struct bpf_insn *ld_imm64)
+{
+	size_t i;
+
+	for (i = 0; i < insn_cnt; i++) {
+		if (insns[i].code == (BPF_LD | BPF_IMM | BPF_DW)) {
+			ld_imm64[0] = insns[i];
+			ld_imm64[1] = insns[i + 1];
+			return i;
+		}
+	}
+
+	return -ENOENT;
+}
+
+/*
+ * Special (internal-only) form of mov, used to resolve per-CPU addrs:
+ * dst_reg = src_reg + <percpu_base_off>
+ * BPF_ADDR_PERCPU is used as a special insn->off value.
+ */
+#define BPF_ADDR_PERCPU	(-1)
+
+#define BPF_MOV64_PERCPU_REG(DST, SRC)				\
+	((struct bpf_insn) {					\
+		.code  = BPF_ALU64 | BPF_MOV | BPF_X,		\
+		.dst_reg = DST,					\
+		.src_reg = SRC,					\
+		.off   = BPF_ADDR_PERCPU,			\
+		.imm   = 0 })
+
+static __u64 ld_imm64_to_u64(const struct bpf_insn *insn)
+{
+	return ((__u64)(__u32) insn[1].imm << 32) | (__u32) insn[0].imm;
+}
+
+static void test_global_percpu_data_xlated(void)
+{
+	struct bpf_insn ld_imm64_raw[2], ld_imm64_xlated[2], mov64_percpu_reg, *insns = NULL;
+	size_t insn_sz = sizeof(struct bpf_insn);
+	struct test_global_percpu_data *skel;
+	struct bpf_program *prog;
+	int idx, err;
+	__u32 cnt;
+
+	skel = test_global_percpu_data__open();
+	if (!ASSERT_OK_PTR(skel, "test_global_percpu_data__open"))
+		return;
+
+	prog = skel->progs.verifier_percpu_read;
+	idx = find_ld_imm64(bpf_program__insns(prog), bpf_program__insn_cnt(prog), ld_imm64_raw);
+	if (!ASSERT_GE(idx, 0, "find_ld_imm64 raw"))
+		goto out;
+
+	err = test_global_percpu_data__load(skel);
+	if (!ASSERT_OK(err, "test_global_percpu_data__load"))
+		goto out;
+
+	err = get_xlated_program(bpf_program__fd(prog), &insns, &cnt);
+	if (!ASSERT_OK(err, "get_xlated_program"))
+		goto out;
+
+	idx = find_ld_imm64(insns, cnt, ld_imm64_xlated);
+	if (!ASSERT_GE(idx, 0, "find_ld_imm64 xlated"))
+		goto out;
+
+	if (!ASSERT_GT(cnt, idx + 2, "xlated insn count"))
+		goto out;
+
+	ASSERT_EQ(ld_imm64_xlated[0].code, ld_imm64_raw[0].code, "ld_imm64 opcode");

tools/testing/selftests/bpf/prog_tests/global_data_init.c static void test_global_percpu_data_verifier_log(void) · line 360

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +10 -0
+	ASSERT_TRUE(ld_imm64_xlated[0].dst_reg == ld_imm64_raw[0].dst_reg, "ld_imm64 dst_reg");
+	/*
+	 * The xlated instruction has the map ID in imm and the offset
+	 * in the next instruction's imm. The raw instruction just has
+	 * the offset in its imm.
+	 */
+	ASSERT_EQ(ld_imm64_xlated[1].imm, ld_imm64_to_u64(ld_imm64_raw), "ld_imm64 off");
+
+	mov64_percpu_reg = BPF_MOV64_PERCPU_REG(ld_imm64_raw[0].dst_reg, ld_imm64_raw[0].dst_reg);
+	ASSERT_MEMEQ(&insns[idx + 2], &mov64_percpu_reg, insn_sz, "mov64_percpu_reg");

If the point of the test was to check that percpu_array_map_direct_value_meta() computes 'off' correctly then it failed to achieve that goal.
It checks that map ID is correct which is a pointless test.
Either make it a real test or drop this patch.

pw-bot: cr

author
#8 lore raw

Re: [PATCH bpf-next v6 06/12] libbpf: Add support for global percpu data

In reply to [email protected]

On 15/6/26 23:41, [email protected] wrote:
[...]

Quoted 14 lines

tools/lib/bpf/gen_loader.c void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue, · line 1201

Quoted 6 lines
tools/lib/bpf/gen_loader.c +1 -0
@@ -1198,6 +1198,7 @@ void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue,
 	int zero = 0;
 
 	memset(&attr, 0, attr_size);
+	attr.flags = flags;

[Severity: High]
Does this assignment need to use tgt_endian() to support cross-compilation?

When libbpf generates a skeleton loader for a target architecture with a different endianness than the host, fields populated into the union bpf_attr blob usually need to be converted to the target byte order.

Quoted 6 lines

For global percpu data maps, this flag will be BPF_F_ALL_CPUS (0x10). If a little-endian host generates a skeleton for a big-endian target, a direct assignment of the 64-bit value could result in it being interpreted as 0x1000000000000000 by the target kernel, which would cause the sys_bpf call to fail with -EINVAL.

Will use tgt_endian() in the next revision.

Thanks,
Leon

author
#14 lore raw

Re: [PATCH bpf-next v6 10/12] selftests/bpf: Add tests to verify verifier log for global percpu data

In reply to [email protected]

On 15/6/26 23:40, [email protected] wrote:
[...]

Quoted 13 lines

tools/testing/selftests/bpf/progs/test_global_percpu_data.c int update_percpu_data(void *ctx) · line 48

Quoted 9 lines
tools/testing/selftests/bpf/progs/test_global_percpu_data.c +9 -0
+SEC("?kprobe")
+__failure __msg("R{{[0-9]+}} points to percpu_array map which cannot be used as const string")
+int verifier_snprintf(void *ctx)
+{
+	u64 args[] = { data };
+	char buf[128];
+	int len;
+
+	len = bpf_snprintf(buf, sizeof(buf), fmt, args, 1);

[Severity: Low]
Should this be passing sizeof(args) instead of 1?

Quoted 5 lines

The bpf_snprintf helper requires data_len to be the size of the data array in bytes. Passing 1 makes the test brittle if the verifier's argument evaluation order changes in the future, as it would fail on the data_len argument before checking the fmt argument.

Will use sizeof(args).

Thanks,
Leon

author
#17 lore raw

Re: [PATCH bpf-next v6 11/12] selftests/bpf: Add test to verify xlated insns for global percpu data

In reply to Alexei Starovoitov
Quoted 1 line · Alexei Starovoitov

On Mon Jun 15, 2026 at 8:26 AM PDT, Leon Hwang wrote:

[...]

Quoted 6 lines
Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +6 -0
+	ASSERT_EQ(ld_imm64_xlated[0].code, ld_imm64_raw[0].code, "ld_imm64 opcode");
+	ASSERT_TRUE(ld_imm64_xlated[0].dst_reg == ld_imm64_raw[0].dst_reg, "ld_imm64 dst_reg");
+	/*
+	 * The xlated instruction has the map ID in imm and the offset
+	 * in the next instruction's imm. The raw instruction just has
+	 * the offset in its imm.
Quoted 8 lines

tools/testing/selftests/bpf/prog_tests/global_data_init.c static void test_global_percpu_data_verifier_log(void) · line 360

Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +5 -0
+	 */
+	ASSERT_EQ(ld_imm64_xlated[1].imm, ld_imm64_to_u64(ld_imm64_raw), "ld_imm64 off");
+
+	mov64_percpu_reg = BPF_MOV64_PERCPU_REG(ld_imm64_raw[0].dst_reg, ld_imm64_raw[0].dst_reg);
+	ASSERT_MEMEQ(&insns[idx + 2], &mov64_percpu_reg, insn_sz, "mov64_percpu_reg");

If the point of the test was to check that percpu_array_map_direct_value_meta() computes 'off' correctly then it failed to achieve that goal.

It was to check the 'off' and the mov64_percpu_reg insn.

To avoid relying on the insns loaded from ELF obj, use raw insns to check the 'off' with a percpu_array map:

	struct bpf_insn raw_insns[] = {
		BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0),
		BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, 0),
		BPF_EXIT_INSN(),
	};

	skel = test_global_percpu_data__open_and_load();
	exp_off = offsetof(struct test_global_percpu_data__percpu,
struct_data.nums[6]);
	raw_insns[0].imm = bpf_map__fd(skel->maps.percpu);
	raw_insns[1].imm = exp_off;

	prog_fd = bpf_prog_load(...);
	err = get_xlated_program(prog_fd, &insns, &cnt);
	idx = find_percpu_ld_imm64(insns, cnt);

	ASSERT_EQ(insns[idx + 1].imm, exp_off, "exp_off");

Would this achieve that goal?

Quoted 3 lines

It checks that map ID is correct which is a pointless test. Either make it a real test or drop this patch.

No map ID check here.

Thanks,
Leon

#18 lore raw

Re: [PATCH bpf-next v6 11/12] selftests/bpf: Add test to verify xlated insns for global percpu data

Quoted 42 lines · Leon Hwang
Quoted 1 line · Alexei Starovoitov

On Mon Jun 15, 2026 at 8:26 AM PDT, Leon Hwang wrote:

[...]

Quoted 12 lines

tools/testing/selftests/bpf/prog_tests/global_data_init.c static void test_global_percpu_data_verifier_log(void) · line 360

Quoted 11 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +11 -0
+    ASSERT_EQ(ld_imm64_xlated[0].code, ld_imm64_raw[0].code, "ld_imm64 opcode");
+    ASSERT_TRUE(ld_imm64_xlated[0].dst_reg == ld_imm64_raw[0].dst_reg, "ld_imm64 dst_reg");
+    /*
+     * The xlated instruction has the map ID in imm and the offset
+     * in the next instruction's imm. The raw instruction just has
+     * the offset in its imm.
+     */
+    ASSERT_EQ(ld_imm64_xlated[1].imm, ld_imm64_to_u64(ld_imm64_raw), "ld_imm64 off");
+
+    mov64_percpu_reg = BPF_MOV64_PERCPU_REG(ld_imm64_raw[0].dst_reg, ld_imm64_raw[0].dst_reg);
+    ASSERT_MEMEQ(&insns[idx + 2], &mov64_percpu_reg, insn_sz, "mov64_percpu_reg");
Quoted 2 lines

If the point of the test was to check that percpu_array_map_direct_value_meta() computes 'off' correctly then it failed to achieve that goal.

It was to check the 'off' and the mov64_percpu_reg insn.

To avoid relying on the insns loaded from ELF obj, use raw insns to check the 'off' with a percpu_array map:

        struct bpf_insn raw_insns[] = {
                BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0),
                BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, 0),
                BPF_EXIT_INSN(),
        };

        skel = test_global_percpu_data__open_and_load();
        exp_off = offsetof(struct test_global_percpu_data__percpu,
struct_data.nums[6]);
        raw_insns[0].imm = bpf_map__fd(skel->maps.percpu);
        raw_insns[1].imm = exp_off;

        prog_fd = bpf_prog_load(...);
        err = get_xlated_program(prog_fd, &insns, &cnt);
        idx = find_percpu_ld_imm64(insns, cnt);

        ASSERT_EQ(insns[idx + 1].imm, exp_off, "exp_off");
Quoted 1 line

Would this achieve that goal?

Hmm. There is nothing like that in the patch. What are you talking about?
Some future improvement to the patch? My question was about the patch you posted. What was the point of it?

author
#19 lore raw

Re: [PATCH bpf-next v6 11/12] selftests/bpf: Add test to verify xlated insns for global percpu data

Quoted 41 lines · Alexei Starovoitov
Quoted 40 lines · Leon Hwang
Quoted 1 line · Alexei Starovoitov

On Mon Jun 15, 2026 at 8:26 AM PDT, Leon Hwang wrote:

[...]

Quoted 12 lines

tools/testing/selftests/bpf/prog_tests/global_data_init.c static void test_global_percpu_data_verifier_log(void) · line 360

Quoted 11 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +11 -0
+    ASSERT_EQ(ld_imm64_xlated[0].code, ld_imm64_raw[0].code, "ld_imm64 opcode");
+    ASSERT_TRUE(ld_imm64_xlated[0].dst_reg == ld_imm64_raw[0].dst_reg, "ld_imm64 dst_reg");
+    /*
+     * The xlated instruction has the map ID in imm and the offset
+     * in the next instruction's imm. The raw instruction just has
+     * the offset in its imm.
+     */
+    ASSERT_EQ(ld_imm64_xlated[1].imm, ld_imm64_to_u64(ld_imm64_raw), "ld_imm64 off");
+
+    mov64_percpu_reg = BPF_MOV64_PERCPU_REG(ld_imm64_raw[0].dst_reg, ld_imm64_raw[0].dst_reg);
+    ASSERT_MEMEQ(&insns[idx + 2], &mov64_percpu_reg, insn_sz, "mov64_percpu_reg");
Quoted 2 lines

If the point of the test was to check that percpu_array_map_direct_value_meta() computes 'off' correctly then it failed to achieve that goal.

It was to check the 'off' and the mov64_percpu_reg insn.

To avoid relying on the insns loaded from ELF obj, use raw insns to check the 'off' with a percpu_array map:

        struct bpf_insn raw_insns[] = {
                BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0),
                BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, 0),
                BPF_EXIT_INSN(),
        };

        skel = test_global_percpu_data__open_and_load();
        exp_off = offsetof(struct test_global_percpu_data__percpu,
struct_data.nums[6]);
        raw_insns[0].imm = bpf_map__fd(skel->maps.percpu);
        raw_insns[1].imm = exp_off;

        prog_fd = bpf_prog_load(...);
        err = get_xlated_program(prog_fd, &insns, &cnt);
        idx = find_percpu_ld_imm64(insns, cnt);
Quoted 7 lines
Quoted 3 lines
        ASSERT_EQ(insns[idx + 1].imm, exp_off, "exp_off");

Would this achieve that goal?

Hmm. There is nothing like that in the patch. What are you talking about?
Some future improvement to the patch?

It is future improvement to the patch. Sorry, it is too far from the patch.

Quoted 2 lines

My question was about the patch you posted. What was the point of it?

The point of the patch was to check that percpu_array_map_direct_value_meta() computes 'off' correctly against the 'off' encoded in ELF obj, and a mov64_percpu_reg insn should be added after loading the prog.

Thanks,
Leon

#20 lore raw

Re: [PATCH bpf-next v6 11/12] selftests/bpf: Add test to verify xlated insns for global percpu data

Quoted 52 lines · Leon Hwang
Quoted 41 lines · Alexei Starovoitov
Quoted 40 lines · Leon Hwang
Quoted 1 line · Alexei Starovoitov

On Mon Jun 15, 2026 at 8:26 AM PDT, Leon Hwang wrote:

[...]

Quoted 12 lines

tools/testing/selftests/bpf/prog_tests/global_data_init.c static void test_global_percpu_data_verifier_log(void) · line 360

Quoted 11 lines
tools/testing/selftests/bpf/prog_tests/global_data_init.c +11 -0
+    ASSERT_EQ(ld_imm64_xlated[0].code, ld_imm64_raw[0].code, "ld_imm64 opcode");
+    ASSERT_TRUE(ld_imm64_xlated[0].dst_reg == ld_imm64_raw[0].dst_reg, "ld_imm64 dst_reg");
+    /*
+     * The xlated instruction has the map ID in imm and the offset
+     * in the next instruction's imm. The raw instruction just has
+     * the offset in its imm.
+     */
+    ASSERT_EQ(ld_imm64_xlated[1].imm, ld_imm64_to_u64(ld_imm64_raw), "ld_imm64 off");
+
+    mov64_percpu_reg = BPF_MOV64_PERCPU_REG(ld_imm64_raw[0].dst_reg, ld_imm64_raw[0].dst_reg);
+    ASSERT_MEMEQ(&insns[idx + 2], &mov64_percpu_reg, insn_sz, "mov64_percpu_reg");
Quoted 2 lines

If the point of the test was to check that percpu_array_map_direct_value_meta() computes 'off' correctly then it failed to achieve that goal.

It was to check the 'off' and the mov64_percpu_reg insn.

To avoid relying on the insns loaded from ELF obj, use raw insns to check the 'off' with a percpu_array map:

        struct bpf_insn raw_insns[] = {
                BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0),
                BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, 0),
                BPF_EXIT_INSN(),
        };

        skel = test_global_percpu_data__open_and_load();
        exp_off = offsetof(struct test_global_percpu_data__percpu,
struct_data.nums[6]);
        raw_insns[0].imm = bpf_map__fd(skel->maps.percpu);
        raw_insns[1].imm = exp_off;

        prog_fd = bpf_prog_load(...);
        err = get_xlated_program(prog_fd, &insns, &cnt);
        idx = find_percpu_ld_imm64(insns, cnt);
Quoted 7 lines
Quoted 3 lines
        ASSERT_EQ(insns[idx + 1].imm, exp_off, "exp_off");

Would this achieve that goal?

Hmm. There is nothing like that in the patch. What are you talking about?
Some future improvement to the patch?

It is future improvement to the patch. Sorry, it is too far from the patch.

Quoted 7 lines
Quoted 2 lines

My question was about the patch you posted. What was the point of it?

The point of the patch was to check that percpu_array_map_direct_value_meta() computes 'off' correctly against the 'off' encoded in ELF obj, and a mov64_percpu_reg insn should be added after loading the prog.

If that was the goal then it wasn't achieved. Just test manually that the 'off' is updated correctly and skip such selftest. There is little value.

author
#21 lore raw

Re: [PATCH bpf-next v6 11/12] selftests/bpf: Add test to verify xlated insns for global percpu data

On 2026/6/20 11:29, Alexei Starovoitov wrote:
[...]

Quoted 10 lines
Quoted 5 lines

The point of the patch was to check that percpu_array_map_direct_value_meta() computes 'off' correctly against the 'off' encoded in ELF obj, and a mov64_percpu_reg insn should be added after loading the prog.

If that was the goal then it wasn't achieved. Just test manually that the 'off' is updated correctly and skip such selftest. There is little value.

Got it. Will drop this patch.

Thanks,
Leon