Skip to contentSkip to conversation

bpf: fix per-cpu initialization of a BPF_F_CPU created hash element

PATCH Applied Donggeun Yoo 2 participants 4 messages Sep 24

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v3 cover bpf: fix per-cpu initialization of a BPF_F_CPU created hash element

A BPF_F_CPU update that creates a [lru_]percpu_hash element writes the named CPU's slot and leaves the others holding the recycled element's values, so a lookup of the new key returns a deleted key's per-cpu values.

Patch 1 zero-fills the other CPUs. Patch 2 adds the selftest: the existing cpu_flag subtests always prime a key with BPF_F_ALL_CPUS first, so the create path is not covered today.

v1: https://lore.kernel.org/bpf/[email protected]/
v2: https://lore.kernel.org/bpf/[email protected]/

Changes in v3:
- patch 1: key init_cpu on BPF_F_CPU rather than on onallcpus (Leon Hwang)
- patch 1: re-flow the paragraph naming pcpu_copy_value() (BPF CI AI review)
- patch 2: pin the thread across the delete and the create, and name a CPU other than the pinned one, so the BPF_F_NO_PREALLOC arm does not depend on staying put (BPF CI AI review)

Changes in v2:
- patch 1: cover the BPF_F_CPU entry condition in the block comment above pcpu_init_value() (BPF CI AI review, Alexei Starovoitov)
- patch 2: skip the new subtests instead of failing them on a uniprocessor machine (Sashiko AI review)

test_progs -t percpu_alloc and fourteen neighboring map tests, x86_64 under QEMU/KVM, 4 vCPUs:

without patch 1 31/112 PASSED, 1/3 FAILED
with patch 1 32/115 PASSED, 0/0 FAILED

Donggeun Yoo (2):
bpf: Zero-fill other CPUs when BPF_F_CPU creates a per-cpu hash
element
selftests/bpf: Test per-cpu initialization of a BPF_F_CPU created
element

 kernel/bpf/hashtab.c                          |  11 +-
 .../selftests/bpf/prog_tests/percpu_alloc.c   | 105 ++++++++++++++++++
 2 files changed, 112 insertions(+), 4 deletions(-)
Signature · 1 line
2.53.0
#1 lore raw

PATCH v3 1/2 bpf: Zero-fill other CPUs when BPF_F_CPU creates a per-cpu hash element

pcpu_init_value() initializes the per-cpu area of a newly created [lru_]percpu_hash element. The area is recycled, so when the value comes from a BPF program (onallcpus == false) it writes the running CPU's slot and zeroes the rest.

bpf_percpu_hash_update() passes onallcpus == true, which delegates to pcpu_copy_value(). pcpu_copy_value() writes only the CPU named in map_flags when BPF_F_CPU is set, so on the create path the other slots keep the recycled element's values:

  update(k1, 0xdeadc0de, BPF_F_ALL_CPUS)  every CPU holds 0xdeadc0de
  delete(k1)                              element back on the freelist
  update(k2, 0xc0ffee, BPF_F_CPU | 0)     creates, writes CPU 0 only
  lookup(k2)                              CPU 0 0xc0ffee, rest 0xdeadc0de

Zero-fill the other CPUs on that arm too.

Fixes c6936161fd55 ("bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_hash and lru_percpu_hash maps")
---
 kernel/bpf/hashtab.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
kernel/bpf/hashtab.c +7 -4
index 4f495dcbf670c..f9464e566f109 100644
@@ -1054,14 +1054,17 @@ static void pcpu_init_value(struct bpf_htab *htab, void __percpu *pptr,
 	/* When not setting the initial value on all cpus, zero-fill element
 	 * values for other cpus. Otherwise, bpf program has no way to ensure
 	 * known initial values for cpus other than current one
-	 * (onallcpus=false always when coming from bpf prog).
+	 * (onallcpus=false always when coming from bpf prog,
+	 *  map_flags & BPF_F_CPU when coming from syscall but setting
+	 *  only one cpu).
 	 */
-	if (!onallcpus) {
-		int current_cpu = raw_smp_processor_id();
+	if (!onallcpus || (map_flags & BPF_F_CPU)) {
+		int init_cpu = (map_flags & BPF_F_CPU) ? map_flags >> 32 :
+			       raw_smp_processor_id();
 		int cpu;
 
 		for_each_possible_cpu(cpu) {
-			if (cpu == current_cpu)
+			if (cpu == init_cpu)
 				copy_map_value(&htab->map, per_cpu_ptr(pptr, cpu), value);
 			else /* Since elem is preallocated, we cannot touch special fields */
 				zero_map_value(&htab->map, per_cpu_ptr(pptr, cpu));
Signature · 1 line
2.53.0
#2 lore raw

PATCH v3 2/2 selftests/bpf: Test per-cpu initialization of a BPF_F_CPU created element

In reply to Donggeun Yoo

The existing cpu_flag subtests always prime a key with BPF_F_ALL_CPUS before any BPF_F_CPU write, so the create path is never covered.

Add a subtest that creates the element with BPF_F_CPU on a map with max_entries 1, so the key can only reuse the element the previous key released, and check that the CPUs the update did not name read back zero. Run it for PERCPU_HASH preallocated and BPF_F_NO_PREALLOC, whose per-cpu areas come from different allocators, and for LRU_PERCPU_HASH.

Under BPF_F_NO_PREALLOC the reuse is only guaranteed on the cpu that ran the delete, so pin the thread across the pair, and name a cpu other than that one in map_flags.

---
 .../selftests/bpf/prog_tests/percpu_alloc.c   | 105 ++++++++++++++++++
 1 file changed, 105 insertions(+)
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c +105 -0
index a72ae0b29f6e9..7b4a1e24363b5 100644
@@ -1,4 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0
+#define _GNU_SOURCE
+#include <sched.h>
 #include <test_progs.h>
 #include "cgroup_helpers.h"
 #include "percpu_alloc_array.skel.h"
@@ -350,6 +352,103 @@ static void test_lru_percpu_hash_cpu_flag(void)
 	test_percpu_map_cpu_flag(BPF_MAP_TYPE_LRU_PERCPU_HASH);
 }
 
+/*
+ * A BPF_F_CPU update that creates an element must zero the value on the other
+ * cpus, rather than leave them holding whatever the recycled element last
+ * contained. max_entries is 1 so the second key can only reuse the element
+ * the first one released.
+ */
+static void test_percpu_map_cpu_flag_create(enum bpf_map_type map_type, __u32 map_flags)
+{
+	LIBBPF_OPTS(bpf_map_create_opts, opts, .map_flags = map_flags);
+	const u32 stale = 0xDEADC0DE, fresh = 0xC0FFEE;
+	int nr_cpus, cpu, map_fd, err, key;
+	int pinned_cpu, value_cpu;
+	cpu_set_t old_mask, new_mask;
+	bool restore_mask = false;
+	u32 value;
+	u64 flags;
+
+	nr_cpus = libbpf_num_possible_cpus();
+	if (!ASSERT_GT(nr_cpus, 0, "libbpf_num_possible_cpus"))
+		return;
+
+	if (nr_cpus < 2) {
+		test__skip();
+		return;
+	}
+
+	map_fd = bpf_map_create(map_type, "cpu_flag_create", sizeof(key), sizeof(value), 1, &opts);
+	if (!ASSERT_GE(map_fd, 0, "bpf_map_create"))
+		return;
+
+	/* NO_PREALLOC recycles per cpu, so keep the delete and the create on one cpu. */
+	err = sched_getaffinity(0, sizeof(old_mask), &old_mask);
+	if (!ASSERT_OK(err, "sched_getaffinity"))
+		goto out;
+
+	pinned_cpu = sched_getcpu();
+	if (!ASSERT_GE(pinned_cpu, 0, "sched_getcpu"))
+		goto out;
+
+	CPU_ZERO(&new_mask);
+	CPU_SET(pinned_cpu, &new_mask);
+	err = sched_setaffinity(0, sizeof(new_mask), &new_mask);
+	if (!ASSERT_OK(err, "sched_setaffinity"))
+		goto out;
+	restore_mask = true;
+
+	value_cpu = pinned_cpu ? 0 : 1;
+
+	key = 1;
+	value = stale;
+	err = bpf_map_update_elem(map_fd, &key, &value, BPF_F_ALL_CPUS);
+	if (!ASSERT_OK(err, "bpf_map_update_elem all_cpus"))
+		goto out;
+
+	err = bpf_map_delete_elem(map_fd, &key);
+	if (!ASSERT_OK(err, "bpf_map_delete_elem"))
+		goto out;
+
+	key = 2;
+	value = fresh;
+	flags = (u64)value_cpu << 32 | BPF_F_CPU;
+	err = bpf_map_update_elem(map_fd, &key, &value, flags);
+	if (!ASSERT_OK(err, "bpf_map_update_elem specified cpu"))
+		goto out;
+
+	for (cpu = 0; cpu < nr_cpus; cpu++) {
+		value = 0;
+		flags = (u64)cpu << 32 | BPF_F_CPU;
+		err = bpf_map_lookup_elem_flags(map_fd, &key, &value, flags);
+		if (!ASSERT_OK(err, "bpf_map_lookup_elem_flags specified cpu"))
+			goto out;
+		if (!ASSERT_EQ(value, cpu == value_cpu ? fresh : 0, "value on specified cpu"))
+			goto out;
+	}
+
+out:
+	if (restore_mask)
+		sched_setaffinity(0, sizeof(old_mask), &old_mask);
+	close(map_fd);
+}
+
+static void test_percpu_hash_cpu_flag_create(void)
+{
+	test_percpu_map_cpu_flag_create(BPF_MAP_TYPE_PERCPU_HASH, 0);
+}
+
+static void test_percpu_hash_cpu_flag_create_malloc(void)
+{
+	test_percpu_map_cpu_flag_create(BPF_MAP_TYPE_PERCPU_HASH, BPF_F_NO_PREALLOC);
+}
+
+static void test_lru_percpu_hash_cpu_flag_create(void)
+{
+	/* lru without prealloc is -ENOTSUPP, so there is no malloc variant */
+	test_percpu_map_cpu_flag_create(BPF_MAP_TYPE_LRU_PERCPU_HASH, 0);
+}
+
 static void test_percpu_cgroup_storage_cpu_flag(void)
 {
 	struct percpu_alloc_array *skel = NULL;
@@ -454,6 +553,12 @@ void test_percpu_alloc(void)
 		test_percpu_hash_cpu_flag();
 	if (test__start_subtest("cpu_flag_lru_percpu_hash"))
 		test_lru_percpu_hash_cpu_flag();
+	if (test__start_subtest("cpu_flag_create_percpu_hash"))
+		test_percpu_hash_cpu_flag_create();
+	if (test__start_subtest("cpu_flag_create_percpu_hash_malloc"))
+		test_percpu_hash_cpu_flag_create_malloc();
+	if (test__start_subtest("cpu_flag_create_lru_percpu_hash"))
+		test_lru_percpu_hash_cpu_flag_create();
 	if (test__start_subtest("cpu_flag_percpu_cgroup_storage"))
 		test_percpu_cgroup_storage_cpu_flag();
 	if (test__start_subtest("cpu_flag_array"))
Signature · 1 line
2.53.0
#3 lore raw
In reply to Donggeun Yoo
This series was applied to bpf/bpf.git (master) by Alexei Starovoitov <[email protected]>:

Hello:

This series was applied to bpf/bpf.git (master)
by Alexei Starovoitov <[email protected]>:

Quoted 10 lines · On Thu, 24 Sep 2026 19:23:19 +0900 you wrote:

A BPF_F_CPU update that creates a [lru_]percpu_hash element writes the named CPU's slot and leaves the others holding the recycled element's values, so a lookup of the new key returns a deleted key's per-cpu values.

Patch 1 zero-fills the other CPUs. Patch 2 adds the selftest: the existing cpu_flag subtests always prime a key with BPF_F_ALL_CPUS first, so the create path is not covered today.

[...]

Here is the summary with links:
- [bpf,v3,1/2] bpf: Zero-fill other CPUs when BPF_F_CPU creates a per-cpu hash element

    https://git.kernel.org/bpf/bpf/c/c3a66e5f5bab
  - [bpf,v3,2/2] selftests/bpf: Test per-cpu initialization of a BPF_F_CPU created element
    https://git.kernel.org/bpf/bpf/c/3422808f4e95

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html