Skip to contentSkip to conversation

bpf: Fix incorrect handling of user flags by percpu map updates

PATCH Applied Masoud Aghasi 3 participants 9 messages Oct 8 – Oct 18

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

v6 cover bpf: Fix incorrect handling of user flags by percpu map updates

For BPF_MAP_TYPE_PERCPU_ARRAY map, bpf_percpu_array_update() is not considering the possibility of a combination of (BPF_NOEXIST, BPF_EXIST) flags with (BPF_F_CPU, BPF_F_ALL_CPUS) flags. This causes the (BPF_NOEXIST, BPF_EXIST) flags to lose their effect in some cases.

For BPF_MAP_TYPE_PERCPU_HASH and BPF_MAP_TYPE_LRU_PERCPU_HASH maps, htab_map_check_update_flags() and check_flags() are not considering the possibility of a combination of (BPF_NOEXIST, BPF_EXIST) flags with (BPF_F_CPU, BPF_F_ALL_CPUS) flags. This causes the (BPF_NOEXIST, BPF_EXIST) flags to lose their effect in some cases.

For example, when using (BPF_F_CPU | BPF_EXIST) flag combination with bpf_map_update_elem() on a BPF_MAP_TYPE_PERCPU_HASH map, the BPF_EXIST flag does not prevent new insertions as expected.

This series fixes the bug by adding proper flag validations and checks and adds regression tests.

V6 changes:
- Improve the style and structure of the selftest

V5 changes: https://lore.kernel.org/bpf/[email protected]/T/
- Simplify flag validations by introducing new macros in bpf.h
- Add helpers to verify the values after updates in the new tests
- Improve the style and structure of the new tests

V4 changes: https://lore.kernel.org/bpf/[email protected]/T/
- Edit selftest to use proper value_sz in the new cgroup map test
- Edit selftest to pin the current pid to the current cpu for LRU map
- Edit commit messages to include user-visible changes

V3 changes: https://lore.kernel.org/bpf/[email protected]/T/
- Split the fix into two separate patches for array map and hash maps
- Extend the selftest to cover all percpu map types
- Extend the selftest to cover all applicable flag combinations
- Extend the selftest to cover the unnecessary delete issue of LRU map

V2 changes: https://lore.kernel.org/bpf/[email protected]/T/
- Fix a BPF_EXIST flag check in __htab_lru_percpu_map_update_elem()
- Add additional test for BPF_MAP_TYPE_LRU_PERCPU_HASH map
- Add check for BPF_EXIST, BPF_NOEXIST combination in percpu array map

V1: https://lore.kernel.org/bpf/[email protected]/T/

Masoud Aghasi (3):
bpf: Fix incorrect handling of user flags in bpf_percpu_array_update
bpf: Fix incorrect handling of user flags by percpu hash map updates selftests/bpf: add tests for percpu map flags combination

 include/linux/bpf.h                           |   3 +
 kernel/bpf/arraymap.c                         |   6 +-
 kernel/bpf/hashtab.c                          |  10 +-
 .../selftests/bpf/prog_tests/percpu_alloc.c   | 254 ++++++++++++++++++
 4 files changed, 266 insertions(+), 7 deletions(-)
Signature · 1 line
2.47.3
#1 lore raw

PATCH v6 1/3 bpf: Fix incorrect handling of user flags in bpf_percpu_array_update

For BPF_MAP_TYPE_PERCPU_ARRAY map, bpf_percpu_array_update() is not considering the possibility of a combination of (BPF_NOEXIST, BPF_EXIST) flags with (BPF_F_CPU, BPF_F_ALL_CPUS) flags. This causes the (BPF_NOEXIST, BPF_EXIST) flags to lose their effect in some cases.

For example, using the (BPF_F_ALL_CPUS | BPF_EXIST) flag combination with bpf_map_update_elem() results in an incorrect EINVAL error response, even though the flag combination is valid.

This patch fixes the bug by adding proper flag validations and checks. Before this patch, bpf_percpu_array_update() rejected BPF_F_ALL_CPUS | BPF_EXIST and BPF_F_ALL_CPUS | BPF_NOEXIST with -EINVAL. Also the BPF_F_CPU | BPF_NOEXIST were accepted. After the patch BPF_F_ALL_CPUS | BPF_EXIST is accepted and BPF_F_ALL_CPUS | BPF_NOEXIST and BPF_F_CPU | BPF_NOEXIST result in -EEXIST.

This patch also starts returning -EINVAL for (BPF_EXIST | BPF_NOEXIST), with or without BPF_F_CPU, which was previously accepted.

Fixes 8eb76cb03f0f ("bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_array maps")
---
 include/linux/bpf.h   | 3 +++
 kernel/bpf/arraymap.c | 6 +++---
 2 files changed, 6 insertions(+), 3 deletions(-)
include/linux/bpf.h +3 -0
index 4bae3796c42f..2efa8fdbb737 100644
@@ -4322,6 +4322,9 @@ static inline bool bpf_map_is_percpu_map(enum bpf_map_type map_type)
 	}
 }
 
+#define BPF_EXIST_FLAGS		(BPF_EXIST | BPF_NOEXIST)
+#define BPF_CPU_FLAGS		(BPF_F_CPU | BPF_F_ALL_CPUS)
+
 static inline int bpf_map_check_op_flags(struct bpf_map *map, u64 flags, u64 allowed_flags)
 {
 	u32 cpu;
kernel/bpf/arraymap.c +3 -3
index 0fe9afd4a591..133f189bf6d2 100644
@@ -438,15 +438,15 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value,
 	u32 size;
 	int cpu, off = 0;
 
-	if (unlikely((map_flags & BPF_F_LOCK) || (u32)map_flags > BPF_F_ALL_CPUS))
-		/* unknown flags */
+	if (unlikely(((map_flags & BPF_EXIST_FLAGS) == BPF_EXIST_FLAGS) ||
+		     ((u32)map_flags & ~(BPF_EXIST_FLAGS | BPF_CPU_FLAGS))))
 		return -EINVAL;
 
 	if (unlikely(index >= array->map.max_entries))
 		/* all elements were pre-allocated, cannot insert a new one */
 		return -E2BIG;
 
-	if (unlikely(map_flags == BPF_NOEXIST))
+	if (unlikely(map_flags & BPF_NOEXIST))
 		/* all elements already exist */
 		return -EEXIST;
 
Signature · 1 line
2.47.3
Acked-by on v6 1/3
#2 lore raw

Re: [PATCH bpf v6 1/3] bpf: Fix incorrect handling of user flags in bpf_percpu_array_update

Quoted 18 lines · Masoud Aghasi

For BPF_MAP_TYPE_PERCPU_ARRAY map, bpf_percpu_array_update() is not considering the possibility of a combination of (BPF_NOEXIST, BPF_EXIST) flags with (BPF_F_CPU, BPF_F_ALL_CPUS) flags. This causes the (BPF_NOEXIST, BPF_EXIST) flags to lose their effect in some cases.

For example, using the (BPF_F_ALL_CPUS | BPF_EXIST) flag combination with bpf_map_update_elem() results in an incorrect EINVAL error response, even though the flag combination is valid.

This patch fixes the bug by adding proper flag validations and checks. Before this patch, bpf_percpu_array_update() rejected BPF_F_ALL_CPUS | BPF_EXIST and BPF_F_ALL_CPUS | BPF_NOEXIST with -EINVAL. Also the BPF_F_CPU | BPF_NOEXIST were accepted. After the patch BPF_F_ALL_CPUS | BPF_EXIST is accepted and BPF_F_ALL_CPUS | BPF_NOEXIST and BPF_F_CPU | BPF_NOEXIST result in -EEXIST.

Quoted 5 lines

This patch also starts returning -EINVAL for (BPF_EXIST | BPF_NOEXIST), with or without BPF_F_CPU, which was previously accepted.

Fixes 8eb76cb03f0f ("bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_array maps")

Should carry my Acked-by in v5.

Quoted 1 line

[...]

#3 lore raw

PATCH v6 2/3 bpf: Fix incorrect handling of user flags by percpu hash map updates

In reply to Masoud Aghasi

For BPF_MAP_TYPE_PERCPU_HASH and BPF_MAP_TYPE_LRU_PERCPU_HASH maps, htab_map_check_update_flags() and check_flags() are not considering the possibility of a combination of (BPF_NOEXIST, BPF_EXIST) flags with (BPF_F_CPU, BPF_F_ALL_CPUS) flags. This causes the (BPF_NOEXIST, BPF_EXIST) flags to lose their effect in some cases.

For example, when using (BPF_F_CPU | BPF_EXIST) or (BPF_F_CPU | BPF_NOEXIST) flag combinations with bpf_map_update_elem() on a percpu hash map, the BPF_EXIST flag does not prevent new insertions as expected and BPF_NOEXIST flag does not prevent modification of existing entries as expected.

This patch fixes the bug by adding proper flag validations and checks. Before this patch, htab_map_check_update_flags() rejected (BPF_F_ALL_CPUS | BPF_EXIST) and (BPF_F_ALL_CPUS | BPF_NOEXIST) with -EINVAL. After the patch those combinations are accepted.

This patch also starts returning -EINVAL for (BPF_EXIST | BPF_NOEXIST), with or without BPF_F_CPU, which was previously accepted.

Fixes c6936161fd55 ("bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_hash and lru_percpu_hash maps")
---
 kernel/bpf/hashtab.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
kernel/bpf/hashtab.c +6 -4
index 53c99fe4f176..0202922b7f37 100644
@@ -1196,11 +1196,11 @@ static struct htab_elem *alloc_htab_elem(struct bpf_htab *htab, void *key,
 static int check_flags(struct bpf_htab *htab, struct htab_elem *l_old,
 		       u64 map_flags)
 {
-	if (l_old && (map_flags & ~BPF_F_LOCK) == BPF_NOEXIST)
+	if (l_old && (map_flags & BPF_NOEXIST))
 		/* elem already exists */
 		return -EEXIST;
 
-	if (!l_old && (map_flags & ~BPF_F_LOCK) == BPF_EXIST)
+	if (!l_old && (map_flags & BPF_EXIST))
 		/* elem doesn't exist, cannot update it */
 		return -ENOENT;
 
@@ -1383,9 +1383,11 @@ static long htab_lru_map_update_elem(struct bpf_map *map, void *key, void *value
 
 static int htab_map_check_update_flags(bool onallcpus, u64 map_flags)
 {
+	if (unlikely((map_flags & BPF_EXIST_FLAGS) == BPF_EXIST_FLAGS))
+		return -EINVAL;
 	if (unlikely(!onallcpus && map_flags > BPF_EXIST))
 		return -EINVAL;
-	if (unlikely(onallcpus && ((map_flags & BPF_F_LOCK) || (u32)map_flags > BPF_F_ALL_CPUS)))
+	if (unlikely(onallcpus && ((u32)map_flags & ~(BPF_EXIST_FLAGS | BPF_CPU_FLAGS))))
 		return -EINVAL;
 	return 0;
 }
@@ -1483,7 +1485,7 @@ static long __htab_lru_percpu_map_update_elem(struct bpf_map *map, void *key,
 	 * to remove older elem from htab and this removal
 	 * operation will need a bucket lock.
 	 */
-	if (map_flags != BPF_EXIST) {
+	if (!(map_flags & BPF_EXIST)) {
 		l_new = prealloc_lru_pop(htab, key, hash);
 		if (!l_new)
 			return -ENOMEM;
Signature · 1 line
2.47.3
Acked-by on v6 2/3
#4 lore raw

Re: [PATCH bpf v6 2/3] bpf: Fix incorrect handling of user flags by percpu hash map updates

Quoted 17 lines · Masoud Aghasi

For BPF_MAP_TYPE_PERCPU_HASH and BPF_MAP_TYPE_LRU_PERCPU_HASH maps, htab_map_check_update_flags() and check_flags() are not considering the possibility of a combination of (BPF_NOEXIST, BPF_EXIST) flags with (BPF_F_CPU, BPF_F_ALL_CPUS) flags. This causes the (BPF_NOEXIST, BPF_EXIST) flags to lose their effect in some cases.

For example, when using (BPF_F_CPU | BPF_EXIST) or (BPF_F_CPU | BPF_NOEXIST) flag combinations with bpf_map_update_elem() on a percpu hash map, the BPF_EXIST flag does not prevent new insertions as expected and BPF_NOEXIST flag does not prevent modification of existing entries as expected.

This patch fixes the bug by adding proper flag validations and checks. Before this patch, htab_map_check_update_flags() rejected (BPF_F_ALL_CPUS | BPF_EXIST) and (BPF_F_ALL_CPUS | BPF_NOEXIST) with -EINVAL. After the patch those combinations are accepted.

Quoted 5 lines

This patch also starts returning -EINVAL for (BPF_EXIST | BPF_NOEXIST), with or without BPF_F_CPU, which was previously accepted.

Fixes c6936161fd55 ("bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_hash and lru_percpu_hash maps")

Should carry my Acked-by in v5.

Quoted 1 line

[...]

#5 lore raw

PATCH v6 3/3 selftests/bpf: add tests for percpu map flags combination

In reply to Masoud Aghasi

All possible combinations of (BPF_EXIST, BPF_NOEXIST) and (BPF_F_ALL_CPUS, BPF_F_CPU) flags are covered for all percpu map types.

As BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE does not support BPF_NOEXIST and also in order to reduce the amount of code duplicates, I added its new test scenarios to the existing cpu_flag_percpu_cgroup_storage test.

But for other percpu map types, I added new tests to be able to exercise all flag combinations and the edge cases such as percpu LRU unnecessary deletion issue.

---
 .../selftests/bpf/prog_tests/percpu_alloc.c   | 254 ++++++++++++++++++
 1 file changed, 254 insertions(+)
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c +254 -0
index 7b4a1e24363b..67124dd4d8a7 100644
@@ -449,6 +449,95 @@ static void test_lru_percpu_hash_cpu_flag_create(void)
 	test_percpu_map_cpu_flag_create(BPF_MAP_TYPE_LRU_PERCPU_HASH, 0);
 }
 
+static bool verify_map_value_all_cpus(int map_fd, void *key, void *values, size_t elem_sz,
+				      u32 expected_value, int nr_cpus)
+{
+	int i, err;
+
+	err = bpf_map_lookup_elem(map_fd, key, values);
+	if (!ASSERT_OK(err, "bpf_map_lookup_elem all_cpus"))
+		return false;
+
+	for (i = 0 ; i < nr_cpus ; i++) {
+		if (!ASSERT_EQ(*(u32 *)(values + elem_sz * i), expected_value,
+			       "bpf_map_lookup_elem value all_cpus"))
+			return false;
+	}
+
+	return true;
+}
+
+static bool verify_map_value_first_cpu(int map_fd, void *key, void *values, size_t elem_sz,
+				       u32 expected_value, int nr_cpus)
+{
+	int i, err;
+
+	err = bpf_map_lookup_elem(map_fd, key, values);
+	if (!ASSERT_OK(err, "bpf_map_lookup_elem cpu"))
+		return false;
+
+	if (!ASSERT_EQ(*(u32 *)values, expected_value, "bpf_map_lookup_elem value cpu"))
+		return false;
+
+	for (i = 1 ; i < nr_cpus ; i++) {
+		if (!ASSERT_NEQ(*(u32 *)(values + elem_sz * i), expected_value,
+				"bpf_map_lookup_elem value cpu"))
+			return false;
+	}
+
+	return true;
+}
+
+static void test_percpu_cgroup_storage_flags_combination(struct bpf_map *map, int nr_cpus,
+							 struct bpf_cgroup_storage_key *key)
+{
+	size_t value_sz = sizeof(u32), key_sz, elem_sz;
+	u32 *values, test_value = 0xAABBCCDD;
+	int err, map_fd;
+	u64 flags;
+
+	key_sz = sizeof(*key);
+	elem_sz = roundup(value_sz, 8);
+	map_fd = bpf_map__fd(map);
+	values = calloc(nr_cpus, elem_sz);
+	if (!ASSERT_OK_PTR(values, "calloc values"))
+		return;
+
+	flags = BPF_NOEXIST | BPF_EXIST;
+	err = bpf_map__update_elem(map, key, key_sz, values, elem_sz * nr_cpus, flags);
+	if (!ASSERT_EQ(err, -EINVAL, "bpf_map__update_elem noexist|exist"))
+		goto out;
+
+	flags = BPF_F_ALL_CPUS | BPF_NOEXIST;
+	err = bpf_map__update_elem(map, key, key_sz, values, value_sz, flags);
+	if (!ASSERT_EQ(err, -EINVAL, "bpf_map__update_elem all_cpus|noexist"))
+		goto out;
+
+	flags = BPF_F_CPU | BPF_NOEXIST;
+	err = bpf_map__update_elem(map, key, key_sz, values, value_sz, flags);
+	if (!ASSERT_EQ(err, -EINVAL, "bpf_map__update_elem cpu|noexist"))
+		goto out;
+
+	flags = BPF_F_ALL_CPUS | BPF_EXIST;
+	values[0] = test_value;
+	err = bpf_map__update_elem(map, key, key_sz, values, value_sz, flags);
+	if (!ASSERT_OK(err, "bpf_map__update_elem all_cpus|exist"))
+		goto out;
+
+	if (!verify_map_value_all_cpus(map_fd, key, values, elem_sz, test_value, nr_cpus))
+		goto out;
+
+	flags = BPF_F_CPU | BPF_EXIST;
+	values[0] = --test_value;
+	err = bpf_map__update_elem(map, key, key_sz, values, value_sz, flags);
+	if (!ASSERT_OK(err, "bpf_map__update_elem cpu|exist"))
+		goto out;
+
+	verify_map_value_first_cpu(map_fd, key, values, elem_sz, test_value, nr_cpus);
+out:
+	free(values);
+}
+
 static void test_percpu_cgroup_storage_cpu_flag(void)
 {
 	struct percpu_alloc_array *skel = NULL;
@@ -489,6 +578,7 @@ static void test_percpu_cgroup_storage_cpu_flag(void)
 		goto out;
 
 	test_percpu_map_op_cpu_flag(map, &key, sizeof(key), 1, nr_cpus, false);
+	test_percpu_cgroup_storage_flags_combination(map, nr_cpus, &key);
 out:
 	bpf_prog_detach2(-1, cgroup, BPF_CGROUP_INET_EGRESS);
 	close(cgroup);
@@ -537,6 +627,164 @@ static void test_hash_cpu_flag(void)
 	test_map_op_cpu_flag(BPF_MAP_TYPE_HASH);
 }
 
+static void test_percpu_map_flags_combination(enum bpf_map_type map_type)
+{
+	u32 max_entries = 3, key = 0, first_value = 0xAABBCCDD, test_value = first_value;
+	size_t value_sz = sizeof(u32), elem_sz;
+	int err, map_fd, nr_cpus;
+	bool is_hash_map;
+	u32 *values;
+	u64 flags;
+
+	is_hash_map = (map_type == BPF_MAP_TYPE_PERCPU_HASH ||
+		       map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH);
+	nr_cpus = libbpf_num_possible_cpus();
+	if (!ASSERT_GT(nr_cpus, 0, "libbpf_num_possible_cpus"))
+		return;
+
+	elem_sz = roundup(value_sz, 8);
+	values = calloc(nr_cpus, elem_sz);
+	if (!ASSERT_OK_PTR(values, "calloc values"))
+		return;
+
+	map_fd = bpf_map_create(map_type, "test_flags_combination_map",
+				sizeof(u32), value_sz, max_entries, NULL);
+	if (!ASSERT_GE(map_fd, 0, "bpf_map_create")) {
+		free(values);
+		return;
+	}
+
+	flags = BPF_NOEXIST | BPF_EXIST;
+	err = bpf_map_update_elem(map_fd, &key, values, flags);
+	if (!ASSERT_EQ(err, -EINVAL, "bpf_map_update_elem noexist|exist"))
+		goto out;
+
+	flags = BPF_F_ALL_CPUS | BPF_NOEXIST;
+	values[0] = test_value;
+	key = 0;
+	err = bpf_map_update_elem(map_fd, &key, values, flags);
+	if (is_hash_map) {
+		if (!ASSERT_OK(err, "bpf_map_update_elem all_cpus|noexist"))
+			goto out;
+
+		if (!verify_map_value_all_cpus(map_fd, &key, values, elem_sz, test_value, nr_cpus))
+			goto out;
+
+		err = bpf_map_update_elem(map_fd, &key, values, flags);
+	}
+	if (!ASSERT_EQ(err, -EEXIST, "bpf_map_update_elem all_cpus|noexist"))
+		goto out;
+
+	flags = BPF_F_CPU | BPF_NOEXIST;
+	values[0] = --test_value;
+	key = 1;
+	err = bpf_map_update_elem(map_fd, &key, values, flags);
+	if (is_hash_map) {
+		if (!ASSERT_OK(err, "bpf_map_update_elem cpu|noexist"))
+			goto out;
+
+		if (!verify_map_value_first_cpu(map_fd, &key, values, elem_sz, test_value, nr_cpus))
+			goto out;
+
+		err = bpf_map_update_elem(map_fd, &key, values, flags);
+	}
+	if (!ASSERT_EQ(err, -EEXIST, "bpf_map_update_elem cpu|noexist"))
+		goto out;
+
+	flags = BPF_F_ALL_CPUS | BPF_EXIST;
+	values[0] = --test_value;
+	key = 2;
+	err = bpf_map_update_elem(map_fd, &key, values, flags);
+	if (is_hash_map) {
+		if (!ASSERT_EQ(err, -ENOENT, "bpf_map_update_elem all_cpus|exist"))
+			goto out;
+	} else {
+		if (!ASSERT_OK(err, "bpf_map_update_elem all_cpus|exist"))
+			goto out;
+
+		if (!verify_map_value_all_cpus(map_fd, &key, values, elem_sz, test_value, nr_cpus))
+			goto out;
+	}
+
+	flags = BPF_F_CPU | BPF_EXIST;
+	values[0] = --test_value;
+	err = bpf_map_update_elem(map_fd, &key, values, flags);
+	if (is_hash_map) {
+		if (!ASSERT_EQ(err, -ENOENT, "bpf_map_update_elem cpu|exist"))
+			goto out;
+	} else {
+		if (!ASSERT_OK(err, "bpf_map_update_elem cpu|exist"))
+			goto out;
+
+		if (!verify_map_value_first_cpu(map_fd, &key, values, elem_sz, test_value, nr_cpus))
+			goto out;
+	}
+
+	if (map_type != BPF_MAP_TYPE_LRU_PERCPU_HASH)
+		goto out;
+
+	/* Percpu LRU hash map should not delete old entries unnecessarily */
+	flags = BPF_F_ALL_CPUS | BPF_NOEXIST;
+	key = 2;
+	err = bpf_map_update_elem(map_fd, &key, values, flags);
+	if (!ASSERT_OK(err, "bpf_map_update_elem lru_percpu_hash all_cpus|noexist"))
+		goto out;
+
+	flags = BPF_F_ALL_CPUS | BPF_EXIST;
+	err = bpf_map_update_elem(map_fd, &key, values, flags);
+	if (!ASSERT_OK(err, "bpf_map_update_elem lru_percpu_hash all_cpus|exist"))
+		goto out;
+
+	key = 0;
+	verify_map_value_all_cpus(map_fd, &key, values, elem_sz, first_value, nr_cpus);
+
+out:
+	close(map_fd);
+	free(values);
+}
+
+static bool pin_current_cpu(cpu_set_t *old_mask)
+{
+	cpu_set_t new_mask;
+	int err, cpu;
+
+	err = sched_getaffinity(0, sizeof(*old_mask), old_mask);
+	if (!ASSERT_OK(err, "sched_getaffinity"))
+		return false;
+
+	cpu = sched_getcpu();
+	if (!ASSERT_GE(cpu, 0, "sched_getcpu"))
+		return false;
+
+	CPU_ZERO(&new_mask);
+	CPU_SET(cpu, &new_mask);
+
+	err = sched_setaffinity(0, sizeof(new_mask), &new_mask);
+	return ASSERT_OK(err, "sched_setaffinity");
+}
+
+static void test_percpu_hash_flags_combination(void)
+{
+	test_percpu_map_flags_combination(BPF_MAP_TYPE_PERCPU_HASH);
+}
+
+static void test_lru_percpu_hash_flags_combination(void)
+{
+	cpu_set_t old_mask;
+
+	if (!pin_current_cpu(&old_mask))
+		return;
+
+	test_percpu_map_flags_combination(BPF_MAP_TYPE_LRU_PERCPU_HASH);
+
+	sched_setaffinity(0, sizeof(old_mask), &old_mask);
+}
+
+static void test_percpu_array_flags_combination(void)
+{
+	test_percpu_map_flags_combination(BPF_MAP_TYPE_PERCPU_ARRAY);
+}
+
 void test_percpu_alloc(void)
 {
 	if (test__start_subtest("array"))
@@ -565,4 +813,10 @@ void test_percpu_alloc(void)
 		test_array_cpu_flag();
 	if (test__start_subtest("cpu_flag_hash"))
 		test_hash_cpu_flag();
+	if (test__start_subtest("cpu_flag_combination_percpu_hash"))
+		test_percpu_hash_flags_combination();
+	if (test__start_subtest("cpu_flag_combination_lru_percpu_hash"))
+		test_lru_percpu_hash_flags_combination();
+	if (test__start_subtest("cpu_flag_combination_percpu_array"))
+		test_percpu_array_flags_combination();
 }
Signature · 1 line
2.47.3
Acked-by on v6 3/3
#6 lore raw

Re: [PATCH bpf v6 3/3] selftests/bpf: add tests for percpu map flags combination

Quoted 7 lines · Masoud Aghasi

All possible combinations of (BPF_EXIST, BPF_NOEXIST) and (BPF_F_ALL_CPUS, BPF_F_CPU) flags are covered for all percpu map types.

As BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE does not support BPF_NOEXIST and also in order to reduce the amount of code duplicates, I added its new test scenarios to the existing cpu_flag_percpu_cgroup_storage test.

Quoted 5 lines

But for other percpu map types, I added new tests to be able to exercise all flag combinations and the edge cases such as percpu LRU unnecessary deletion issue.

Two concerns below, others lgtm.

Quoted 4 lines
---
 .../selftests/bpf/prog_tests/percpu_alloc.c   | 254 ++++++++++++++++++
 1 file changed, 254 insertions(+)

[...]

tools/testing/selftests/bpf/prog_tests/percpu_alloc.c static void test_hash_cpu_flag(void) · line 731

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c +10 -0
+
+	if (map_type != BPF_MAP_TYPE_LRU_PERCPU_HASH)
+		goto out;
+
+	/* Percpu LRU hash map should not delete old entries unnecessarily */
+	flags = BPF_F_ALL_CPUS | BPF_NOEXIST;
+	key = 2;
+	err = bpf_map_update_elem(map_fd, &key, values, flags);
+	if (!ASSERT_OK(err, "bpf_map_update_elem lru_percpu_hash all_cpus|noexist"))
+		goto out;

Update values[0] before update_elem(), and verify_map_value_all_cpus()?

Also update values[0] before the below update_elem().

tools/testing/selftests/bpf/prog_tests/percpu_alloc.c static void test_hash_cpu_flag(void) · line 738

Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c +7 -0
+
+	flags = BPF_F_ALL_CPUS | BPF_EXIST;
+	err = bpf_map_update_elem(map_fd, &key, values, flags);
+	if (!ASSERT_OK(err, "bpf_map_update_elem lru_percpu_hash all_cpus|exist"))
+		goto out;
+
+	key = 0;

Why do you use 'k = 0' instead of '2'?

I think we should update and verify using the same key.

Thanks,
Leon

Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c +7 -0
+	verify_map_value_all_cpus(map_fd, &key, values, elem_sz, first_value, nr_cpus);
+
+out:
+	close(map_fd);
+	free(values);
+}
+

[...]

#7 lore raw

Re: [PATCH bpf v6 3/3] selftests/bpf: add tests for percpu map flags combination

Quoted 56 lines · Leon Hwang
Quoted 7 lines · Masoud Aghasi

All possible combinations of (BPF_EXIST, BPF_NOEXIST) and (BPF_F_ALL_CPUS, BPF_F_CPU) flags are covered for all percpu map types.

As BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE does not support BPF_NOEXIST and also in order to reduce the amount of code duplicates, I added its new test scenarios to the existing cpu_flag_percpu_cgroup_storage test.

Quoted 5 lines

But for other percpu map types, I added new tests to be able to exercise all flag combinations and the edge cases such as percpu LRU unnecessary deletion issue.

Two concerns below, others lgtm.

Quoted 4 lines
---
 .../selftests/bpf/prog_tests/percpu_alloc.c   | 254 ++++++++++++++++++
 1 file changed, 254 insertions(+)

[...]

tools/testing/selftests/bpf/prog_tests/percpu_alloc.c static void test_hash_cpu_flag(void) · line 731

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c +10 -0
+
+	if (map_type != BPF_MAP_TYPE_LRU_PERCPU_HASH)
+		goto out;
+
+	/* Percpu LRU hash map should not delete old entries unnecessarily */
+	flags = BPF_F_ALL_CPUS | BPF_NOEXIST;
+	key = 2;
+	err = bpf_map_update_elem(map_fd, &key, values, flags);
+	if (!ASSERT_OK(err, "bpf_map_update_elem lru_percpu_hash all_cpus|noexist"))
+		goto out;

Update values[0] before update_elem(), and verify_map_value_all_cpus()?

Also update values[0] before the below update_elem().

tools/testing/selftests/bpf/prog_tests/percpu_alloc.c static void test_hash_cpu_flag(void) · line 738

Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c +7 -0
+
+	flags = BPF_F_ALL_CPUS | BPF_EXIST;
+	err = bpf_map_update_elem(map_fd, &key, values, flags);
+	if (!ASSERT_OK(err, "bpf_map_update_elem lru_percpu_hash all_cpus|exist"))
+		goto out;
+
+	key = 0;

Why do you use 'k = 0' instead of '2'?

I think we should update and verify using the same key.

Thanks,
Leon

Quoted 9 lines
Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c +7 -0
+	verify_map_value_all_cpus(map_fd, &key, values, elem_sz, first_value, nr_cpus);
+
+out:
+	close(map_fd);
+	free(values);
+}
+

[...]

Thanks for the review.

Regarding your remaining concerns, I designed these new tests to cover all changes made in arraymap.c and hashtab.c. For example, if we revert my change to __htab_lru_percpu_map_update_elem() in hashtab.c, the last verification in this test for key '0' will fail.

It shows that the change is necessary to prevent unnecessary deletion of old keys by percpu LRU maps. In the case of this test, inserting and updating the last element (key '2') of a map of 3 entries should not cause unnecessary deletion of the first key (key '0').

And regarding not verifying the last 2 updates, I left those out because we already tested the (BPF_F_ALL_CPUS | BPF_NOEXIST) and (BPF_F_ALL_CPUS | BPF_EXIST) flag combinations for LRU maps (and other types) in previous code, so there was no need to test them again. the final part of this test is only meant to cover a specific LRU map scenario that I discussed with Alexei in V2.

#8 lore raw
In reply to Masoud Aghasi
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:

Hello:

This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:

Quoted 6 lines · On Thu, 8 Oct 2026 13:07:31 +0100 you wrote:

For BPF_MAP_TYPE_PERCPU_ARRAY map, bpf_percpu_array_update() is not considering the possibility of a combination of (BPF_NOEXIST, BPF_EXIST) flags with (BPF_F_CPU, BPF_F_ALL_CPUS) flags. This causes the (BPF_NOEXIST, BPF_EXIST) flags to lose their effect in some cases.

Quoted 7 lines

For BPF_MAP_TYPE_PERCPU_HASH and BPF_MAP_TYPE_LRU_PERCPU_HASH maps, htab_map_check_update_flags() and check_flags() are not considering the possibility of a combination of (BPF_NOEXIST, BPF_EXIST) flags with (BPF_F_CPU, BPF_F_ALL_CPUS) flags. This causes the (BPF_NOEXIST, BPF_EXIST) flags to lose their effect in some cases.

[...]

Here is the summary with links:
- [bpf,v6,1/3] bpf: Fix incorrect handling of user flags in bpf_percpu_array_update

    https://git.kernel.org/bpf/bpf-next/c/a898ff0f3e99
  - [bpf,v6,2/3] bpf: Fix incorrect handling of user flags by percpu hash map updates
    https://git.kernel.org/bpf/bpf-next/c/4a9b25fe8d13
  - [bpf,v6,3/3] selftests/bpf: add tests for percpu map flags combination
    https://git.kernel.org/bpf/bpf-next/c/22392799e251

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html