Skip to contentSkip to conversation

bpf: Fix incorrect handling of user flags by percpu map updates

PATCH Superseded Masoud Aghasi 4 participants 10 messages Sep 30 – Oct 2

Subsystems (3)

v2 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v2 cover bpf: Fix incorrect handling of user flags by percpu map updates

For BPF_MAP_TYPE_PERCPU_HASH map, htab_map_check_update_flags() and check_flags() are not considering the possibility of a combination of (BPF_NOEXIST, BPF_EXIST) flags with (BPF_F_CPU, BPF_F_ALL_CPUS) flags. This causes the (BPF_NOEXIST, BPF_EXIST) flags to lose their effect.

For example, when using (BPF_F_CPU | BPF_EXIST) flag combination with bpf_map_update_elem() on a BPF_MAP_TYPE_PERCPU_HASH map, the BPF_EXIST flag does not prevent new insertions as expected.

Also for BPF_MAP_TYPE_PERCPU_ARRAY map, using the (BPF_F_ALL_CPUS | BPF_EXIST) flag combination with bpf_map_update_elem() results in an incorrect EINVAL error response, even though the flag combination is valid.

This series fixes the bug by adding proper flag validations and checks and adds regression tests.

Changes since v1:
- Fix a BPF_EXIST flag check in __htab_lru_percpu_map_update_elem()
- Add additional test for BPF_MAP_TYPE_LRU_PERCPU_HASH map
- Add check for BPF_EXIST, BPF_NOEXIST combination in percpu array map

Masoud Aghasi (2):
bpf: Fix incorrect handling of user flags by percpu map updates selftests/bpf: add test for percpu map flags combination

 kernel/bpf/arraymap.c                         |  5 +--
 kernel/bpf/hashtab.c                          | 11 ++++---
 .../selftests/bpf/prog_tests/percpu_alloc.c   | 32 +++++++++++++++++++
 3 files changed, 42 insertions(+), 6 deletions(-)
Signature · 1 line
2.47.3
#1 lore raw

PATCH v2 1/2 bpf: Fix incorrect handling of user flags by percpu map updates

For BPF_MAP_TYPE_PERCPU_HASH map, htab_map_check_update_flags() and check_flags() are not considering the possibility of a combination of (BPF_NOEXIST, BPF_EXIST) flags with (BPF_F_CPU, BPF_F_ALL_CPUS) flags. This causes the (BPF_NOEXIST, BPF_EXIST) flags to lose their effect.

For example, when using (BPF_F_CPU | BPF_EXIST) flag combination with bpf_map_update_elem() on a BPF_MAP_TYPE_PERCPU_HASH map, the BPF_EXIST flag does not prevent new insertions as expected.

Also for BPF_MAP_TYPE_PERCPU_ARRAY map, using the (BPF_F_ALL_CPUS | BPF_EXIST) flag combination with bpf_map_update_elem() results in an incorrect EINVAL error response, even though the flag combination is valid.

This patch fixes the bug by adding proper flag validations and checks.

Fixes c6936161fd55 ("bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_hash and lru_percpu_hash maps")
Fixes 8eb76cb03f0f ("bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_array maps")
---
 kernel/bpf/arraymap.c |  5 +++--
 kernel/bpf/hashtab.c  | 11 +++++++----
 2 files changed, 10 insertions(+), 6 deletions(-)
kernel/bpf/arraymap.c +3 -2
index 0fe9afd4a591..4edfde6a624c 100644
@@ -438,7 +438,8 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value,
 	u32 size;
 	int cpu, off = 0;
 
-	if (unlikely((map_flags & BPF_F_LOCK) || (u32)map_flags > BPF_F_ALL_CPUS))
+	if (unlikely((map_flags & BPF_EXIST) && (map_flags & BPF_NOEXIST)) ||
+	    unlikely((u32)map_flags & ~(BPF_EXIST | BPF_NOEXIST | BPF_F_CPU | BPF_F_ALL_CPUS)))
 		/* unknown flags */
 		return -EINVAL;
 
@@ -446,7 +447,7 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value,
 		/* all elements were pre-allocated, cannot insert a new one */
 		return -E2BIG;
 
-	if (unlikely(map_flags == BPF_NOEXIST))
+	if (unlikely(map_flags & BPF_NOEXIST))
 		/* all elements already exist */
 		return -EEXIST;
 
kernel/bpf/hashtab.c +7 -4
index 53c99fe4f176..2106b82894b2 100644
@@ -1196,11 +1196,11 @@ static struct htab_elem *alloc_htab_elem(struct bpf_htab *htab, void *key,
 static int check_flags(struct bpf_htab *htab, struct htab_elem *l_old,
 		       u64 map_flags)
 {
-	if (l_old && (map_flags & ~BPF_F_LOCK) == BPF_NOEXIST)
+	if (l_old && (map_flags & BPF_NOEXIST))
 		/* elem already exists */
 		return -EEXIST;
 
-	if (!l_old && (map_flags & ~BPF_F_LOCK) == BPF_EXIST)
+	if (!l_old && (map_flags & BPF_EXIST))
 		/* elem doesn't exist, cannot update it */
 		return -ENOENT;
 
@@ -1383,9 +1383,12 @@ static long htab_lru_map_update_elem(struct bpf_map *map, void *key, void *value
 
 static int htab_map_check_update_flags(bool onallcpus, u64 map_flags)
 {
+	if (unlikely((map_flags & BPF_EXIST) && (map_flags & BPF_NOEXIST)))
+		return -EINVAL;
 	if (unlikely(!onallcpus && map_flags > BPF_EXIST))
 		return -EINVAL;
-	if (unlikely(onallcpus && ((map_flags & BPF_F_LOCK) || (u32)map_flags > BPF_F_ALL_CPUS)))
+	if (unlikely(onallcpus &&
+	    ((u32)map_flags & ~(BPF_EXIST | BPF_NOEXIST | BPF_F_CPU | BPF_F_ALL_CPUS))))
 		return -EINVAL;
 	return 0;
 }
@@ -1483,7 +1486,7 @@ static long __htab_lru_percpu_map_update_elem(struct bpf_map *map, void *key,
 	 * to remove older elem from htab and this removal
 	 * operation will need a bucket lock.
 	 */
-	if (map_flags != BPF_EXIST) {
+	if (!(map_flags & BPF_EXIST)) {
 		l_new = prealloc_lru_pop(htab, key, hash);
 		if (!l_new)
 			return -ENOMEM;
Signature · 1 line
2.47.3
#2 lore raw

Hi Masoud,

Thanks for the fix.

Pls add target tree in subject, like
[PATCH bpf v3 ...] bpf: ...

Quoted 14 lines · Masoud Aghasi

For BPF_MAP_TYPE_PERCPU_HASH map, htab_map_check_update_flags() and check_flags() are not considering the possibility of a combination of (BPF_NOEXIST, BPF_EXIST) flags with (BPF_F_CPU, BPF_F_ALL_CPUS) flags. This causes the (BPF_NOEXIST, BPF_EXIST) flags to lose their effect.

For example, when using (BPF_F_CPU | BPF_EXIST) flag combination with bpf_map_update_elem() on a BPF_MAP_TYPE_PERCPU_HASH map, the BPF_EXIST flag does not prevent new insertions as expected.

Also for BPF_MAP_TYPE_PERCPU_ARRAY map, using the (BPF_F_ALL_CPUS | BPF_EXIST) flag combination with bpf_map_update_elem() results in an incorrect EINVAL error response, even though the flag combination is valid.

Quoted 4 lines

This patch fixes the bug by adding proper flag validations and checks.

Fixes c6936161fd55 ("bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_hash and lru_percpu_hash maps")
Fixes 8eb76cb03f0f ("bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_array maps")

I'd like to split this patch into two patches: one for lru_/percpu_hash maps, another one for percpu_array maps.

Is there similar issue to percpu_cgroup_storage maps?

Thanks,
Leon

Quoted 1 line

[...]

#3 lore raw
Quoted 9 lines · Leon Hwang

Hi Masoud,

Thanks for the fix.

Pls add target tree in subject, like
[PATCH bpf v3 ...] bpf: ...

Quoted 9 lines

I'd like to split this patch into two patches: one for lru_/percpu_hash maps, another one for percpu_array maps.

Is there similar issue to percpu_cgroup_storage maps?

Thanks,
Leon

Quoted 1 line

[...]

Thanks for your time and review.

I will apply your notes to v3.
Regarding the percpu_cgroup_storage update, it checks flags differently and I did not notice any issues with the way it handles flags.

#4 lore raw

PATCH v2 2/2 selftests/bpf: add test for percpu map flags combination

In reply to Masoud Aghasi

Add tests to exercise usage of a combination of flags for percpu hash map updates.

Before the fix, these tests fail to respect user flags combination of (BPF_F_CPU | BPF_EXIST). With the fix applied these tests pass.

---
 .../selftests/bpf/prog_tests/percpu_alloc.c   | 32 +++++++++++++++++++
 1 file changed, 32 insertions(+)
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c +32 -0
index 7b4a1e24363b..e14a476da5c8 100644
@@ -537,6 +537,34 @@ static void test_hash_cpu_flag(void)
 	test_map_op_cpu_flag(BPF_MAP_TYPE_HASH);
 }
 
+static void test_percpu_map_flags_combination(enum bpf_map_type map_type)
+{
+	u32 max_entries = 1;
+	u64 flags, val = 0;
+	int err, map_fd, key = 0;
+
+	map_fd = bpf_map_create(map_type, "test_flags_combo",
+				sizeof(int), sizeof(u64), max_entries, NULL);
+	if (!ASSERT_GE(map_fd, 0, "bpf_map_create"))
+		return;
+
+	flags = BPF_F_CPU | BPF_EXIST;
+	err = bpf_map_update_elem(map_fd, &key, &val, flags);
+	ASSERT_EQ(err, -ENOENT, "bpf_map_update_elem flags_combo");
+
+	close(map_fd);
+}
+
+static void test_percpu_hash_flags_combination(void)
+{
+	test_percpu_map_flags_combination(BPF_MAP_TYPE_PERCPU_HASH);
+}
+
+static void test_percpu_lru_hash_flags_combination(void)
+{
+	test_percpu_map_flags_combination(BPF_MAP_TYPE_LRU_PERCPU_HASH);
+}
+
 void test_percpu_alloc(void)
 {
 	if (test__start_subtest("array"))
@@ -565,4 +593,8 @@ void test_percpu_alloc(void)
 		test_array_cpu_flag();
 	if (test__start_subtest("cpu_flag_hash"))
 		test_hash_cpu_flag();
+	if (test__start_subtest("percpu_hash_flags_combination"))
+		test_percpu_hash_flags_combination();
+	if (test__start_subtest("percpu_lru_hash_flags_combination"))
+		test_percpu_lru_hash_flags_combination();
 }
Signature · 1 line
2.47.3
#5 lore raw

Re: [PATCH v2 2/2] selftests/bpf: add test for percpu map flags combination

TL;DR: Nit only, not a bug: the new percpu_alloc selftest names (lru_percpu word order, missing cpu_flag_ prefix) differ from existing subtests, so '-t percpu_alloc/cpu_flag*' won't select them.
Quoted 1 line

selftests/bpf: add test for percpu map flags combination

TL;DR: Nit only, not a bug: the new percpu_alloc selftest names (lru_percpu word order, missing cpu_flag_ prefix) differ from existing subtests, so '-t percpu_alloc/cpu_flag*' won't select them.

Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c +0 -0

[ ... ]

Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c +5 -0
+static void test_percpu_lru_hash_flags_combination(void)
+{
+	test_percpu_map_flags_combination(BPF_MAP_TYPE_LRU_PERCPU_HASH);
+}
+

[ ... ]

tools/testing/selftests/bpf/prog_tests/percpu_alloc.c void test_percpu_alloc(void) · line 600

Quoted 9 lines
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c +4 -0
@@ -565,4 +593,8 @@ void test_percpu_alloc(void)
 		test_array_cpu_flag();
 	if (test__start_subtest("cpu_flag_hash"))
 		test_hash_cpu_flag();
+	if (test__start_subtest("percpu_hash_flags_combination"))
+		test_percpu_hash_flags_combination();
+	if (test__start_subtest("percpu_lru_hash_flags_combination"))
+		test_percpu_lru_hash_flags_combination();
 }

This isn't a bug, but would test_lru_percpu_hash_flags_combination() and a subtest name like "cpu_flag_exist_lru_percpu_hash" match the naming the other percpu_alloc subtests use?

The map type is BPF_MAP_TYPE_LRU_PERCPU_HASH, and the existing helpers and subtests in this file use the 'lru_percpu_hash' word order:
test_lru_percpu_hash_cpu_flag(), test_lru_percpu_hash_cpu_flag_create(), "cpu_flag_lru_percpu_hash" and "cpu_flag_create_lru_percpu_hash".

The new subtest names also don't share the 'cpu_flag_' prefix, so filtering with '-t percpu_alloc/cpu_flag*' will not select them even though they exercise BPF_F_CPU.

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/36727325577

#6 lore raw

Re: [PATCH v2 2/2] selftests/bpf: add test for percpu map flags combination

In reply to Masoud Aghasi
Quoted 5 lines · Masoud Aghasi

Add tests to exercise usage of a combination of flags for percpu hash map updates.

Before the fix, these tests fail to respect user flags combination of (BPF_F_CPU | BPF_EXIST). With the fix applied these tests pass.

I think combinations of (BPF_F_CPU, BPF_F_ALL_CPUS) and (BPF_EXIST, BPF_NOEXIST) should be covered for all per-CPU map types where applicable.

Thanks,
Leon

Quoted 1 line

[...]

#7 lore raw

Re: [PATCH v2 2/2] selftests/bpf: add test for percpu map flags combination

Quoted 4 lines · Leon Hwang
Quoted 3 lines · Masoud Aghasi

Add tests to exercise usage of a combination of flags for percpu hash map updates.

Quoted 11 lines
Quoted 2 lines

Before the fix, these tests fail to respect user flags combination of (BPF_F_CPU | BPF_EXIST). With the fix applied these tests pass.

I think combinations of (BPF_F_CPU, BPF_F_ALL_CPUS) and (BPF_EXIST, BPF_NOEXIST) should be covered for all per-CPU map types where applicable.

Thanks,
Leon

Quoted 1 line

[...]

Thanks.
I will add full test coverage for these to v3.

#8 lore raw

Re: [PATCH v2 2/2] selftests/bpf: add test for percpu map flags combination

In reply to Masoud Aghasi

tools/testing/selftests/bpf/prog_tests/percpu_alloc.c static void test_hash_cpu_flag(void) · line 553

Quoted 3 lines · Masoud Aghasi
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c +3 -0
+	flags = BPF_F_CPU | BPF_EXIST;
+	err = bpf_map_update_elem(map_fd, &key, &val, flags);
+	ASSERT_EQ(err, -ENOENT, "bpf_map_update_elem flags_combo");

The map is empty, so lru_percpu_hash test passes without the __htab_lru_percpu_map_update_elem() hunk that v2 added to patch 1. The check_flags() change alone is enough to get -ENOENT. Insert the key first and then update it with BPF_F_CPU | BPF_EXIST. With max_entries == 1 prealloc_lru_pop() evicts that key and the update fails with -ENOENT without that hunk.

percpu_array changes are not tested at all.
Not even BPF_F_ALL_CPUS | BPF_EXIST from the commit log of patch 1.

pw-bot: cr

#9 lore raw

Re: [PATCH v2 2/2] selftests/bpf: add test for percpu map flags combination

Quoted 12 lines · Alexei Starovoitov

tools/testing/selftests/bpf/prog_tests/percpu_alloc.c static void test_hash_cpu_flag(void) · line 553

Quoted 3 lines · Masoud Aghasi
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c +3 -0
+	flags = BPF_F_CPU | BPF_EXIST;
+	err = bpf_map_update_elem(map_fd, &key, &val, flags);
+	ASSERT_EQ(err, -ENOENT, "bpf_map_update_elem flags_combo");

The map is empty, so lru_percpu_hash test passes without the __htab_lru_percpu_map_update_elem() hunk that v2 added to patch 1. The check_flags() change alone is enough to get -ENOENT. Insert the key first and then update it with BPF_F_CPU | BPF_EXIST. With max_entries == 1 prealloc_lru_pop() evicts that key and the update fails with -ENOENT without that hunk.

Quoted 4 lines

percpu_array changes are not tested at all.
Not even BPF_F_ALL_CPUS | BPF_EXIST from the commit log of patch 1.

pw-bot: cr

Thanks for your time and review.

I will add test coverage for all my changes to v3.

Regarding the v2 change in __htab_lru_percpu_map_update_elem(), you are right in case of empty map or max_entries 1.
But consider this scenario: an lru map with max_entries 2, first inserting key1 then inserting key2 and then calling update on key2 with (BPF_F_CPU | BPF_EXIST) flags. without v2 change, this sequence cause the unnecessary deletion of key1 by prealloc_lru_pop(). But with the v2 change, no key get deleted unnecessarily in all cases.

I will include this scenario to testcases as well.