bpf: Fix incorrect handling of user flags by percpu map updates
Subsystems (3)
v2 cover bpf: Fix incorrect handling of user flags by percpu map updates
For BPF_MAP_TYPE_PERCPU_HASH map, htab_map_check_update_flags() and check_flags() are not considering the possibility of a combination of (BPF_NOEXIST, BPF_EXIST) flags with (BPF_F_CPU, BPF_F_ALL_CPUS) flags. This causes the (BPF_NOEXIST, BPF_EXIST) flags to lose their effect.
For example, when using (BPF_F_CPU | BPF_EXIST) flag combination with bpf_map_update_elem() on a BPF_MAP_TYPE_PERCPU_HASH map, the BPF_EXIST flag does not prevent new insertions as expected.
Also for BPF_MAP_TYPE_PERCPU_ARRAY map, using the (BPF_F_ALL_CPUS | BPF_EXIST) flag combination with bpf_map_update_elem() results in an incorrect EINVAL error response, even though the flag combination is valid.
This series fixes the bug by adding proper flag validations and checks and adds regression tests.
Changes since v1:
- Fix a BPF_EXIST flag check in __htab_lru_percpu_map_update_elem()
- Add additional test for BPF_MAP_TYPE_LRU_PERCPU_HASH map
- Add check for BPF_EXIST, BPF_NOEXIST combination in percpu array map
Masoud Aghasi (2):
bpf: Fix incorrect handling of user flags by percpu map updates selftests/bpf: add test for percpu map flags combination
kernel/bpf/arraymap.c | 5 +-- kernel/bpf/hashtab.c | 11 ++++--- .../selftests/bpf/prog_tests/percpu_alloc.c | 32 +++++++++++++++++++ 3 files changed, 42 insertions(+), 6 deletions(-)
Signature · 1 line
2.47.3
PATCH v2 1/2 bpf: Fix incorrect handling of user flags by percpu map updates
For BPF_MAP_TYPE_PERCPU_HASH map, htab_map_check_update_flags() and check_flags() are not considering the possibility of a combination of (BPF_NOEXIST, BPF_EXIST) flags with (BPF_F_CPU, BPF_F_ALL_CPUS) flags. This causes the (BPF_NOEXIST, BPF_EXIST) flags to lose their effect.
For example, when using (BPF_F_CPU | BPF_EXIST) flag combination with bpf_map_update_elem() on a BPF_MAP_TYPE_PERCPU_HASH map, the BPF_EXIST flag does not prevent new insertions as expected.
Also for BPF_MAP_TYPE_PERCPU_ARRAY map, using the (BPF_F_ALL_CPUS | BPF_EXIST) flag combination with bpf_map_update_elem() results in an incorrect EINVAL error response, even though the flag combination is valid.
This patch fixes the bug by adding proper flag validations and checks.
--- kernel/bpf/arraymap.c | 5 +++-- kernel/bpf/hashtab.c | 11 +++++++---- 2 files changed, 10 insertions(+), 6 deletions(-)
kernel/bpf/arraymap.c +3 -2
@@ -438,7 +438,8 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value,u32 size;int cpu, off =0 ; -if (unlikely ((map_flags &BPF_F_LOCK ) || (u32 )map_flags >BPF_F_ALL_CPUS )) +if (unlikely ((map_flags &BPF_EXIST ) && (map_flags &BPF_NOEXIST )) || +unlikely ((u32 )map_flags & ~(BPF_EXIST |BPF_NOEXIST |BPF_F_CPU |BPF_F_ALL_CPUS )))/* unknown flags */ return -EINVAL ;
@@ -446,7 +447,7 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value,/* all elements were pre-allocated, cannot insert a new one */ return -E2BIG ; -if (unlikely (map_flags ==BPF_NOEXIST )) +if (unlikely (map_flags &BPF_NOEXIST ))/* all elements already exist */ return -EEXIST ;
kernel/bpf/hashtab.c +7 -4
@@ -1196,11 +1196,11 @@ static struct htab_elem *alloc_htab_elem(struct bpf_htab *htab, void *key,static int check_flags (struct bpf_htab *htab,struct htab_elem *l_old,u64 map_flags) { -if (l_old && (map_flags & ~BPF_F_LOCK ) ==BPF_NOEXIST ) +if (l_old && (map_flags &BPF_NOEXIST ))/* elem already exists */ return -EEXIST ; -if (!l_old && (map_flags & ~BPF_F_LOCK ) ==BPF_EXIST ) +if (!l_old && (map_flags &BPF_EXIST ))/* elem doesn't exist, cannot update it */ return -ENOENT ;
@@ -1383,9 +1383,12 @@ static long htab_lru_map_update_elem(struct bpf_map *map, void *key, void *valuestatic int htab_map_check_update_flags (bool onallcpus,u64 map_flags) { +if (unlikely ((map_flags &BPF_EXIST ) && (map_flags &BPF_NOEXIST ))) +return -EINVAL ;if (unlikely (!onallcpus && map_flags >BPF_EXIST ))return -EINVAL ; -if (unlikely (onallcpus && ((map_flags &BPF_F_LOCK ) || (u32 )map_flags >BPF_F_ALL_CPUS ))) +if (unlikely (onallcpus && + ((u32 )map_flags & ~(BPF_EXIST |BPF_NOEXIST |BPF_F_CPU |BPF_F_ALL_CPUS ))))return -EINVAL ;return 0 ; }
@@ -1483,7 +1486,7 @@ static long __htab_lru_percpu_map_update_elem(struct bpf_map *map, void *key,* to remove older elem from htab and this removal * operation will need a bucket lock. */ -if (map_flags !=BPF_EXIST ) { +if (!(map_flags &BPF_EXIST )) { l_new =prealloc_lru_pop (htab, key, hash);if (!l_new)return -ENOMEM ;
Signature · 1 line
2.47.3
Hi Masoud,
Thanks for the fix.
Pls add target tree in subject, like
[PATCH bpf v3 ...] bpf: ...
Quoted 14 lines · Masoud Aghasi
For
BPF_MAP_TYPE_PERCPU_HASHmap,htab_map_check_update_flags()andcheck_flags()are not considering the possibility of a combination of (BPF_NOEXIST,BPF_EXIST) flags with (BPF_F_CPU,BPF_F_ALL_CPUS) flags. This causes the (BPF_NOEXIST,BPF_EXIST) flags to lose their effect.For example, when using (
BPF_F_CPU|BPF_EXIST) flag combination withbpf_map_update_elem()on aBPF_MAP_TYPE_PERCPU_HASHmap, theBPF_EXISTflag does not prevent new insertions as expected.Also for
BPF_MAP_TYPE_PERCPU_ARRAYmap, using the (BPF_F_ALL_CPUS|BPF_EXIST) flag combination withbpf_map_update_elem()results in an incorrect EINVAL error response, even though the flag combination is valid.
Quoted 4 lines
This patch fixes the bug by adding proper flag validations and checks.
Fixes c6936161fd55 ("bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_hash and lru_percpu_hash maps")Fixes 8eb76cb03f0f ("bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_array maps")
I'd like to split this patch into two patches: one for lru_/percpu_hash maps, another one for percpu_array maps.
Is there similar issue to percpu_cgroup_storage maps?
Thanks,
Leon
Quoted 1 line
[...]
Quoted 9 lines · Leon Hwang
Hi Masoud,
Thanks for the fix.
Pls add target tree in subject, like
[PATCH bpf v3 ...] bpf: ...
Quoted 9 lines
I'd like to split this patch into two patches: one for lru_/percpu_hash maps, another one for
percpu_arraymaps.Is there similar issue to
percpu_cgroup_storagemaps?Thanks,
LeonQuoted 1 line
[...]
Thanks for your time and review.
I will apply your notes to v3.
Regarding the percpu_cgroup_storage update, it checks flags differently and I did not notice any issues with the way it handles flags.
PATCH v2 2/2 selftests/bpf: add test for percpu map flags combination
In reply to Masoud AghasiAdd tests to exercise usage of a combination of flags for percpu hash map updates.
Before the fix, these tests fail to respect user flags combination of (BPF_F_CPU | BPF_EXIST). With the fix applied these tests pass.
--- .../selftests/bpf/prog_tests/percpu_alloc.c | 32 +++++++++++++++++++ 1 file changed, 32 insertions(+)
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c +32 -0
@@ -537,6 +537,34 @@ static void test_hash_cpu_flag(void)test_map_op_cpu_flag (BPF_MAP_TYPE_HASH ); } +static void test_percpu_map_flags_combination (enum bpf_map_type map_type) +{ +u32 max_entries =1 ; +u64 flags, val =0 ; +int err, map_fd, key =0 ; + + map_fd =bpf_map_create (map_type,"test_flags_combo" , +sizeof (int ),sizeof (u64), max_entries,NULL ); +if (!ASSERT_GE (map_fd,0 ,"bpf_map_create" )) +return ; + + flags =BPF_F_CPU |BPF_EXIST ; + err =bpf_map_update_elem (map_fd, &key, &val, flags); +ASSERT_EQ (err, -ENOENT ,"bpf_map_update_elem flags_combo" ); + +close (map_fd); +} + +static void test_percpu_hash_flags_combination (void ) +{ +test_percpu_map_flags_combination (BPF_MAP_TYPE_PERCPU_HASH ); +} + +static void test_percpu_lru_hash_flags_combination (void ) +{ +test_percpu_map_flags_combination (BPF_MAP_TYPE_LRU_PERCPU_HASH ); +} +void test_percpu_alloc (void ) {if (test__start_subtest ("array" ))
@@ -565,4 +593,8 @@ void test_percpu_alloc(void)test_array_cpu_flag ();if (test__start_subtest ("cpu_flag_hash" ))test_hash_cpu_flag (); +if (test__start_subtest ("percpu_hash_flags_combination" )) +test_percpu_hash_flags_combination (); +if (test__start_subtest ("percpu_lru_hash_flags_combination" )) +test_percpu_lru_hash_flags_combination (); }
Signature · 1 line
2.47.3
[email protected]
AI reviewRe: [PATCH v2 2/2] selftests/bpf: add test for percpu map flags combination
TL;DR: Nit only, not a bug: the new percpu_alloc selftest names (lru_percpu word order, missing cpu_flag_ prefix) differ from existing subtests, so '-t percpu_alloc/cpu_flag*' won't select them.
Quoted 1 line
selftests/bpf: add test for percpu map flags combination
TL;DR: Nit only, not a bug: the new percpu_alloc selftest names (lru_percpu word order, missing cpu_flag_ prefix) differ from existing subtests, so '-t percpu_alloc/cpu_flag*' won't select them.
Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c +0
-0
[ ... ]
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c +5
-0+static void test_percpu_lru_hash_flags_combination (void ) +{ +test_percpu_map_flags_combination (BPF_MAP_TYPE_LRU_PERCPU_HASH ); +} +
[ ... ]
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c void test_percpu_alloc(void) · line 600
Quoted 9 lines
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c +4
-0@@ -565,4 +593,8 @@ void test_percpu_alloc(void)test_array_cpu_flag ();if (test__start_subtest ("cpu_flag_hash" ))test_hash_cpu_flag (); +if (test__start_subtest ("percpu_hash_flags_combination" )) +test_percpu_hash_flags_combination (); +if (test__start_subtest ("percpu_lru_hash_flags_combination" )) +test_percpu_lru_hash_flags_combination (); }
This isn't a bug, but would test_lru_percpu_hash_flags_combination() and a subtest name like "cpu_flag_exist_lru_percpu_hash" match the naming the other percpu_alloc subtests use?
The map type is BPF_MAP_TYPE_LRU_PERCPU_HASH, and the existing helpers and subtests in this file use the 'lru_percpu_hash' word order:test_lru_percpu_hash_cpu_flag(), test_lru_percpu_hash_cpu_flag_create(), "cpu_flag_lru_percpu_hash" and "cpu_flag_create_lru_percpu_hash".
The new subtest names also don't share the 'cpu_flag_' prefix, so filtering with '-t percpu_alloc/cpu_flag*' will not select them even though they exercise BPF_F_CPU.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/36727325577
Re: [PATCH v2 2/2] selftests/bpf: add test for percpu map flags combination
In reply to Masoud AghasiQuoted 5 lines · Masoud Aghasi
Add tests to exercise usage of a combination of flags for percpu hash map updates.
Before the fix, these tests fail to respect user flags combination of (
BPF_F_CPU|BPF_EXIST). With the fix applied these tests pass.
I think combinations of (BPF_F_CPU, BPF_F_ALL_CPUS) and (BPF_EXIST, BPF_NOEXIST) should be covered for all per-CPU map types where applicable.
Thanks,
Leon
Quoted 1 line
[...]
Re: [PATCH v2 2/2] selftests/bpf: add test for percpu map flags combination
Quoted 4 lines · Leon Hwang
Quoted 3 lines · Masoud Aghasi
Add tests to exercise usage of a combination of flags for percpu hash map updates.
Quoted 11 lines
Quoted 2 lines
Before the fix, these tests fail to respect user flags combination of (
BPF_F_CPU|BPF_EXIST). With the fix applied these tests pass.I think combinations of (
BPF_F_CPU,BPF_F_ALL_CPUS) and (BPF_EXIST,BPF_NOEXIST) should be covered for all per-CPU map types where applicable.Thanks,
LeonQuoted 1 line
[...]
Thanks.
I will add full test coverage for these to v3.
Re: [PATCH v2 2/2] selftests/bpf: add test for percpu map flags combination
In reply to Masoud Aghasitools/testing/selftests/bpf/prog_tests/percpu_alloc.c static void test_hash_cpu_flag(void) · line 553
Quoted 3 lines · Masoud Aghasi
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c +3
-0+ flags =BPF_F_CPU |BPF_EXIST ; + err =bpf_map_update_elem (map_fd, &key, &val, flags); +ASSERT_EQ (err, -ENOENT ,"bpf_map_update_elem flags_combo" );
The map is empty, so lru_percpu_hash test passes without the __htab_lru_percpu_map_update_elem() hunk that v2 added to patch 1. The check_flags() change alone is enough to get -ENOENT. Insert the key first and then update it with BPF_F_CPU | BPF_EXIST. With max_entries == 1 prealloc_lru_pop() evicts that key and the update fails with -ENOENT without that hunk.
percpu_array changes are not tested at all.
Not even BPF_F_ALL_CPUS | BPF_EXIST from the commit log of patch 1.
pw-bot: cr
Re: [PATCH v2 2/2] selftests/bpf: add test for percpu map flags combination
Quoted 12 lines · Alexei Starovoitov
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c static void test_hash_cpu_flag(void) · line 553
Quoted 3 lines · Masoud Aghasi
tools/testing/selftests/bpf/prog_tests/percpu_alloc.c +3
-0+ flags =BPF_F_CPU |BPF_EXIST ; + err =bpf_map_update_elem (map_fd, &key, &val, flags); +ASSERT_EQ (err, -ENOENT ,"bpf_map_update_elem flags_combo" );The map is empty, so
lru_percpu_hashtest passes without the__htab_lru_percpu_map_update_elem()hunk that v2 added to patch 1. Thecheck_flags()change alone is enough to get-ENOENT. Insert the key first and then update it withBPF_F_CPU|BPF_EXIST. Withmax_entries== 1prealloc_lru_pop()evicts that key and the update fails with-ENOENTwithout that hunk.
Quoted 4 lines
percpu_arraychanges are not tested at all.
Not evenBPF_F_ALL_CPUS|BPF_EXISTfrom the commit log of patch 1.pw-bot: cr
Thanks for your time and review.
I will add test coverage for all my changes to v3.
Regarding the v2 change in __htab_lru_percpu_map_update_elem(), you are right in case of empty map or max_entries 1.
But consider this scenario: an lru map with max_entries 2, first inserting key1 then inserting key2 and then calling update on key2 with (BPF_F_CPU | BPF_EXIST) flags. without v2 change, this sequence cause the unnecessary deletion of key1 by prealloc_lru_pop(). But with the v2 change, no key get deleted unnecessarily in all cases.
I will include this scenario to testcases as well.