Close race in freeing special fields and map value
Subsystems (4)
Kumar Kartikeya Dwivedi
maintainerv2 cover Close race in freeing special fields and map value
There exists a race across various map types where the freeing of special fields (tw, timer, wq, kptr, etc.) can be done eagerly when a logical delete operation is done on a map value, such that the program which continues to have access to such a map value can recreate the fields and cause them to leak.
The set contains fixes for this case. It is a continuation of Mykyta's previous attempt in [0], but applies to all fields. A test is included which reproduces the bug reliably in absence of the fixes.
Local Storage Benchmarks
------------------------
Evaluation Setup: Benchmarked on a dual-socket Intel Xeon Gold 6348 (Ice Lake) @ 2.60GHz (56 cores / 112 threads), with the CPU governor set to performance. Bench was pinned to a single NUMA node throughout the test.
Benchmark comes from [1] using the following command:
./bench -p 1 local-storage-create --storage-type <socket,task> --batch-size <16,32,64>
Before the test, 10 runs of all cases ([socket|task] x 3 batch sizes x 7 iterations per batch size) are done to warm up and prime the machine.
Then, 3 runs of all cases are done (with and without the patch, across reboots).
For each comparison, we have 21 samples, i.e. per batch size (e.g. socket 16) of a given local storage, we have 3 runs x 7 iterations.
The statistics (mean, median, stddev) and t-test is done for each scenario (local storage and batch size pair) individually (21 samples for either case). All values are for local storage creations in thousand creations / sec (k/s).
Baseline (without patch) With patch Delta
Case Median Mean Std. Dev. Median Mean Std. Dev. Median %
---------------------------------------------------------------------------------------------------
socket 16 432.026 431.941 1.047 431.347 431.953 1.635 -0.679 -0.16% socket 32 432.641 432.818 1.535 432.488 432.302 1.508 -0.153 -0.04% socket 64 431.504 431.996 1.337 429.145 430.326 2.469 -2.359 -0.55% task 16 38.816 39.382 1.456 39.657 39.337 1.831 +0.841 +2.17% task 32 38.815 39.644 2.690 38.721 39.122 1.636 -0.094 -0.24% task 64 37.562 38.080 1.701 39.554 38.563 1.689 +1.992 +5.30%
The cases for socket are within the range of noise, and improvements in task local storage are due to high variance (CV ~4%-6% across batch sizes). The only statistically significant case worth mentioning is socket with batch size 64 with p-value from t-test < 0.05, but the absolute difference is small (~2k/s).
TL;DR there doesn't appear to be any significant regression or improvement.
[0]: https://lore.kernel.org/bpf/[email protected]
[1]: https://lore.kernel.org/bpf/[email protected]
Changelog:
----------
v1 -> v2
v1: https://lore.kernel.org/bpf/[email protected]
* Add Paul's Reviewed-by.
* Fix use-after-free in accessing bpf_mem_alloc embedded in map. (szybot CI)
* Add benchmark numbers for local storage.
* Add extra test case for per-cpu hashmap coverage with up to 16 refcount leaks.
* Target bpf tree.
Kumar Kartikeya Dwivedi (5):
bpf: Register dtor for freeing special fields
bpf: Lose const-ness of map in map_check_btf()
bpf: Delay freeing fields in local storage
bpf: Retire rcu_trace_implies_rcu_gp() from local storage selftests/bpf: Add tests for special fields races
include/linux/bpf.h | 4 +- include/linux/bpf_local_storage.h | 2 +- include/linux/bpf_mem_alloc.h | 6 + kernel/bpf/arena.c | 2 +- kernel/bpf/arraymap.c | 2 +- kernel/bpf/bloom_filter.c | 2 +- kernel/bpf/bpf_insn_array.c | 2 +- kernel/bpf/bpf_local_storage.c | 67 +++--- kernel/bpf/hashtab.c | 86 +++++++ kernel/bpf/local_storage.c | 2 +- kernel/bpf/lpm_trie.c | 2 +- kernel/bpf/memalloc.c | 58 ++++- kernel/bpf/syscall.c | 2 +- .../selftests/bpf/prog_tests/map_kptr_race.c | 218 ++++++++++++++++++ .../selftests/bpf/progs/map_kptr_race.c | 197 ++++++++++++++++ 15 files changed, 599 insertions(+), 53 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/map_kptr_race.c create mode 100644 tools/testing/selftests/bpf/progs/map_kptr_race.c
base-commit: 6881af27f9ea0f5ca8f606f573ef5cc25ca31fe4
Signature · 1 line
2.47.3
Kumar Kartikeya Dwivedi
maintainerPATCH v2 1/5 bpf: Register dtor for freeing special fields
There is a race window where BPF hash map elements can leak special fields if the program with access to the map value recreates these special fields between the check_and_free_fields done on the map value and its eventual return to the memory allocator.
Several ways were explored prior to this patch, most notably [0] tried to use a poison value to reject attempts to recreate special fields for map values that have been logically deleted but still accessible to BPF programs (either while sitting in the free list or when reused). While this approach works well for task work, timers, wq, etc., it is harder to apply the idea to kptrs, which have a similar race and failure mode.
Instead, we change bpf_mem_alloc to allow registering destructor for allocated elements, such that when they are returned to the allocator, any special fields created while they were accessible to programs in the mean time will be freed. If these values get reused, we do not free the fields again before handing the element back. The special fields thus may remain initialized while the map value sits in a free list.
When bpf_mem_alloc is retired in the future, a similar concept can be introduced to kmalloc_nolock-backed kmem_cache, paired with the existing idea of a constructor.
Note that the destructor registration happens in map_check_btf, after the BTF record is populated and (at that point) avaiable for inspection and duplication. Duplication is necessary since the freeing of embedded bpf_mem_alloc can be decoupled from actual map lifetime due to logic introduced to reduce the cost of rcu_barrier()s in mem alloc free path in 9f2c6e96c65e ("bpf: Optimize rcu_barrier usage between hash map and bpf_mem_alloc.").
As such, once all callbacks are done, we must also free the duplicated record. To remove dependency on the bpf_map itself, also stash the key size of the map to obtain value from htab_elem long after the map is gone.
[0]: https://lore.kernel.org/bpf/[email protected]
--- include/linux/bpf_mem_alloc.h | 6 +++ kernel/bpf/hashtab.c | 87 +++++++++++++++++++++++++++++++++++ kernel/bpf/memalloc.c | 58 ++++++++++++++++++----- 3 files changed, 140 insertions(+), 11 deletions(-)
include/linux/bpf_mem_alloc.h +6 -0
@@ -14,6 +14,8 @@ struct bpf_mem_alloc {struct obj_cgroup *objcg;bool percpu;struct work_struct work; +void (*dtor_ctx_free)(void *ctx); +void *dtor_ctx; };/* 'size != 0' is for bpf_mem_alloc which manages fixed-size objects.
@@ -32,6 +34,10 @@ int bpf_mem_alloc_percpu_init(struct bpf_mem_alloc *ma, struct obj_cgroup *objcg/* The percpu allocation with a specific unit size. */ int bpf_mem_alloc_percpu_unit_init (struct bpf_mem_alloc *ma,int size);void bpf_mem_alloc_destroy (struct bpf_mem_alloc *ma); +void bpf_mem_alloc_set_dtor (struct bpf_mem_alloc *ma, +void (*dtor)(void *obj,void *ctx), +void (*dtor_ctx_free)(void *ctx), +void *ctx);/* Check the allocation size for kmalloc equivalent allocator */ int bpf_mem_alloc_check_size (bool percpu,size_t size);
kernel/bpf/hashtab.c +87 -0
123 lines not on this page · raw message
kernel/bpf/memalloc.c +47 -11
108 lines not on this page · raw message
Signature · 1 line
2.47.3
Kumar Kartikeya Dwivedi
maintainerPATCH v2 2/5 bpf: Lose const-ness of map in map_check_btf()
In reply to Kumar Kartikeya DwivediBPF hash map may now use the map_check_btf() callback to decide whether to set a dtor on its bpf_mem_alloc or not. Unlike C++ where members can opt out of const-ness using mutable, we must lose the const qualifier on the callback such that we can avoid the ugly cast. Make the change and adjust all existing users, and lose the comment in hashtab.c.
--- include/linux/bpf.h | 4 ++-- include/linux/bpf_local_storage.h | 2 +- kernel/bpf/arena.c | 2 +- kernel/bpf/arraymap.c | 2 +- kernel/bpf/bloom_filter.c | 2 +- kernel/bpf/bpf_insn_array.c | 2 +- kernel/bpf/bpf_local_storage.c | 2 +- kernel/bpf/hashtab.c | 9 ++++----- kernel/bpf/local_storage.c | 2 +- kernel/bpf/lpm_trie.c | 2 +- kernel/bpf/syscall.c | 2 +- 11 files changed, 15 insertions(+), 16 deletions(-)
include/linux/bpf.h +2 -2
@@ -124,7 +124,7 @@ struct bpf_map_ops {u32 (*map_fd_sys_lookup_elem)(void *ptr);void (*map_seq_show_elem)(struct bpf_map *map,void *key,struct seq_file *m); -int (*map_check_btf)(const struct bpf_map *map, +int (*map_check_btf)(struct bpf_map *map,const struct btf *btf,const struct btf_type *key_type,const struct btf_type *value_type);
@@ -656,7 +656,7 @@ static inline bool bpf_map_support_seq_show(const struct bpf_map *map) map->ops->map_seq_show_elem; } -int map_check_no_btf (const struct bpf_map *map, +int map_check_no_btf (struct bpf_map *map,const struct btf *btf,const struct btf_type *key_type,const struct btf_type *value_type);
include/linux/bpf_local_storage.h +1 -1
@@ -176,7 +176,7 @@ u32 bpf_local_storage_destroy(struct bpf_local_storage *local_storage);void bpf_local_storage_map_free (struct bpf_map *map,struct bpf_local_storage_cache *cache); -int bpf_local_storage_map_check_btf (const struct bpf_map *map, +int bpf_local_storage_map_check_btf (struct bpf_map *map,const struct btf *btf,const struct btf_type *key_type,const struct btf_type *value_type);
kernel/bpf/arena.c +1 -1
@@ -303,7 +303,7 @@ static long arena_map_update_elem(struct bpf_map *map, void *key,return -EOPNOTSUPP ; } -static int arena_map_check_btf (const struct bpf_map *map,const struct btf *btf, +static int arena_map_check_btf (struct bpf_map *map,const struct btf *btf,const struct btf_type *key_type,const struct btf_type *value_type) {return 0 ;
kernel/bpf/arraymap.c +1 -1
@@ -548,7 +548,7 @@ static void percpu_array_map_seq_show_elem(struct bpf_map *map, void *key,rcu_read_unlock (); } -static int array_map_check_btf (const struct bpf_map *map, +static int array_map_check_btf (struct bpf_map *map,const struct btf *btf,const struct btf_type *key_type,const struct btf_type *value_type)
kernel/bpf/bloom_filter.c +1 -1
@@ -180,7 +180,7 @@ static long bloom_map_update_elem(struct bpf_map *map, void *key,return -EINVAL ; } -static int bloom_map_check_btf (const struct bpf_map *map, +static int bloom_map_check_btf (struct bpf_map *map,const struct btf *btf,const struct btf_type *key_type,const struct btf_type *value_type)
kernel/bpf/bpf_insn_array.c +1 -1
@@ -98,7 +98,7 @@ static long insn_array_delete_elem(struct bpf_map *map, void *key)return -EINVAL ; } -static int insn_array_check_btf (const struct bpf_map *map, +static int insn_array_check_btf (struct bpf_map *map,const struct btf *btf,const struct btf_type *key_type,const struct btf_type *value_type)
kernel/bpf/bpf_local_storage.c +1 -1
@@ -797,7 +797,7 @@ int bpf_local_storage_map_alloc_check(union bpf_attr *attr)return 0 ; } -int bpf_local_storage_map_check_btf (const struct bpf_map *map, +int bpf_local_storage_map_check_btf (struct bpf_map *map,const struct btf *btf,const struct btf_type *key_type,const struct btf_type *value_type)
kernel/bpf/hashtab.c +4 -5
@@ -496,10 +496,10 @@ static void htab_dtor_ctx_free(void *ctx)kfree (ctx); } -static int htab_set_dtor (const struct bpf_htab *htab,void (*dtor)(void *,void *)) +static int htab_set_dtor (struct bpf_htab *htab,void (*dtor)(void *,void *)) {u32 key_size = htab->map.key_size; -const struct bpf_mem_alloc *ma; +struct bpf_mem_alloc *ma;struct htab_btf_record *hrec;int err;
@@ -518,12 +518,11 @@ static int htab_set_dtor(const struct bpf_htab *htab, void (*dtor)(void *, voidreturn err; } ma =htab_is_percpu (htab) ? &htab->pcpu_ma : &htab->ma; -/* Kinda sad, but cast away const-ness since we change ma->dtor. */ -bpf_mem_alloc_set_dtor ((struct bpf_mem_alloc *)ma, dtor, htab_dtor_ctx_free, hrec); +bpf_mem_alloc_set_dtor (ma, dtor, htab_dtor_ctx_free, hrec);return 0 ; } -static int htab_map_check_btf (const struct bpf_map *map,const struct btf *btf, +static int htab_map_check_btf (struct bpf_map *map,const struct btf *btf,const struct btf_type *key_type,const struct btf_type *value_type) { struct bpf_htab *htab = container_of(map, struct bpf_htab, map);
kernel/bpf/local_storage.c +1 -1
@@ -364,7 +364,7 @@ static long cgroup_storage_delete_elem(struct bpf_map *map, void *key)return -EINVAL ; } -static int cgroup_storage_check_btf (const struct bpf_map *map, +static int cgroup_storage_check_btf (struct bpf_map *map,const struct btf *btf,const struct btf_type *key_type,const struct btf_type *value_type)
kernel/bpf/lpm_trie.c +1 -1
@@ -751,7 +751,7 @@ static int trie_get_next_key(struct bpf_map *map, void *_key, void *_next_key)return err; } -static int trie_check_btf (const struct bpf_map *map, +static int trie_check_btf (struct bpf_map *map,const struct btf *btf,const struct btf_type *key_type,const struct btf_type *value_type)
kernel/bpf/syscall.c +1 -1
@@ -1234,7 +1234,7 @@ int bpf_obj_name_cpy(char *dst, const char *src, unsigned int size) }EXPORT_SYMBOL_GPL (bpf_obj_name_cpy); -int map_check_no_btf (const struct bpf_map *map, +int map_check_no_btf (struct bpf_map *map,const struct btf *btf,const struct btf_type *key_type,const struct btf_type *value_type)
Signature · 1 line
2.47.3
Kumar Kartikeya Dwivedi
maintainerPATCH v2 3/5 bpf: Delay freeing fields in local storage
In reply to Kumar Kartikeya DwivediCurrently, when use_kmalloc_nolock is false, the freeing of fields for a local storage selem is done eagerly before waiting for the RCU or RCU tasks trace grace period to elapse. This opens up a window where the program which has access to the selem can recreate the fields after the freeing of fields is done eagerly, causing memory leaks when the element is finally freed and returned to the kernel.
Make a few changes to address this. First, delay the freeing of fields until after the grace periods have expired using a __bpf_selem_free_rcu wrapper which is eventually invoked after transitioning through the necessary number of grace period waits. Replace usage of the kfree_rcu with call_rcu to be able to take a custom callback. Finally, care needs to be taken to extend the rcu barriers for all cases, and not just when use_kmalloc_nolock is true, as RCU and RCU tasks trace callbacks can be in flight for either case and access the smap field, which is used to obtain the BTF record to walk over special fields in the map value.
--- kernel/bpf/bpf_local_storage.c | 32 +++++++++++++++++--------------- 1 file changed, 17 insertions(+), 15 deletions(-)
kernel/bpf/bpf_local_storage.c +17 -15
@@ -164,16 +164,25 @@ static void bpf_local_storage_free(struct bpf_local_storage *local_storage, bpf_local_storage_free_trace_rcu); } -/* rcu tasks trace callback for use_kmalloc_nolock == false */ -static void __bpf_selem_free_trace_rcu (struct rcu_head *rcu) +/* rcu callback for use_kmalloc_nolock == false */ +static void __bpf_selem_free_rcu (struct rcu_head *rcu) {struct bpf_local_storage_elem *selem; selem = container_of(rcu, struct bpf_local_storage_elem, rcu); +/* bpf_selem_unlink_nofail may have already cleared smap and freed fields. */ +if (SDATA (selem)->smap) +bpf_obj_free_fields (SDATA (selem)->smap->map.record,SDATA (selem)->data); +kfree (selem); +} + +/* rcu tasks trace callback for use_kmalloc_nolock == false */ +static void __bpf_selem_free_trace_rcu (struct rcu_head *rcu) +{if (rcu_trace_implies_rcu_gp ()) -kfree (selem); +__bpf_selem_free_rcu (rcu);else -kfree_rcu (selem, rcu); +call_rcu (rcu, __bpf_selem_free_rcu); }/* Handle use_kmalloc_nolock == false */
@@ -181,7 +190,7 @@ static void __bpf_selem_free(struct bpf_local_storage_elem *selem, bool vanilla_rcu) {if (vanilla_rcu) -kfree_rcu (selem, rcu); +call_rcu (&selem->rcu, __bpf_selem_free_rcu);else call_rcu_tasks_trace (&selem->rcu, __bpf_selem_free_trace_rcu); }
@@ -214,18 +223,12 @@ static void bpf_selem_free_trace_rcu(struct rcu_head *rcu)void bpf_selem_free (struct bpf_local_storage_elem *selem,bool reuse_now) { -struct bpf_local_storage_map *smap; - - smap =rcu_dereference_check (SDATA (selem)->smap,bpf_rcu_lock_held ()); -if (!selem->use_kmalloc_nolock) {/* * No uptr will be unpin even when reuse_now == false since uptr * is only supported in task local storage, where * smap->use_kmalloc_nolock == true. */ -if (smap) -bpf_obj_free_fields (smap->map.record,SDATA (selem)->data);__bpf_selem_free (selem, reuse_now);return ; }
@@ -958,10 +961,9 @@ void bpf_local_storage_map_free(struct bpf_map *map,*/ synchronize_rcu (); -if (smap->use_kmalloc_nolock) { -rcu_barrier_tasks_trace (); -rcu_barrier (); - } +/* smap remains in use regardless of kmalloc_nolock, so wait unconditionally. */ +rcu_barrier_tasks_trace (); +rcu_barrier ();kvfree (smap->buckets);bpf_map_area_free (smap); }
Signature · 1 line
2.47.3
[email protected]
AI reviewRe: [PATCH bpf v2 3/5] bpf: Delay freeing fields in local storage
The smap field is declared as __rcu in struct bpf_local_storage_data:
Quoted 4 lines
kernel/bpf/bpf_local_storage.c +0
-0@@ -164,16 +164,25 @@ static void bpf_local_storage_free(struct bpf_local_storage *local_storage,
[ ... ]
kernel/bpf/bpf_local_storage.c
Quoted 11 lines
kernel/bpf/bpf_local_storage.c +11
-0+/* rcu callback for use_kmalloc_nolock == false */ +static void __bpf_selem_free_rcu (struct rcu_head *rcu) +{ +struct bpf_local_storage_elem *selem; + + selem = container_of(rcu, struct bpf_local_storage_elem, rcu); +/* bpf_selem_unlink_nofail may have already cleared smap and freed fields. */ +if (SDATA (selem)->smap) +bpf_obj_free_fields (SDATA (selem)->smap->map.record,SDATA (selem)->data); +kfree (selem); +}
The smap field is declared as __rcu in struct bpf_local_storage_data:
struct bpf_local_storage_map __rcu *smap;
Here it is read directly twice without rcu_dereference_check, once for the NULL check and once for the ->map.record access.
The sibling function bpf_selem_free_rcu() in the same file handles this with a local variable and the proper accessor:
smap = rcu_dereference_check(SDATA(selem)->smap, 1);
if (smap) {
...
bpf_obj_free_fields(smap->map.record, SDATA(selem)->data);
...
}Should __bpf_selem_free_rcu() follow the same pattern for consistency and to avoid sparse warnings?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/22474216888
AI-authorship-score: low
AI-authorship-explanation: Deep domain-specific BPF local storage and RCU chaining knowledge, tightly integrated with existing subsystem patterns, characteristic of a domain expert.
issues-found: 1
issue-severity-score: low
issue-severity-explanation: Inconsistent access of __rcu-annotated pointer without rcu_dereference_check and double read instead of local variable, will trigger sparse warnings but is not a correctness bug.
Amery Hung
Reviewed-byRe: [PATCH bpf v2 3/5] bpf: Delay freeing fields in local storage
Quoted 37 lines · On Thu, Feb 26, 2026 at 10:00 PM <[email protected]> wrote:
Quoted 5 lines
kernel/bpf/bpf_local_storage.c +0
-0@@ -164,16 +164,25 @@ static void bpf_local_storage_free(struct bpf_local_storage *local_storage,[ ... ]
kernel/bpf/bpf_local_storage.c
Quoted 11 lines
kernel/bpf/bpf_local_storage.c +11
-0+/* rcu callback for use_kmalloc_nolock == false */ +static void __bpf_selem_free_rcu (struct rcu_head *rcu) +{ +struct bpf_local_storage_elem *selem; + + selem = container_of(rcu, struct bpf_local_storage_elem, rcu); +/* bpf_selem_unlink_nofail may have already cleared smap and freed fields. */ +if (SDATA (selem)->smap) +bpf_obj_free_fields (SDATA (selem)->smap->map.record,SDATA (selem)->data); +kfree (selem); +}The smap field is declared as
__rcuin structbpf_local_storage_data:struct bpf_local_storage_map __rcu *smap;Here it is read directly twice without
rcu_dereference_check, once for the NULL check and once for the ->map.record access.The sibling function
bpf_selem_free_rcu()in the same file handles this with a local variable and the proper accessor:smap = rcu_dereference_check(SDATA(selem)->smap, 1); if (smap) { ... bpf_obj_free_fields(smap->map.record, SDATA(selem)->data); ... }
Quoted 3 lines
Should
__bpf_selem_free_rcu()follow the same pattern for consistency and to avoid sparse warnings?
AI is right, but the rest looks good to me.
Quoted 12 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/22474216888
AI-authorship-score: low
AI-authorship-explanation: Deep domain-specific BPF local storage and RCU chaining knowledge, tightly integrated with existing subsystem patterns, characteristic of a domain expert.
issues-found: 1
issue-severity-score: low
issue-severity-explanation: Inconsistent access of __rcu-annotated pointer withoutrcu_dereference_checkand double read instead of local variable, will trigger sparse warnings but is not a correctness bug.
Kumar Kartikeya Dwivedi
maintainerPATCH v2 4/5 bpf: Retire rcu_trace_implies_rcu_gp() from local storage
In reply to Kumar Kartikeya DwivediThis assumption will always hold going forward, hence just remove the various checks and assume it is true with a comment for the uninformed reader.
--- kernel/bpf/bpf_local_storage.c | 37 +++++++++++++++++----------------- 1 file changed, 19 insertions(+), 18 deletions(-)
kernel/bpf/bpf_local_storage.c +19 -18
@@ -107,14 +107,12 @@ static void __bpf_local_storage_free_trace_rcu(struct rcu_head *rcu) {struct bpf_local_storage *local_storage; -/* If RCU Tasks Trace grace period implies RCU grace period, do -* kfree(), else do kfree_rcu(). +/* +* RCU Tasks Trace grace period implies RCU grace period, do +* kfree() directly. */ local_storage = container_of(rcu, struct bpf_local_storage, rcu); -if (rcu_trace_implies_rcu_gp ()) -kfree (local_storage); -else -kfree_rcu (local_storage, rcu); +kfree (local_storage); }/* Handle use_kmalloc_nolock == false */
@@ -138,10 +136,11 @@ static void bpf_local_storage_free_rcu(struct rcu_head *rcu)static void bpf_local_storage_free_trace_rcu (struct rcu_head *rcu) { -if (rcu_trace_implies_rcu_gp ()) -bpf_local_storage_free_rcu (rcu); -else -call_rcu (rcu, bpf_local_storage_free_rcu); +/* +* RCU Tasks Trace grace period implies RCU grace period, do +* kfree() directly. +*/ +bpf_local_storage_free_rcu (rcu); } static void bpf_local_storage_free(struct bpf_local_storage *local_storage,
@@ -179,10 +178,11 @@ static void __bpf_selem_free_rcu(struct rcu_head *rcu)/* rcu tasks trace callback for use_kmalloc_nolock == false */ static void __bpf_selem_free_trace_rcu (struct rcu_head *rcu) { -if (rcu_trace_implies_rcu_gp ()) -__bpf_selem_free_rcu (rcu); -else -call_rcu (rcu, __bpf_selem_free_rcu); +/* +* RCU Tasks Trace grace period implies RCU grace period, do +* kfree() directly. +*/ +__bpf_selem_free_rcu (rcu); }/* Handle use_kmalloc_nolock == false */
@@ -214,10 +214,11 @@ static void bpf_selem_free_rcu(struct rcu_head *rcu)static void bpf_selem_free_trace_rcu (struct rcu_head *rcu) { -if (rcu_trace_implies_rcu_gp ()) -bpf_selem_free_rcu (rcu); -else -call_rcu (rcu, bpf_selem_free_rcu); +/* +* RCU Tasks Trace grace period implies RCU grace period, do +* kfree() directly. +*/ +bpf_selem_free_rcu (rcu); } void bpf_selem_free(struct bpf_local_storage_elem *selem,
Signature · 1 line
2.47.3
Re: [PATCH bpf v2 4/5] bpf: Retire rcu_trace_implies_rcu_gp() from local storage
Quoted 10 lines · Kumar Kartikeya Dwivedi
This assumption will always hold going forward, hence just remove the various checks and assume it is true with a comment for the uninformed reader.
Reviewed-by Paul E. McKenney <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- kernel/bpf/bpf_local_storage.c | 37 +++++++++++++++++----------------- 1 file changed, 19 insertions(+), 18 deletions(-)
Quoted 76 lines
kernel/bpf/bpf_local_storage.c +19
-19@@ -107,14 +107,12 @@ static void __bpf_local_storage_free_trace_rcu(struct rcu_head *rcu) {struct bpf_local_storage *local_storage; -/* If RCU Tasks Trace grace period implies RCU grace period, do -* kfree(), else do kfree_rcu(). +/* +* RCU Tasks Trace grace period implies RCU grace period, do +* kfree() directly. */ local_storage = container_of(rcu, struct bpf_local_storage, rcu); -if (rcu_trace_implies_rcu_gp ()) -kfree (local_storage); -else -kfree_rcu (local_storage, rcu); +kfree (local_storage); }/* Handle use_kmalloc_nolock == false */ @@ -138,10 +136,11 @@ static void bpf_local_storage_free_rcu(struct rcu_head *rcu)static void bpf_local_storage_free_trace_rcu (struct rcu_head *rcu) { -if (rcu_trace_implies_rcu_gp ()) -bpf_local_storage_free_rcu (rcu); -else -call_rcu (rcu, bpf_local_storage_free_rcu); +/* +* RCU Tasks Trace grace period implies RCU grace period, do +* kfree() directly. +*/ +bpf_local_storage_free_rcu (rcu); } static void bpf_local_storage_free(struct bpf_local_storage *local_storage,@@ -179,10 +178,11 @@ static void __bpf_selem_free_rcu(struct rcu_head *rcu)/* rcu tasks trace callback for use_kmalloc_nolock == false */ static void __bpf_selem_free_trace_rcu (struct rcu_head *rcu) { -if (rcu_trace_implies_rcu_gp ()) -__bpf_selem_free_rcu (rcu); -else -call_rcu (rcu, __bpf_selem_free_rcu); +/* +* RCU Tasks Trace grace period implies RCU grace period, do +* kfree() directly. +*/ +__bpf_selem_free_rcu (rcu); }/* Handle use_kmalloc_nolock == false */ @@ -214,10 +214,11 @@ static void bpf_selem_free_rcu(struct rcu_head *rcu)static void bpf_selem_free_trace_rcu (struct rcu_head *rcu) { -if (rcu_trace_implies_rcu_gp ()) -bpf_selem_free_rcu (rcu); -else -call_rcu (rcu, bpf_selem_free_rcu); +/* +* RCU Tasks Trace grace period implies RCU grace period, do +* kfree() directly. +*/ +bpf_selem_free_rcu (rcu); } void bpf_selem_free(struct bpf_local_storage_elem *selem, --2.47.3
Kumar Kartikeya Dwivedi
maintainerPATCH v2 5/5 selftests/bpf: Add tests for special fields races
In reply to Kumar Kartikeya DwivediAdd a couple of tests to ensure that the refcount drops to zero when we exercise the race where creation of a special field succeeds the logical bpf_obj_free_fields done when deleting an element. Prior to previous changes, the fields would be freed eagerly and repopulate and end up leaking, causing the reference to not drop down correctly. Running this test on a kernel without fixes will cause a hang in delete_module, since the module reference stays active due to the leaked kptr not dropping it. After the fixes tests succeed as expected.
--- .../selftests/bpf/prog_tests/map_kptr_race.c | 218 ++++++++++++++++++ .../selftests/bpf/progs/map_kptr_race.c | 197 ++++++++++++++++ 2 files changed, 415 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/map_kptr_race.c create mode 100644 tools/testing/selftests/bpf/progs/map_kptr_race.c
tools/testing/selftests/bpf/prog_tests/map_kptr_race.c new file +218 -0
@@ -0,0 +1,218 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <test_progs.h> +#include <network_helpers.h> + +#include "map_kptr_race.skel.h" + +static int get_map_id (int map_fd) +{ +struct bpf_map_info info = {}; +__u32 len =sizeof (info); + +if (!ASSERT_OK (bpf_map_get_info_by_fd (map_fd, &info, &len),"get_map_info" )) +return -1 ; +return info.id; +} + +static int read_refs (struct map_kptr_race *skel) +{ +LIBBPF_OPTS (bpf_test_run_opts, opts); +int ret; + + ret =bpf_prog_test_run_opts (bpf_program__fd (skel->progs.count_ref), &opts); +if (!ASSERT_OK (ret,"count_ref run" )) +return -1 ; +if (!ASSERT_OK (opts.retval,"count_ref retval" )) +return -1 ; +return skel->bss->num_of_refs; +} + +static void test_htab_leak (void ) +{ +LIBBPF_OPTS (bpf_test_run_opts, opts, + .data_in = &pkt_v4, + .data_size_in = sizeof(pkt_v4), + .repeat = 1, + ); +struct map_kptr_race *skel, *watcher; +int ret, map_id; + + skel =map_kptr_race__open_and_load (); +if (!ASSERT_OK_PTR (skel,"open_and_load" )) +return ; + + ret =bpf_prog_test_run_opts (bpf_program__fd (skel->progs.test_htab_leak), &opts); +if (!ASSERT_OK (ret,"test_htab_leak run" )) + goto out_skel; +if (!ASSERT_OK (opts.retval,"test_htab_leak retval" )) + goto out_skel; + + map_id =get_map_id (bpf_map__fd (skel->maps.race_hash_map)); +if (!ASSERT_GE (map_id,0 ,"map_id" )) + goto out_skel; + + watcher =map_kptr_race__open_and_load (); +if (!ASSERT_OK_PTR (watcher,"watcher open_and_load" )) + goto out_skel; + + watcher->bss->target_map_id = map_id; + watcher->links.map_put =bpf_program__attach (watcher->progs.map_put); +if (!ASSERT_OK_PTR (watcher->links.map_put,"attach fentry" )) + goto out_watcher; + watcher->links.htab_map_free =bpf_program__attach (watcher->progs.htab_map_free); +if (!ASSERT_OK_PTR (watcher->links.htab_map_free,"attach fexit" )) + goto out_watcher; + +map_kptr_race__destroy (skel); + skel =NULL ; + +kern_sync_rcu (); + +while (!READ_ONCE (watcher->bss->map_freed)) +sched_yield (); + +ASSERT_EQ (watcher->bss->map_freed,1 ,"map_freed" ); +ASSERT_EQ (read_refs (watcher),2 ,"htab refcount" ); + +out_watcher: +map_kptr_race__destroy (watcher); +out_skel: +map_kptr_race__destroy (skel); +} + +static void test_percpu_htab_leak (void ) +{ +LIBBPF_OPTS (bpf_test_run_opts, opts, + .data_in = &pkt_v4, + .data_size_in = sizeof(pkt_v4), + .repeat = 1, + ); +struct map_kptr_race *skel, *watcher; +int ret, map_id; + + skel =map_kptr_race__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return ; + + skel->rodata->nr_cpus =libbpf_num_possible_cpus (); +if (skel->rodata->nr_cpus >16 ) + skel->rodata->nr_cpus =16 ; + + ret =map_kptr_race__load (skel); +if (!ASSERT_OK (ret,"load" )) + goto out_skel; + + ret =bpf_prog_test_run_opts (bpf_program__fd (skel->progs.test_percpu_htab_leak), &opts); +if (!ASSERT_OK (ret,"test_percpu_htab_leak run" )) + goto out_skel; +if (!ASSERT_OK (opts.retval,"test_percpu_htab_leak retval" )) + goto out_skel; + + map_id =get_map_id (bpf_map__fd (skel->maps.race_percpu_hash_map)); +if (!ASSERT_GE (map_id,0 ,"map_id" )) + goto out_skel; + + watcher =map_kptr_race__open_and_load (); +if (!ASSERT_OK_PTR (watcher,"watcher open_and_load" )) + goto out_skel; + + watcher->bss->target_map_id = map_id; + watcher->links.map_put =bpf_program__attach (watcher->progs.map_put); +if (!ASSERT_OK_PTR (watcher->links.map_put,"attach fentry" )) + goto out_watcher; + watcher->links.htab_map_free =bpf_program__attach (watcher->progs.htab_map_free); +if (!ASSERT_OK_PTR (watcher->links.htab_map_free,"attach fexit" )) + goto out_watcher; + +map_kptr_race__destroy (skel); + skel =NULL ; + +kern_sync_rcu (); + +while (!READ_ONCE (watcher->bss->map_freed)) +sched_yield (); + +ASSERT_EQ (watcher->bss->map_freed,1 ,"map_freed" ); +ASSERT_EQ (read_refs (watcher),2 ,"percpu_htab refcount" ); + +out_watcher: +map_kptr_race__destroy (watcher); +out_skel: +map_kptr_race__destroy (skel); +} + +static void test_sk_ls_leak (void ) +{ +struct map_kptr_race *skel, *watcher; +int listen_fd =-1 , client_fd =-1 , map_id; + + skel =map_kptr_race__open_and_load (); +if (!ASSERT_OK_PTR (skel,"open_and_load" )) +return ; + +if (!ASSERT_OK (map_kptr_race__attach (skel),"attach" )) + goto out_skel; + + listen_fd =start_server (AF_INET6 ,SOCK_STREAM ,"::1" ,0 ,0 ); +if (!ASSERT_GE (listen_fd,0 ,"start_server" )) + goto out_skel; + + client_fd =connect_to_fd (listen_fd,0 ); +if (!ASSERT_GE (client_fd,0 ,"connect_to_fd" )) + goto out_skel; + +if (!ASSERT_EQ (skel->bss->sk_ls_leak_done,1 ,"sk_ls_leak_done" )) + goto out_skel; + +close (client_fd); + client_fd =-1 ; +close (listen_fd); + listen_fd =-1 ; + + map_id =get_map_id (bpf_map__fd (skel->maps.race_sk_ls_map)); +if (!ASSERT_GE (map_id,0 ,"map_id" )) + goto out_skel; + + watcher =map_kptr_race__open_and_load (); +if (!ASSERT_OK_PTR (watcher,"watcher open_and_load" )) + goto out_skel; + + watcher->bss->target_map_id = map_id; + watcher->links.map_put =bpf_program__attach (watcher->progs.map_put); +if (!ASSERT_OK_PTR (watcher->links.map_put,"attach fentry" )) + goto out_watcher; + watcher->links.sk_map_free =bpf_program__attach (watcher->progs.sk_map_free); +if (!ASSERT_OK_PTR (watcher->links.sk_map_free,"attach fexit" )) + goto out_watcher; + +map_kptr_race__destroy (skel); + skel =NULL ; + +kern_sync_rcu (); + +while (!READ_ONCE (watcher->bss->map_freed)) +sched_yield (); + +ASSERT_EQ (watcher->bss->map_freed,1 ,"map_freed" ); +ASSERT_EQ (read_refs (watcher),2 ,"sk_ls refcount" ); + +out_watcher: +map_kptr_race__destroy (watcher); +out_skel: +if (client_fd >=0 ) +close (client_fd); +if (listen_fd >=0 ) +close (listen_fd); +map_kptr_race__destroy (skel); +} + +void serial_test_map_kptr_race (void ) +{ +if (test__start_subtest ("htab_leak" )) +test_htab_leak (); +if (test__start_subtest ("percpu_htab_leak" )) +test_percpu_htab_leak (); +if (test__start_subtest ("sk_ls_leak" )) +test_sk_ls_leak (); +}
tools/testing/selftests/bpf/progs/map_kptr_race.c new file +197 -0
197 lines not on this page · raw message
Signature · 1 line
2.47.3
Re: [PATCH bpf v2 5/5] selftests/bpf: Add tests for special fields races
Quoted 10 lines · Kumar Kartikeya Dwivedi
Add a couple of tests to ensure that the refcount drops to zero when we exercise the race where creation of a special field succeeds the logical
bpf_obj_free_fieldsdone when deleting an element. Prior to previous changes, the fields would be freed eagerly and repopulate and end up leaking, causing the reference to not drop down correctly. Running this test on a kernel without fixes will cause a hang indelete_module, since the module reference stays active due to the leaked kptr not dropping it. After the fixes tests succeed as expected.
Quoted 7 lines
Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- .../selftests/bpf/prog_tests/map_kptr_race.c | 218 ++++++++++++++++++ .../selftests/bpf/progs/map_kptr_race.c | 197 ++++++++++++++++ 2 files changed, 415 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/map_kptr_race.c create mode 100644 tools/testing/selftests/bpf/progs/map_kptr_race.c
Quoted 432 lines
tools/testing/selftests/bpf/prog_tests/map_kptr_race.c new file +218
-0@@ -0,0 +1,218 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <test_progs.h> +#include <network_helpers.h> + +#include "map_kptr_race.skel.h" + +static int get_map_id (int map_fd) +{ +struct bpf_map_info info = {}; +__u32 len =sizeof (info); + +if (!ASSERT_OK (bpf_map_get_info_by_fd (map_fd, &info, &len),"get_map_info" )) +return -1 ; +return info.id; +} + +static int read_refs (struct map_kptr_race *skel) +{ +LIBBPF_OPTS (bpf_test_run_opts, opts); +int ret; + + ret =bpf_prog_test_run_opts (bpf_program__fd (skel->progs.count_ref), &opts); +if (!ASSERT_OK (ret,"count_ref run" )) +return -1 ; +if (!ASSERT_OK (opts.retval,"count_ref retval" )) +return -1 ; +return skel->bss->num_of_refs; +} + +static void test_htab_leak (void ) +{ +LIBBPF_OPTS (bpf_test_run_opts, opts, + .data_in = &pkt_v4, + .data_size_in = sizeof(pkt_v4), + .repeat = 1, + ); +struct map_kptr_race *skel, *watcher; +int ret, map_id; + + skel =map_kptr_race__open_and_load (); +if (!ASSERT_OK_PTR (skel,"open_and_load" )) +return ; + + ret =bpf_prog_test_run_opts (bpf_program__fd (skel->progs.test_htab_leak), &opts); +if (!ASSERT_OK (ret,"test_htab_leak run" )) + goto out_skel; +if (!ASSERT_OK (opts.retval,"test_htab_leak retval" )) + goto out_skel; + + map_id =get_map_id (bpf_map__fd (skel->maps.race_hash_map)); +if (!ASSERT_GE (map_id,0 ,"map_id" )) + goto out_skel; + + watcher =map_kptr_race__open_and_load (); +if (!ASSERT_OK_PTR (watcher,"watcher open_and_load" )) + goto out_skel; + + watcher->bss->target_map_id = map_id; + watcher->links.map_put =bpf_program__attach (watcher->progs.map_put); +if (!ASSERT_OK_PTR (watcher->links.map_put,"attach fentry" )) + goto out_watcher; + watcher->links.htab_map_free =bpf_program__attach (watcher->progs.htab_map_free); +if (!ASSERT_OK_PTR (watcher->links.htab_map_free,"attach fexit" )) + goto out_watcher; + +map_kptr_race__destroy (skel); + skel =NULL ; + +kern_sync_rcu (); + +while (!READ_ONCE (watcher->bss->map_freed)) +sched_yield (); + +ASSERT_EQ (watcher->bss->map_freed,1 ,"map_freed" ); +ASSERT_EQ (read_refs (watcher),2 ,"htab refcount" ); + +out_watcher: +map_kptr_race__destroy (watcher); +out_skel: +map_kptr_race__destroy (skel); +} + +static void test_percpu_htab_leak (void ) +{ +LIBBPF_OPTS (bpf_test_run_opts, opts, + .data_in = &pkt_v4, + .data_size_in = sizeof(pkt_v4), + .repeat = 1, + ); +struct map_kptr_race *skel, *watcher; +int ret, map_id; + + skel =map_kptr_race__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return ; + + skel->rodata->nr_cpus =libbpf_num_possible_cpus (); +if (skel->rodata->nr_cpus >16 ) + skel->rodata->nr_cpus =16 ; + + ret =map_kptr_race__load (skel); +if (!ASSERT_OK (ret,"load" )) + goto out_skel; + + ret =bpf_prog_test_run_opts (bpf_program__fd (skel->progs.test_percpu_htab_leak), &opts); +if (!ASSERT_OK (ret,"test_percpu_htab_leak run" )) + goto out_skel; +if (!ASSERT_OK (opts.retval,"test_percpu_htab_leak retval" )) + goto out_skel; + + map_id =get_map_id (bpf_map__fd (skel->maps.race_percpu_hash_map)); +if (!ASSERT_GE (map_id,0 ,"map_id" )) + goto out_skel; + + watcher =map_kptr_race__open_and_load (); +if (!ASSERT_OK_PTR (watcher,"watcher open_and_load" )) + goto out_skel; + + watcher->bss->target_map_id = map_id; + watcher->links.map_put =bpf_program__attach (watcher->progs.map_put); +if (!ASSERT_OK_PTR (watcher->links.map_put,"attach fentry" )) + goto out_watcher; + watcher->links.htab_map_free =bpf_program__attach (watcher->progs.htab_map_free); +if (!ASSERT_OK_PTR (watcher->links.htab_map_free,"attach fexit" )) + goto out_watcher; + +map_kptr_race__destroy (skel); + skel =NULL ; + +kern_sync_rcu (); + +while (!READ_ONCE (watcher->bss->map_freed)) +sched_yield (); + +ASSERT_EQ (watcher->bss->map_freed,1 ,"map_freed" ); +ASSERT_EQ (read_refs (watcher),2 ,"percpu_htab refcount" ); + +out_watcher: +map_kptr_race__destroy (watcher); +out_skel: +map_kptr_race__destroy (skel); +} + +static void test_sk_ls_leak (void ) +{ +struct map_kptr_race *skel, *watcher; +int listen_fd =-1 , client_fd =-1 , map_id; + + skel =map_kptr_race__open_and_load (); +if (!ASSERT_OK_PTR (skel,"open_and_load" )) +return ; + +if (!ASSERT_OK (map_kptr_race__attach (skel),"attach" )) + goto out_skel; + + listen_fd =start_server (AF_INET6 ,SOCK_STREAM ,"::1" ,0 ,0 ); +if (!ASSERT_GE (listen_fd,0 ,"start_server" )) + goto out_skel; + + client_fd =connect_to_fd (listen_fd,0 ); +if (!ASSERT_GE (client_fd,0 ,"connect_to_fd" )) + goto out_skel; + +if (!ASSERT_EQ (skel->bss->sk_ls_leak_done,1 ,"sk_ls_leak_done" )) + goto out_skel; + +close (client_fd); + client_fd =-1 ; +close (listen_fd); + listen_fd =-1 ; + + map_id =get_map_id (bpf_map__fd (skel->maps.race_sk_ls_map)); +if (!ASSERT_GE (map_id,0 ,"map_id" )) + goto out_skel; + + watcher =map_kptr_race__open_and_load (); +if (!ASSERT_OK_PTR (watcher,"watcher open_and_load" )) + goto out_skel; + + watcher->bss->target_map_id = map_id; + watcher->links.map_put =bpf_program__attach (watcher->progs.map_put); +if (!ASSERT_OK_PTR (watcher->links.map_put,"attach fentry" )) + goto out_watcher; + watcher->links.sk_map_free =bpf_program__attach (watcher->progs.sk_map_free); +if (!ASSERT_OK_PTR (watcher->links.sk_map_free,"attach fexit" )) + goto out_watcher; + +map_kptr_race__destroy (skel); + skel =NULL ; + +kern_sync_rcu (); + +while (!READ_ONCE (watcher->bss->map_freed)) +sched_yield (); + +ASSERT_EQ (watcher->bss->map_freed,1 ,"map_freed" ); +ASSERT_EQ (read_refs (watcher),2 ,"sk_ls refcount" ); + +out_watcher: +map_kptr_race__destroy (watcher); +out_skel: +if (client_fd >=0 ) +close (client_fd); +if (listen_fd >=0 ) +close (listen_fd); +map_kptr_race__destroy (skel); +} + +void serial_test_map_kptr_race (void ) +{ +if (test__start_subtest ("htab_leak" )) +test_htab_leak (); +if (test__start_subtest ("percpu_htab_leak" )) +test_percpu_htab_leak (); +if (test__start_subtest ("sk_ls_leak" )) +test_sk_ls_leak (); +}tools/testing/selftests/bpf/progs/map_kptr_race.c new file +197
-1@@ -0,0 +1,197 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_tracing.h> +#include "../test_kmods/bpf_testmod_kfunc.h" + +struct map_value { + struct prog_test_ref_kfunc __kptr *ref_ptr; +}; + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(map_flags, BPF_F_NO_PREALLOC); + __type(key, int); + __type(value, struct map_value); + __uint(max_entries, 1); +} race_hash_map SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_PERCPU_HASH); + __uint(map_flags, BPF_F_NO_PREALLOC); + __type(key, int); + __type(value, struct map_value); + __uint(max_entries, 1); +} race_percpu_hash_map SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_SK_STORAGE); + __uint(map_flags, BPF_F_NO_PREALLOC); + __type(key, int); + __type(value, struct map_value); +} race_sk_ls_map SEC(".maps"); + +int num_of_refs; +int sk_ls_leak_done; +int target_map_id; +int map_freed; +const volatile int nr_cpus; + +SEC("tc") +int test_htab_leak (struct __sk_buff *skb) +{ +struct prog_test_ref_kfunc *p, *old; +struct map_value val = {}; +struct map_value *v; +int key =0 ; + +if (bpf_map_update_elem (&race_hash_map, &key, &val,BPF_ANY )) +return 1 ; + + v =bpf_map_lookup_elem (&race_hash_map, &key); +if (!v) +return 2 ; + + p =bpf_kfunc_call_test_acquire (&(unsigned long ){0 }); +if (!p) +return 3 ; + old =bpf_kptr_xchg (&v->ref_ptr, p); +if (old) +bpf_kfunc_call_test_release (old); + +bpf_map_delete_elem (&race_hash_map, &key); + + p =bpf_kfunc_call_test_acquire (&(unsigned long ){0 }); +if (!p) +return 4 ; + old =bpf_kptr_xchg (&v->ref_ptr, p); +if (old) +bpf_kfunc_call_test_release (old); + +return 0 ; +} + +static int fill_percpu_kptr (struct map_value *v) +{ +struct prog_test_ref_kfunc *p, *old; + + p =bpf_kfunc_call_test_acquire (&(unsigned long ){0 }); +if (!p) +return 1 ; + old =bpf_kptr_xchg (&v->ref_ptr, p); +if (old) +bpf_kfunc_call_test_release (old); +return 0 ; +} + +SEC("tc") +int test_percpu_htab_leak (struct __sk_buff *skb) +{ +struct map_value *v, *arr[16 ] = {}; +struct map_value val = {}; +int key =0 ; +int err =0 ; + +if (bpf_map_update_elem (&race_percpu_hash_map, &key, &val,BPF_ANY )) +return 1 ; + +for (int i =0 ; i < nr_cpus; i++) { + v =bpf_map_lookup_percpu_elem (&race_percpu_hash_map, &key, i); +if (!v) +return 2 ; + arr[i] = v; + } + +bpf_map_delete_elem (&race_percpu_hash_map, &key); + +for (int i =0 ; i < nr_cpus; i++) { + v = arr[i]; + err =fill_percpu_kptr (v); +if (err) +return 3 ; + } + +return 0 ; +} + +SEC("tp_btf/inet_sock_set_state") +int BPF_PROG (test_sk_ls_leak ,struct sock *sk,int oldstate,int newstate) +{ +struct prog_test_ref_kfunc *p, *old; +struct map_value *v; + +if (newstate !=BPF_TCP_SYN_SENT ) +return 0 ; + +if (sk_ls_leak_done) +return 0 ; + + v =bpf_sk_storage_get (&race_sk_ls_map, sk,NULL , +BPF_SK_STORAGE_GET_F_CREATE ); +if (!v) +return 0 ; + + p =bpf_kfunc_call_test_acquire (&(unsigned long ){0 }); +if (!p) +return 0 ; + old =bpf_kptr_xchg (&v->ref_ptr, p); +if (old) +bpf_kfunc_call_test_release (old); + +bpf_sk_storage_delete (&race_sk_ls_map, sk); + + p =bpf_kfunc_call_test_acquire (&(unsigned long ){0 }); +if (!p) +return 0 ; + old =bpf_kptr_xchg (&v->ref_ptr, p); +if (old) +bpf_kfunc_call_test_release (old); + + sk_ls_leak_done =1 ; +return 0 ; +} + +long target_map_ptr; + +SEC("fentry/bpf_map_put") +int BPF_PROG (map_put ,struct bpf_map *map) +{ +if (target_map_id && map->id == (u32 )target_map_id) + target_map_ptr = (long )map; +return 0 ; +} + +SEC("fexit/htab_map_free") +int BPF_PROG (htab_map_free ,struct bpf_map *map) +{ +if (target_map_ptr && (long )map == target_map_ptr) + map_freed =1 ; +return 0 ; +} + +SEC("fexit/bpf_sk_storage_map_free") +int BPF_PROG (sk_map_free ,struct bpf_map *map) +{ +if (target_map_ptr && (long )map == target_map_ptr) + map_freed =1 ; +return 0 ; +} + +SEC("syscall") +int count_ref (void *ctx) +{ +struct prog_test_ref_kfunc *p; +unsigned long arg =0 ; + + p =bpf_kfunc_call_test_acquire (&arg); +if (!p) +return 1 ; + + num_of_refs = p->cnt.refs.counter; + +bpf_kfunc_call_test_release (p); +return 0 ; +} + +char _license[] SEC("license") = "GPL"; --2.47.3