bpf: Use bpf_mem_cache_alloc/free in bpf_local_storage
Subsystems (5)
Martin KaFai Lau
reviewerv3 cover bpf: Use bpf_mem_cache_alloc/free in bpf_local_storage
From: Martin KaFai Lau <[email protected]>
This set is a continuation of the effort in using
bpf_mem_cache_alloc/free in bpf_local_storage [1]
Major change is only using bpf_mem_alloc for task and cgrp storage while sk and inode stay with kzalloc/kfree. The details is in patch 2.
[1]: https://lore.kernel.org/bpf/[email protected]/
v3:
- Only use bpf_mem_alloc for task and cgrp storage.
- sk and inode storage stay with kzalloc/kfree.
- Check NULL and add comments in bpf_mem_cache_raw_free() in patch 1.
- Added test and benchmark for task storage.
v2:
- Added bpf_mem_cache_alloc_flags() and bpf_mem_cache_raw_free() to hide the internal data structure of the bpf allocator.
- Fixed a typo bug in bpf_selem_free()
- Simplified the test_local_storage test by directly using err returned from libbpf
Martin KaFai Lau (5):
bpf: Add a few bpf mem allocator functions
bpf: Use bpf_mem_cache_alloc/free in bpf_local_storage_elem
bpf: Use bpf_mem_cache_alloc/free for bpf_local_storage selftests/bpf: Test task storage when local_storage->smap is NULL selftests/bpf: Add bench for task storage creation
include/linux/bpf_local_storage.h | 7 +- include/linux/bpf_mem_alloc.h | 2 + kernel/bpf/bpf_cgrp_storage.c | 2 +- kernel/bpf/bpf_inode_storage.c | 2 +- kernel/bpf/bpf_local_storage.c | 223 ++++++++++++++++-- kernel/bpf/bpf_task_storage.c | 2 +- kernel/bpf/memalloc.c | 59 ++++- net/core/bpf_sk_storage.c | 2 +- tools/testing/selftests/bpf/bench.c | 2 + .../bpf/benchs/bench_local_storage_create.c | 151 ++++++++++-- .../bpf/prog_tests/test_local_storage.c | 7 +- .../bpf/progs/bench_local_storage_create.c | 25 ++ .../selftests/bpf/progs/local_storage.c | 56 +++-- 13 files changed, 472 insertions(+), 68 deletions(-)
Signature · 1 line
2.34.1
Martin KaFai Lau
reviewerPATCH v3 1/5 bpf: Add a few bpf mem allocator functions
From: Martin KaFai Lau <[email protected]>
This patch adds a few bpf mem allocator functions which will be used in the bpf_local_storage in a later patch.
bpf_mem_cache_alloc_flags(..., gfp_t flags) is added. When the flags == GFP_KERNEL, it will fallback to __alloc(..., GFP_KERNEL). bpf_local_storage knows its running context is sleepable (GFP_KERNEL) and provides a better guarantee on memory allocation.
bpf_local_storage has some uncommon cases that its selem cannot be reused immediately. It handles its own rcu_head and goes through a rcu_trace gp and then free it. bpf_mem_cache_raw_free() is added for direct free purpose without leaking the LLIST_NODE_SZ internal knowledge. During free time, the 'struct bpf_mem_alloc *ma' is no longer available. However, the caller should know if it is percpu memory or not and it can call different raw_free functions. bpf_local_storage does not support percpu value, so only the non-percpu 'bpf_mem_cache_raw_free()' is added in this patch.
--- include/linux/bpf_mem_alloc.h | 2 ++ kernel/bpf/memalloc.c | 59 +++++++++++++++++++++++++++++------ 2 files changed, 52 insertions(+), 9 deletions(-)
include/linux/bpf_mem_alloc.h +2 -0
@@ -31,5 +31,7 @@ void bpf_mem_free(struct bpf_mem_alloc *ma, void *ptr);/* kmem_cache_alloc/free equivalent: */ void *bpf_mem_cache_alloc (struct bpf_mem_alloc *ma);void bpf_mem_cache_free (struct bpf_mem_alloc *ma,void *ptr); +void bpf_mem_cache_raw_free (void *ptr); +void *bpf_mem_cache_alloc_flags (struct bpf_mem_alloc *ma,gfp_t flags);#endif /* _BPF_MEM_ALLOC_H */
kernel/bpf/memalloc.c +50 -9
75 lines not on this page · raw message
Signature · 1 line
2.34.1
Martin KaFai Lau
reviewerPATCH v3 2/5 bpf: Use bpf_mem_cache_alloc/free in bpf_local_storage_elem
In reply to Martin KaFai LauFrom: Martin KaFai Lau <[email protected]>
This patch uses bpf_mem_alloc for the task and cgroup local storage that the bpf prog can easily get a hold of the storage owner's PTR_TO_BTF_ID. eg. bpf_get_current_task_btf() can be used in some of the kmalloc code path which will cause deadlock/recursion. bpf_mem_cache_alloc is deadlock free and will solve a legit use case in [1].
For sk storage, its batch creation benchmark shows a few percent regression when the sk create/destroy batch size is larger than 32. The sk creation/destruction happens much more often and depends on external traffic. Considering it is hypothetical to be able to cause deadlock with sk storage, it can cross the bridge to use bpf_mem_alloc till a legit (ie. useful) use case comes up.
For inode storage, bpf_local_storage_destroy() is called before waiting for a rcu gp and its memory cannot be reused immediately. inode stays with kmalloc/kfree after the rcu [or tasks_trace] gp.
A 'bool bpf_ma' argument is added to bpf_local_storage_map_alloc(). Only task and cgroup storage have 'bpf_ma == true' which means to use bpf_mem_cache_alloc/free(). This patch only changes selem to use bpf_mem_alloc for task and cgroup. The next patch will change the local_storage to use bpf_mem_alloc also for task and cgroup.
Here is some more details on the changes:
* memory allocation:
After bpf_mem_cache_alloc(), the SDATA(selem)->data is zero-ed because bpf_mem_cache_alloc() could return a reused selem. It is to keep the existing bpf_map_kzalloc() behavior. Only SDATA(selem)->data is zero-ed. SDATA(selem)->data is the visible part to the bpf prog. No need to use zero_map_value() to do the zeroing because bpf_selem_free(..., reuse_now = true) ensures no bpf prog is using the selem before returning the selem through bpf_mem_cache_free(). For the internal fields of selem, they will be initialized when linking to the new smap and the new local_storage.
When 'bpf_ma == false', nothing changes in this patch. It will stay with the bpf_map_kzalloc().
* memory free:
The bpf_selem_free() and bpf_selem_free_rcu() are modified to handle the bpf_ma == true case.
For the common selem free path where its owner is also being destroyed, the mem is freed in bpf_local_storage_destroy(), the owner (task and cgroup) has gone through a rcu gp. The memory can be reused immediately, so bpf_local_storage_destroy() will call bpf_selem_free(..., reuse_now = true) which will do bpf_mem_cache_free() for immediate reuse consideration.
An exception is the delete elem code path. The delete elem code path is called from the helper bpf_*_storage_delete() and the syscall bpf_map_delete_elem(). This path is an unusual case for local storage because the common use case is to have the local storage staying with its owner life time so that the bpf prog and the user space does not have to monitor the owner's destruction. For the delete elem path, the selem cannot be reused immediately because there could be bpf prog using it. It will call bpf_selem_free(..., reuse_now = false) and it will wait for a rcu tasks trace gp before freeing the elem. The rcu callback is changed to do bpf_mem_cache_raw_free() instead of kfree().
When 'bpf_ma == false', it should be the same as before. __bpf_selem_free() is added to do the kfree_rcu and call_tasks_trace_rcu(). A few words on the 'reuse_now == true'. When 'reuse_now == true', it is still racing with bpf_local_storage_map_free which is under rcu protection, so it still needs to wait for a rcu gp instead of kfree(). Otherwise, the selem may be reused by slab for a totally different struct while the bpf_local_storage_map_free() is still using it (as a rcu reader). For the inode case, there may be other rcu readers also. In short, when bpf_ma == false and reuse_now == true => vanilla rcu.
[1]: https://lore.kernel.org/bpf/[email protected]/
--- include/linux/bpf_local_storage.h | 6 +- kernel/bpf/bpf_cgrp_storage.c | 2 +- kernel/bpf/bpf_inode_storage.c | 2 +- kernel/bpf/bpf_local_storage.c | 95 ++++++++++++++++++++++++++++--- kernel/bpf/bpf_task_storage.c | 2 +- net/core/bpf_sk_storage.c | 2 +- 6 files changed, 95 insertions(+), 14 deletions(-)
include/linux/bpf_local_storage.h +5 -1
@@ -13,6 +13,7 @@#include <linux/list.h> #include <linux/hash.h> #include <linux/types.h> +#include <linux/bpf_mem_alloc.h> #include <uapi/linux/btf.h> #define BPF_LOCAL_STORAGE_CACHE_SIZE 16
@@ -55,6 +56,8 @@ struct bpf_local_storage_map {u32 bucket_log;u16 elem_size;u16 cache_idx; +struct bpf_mem_alloc selem_ma; +bool bpf_ma; };struct bpf_local_storage_data {
@@ -122,7 +125,8 @@ int bpf_local_storage_map_alloc_check(union bpf_attr *attr);struct bpf_map *bpf_local_storage_map_alloc (union bpf_attr *attr, -struct bpf_local_storage_cache *cache); +struct bpf_local_storage_cache *cache, +bool bpf_ma);struct bpf_local_storage_data * bpf_local_storage_lookup(struct bpf_local_storage *local_storage,
kernel/bpf/bpf_cgrp_storage.c +1 -1
@@ -149,7 +149,7 @@ static int notsupp_get_next_key(struct bpf_map *map, void *key, void *next_key)static struct bpf_map *cgroup_storage_map_alloc (union bpf_attr *attr) { -return bpf_local_storage_map_alloc (attr, &cgroup_cache); +return bpf_local_storage_map_alloc (attr, &cgroup_cache, true); }static void cgroup_storage_map_free (struct bpf_map *map)
kernel/bpf/bpf_inode_storage.c +1 -1
@@ -199,7 +199,7 @@ static int notsupp_get_next_key(struct bpf_map *map, void *key,static struct bpf_map *inode_storage_map_alloc (union bpf_attr *attr) { -return bpf_local_storage_map_alloc (attr, &inode_cache); +return bpf_local_storage_map_alloc (attr, &inode_cache, false); }static void inode_storage_map_free (struct bpf_map *map)
kernel/bpf/bpf_local_storage.c +86 -9
151 lines not on this page · raw message
kernel/bpf/bpf_task_storage.c +1 -1
@@ -309,7 +309,7 @@ static int notsupp_get_next_key(struct bpf_map *map, void *key, void *next_key)static struct bpf_map *task_storage_map_alloc (union bpf_attr *attr) { -return bpf_local_storage_map_alloc (attr, &task_cache); +return bpf_local_storage_map_alloc (attr, &task_cache, true); }static void task_storage_map_free (struct bpf_map *map)
net/core/bpf_sk_storage.c +1 -1
@@ -68,7 +68,7 @@ static void bpf_sk_storage_map_free(struct bpf_map *map)static struct bpf_map *bpf_sk_storage_map_alloc (union bpf_attr *attr) { -return bpf_local_storage_map_alloc (attr, &sk_cache); +return bpf_local_storage_map_alloc (attr, &sk_cache, false); } static int notsupp_get_next_key(struct bpf_map *map, void *key,
Signature · 1 line
2.34.1
Martin KaFai Lau
reviewerPATCH v3 3/5 bpf: Use bpf_mem_cache_alloc/free for bpf_local_storage
In reply to Martin KaFai LauFrom: Martin KaFai Lau <[email protected]>
This patch uses bpf_mem_cache_alloc/free for allocating and freeing bpf_local_storage for task and cgroup storage.
The changes are similar to the previous patch. A few things that worth to mention for bpf_local_storage:
The local_storage is freed when the last selem is deleted. Before deleting a selem from local_storage, it needs to retrieve the local_storage->smap because the bpf_selem_unlink_storage_nolock() may have set it to NULL. Note that local_storage->smap may have already been NULL when the selem created this local_storage has been removed. In this case, call_rcu will be used to free the local_storage.
Also, the bpf_ma (true or false) value is needed before calling bpf_local_storage_free(). The bpf_ma can either be obtained from the local_storage->smap (if available) or any of its selem's smap. A new helper check_storage_bpf_ma() is added to obtain bpf_ma for a deleting bpf_local_storage.
When bpf_local_storage_alloc getting a reused memory, all fields are either in the correct values or will be initialized. 'cache[]' must already be all NULLs. 'list' must be empty. Others will be initialized.
--- include/linux/bpf_local_storage.h | 1 + kernel/bpf/bpf_local_storage.c | 130 ++++++++++++++++++++++++++---- 2 files changed, 116 insertions(+), 15 deletions(-)
include/linux/bpf_local_storage.h +1 -0
@@ -57,6 +57,7 @@ struct bpf_local_storage_map {u16 elem_size;u16 cache_idx;struct bpf_mem_alloc selem_ma; +struct bpf_mem_alloc storage_ma;bool bpf_ma; };
kernel/bpf/bpf_local_storage.c +115 -15
219 lines not on this page · raw message
Signature · 1 line
2.34.1
Martin KaFai Lau
reviewerPATCH v3 4/5 selftests/bpf: Test task storage when local_storage->smap is NULL
In reply to Martin KaFai LauFrom: Martin KaFai Lau <[email protected]>
The current sk storage test ensures the memory free works when the local_storage->smap is NULL.
This patch adds a task storage test to ensure the memory free code path works when local_storage->smap is NULL.
--- .../bpf/prog_tests/test_local_storage.c | 7 ++- .../selftests/bpf/progs/local_storage.c | 56 ++++++++++++++----- 2 files changed, 46 insertions(+), 17 deletions(-)
tools/testing/selftests/bpf/prog_tests/test_local_storage.c +4 -3
@@ -23,7 +23,7 @@ struct storage {/* Fork and exec the provided rm binary and return the exit code of the * forked process and its pid. */ -static int run_self_unlink (int *monitored_pid,const char *rm_path) +static int run_self_unlink (struct local_storage *skel,const char *rm_path) {int child_pid, child_status, ret;int null_fd;
@@ -35,7 +35,7 @@ static int run_self_unlink(int *monitored_pid, const char *rm_path)dup2 (null_fd,STDERR_FILENO );close (null_fd); - *monitored_pid =getpid (); + skel->bss->monitored_pid =getpid ();/* Use the copied /usr/bin/rm to delete itself * /tmp/copy_of_rm /tmp/copy_of_rm. */
@@ -44,6 +44,7 @@ static int run_self_unlink(int *monitored_pid, const char *rm_path)exit (errno); }else if (child_pid >0 ) {waitpid (child_pid, &child_status,0 ); +ASSERT_EQ (skel->data->task_storage_result,0 ,"task_storage_result" );return WEXITSTATUS (child_status); }
@@ -133,7 +134,7 @@ void test_test_local_storage(void)* unlink its executable. This operation should be denied by the loaded * LSM program. */ - err =run_self_unlink (&skel->bss->monitored_pid, tmp_exec_path); + err =run_self_unlink (skel, tmp_exec_path);if (!ASSERT_EQ (err,EPERM ,"run_self_unlink" )) goto close_prog_rmdir;
tools/testing/selftests/bpf/progs/local_storage.c +42 -14
85 lines not on this page · raw message
Signature · 1 line
2.34.1
Martin KaFai Lau
reviewerPATCH v3 5/5 selftests/bpf: Add bench for task storage creation
In reply to Martin KaFai LauFrom: Martin KaFai Lau <[email protected]>
This patch adds a task storage benchmark to the existing local-storage-create benchmark.
For task storage,
./bench --storage-type task --batch-size 32:
bpf_ma: Summary: creates 30.456 ± 0.507k/s ( 30.456k/prod), 6.08 kmallocs/create
no bpf_ma: Summary: creates 31.962 ± 0.486k/s ( 31.962k/prod), 6.13 kmallocs/create
./bench --storage-type task --batch-size 64:
bpf_ma: Summary: creates 30.197 ± 1.476k/s ( 30.197k/prod), 6.08 kmallocs/create
no bpf_ma: Summary: creates 31.103 ± 0.297k/s ( 31.103k/prod), 6.13 kmallocs/create
--- tools/testing/selftests/bpf/bench.c | 2 + .../bpf/benchs/bench_local_storage_create.c | 151 ++++++++++++++++-- .../bpf/progs/bench_local_storage_create.c | 25 +++ 3 files changed, 164 insertions(+), 14 deletions(-)
tools/testing/selftests/bpf/bench.c +2 -0
@@ -278,6 +278,7 @@ extern struct argp bench_local_storage_argp;extern struct argp bench_local_storage_rcu_tasks_trace_argp;extern struct argp bench_strncmp_argp;extern struct argp bench_hashmap_lookup_argp; +extern struct argp bench_local_storage_create_argp;static const struct argp_child bench_parsers[] = { { &bench_ringbufs_argp,0 ,"Ring buffers benchmark" ,0 },
@@ -288,6 +289,7 @@ static const struct argp_child bench_parsers[] = { { &bench_local_storage_rcu_tasks_trace_argp,0 , "local_storage RCU Tasks Trace slowdown benchmark", 0 }, { &bench_hashmap_lookup_argp, 0, "Hashmap lookup benchmark", 0 }, + { &bench_local_storage_create_argp, 0, "local-storage-create benchmark", 0 }, {}, };
tools/testing/selftests/bpf/benchs/bench_local_storage_create.c +137 -14
218 lines not on this page · raw message
tools/testing/selftests/bpf/progs/bench_local_storage_create.c +25 -0
@@ -22,6 +22,13 @@ struct { __type(value, struct storage); } sk_storage_map SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_TASK_STORAGE); + __uint(map_flags, BPF_F_NO_PREALLOC); + __type(key, int); + __type(value, struct storage); +} task_storage_map SEC(".maps"); + SEC("raw_tp/kmalloc") int BPF_PROG(kmalloc, unsigned long call_site, const void *ptr, size_t bytes_req, size_t bytes_alloc, gfp_t gfp_flags,
@@ -32,6 +39,24 @@ int BPF_PROG(kmalloc, unsigned long call_site, const void *ptr,return 0 ; } +SEC("tp_btf/sched_process_fork") +int BPF_PROG (fork ,struct task_struct *parent,struct task_struct *child) +{ +struct storage *stg; + +if (parent->tgid != bench_pid) +return 0 ; + + stg =bpf_task_storage_get (&task_storage_map, child,NULL , +BPF_LOCAL_STORAGE_GET_F_CREATE ); +if (stg) +__sync_fetch_and_add (&create_cnts,1 ); +else +__sync_fetch_and_add (&create_errs,1 ); + +return 0 ; +} + SEC("lsm.s/socket_post_create")int BPF_PROG (socket_post_create ,struct socket *sock,int family,int type,int protocol,int kern)
Signature · 1 line
2.34.1
Re: [PATCH v3 bpf-next 5/5] selftests/bpf: Add bench for task storage creation
Quoted 290 lines · Martin KaFai Lau
From: Martin KaFai Lau <[email protected]>
This patch adds a task storage benchmark to the existing local-storage-create benchmark.
For task storage,
./bench --storage-type task --batch-size 32:bpf_ma: Summary: creates 30.456 ± 0.507k/s ( 30.456k/prod), 6.08 kmallocs/createno
bpf_ma: Summary: creates 31.962 ± 0.486k/s ( 31.962k/prod), 6.13 kmallocs/create./bench --storage-type task --batch-size 64:
bpf_ma: Summary: creates 30.197 ± 1.476k/s ( 30.197k/prod), 6.08 kmallocs/createno
bpf_ma: Summary: creates 31.103 ± 0.297k/s ( 31.103k/prod), 6.13 kmallocs/createSigned-off-by Martin KaFai Lau <[email protected]>--- tools/testing/selftests/bpf/bench.c | 2 + .../bpf/benchs/bench_local_storage_create.c | 151 ++++++++++++++++-- .../bpf/progs/bench_local_storage_create.c | 25 +++ 3 files changed, 164 insertions(+), 14 deletions(-)tools/testing/selftests/bpf/bench.c +2
-0@@ -278,6 +278,7 @@ extern struct argp bench_local_storage_argp;extern struct argp bench_local_storage_rcu_tasks_trace_argp;extern struct argp bench_strncmp_argp;extern struct argp bench_hashmap_lookup_argp; +extern struct argp bench_local_storage_create_argp;static const struct argp_child bench_parsers[] = { { &bench_ringbufs_argp,0 ,"Ring buffers benchmark" ,0 },@@ -288,6 +289,7 @@ static const struct argp_child bench_parsers[] = { { &bench_local_storage_rcu_tasks_trace_argp,0 , "local_storage RCU Tasks Trace slowdown benchmark", 0 }, { &bench_hashmap_lookup_argp, 0, "Hashmap lookup benchmark", 0 }, + { &bench_local_storage_create_argp, 0, "local-storage-create benchmark", 0 }, {}, };tools/testing/selftests/bpf/benchs/bench_local_storage_create.c +137
-14@@ -3,19 +3,71 @@#include <sys/types.h> #include <sys/socket.h> +#include <pthread.h> +#include <argp.h> #include "bench.h" #include "bench_local_storage_create.skel.h" -#define BATCH_SZ 32 -struct thread { -int fds[BATCH_SZ ]; +int *fds; +pthread_t *pthds; +int *pthd_results; };static struct bench_local_storage_create *skel;static struct thread *threads; -static long socket_errs; +static long create_owner_errs; +static int storage_type =BPF_MAP_TYPE_SK_STORAGE ; +static int batch_sz =32 ; + +enum { +ARG_BATCH_SZ =9000 , +ARG_STORAGE_TYPE =9001 , +}; + +static const struct argp_option opts[] = { + {"batch-size" ,ARG_BATCH_SZ ,"BATCH_SIZE" ,0 , +"The number of storage creations in each batch" }, + {"storage-type" ,ARG_STORAGE_TYPE ,"STORAGE_TYPE" ,0 , +"The type of local storage to test (socket or task)" }, + {}, +}; + +static error_t parse_arg (int key,char *arg,struct argp_state *state) +{ +int ret; + +switch (key) { +case ARG_BATCH_SZ : + ret =atoi (arg); +if (ret <1 ) { +fprintf (stderr,"invalid batch-size\n" ); +argp_usage (state); + } + batch_sz = ret; +break ; +case ARG_STORAGE_TYPE : +if (!strcmp (arg,"task" )) { + storage_type =BPF_MAP_TYPE_TASK_STORAGE ; + }else if (!strcmp (arg,"socket" )) { + storage_type =BPF_MAP_TYPE_SK_STORAGE ; + }else { +fprintf (stderr,"invalid storage-type (socket or task)\n" ); +argp_usage (state); + } +break ; +default : +return ARGP_ERR_UNKNOWN ; + } + +return 0 ; +} + +const struct argp bench_local_storage_create_argp = { + .options = opts, + .parser = parse_arg, +};static void validate (void ) {@@ -28,6 +80,8 @@ static void validate(void)static void setup (void ) { +int i; + skel =bench_local_storage_create__open_and_load ();if (!skel) {fprintf (stderr,"error loading skel\n" );@@ -35,10 +89,16 @@ static void setup(void) } skel->bss->bench_pid =getpid (); - -if (!bpf_program__attach (skel->progs.socket_post_create)) { -fprintf (stderr,"Error attaching bpf program\n" ); -exit (1 ); +if (storage_type ==BPF_MAP_TYPE_SK_STORAGE ) { +if (!bpf_program__attach (skel->progs.socket_post_create)) { +fprintf (stderr,"Error attaching bpf program\n" ); +exit (1 ); + } + }else { +if (!bpf_program__attach (skel->progs.fork)) { +fprintf (stderr,"Error attaching bpf program\n" ); +exit (1 ); + } }if (!bpf_program__attach (skel->progs.kmalloc)) {@@ -52,6 +112,29 @@ static void setup(void)fprintf (stderr,"cannot alloc thread_res\n" );exit (1 ); } + +for (i =0 ; i < env.producer_cnt; i++) { +struct thread *t = &threads[i]; + +if (storage_type ==BPF_MAP_TYPE_SK_STORAGE ) { + t->fds =malloc (batch_sz *sizeof (*t->fds)); +if (!t->fds) { +fprintf (stderr,"cannot alloc t->fds\n" ); +exit (1 ); + } + }else { + t->pthds =malloc (batch_sz *sizeof (*t->pthds)); +if (!t->pthds) { +fprintf (stderr,"cannot alloc t->pthds\n" ); +exit (1 ); + } + t->pthd_results =malloc (batch_sz *sizeof (*t->pthd_results)); +if (!t->pthd_results) { +fprintf (stderr,"cannot alloc t->pthd_results\n" ); +exit (1 ); + } + } + } }static void measure (struct bench_res *res)@@ -65,20 +148,20 @@ static void *consumer(void *input)return NULL ; } -static void *producer (void *input) +static void *sk_producer (void *input) {struct thread *t = &threads[(long )(input)];int *fds = t->fds;int i;while (true) { -for (i =0 ; i <BATCH_SZ ; i++) { +for (i =0 ; i < batch_sz; i++) { fds[i] =socket (AF_INET6 ,SOCK_DGRAM ,0 );if (fds[i] ==-1 ) -atomic_inc (&socket_errs); +atomic_inc (&create_owner_errs); } -for (i =0 ; i <BATCH_SZ ; i++) { +for (i =0 ; i < batch_sz; i++) {if (fds[i] !=-1 )close (fds[i]); }@@ -87,6 +170,42 @@ static void *producer(void *input)return NULL ; } +static void *thread_func (void *arg) +{ +return NULL ; +} + +static void *task_producer (void *input) +{ +struct thread *t = &threads[(long )(input)]; +pthread_t *pthds = t->pthds; +int *pthd_results = t->pthd_results; +int i; + +while (true) { +for (i =0 ; i < batch_sz; i++) { + pthd_results[i] =pthread_create (&pthds[i],NULL , thread_func,NULL ); +if (pthd_results[i]) +atomic_inc (&create_owner_errs); + } + +for (i =0 ; i < batch_sz; i++) { +if (!pthd_results[i]) +pthread_join (pthds[i],NULL );; + } + } + +return NULL ; +} + +static void *producer (void *input) +{ +if (storage_type ==BPF_MAP_TYPE_SK_STORAGE ) +return sk_producer (input); +else +return task_producer (input); +} +static void report_progress (int iter,struct bench_res *res,long delta_ns) {double creates_per_sec, kmallocs_per_create;@@ -123,14 +242,18 @@ static void report_final(struct bench_res res[], int res_cnt)printf ("Summary: creates %8.3lf \u00B1 %5.3lfk/s (%7.3lfk/prod), " , creates_mean, creates_stddev, creates_mean / env.producer_cnt);printf ("%4.2lf kmallocs/create\n" , (double )total_kmallocs / total_creates); -if (socket_errs || skel->bss->create_errs) -printf ("socket() errors %ld create_errs %ld\n" , socket_errs, +if (create_owner_errs || skel->bss->create_errs) +printf ("%s() errors %ld create_errs %ld\n" , + storage_type ==BPF_MAP_TYPE_SK_STORAGE ? +"socket" :"pthread_create" , + create_owner_errs, skel->bss->create_errs); }/* Benchmark performance of creating bpf local storage */ const struct bench bench_local_storage_create = { .name ="local-storage-create" , + .argp = &bench_local_storage_create_argp, .validate = validate, .setup = setup, .producer_thread = producer,tools/testing/selftests/bpf/progs/bench_local_storage_create.c +7
-0@@ -22,6 +22,13 @@ struct { __type(value, struct storage); } sk_storage_map SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_TASK_STORAGE); + __uint(map_flags, BPF_F_NO_PREALLOC); + __type(key, int); + __type(value, struct storage); +} task_storage_map SEC(".maps"); + SEC("raw_tp/kmalloc") int BPF_PROG(kmalloc, unsigned long call_site, const void *ptr, size_t bytes_req, size_t bytes_alloc, gfp_t gfp_flags,
tools/testing/selftests/bpf/progs/bench_local_storage_create.c int BPF_PROG(kmalloc, unsigned long call_site, const void *ptr, · line 43
Quoted 5 lines
tools/testing/selftests/bpf/progs/bench_local_storage_create.c +2
-0@@ -32,6 +39,24 @@ int BPF_PROG(kmalloc, unsigned long call_site, const void *ptr,return 0 ; } +SEC("tp_btf/sched_process_fork") +int BPF_PROG (fork ,struct task_struct *parent,struct task_struct *child)
Apparently fork is a built-in function in bpf-gcc:
In file included from progs/bench_local_storage_create.c:6:progs/bench_local_storage_create.c:43:14: error: conflicting types for built-in function 'fork'; expected 'int(void)' [-Werror=builtin-declaration-mismatch]
43 | int BPF_PROG(fork, struct task_struct *parent, structtask_struct *child)
| ^~~~
I haven't been able to find this documented anywhere however.
Quoted 23 lines
tools/testing/selftests/bpf/progs/bench_local_storage_create.c +16
-1+{ +struct storage *stg; + +if (parent->tgid != bench_pid) +return 0 ; + + stg =bpf_task_storage_get (&task_storage_map, child,NULL , +BPF_LOCAL_STORAGE_GET_F_CREATE ); +if (stg) +__sync_fetch_and_add (&create_cnts,1 ); +else +__sync_fetch_and_add (&create_errs,1 ); + +return 0 ; +} + SEC("lsm.s/socket_post_create")int BPF_PROG (socket_post_create ,struct socket *sock,int family,int type,int protocol,int kern) --2.34.1
Martin KaFai Lau
reviewerRe: [PATCH v3 bpf-next 5/5] selftests/bpf: Add bench for task storage creation
Quoted 22 lines · James Hilliard
Quoted 22 lines
tools/testing/selftests/bpf/progs/bench_local_storage_create.c +7
-0@@ -22,6 +22,13 @@ struct { __type(value, struct storage); } sk_storage_map SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_TASK_STORAGE); + __uint(map_flags, BPF_F_NO_PREALLOC); + __type(key, int); + __type(value, struct storage); +} task_storage_map SEC(".maps"); + SEC("raw_tp/kmalloc") int BPF_PROG(kmalloc, unsigned long call_site, const void *ptr, size_t bytes_req, size_t bytes_alloc, gfp_t gfp_flags,@@ -32,6 +39,24 @@ int BPF_PROG(kmalloc, unsigned long call_site, const void *ptr,return 0 ; }
Quoted 4 lines
tools/testing/selftests/bpf/progs/bench_local_storage_create.c int BPF_PROG(kmalloc, unsigned long call_site, const void *ptr, · line 43
Quoted 2 lines
tools/testing/selftests/bpf/progs/bench_local_storage_create.c +2
-0+SEC("tp_btf/sched_process_fork") +int BPF_PROG (fork ,struct task_struct *parent,struct task_struct *child)Apparently fork is a built-in function in bpf-gcc:
It is also failing in a plain C program
#> gcc -Werror=builtin-declaration-mismatch -o test test.c
test.c:14:35: error: conflicting types for built-in function ‘fork’; expected ‘int(void)’ [-Werror=builtin-declaration-mismatch]
14 | int __attribute__((__noinline__)) fork(long x, long y)
| ^~~~
cc1: some warnings being treated as errors
#> clang -o test test.c
succeed
I am not too attached to the name but it seems something should be addressed in the gcc instead.
Quoted 10 lines
In file included from
progs/bench_local_storage_create.c:6:progs/bench_local_storage_create.c:43:14: error: conflicting types for built-in function 'fork'; expected 'int(void)' [-Werror=builtin-declaration-mismatch]
43 | int BPF_PROG(fork, structtask_struct*parent, structtask_struct*child)| ^~~~I haven't been able to find this documented anywhere however.
Re: [PATCH v3 bpf-next 5/5] selftests/bpf: Add bench for task storage creation
Quoted 36 lines · Martin KaFai Lau
Quoted 22 lines · James Hilliard
Quoted 22 lines
tools/testing/selftests/bpf/progs/bench_local_storage_create.c +7
-0@@ -22,6 +22,13 @@ struct { __type(value, struct storage); } sk_storage_map SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_TASK_STORAGE); + __uint(map_flags, BPF_F_NO_PREALLOC); + __type(key, int); + __type(value, struct storage); +} task_storage_map SEC(".maps"); + SEC("raw_tp/kmalloc") int BPF_PROG(kmalloc, unsigned long call_site, const void *ptr, size_t bytes_req, size_t bytes_alloc, gfp_t gfp_flags,@@ -32,6 +39,24 @@ int BPF_PROG(kmalloc, unsigned long call_site, const void *ptr,return 0 ; }Quoted 4 lines
tools/testing/selftests/bpf/progs/bench_local_storage_create.c int BPF_PROG(kmalloc, unsigned long call_site, const void *ptr, · line 43
Quoted 2 lines
tools/testing/selftests/bpf/progs/bench_local_storage_create.c +2
-0+SEC("tp_btf/sched_process_fork") +int BPF_PROG (fork ,struct task_struct *parent,struct task_struct *child)Apparently fork is a built-in function in bpf-gcc:
It is also failing in a plain C program
#> gcc -Werror=builtin-declaration-mismatch -o test
test.ctest.c:14:35: error: conflicting types for built-in function ‘fork’; expected ‘int(void)’ [-Werror=builtin-declaration-mismatch]14 | int __attribute__((__noinline__)) fork(long x, long y) | ^~~~
Quoted 7 lines
cc1: some warnings being treated as errors
#> clang -o test
test.c
succeedI am not too attached to the name but it seems something should be addressed in the gcc instead.
Hmm, so it looks like it's marked as a builtin here:
https://github.com/gcc-mirror/gcc/blob/releases/gcc-12.1.0/gcc/builtins.def#L875
The macro for that is here:
https://github.com/gcc-mirror/gcc/blob/releases/gcc-12.1.0/gcc/builtins.def#L104-L111
Which has this comment:
/* Like DEF_LIB_BUILTIN, except that the function is not one that is specified by ANSI/ISO C. So, when we're being fully conformant we ignore the version of these builtins that does not begin with __builtin. */
Looks like this builtin was originally added here:
https://github.com/gcc-mirror/gcc/commit/d1c38823924506d389ca58d02926ace21bdf82fa
Based on this issue it looks like fork is treated as a builtin for libgcov support:
https://gcc.gnu.org/bugzilla//show_bug.cgi?id=82457
So from my understanding fork is a gcc builtin when building with -std=gnu11 but is not a builtin when building with -std=c11.
So it looks like fork is translated to __gcov_fork when -std=gnu* is set which is why we get this error.
As this appears to be intended behavior for gcc I think the best option is to just rename the function so that we don't run into issues when building with gnu extensions like -std=gnu11.
Quoted 12 lines
Quoted 11 lines
In file included from
progs/bench_local_storage_create.c:6:progs/bench_local_storage_create.c:43:14: error: conflicting types for built-in function 'fork'; expected 'int(void)' [-Werror=builtin-declaration-mismatch]
43 | int BPF_PROG(fork, structtask_struct*parent, structtask_struct*child)| ^~~~I haven't been able to find this documented anywhere however.
Martin KaFai Lau
reviewerRe: [PATCH v3 bpf-next 5/5] selftests/bpf: Add bench for task storage creation
Quoted 60 lines · James Hilliard
Quoted 36 lines · Martin KaFai Lau
Quoted 22 lines · James Hilliard
Quoted 22 lines
tools/testing/selftests/bpf/progs/bench_local_storage_create.c +7
-0@@ -22,6 +22,13 @@ struct { __type(value, struct storage); } sk_storage_map SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_TASK_STORAGE); + __uint(map_flags, BPF_F_NO_PREALLOC); + __type(key, int); + __type(value, struct storage); +} task_storage_map SEC(".maps"); + SEC("raw_tp/kmalloc") int BPF_PROG(kmalloc, unsigned long call_site, const void *ptr, size_t bytes_req, size_t bytes_alloc, gfp_t gfp_flags,@@ -32,6 +39,24 @@ int BPF_PROG(kmalloc, unsigned long call_site, const void *ptr,return 0 ; }Quoted 4 lines
tools/testing/selftests/bpf/progs/bench_local_storage_create.c int BPF_PROG(kmalloc, unsigned long call_site, const void *ptr, · line 43
Quoted 2 lines
tools/testing/selftests/bpf/progs/bench_local_storage_create.c +2
-0+SEC("tp_btf/sched_process_fork") +int BPF_PROG (fork ,struct task_struct *parent,struct task_struct *child)Apparently fork is a built-in function in bpf-gcc:
It is also failing in a plain C program
#> gcc -Werror=builtin-declaration-mismatch -o test
test.ctest.c:14:35: error: conflicting types for built-in function ‘fork’; expected ‘int(void)’ [-Werror=builtin-declaration-mismatch]14 | int __attribute__((__noinline__)) fork(long x, long y) | ^~~~Quoted 7 lines
cc1: some warnings being treated as errors
#> clang -o test
test.c
succeedI am not too attached to the name but it seems something should be addressed in the gcc instead.
Hmm, so it looks like it's marked as a builtin here:
https://github.com/gcc-mirror/gcc/blob/releases/gcc-12.1.0/gcc/builtins.def#L875The macro for that is here:
https://github.com/gcc-mirror/gcc/blob/releases/gcc-12.1.0/gcc/builtins.def#L104-L111Which has this comment:
/* LikeDEF_LIB_BUILTIN, except that the function is not one that is specified by ANSI/ISO C. So, when we're being fully conformant we ignore the version of these builtins that does not begin with__builtin. */Looks like this builtin was originally added here:
https://github.com/gcc-mirror/gcc/commit/d1c38823924506d389ca58d02926ace21bdf82fa
Quoted 6 lines
Based on this issue it looks like fork is treated as a builtin for libgcov support:
https://gcc.gnu.org/bugzilla//show_bug.cgi?id=82457So from my understanding fork is a gcc builtin when building with -std=gnu11 but is not a builtin when building with -std=c11.
That sounds like there is a knob to turn this behavior on and off. Do the same for the bpf target?
Quoted 7 lines
So it looks like fork is translated to
__gcov_forkwhen -std=gnu* is set which is why we get this error.As this appears to be intended behavior for gcc I think the best option is to just rename the function so that we don't run into issues when building with gnu extensions like -std=gnu11.
Is it sure 'fork' is the only culprit? If not, it is better to address it properly because this unnecessary name change is annoying when switching bpf prog from clang to gcc. Like changing the name in this .c here has to make another change to the .c in the prog_tests/ directory.
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
In file included from
progs/bench_local_storage_create.c:6:progs/bench_local_storage_create.c:43:14: error: conflicting types for built-in function 'fork'; expected 'int(void)' [-Werror=builtin-declaration-mismatch]
43 | int BPF_PROG(fork, structtask_struct*parent, structtask_struct*child)| ^~~~I haven't been able to find this documented anywhere however.
Re: [PATCH v3 bpf-next 5/5] selftests/bpf: Add bench for task storage creation
Quoted 66 lines · Martin KaFai Lau
Quoted 64 lines · James Hilliard
Quoted 36 lines · Martin KaFai Lau
Quoted 22 lines · James Hilliard
Quoted 22 lines
tools/testing/selftests/bpf/progs/bench_local_storage_create.c +7
-0@@ -22,6 +22,13 @@ struct { __type(value, struct storage); } sk_storage_map SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_TASK_STORAGE); + __uint(map_flags, BPF_F_NO_PREALLOC); + __type(key, int); + __type(value, struct storage); +} task_storage_map SEC(".maps"); + SEC("raw_tp/kmalloc") int BPF_PROG(kmalloc, unsigned long call_site, const void *ptr, size_t bytes_req, size_t bytes_alloc, gfp_t gfp_flags,@@ -32,6 +39,24 @@ int BPF_PROG(kmalloc, unsigned long call_site, const void *ptr,return 0 ; }Quoted 4 lines
tools/testing/selftests/bpf/progs/bench_local_storage_create.c int BPF_PROG(kmalloc, unsigned long call_site, const void *ptr, · line 43
Quoted 2 lines
tools/testing/selftests/bpf/progs/bench_local_storage_create.c +2
-0+SEC("tp_btf/sched_process_fork") +int BPF_PROG (fork ,struct task_struct *parent,struct task_struct *child)Apparently fork is a built-in function in bpf-gcc:
It is also failing in a plain C program
#> gcc -Werror=builtin-declaration-mismatch -o test
test.ctest.c:14:35: error: conflicting types for built-in function ‘fork’; expected ‘int(void)’ [-Werror=builtin-declaration-mismatch]14 | int __attribute__((__noinline__)) fork(long x, long y) | ^~~~Quoted 7 lines
cc1: some warnings being treated as errors
#> clang -o test
test.c
succeedI am not too attached to the name but it seems something should be addressed in the gcc instead.
Hmm, so it looks like it's marked as a builtin here:
https://github.com/gcc-mirror/gcc/blob/releases/gcc-12.1.0/gcc/builtins.def#L875The macro for that is here:
https://github.com/gcc-mirror/gcc/blob/releases/gcc-12.1.0/gcc/builtins.def#L104-L111Which has this comment:
/* LikeDEF_LIB_BUILTIN, except that the function is not one that is specified by ANSI/ISO C. So, when we're being fully conformant we ignore the version of these builtins that does not begin with__builtin. */Looks like this builtin was originally added here:
https://github.com/gcc-mirror/gcc/commit/d1c38823924506d389ca58d02926ace21bdf82faBased on this issue it looks like fork is treated as a builtin for libgcov support:
https://gcc.gnu.org/bugzilla//show_bug.cgi?id=82457
Quoted 5 lines
Quoted 2 lines
So from my understanding fork is a gcc builtin when building with -std=gnu11 but is not a builtin when building with -std=c11.
That sounds like there is a knob to turn this behavior on and off. Do the same for the bpf target?
I don't think we want to have to do that.
Quoted 5 lines
Quoted 5 lines
So it looks like fork is translated to
__gcov_forkwhen -std=gnu* is set which is why we get this error.
Quoted 8 lines
Quoted 3 lines
As this appears to be intended behavior for gcc I think the best option is to just rename the function so that we don't run into issues when building with gnu extensions like -std=gnu11.
Is it sure 'fork' is the only culprit? If not, it is better to address it properly because this unnecessary name change is annoying when switching bpf prog from clang to gcc. Like changing the name in this .c here has to make another change to the .c in the prog_tests/ directory.
We've fixed a similar issue in the past by renaming to avoid a conflict with the builtin:
https://github.com/torvalds/linux/commit/ab0350c743d5c93fd88742f02b3dff12168ab435
Quoted 15 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
In file included from
progs/bench_local_storage_create.c:6:progs/bench_local_storage_create.c:43:14: error: conflicting types for built-in function 'fork'; expected 'int(void)' [-Werror=builtin-declaration-mismatch]
43 | int BPF_PROG(fork, structtask_struct*parent, structtask_struct*child)| ^~~~I haven't been able to find this documented anywhere however.
Martin KaFai Lau
reviewerRe: [PATCH v3 bpf-next 5/5] selftests/bpf: Add bench for task storage creation
Quoted 6 lines · James Hilliard
Quoted 6 lines
Quoted 6 lines
So it looks like fork is translated to
__gcov_forkwhen -std=gnu* is set which is why we get this error.As this appears to be intended behavior for gcc I think the best option is to just rename the function so that we don't run into issues when building with gnu extensions like -std=gnu11.
Quoted 8 lines
Quoted 4 lines
Is it sure 'fork' is the only culprit? If not, it is better to address it properly because this unnecessary name change is annoying when switching bpf prog from clang to gcc. Like changing the name in this .c here has to make another change to the .c in the prog_tests/ directory.
We've fixed a similar issue in the past by renaming to avoid a conflict with the builtin:
https://github.com/torvalds/linux/commit/ab0350c743d5c93fd88742f02b3dff12168ab435
Fair enough. Please post a patch for the name change.
Re: [PATCH v3 bpf-next 5/5] selftests/bpf: Add bench for task storage creation
Quoted 12 lines · Martin KaFai Lau
Quoted 10 lines · James Hilliard
Quoted 10 lines
Quoted 6 lines
So it looks like fork is translated to
__gcov_forkwhen -std=gnu* is set which is why we get this error.As this appears to be intended behavior for gcc I think the best option is to just rename the function so that we don't run into issues when building with gnu extensions like -std=gnu11.
Is it sure 'fork' is the only culprit? If not, it is better to address it properly because this unnecessary name change is annoying when switching bpf prog from clang to gcc. Like changing the name in this .c here has to make another change to the .c in the prog_tests/ directory.
Quoted 6 lines
Quoted 4 lines
We've fixed a similar issue in the past by renaming to avoid a conflict with the builtin:
https://github.com/torvalds/linux/commit/ab0350c743d5c93fd88742f02b3dff12168ab435Fair enough. Please post a patch for the name change.
Any suggestions/preferences on what name I should use instead?
Martin KaFai Lau
reviewerRe: [PATCH v3 bpf-next 5/5] selftests/bpf: Add bench for task storage creation
Quoted 13 lines · James Hilliard
Quoted 12 lines · Martin KaFai Lau
Quoted 10 lines · James Hilliard
Quoted 10 lines
Quoted 6 lines
So it looks like fork is translated to
__gcov_forkwhen -std=gnu* is set which is why we get this error.As this appears to be intended behavior for gcc I think the best option is to just rename the function so that we don't run into issues when building with gnu extensions like -std=gnu11.
Is it sure 'fork' is the only culprit? If not, it is better to address it properly because this unnecessary name change is annoying when switching bpf prog from clang to gcc. Like changing the name in this .c here has to make another change to the .c in the prog_tests/ directory.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
We've fixed a similar issue in the past by renaming to avoid a conflict with the builtin:
https://github.com/torvalds/linux/commit/ab0350c743d5c93fd88742f02b3dff12168ab435Fair enough. Please post a patch for the name change.
Any suggestions/preferences on what name I should use instead?
May be 'sched_process_fork'?
that will make it the same as the tracepoint's name.
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Quoted 10 lines · On Wed, 22 Mar 2023 14:52:41 -0700 you wrote:
From: Martin KaFai Lau <[email protected]>
This set is a continuation of the effort in using
bpf_mem_cache_alloc/free inbpf_local_storage[1]Major change is only using
bpf_mem_allocfor task and cgrp storage while sk and inode stay with kzalloc/kfree. The details is in patch 2.[...]
Here is the summary with links:
- [v3,bpf-next,1/5] bpf: Add a few bpf mem allocator functions
https://git.kernel.org/bpf/bpf-next/c/e65a5c6edbc6 - [v3,bpf-next,2/5] bpf: Use bpf_mem_cache_alloc/free in bpf_local_storage_elem https://git.kernel.org/bpf/bpf-next/c/08a7ce384e33 - [v3,bpf-next,3/5] bpf: Use bpf_mem_cache_alloc/free for bpf_local_storage https://git.kernel.org/bpf/bpf-next/c/6ae9d5e99e1d - [v3,bpf-next,4/5] selftests/bpf: Test task storage when local_storage->smap is NULL https://git.kernel.org/bpf/bpf-next/c/d8db84d71c0e - [v3,bpf-next,5/5] selftests/bpf: Add bench for task storage creation https://git.kernel.org/bpf/bpf-next/c/cbe9d93d58b1
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html