bpf: Reduce memory usage for bpf_global_percpu_ma
Subsystems (4)
Yonghong Song
reviewerv2 cover bpf: Reduce memory usage for bpf_global_percpu_ma
Currently when a bpf program intends to allocate memory for percpu kptr, the verifier will call bpf_mem_alloc_init() to prefill all supported unit sizes and this caused memory consumption very big for large number of cpus. For example, for 128-cpu system, the total memory consumption with initial prefill is ~175MB. Things will become worse for systems with even more cpus.
Patch 1 is a preparatory patch.
Patch 2 avoids unnecessary extra percpu memory allocation. Patch 3 addresses memory consumption issue by avoiding to prefill with all unit sizes, i.e. only prefilling with user specified size. Patch 4 further reduces memory consumption by limiting the number of prefill entries for percpu memory allocation. Patch 5 rejects percpu memory allocation with bpf_global_percpu_ma when allocation size is greater than 512 bytes.
Patch 6 fixed one test due to Patch 5 and added one test to show the verification failure log message.
Changelogs:
v1 -> v2:
. Avoid unnecessary extra percpu memory allocation.
. Add a separate function to do bpf_global_percpu_ma initialization . promote.
. Promote function static 'sizes' array to file static. . Add comments to explain to refill only one item for percpu alloc.
Yonghong Song (6):
bpf: Refactor to have a memalloc cache destroying function
bpf: Avoid unnecessary extra percpu memory allocation
bpf: Allow per unit prefill for non-fix-size percpu memory allocator
bpf: Refill only one percpu element in memalloc
bpf: Limit up to 512 bytes for bpf_global_percpu_ma allocation selftests/bpf: Cope with 512 bytes limit with bpf_global_percpu_ma
include/linux/bpf.h | 2 +- include/linux/bpf_mem_alloc.h | 7 ++ kernel/bpf/core.c | 8 +- kernel/bpf/memalloc.c | 100 +++++++++++++++--- kernel/bpf/verifier.c | 36 ++++--- .../selftests/bpf/progs/percpu_alloc_fail.c | 18 ++++ .../testing/selftests/bpf/progs/test_bpf_ma.c | 9 -- 7 files changed, 138 insertions(+), 42 deletions(-)
Signature · 1 line
2.34.1
Yonghong Song
reviewerPATCH v2 1/6 bpf: Refactor to have a memalloc cache destroying function
The function, named as bpf_mem_alloc_destroy_cache(), will be used in the subsequent patch.
--- kernel/bpf/memalloc.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
kernel/bpf/memalloc.c +9 -6
@@ -618,6 +618,13 @@ static void drain_mem_cache(struct bpf_mem_cache *c)free_all (llist_del_all (&c->waiting_for_gp), percpu); } +static void bpf_mem_alloc_destroy_cache (struct bpf_mem_cache *c) +{ +WRITE_ONCE (c->draining, true); +irq_work_sync (&c->refill_work); +drain_mem_cache (c); +} +static void check_mem_cache (struct bpf_mem_cache *c) {WARN_ON_ONCE (!llist_empty (&c->free_by_rcu_ttrace));
@@ -723,9 +730,7 @@ void bpf_mem_alloc_destroy(struct bpf_mem_alloc *ma) rcu_in_progress =0 ;for_each_possible_cpu (cpu) { c =per_cpu_ptr (ma->cache, cpu); -WRITE_ONCE (c->draining, true); -irq_work_sync (&c->refill_work); -drain_mem_cache (c); +bpf_mem_alloc_destroy_cache (c); rcu_in_progress +=atomic_read (&c->call_rcu_ttrace_in_progress); rcu_in_progress +=atomic_read (&c->call_rcu_in_progress); }
@@ -740,9 +745,7 @@ void bpf_mem_alloc_destroy(struct bpf_mem_alloc *ma) cc =per_cpu_ptr (ma->caches, cpu);for (i =0 ; i <NUM_CACHES ; i++) { c = &cc->cache[i]; -WRITE_ONCE (c->draining, true); -irq_work_sync (&c->refill_work); -drain_mem_cache (c); +bpf_mem_alloc_destroy_cache (c); rcu_in_progress +=atomic_read (&c->call_rcu_ttrace_in_progress); rcu_in_progress +=atomic_read (&c->call_rcu_in_progress); }
Signature · 1 line
2.34.1
Yonghong Song
reviewerPATCH v2 2/6 bpf: Avoid unnecessary extra percpu memory allocation
In reply to Yonghong SongCurrently, for percpu memory allocation, say if the user requests allocation size to be 32 bytes, the actually calculated size will be 40 bytes and it further rounds to 64 bytes, and eventually 64 bytes are allocated, wasting 32-byte memory.
Change bpf_mem_alloc() to calculate the cache index based on the user-provided allocation size so unnecessary extra memory can be avoided.
--- kernel/bpf/memalloc.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/bpf/memalloc.c +3 -1
@@ -874,7 +874,9 @@ void notrace *bpf_mem_alloc(struct bpf_mem_alloc *ma, size_t size)if (!size)return ZERO_SIZE_PTR ; - idx =bpf_mem_cache_idx (size +LLIST_NODE_SZ ); +if (!ma->percpu) + size +=LLIST_NODE_SZ ; + idx =bpf_mem_cache_idx (size);if (idx <0 )return NULL ;
Signature · 1 line
2.34.1
Re: [PATCH bpf-next v2 2/6] bpf: Avoid unnecessary extra percpu memory allocation
Quoted 6 lines · Yonghong Song
Currently, for percpu memory allocation, say if the user requests allocation size to be 32 bytes, the actually calculated size will be 40 bytes and it further rounds to 64 bytes, and eventually 64 bytes are allocated, wasting 32-byte memory.
Quoted 6 lines
Change
bpf_mem_alloc()to calculate the cache index based on the user-provided allocation size so unnecessary extra memory can be avoided.Suggested-by Hou Tao <[email protected]>Signed-off-by Yonghong Song <[email protected]>
Yonghong Song
reviewerPATCH v2 3/6 bpf: Allow per unit prefill for non-fix-size percpu memory allocator
In reply to Yonghong SongCommit 41a5db8d8161 ("Add support for non-fix-size percpu mem allocation") added support for non-fix-size percpu memory allocation.
Such allocation will allocate percpu memory for all buckets on all cpus and the memory consumption is in the order to quadratic. For example, let us say, 4 cpus, unit size 16 bytes, so each cpu has 16 * 4 = 64 bytes, with 4 cpus, total will be 64 * 4 = 256 bytes. Then let us say, 8 cpus with the same unit size, each cpu
has 16 * 8 = 128 bytes, with 8 cpus, total will be 128 * 8 = 1024 bytes. So if the number of cpus doubles, the number of memory consumption will be 4 times. So for a system with large number of cpus, the memory consumption goes up quickly with quadratic order.
For example, for 4KB percpu allocation, 128 cpus. The total memory consumption will 4KB * 128 * 128 = 64MB. Things will become worse if the number of cpus is bigger (e.g., 512, 1024, etc.)
In Commit 41a5db8d8161, the non-fix-size percpu memory allocation is done in boot time, so for system with large number of cpus, the initial percpu memory consumption is very visible. For example, for 128 cpu system, the total percpu memory allocation will be at least (16 + 32 + 64 + 96 + 128 + 196 + 256 + 512 + 1024 + 2048 + 4096)
* 128 * 128 = ~138MB.
which is pretty big. It will be even bigger for larger number of cpus.
Note that the current prefill also allocates 4 entries if the unit size is less than 256. So on top of 138MB memory consumption, this will add more consumption with
3 * (16 + 32 + 64 + 96 + 128 + 196 + 256) * 128 * 128 = ~38MB. Next patch will try to reduce this memory consumption.
Later on, Commit 1fda5bb66ad8 ("bpf: Do not allocate percpu memory at init stage") moved the non-fix-size percpu memory allocation to bpf verificaiton stage. Once a particular bpf_percpu_obj_new() is called by bpf program, the memory allocator will try to fill in the cache with all sizes, causing the same amount of percpu memory consumption as in the boot stage.
To reduce the initial percpu memory consumption for non-fix-size percpu memory allocation, instead of filling the cache with all supported allocation sizes, this patch intends to fill the cache only for the requested size. As typically users will not use large percpu data structure, this can save memory significantly. For example, the allocation size is 64 bytes with 128 cpus. Then total percpu memory amount will be 64 * 128 * 128 = 1MB, much less than previous 138MB.
--- include/linux/bpf.h | 2 +- include/linux/bpf_mem_alloc.h | 7 ++++ kernel/bpf/core.c | 8 +++-- kernel/bpf/memalloc.c | 68 ++++++++++++++++++++++++++++++++++- kernel/bpf/verifier.c | 28 ++++++--------- 5 files changed, 91 insertions(+), 22 deletions(-)
include/linux/bpf.h +1 -1
@@ -60,7 +60,7 @@ extern struct idr btf_idr;extern spinlock_t btf_idr_lock;extern struct kobject *btf_kobj;extern struct bpf_mem_alloc bpf_global_ma, bpf_global_percpu_ma; -extern bool bpf_global_ma_set; +extern bool bpf_global_ma_set, bpf_global_percpu_ma_set;typedef u64 (*bpf_callback_t )(u64 ,u64 ,u64 ,u64 ,u64 ); typedef int (*bpf_iter_init_seq_priv_t)(void *private_data,
include/linux/bpf_mem_alloc.h +7 -0
@@ -21,8 +21,15 @@ struct bpf_mem_alloc {* 'size = 0' is for bpf_mem_alloc which manages many fixed-size objects. * Alloc and free are done with bpf_mem_{alloc,free}() and the size of * the returned object is given by the size argument of bpf_mem_alloc(). +* If percpu equals true, error will be returned in order to avoid +* large memory consumption and the below bpf_mem_alloc_percpu_unit_init() +* should be used to do on-demand per-cpu allocation for each size. */ int bpf_mem_alloc_init (struct bpf_mem_alloc *ma,int size,bool percpu); +/* Initialize a non-fix-size percpu memory allocator */ +int bpf_mem_alloc_percpu_init (struct bpf_mem_alloc *ma); +/* The percpu allocation with a specific unit size. */ +int bpf_mem_alloc_percpu_unit_init (struct bpf_mem_alloc *ma,int size);void bpf_mem_alloc_destroy (struct bpf_mem_alloc *ma);/* kmalloc/kfree equivalent: */
kernel/bpf/core.c +5 -3
@@ -64,8 +64,8 @@#define OFF insn->off#define IMM insn->imm -struct bpf_mem_alloc bpf_global_ma; -bool bpf_global_ma_set; +struct bpf_mem_alloc bpf_global_ma, bpf_global_percpu_ma; +bool bpf_global_ma_set, bpf_global_percpu_ma_set;/* No hurry in this branch *
@@ -2938,7 +2938,9 @@ static int __init bpf_global_ma_init(void) ret =bpf_mem_alloc_init (&bpf_global_ma,0 , false); bpf_global_ma_set = !ret; - return ret; + ret =bpf_mem_alloc_percpu_init (&bpf_global_percpu_ma); + bpf_global_percpu_ma_set = !ret; + return !bpf_global_ma_set || !bpf_global_percpu_ma_set; }late_initcall (bpf_global_ma_init);#endif
kernel/bpf/memalloc.c +67 -1
91 lines not on this page · raw message
kernel/bpf/verifier.c +11 -17
@@ -42,9 +42,6 @@ static const struct bpf_verifier_ops * const bpf_verifier_ops[] = {#undef BPF_LINK_TYPE }; -struct bpf_mem_alloc bpf_global_percpu_ma; -static bool bpf_global_percpu_ma_set; -/* bpf_check() is a static code analyzer that walks eBPF program * instruction by instruction and updates register/stack state. * All paths of conditional branches are analyzed until 'bpf_exit' insn.
@@ -12062,20 +12059,6 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,if (meta.func_id == special_kfunc_list[KF_bpf_obj_new_impl] && !bpf_global_ma_set)return -ENOMEM ; -if (meta.func_id == special_kfunc_list[KF_bpf_percpu_obj_new_impl]) { -if (!bpf_global_percpu_ma_set) { -mutex_lock (&bpf_percpu_ma_lock); -if (!bpf_global_percpu_ma_set) { - err =bpf_mem_alloc_init (&bpf_global_percpu_ma,0 , true); -if (!err) - bpf_global_percpu_ma_set = true; - } -mutex_unlock (&bpf_percpu_ma_lock); -if (err) -return err; - } - } -if (((u64 )(u32 )meta.arg_constant.value) != meta.arg_constant.value) {verbose (env,"local type ID argument must be in range [0, U32_MAX]\n" );return -EINVAL ;
@@ -12096,6 +12079,17 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,return -EINVAL ; } +if (meta.func_id == special_kfunc_list[KF_bpf_percpu_obj_new_impl]) { +if (!bpf_global_percpu_ma_set) +return -ENOMEM ; + +mutex_lock (&bpf_percpu_ma_lock); + err =bpf_mem_alloc_percpu_unit_init (&bpf_global_percpu_ma, ret_t->size); +mutex_unlock (&bpf_percpu_ma_lock); +if (err) +return err; + } + struct_meta =btf_find_struct_meta (ret_btf, ret_btf_id);if (meta.func_id == special_kfunc_list[KF_bpf_percpu_obj_new_impl]) {if (!__btf_type_is_scalar_struct (env, ret_btf, ret_t,0 )) {
Signature · 1 line
2.34.1
Yonghong Song
reviewerRe: [PATCH bpf-next v2 3/6] bpf: Allow per unit prefill for non-fix-size percpu memory allocator
Quoted 152 lines · Yonghong Song
Commit
41a5db8d8161("Add support for non-fix-size percpu mem allocation") added support for non-fix-size percpu memory allocation.
Such allocation will allocate percpu memory for all buckets on all cpus and the memory consumption is in the order to quadratic. For example, let us say, 4 cpus, unit size 16 bytes, so each cpu has 16 * 4 = 64 bytes, with 4 cpus, total will be 64 * 4 = 256 bytes. Then let us say, 8 cpus with the same unit size, each cpu
has 16 * 8 = 128 bytes, with 8 cpus, total will be 128 * 8 = 1024 bytes. So if the number of cpus doubles, the number of memory consumption will be 4 times. So for a system with large number of cpus, the memory consumption goes up quickly with quadratic order.
For example, for 4KB percpu allocation, 128 cpus. The total memory consumption will 4KB * 128 * 128 = 64MB. Things will become worse if the number of cpus is bigger (e.g., 512, 1024, etc.)In Commit
41a5db8d8161, the non-fix-size percpu memory allocation is done in boot time, so for system with large number of cpus, the initial percpu memory consumption is very visible. For example, for 128 cpu system, the total percpu memory allocation will be at least (16 + 32 + 64 + 96 + 128 + 196 + 256 + 512 + 1024 + 2048 + 4096)
* 128 * 128 = ~138MB.
which is pretty big. It will be even bigger for larger number of cpus.Note that the current prefill also allocates 4 entries if the unit size is less than 256. So on top of 138MB memory consumption, this will add more consumption with
3 * (16 + 32 + 64 + 96 + 128 + 196 + 256) * 128 * 128 = ~38MB. Next patch will try to reduce this memory consumption.Later on, Commit
1fda5bb66ad8("bpf: Do not allocate percpu memory at init stage") moved the non-fix-size percpu memory allocation to bpf verificaiton stage. Once a particularbpf_percpu_obj_new()is called by bpf program, the memory allocator will try to fill in the cache with all sizes, causing the same amount of percpu memory consumption as in the boot stage.To reduce the initial percpu memory consumption for non-fix-size percpu memory allocation, instead of filling the cache with all supported allocation sizes, this patch intends to fill the cache only for the requested size. As typically users will not use large percpu data structure, this can save memory significantly. For example, the allocation size is 64 bytes with 128 cpus. Then total percpu memory amount will be 64 * 128 * 128 = 1MB, much less than previous 138MB.
Signed-off-by Yonghong Song <[email protected]>--- include/linux/bpf.h | 2 +- include/linux/bpf_mem_alloc.h | 7 ++++ kernel/bpf/core.c | 8 +++-- kernel/bpf/memalloc.c | 68 ++++++++++++++++++++++++++++++++++- kernel/bpf/verifier.c | 28 ++++++--------- 5 files changed, 91 insertions(+), 22 deletions(-)include/linux/bpf.h +1
-1@@ -60,7 +60,7 @@ extern struct idr btf_idr;extern spinlock_t btf_idr_lock;extern struct kobject *btf_kobj;extern struct bpf_mem_alloc bpf_global_ma, bpf_global_percpu_ma; -extern bool bpf_global_ma_set; +extern bool bpf_global_ma_set, bpf_global_percpu_ma_set;typedef u64 (*bpf_callback_t )(u64 ,u64 ,u64 ,u64 ,u64 ); typedef int (*bpf_iter_init_seq_priv_t)(void *private_data,include/linux/bpf_mem_alloc.h +7
-0@@ -21,8 +21,15 @@ struct bpf_mem_alloc {* 'size = 0' is for bpf_mem_alloc which manages many fixed-size objects. * Alloc and free are done with bpf_mem_{alloc,free}() and the size of * the returned object is given by the size argument of bpf_mem_alloc(). +* If percpu equals true, error will be returned in order to avoid +* large memory consumption and the below bpf_mem_alloc_percpu_unit_init() +* should be used to do on-demand per-cpu allocation for each size. */ int bpf_mem_alloc_init (struct bpf_mem_alloc *ma,int size,bool percpu); +/* Initialize a non-fix-size percpu memory allocator */ +int bpf_mem_alloc_percpu_init (struct bpf_mem_alloc *ma); +/* The percpu allocation with a specific unit size. */ +int bpf_mem_alloc_percpu_unit_init (struct bpf_mem_alloc *ma,int size);void bpf_mem_alloc_destroy (struct bpf_mem_alloc *ma);/* kmalloc/kfree equivalent: */ kernel/bpf/core.c +5
-3@@ -64,8 +64,8 @@#define OFF insn->off#define IMM insn->imm -struct bpf_mem_alloc bpf_global_ma; -bool bpf_global_ma_set; +struct bpf_mem_alloc bpf_global_ma, bpf_global_percpu_ma; +bool bpf_global_ma_set, bpf_global_percpu_ma_set;/* No hurry in this branch * @@ -2938,7 +2938,9 @@ static int __init bpf_global_ma_init(void) ret =bpf_mem_alloc_init (&bpf_global_ma,0 , false); bpf_global_ma_set = !ret; - return ret; + ret =bpf_mem_alloc_percpu_init (&bpf_global_percpu_ma); + bpf_global_percpu_ma_set = !ret; + return !bpf_global_ma_set || !bpf_global_percpu_ma_set; }late_initcall (bpf_global_ma_init);#endif kernel/bpf/memalloc.c +11
-1@@ -121,6 +121,8 @@ struct bpf_mem_caches {struct bpf_mem_cache cache[NUM_CACHES ]; }; +static u16 sizes[NUM_CACHES ] = {96 ,192 ,16 ,32 ,64 ,128 ,256 ,512 ,1024 ,2048 ,4096 }; + static struct llist_node notrace *__llist_del_first(struct llist_head *head) {struct llist_node *entry, *next;@@ -520,12 +522,14 @@ static int check_obj_size(struct bpf_mem_cache *c, unsigned int idx)*/ int bpf_mem_alloc_init (struct bpf_mem_alloc *ma,int size,bool percpu) { -static u16 sizes[NUM_CACHES ] = {96 ,192 ,16 ,32 ,64 ,128 ,256 ,512 ,1024 ,2048 ,4096 };int cpu, i, err, unit_size, percpu_size =0 ;struct bpf_mem_caches *cc,__percpu *pcc;struct bpf_mem_cache *c,__percpu *pc;struct obj_cgroup *objcg =NULL ; +if (percpu && size ==0 ) +return -EINVAL ; +/* room for llist_node and per-cpu pointer */ if (percpu) percpu_size =LLIST_NODE_SZ +sizeof (void *);@@ -625,6 +629,68 @@ static void bpf_mem_alloc_destroy_cache(struct bpf_mem_cache *c)drain_mem_cache (c); } +int bpf_mem_alloc_percpu_init (struct bpf_mem_alloc *ma) +{ +struct bpf_mem_caches __percpu *pcc; + + pcc =__alloc_percpu_gfp (sizeof (struct bpf_mem_caches ),8 ,GFP_KERNEL | __GFP_ZERO); +if (!pcc)
kernel/bpf/memalloc.c static void bpf_mem_alloc_destroy_cache(struct bpf_mem_cache *c) · line 647
Quoted 10 lines
kernel/bpf/memalloc.c +10
-0+return -ENOMEM ; + + ma->caches = pcc; + ma->percpu = true; +return 0 ; +} + +int bpf_mem_alloc_percpu_unit_init (struct bpf_mem_alloc *ma,int size) +{ +static u16 sizes[NUM_CACHES ] = {96 ,192 ,16 ,32 ,64 ,128 ,256 ,512 ,1024 ,2048 ,4096 };
Sorry, a oversight here. The above should be removed. Will fix in the next revision.
Quoted 49 lines
kernel/bpf/memalloc.c +46
-0+int cpu, i, err, unit_size, percpu_size =0 ; +struct bpf_mem_caches *cc,__percpu *pcc; +struct obj_cgroup *objcg =NULL ; +struct bpf_mem_cache *c; + +/* room for llist_node and per-cpu pointer */ + percpu_size =LLIST_NODE_SZ +sizeof (void *); + + i =bpf_mem_cache_idx (size); +if (i <0 ) +return -EINVAL ; + + err =0 ; + pcc = ma->caches; + unit_size = sizes[i]; + +#ifdef CONFIG_MEMCG_KMEM + objcg =get_obj_cgroup_from_current (); +#endif +for_each_possible_cpu (cpu) { + cc =per_cpu_ptr (pcc, cpu); + c = &cc->cache[i]; +if (cpu ==0 && c->unit_size) + goto out; + + c->unit_size = unit_size; + c->objcg = objcg; + c->percpu_size = percpu_size; + c->tgt = c; + +init_refill_work (c); +prefill_mem_cache (c, cpu); + +if (cpu ==0 ) { + err =check_obj_size (c, i); +if (err) { +bpf_mem_alloc_destroy_cache (c); + goto out; + } + } + } + +out: +return err; +} +static void check_mem_cache (struct bpf_mem_cache *c) {WARN_ON_ONCE (!llist_empty (&c->free_by_rcu_ttrace));
[...]
Re: [PATCH bpf-next v2 3/6] bpf: Allow per unit prefill for non-fix-size percpu memory allocator
In reply to Yonghong SongQuoted 23 lines · Yonghong Song
Commit
41a5db8d8161("Add support for non-fix-size percpu mem allocation") added support for non-fix-size percpu memory allocation.
Such allocation will allocate percpu memory for all buckets on all cpus and the memory consumption is in the order to quadratic. For example, let us say, 4 cpus, unit size 16 bytes, so each cpu has 16 * 4 = 64 bytes, with 4 cpus, total will be 64 * 4 = 256 bytes. Then let us say, 8 cpus with the same unit size, each cpu
has 16 * 8 = 128 bytes, with 8 cpus, total will be 128 * 8 = 1024 bytes. So if the number of cpus doubles, the number of memory consumption will be 4 times. So for a system with large number of cpus, the memory consumption goes up quickly with quadratic order.
For example, for 4KB percpu allocation, 128 cpus. The total memory consumption will 4KB * 128 * 128 = 64MB. Things will become worse if the number of cpus is bigger (e.g., 512, 1024, etc.)In Commit
41a5db8d8161, the non-fix-size percpu memory allocation is done in boot time, so for system with large number of cpus, the initial percpu memory consumption is very visible. For example, for 128 cpu system, the total percpu memory allocation will be at least (16 + 32 + 64 + 96 + 128 + 196 + 256 + 512 + 1024 + 2048 + 4096)
* 128 * 128 = ~138MB.
which is pretty big. It will be even bigger for larger number of cpus.
SNIP
Quoted 50 lines
index
bb1223b21308..43e635c67150100644include/linux/bpf_mem_alloc.h +7
-0@@ -21,8 +21,15 @@ struct bpf_mem_alloc {* 'size = 0' is for bpf_mem_alloc which manages many fixed-size objects. * Alloc and free are done with bpf_mem_{alloc,free}() and the size of * the returned object is given by the size argument of bpf_mem_alloc(). +* If percpu equals true, error will be returned in order to avoid +* large memory consumption and the below bpf_mem_alloc_percpu_unit_init() +* should be used to do on-demand per-cpu allocation for each size. */ int bpf_mem_alloc_init (struct bpf_mem_alloc *ma,int size,bool percpu); +/* Initialize a non-fix-size percpu memory allocator */ +int bpf_mem_alloc_percpu_init (struct bpf_mem_alloc *ma); +/* The percpu allocation with a specific unit size. */ +int bpf_mem_alloc_percpu_unit_init (struct bpf_mem_alloc *ma,int size);void bpf_mem_alloc_destroy (struct bpf_mem_alloc *ma);/* kmalloc/kfree equivalent: */ kernel/bpf/core.c +5
-3@@ -64,8 +64,8 @@#define OFF insn->off#define IMM insn->imm -struct bpf_mem_alloc bpf_global_ma; -bool bpf_global_ma_set; +struct bpf_mem_alloc bpf_global_ma, bpf_global_percpu_ma; +bool bpf_global_ma_set, bpf_global_percpu_ma_set;/* No hurry in this branch * @@ -2938,7 +2938,9 @@ static int __init bpf_global_ma_init(void) ret =bpf_mem_alloc_init (&bpf_global_ma,0 , false); bpf_global_ma_set = !ret; - return ret; + ret =bpf_mem_alloc_percpu_init (&bpf_global_percpu_ma); + bpf_global_percpu_ma_set = !ret; + return !bpf_global_ma_set || !bpf_global_percpu_ma_set; }late_initcall (bpf_global_ma_init);#endif kernel/bpf/memalloc.c +0
-0
kernel/bpf/memalloc.c struct bpf_mem_caches { · line 124
Quoted 4 lines
kernel/bpf/memalloc.c +1
-0@@ -121,6 +121,8 @@ struct bpf_mem_caches {struct bpf_mem_cache cache[NUM_CACHES ]; }; +static u16 sizes[NUM_CACHES ] = {96 ,192 ,16 ,32 ,64 ,128 ,256 ,512 ,1024 ,2048 ,4096 };
Is it better to make it being const ?
Quoted 21 lines
kernel/bpf/memalloc.c +4
-1+ static struct llist_node notrace *__llist_del_first(struct llist_head *head) {struct llist_node *entry, *next;@@ -520,12 +522,14 @@ static int check_obj_size(struct bpf_mem_cache *c, unsigned int idx)*/ int bpf_mem_alloc_init (struct bpf_mem_alloc *ma,int size,bool percpu) { -static u16 sizes[NUM_CACHES ] = {96 ,192 ,16 ,32 ,64 ,128 ,256 ,512 ,1024 ,2048 ,4096 };int cpu, i, err, unit_size, percpu_size =0 ;struct bpf_mem_caches *cc,__percpu *pcc;struct bpf_mem_cache *c,__percpu *pc;struct obj_cgroup *objcg =NULL ; +if (percpu && size ==0 ) +return -EINVAL ; +/* room for llist_node and per-cpu pointer */ if (percpu) percpu_size =LLIST_NODE_SZ +sizeof (void *);
kernel/bpf/memalloc.c static void bpf_mem_alloc_destroy_cache(struct bpf_mem_cache *c) · line 638
Quoted 10 lines
kernel/bpf/memalloc.c +7
-0@@ -625,6 +629,68 @@ static void bpf_mem_alloc_destroy_cache(struct bpf_mem_cache *c)drain_mem_cache (c); } +int bpf_mem_alloc_percpu_init (struct bpf_mem_alloc *ma) +{ +struct bpf_mem_caches __percpu *pcc; + + pcc =__alloc_percpu_gfp (sizeof (struct bpf_mem_caches ),8 ,GFP_KERNEL | __GFP_ZERO); +if (!pcc) +return -ENOMEM ;
__GFP_ZERO is not needed. __alloc_percpu_gfp() will zero the returned area by default.
Quoted 36 lines
kernel/bpf/memalloc.c +36
-0+ + ma->caches = pcc; + ma->percpu = true; +return 0 ; +} + +int bpf_mem_alloc_percpu_unit_init (struct bpf_mem_alloc *ma,int size) +{ +static u16 sizes[NUM_CACHES ] = {96 ,192 ,16 ,32 ,64 ,128 ,256 ,512 ,1024 ,2048 ,4096 }; +int cpu, i, err, unit_size, percpu_size =0 ; +struct bpf_mem_caches *cc,__percpu *pcc; +struct obj_cgroup *objcg =NULL ; +struct bpf_mem_cache *c; + +/* room for llist_node and per-cpu pointer */ + percpu_size =LLIST_NODE_SZ +sizeof (void *); + + i =bpf_mem_cache_idx (size); +if (i <0 ) +return -EINVAL ; + + err =0 ; + pcc = ma->caches; + unit_size = sizes[i]; + +#ifdef CONFIG_MEMCG_KMEM + objcg =get_obj_cgroup_from_current (); +#endif +for_each_possible_cpu (cpu) { + cc =per_cpu_ptr (pcc, cpu); + c = &cc->cache[i]; +if (cpu ==0 && c->unit_size) + goto out; + + c->unit_size = unit_size; + c->objcg = objcg;
kernel/bpf/memalloc.c static void bpf_mem_alloc_destroy_cache(struct bpf_mem_cache *c) · line 684
Quoted 10 lines
kernel/bpf/memalloc.c +10
-0+ c->percpu_size = percpu_size; + c->tgt = c; + +init_refill_work (c); +prefill_mem_cache (c, cpu); + +if (cpu ==0 ) { + err =check_obj_size (c, i); +if (err) { +bpf_mem_alloc_destroy_cache (c);
It seems drain_mem_cache() will be enough. Have you considered setting low_watermark as 0 to prevent potential refill in unit_alloc() if the initialization of the current unit fails ?
kernel/bpf/memalloc.c static void bpf_mem_alloc_destroy_cache(struct bpf_mem_cache *c) · line 696
Quoted 13 lines
kernel/bpf/memalloc.c +9
-0+ goto out; + } + } + } + +out: +return err; +} +static void check_mem_cache (struct bpf_mem_cache *c) {WARN_ON_ONCE (!llist_empty (&c->free_by_rcu_ttrace));
.
Yonghong Song
reviewerRe: [PATCH bpf-next v2 3/6] bpf: Allow per unit prefill for non-fix-size percpu memory allocator
Quoted 79 lines · Hou Tao
Quoted 23 lines · Yonghong Song
Commit
41a5db8d8161("Add support for non-fix-size percpu mem allocation") added support for non-fix-size percpu memory allocation.
Such allocation will allocate percpu memory for all buckets on all cpus and the memory consumption is in the order to quadratic. For example, let us say, 4 cpus, unit size 16 bytes, so each cpu has 16 * 4 = 64 bytes, with 4 cpus, total will be 64 * 4 = 256 bytes. Then let us say, 8 cpus with the same unit size, each cpu
has 16 * 8 = 128 bytes, with 8 cpus, total will be 128 * 8 = 1024 bytes. So if the number of cpus doubles, the number of memory consumption will be 4 times. So for a system with large number of cpus, the memory consumption goes up quickly with quadratic order.
For example, for 4KB percpu allocation, 128 cpus. The total memory consumption will 4KB * 128 * 128 = 64MB. Things will become worse if the number of cpus is bigger (e.g., 512, 1024, etc.)In Commit
41a5db8d8161, the non-fix-size percpu memory allocation is done in boot time, so for system with large number of cpus, the initial percpu memory consumption is very visible. For example, for 128 cpu system, the total percpu memory allocation will be at least (16 + 32 + 64 + 96 + 128 + 196 + 256 + 512 + 1024 + 2048 + 4096)
* 128 * 128 = ~138MB.
which is pretty big. It will be even bigger for larger number of cpus.SNIP
Quoted 53 lines
index
bb1223b21308..43e635c67150100644include/linux/bpf_mem_alloc.h +7
-0@@ -21,8 +21,15 @@ struct bpf_mem_alloc {* 'size = 0' is for bpf_mem_alloc which manages many fixed-size objects. * Alloc and free are done with bpf_mem_{alloc,free}() and the size of * the returned object is given by the size argument of bpf_mem_alloc(). +* If percpu equals true, error will be returned in order to avoid +* large memory consumption and the below bpf_mem_alloc_percpu_unit_init() +* should be used to do on-demand per-cpu allocation for each size. */ int bpf_mem_alloc_init (struct bpf_mem_alloc *ma,int size,bool percpu); +/* Initialize a non-fix-size percpu memory allocator */ +int bpf_mem_alloc_percpu_init (struct bpf_mem_alloc *ma); +/* The percpu allocation with a specific unit size. */ +int bpf_mem_alloc_percpu_unit_init (struct bpf_mem_alloc *ma,int size);void bpf_mem_alloc_destroy (struct bpf_mem_alloc *ma);/* kmalloc/kfree equivalent: */ kernel/bpf/core.c +5
-3@@ -64,8 +64,8 @@#define OFF insn->off#define IMM insn->imm -struct bpf_mem_alloc bpf_global_ma; -bool bpf_global_ma_set; +struct bpf_mem_alloc bpf_global_ma, bpf_global_percpu_ma; +bool bpf_global_ma_set, bpf_global_percpu_ma_set;/* No hurry in this branch * @@ -2938,7 +2938,9 @@ static int __init bpf_global_ma_init(void) ret =bpf_mem_alloc_init (&bpf_global_ma,0 , false); bpf_global_ma_set = !ret; - return ret; + ret =bpf_mem_alloc_percpu_init (&bpf_global_percpu_ma); + bpf_global_percpu_ma_set = !ret; + return !bpf_global_ma_set || !bpf_global_percpu_ma_set; }late_initcall (bpf_global_ma_init);#endif kernel/bpf/memalloc.c +0
-0@@ -121,6 +121,8 @@ struct bpf_mem_caches {struct bpf_mem_cache cache[NUM_CACHES ]; };
Quoted 2 lines
kernel/bpf/memalloc.c struct bpf_mem_caches { · line 124
Quoted 1 line
kernel/bpf/memalloc.c +1
-0+static u16 sizes[NUM_CACHES ] = {96 ,192 ,16 ,32 ,64 ,128 ,256 ,512 ,1024 ,2048 ,4096 };Is it better to make it being const ?
Right. We can make it as const.
Quoted 24 lines
Quoted 24 lines
kernel/bpf/memalloc.c +4
-1+ static struct llist_node notrace *__llist_del_first(struct llist_head *head) {struct llist_node *entry, *next;@@ -520,12 +522,14 @@ static int check_obj_size(struct bpf_mem_cache *c, unsigned int idx)*/ int bpf_mem_alloc_init (struct bpf_mem_alloc *ma,int size,bool percpu) { -static u16 sizes[NUM_CACHES ] = {96 ,192 ,16 ,32 ,64 ,128 ,256 ,512 ,1024 ,2048 ,4096 };int cpu, i, err, unit_size, percpu_size =0 ;struct bpf_mem_caches *cc,__percpu *pcc;struct bpf_mem_cache *c,__percpu *pc;struct obj_cgroup *objcg =NULL ; +if (percpu && size ==0 ) +return -EINVAL ; +/* room for llist_node and per-cpu pointer */ if (percpu) percpu_size =LLIST_NODE_SZ +sizeof (void *);@@ -625,6 +629,68 @@ static void bpf_mem_alloc_destroy_cache(struct bpf_mem_cache *c)drain_mem_cache (c); }
Quoted 9 lines
kernel/bpf/memalloc.c static void bpf_mem_alloc_destroy_cache(struct bpf_mem_cache *c) · line 638
Quoted 7 lines
kernel/bpf/memalloc.c +7
-0+int bpf_mem_alloc_percpu_init (struct bpf_mem_alloc *ma) +{ +struct bpf_mem_caches __percpu *pcc; + + pcc =__alloc_percpu_gfp (sizeof (struct bpf_mem_caches ),8 ,GFP_KERNEL | __GFP_ZERO); +if (!pcc) +return -ENOMEM ;
__GFP_ZEROis not needed.__alloc_percpu_gfp()will zero the returned area by default.
Thanks. Checked the comments in __alloc_percpu_gfp() and indeed, the returned buffer has been zeroed.
Quoted 39 lines
Quoted 39 lines
kernel/bpf/memalloc.c +39
-0+ + ma->caches = pcc; + ma->percpu = true; +return 0 ; +} + +int bpf_mem_alloc_percpu_unit_init (struct bpf_mem_alloc *ma,int size) +{ +static u16 sizes[NUM_CACHES ] = {96 ,192 ,16 ,32 ,64 ,128 ,256 ,512 ,1024 ,2048 ,4096 }; +int cpu, i, err, unit_size, percpu_size =0 ; +struct bpf_mem_caches *cc,__percpu *pcc; +struct obj_cgroup *objcg =NULL ; +struct bpf_mem_cache *c; + +/* room for llist_node and per-cpu pointer */ + percpu_size =LLIST_NODE_SZ +sizeof (void *); + + i =bpf_mem_cache_idx (size); +if (i <0 ) +return -EINVAL ; + + err =0 ; + pcc = ma->caches; + unit_size = sizes[i]; + +#ifdef CONFIG_MEMCG_KMEM + objcg =get_obj_cgroup_from_current (); +#endif +for_each_possible_cpu (cpu) { + cc =per_cpu_ptr (pcc, cpu); + c = &cc->cache[i]; +if (cpu ==0 && c->unit_size) + goto out; + + c->unit_size = unit_size; + c->objcg = objcg; + c->percpu_size = percpu_size; + c->tgt = c; +
Quoted 8 lines
kernel/bpf/memalloc.c static void bpf_mem_alloc_destroy_cache(struct bpf_mem_cache *c) · line 684
Quoted 7 lines
kernel/bpf/memalloc.c +7
-0+init_refill_work (c); +prefill_mem_cache (c, cpu); + +if (cpu ==0 ) { + err =check_obj_size (c, i); +if (err) { +bpf_mem_alloc_destroy_cache (c);It seems
drain_mem_cache()will be enough. Have you considered setting
At prefill stage, looks like the following is enough:
free_all(__llist_del_all(&c->free_llist), percpu);
But I agree that drain_mem_cache() is simpler and is easier for future potential code change.
Quoted 2 lines
low_watermarkas 0 to prevent potential refill inunit_alloc()if the initialization of the current unit fails ?
I think it does make sense. For non-fix-size non-percpu prefill, if check_obj_size() failed, the prefill will fail, which include all buckets.
In this case, if it fails for a particular bucket, we should make sure that bucket always return NULL ptr, so setting the low_watermark to 0 does make sense.
Quoted 15 lines
kernel/bpf/memalloc.c static void bpf_mem_alloc_destroy_cache(struct bpf_mem_cache *c) · line 696
Quoted 13 lines
kernel/bpf/memalloc.c +9
-0+ goto out; + } + } + } + +out: +return err; +} +static void check_mem_cache (struct bpf_mem_cache *c) {WARN_ON_ONCE (!llist_empty (&c->free_by_rcu_ttrace));.
Yonghong Song
reviewerRe: [PATCH bpf-next v2 3/6] bpf: Allow per unit prefill for non-fix-size percpu memory allocator
Quoted 179 lines · Yonghong Song
Quoted 70 lines · Hou Tao
Quoted 23 lines · Yonghong Song
Commit
41a5db8d8161("Add support for non-fix-size percpu mem allocation") added support for non-fix-size percpu memory allocation.
Such allocation will allocate percpu memory for all buckets on all cpus and the memory consumption is in the order to quadratic. For example, let us say, 4 cpus, unit size 16 bytes, so each cpu has 16 * 4 = 64 bytes, with 4 cpus, total will be 64 * 4 = 256 bytes. Then let us say, 8 cpus with the same unit size, each cpu
has 16 * 8 = 128 bytes, with 8 cpus, total will be 128 * 8 = 1024 bytes. So if the number of cpus doubles, the number of memory consumption will be 4 times. So for a system with large number of cpus, the memory consumption goes up quickly with quadratic order.
For example, for 4KB percpu allocation, 128 cpus. The total memory consumption will 4KB * 128 * 128 = 64MB. Things will become worse if the number of cpus is bigger (e.g., 512, 1024, etc.)In Commit
41a5db8d8161, the non-fix-size percpu memory allocation is done in boot time, so for system with large number of cpus, the initial percpu memory consumption is very visible. For example, for 128 cpu system, the total percpu memory allocation will be at least (16 + 32 + 64 + 96 + 128 + 196 + 256 + 512 + 1024 + 2048 + 4096)
* 128 * 128 = ~138MB.
which is pretty big. It will be even bigger for larger number of cpus.SNIP
Quoted 44 lines
index
bb1223b21308..43e635c67150100644include/linux/bpf_mem_alloc.h +7
-0@@ -21,8 +21,15 @@ struct bpf_mem_alloc {* 'size = 0' is for bpf_mem_alloc which manages many fixed-size objects. * Alloc and free are done with bpf_mem_{alloc,free}() and the size of * the returned object is given by the size argument of bpf_mem_alloc(). +* If percpu equals true, error will be returned in order to avoid +* large memory consumption and the below bpf_mem_alloc_percpu_unit_init() +* should be used to do on-demand per-cpu allocation for each size. */ int bpf_mem_alloc_init(struct bpf_mem_alloc *ma, int size, bool percpu); +/* Initialize a non-fix-size percpu memory allocator */ +int bpf_mem_alloc_percpu_init (struct bpf_mem_alloc *ma); +/* The percpu allocation with a specific unit size. */ +int bpf_mem_alloc_percpu_unit_init (struct bpf_mem_alloc *ma,int size); void bpf_mem_alloc_destroy(struct bpf_mem_alloc *ma);/* kmalloc/kfree equivalent: */
kernel/bpf/core.c +5
-2@@ -64,8 +64,8 @@ #define OFF insn->off #define IMM insn->imm -struct bpf_mem_alloc bpf_global_ma; -bool bpf_global_ma_set; +struct bpf_mem_alloc bpf_global_ma, bpf_global_percpu_ma; +bool bpf_global_ma_set, bpf_global_percpu_ma_set; /* No hurry in this branch* @@ -2938,7 +2938,9 @@ static int __init bpf_global_ma_init(void) ret = bpf_mem_alloc_init(&bpf_global_ma, 0, false); bpf_global_ma_set = !ret; - return ret; + ret = bpf_mem_alloc_percpu_init(&bpf_global_percpu_ma); + bpf_global_percpu_ma_set = !ret; + return !bpf_global_ma_set || !bpf_global_percpu_ma_set; } late_initcall(bpf_global_ma_init); #endifkernel/bpf/memalloc.c +0
-0Quoted 8 lines
kernel/bpf/memalloc.c struct bpf_mem_caches { · line 123
Quoted 7 lines
index
472158f1fb08..aea4cd07c7b6100644kernel/bpf/memalloc.c +0
-0@@ -121,6 +121,8 @@ struct bpf_mem_caches { struct bpf_mem_cache cache[NUM_CACHES]; }; +static u16 sizes[NUM_CACHES] = {96, 192, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096};Is it better to make it being const ?
Right. We can make it as const.
Quoted 27 lines
Quoted 27 lines
kernel/bpf/memalloc.c +3
-1+ static struct llist_node notrace *__llist_del_first(struct llist_head *head) { struct llist_node *entry, *next;@@ -520,12 +522,14 @@ static int check_obj_size(struct bpf_mem_cache*c, unsigned int idx) */ int bpf_mem_alloc_init(struct bpf_mem_alloc *ma, int size, bool percpu) { - static u16 sizes[NUM_CACHES] = {96, 192, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096}; int cpu, i, err, unit_size, percpu_size = 0; struct bpf_mem_caches *cc, __percpu *pcc; struct bpf_mem_cache *c, __percpu *pc; struct obj_cgroup *objcg = NULL; + if (percpu && size == 0) + return -EINVAL; + /* room for llist_node and per-cpu pointer */if (percpu)
percpu_size=LLIST_NODE_SZ+sizeof(void *);kernel/bpf/memalloc.c +1
-0@@ -625,6 +629,68 @@ static void bpf_mem_alloc_destroy_cache(struct bpf_mem_cache *c) drain_mem_cache(c); } +int bpf_mem_alloc_percpu_init(struct bpf_mem_alloc *ma) +{Quoted 8 lines
kernel/bpf/memalloc.c static void bpf_mem_alloc_destroy_cache(struct bpf_mem_cache *c) · line 638
Quoted 6 lines
kernel/bpf/memalloc.c +5
-0+ struct bpf_mem_caches __percpu *pcc; + + pcc = __alloc_percpu_gfp(sizeof(struct bpf_mem_caches), 8,GFP_KERNEL | __GFP_ZERO); + if (!pcc) + return -ENOMEM;
__GFP_ZEROis not needed.__alloc_percpu_gfp()will zero the returned area by default.Thanks. Checked the comments in
__alloc_percpu_gfp()and indeed, the returned buffer has been zeroed.Quoted 40 lines
Quoted 40 lines
kernel/bpf/memalloc.c +39
-0+ + ma->caches = pcc; + ma->percpu = true; + return 0; +} + +int bpf_mem_alloc_percpu_unit_init (struct bpf_mem_alloc *ma,int size) +{ + static u16 sizes[NUM_CACHES] = {96, 192, 16, 32, 64, 128, 256,512 ,1024 ,2048 ,4096 }; + int cpu, i, err, unit_size, percpu_size = 0; + struct bpf_mem_caches *cc, __percpu *pcc; + struct obj_cgroup *objcg = NULL; + struct bpf_mem_cache *c; + + /* room for llist_node and per-cpu pointer */ + percpu_size = LLIST_NODE_SZ + sizeof(void *); + + i = bpf_mem_cache_idx(size); + if (i < 0) + return -EINVAL; + + err = 0; + pcc = ma->caches; + unit_size = sizes[i]; + +#ifdef CONFIG_MEMCG_KMEM + objcg = get_obj_cgroup_from_current(); +#endif + for_each_possible_cpu(cpu) { + cc = per_cpu_ptr(pcc, cpu); + c = &cc->cache[i]; + if (cpu == 0 && c->unit_size) + goto out; + + c->unit_size = unit_size; + c->objcg = objcg; + c->percpu_size = percpu_size; + c->tgt = c; +Quoted 8 lines
kernel/bpf/memalloc.c static void bpf_mem_alloc_destroy_cache(struct bpf_mem_cache *c) · line 684
Quoted 7 lines
kernel/bpf/memalloc.c +7
-0+ init_refill_work(c); + prefill_mem_cache(c, cpu); + + if (cpu == 0) { + err = check_obj_size(c, i); + if (err) { + bpf_mem_alloc_destroy_cache(c);It seems
drain_mem_cache()will be enough. Have you considered settingAt prefill stage, looks like the following is enough:
free_all(__llist_del_all(&c->free_llist), percpu); But I agree thatdrain_mem_cache()is simpler and is easier for future potential code change.Quoted 2 lines
low_watermarkas 0 to prevent potential refill inunit_alloc()if the initialization of the current unit fails ?
Quoted 7 lines
I think it does make sense. For non-fix-size non-percpu prefill, if
check_obj_size()failed, the prefill will fail, which include all buckets.In this case, if it fails for a particular bucket, we should make sure that bucket always return NULL ptr, so setting the
low_watermarkto 0 does make sense.
Thinking again. If the initialization of the current unit failed, the verification will fail and the corresponding bpf program will not be able to do memory alloc, so we should be fine.
But it is totally possible that some prog later may call bpf_mem_alloc_percpu_unit_init() again with the same size/bucket. So we should simply reset bpf_mem_cache to 0 during the previous failed bpf_mem_alloc_percpu_unit_init() call. Is it possible that check_obj_size() may initially returns an error but sometime later something in
the kernel changed and the check_obj_size() with the same size could return true?
Quoted 17 lines
Quoted 16 lines
Quoted 10 lines
kernel/bpf/memalloc.c +9
-0+ goto out; + } + } + } + +out: + return err; +} +Quoted 4 lines
static void
check_mem_cache(struct bpf_mem_cache *c){ WARN_ON_ONCE(!llist_empty(&c->free_by_rcu_ttrace));.
Re: [PATCH bpf-next v2 3/6] bpf: Allow per unit prefill for non-fix-size percpu memory allocator
Hi,
Quoted 23 lines · Yonghong Song
Quoted 21 lines · Yonghong Song
Quoted 19 lines · Hou Tao
Quoted 17 lines · Yonghong Song
Commit
41a5db8d8161("Add support for non-fix-size percpu mem allocation")
added support for non-fix-size percpu memory allocation. Such allocation will allocate percpu memory for all buckets on all cpus and the memory consumption is in the order to quadratic. For example, let us say, 4 cpus, unit size 16 bytes, so each cpu has 16 * 4 = 64 bytes, with 4 cpus, total will be 64 * 4 = 256 bytes.
Then let us say, 8 cpus with the same unit size, each cpu has 16 * 8 = 128 bytes, with 8 cpus, total will be 128 * 8 = 1024 bytes.
So if the number of cpus doubles, the number of memory consumption will be 4 times. So for a system with large number of cpus, the memory consumption goes up quickly with quadratic order. For example, for 4KB percpu allocation, 128 cpus. The total memory consumption will 4KB * 128 * 128 = 64MB. Things will become worse if the number of cpus is bigger (e.g., 512, 1024, etc.)
SNIP
Quoted 44 lines
Quoted 31 lines
Quoted 14 lines
Quoted 14 lines
kernel/bpf/memalloc.c +14
-0+#ifdef CONFIG_MEMCG_KMEM + objcg = get_obj_cgroup_from_current(); +#endif + for_each_possible_cpu(cpu) { + cc = per_cpu_ptr(pcc, cpu); + c = &cc->cache[i]; + if (cpu == 0 && c->unit_size) + goto out; + + c->unit_size = unit_size; + c->objcg = objcg; + c->percpu_size = percpu_size; + c->tgt = c; +Quoted 8 lines
Quoted 7 lines
+
init_refill_work(c);
+prefill_mem_cache(c, cpu);
+
+ if (cpu == 0) {
+ err =check_obj_size(c, i);
+ if (err) {
+bpf_mem_alloc_destroy_cache(c);It seems
drain_mem_cache()will be enough. Have you considered settingAt prefill stage, looks like the following is enough:
free_all(__llist_del_all(&c->free_llist), percpu); But I agree thatdrain_mem_cache()is simpler and is easier for future potential code change.Quoted 2 lines
low_watermarkas 0 to prevent potential refill inunit_alloc()if the initialization of the current unit fails ?Quoted 7 lines
I think it does make sense. For non-fix-size non-percpu prefill, if
check_obj_size()failed, the prefill will fail, which include all buckets.In this case, if it fails for a particular bucket, we should make sure that bucket always return NULL ptr, so setting the
low_watermarkto 0 does make sense.Thinking again. If the initialization of the current unit failed, the verification will fail and the corresponding bpf program will not be able to do memory alloc, so we should be fine.
Quoted 8 lines
But it is totally possible that some prog later may call
bpf_mem_alloc_percpu_unit_init()again with the same size/bucket. So we should simply resetbpf_mem_cacheto 0 during the previous failedbpf_mem_alloc_percpu_unit_init()call. Is it possible thatcheck_obj_size()may initially returns an error but sometime later something in
the kernel changed and thecheck_obj_size()with the same size could return true?
Resetting bpf_mem_cache as 0 is much simpler and easier to understand than resetting low_watermark as 0. For per-cpu allocation, the return value of pcpu_alloc_size() is stable and I don't think it will change like ksize() does(), so it is not possible that the previous check_obj_size() failed, but the new check_obj_size() for the same unit_size succeeds.
Quoted 19 lines
Quoted 19 lines
Quoted 18 lines
Quoted 12 lines
+ goto out;
+ }
+ }
+ }kernel/bpf/memalloc.c +5
-0+ +out: + return err; +} +Quoted 4 lines
static void
check_mem_cache(struct bpf_mem_cache *c){ WARN_ON_ONCE(!llist_empty(&c->free_by_rcu_ttrace));.
Yonghong Song
reviewerRe: [PATCH bpf-next v2 3/6] bpf: Allow per unit prefill for non-fix-size percpu memory allocator
Quoted 73 lines · Hou Tao
Hi,
Quoted 20 lines · Yonghong Song
Quoted 19 lines · Yonghong Song
Quoted 18 lines · Hou Tao
Quoted 17 lines · Yonghong Song
Commit
41a5db8d8161("Add support for non-fix-size percpu mem allocation")
added support for non-fix-size percpu memory allocation. Such allocation will allocate percpu memory for all buckets on all cpus and the memory consumption is in the order to quadratic. For example, let us say, 4 cpus, unit size 16 bytes, so each cpu has 16 * 4 = 64 bytes, with 4 cpus, total will be 64 * 4 = 256 bytes.
Then let us say, 8 cpus with the same unit size, each cpu has 16 * 8 = 128 bytes, with 8 cpus, total will be 128 * 8 = 1024 bytes.
So if the number of cpus doubles, the number of memory consumption will be 4 times. So for a system with large number of cpus, the memory consumption goes up quickly with quadratic order. For example, for 4KB percpu allocation, 128 cpus. The total memory consumption will 4KB * 128 * 128 = 64MB. Things will become worse if the number of cpus is bigger (e.g., 512, 1024, etc.)SNIP
Quoted 41 lines
Quoted 29 lines
Quoted 14 lines
Quoted 14 lines
kernel/bpf/memalloc.c +14
-0+#ifdef CONFIG_MEMCG_KMEM + objcg = get_obj_cgroup_from_current(); +#endif + for_each_possible_cpu(cpu) { + cc = per_cpu_ptr(pcc, cpu); + c = &cc->cache[i]; + if (cpu == 0 && c->unit_size) + goto out; + + c->unit_size = unit_size; + c->objcg = objcg; + c->percpu_size = percpu_size; + c->tgt = c; +Quoted 8 lines
Quoted 7 lines
+
init_refill_work(c);
+prefill_mem_cache(c, cpu);
+
+ if (cpu == 0) {
+ err =check_obj_size(c, i);
+ if (err) {
+bpf_mem_alloc_destroy_cache(c);It seems
drain_mem_cache()will be enough. Have you considered settingAt prefill stage, looks like the following is enough:
free_all(__llist_del_all(&c->free_llist), percpu);
But I agree thatdrain_mem_cache()is simpler and is easier for future potential code change.Quoted 2 lines
low_watermarkas 0 to prevent potential refill inunit_alloc()if the initialization of the current unit fails ?Quoted 7 lines
I think it does make sense. For non-fix-size non-percpu prefill, if
check_obj_size()failed, the prefill will fail, which include all buckets.In this case, if it fails for a particular bucket, we should make sure that bucket always return NULL ptr, so setting the
low_watermarkto 0 does make sense.Thinking again. If the initialization of the current unit failed, the verification will fail and the corresponding bpf program will not be able to do memory alloc, so we should be fine.
Quoted 8 lines
But it is totally possible that some prog later may call
bpf_mem_alloc_percpu_unit_init()again with the same size/bucket. So we should simply resetbpf_mem_cacheto 0 during the previous failedbpf_mem_alloc_percpu_unit_init()call. Is it possible thatcheck_obj_size()may initially returns an error but sometime later something in
the kernel changed and thecheck_obj_size()with the same size could return true?
Quoted 6 lines
Resetting
bpf_mem_cacheas 0 is much simpler and easier to understand than resettinglow_watermarkas 0. For per-cpu allocation, the return value ofpcpu_alloc_size()is stable and I don't think it will change likeksize()does(), so it is not possible that the previouscheck_obj_size()failed, but the newcheck_obj_size()for the sameunit_sizesucceeds.
Thanks for clarification. Let me just do resetting bpf_mem_cache to 0 then.
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 13 lines
+ goto out;
+ }
+ }
+ }kernel/bpf/memalloc.c +5
-0+ +out: + return err; +} + static void check_mem_cache(struct bpf_mem_cache *c) {Quoted 2 lines
WARN_ON_ONCE(!llist_empty(&c->free_by_rcu_ttrace));
.
Yonghong Song
reviewerPATCH v2 4/6 bpf: Refill only one percpu element in memalloc
In reply to Yonghong SongTypically for percpu map element or data structure, once allocated, most operations are lookup or in-place update. Deletion are really rare. Currently, for percpu data strcture, 4 elements will be refilled if the size is <= 256. Let us just do with one element for percpu data. For example, for size 256 and 128 cpus, the potential saving will be 3 * 256 * 128 * 128 = 12MB.
--- kernel/bpf/memalloc.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
kernel/bpf/memalloc.c +9 -4
@@ -485,11 +485,16 @@ static void init_refill_work(struct bpf_mem_cache *c)static void prefill_mem_cache (struct bpf_mem_cache *c,int cpu) { -/* To avoid consuming memory assume that 1st run of bpf -* prog won't be doing more than 4 map_update_elem from -* irq disabled region +int cnt =1 ; + +/* To avoid consuming memory, for non-percpu allocation, assume that +* 1st run of bpf prog won't be doing more than 4 map_update_elem from +* irq disabled region if unit size is less than or equal to 256. +* For all other cases, let us just do one allocation. */ -alloc_bulk (c, c->unit_size <=256 ?4 :1 ,cpu_to_node (cpu), false); +if (!c->percpu_size && c->unit_size <=256 ) + cnt =4 ; +alloc_bulk (c, cnt,cpu_to_node (cpu), false); }static int check_obj_size (struct bpf_mem_cache *c,unsigned int idx)
Signature · 1 line
2.34.1
Yonghong Song
reviewerPATCH v2 5/6 bpf: Limit up to 512 bytes for bpf_global_percpu_ma allocation
In reply to Yonghong SongFor percpu data structure allocation with bpf_global_percpu_ma, the maximum data size is 4K. But for a system with large number of cpus, bigger data size (e.g., 2K, 4K) might consume a lot of memory. For example, the percpu memory consumption with unit size 2K and 1024 cpus will be 2K * 1K * 1k = 2GB memory.
We should discourage such usage. Let us limit the maximum data size to be 512 for bpf_global_percpu_ma allocation.
--- kernel/bpf/verifier.c | 8 ++++++++ 1 file changed, 8 insertions(+)
kernel/bpf/verifier.c +8 -0
@@ -192,6 +192,8 @@ struct bpf_verifier_stack_elem { POISON_POINTER_DELTA))#define BPF_MAP_PTR (X ) ((struct bpf_map *)((X) & ~BPF_MAP_PTR_UNPRIV)) +#define BPF_GLOBAL_PERCPU_MA_MAX_SIZE 512 +static int acquire_reference_state (struct bpf_verifier_env *env,int insn_idx);static int release_reference (struct bpf_verifier_env *env,int ref_obj_id);static void invalidate_non_owning_refs (struct bpf_verifier_env *env);
@@ -12083,6 +12085,12 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,if (!bpf_global_percpu_ma_set)return -ENOMEM ; +if (ret_t->size >BPF_GLOBAL_PERCPU_MA_MAX_SIZE ) { +verbose (env,"bpf_percpu_obj_new type size (%d) is greater than %d\n" , + ret_t->size,BPF_GLOBAL_PERCPU_MA_MAX_SIZE ); +return -EINVAL ; + } +mutex_lock (&bpf_percpu_ma_lock); err =bpf_mem_alloc_percpu_unit_init (&bpf_global_percpu_ma, ret_t->size);mutex_unlock (&bpf_percpu_ma_lock);
Signature · 1 line
2.34.1
Yonghong Song
reviewerPATCH v2 6/6 selftests/bpf: Cope with 512 bytes limit with bpf_global_percpu_ma
In reply to Yonghong SongIn the previous patch, the maximum data size for bpf_global_percpu_ma is 512 bytes. This breaks selftest test_bpf_ma. Let us adjust it accordingly. Also added a selftest to capture the verification failure when the allocation size is greater than 512.
--- .../selftests/bpf/progs/percpu_alloc_fail.c | 18 ++++++++++++++++++ .../testing/selftests/bpf/progs/test_bpf_ma.c | 9 --------- 2 files changed, 18 insertions(+), 9 deletions(-)
tools/testing/selftests/bpf/progs/percpu_alloc_fail.c +18 -0
@@ -17,6 +17,10 @@ struct val_with_rb_root_t {struct bpf_spin_lock lock; }; +struct val_600b_t { +char b[600 ]; +}; +struct elem {long sum; struct val_t __percpu_kptr *pc;
@@ -161,4 +165,18 @@ int BPF_PROG(test_array_map_7)return 0 ; } +SEC("?fentry.s/bpf_fentry_test1") +__failure __msg("bpf_percpu_obj_new type size (600) is greater than 512") +int BPF_PROG(test_array_map_8) +{ + struct val_600b_t __percpu_kptr *p; + + p = bpf_percpu_obj_new(struct val_600b_t); +if (!p) +return 0 ; + +bpf_percpu_obj_drop (p); +return 0 ; +} + char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/progs/test_bpf_ma.c +0 -9
@@ -188,9 +188,6 @@ DEFINE_ARRAY_WITH_PERCPU_KPTR(128);DEFINE_ARRAY_WITH_PERCPU_KPTR (192 );DEFINE_ARRAY_WITH_PERCPU_KPTR (256 );DEFINE_ARRAY_WITH_PERCPU_KPTR (512 ); -DEFINE_ARRAY_WITH_PERCPU_KPTR (1024 ); -DEFINE_ARRAY_WITH_PERCPU_KPTR (2048 ); -DEFINE_ARRAY_WITH_PERCPU_KPTR (4096 ); SEC("?fentry/" SYS_PREFIX "sys_nanosleep")int test_batch_alloc_free (void *ctx)
@@ -259,9 +256,6 @@ int test_batch_percpu_alloc_free(void *ctx)CALL_BATCH_PERCPU_ALLOC_FREE (192 ,128 ,6 );CALL_BATCH_PERCPU_ALLOC_FREE (256 ,128 ,7 );CALL_BATCH_PERCPU_ALLOC_FREE (512 ,64 ,8 ); -CALL_BATCH_PERCPU_ALLOC_FREE (1024 ,32 ,9 ); -CALL_BATCH_PERCPU_ALLOC_FREE (2048 ,16 ,10 ); -CALL_BATCH_PERCPU_ALLOC_FREE (4096 ,8 ,11 );return 0 ; }
@@ -283,9 +277,6 @@ int test_percpu_free_through_map_free(void *ctx)CALL_BATCH_PERCPU_ALLOC (192 ,128 ,6 );CALL_BATCH_PERCPU_ALLOC (256 ,128 ,7 );CALL_BATCH_PERCPU_ALLOC (512 ,64 ,8 ); -CALL_BATCH_PERCPU_ALLOC (1024 ,32 ,9 ); -CALL_BATCH_PERCPU_ALLOC (2048 ,16 ,10 ); -CALL_BATCH_PERCPU_ALLOC (4096 ,8 ,11 );return 0 ; }
Signature · 1 line
2.34.1
Re: [PATCH bpf-next v2 6/6] selftests/bpf: Cope with 512 bytes limit with bpf_global_percpu_ma
Hi,
Quoted 50 lines · Yonghong Song
In the previous patch, the maximum data size for
bpf_global_percpu_mais 512 bytes. This breaks selftesttest_bpf_ma. Let us adjust it accordingly. Also added a selftest to capture the verification failure when the allocation size is greater than 512.Signed-off-by Yonghong Song <[email protected]>--- .../selftests/bpf/progs/percpu_alloc_fail.c | 18 ++++++++++++++++++ .../testing/selftests/bpf/progs/test_bpf_ma.c | 9 --------- 2 files changed, 18 insertions(+), 9 deletions(-)tools/testing/selftests/bpf/progs/percpu_alloc_fail.c +18
-0@@ -17,6 +17,10 @@ struct val_with_rb_root_t {struct bpf_spin_lock lock; }; +struct val_600b_t { +char b[600 ]; +}; +struct elem {long sum; struct val_t __percpu_kptr *pc;@@ -161,4 +165,18 @@ int BPF_PROG(test_array_map_7)return 0 ; } +SEC("?fentry.s/bpf_fentry_test1") +__failure __msg("bpf_percpu_obj_new type size (600) is greater than 512") +int BPF_PROG(test_array_map_8) +{ + struct val_600b_t __percpu_kptr *p; + + p = bpf_percpu_obj_new(struct val_600b_t); +if (!p) +return 0 ; + +bpf_percpu_obj_drop (p); +return 0 ; +} + char _license[] SEC("license") = "GPL";tools/testing/selftests/bpf/progs/test_bpf_ma.c +0
-0
tools/testing/selftests/bpf/progs/test_bpf_ma.c DEFINE_ARRAY_WITH_PERCPU_KPTR(128); · line 193
Quoted 6 lines
tools/testing/selftests/bpf/progs/test_bpf_ma.c +0
-3@@ -188,9 +188,6 @@ DEFINE_ARRAY_WITH_PERCPU_KPTR(128);DEFINE_ARRAY_WITH_PERCPU_KPTR (192 );DEFINE_ARRAY_WITH_PERCPU_KPTR (256 );DEFINE_ARRAY_WITH_PERCPU_KPTR (512 ); -DEFINE_ARRAY_WITH_PERCPU_KPTR (1024 ); -DEFINE_ARRAY_WITH_PERCPU_KPTR (2048 ); -DEFINE_ARRAY_WITH_PERCPU_KPTR (4096 );
Considering the update in patch "bpf: Avoid unnecessary extra percpu memory allocation", the definition of DEFINE_ARRAY_WITH_PERCPU_KPTR() needs update as well, because for 512-sized per-cpu kptr, the tests only allocate for (512 - sizeof(void *)) bytes. And we could do DEFINE_ARRAY_WITH_PERCPU_KPTR(8) test after the update. I could do that after the patch-set is landed if you don't have time to do that.
A bit of off-topic, but it is still relevant. I have a question about how to forcibly generate BTF info for struct definition in the test ? Currently, I have to include bin_data_xx in the definition of map_value, but I don't want to increase the size of map_value. I had tried to use BTF_TYPE_EMIT() in prog just like in linux kernel, but it didn't work.
Quoted 23 lines
tools/testing/selftests/bpf/progs/test_bpf_ma.c +0
-6SEC("?fentry/" SYS_PREFIX "sys_nanosleep")int test_batch_alloc_free (void *ctx)@@ -259,9 +256,6 @@ int test_batch_percpu_alloc_free(void *ctx)CALL_BATCH_PERCPU_ALLOC_FREE (192 ,128 ,6 );CALL_BATCH_PERCPU_ALLOC_FREE (256 ,128 ,7 );CALL_BATCH_PERCPU_ALLOC_FREE (512 ,64 ,8 ); -CALL_BATCH_PERCPU_ALLOC_FREE (1024 ,32 ,9 ); -CALL_BATCH_PERCPU_ALLOC_FREE (2048 ,16 ,10 ); -CALL_BATCH_PERCPU_ALLOC_FREE (4096 ,8 ,11 );return 0 ; }@@ -283,9 +277,6 @@ int test_percpu_free_through_map_free(void *ctx)CALL_BATCH_PERCPU_ALLOC (192 ,128 ,6 );CALL_BATCH_PERCPU_ALLOC (256 ,128 ,7 );CALL_BATCH_PERCPU_ALLOC (512 ,64 ,8 ); -CALL_BATCH_PERCPU_ALLOC (1024 ,32 ,9 ); -CALL_BATCH_PERCPU_ALLOC (2048 ,16 ,10 ); -CALL_BATCH_PERCPU_ALLOC (4096 ,8 ,11 );return 0 ; }
Yonghong Song
reviewerRe: [PATCH bpf-next v2 6/6] selftests/bpf: Cope with 512 bytes limit with bpf_global_percpu_ma
Quoted 66 lines · Hou Tao
Hi,
Quoted 50 lines · Yonghong Song
In the previous patch, the maximum data size for
bpf_global_percpu_mais 512 bytes. This breaks selftesttest_bpf_ma. Let us adjust it accordingly. Also added a selftest to capture the verification failure when the allocation size is greater than 512.Signed-off-by Yonghong Song <[email protected]>--- .../selftests/bpf/progs/percpu_alloc_fail.c | 18 ++++++++++++++++++ .../testing/selftests/bpf/progs/test_bpf_ma.c | 9 --------- 2 files changed, 18 insertions(+), 9 deletions(-)tools/testing/selftests/bpf/progs/percpu_alloc_fail.c +18
-0@@ -17,6 +17,10 @@ struct val_with_rb_root_t {struct bpf_spin_lock lock; }; +struct val_600b_t { +char b[600 ]; +}; +struct elem {long sum; struct val_t __percpu_kptr *pc;@@ -161,4 +165,18 @@ int BPF_PROG(test_array_map_7)return 0 ; } +SEC("?fentry.s/bpf_fentry_test1") +__failure __msg("bpf_percpu_obj_new type size (600) is greater than 512") +int BPF_PROG(test_array_map_8) +{ + struct val_600b_t __percpu_kptr *p; + + p = bpf_percpu_obj_new(struct val_600b_t); +if (!p) +return 0 ; + +bpf_percpu_obj_drop (p); +return 0 ; +} + char _license[] SEC("license") = "GPL";tools/testing/selftests/bpf/progs/test_bpf_ma.c +0
-0tools/testing/selftests/bpf/progs/test_bpf_ma.c DEFINE_ARRAY_WITH_PERCPU_KPTR(128); · line 193
Quoted 6 lines
tools/testing/selftests/bpf/progs/test_bpf_ma.c +0
-3@@ -188,9 +188,6 @@ DEFINE_ARRAY_WITH_PERCPU_KPTR(128);DEFINE_ARRAY_WITH_PERCPU_KPTR (192 );DEFINE_ARRAY_WITH_PERCPU_KPTR (256 );DEFINE_ARRAY_WITH_PERCPU_KPTR (512 ); -DEFINE_ARRAY_WITH_PERCPU_KPTR (1024 ); -DEFINE_ARRAY_WITH_PERCPU_KPTR (2048 ); -DEFINE_ARRAY_WITH_PERCPU_KPTR (4096 );Considering the update in patch "bpf: Avoid unnecessary extra percpu memory allocation", the definition of
DEFINE_ARRAY_WITH_PERCPU_KPTR()needs update as well, because for 512-sized per-cpu kptr, the tests only allocate for (512 -sizeof(void *)) bytes. And we could doDEFINE_ARRAY_WITH_PERCPU_KPTR(8)test after the update. I could do that after the patch-set is landed if you don't have time to do that.
Quoted 6 lines
A bit of off-topic, but it is still relevant. I have a question about how to forcibly generate BTF info for struct definition in the test ? Currently, I have to include
bin_data_xxin the definition ofmap_value, but I don't want to increase the size ofmap_value. I had tried to useBTF_TYPE_EMIT()in prog just like in linux kernel, but it didn't work.
Since you mentioned the btf generation issue, I did some investigation. To workaround btf generation issue, we can use the method in prog_tests/local_kptr_stash.c:
====
/* This is necessary so that LLVM generates BTF for node_data struct
* If it's not included, a fwd reference for node_data will be generated but
* no struct. Example BTF of "node" field in map_value when not included:
*
* [10] PTR '(anon)' type_id=35
* [34] FWD 'node_data' fwd_kind=struct
* [35] TYPE_TAG 'kptr_ref' type_id=34
*
* (with no node_data struct defined)
* Had to do the same w/ bpf_kfunc_call_test_release below
*/
struct node_data *just_here_because_btf_bug;
struct refcounted_node *just_here_because_btf_bug2;
====
I have hacked the test_bpf_ma.c files and something like below should work to generate btf types:
struct bin_data_##_size { \
char data[_size - sizeof(void *)]; \
}; \
+ /* See Commit 5d8d6634ccc, force btf generation for type bin_data_##_size */ \
+ struct bin_data_##_size *__bin_data_##_size; \
struct map_value_##_size { \
struct bin_data_##_size __kptr * data; \
- /* To emit BTF info for bin_data_xx */ \
- struct bin_data_##_size not_used; \
}; \
struct { \
__uint(type, BPF_MAP_TYPE_ARRAY); \
@@ -40,8 +43,12 @@ int pid = 0;
} array_##_size SEC(".maps") #define DEFINE_ARRAY_WITH_PERCPU_KPTR(_size) \
+ struct percpu_bin_data_##_size { \
+ char data[_size]; \
+ }; \
+ struct percpu_bin_data_##_size *__percpu_bin_data_##_size; \
struct map_value_percpu_##_size { \
- struct bin_data_##_size __percpu_kptr * data; \
+ struct percpu_bin_data_##_size __percpu_kptr * data; \
}; \
struct { \
__uint(type, BPF_MAP_TYPE_ARRAY); \I have a prototype to ensure the type (for percpu kptr) removing these '- sizeof(void *)' and enabling DEFINE_ARRAY_WITH_PERCPU_KPTR(). Once we resolved the check_obj_size() issue, I can then post v3.
Quoted 23 lines
Quoted 23 lines
tools/testing/selftests/bpf/progs/test_bpf_ma.c +0
-6SEC("?fentry/" SYS_PREFIX "sys_nanosleep")int test_batch_alloc_free (void *ctx)@@ -259,9 +256,6 @@ int test_batch_percpu_alloc_free(void *ctx)CALL_BATCH_PERCPU_ALLOC_FREE (192 ,128 ,6 );CALL_BATCH_PERCPU_ALLOC_FREE (256 ,128 ,7 );CALL_BATCH_PERCPU_ALLOC_FREE (512 ,64 ,8 ); -CALL_BATCH_PERCPU_ALLOC_FREE (1024 ,32 ,9 ); -CALL_BATCH_PERCPU_ALLOC_FREE (2048 ,16 ,10 ); -CALL_BATCH_PERCPU_ALLOC_FREE (4096 ,8 ,11 );return 0 ; }@@ -283,9 +277,6 @@ int test_percpu_free_through_map_free(void *ctx)CALL_BATCH_PERCPU_ALLOC (192 ,128 ,6 );CALL_BATCH_PERCPU_ALLOC (256 ,128 ,7 );CALL_BATCH_PERCPU_ALLOC (512 ,64 ,8 ); -CALL_BATCH_PERCPU_ALLOC (1024 ,32 ,9 ); -CALL_BATCH_PERCPU_ALLOC (2048 ,16 ,10 ); -CALL_BATCH_PERCPU_ALLOC (4096 ,8 ,11 );return 0 ; }
Re: [PATCH bpf-next v2 6/6] selftests/bpf: Cope with 512 bytes limit with bpf_global_percpu_ma
Hi,
Quoted 12 lines · Yonghong Song
Quoted 10 lines · Hou Tao
Hi,
Quoted 7 lines · Yonghong Song
In the previous patch, the maximum data size for
bpf_global_percpu_mais 512 bytes. This breaks selftesttest_bpf_ma. Let us adjust it accordingly. Also added a selftest to capture the verification failure when the allocation size is greater than 512.Signed-off-by Yonghong Song <[email protected]>
SNIP
Quoted 24 lines
Quoted 13 lines
tools/testing/selftests/bpf/progs/test_bpf_ma.c DEFINE_ARRAY_WITH_PERCPU_KPTR(128); · line 193
Quoted 6 lines
DEFINE_ARRAY_WITH_PERCPU_KPTR(192);
DEFINE_ARRAY_WITH_PERCPU_KPTR(256);
DEFINE_ARRAY_WITH_PERCPU_KPTR(512);tools/testing/selftests/bpf/progs/test_bpf_ma.c +0
-3-DEFINE_ARRAY_WITH_PERCPU_KPTR (1024 ); -DEFINE_ARRAY_WITH_PERCPU_KPTR (2048 ); -DEFINE_ARRAY_WITH_PERCPU_KPTR (4096 );Considering the update in patch "bpf: Avoid unnecessary extra percpu memory allocation", the definition of
DEFINE_ARRAY_WITH_PERCPU_KPTR()needs update as well, because for 512-sized per-cpu kptr, the tests only allocate for (512 -sizeof(void *)) bytes. And we could doDEFINE_ARRAY_WITH_PERCPU_KPTR(8)test after the update. I could do that after the patch-set is landed if you don't have time to do that.Quoted 6 lines
A bit of off-topic, but it is still relevant. I have a question about how to forcibly generate BTF info for struct definition in the test ? Currently, I have to include
bin_data_xxin the definition ofmap_value, but I don't want to increase the size ofmap_value. I had tried to useBTF_TYPE_EMIT()in prog just like in linux kernel, but it didn't work.Since you mentioned the btf generation issue, I did some investigation. To workaround btf generation issue, we can use the method in
prog_tests/local_kptr_stash.c:
Quoted 16 lines
====
/* This is necessary so that LLVM generates BTF fornode_datastruct
* If it's not included, a fwd reference fornode_datawill be generated but
* no struct. Example BTF of "node" field inmap_valuewhen not included:
*
* [10] PTR '(anon)' type_id=35
* [34] FWD 'node_data' fwd_kind=struct
* [35]TYPE_TAG'kptr_ref' type_id=34
*
* (with nonode_datastruct defined)
* Had to do the same w/bpf_kfunc_call_test_releasebelow
*/
structnode_data*just_here_because_btf_bug;
structrefcounted_node*just_here_because_btf_bug2;
====
Totally missed it. Thanks for pointing it out to me.
Quoted 32 lines
I have hacked the
test_bpf_ma.cfiles and something like below should work to generate btf types:struct bin_data_##_size { \
char data[_size -sizeof(void *)]; \
}; \
+ /* See Commit5d8d6634ccc, force btf generation for type bin_data_##_size */ \
+ struct bin_data_##_size *__bin_data_##_size; \
struct map_value_##_size { \
struct bin_data_##_size__kptr* data; \
- /* To emit BTF info forbin_data_xx*/ \
- struct bin_data_##_sizenot_used; \
}; \
struct { \
__uint(type, BPF_MAP_TYPE_ARRAY); \(quoted diff) +0
-0@@ -40,8 +43,12 @@ int pid = 0; } array_##_size SEC(".maps")
#defineDEFINE_ARRAY_WITH_PERCPU_KPTR(_size)\
+ struct percpu_bin_data_##_size { \
+ char data[_size]; \
+ }; \
+ struct percpu_bin_data_##_size *__percpu_bin_data_##_size; \
struct map_value_percpu_##_size { \
- struct bin_data_##_size__percpu_kptr* data; \
+ struct percpu_bin_data_##_size__percpu_kptr* data; \
}; \
struct { \
__uint(type, BPF_MAP_TYPE_ARRAY); \
Quoted 3 lines
I have a prototype to ensure the type (for percpu kptr) removing these '-
sizeof(void *)' and enablingDEFINE_ARRAY_WITH_PERCPU_KPTR(). Once we resolved thecheck_obj_size()issue, I can then post v3.
Thanks for the update and it looks fine to me. Looking forwards to v3.
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
SEC("?fentry/" SYS_PREFIX "sys_nanosleep")
inttest_batch_alloc_free(void *ctx)tools/testing/selftests/bpf/progs/test_bpf_ma.c +0
-6@@ -259,9 +256,6 @@ int test_batch_percpu_alloc_free(void *ctx) CALL_BATCH_PERCPU_ALLOC_FREE(192, 128, 6); CALL_BATCH_PERCPU_ALLOC_FREE(256, 128, 7); CALL_BATCH_PERCPU_ALLOC_FREE(512, 64, 8); - CALL_BATCH_PERCPU_ALLOC_FREE(1024, 32, 9); - CALL_BATCH_PERCPU_ALLOC_FREE(2048, 16, 10); - CALL_BATCH_PERCPU_ALLOC_FREE(4096, 8, 11); return 0; }@@ -283,9 +277,6 @@ int test_percpu_free_through_map_free(void *ctx) CALL_BATCH_PERCPU_ALLOC(192, 128, 6); CALL_BATCH_PERCPU_ALLOC(256, 128, 7); CALL_BATCH_PERCPU_ALLOC(512, 64, 8); - CALL_BATCH_PERCPU_ALLOC(1024, 32, 9); - CALL_BATCH_PERCPU_ALLOC(2048, 16, 10); - CALL_BATCH_PERCPU_ALLOC(4096, 8, 11); return 0; }