bpf: Reduce memory usage for bpf_global_percpu_ma
Subsystems (4)
Yonghong Song
reviewerv3 cover bpf: Reduce memory usage for bpf_global_percpu_ma
Currently when a bpf program intends to allocate memory for percpu kptr, the verifier will call bpf_mem_alloc_init() to prefill all supported unit sizes and this caused memory consumption very big for large number of cpus. For example, for 128-cpu system, the total memory consumption with initial prefill is ~175MB. Things will become worse for systems with even more cpus.
Patch 1 avoids unnecessary extra percpu memory allocation. Patch 2 addresses memory consumption issue by avoiding to prefill with all unit sizes, i.e. only prefilling with user specified size. Patch 3 further reduces memory consumption by limiting the number of prefill entries for percpu memory allocation. Patch 4 rejects percpu memory allocation with bpf_global_percpu_ma when allocation size is greater than 512 bytes.
Patch 5 fixed test_bpf_ma test due to Patch 5.
Patch 6 added one test to show the verification failure log message.
Changelogs:
v2 -> v3:
. Clear the bpf_mem_cache if prefill fails.
. Change test_bpf_ma percpu allocation tests to use bucket_size
as allocation size instead of bucket_size - 8.
. Remove __GFP_ZERO flag from __alloc_percpu_gfp() call.
v1 -> v2:
. Avoid unnecessary extra percpu memory allocation.
. Add a separate function to do bpf_global_percpu_ma initialization . promote.
. Promote function static 'sizes' array to file static. . Add comments to explain to refill only one item for percpu alloc.
Yonghong Song (6):
bpf: Avoid unnecessary extra percpu memory allocation
bpf: Allow per unit prefill for non-fix-size percpu memory allocator
bpf: Refill only one percpu element in memalloc
bpf: Limit up to 512 bytes for bpf_global_percpu_ma allocation selftests/bpf: Cope with 512 bytes limit with bpf_global_percpu_ma selftests/bpf: Add a selftest with > 512-byte percpu allocation size
include/linux/bpf.h | 2 +- include/linux/bpf_mem_alloc.h | 7 ++ kernel/bpf/core.c | 8 +- kernel/bpf/memalloc.c | 84 +++++++++++++++++-- kernel/bpf/verifier.c | 36 ++++---- .../selftests/bpf/prog_tests/test_bpf_ma.c | 20 +++-- .../selftests/bpf/progs/percpu_alloc_fail.c | 18 ++++ .../testing/selftests/bpf/progs/test_bpf_ma.c | 34 ++++---- 8 files changed, 158 insertions(+), 51 deletions(-)
Signature · 1 line
2.34.1
Yonghong Song
reviewerPATCH v3 1/6 bpf: Avoid unnecessary extra percpu memory allocation
Currently, for percpu memory allocation, say if the user requests allocation size to be 32 bytes, the actually calculated size will be 40 bytes and it further rounds to 64 bytes, and eventually 64 bytes are allocated, wasting 32-byte memory.
Change bpf_mem_alloc() to calculate the cache index based on the user-provided allocation size so unnecessary extra memory can be avoided.
--- kernel/bpf/memalloc.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/bpf/memalloc.c +3 -1
@@ -871,7 +871,9 @@ void notrace *bpf_mem_alloc(struct bpf_mem_alloc *ma, size_t size)if (!size)return ZERO_SIZE_PTR ; - idx =bpf_mem_cache_idx (size +LLIST_NODE_SZ ); +if (!ma->percpu) + size +=LLIST_NODE_SZ ; + idx =bpf_mem_cache_idx (size);if (idx <0 )return NULL ;
Signature · 1 line
2.34.1
Yonghong Song
reviewerPATCH v3 2/6 bpf: Allow per unit prefill for non-fix-size percpu memory allocator
In reply to Yonghong SongCommit 41a5db8d8161 ("Add support for non-fix-size percpu mem allocation") added support for non-fix-size percpu memory allocation.
Such allocation will allocate percpu memory for all buckets on all cpus and the memory consumption is in the order to quadratic. For example, let us say, 4 cpus, unit size 16 bytes, so each cpu has 16 * 4 = 64 bytes, with 4 cpus, total will be 64 * 4 = 256 bytes. Then let us say, 8 cpus with the same unit size, each cpu
has 16 * 8 = 128 bytes, with 8 cpus, total will be 128 * 8 = 1024 bytes. So if the number of cpus doubles, the number of memory consumption will be 4 times. So for a system with large number of cpus, the memory consumption goes up quickly with quadratic order.
For example, for 4KB percpu allocation, 128 cpus. The total memory consumption will 4KB * 128 * 128 = 64MB. Things will become worse if the number of cpus is bigger (e.g., 512, 1024, etc.)
In Commit 41a5db8d8161, the non-fix-size percpu memory allocation is done in boot time, so for system with large number of cpus, the initial percpu memory consumption is very visible. For example, for 128 cpu system, the total percpu memory allocation will be at least (16 + 32 + 64 + 96 + 128 + 196 + 256 + 512 + 1024 + 2048 + 4096)
* 128 * 128 = ~138MB.
which is pretty big. It will be even bigger for larger number of cpus.
Note that the current prefill also allocates 4 entries if the unit size is less than 256. So on top of 138MB memory consumption, this will add more consumption with
3 * (16 + 32 + 64 + 96 + 128 + 196 + 256) * 128 * 128 = ~38MB. Next patch will try to reduce this memory consumption.
Later on, Commit 1fda5bb66ad8 ("bpf: Do not allocate percpu memory at init stage") moved the non-fix-size percpu memory allocation to bpf verificaiton stage. Once a particular bpf_percpu_obj_new() is called by bpf program, the memory allocator will try to fill in the cache with all sizes, causing the same amount of percpu memory consumption as in the boot stage.
To reduce the initial percpu memory consumption for non-fix-size percpu memory allocation, instead of filling the cache with all supported allocation sizes, this patch intends to fill the cache only for the requested size. As typically users will not use large percpu data structure, this can save memory significantly. For example, the allocation size is 64 bytes with 128 cpus. Then total percpu memory amount will be 64 * 128 * 128 = 1MB, much less than previous 138MB.
--- include/linux/bpf.h | 2 +- include/linux/bpf_mem_alloc.h | 7 ++++ kernel/bpf/core.c | 8 +++-- kernel/bpf/memalloc.c | 67 ++++++++++++++++++++++++++++++++++- kernel/bpf/verifier.c | 28 ++++++--------- 5 files changed, 90 insertions(+), 22 deletions(-)
include/linux/bpf.h +1 -1
@@ -61,7 +61,7 @@ extern struct idr btf_idr;extern spinlock_t btf_idr_lock;extern struct kobject *btf_kobj;extern struct bpf_mem_alloc bpf_global_ma, bpf_global_percpu_ma; -extern bool bpf_global_ma_set; +extern bool bpf_global_ma_set, bpf_global_percpu_ma_set;typedef u64 (*bpf_callback_t )(u64 ,u64 ,u64 ,u64 ,u64 ); typedef int (*bpf_iter_init_seq_priv_t)(void *private_data,
include/linux/bpf_mem_alloc.h +7 -0
@@ -21,8 +21,15 @@ struct bpf_mem_alloc {* 'size = 0' is for bpf_mem_alloc which manages many fixed-size objects. * Alloc and free are done with bpf_mem_{alloc,free}() and the size of * the returned object is given by the size argument of bpf_mem_alloc(). +* If percpu equals true, error will be returned in order to avoid +* large memory consumption and the below bpf_mem_alloc_percpu_unit_init() +* should be used to do on-demand per-cpu allocation for each size. */ int bpf_mem_alloc_init (struct bpf_mem_alloc *ma,int size,bool percpu); +/* Initialize a non-fix-size percpu memory allocator */ +int bpf_mem_alloc_percpu_init (struct bpf_mem_alloc *ma); +/* The percpu allocation with a specific unit size. */ +int bpf_mem_alloc_percpu_unit_init (struct bpf_mem_alloc *ma,int size);void bpf_mem_alloc_destroy (struct bpf_mem_alloc *ma);/* kmalloc/kfree equivalent: */
kernel/bpf/core.c +5 -3
@@ -64,8 +64,8 @@#define OFF insn->off#define IMM insn->imm -struct bpf_mem_alloc bpf_global_ma; -bool bpf_global_ma_set; +struct bpf_mem_alloc bpf_global_ma, bpf_global_percpu_ma; +bool bpf_global_ma_set, bpf_global_percpu_ma_set;/* No hurry in this branch *
@@ -2963,7 +2963,9 @@ static int __init bpf_global_ma_init(void) ret =bpf_mem_alloc_init (&bpf_global_ma,0 , false); bpf_global_ma_set = !ret; - return ret; + ret =bpf_mem_alloc_percpu_init (&bpf_global_percpu_ma); + bpf_global_percpu_ma_set = !ret; + return !bpf_global_ma_set || !bpf_global_percpu_ma_set; }late_initcall (bpf_global_ma_init);#endif
kernel/bpf/memalloc.c +66 -1
90 lines not on this page · raw message
kernel/bpf/verifier.c +11 -17
@@ -42,9 +42,6 @@ static const struct bpf_verifier_ops * const bpf_verifier_ops[] = {#undef BPF_LINK_TYPE }; -struct bpf_mem_alloc bpf_global_percpu_ma; -static bool bpf_global_percpu_ma_set; -/* bpf_check() is a static code analyzer that walks eBPF program * instruction by instruction and updates register/stack state. * All paths of conditional branches are analyzed until 'bpf_exit' insn.
@@ -12062,20 +12059,6 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,if (meta.func_id == special_kfunc_list[KF_bpf_obj_new_impl] && !bpf_global_ma_set)return -ENOMEM ; -if (meta.func_id == special_kfunc_list[KF_bpf_percpu_obj_new_impl]) { -if (!bpf_global_percpu_ma_set) { -mutex_lock (&bpf_percpu_ma_lock); -if (!bpf_global_percpu_ma_set) { - err =bpf_mem_alloc_init (&bpf_global_percpu_ma,0 , true); -if (!err) - bpf_global_percpu_ma_set = true; - } -mutex_unlock (&bpf_percpu_ma_lock); -if (err) -return err; - } - } -if (((u64 )(u32 )meta.arg_constant.value) != meta.arg_constant.value) {verbose (env,"local type ID argument must be in range [0, U32_MAX]\n" );return -EINVAL ;
@@ -12096,6 +12079,17 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,return -EINVAL ; } +if (meta.func_id == special_kfunc_list[KF_bpf_percpu_obj_new_impl]) { +if (!bpf_global_percpu_ma_set) +return -ENOMEM ; + +mutex_lock (&bpf_percpu_ma_lock); + err =bpf_mem_alloc_percpu_unit_init (&bpf_global_percpu_ma, ret_t->size); +mutex_unlock (&bpf_percpu_ma_lock); +if (err) +return err; + } + struct_meta =btf_find_struct_meta (ret_btf, ret_btf_id);if (meta.func_id == special_kfunc_list[KF_bpf_percpu_obj_new_impl]) {if (!__btf_type_is_scalar_struct (env, ret_btf, ret_t,0 )) {
Signature · 1 line
2.34.1
Re: [PATCH bpf-next v3 2/6] bpf: Allow per unit prefill for non-fix-size percpu memory allocator
Hi,
Quoted 14 lines · Yonghong Song
Commit
41a5db8d8161("Add support for non-fix-size percpu mem allocation") added support for non-fix-size percpu memory allocation.
Such allocation will allocate percpu memory for all buckets on all cpus and the memory consumption is in the order to quadratic. For example, let us say, 4 cpus, unit size 16 bytes, so each cpu has 16 * 4 = 64 bytes, with 4 cpus, total will be 64 * 4 = 256 bytes. Then let us say, 8 cpus with the same unit size, each cpu
has 16 * 8 = 128 bytes, with 8 cpus, total will be 128 * 8 = 1024 bytes. So if the number of cpus doubles, the number of memory consumption will be 4 times. So for a system with large number of cpus, the memory consumption goes up quickly with quadratic order.
For example, for 4KB percpu allocation, 128 cpus. The total memory consumption will 4KB * 128 * 128 = 64MB. Things will become worse if the number of cpus is bigger (e.g., 512, 1024, etc.)
SNIP
Quoted 23 lines
kernel/bpf/memalloc.c +23
-0+__init int bpf_mem_alloc_percpu_init (struct bpf_mem_alloc *ma) +{ +struct bpf_mem_caches __percpu *pcc; + + pcc =__alloc_percpu_gfp (sizeof (struct bpf_mem_caches ),8 ,GFP_KERNEL ); +if (!pcc) +return -ENOMEM ; + + ma->caches = pcc; + ma->percpu = true; +return 0 ; +} + +int bpf_mem_alloc_percpu_unit_init (struct bpf_mem_alloc *ma,int size) +{ +int cpu, i, err =0 , unit_size, percpu_size; +struct bpf_mem_caches *cc,__percpu *pcc; +struct obj_cgroup *objcg; +struct bpf_mem_cache *c; + + i =bpf_mem_cache_idx (size); +if (i <0 ) +return -EINVAL ;
kernel/bpf/memalloc.c static void drain_mem_cache(struct bpf_mem_cache *c) · line 657
Quoted 10 lines
kernel/bpf/memalloc.c +10
-0+ +/* room for llist_node and per-cpu pointer */ + percpu_size =LLIST_NODE_SZ +sizeof (void *); + + pcc = ma->caches; + unit_size = sizes[i]; + +#ifdef CONFIG_MEMCG_KMEM + objcg =get_obj_cgroup_from_current (); +#endif
For bpf_global_percpu_ma, we also need to account the allocated memory to root memory cgroup just like bpf_global_ma did, do we ? So it seems that we need to initialize c->objcg early in bpf_mem_alloc_percpu_init ().
Quoted 9 lines
kernel/bpf/memalloc.c +9
-0+for_each_possible_cpu (cpu) { + cc =per_cpu_ptr (pcc, cpu); + c = &cc->cache[i]; +if (cpu ==0 && c->unit_size) + goto out; + + c->unit_size = unit_size; + c->objcg = objcg; + c->percpu_size = percpu_size;
kernel/bpf/memalloc.c static void drain_mem_cache(struct bpf_mem_cache *c) · line 676
Quoted 10 lines
kernel/bpf/memalloc.c +10
-0+ c->tgt = c; + +init_refill_work (c); +prefill_mem_cache (c, cpu); + +if (cpu ==0 ) { + err =check_obj_size (c, i); +if (err) { +drain_mem_cache (c); +memset (c,0 ,sizeof (*c));
I also forgot about c->objcg. objcg may be leaked if we do memset() here.
kernel/bpf/memalloc.c static void drain_mem_cache(struct bpf_mem_cache *c) · line 685
Quoted 10 lines
kernel/bpf/memalloc.c +9
-0+ goto out; + } + } + } + +out: +return err; +} +
.
Yonghong Song
reviewerRe: [PATCH bpf-next v3 2/6] bpf: Allow per unit prefill for non-fix-size percpu memory allocator
Quoted 46 lines · Hou Tao
Hi,
Quoted 14 lines · Yonghong Song
Commit
41a5db8d8161("Add support for non-fix-size percpu mem allocation") added support for non-fix-size percpu memory allocation.
Such allocation will allocate percpu memory for all buckets on all cpus and the memory consumption is in the order to quadratic. For example, let us say, 4 cpus, unit size 16 bytes, so each cpu has 16 * 4 = 64 bytes, with 4 cpus, total will be 64 * 4 = 256 bytes. Then let us say, 8 cpus with the same unit size, each cpu
has 16 * 8 = 128 bytes, with 8 cpus, total will be 128 * 8 = 1024 bytes. So if the number of cpus doubles, the number of memory consumption will be 4 times. So for a system with large number of cpus, the memory consumption goes up quickly with quadratic order.
For example, for 4KB percpu allocation, 128 cpus. The total memory consumption will 4KB * 128 * 128 = 64MB. Things will become worse if the number of cpus is bigger (e.g., 512, 1024, etc.)SNIP
Quoted 28 lines
kernel/bpf/memalloc.c +28
-0+__init int bpf_mem_alloc_percpu_init (struct bpf_mem_alloc *ma) +{ +struct bpf_mem_caches __percpu *pcc; + + pcc =__alloc_percpu_gfp (sizeof (struct bpf_mem_caches ),8 ,GFP_KERNEL ); +if (!pcc) +return -ENOMEM ; + + ma->caches = pcc; + ma->percpu = true; +return 0 ; +} + +int bpf_mem_alloc_percpu_unit_init (struct bpf_mem_alloc *ma,int size) +{ +int cpu, i, err =0 , unit_size, percpu_size; +struct bpf_mem_caches *cc,__percpu *pcc; +struct obj_cgroup *objcg; +struct bpf_mem_cache *c; + + i =bpf_mem_cache_idx (size); +if (i <0 ) +return -EINVAL ; + +/* room for llist_node and per-cpu pointer */ + percpu_size =LLIST_NODE_SZ +sizeof (void *); + + pcc = ma->caches;
Quoted 8 lines
kernel/bpf/memalloc.c static void drain_mem_cache(struct bpf_mem_cache *c) · line 657
Quoted 5 lines
kernel/bpf/memalloc.c +5
-0+ unit_size = sizes[i]; + +#ifdef CONFIG_MEMCG_KMEM + objcg =get_obj_cgroup_from_current (); +#endif For
bpf_global_percpu_ma, we also need to account the allocated memory to root memory cgroup just likebpf_global_madid, do we ? So it seems that we need to initializec->objcgearly inbpf_mem_alloc_percpu_init().
Good point. Agree. the original behavior percpu non-fix-size mem allocation is to do get_obj_cgroup_from_current() at init stage and charge to root memory cgroup, and we indeed should move the above bpf_mem_alloc_percpu_init().
Quoted 12 lines
Quoted 12 lines
kernel/bpf/memalloc.c +12
-0+for_each_possible_cpu (cpu) { + cc =per_cpu_ptr (pcc, cpu); + c = &cc->cache[i]; +if (cpu ==0 && c->unit_size) + goto out; + + c->unit_size = unit_size; + c->objcg = objcg; + c->percpu_size = percpu_size; + c->tgt = c; + +init_refill_work (c);
Quoted 8 lines
kernel/bpf/memalloc.c static void drain_mem_cache(struct bpf_mem_cache *c) · line 676
Quoted 7 lines
kernel/bpf/memalloc.c +7
-0+prefill_mem_cache (c, cpu); + +if (cpu ==0 ) { + err =check_obj_size (c, i); +if (err) { +drain_mem_cache (c); +memset (c,0 ,sizeof (*c));I also forgot about
c->objcg. objcg may be leaked if we domemset()here.
The objcg gets a reference at init bpf_mem_alloc_init() stage and released at bpf_mem_alloc_destroy(). For bpf_global_ma, if there is a failure, indeed bpf_mem_alloc_destroy() will be called and the reference c->objcg will be released.
So if we move get_obj_cgroup_from_current() to bpf_mem_alloc_percpu_init() stage, we should be okay here.
BTW, is check_obj_size() really necessary here? My answer is no since as you mentioned, the size->cache_index is pretty stable, so check_obj_size() should not return error in such cases. What do you think?
Quoted 12 lines
kernel/bpf/memalloc.c static void drain_mem_cache(struct bpf_mem_cache *c) · line 685
Quoted 10 lines
kernel/bpf/memalloc.c +9
-0+ goto out; + } + } + } + +out: +return err; +} +.
Yonghong Song
reviewerRe: [PATCH bpf-next v3 2/6] bpf: Allow per unit prefill for non-fix-size percpu memory allocator
Quoted 89 lines · Yonghong Song
Quoted 47 lines · Hou Tao
Hi,
Quoted 14 lines · Yonghong Song
Commit
41a5db8d8161("Add support for non-fix-size percpu mem allocation") added support for non-fix-size percpu memory allocation.
Such allocation will allocate percpu memory for all buckets on all cpus and the memory consumption is in the order to quadratic. For example, let us say, 4 cpus, unit size 16 bytes, so each cpu has 16 * 4 = 64 bytes, with 4 cpus, total will be 64 * 4 = 256 bytes. Then let us say, 8 cpus with the same unit size, each cpu
has 16 * 8 = 128 bytes, with 8 cpus, total will be 128 * 8 = 1024 bytes. So if the number of cpus doubles, the number of memory consumption will be 4 times. So for a system with large number of cpus, the memory consumption goes up quickly with quadratic order.
For example, for 4KB percpu allocation, 128 cpus. The total memory consumption will 4KB * 128 * 128 = 64MB. Things will become worse if the number of cpus is bigger (e.g., 512, 1024, etc.)SNIP
Quoted 29 lines
(quoted diff) +5
-0+__init int bpf_mem_alloc_percpu_init(struct bpf_mem_alloc *ma) +{ + struct bpf_mem_caches __percpu *pcc; + + pcc = __alloc_percpu_gfp(sizeof(struct bpf_mem_caches), 8,
GFP_KERNEL);
+ if (!pcc)
+ return-ENOMEM;(quoted diff) +21
-0+ + ma->caches = pcc; + ma->percpu = true; + return 0; +} + +int bpf_mem_alloc_percpu_unit_init(struct bpf_mem_alloc *ma, int size) +{ + int cpu, i, err = 0, unit_size, percpu_size; + struct bpf_mem_caches *cc, __percpu *pcc; + struct obj_cgroup *objcg; + struct bpf_mem_cache *c; + + i = bpf_mem_cache_idx(size); + if (i < 0) + return -EINVAL; + + /* room for llist_node and per-cpu pointer */ + percpu_size = LLIST_NODE_SZ + sizeof(void *); + + pcc = ma->caches;Quoted 8 lines
Review
Quoted 5 lines
+
unit_size= sizes[i];(quoted diff) +4
-0+ +#ifdef CONFIG_MEMCG_KMEM + objcg = get_obj_cgroup_from_current(); +#endifFor
bpf_global_percpu_ma, we also need to account the allocated memory to root memory cgroup just likebpf_global_madid, do we ? So it seems that we need to initializec->objcgearly inbpf_mem_alloc_percpu_init().Good point. Agree. the original behavior percpu non-fix-size mem allocation is to do
get_obj_cgroup_from_current()at init stage and charge to root memory cgroup, and we indeed should move the abovebpf_mem_alloc_percpu_init().Quoted 12 lines
Quoted 12 lines
+
for_each_possible_cpu(cpu){
+ cc =per_cpu_ptr(pcc, cpu);
+ c = &cc->cache[i];
+ if (cpu == 0 &&c->unit_size)
+ goto out;(quoted diff) +7
-0+ + c->unit_size = unit_size; + c->objcg = objcg; + c->percpu_size = percpu_size; + c->tgt = c; + + init_refill_work(c);Quoted 8 lines
Quoted 7 lines
+
prefill_mem_cache(c, cpu);
+
+ if (cpu == 0) {
+ err =check_obj_size(c, i);
+ if (err) {
+drain_mem_cache(c);
+memset(c, 0, sizeof(*c));I also forgot about
c->objcg. objcg may be leaked if we domemset()here.The objcg gets a reference at init
bpf_mem_alloc_init()stage and released atbpf_mem_alloc_destroy(). Forbpf_global_ma, if there is a failure, indeedbpf_mem_alloc_destroy()will be called and the referencec->objcgwill be released.
Quoted 7 lines
So if we move
get_obj_cgroup_from_current()tobpf_mem_alloc_percpu_init()stage, we should be okay here.BTW, is
check_obj_size()really necessary here? My answer is no since as you mentioned, thesize->cache_indexis pretty stable, socheck_obj_size()should not return error in such cases. What do you think?
How about the following change on top of this patch?
include/linux/bpf_mem_alloc.h +1 -0
@@ -11,6 +11,7 @@ struct bpf_mem_caches;struct bpf_mem_alloc {struct bpf_mem_caches __percpu *caches;struct bpf_mem_cache __percpu *cache; +struct obj_cgroup *objcg;bool percpu;struct work_struct work; };
kernel/bpf/memalloc.c +14 -8
@@ -553,6 +553,8 @@ int bpf_mem_alloc_init(struct bpf_mem_alloc *ma, int size, bool percpu)if (memcg_bpf_enabled ()) objcg =get_obj_cgroup_from_current ();#endif + ma->objcg = objcg; +for_each_possible_cpu (cpu) { c =per_cpu_ptr (pc, cpu); c->unit_size = unit_size;
@@ -573,6 +575,7 @@ int bpf_mem_alloc_init(struct bpf_mem_alloc *ma, int size, bool percpu)#ifdef CONFIG_MEMCG_KMEM objcg =get_obj_cgroup_from_current ();#endif + ma->objcg = objcg;for_each_possible_cpu (cpu) { cc =per_cpu_ptr (pcc, cpu);for (i =0 ; i <NUM_CACHES ; i++) {
@@ -637,6 +640,12 @@ __init int bpf_mem_alloc_percpu_init(struct bpf_mem_alloc *ma) ma->caches = pcc; ma->percpu = true; + +#ifdef CONFIG_MEMCG_KMEM + ma->objcg =get_obj_cgroup_from_current (); +#else + ma->objcg =NULL ; +#endif return 0 ; }
@@ -656,10 +665,8 @@ int bpf_mem_alloc_percpu_unit_init(struct bpf_mem_alloc *ma, int size) pcc = ma->caches; unit_size = sizes[i]; + objcg = ma->objcg; -#ifdef CONFIG_MEMCG_KMEM - objcg =get_obj_cgroup_from_current (); -#endif for_each_possible_cpu (cpu) { cc =per_cpu_ptr (pcc, cpu); c = &cc->cache[i];
@@ -799,9 +806,8 @@ void bpf_mem_alloc_destroy(struct bpf_mem_alloc *ma) rcu_in_progress +=atomic_read (&c->call_rcu_ttrace_in_progress); rcu_in_progress +=atomic_read (&c->call_rcu_in_progress); } -/* objcg is the same across cpus */ -if (c->objcg) -obj_cgroup_put (c->objcg); +if (ma->objcg) +obj_cgroup_put (ma->objcg);destroy_mem_alloc (ma, rcu_in_progress); }if (ma->caches) {
@@ -817,8 +823,8 @@ void bpf_mem_alloc_destroy(struct bpf_mem_alloc *ma) rcu_in_progress +=atomic_read (&c->call_rcu_in_progress); } } -if (c->objcg) -obj_cgroup_put (c->objcg); +if (ma->objcg) +obj_cgroup_put (ma->objcg);destroy_mem_alloc (ma, rcu_in_progress); } }
I still think check_obj_size for percpu allocation is not needed. But I guess we can address that issue later on.
Quoted 13 lines
Quoted 12 lines
Review
Quoted 10 lines
+ goto out;
+ }
+ }
+ }(quoted diff) +5
-0+ +out: + return err; +} +.
Re: [PATCH bpf-next v3 2/6] bpf: Allow per unit prefill for non-fix-size percpu memory allocator
Hi,
Quoted 100 lines · Yonghong Song
Quoted 98 lines · Yonghong Song
Quoted 52 lines · Hou Tao
Hi,
Quoted 17 lines · Yonghong Song
Commit
41a5db8d8161("Add support for non-fix-size percpu mem allocation")
added support for non-fix-size percpu memory allocation. Such allocation will allocate percpu memory for all buckets on all cpus and the memory consumption is in the order to quadratic. For example, let us say, 4 cpus, unit size 16 bytes, so each cpu has 16 * 4 = 64 bytes, with 4 cpus, total will be 64 * 4 = 256 bytes.
Then let us say, 8 cpus with the same unit size, each cpu has 16 * 8 = 128 bytes, with 8 cpus, total will be 128 * 8 = 1024 bytes.
So if the number of cpus doubles, the number of memory consumption will be 4 times. So for a system with large number of cpus, the memory consumption goes up quickly with quadratic order. For example, for 4KB percpu allocation, 128 cpus. The total memory consumption will 4KB * 128 * 128 = 64MB. Things will become worse if the number of cpus is bigger (e.g., 512, 1024, etc.)SNIP
Quoted 31 lines
(quoted diff) +5
-0+__init int bpf_mem_alloc_percpu_init(struct bpf_mem_alloc *ma) +{ + struct bpf_mem_caches __percpu *pcc; + + pcc = __alloc_percpu_gfp(sizeof(struct bpf_mem_caches), 8,
GFP_KERNEL);
+ if (!pcc)
+ return-ENOMEM;(quoted diff) +7
-0+ + ma->caches = pcc; + ma->percpu = true; + return 0; +} + +int bpf_mem_alloc_percpu_unit_init(struct bpf_mem_alloc *ma, intsize)
(quoted diff) +15
-0+{ + int cpu, i, err = 0, unit_size, percpu_size; + struct bpf_mem_caches *cc, __percpu *pcc; + struct obj_cgroup *objcg; + struct bpf_mem_cache *c; + + i = bpf_mem_cache_idx(size); + if (i < 0) + return -EINVAL; + + /* room for llist_node and per-cpu pointer */ + percpu_size = LLIST_NODE_SZ + sizeof(void *); + + pcc = ma->caches; + unit_size = sizes[i];Quoted 8 lines
kernel/bpf/memalloc.c __init int bpf_mem_alloc_percpu_init(struct bpf_mem_alloc *ma) · line 648
Quoted 4 lines
kernel/bpf/memalloc.c +4
-0+ +#ifdef CONFIG_MEMCG_KMEM + objcg = get_obj_cgroup_from_current(); +#endif For
bpf_global_percpu_ma, we also need to account the allocated memory to root memory cgroup just likebpf_global_madid, do we ? So it seems that we need to initializec->objcgearly inbpf_mem_alloc_percpu_init().Good point. Agree. the original behavior percpu non-fix-size mem allocation is to do
get_obj_cgroup_from_current()at init stage and charge to root memory cgroup, and we indeed should move the abovebpf_mem_alloc_percpu_init().Quoted 13 lines
Quoted 13 lines
+
for_each_possible_cpu(cpu){
+ cc =per_cpu_ptr(pcc, cpu);
+ c = &cc->cache[i];
+ if (cpu == 0 &&c->unit_size)
+ goto out;(quoted diff) +8
-0+ + c->unit_size = unit_size; + c->objcg = objcg; + c->percpu_size = percpu_size; + c->tgt = c; + + init_refill_work(c); + prefill_mem_cache(c, cpu);Quoted 8 lines
Quoted 6 lines
+
+ if (cpu == 0) {
+ err =check_obj_size(c, i);
+ if (err) {
+drain_mem_cache(c);
+memset(c, 0, sizeof(*c));I also forgot about
c->objcg. objcg may be leaked if we domemset()here.The objcg gets a reference at init
bpf_mem_alloc_init()stage and released atbpf_mem_alloc_destroy(). Forbpf_global_ma, if there is a failure, indeedbpf_mem_alloc_destroy()will be called and the referencec->objcgwill be released.So if we move
get_obj_cgroup_from_current()tobpf_mem_alloc_percpu_init()stage, we should be okay here.
Quoted 6 lines
Quoted 4 lines
BTW, is
check_obj_size()really necessary here? My answer is no since as you mentioned, thesize->cache_indexis pretty stable, socheck_obj_size()should not return error in such cases. What do you think?How about the following change on top of this patch?
I think the patch below is fine. Before the change, objcg is a per-bpf_mem_alloc object, but the implementation doesn't make it being explicit. The change below make the objcg being a a per-bpf_mem_alloc object.
Quoted 91 lines
include/linux/bpf_mem_alloc.h +1
-0@@ -11,6 +11,7 @@ struct bpf_mem_caches; struct bpf_mem_alloc { struct bpf_mem_caches __percpu *caches; struct bpf_mem_cache __percpu *cache; + struct obj_cgroup *objcg; bool percpu; struct work_struct work; };kernel/bpf/memalloc.c +2
-0@@ -553,6 +553,8 @@ int bpf_mem_alloc_init(struct bpf_mem_alloc *ma, int size, bool percpu) if (memcg_bpf_enabled()) objcg = get_obj_cgroup_from_current(); #endif + ma->objcg = objcg; + for_each_possible_cpu(cpu) { c = per_cpu_ptr(pc, cpu);
c->unit_size=unit_size;kernel/bpf/memalloc.c +1
-0@@ -573,6 +575,7 @@ int bpf_mem_alloc_init(struct bpf_mem_alloc *ma, int size, bool percpu) #ifdef CONFIG_MEMCG_KMEM objcg = get_obj_cgroup_from_current(); #endif + ma->objcg = objcg; for_each_possible_cpu(cpu) { cc = per_cpu_ptr(pcc, cpu);for (i = 0; i <
NUM_CACHES; i++) {kernel/bpf/memalloc.c +0
-0@@ -637,6 +640,12 @@ __init int bpf_mem_alloc_percpu_init(struct bpf_mem_alloc *ma)
ma->caches= pcc;
ma->percpu= true;kernel/bpf/memalloc.c +6
-0+ +#ifdef CONFIG_MEMCG_KMEM + ma->objcg = get_obj_cgroup_from_current(); +#else + ma->objcg = NULL; +#endif return 0;
}kernel/bpf/memalloc.c +0
-0@@ -656,10 +665,8 @@ int bpf_mem_alloc_percpu_unit_init(struct bpf_mem_alloc *ma, int size)
pcc =ma->caches;
unit_size= sizes[i];
+ objcg =ma->objcg;kernel/bpf/memalloc.c +0
-3-#ifdef CONFIG_MEMCG_KMEM - objcg = get_obj_cgroup_from_current(); -#endif
for_each_possible_cpu(cpu){
cc =per_cpu_ptr(pcc, cpu);
c = &cc->cache[i];kernel/bpf/memalloc.c +0
-0@@ -799,9 +806,8 @@ void bpf_mem_alloc_destroy(struct bpf_mem_alloc *ma)
rcu_in_progress+=atomic_read(&c->call_rcu_ttrace_in_progress);rcu_in_progress+=atomic_read(&c->call_rcu_in_progress);
}
- /* objcg is the same across cpus */ - if (c->objcg)
-obj_cgroup_put(c->objcg); + if (ma->objcg)
+obj_cgroup_put(ma->objcg);destroy_mem_alloc(ma, rcu_in_progress); }
if (ma->caches) {kernel/bpf/memalloc.c +0
-0@@ -817,8 +823,8 @@ void bpf_mem_alloc_destroy(struct bpf_mem_alloc *ma)
rcu_in_progress+=atomic_read(&c->call_rcu_in_progress);
}
}
- if (c->objcg)
-obj_cgroup_put(c->objcg); + if (ma->objcg)
+obj_cgroup_put(ma->objcg);destroy_mem_alloc(ma, rcu_in_progress); }
}
Quoted 2 lines
I still think
check_obj_sizefor percpu allocation is not needed. But I guess we can address that issue later on.
You are right. check_obj_size() is not needed for per-cpu allocation, so it is OK to just remove it. I also remove check_obj_size() for kmalloc allocation in [1].
[1]:
https://lore.kernel.org/bpf/[email protected]/
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Review
Quoted 11 lines
+ goto out;
+ }
+ }
+ }(quoted diff) +5
-0+ +out: + return err; +} +.
kernel test robot
build botRe: [PATCH bpf-next v3 2/6] bpf: Allow per unit prefill for non-fix-size percpu memory allocator
In reply to Yonghong Songkernel test robot noticed the following build warnings:
Hi Yonghong,
kernel test robot noticed the following build warnings:
[auto build test WARNING on bpf-next/master]
url: https://github.com/intel-lab-lkp/linux/commits/Yonghong-Song/bpf-Avoid-unnecessary-extra-percpu-memory-allocation/20231216-103310 base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master patch link: https://lore.kernel.org/r/20231216023015.3741144-1-yonghong.song%40linux.dev patch subject: [PATCH bpf-next v3 2/6] bpf: Allow per unit prefill for non-fix-size percpu memory allocator config: x86_64-randconfig-003-20231216 (https://download.01.org/0day-ci/archive/20231216/[email protected]/config) compiler: clang version 16.0.4 (https://github.com/llvm/llvm-project.git ae42196bc493ffe877a7e3dff8be32035dea4d07) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231216/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> kernel/bpf/memalloc.c:665:14: warning: variable 'objcg' is uninitialized when used here [-Wuninitialized]
c->objcg = objcg;
^~~~~
kernel/bpf/memalloc.c:642:26: note: initialize the variable 'objcg' to silence this warning
struct obj_cgroup *objcg;
^
= NULL1 warning generated.
vim +/objcg +665 kernel/bpf/memalloc.c
637
638 int bpf_mem_alloc_percpu_unit_init(struct bpf_mem_alloc *ma, int size)
639 {
640 int cpu, i, err = 0, unit_size, percpu_size;
641 struct bpf_mem_caches *cc, __percpu *pcc;
642 struct obj_cgroup *objcg;
643 struct bpf_mem_cache *c;
644
645 i = bpf_mem_cache_idx(size);
646 if (i < 0)
647 return -EINVAL;
648
649 /* room for llist_node and per-cpu pointer */
650 percpu_size = LLIST_NODE_SZ + sizeof(void *);
651
652 pcc = ma->caches;
653 unit_size = sizes[i];
654
655 #ifdef CONFIG_MEMCG_KMEM
656 objcg = get_obj_cgroup_from_current();
657 #endif
658 for_each_possible_cpu(cpu) {
659 cc = per_cpu_ptr(pcc, cpu);
660 c = &cc->cache[i];
661 if (cpu == 0 && c->unit_size)
662 goto out;
663
664 c->unit_size = unit_size;
> 665 c->objcg = objcg;
666 c->percpu_size = percpu_size;
667 c->tgt = c;
668
669 init_refill_work(c);
670 prefill_mem_cache(c, cpu);
671
672 if (cpu == 0) {
673 err = check_obj_size(c, i);
674 if (err) {
675 drain_mem_cache(c);
676 memset(c, 0, sizeof(*c));
677 goto out;
678 }
679 }
680 }
681
682 out:
683 return err;
684 }
685
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Yonghong Song
reviewerPATCH v3 3/6 bpf: Refill only one percpu element in memalloc
In reply to Yonghong SongTypically for percpu map element or data structure, once allocated, most operations are lookup or in-place update. Deletion are really rare. Currently, for percpu data strcture, 4 elements will be refilled if the size is <= 256. Let us just do with one element for percpu data. For example, for size 256 and 128 cpus, the potential saving will be 3 * 256 * 128 * 128 = 12MB.
--- kernel/bpf/memalloc.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
kernel/bpf/memalloc.c +9 -4
@@ -485,11 +485,16 @@ static void init_refill_work(struct bpf_mem_cache *c)static void prefill_mem_cache (struct bpf_mem_cache *c,int cpu) { -/* To avoid consuming memory assume that 1st run of bpf -* prog won't be doing more than 4 map_update_elem from -* irq disabled region +int cnt =1 ; + +/* To avoid consuming memory, for non-percpu allocation, assume that +* 1st run of bpf prog won't be doing more than 4 map_update_elem from +* irq disabled region if unit size is less than or equal to 256. +* For all other cases, let us just do one allocation. */ -alloc_bulk (c, c->unit_size <=256 ?4 :1 ,cpu_to_node (cpu), false); +if (!c->percpu_size && c->unit_size <=256 ) + cnt =4 ; +alloc_bulk (c, cnt,cpu_to_node (cpu), false); }static int check_obj_size (struct bpf_mem_cache *c,unsigned int idx)
Signature · 1 line
2.34.1
Re: [PATCH bpf-next v3 3/6] bpf: Refill only one percpu element in memalloc
Quoted 8 lines · Yonghong Song
Typically for percpu map element or data structure, once allocated, most operations are lookup or in-place update. Deletion are really rare. Currently, for percpu data strcture, 4 elements will be refilled if the size is <= 256. Let us just do with one element for percpu data. For example, for size 256 and 128 cpus, the potential saving will be 3 * 256 * 128 * 128 = 12MB.
Signed-off-by Yonghong Song <[email protected]>
Yonghong Song
reviewerPATCH v3 4/6 bpf: Limit up to 512 bytes for bpf_global_percpu_ma allocation
In reply to Yonghong SongFor percpu data structure allocation with bpf_global_percpu_ma, the maximum data size is 4K. But for a system with large number of cpus, bigger data size (e.g., 2K, 4K) might consume a lot of memory. For example, the percpu memory consumption with unit size 2K and 1024 cpus will be 2K * 1K * 1k = 2GB memory.
We should discourage such usage. Let us limit the maximum data size to be 512 for bpf_global_percpu_ma allocation.
--- kernel/bpf/verifier.c | 8 ++++++++ 1 file changed, 8 insertions(+)
kernel/bpf/verifier.c +8 -0
@@ -192,6 +192,8 @@ struct bpf_verifier_stack_elem { POISON_POINTER_DELTA))#define BPF_MAP_PTR (X ) ((struct bpf_map *)((X) & ~BPF_MAP_PTR_UNPRIV)) +#define BPF_GLOBAL_PERCPU_MA_MAX_SIZE 512 +static int acquire_reference_state (struct bpf_verifier_env *env,int insn_idx);static int release_reference (struct bpf_verifier_env *env,int ref_obj_id);static void invalidate_non_owning_refs (struct bpf_verifier_env *env);
@@ -12083,6 +12085,12 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,if (!bpf_global_percpu_ma_set)return -ENOMEM ; +if (ret_t->size >BPF_GLOBAL_PERCPU_MA_MAX_SIZE ) { +verbose (env,"bpf_percpu_obj_new type size (%d) is greater than %d\n" , + ret_t->size,BPF_GLOBAL_PERCPU_MA_MAX_SIZE ); +return -EINVAL ; + } +mutex_lock (&bpf_percpu_ma_lock); err =bpf_mem_alloc_percpu_unit_init (&bpf_global_percpu_ma, ret_t->size);mutex_unlock (&bpf_percpu_ma_lock);
Signature · 1 line
2.34.1
Re: [PATCH bpf-next v3 4/6] bpf: Limit up to 512 bytes for bpf_global_percpu_ma allocation
Quoted 11 lines · Yonghong Song
For percpu data structure allocation with
bpf_global_percpu_ma, the maximum data size is 4K. But for a system with large number of cpus, bigger data size (e.g., 2K, 4K) might consume a lot of memory. For example, the percpu memory consumption with unit size 2K and 1024 cpus will be 2K * 1K * 1k = 2GB memory.We should discourage such usage. Let us limit the maximum data size to be 512 for
bpf_global_percpu_maallocation.Signed-off-by Yonghong Song <[email protected]>
Make sense to me, so
Yonghong Song
reviewerPATCH v3 5/6 selftests/bpf: Cope with 512 bytes limit with bpf_global_percpu_ma
In reply to Yonghong SongIn the previous patch, the maximum data size for bpf_global_percpu_ma is 512 bytes. This breaks selftest test_bpf_ma. The test is adjusted in two aspects:
- Since the maximum allowed data size for bpf_global_percpu_ma is
512, remove all tests beyond that, names sizes 1024, 2048 and 4096.
- Previously the percpu data size is bucket_size - 8 in order to avoid percpu allocation into the next bucket. This patch removed such data size adjustment thanks to Patch 1.
Also, a better way to generate BTF type is used than adding a member to the value struct.
--- .../selftests/bpf/prog_tests/test_bpf_ma.c | 20 +++++++---- .../testing/selftests/bpf/progs/test_bpf_ma.c | 34 +++++++++---------- 2 files changed, 30 insertions(+), 24 deletions(-)
tools/testing/selftests/bpf/prog_tests/test_bpf_ma.c +13 -7
@@ -14,7 +14,8 @@ static void do_bpf_ma_test(const char *name)struct test_bpf_ma *skel;struct bpf_program *prog;struct btf *btf; -int i, err; +int i, err, id; +char tname[32 ]; skel =test_bpf_ma__open ();if (!ASSERT_OK_PTR (skel,"open" ))
@@ -25,16 +26,21 @@ static void do_bpf_ma_test(const char *name) goto out;for (i =0 ; i <ARRAY_SIZE (skel->rodata->data_sizes); i++) { -char name[32 ]; -int id; - -snprintf (name,sizeof (name),"bin_data_%u" , skel->rodata->data_sizes[i]); - id =btf__find_by_name_kind (btf, name,BTF_KIND_STRUCT ); -if (!ASSERT_GT (id,0 ,"bin_data" )) +snprintf (tname,sizeof (tname),"bin_data_%u" , skel->rodata->data_sizes[i]); + id =btf__find_by_name_kind (btf, tname,BTF_KIND_STRUCT ); +if (!ASSERT_GT (id,0 , tname)) goto out; skel->rodata->data_btf_ids[i] = id; } +for (i =0 ; i <ARRAY_SIZE (skel->rodata->percpu_data_sizes); i++) { +snprintf (tname,sizeof (tname),"percpu_bin_data_%u" , skel->rodata->percpu_data_sizes[i]); + id =btf__find_by_name_kind (btf, tname,BTF_KIND_STRUCT ); +if (!ASSERT_GT (id,0 , tname)) + goto out; + skel->rodata->percpu_data_btf_ids[i] = id; + } + prog =bpf_object__find_program_by_name (skel->obj, name);if (!ASSERT_OK_PTR (prog,"invalid prog name" )) goto out;
tools/testing/selftests/bpf/progs/test_bpf_ma.c +17 -17
@@ -20,6 +20,9 @@ char _license[] SEC("license") = "GPL";const unsigned int data_sizes[] = {8 ,16 ,32 ,64 ,96 ,128 ,192 ,256 ,512 ,1024 ,2048 ,4096 };const volatile unsigned int data_btf_ids[ARRAY_SIZE (data_sizes)] = {}; +const unsigned int percpu_data_sizes[] = {8 ,16 ,32 ,64 ,96 ,128 ,192 ,256 ,512 }; +const volatile unsigned int percpu_data_btf_ids[ARRAY_SIZE (data_sizes)] = {}; +int err =0 ;int pid =0 ;
@@ -27,10 +30,10 @@ int pid = 0; struct bin_data_##_size { \char data[_size -sizeof (void *)]; \ }; \ +/* See Commit 5d8d6634ccc, force btf generation for type bin_data_##_size */ \ + struct bin_data_##_size *__bin_data_##_size; \ struct map_value_##_size { \ struct bin_data_##_size __kptr * data; \ -/* To emit BTF info for bin_data_xx */ \ - struct bin_data_##_size not_used; \ }; \struct { \ __uint(type, BPF_MAP_TYPE_ARRAY); \
@@ -40,8 +43,12 @@ int pid = 0; } array_##_size SEC(".maps")#define DEFINE_ARRAY_WITH_PERCPU_KPTR (_size) \ + struct percpu_bin_data_##_size { \ + char data[_size]; \ + }; \ + struct percpu_bin_data_##_size *__percpu_bin_data_##_size; \ struct map_value_percpu_##_size { \ - struct bin_data_##_size __percpu_kptr * data; \ + struct percpu_bin_data_##_size __percpu_kptr * data; \ }; \ struct { \ __uint(type, BPF_MAP_TYPE_ARRAY); \
@@ -114,7 +121,7 @@ static __always_inline void batch_percpu_alloc(struct bpf_map *map, unsigned intreturn ; }/* per-cpu allocator may not be able to refill in time */ - new =bpf_percpu_obj_new_impl (data_btf_ids[idx],NULL ); + new =bpf_percpu_obj_new_impl (percpu_data_btf_ids[idx],NULL );if (!new)continue ;
@@ -179,7 +186,7 @@ DEFINE_ARRAY_WITH_KPTR(1024);DEFINE_ARRAY_WITH_KPTR (2048 );DEFINE_ARRAY_WITH_KPTR (4096 ); -/* per-cpu kptr doesn't support bin_data_8 which is a zero-sized array */ +DEFINE_ARRAY_WITH_PERCPU_KPTR (8 );DEFINE_ARRAY_WITH_PERCPU_KPTR (16 );DEFINE_ARRAY_WITH_PERCPU_KPTR (32 );DEFINE_ARRAY_WITH_PERCPU_KPTR (64 );
@@ -188,9 +195,6 @@ DEFINE_ARRAY_WITH_PERCPU_KPTR(128);DEFINE_ARRAY_WITH_PERCPU_KPTR (192 );DEFINE_ARRAY_WITH_PERCPU_KPTR (256 );DEFINE_ARRAY_WITH_PERCPU_KPTR (512 ); -DEFINE_ARRAY_WITH_PERCPU_KPTR (1024 ); -DEFINE_ARRAY_WITH_PERCPU_KPTR (2048 ); -DEFINE_ARRAY_WITH_PERCPU_KPTR (4096 ); SEC("?fentry/" SYS_PREFIX "sys_nanosleep")int test_batch_alloc_free (void *ctx)
@@ -248,9 +252,10 @@ int test_batch_percpu_alloc_free(void *ctx)if ((u32 )bpf_get_current_pid_tgid () != pid)return 0 ; -/* Alloc 128 16-bytes per-cpu objects in batch to trigger refilling, -* then free 128 16-bytes per-cpu objects in batch to trigger freeing. +/* Alloc 128 8-bytes per-cpu objects in batch to trigger refilling, +* then free 128 8-bytes per-cpu objects in batch to trigger freeing. */ +CALL_BATCH_PERCPU_ALLOC_FREE (8 ,128 ,0 );CALL_BATCH_PERCPU_ALLOC_FREE (16 ,128 ,1 );CALL_BATCH_PERCPU_ALLOC_FREE (32 ,128 ,2 );CALL_BATCH_PERCPU_ALLOC_FREE (64 ,128 ,3 );
@@ -259,9 +264,6 @@ int test_batch_percpu_alloc_free(void *ctx)CALL_BATCH_PERCPU_ALLOC_FREE (192 ,128 ,6 );CALL_BATCH_PERCPU_ALLOC_FREE (256 ,128 ,7 );CALL_BATCH_PERCPU_ALLOC_FREE (512 ,64 ,8 ); -CALL_BATCH_PERCPU_ALLOC_FREE (1024 ,32 ,9 ); -CALL_BATCH_PERCPU_ALLOC_FREE (2048 ,16 ,10 ); -CALL_BATCH_PERCPU_ALLOC_FREE (4096 ,8 ,11 );return 0 ; }
@@ -272,9 +274,10 @@ int test_percpu_free_through_map_free(void *ctx)if ((u32 )bpf_get_current_pid_tgid () != pid)return 0 ; -/* Alloc 128 16-bytes per-cpu objects in batch to trigger refilling, +/* Alloc 128 8-bytes per-cpu objects in batch to trigger refilling, * then free these object through map free. */ +CALL_BATCH_PERCPU_ALLOC (8 ,128 ,0 );CALL_BATCH_PERCPU_ALLOC (16 ,128 ,1 );CALL_BATCH_PERCPU_ALLOC (32 ,128 ,2 );CALL_BATCH_PERCPU_ALLOC (64 ,128 ,3 );
@@ -283,9 +286,6 @@ int test_percpu_free_through_map_free(void *ctx)CALL_BATCH_PERCPU_ALLOC (192 ,128 ,6 );CALL_BATCH_PERCPU_ALLOC (256 ,128 ,7 );CALL_BATCH_PERCPU_ALLOC (512 ,64 ,8 ); -CALL_BATCH_PERCPU_ALLOC (1024 ,32 ,9 ); -CALL_BATCH_PERCPU_ALLOC (2048 ,16 ,10 ); -CALL_BATCH_PERCPU_ALLOC (4096 ,8 ,11 );return 0 ; }
Signature · 1 line
2.34.1
Re: [PATCH bpf-next v3 5/6] selftests/bpf: Cope with 512 bytes limit with bpf_global_percpu_ma
Quoted 5 lines · Yonghong Song
In the previous patch, the maximum data size for
bpf_global_percpu_mais 512 bytes. This breaks selftesttest_bpf_ma. The test is adjusted in two aspects:
- Since the maximum allowed data size forbpf_global_percpu_mais512, remove all tests beyond that, names sizes 1024, 2048 and 4096.
Quoted 8 lines
- Previously the percpu data size is
bucket_size- 8 in order to avoid percpu allocation into the next bucket. This patch removed such data size adjustment thanks to Patch 1.Also, a better way to generate BTF type is used than adding a member to the value struct.
Signed-off-by Yonghong Song <[email protected]>
And thanks for the update.
Yonghong Song
reviewerPATCH v3 6/6 selftests/bpf: Add a selftest with > 512-byte percpu allocation size
In reply to Yonghong SongAdd a selftest to capture the verification failure when the allocation size is greater than 512.
--- .../selftests/bpf/progs/percpu_alloc_fail.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
tools/testing/selftests/bpf/progs/percpu_alloc_fail.c +18 -0
@@ -17,6 +17,10 @@ struct val_with_rb_root_t {struct bpf_spin_lock lock; }; +struct val_600b_t { +char b[600 ]; +}; +struct elem {long sum; struct val_t __percpu_kptr *pc;
@@ -161,4 +165,18 @@ int BPF_PROG(test_array_map_7)return 0 ; } +SEC("?fentry.s/bpf_fentry_test1") +__failure __msg("bpf_percpu_obj_new type size (600) is greater than 512") +int BPF_PROG(test_array_map_8) +{ + struct val_600b_t __percpu_kptr *p; + + p = bpf_percpu_obj_new(struct val_600b_t); +if (!p) +return 0 ; + +bpf_percpu_obj_drop (p); +return 0 ; +} + char _license[] SEC("license") = "GPL";
Signature · 1 line
2.34.1
Re: [PATCH bpf-next v3 6/6] selftests/bpf: Add a selftest with > 512-byte percpu allocation size
Quoted 4 lines · Yonghong Song
Add a selftest to capture the verification failure when the allocation size is greater than 512.
Signed-off-by Yonghong Song <[email protected]>