Allow struct_ops maps with a large number of programs
Subsystems (7)
v2 cover Allow struct_ops maps with a large number of programs
From: Kui-Feng Lee <[email protected]>
The BPF struct_ops previously only allowed for one page to be used for the trampolines of all links in a map. However, we have recently run out of space due to the large number of BPF program links. By allocating additional pages when we exhaust an existing page, we can accommodate more links in a single map.
The variable st_map->image has been changed to st_map->image_pages, and its type has been changed to an array of pointers to buffers of PAGE_SIZE. The array is dynamically resized and additional pages are allocated when all existing pages are exhausted.
The test case loads a struct_ops maps having 40 programs. Their trampolines takes about 6.6k+ bytes over 1.5 pages on x86.
Kui-Feng Lee (3):
bpf, net: validate struct_ops when updating value.
bpf: struct_ops supports more than one page for trampolines. selftests/bpf: Test struct_ops maps with a large number of program
links.
kernel/bpf/bpf_struct_ops.c | 123 ++++++++++++------ net/ipv4/tcp_cong.c | 6 +- .../selftests/bpf/bpf_testmod/bpf_testmod.h | 44 +++++++ .../prog_tests/test_struct_ops_multi_pages.c | 31 +++++ .../bpf/progs/struct_ops_multi_pages.c | 102 +++++++++++++++ 5 files changed, 263 insertions(+), 43 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/test_struct_ops_multi_pages.c create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_multi_pages.c
Signature · 1 line
2.34.1
PATCH v2 1/3 bpf, net: validate struct_ops when updating value.
From: Kui-Feng Lee <[email protected]>
Perform all validations when updating values of struct_ops maps. Doing validation in st_ops->reg() and st_ops->update() is not necessary anymore. However, tcp_register_congestion_control() has been called in various places. It still needs to do validations.
--- kernel/bpf/bpf_struct_ops.c | 11 ++++++----- net/ipv4/tcp_cong.c | 6 +----- 2 files changed, 7 insertions(+), 10 deletions(-)
kernel/bpf/bpf_struct_ops.c +6 -5
@@ -667,13 +667,14 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, *(unsigned long *)(udata + moff) = prog->aux->id; } +if (st_ops->validate) { + err = st_ops->validate (kdata); +if (err) + goto reset_unlock; + } +if (st_map->map.map_flags &BPF_F_LINK ) { err =0 ; -if (st_ops->validate) { - err = st_ops->validate (kdata); -if (err) - goto reset_unlock; - }arch_protect_bpf_trampoline (st_map->image,PAGE_SIZE );/* Let bpf_link handle registration & unregistration. *
net/ipv4/tcp_cong.c +1 -5
@@ -146,11 +146,7 @@ EXPORT_SYMBOL_GPL(tcp_unregister_congestion_control);int tcp_update_congestion_control (struct tcp_congestion_ops *ca,struct tcp_congestion_ops *old_ca) {struct tcp_congestion_ops *existing; -int ret; - - ret =tcp_validate_congestion_control (ca); -if (ret) -return ret; +int ret =0 ; ca->key =jhash (ca->name,sizeof (ca->name),strlen (ca->name));
Signature · 1 line
2.34.1
PATCH v2 2/3 bpf: struct_ops supports more than one page for trampolines.
In reply to [email protected]From: Kui-Feng Lee <[email protected]>
The BPF struct_ops previously only allowed for one page to be used for the trampolines of all links in a map. However, we have recently run out of space due to the large number of BPF program links. By allocating additional pages when we exhaust an existing page, we can accommodate more links in a single map.
The variable st_map->image has been changed to st_map->image_pages, and its type has been changed to an array of pointers to buffers of PAGE_SIZE. Every struct_ops map can have MAX_IMAGE_PAGES (8) pages for trampolines at most.
--- kernel/bpf/bpf_struct_ops.c | 112 +++++++++++++++++++++++++----------- 1 file changed, 79 insertions(+), 33 deletions(-)
kernel/bpf/bpf_struct_ops.c +79 -33
@@ -18,6 +18,8 @@ struct bpf_struct_ops_value {char data[]____cacheline_aligned_in_smp ; }; +#define MAX_TRAMP_IMAGE_PAGES 8 +struct bpf_struct_ops_map {struct bpf_map map;struct rcu_head rcu;
@@ -30,12 +32,11 @@ struct bpf_struct_ops_map {*/ struct bpf_link **links;u32 links_cnt; -/* image is a page that has all the trampolines +u32 image_pages_cnt; +/* image_pages is an array of pages that has all the trampolines * that stores the func args before calling the bpf_prog. -* A PAGE_SIZE "image" is enough to store all trampoline for -* "links[]". */ -void *image; +void *image_pages[MAX_TRAMP_IMAGE_PAGES ];/* The owner moduler's btf. */ struct btf *btf;/* uvalue->data stores the kernel struct
@@ -456,6 +457,41 @@ static void bpf_struct_ops_map_put_progs(struct bpf_struct_ops_map *st_map) } } +static void bpf_struct_ops_map_free_image (struct bpf_struct_ops_map *st_map) +{ +int i; +void *image; + +bpf_jit_uncharge_modmem (PAGE_SIZE * st_map->image_pages_cnt); +for (i =0 ; i < st_map->image_pages_cnt; i++) { + image = st_map->image_pages[i]; +arch_free_bpf_trampoline (image,PAGE_SIZE ); + } + st_map->image_pages_cnt =0 ; +} + +static void *bpf_struct_ops_map_inc_image (struct bpf_struct_ops_map *st_map) +{ +void *image; +int err; + +if (st_map->image_pages_cnt >=MAX_TRAMP_IMAGE_PAGES ) +return ERR_PTR (-E2BIG ); + + err =bpf_jit_charge_modmem (PAGE_SIZE ); +if (err) +return ERR_PTR (err); + + image =arch_alloc_bpf_trampoline (PAGE_SIZE ); +if (!image) { +bpf_jit_uncharge_modmem (PAGE_SIZE ); +return ERR_PTR (-ENOMEM ); + } + st_map->image_pages[st_map->image_pages_cnt++] = image; + +return image; +} +static int check_zero_holes (const struct btf *btf,const struct btf_type *t,void *data) {const struct btf_member *member;
@@ -531,10 +567,10 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key,const struct btf_type *module_type;const struct btf_member *member;const struct btf_type *t = st_ops_desc->type; +void *image =NULL , *image_end =NULL ;struct bpf_tramp_links *tlinks;void *udata, *kdata;int prog_fd, err; -void *image, *image_end;u32 i;if (flags)
@@ -573,15 +609,14 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, udata = &uvalue->data; kdata = &kvalue->data; - image = st_map->image; - image_end = st_map->image +PAGE_SIZE ; module_type =btf_type_by_id (btf_vmlinux, st_ops_ids[IDX_MODULE_ID ]);for_each_member (i, t, member) {const struct btf_type *mtype, *ptype;struct bpf_prog *prog;struct bpf_tramp_link *link; -u32 moff; +u32 moff, tflags; +int tsize; moff =__btf_member_bit_offset (t, member) /8 ; ptype =btf_type_resolve_ptr (st_map->btf, member->type,NULL );
@@ -653,10 +688,38 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, &bpf_struct_ops_link_lops, prog); st_map->links[i] = &link->link; - err =bpf_struct_ops_prepare_trampoline (tlinks, link, - &st_ops->func_models[i], - *(void **)(st_ops->cfi_stubs + moff), - image, image_end); + tflags =BPF_TRAMP_F_INDIRECT ; +if (st_ops->func_models[i].ret_size >0 ) + tflags |=BPF_TRAMP_F_RET_FENTRY_RET ; + +/* Compute the size of the trampoline */ + tlinks[BPF_TRAMP_FENTRY ].links[0 ] = link; + tlinks[BPF_TRAMP_FENTRY ].nr_links =1 ; + tsize =arch_bpf_trampoline_size (&st_ops->func_models[i], + tflags, tlinks,NULL ); +if (tsize <0 ) { + err = tsize; + goto reset_unlock; + } + +/* Allocate pages */ +if (tsize > (unsigned long )image_end - (unsigned long )image) { +if (tsize >PAGE_SIZE ) { + err = -E2BIG ; + goto reset_unlock; + } + image =bpf_struct_ops_map_inc_image (st_map); +if (IS_ERR (image)) { + err =PTR_ERR (image); + goto reset_unlock; + } + image_end = image +PAGE_SIZE ; + } + + err =arch_prepare_bpf_trampoline (NULL , image, image_end, + &st_ops->func_models[i], + tflags, tlinks, + *(void **)(st_ops->cfi_stubs + moff));if (err <0 ) goto reset_unlock;
@@ -672,10 +735,11 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key,if (err) goto reset_unlock; } +for (i =0 ; i < st_map->image_pages_cnt; i++) +arch_protect_bpf_trampoline (st_map->image_pages[i],PAGE_SIZE );if (st_map->map.map_flags &BPF_F_LINK ) { err =0 ; -arch_protect_bpf_trampoline (st_map->image,PAGE_SIZE );/* Let bpf_link handle registration & unregistration. * * Pair with smp_load_acquire() during lookup_elem().
@@ -684,7 +748,6 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, goto unlock; } -arch_protect_bpf_trampoline (st_map->image,PAGE_SIZE ); err = st_ops->reg (kdata);if (likely (!err)) {/* This refcnt increment on the map here after
@@ -707,9 +770,9 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key,* there was a race in registering the struct_ops (under the same name) to * a sub-system through different struct_ops's maps. */ -arch_unprotect_bpf_trampoline (st_map->image,PAGE_SIZE ); reset_unlock: +bpf_struct_ops_map_free_image (st_map);bpf_struct_ops_map_put_progs (st_map);memset (uvalue,0 , map->value_size);memset (kvalue,0 , map->value_size);
@@ -776,10 +839,7 @@ static void __bpf_struct_ops_map_free(struct bpf_map *map)if (st_map->links)bpf_struct_ops_map_put_progs (st_map);bpf_map_area_free (st_map->links); -if (st_map->image) { -arch_free_bpf_trampoline (st_map->image,PAGE_SIZE ); -bpf_jit_uncharge_modmem (PAGE_SIZE ); - } +bpf_struct_ops_map_free_image (st_map);bpf_map_area_free (st_map->uvalue);bpf_map_area_free (st_map); }
@@ -889,20 +949,6 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) st_map->st_ops_desc = st_ops_desc; map = &st_map->map; - ret =bpf_jit_charge_modmem (PAGE_SIZE ); -if (ret) - goto errout_free; - - st_map->image =arch_alloc_bpf_trampoline (PAGE_SIZE ); -if (!st_map->image) { -/* __bpf_struct_ops_map_free() uses st_map->image as flag -* for "charged or not". In this case, we need to unchange -* here. -*/ -bpf_jit_uncharge_modmem (PAGE_SIZE ); - ret = -ENOMEM ; - goto errout_free; - } st_map->uvalue =bpf_map_area_alloc (vt->size,NUMA_NO_NODE ); st_map->links_cnt =btf_type_vlen (t); st_map->links =
Signature · 1 line
2.34.1
PATCH v2 3/3 selftests/bpf: Test struct_ops maps with a large number of program links.
In reply to [email protected]From: Kui-Feng Lee <[email protected]>
Create and load a struct_ops map with a large number of programs to generate trampolines taking a size over multiple pages. The map includes 40 programs. Their trampolines takes 6.6k+, more than 1.5 pages, on x86.
--- .../selftests/bpf/bpf_testmod/bpf_testmod.h | 44 ++++++++ .../prog_tests/test_struct_ops_multi_pages.c | 31 ++++++ .../bpf/progs/struct_ops_multi_pages.c | 102 ++++++++++++++++++ 3 files changed, 177 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/test_struct_ops_multi_pages.c create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_multi_pages.c
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.h +44 -0
@@ -35,6 +35,50 @@ struct bpf_testmod_ops {void (*test_2)(int a,int b);/* Used to test nullable arguments. */ int (*test_maybe_null)(int dummy,struct task_struct *task); + +/* The following pointers are used to test the maps having multiple +* pages of trampolines. +*/ +int (*tramp_1)(int value); +int (*tramp_2)(int value); +int (*tramp_3)(int value); +int (*tramp_4)(int value); +int (*tramp_5)(int value); +int (*tramp_6)(int value); +int (*tramp_7)(int value); +int (*tramp_8)(int value); +int (*tramp_9)(int value); +int (*tramp_10)(int value); +int (*tramp_11)(int value); +int (*tramp_12)(int value); +int (*tramp_13)(int value); +int (*tramp_14)(int value); +int (*tramp_15)(int value); +int (*tramp_16)(int value); +int (*tramp_17)(int value); +int (*tramp_18)(int value); +int (*tramp_19)(int value); +int (*tramp_20)(int value); +int (*tramp_21)(int value); +int (*tramp_22)(int value); +int (*tramp_23)(int value); +int (*tramp_24)(int value); +int (*tramp_25)(int value); +int (*tramp_26)(int value); +int (*tramp_27)(int value); +int (*tramp_28)(int value); +int (*tramp_29)(int value); +int (*tramp_30)(int value); +int (*tramp_31)(int value); +int (*tramp_32)(int value); +int (*tramp_33)(int value); +int (*tramp_34)(int value); +int (*tramp_35)(int value); +int (*tramp_36)(int value); +int (*tramp_37)(int value); +int (*tramp_38)(int value); +int (*tramp_39)(int value); + int (*tramp_40)(int value); };#endif /* _BPF_TESTMOD_H */
tools/testing/selftests/bpf/prog_tests/test_struct_ops_multi_pages.c new file +31 -0
@@ -0,0 +1,31 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ +#include <test_progs.h> + +#include "struct_ops_multi_pages.skel.h" + +static void do_struct_ops_multi_pages (void ) +{ +struct struct_ops_multi_pages *skel; +struct bpf_link *link; + +/* The size of all trampolines of skel->maps.multi_pages should be +* over 1 page (at least for x86). +*/ + skel =struct_ops_multi_pages__open_and_load (); +if (!ASSERT_OK_PTR (skel,"struct_ops_multi_pages_open_and_load" )) +return ; + + link =bpf_map__attach_struct_ops (skel->maps.multi_pages); +ASSERT_OK_PTR (link,"attach_multi_pages" ); + +bpf_link__destroy (link); +struct_ops_multi_pages__destroy (skel); +} + +void test_struct_ops_multi_pages (void ) +{ + +if (test__start_subtest ("multi_pages" )) +do_struct_ops_multi_pages (); +}
tools/testing/selftests/bpf/progs/struct_ops_multi_pages.c new file +102 -0
102 lines not on this page · raw message
Signature · 1 line
2.34.1
Quoted 8 lines · On 2/21/24 14:59, [email protected] wrote:
From: Kui-Feng Lee <[email protected]>
The BPF
struct_opspreviously only allowed for one page to be used for the trampolines of all links in a map. However, we have recently run out of space due to the large number of BPF program links. By allocating additional pages when we exhaust an existing page, we can accommodate more links in a single map.
Quoted 4 lines
The variable
st_map->imagehas been changed tost_map->image_pages, and its type has been changed to an array of pointers to buffers ofPAGE_SIZE. The array is dynamically resized and additional pages are allocated when all existing pages are exhausted.
Sorry for not updating this part about resizing. The array is not resized anymore. But, additional pages are allocated.
Quoted 19 lines
The test case loads a
struct_opsmaps having 40 programs. Their trampolines takes about 6.6k+ bytes over 1.5 pages on x86.Kui-Feng Lee (3):
bpf, net: validatestruct_opswhen updating value.
bpf:struct_opssupports more than one page for trampolines. selftests/bpf: Teststruct_opsmaps with a large number of program
links.kernel/bpf/bpf_struct_ops.c | 123 ++++++++++++------ net/ipv4/tcp_cong.c | 6 +- .../selftests/bpf/bpf_testmod/bpf_testmod.h | 44 +++++++ .../prog_tests/test_struct_ops_multi_pages.c | 31 +++++ .../bpf/progs/struct_ops_multi_pages.c | 102 +++++++++++++++ 5 files changed, 263 insertions(+), 43 deletions(-)create mode 100644
tools/testing/selftests/bpf/prog_tests/test_struct_ops_multi_pages.ccreate mode 100644tools/testing/selftests/bpf/progs/struct_ops_multi_pages.c
Martin KaFai Lau
reviewerRe: [PATCH bpf-next v2 2/3] bpf: struct_ops supports more than one page for trampolines.
In reply to [email protected]Quoted 60 lines · On 2/21/24 2:59 PM, [email protected] wrote:
kernel/bpf/bpf_struct_ops.c +25
-8@@ -531,10 +567,10 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key,const struct btf_type *module_type;const struct btf_member *member;const struct btf_type *t = st_ops_desc->type; +void *image =NULL , *image_end =NULL ;struct bpf_tramp_links *tlinks;void *udata, *kdata;int prog_fd, err; -void *image, *image_end;u32 i;if (flags)@@ -573,15 +609,14 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, udata = &uvalue->data; kdata = &kvalue->data; - image = st_map->image; - image_end = st_map->image +PAGE_SIZE ; module_type =btf_type_by_id (btf_vmlinux, st_ops_ids[IDX_MODULE_ID ]);for_each_member (i, t, member) {const struct btf_type *mtype, *ptype;struct bpf_prog *prog;struct bpf_tramp_link *link; -u32 moff; +u32 moff, tflags; +int tsize; moff =__btf_member_bit_offset (t, member) /8 ; ptype =btf_type_resolve_ptr (st_map->btf, member->type,NULL );@@ -653,10 +688,38 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, &bpf_struct_ops_link_lops, prog); st_map->links[i] = &link->link; - err =bpf_struct_ops_prepare_trampoline (tlinks, link, - &st_ops->func_models[i], - *(void **)(st_ops->cfi_stubs + moff), - image, image_end); + tflags =BPF_TRAMP_F_INDIRECT ; +if (st_ops->func_models[i].ret_size >0 ) + tflags |=BPF_TRAMP_F_RET_FENTRY_RET ; + +/* Compute the size of the trampoline */ + tlinks[BPF_TRAMP_FENTRY ].links[0 ] = link; + tlinks[BPF_TRAMP_FENTRY ].nr_links =1 ; + tsize =arch_bpf_trampoline_size (&st_ops->func_models[i], + tflags, tlinks,NULL ); +if (tsize <0 ) { + err = tsize; + goto reset_unlock; + } + +/* Allocate pages */ +if (tsize > (unsigned long )image_end - (unsigned long )image) { +if (tsize >PAGE_SIZE ) { + err = -E2BIG ; + goto reset_unlock; + } + image =bpf_struct_ops_map_inc_image (st_map); +if (IS_ERR (image)) {
kernel/bpf/bpf_struct_ops.c static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, · line 722
Quoted 10 lines
kernel/bpf/bpf_struct_ops.c +10
-0+ err =PTR_ERR (image); + goto reset_unlock; + } + image_end = image +PAGE_SIZE ; + } + + err =arch_prepare_bpf_trampoline (NULL , image, image_end, + &st_ops->func_models[i], + tflags, tlinks, + *(void **)(st_ops->cfi_stubs + moff));
I don't prefer to copy the BPF_TRAMP_F_* setting on tflags, tlinks, and the arch_*_trampoline_*() logic from bpf_struct_ops_prepare_trampoline() which is used by the bpf_dummy_ops for testing also. Considering struct_ops supports kernel module now, in the future, it is better to move bpf_dummy_ops out to the bpf_testmod somehow and avoid its bpf_struct_ops_prepare_trampoline() usage. For now, it is still better to keep bpf_struct_ops_prepare_trampoline() to be reusable by both.
Have you thought about the earlier suggestion in v1 to do arch_alloc_bpf_trampoline() in bpf_struct_ops_prepare_trampoline() instead of copying codes from bpf_struct_ops_prepare_trampoline() to bpf_struct_ops_map_update_elem()?
Something like this (untested code):
void *bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks,
struct bpf_tramp_link *link, const struct btf_func_model *model, void *stub_func, void *image, u32 *image_off,
bool allow_alloc)
{
u32 flags = BPF_TRAMP_F_INDIRECT;
void *new_image = NULL;
int size;
tlinks[BPF_TRAMP_FENTRY].links[0] = link;
tlinks[BPF_TRAMP_FENTRY].nr_links = 1;
if (model->ret_size > 0)
flags |= BPF_TRAMP_F_RET_FENTRY_RET;
size = arch_bpf_trampoline_size(model, flags, tlinks, NULL);
if (size < 0)
return ERR_PTR(size);
if (!image || size > PAGE_SIZE - *image_off) {
int err;
if (!allow_alloc)
return ERR_PTR(-E2BIG);
err = bpf_jit_charge_modmem(PAGE_SIZE);
if (err)
return ERR_PTR(err);
new_image = image = arch_alloc_bpf_trampoline(PAGE_SIZE);
if (!new_image) {
bpf_jit_uncharge_modmem(PAGE_SIZE);
return ERR_PTR(-ENOMEM);
}
*image_off = 0;
}
size = arch_prepare_bpf_trampoline(NULL, image + *image_off,
image + PAGE_SIZE,
model, flags, tlinks, stub_func);
if (size >= 0) {
*image_off += size;
return image;
}
if (new_image) {
bpf_jit_uncharge_modmem(PAGE_SIZE);
arch_free_bpf_trampoline(new_image, PAGE_SIZE);
}
return ERR_PTR(size);
}----
pw-bot: cr
Quoted 68 lines
kernel/bpf/bpf_struct_ops.c +4
-21if (err <0 ) goto reset_unlock;@@ -672,10 +735,11 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key,if (err) goto reset_unlock; } +for (i =0 ; i < st_map->image_pages_cnt; i++) +arch_protect_bpf_trampoline (st_map->image_pages[i],PAGE_SIZE );if (st_map->map.map_flags &BPF_F_LINK ) { err =0 ; -arch_protect_bpf_trampoline (st_map->image,PAGE_SIZE );/* Let bpf_link handle registration & unregistration. * * Pair with smp_load_acquire() during lookup_elem(). @@ -684,7 +748,6 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, goto unlock; } -arch_protect_bpf_trampoline (st_map->image,PAGE_SIZE ); err = st_ops->reg (kdata);if (likely (!err)) {/* This refcnt increment on the map here after @@ -707,9 +770,9 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key,* there was a race in registering the struct_ops (under the same name) to * a sub-system through different struct_ops's maps. */ -arch_unprotect_bpf_trampoline (st_map->image,PAGE_SIZE ); reset_unlock: +bpf_struct_ops_map_free_image (st_map);bpf_struct_ops_map_put_progs (st_map);memset (uvalue,0 , map->value_size);memset (kvalue,0 , map->value_size);@@ -776,10 +839,7 @@ static void __bpf_struct_ops_map_free(struct bpf_map *map)if (st_map->links)bpf_struct_ops_map_put_progs (st_map);bpf_map_area_free (st_map->links); -if (st_map->image) { -arch_free_bpf_trampoline (st_map->image,PAGE_SIZE ); -bpf_jit_uncharge_modmem (PAGE_SIZE ); - } +bpf_struct_ops_map_free_image (st_map);bpf_map_area_free (st_map->uvalue);bpf_map_area_free (st_map); }@@ -889,20 +949,6 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) st_map->st_ops_desc = st_ops_desc; map = &st_map->map; - ret =bpf_jit_charge_modmem (PAGE_SIZE ); -if (ret) - goto errout_free; - - st_map->image =arch_alloc_bpf_trampoline (PAGE_SIZE ); -if (!st_map->image) { -/* __bpf_struct_ops_map_free() uses st_map->image as flag -* for "charged or not". In this case, we need to unchange -* here. -*/ -bpf_jit_uncharge_modmem (PAGE_SIZE ); - ret = -ENOMEM ; - goto errout_free; - } st_map->uvalue =bpf_map_area_alloc (vt->size,NUMA_NO_NODE ); st_map->links_cnt =btf_type_vlen (t); st_map->links =
Re: [PATCH bpf-next v2 2/3] bpf: struct_ops supports more than one page for trampolines.
Quoted 74 lines · Martin KaFai Lau
Quoted 58 lines · On 2/21/24 2:59 PM, [email protected] wrote:
kernel/bpf/bpf_struct_ops.c +25
-8@@ -531,10 +567,10 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, const struct btf_type *module_type; const struct btf_member *member; const struct btf_type *t = st_ops_desc->type; + void *image = NULL, *image_end = NULL; struct bpf_tramp_links *tlinks; void *udata, *kdata; int prog_fd, err; - void *image, *image_end; u32 i; if (flags)@@ -573,15 +609,14 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, udata = &uvalue->data; kdata = &kvalue->data; - image = st_map->image; - image_end = st_map->image + PAGE_SIZE; module_type = btf_type_by_id(btf_vmlinux, st_ops_ids[IDX_MODULE_ID]); for_each_member(i, t, member) { const struct btf_type *mtype, *ptype; struct bpf_prog *prog; struct bpf_tramp_link *link; - u32 moff; + u32 moff, tflags; + int tsize; moff = __btf_member_bit_offset(t, member) / 8; ptype = btf_type_resolve_ptr(st_map->btf, member->type, NULL);@@ -653,10 +688,38 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, &bpf_struct_ops_link_lops, prog); st_map->links[i] = &link->link; - err = bpf_struct_ops_prepare_trampoline(tlinks, link, - &st_ops->func_models[i], - *(void **)(st_ops->cfi_stubs + moff), - image, image_end); + tflags = BPF_TRAMP_F_INDIRECT; + if (st_ops->func_models[i].ret_size > 0) + tflags |= BPF_TRAMP_F_RET_FENTRY_RET; + + /* Compute the size of the trampoline */ + tlinks[BPF_TRAMP_FENTRY].links[0] = link; + tlinks[BPF_TRAMP_FENTRY].nr_links = 1; + tsize = arch_bpf_trampoline_size(&st_ops->func_models[i], + tflags, tlinks, NULL); + if (tsize < 0) { + err = tsize; + goto reset_unlock; + } + + /* Allocate pages */ + if (tsize > (unsigned long)image_end - (unsigned long)image) { + if (tsize > PAGE_SIZE) { + err = -E2BIG; + goto reset_unlock; + } + image = bpf_struct_ops_map_inc_image(st_map); + if (IS_ERR(image)) {kernel/bpf/bpf_struct_ops.c static long · line 722
Quoted 10 lines
kernel/bpf/bpf_struct_ops.c +10
-0+ err = PTR_ERR(image); + goto reset_unlock; + } + image_end = image + PAGE_SIZE; + } + + err = arch_prepare_bpf_trampoline(NULL, image, image_end, + &st_ops->func_models[i], + tflags, tlinks, + *(void **)(st_ops->cfi_stubs + moff));I don't prefer to copy the BPF_TRAMP_F_* setting on tflags, tlinks, and the arch_*_trampoline_*() logic from
bpf_struct_ops_prepare_trampoline()which is used by thebpf_dummy_opsfor testing also. Consideringstruct_opssupports kernel module now, in the future, it is better to movebpf_dummy_opsout to thebpf_testmodsomehow and avoid itsbpf_struct_ops_prepare_trampoline()usage. For now, it is still better to keepbpf_struct_ops_prepare_trampoline()to be reusable by both.Have you thought about the earlier suggestion in v1 to do
arch_alloc_bpf_trampoline()inbpf_struct_ops_prepare_trampoline()instead of copying codes frombpf_struct_ops_prepare_trampoline()tobpf_struct_ops_map_update_elem()?
Quoted 8 lines
Something like this (untested code):
void *bpf_struct_ops_prepare_trampoline(struct
bpf_tramp_links*tlinks,
structbpf_tramp_link*link,
const structbtf_func_model*model, void*stub_func, void *image,
u32*image_off,
boolallow_alloc)
How about pass a struct bpf_struct_ops_map to bpf_struct_ops_prepare_trampoline(). If the pointer of struct bpf_struct_ops_map is not NULL, try to allocate new pages for the map?
For example,
static int
_bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks,
struct bpf_tramp_link *link,
const struct btf_func_model *model,
void *stub_func, void *image, void *image_end,
struct bpf_struct_ops_map *st_map)
{
...
if (!image || size > PAGE_SIZE - *image_off) {
if (!st_map)
return -E2BIG;
image = bpf_struct_ops_map_inc_image(st_map);
if (IS_ERR(image))
return PTR_ERR(image);
image_end = image + PAGE_SIZE;
}...
}
int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks,
struct bpf_tramp_link *link,
const struct
btf_func_model*model,
void *stub_func, void *image, void *image_end)
{
return _bpf_struct_ops_prepare_trampoline(tlinks, link, model,
stub_func, image, image_end, NULL);
}
Quoted 122 lines
{
u32 flags =BPF_TRAMP_F_INDIRECT;
void*new_image= NULL;
int size;tlinks[BPF_TRAMP_FENTRY].links[0] = link;
tlinks[BPF_TRAMP_FENTRY].nr_links = 1;if (
model->ret_size> 0)
flags |=BPF_TRAMP_F_RET_FENTRY_RET;size =
arch_bpf_trampoline_size(model, flags, tlinks, NULL); if (size < 0)
returnERR_PTR(size);
if (!image || size >PAGE_SIZE-*image_off) {
int err;if (!allow_alloc)
returnERR_PTR(-E2BIG);err =
bpf_jit_charge_modmem(PAGE_SIZE);
if (err)
returnERR_PTR(err);
new_image= image =arch_alloc_bpf_trampoline(PAGE_SIZE); if (!new_image) {bpf_jit_uncharge_modmem(PAGE_SIZE);
returnERR_PTR(-ENOMEM);
}
*image_off= 0;
}size = arch_prepare_bpf_trampoline(NULL, image +
*image_off, image +PAGE_SIZE,
model, flags, tlinks,stub_func); if (size >= 0) {
*image_off+= size;
return image;
}if (
new_image) {bpf_jit_uncharge_modmem(PAGE_SIZE);arch_free_bpf_trampoline(new_image, PAGE_SIZE); }return
ERR_PTR(size);
}----
pw-bot: cr
Quoted 68 lines
kernel/bpf/bpf_struct_ops.c +2
-3if (err < 0) goto reset_unlock;@@ -672,10 +735,11 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, if (err) goto reset_unlock; } + for (i = 0; i < st_map->image_pages_cnt; i++) + arch_protect_bpf_trampoline(st_map->image_pages[i], PAGE_SIZE); if (st_map->map.map_flags & BPF_F_LINK) { err = 0; - arch_protect_bpf_trampoline(st_map->image, PAGE_SIZE); /* Let bpf_link handle registration & unregistration.* * Pair with smp_load_acquire() during lookup_elem(). @@ -684,7 +748,6 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, goto unlock; } - arch_protect_bpf_trampoline(st_map->image, PAGE_SIZE); err = st_ops->reg(kdata); if (likely(!err)) { /* This refcnt increment on the map here after@@ -707,9 +770,9 @@ static long bpf_struct_ops_map_update_elem(structbpf_map *map, void *key, * there was a race in registering the struct_ops (under the same name) to * a sub-system through different struct_ops's maps. */ - arch_unprotect_bpf_trampoline(st_map->image, PAGE_SIZE);
reset_unlock:
+bpf_struct_ops_map_free_image(st_map);
bpf_struct_ops_map_put_progs(st_map);
memset(uvalue, 0, map->value_size);
memset(kvalue, 0, map->value_size);kernel/bpf/bpf_struct_ops.c +1
-4@@ -776,10 +839,7 @@ static void __bpf_struct_ops_map_free(struct bpf_map *map) if (st_map->links) bpf_struct_ops_map_put_progs(st_map); bpf_map_area_free(st_map->links); - if (st_map->image) { - arch_free_bpf_trampoline(st_map->image, PAGE_SIZE); - bpf_jit_uncharge_modmem(PAGE_SIZE); - } + bpf_struct_ops_map_free_image(st_map); bpf_map_area_free(st_map->uvalue); bpf_map_area_free(st_map);}
kernel/bpf/bpf_struct_ops.c +0
-14@@ -889,20 +949,6 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) st_map->st_ops_desc = st_ops_desc; map = &st_map->map; - ret = bpf_jit_charge_modmem(PAGE_SIZE); - if (ret) - goto errout_free; - - st_map->image = arch_alloc_bpf_trampoline(PAGE_SIZE); - if (!st_map->image) { - /* __bpf_struct_ops_map_free() uses st_map->image as flag - * for "charged or not". In this case, we need to unchange - * here. - */ - bpf_jit_uncharge_modmem(PAGE_SIZE); - ret = -ENOMEM; - goto errout_free; - } st_map->uvalue = bpf_map_area_alloc(vt->size, NUMA_NO_NODE); st_map->links_cnt = btf_type_vlen(t);
st_map->links=
Martin KaFai Lau
reviewerRe: [PATCH bpf-next v2 2/3] bpf: struct_ops supports more than one page for trampolines.
Quoted 77 lines · Kui-Feng Lee
Quoted 74 lines · Martin KaFai Lau
Quoted 58 lines · On 2/21/24 2:59 PM, [email protected] wrote:
kernel/bpf/bpf_struct_ops.c +25
-8@@ -531,10 +567,10 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, const struct btf_type *module_type; const struct btf_member *member; const struct btf_type *t = st_ops_desc->type; + void *image = NULL, *image_end = NULL; struct bpf_tramp_links *tlinks; void *udata, *kdata; int prog_fd, err; - void *image, *image_end; u32 i; if (flags)@@ -573,15 +609,14 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, udata = &uvalue->data; kdata = &kvalue->data; - image = st_map->image; - image_end = st_map->image + PAGE_SIZE; module_type = btf_type_by_id(btf_vmlinux, st_ops_ids[IDX_MODULE_ID]); for_each_member(i, t, member) { const struct btf_type *mtype, *ptype; struct bpf_prog *prog; struct bpf_tramp_link *link; - u32 moff; + u32 moff, tflags; + int tsize; moff = __btf_member_bit_offset(t, member) / 8; ptype = btf_type_resolve_ptr(st_map->btf, member->type, NULL);@@ -653,10 +688,38 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, &bpf_struct_ops_link_lops, prog); st_map->links[i] = &link->link; - err = bpf_struct_ops_prepare_trampoline(tlinks, link, - &st_ops->func_models[i], - *(void **)(st_ops->cfi_stubs + moff), - image, image_end); + tflags = BPF_TRAMP_F_INDIRECT; + if (st_ops->func_models[i].ret_size > 0) + tflags |= BPF_TRAMP_F_RET_FENTRY_RET; + + /* Compute the size of the trampoline */ + tlinks[BPF_TRAMP_FENTRY].links[0] = link; + tlinks[BPF_TRAMP_FENTRY].nr_links = 1; + tsize = arch_bpf_trampoline_size(&st_ops->func_models[i], + tflags, tlinks, NULL); + if (tsize < 0) { + err = tsize; + goto reset_unlock; + } + + /* Allocate pages */ + if (tsize > (unsigned long)image_end - (unsigned long)image) { + if (tsize > PAGE_SIZE) { + err = -E2BIG; + goto reset_unlock; + } + image = bpf_struct_ops_map_inc_image(st_map); + if (IS_ERR(image)) {kernel/bpf/bpf_struct_ops.c static long bpf_struct_ops_map_update_elem(struct · line 722
Quoted 10 lines
kernel/bpf/bpf_struct_ops.c +10
-0+ err = PTR_ERR(image); + goto reset_unlock; + } + image_end = image + PAGE_SIZE; + } + + err = arch_prepare_bpf_trampoline(NULL, image, image_end, + &st_ops->func_models[i], + tflags, tlinks, + *(void **)(st_ops->cfi_stubs + moff));I don't prefer to copy the BPF_TRAMP_F_* setting on tflags, tlinks, and the arch_*_trampoline_*() logic from
bpf_struct_ops_prepare_trampoline()which is used by thebpf_dummy_opsfor testing also. Consideringstruct_opssupports kernel module now, in the future, it is better to movebpf_dummy_opsout to thebpf_testmodsomehow and avoid itsbpf_struct_ops_prepare_trampoline()usage. For now, it is still better to keepbpf_struct_ops_prepare_trampoline()to be reusable by both.Have you thought about the earlier suggestion in v1 to do
arch_alloc_bpf_trampoline()inbpf_struct_ops_prepare_trampoline()instead of copying codes frombpf_struct_ops_prepare_trampoline()tobpf_struct_ops_map_update_elem()?
Quoted 8 lines
Quoted 8 lines
Something like this (untested code):
void *bpf_struct_ops_prepare_trampoline(struct
bpf_tramp_links*tlinks, structbpf_tramp_link*link,
const structbtf_func_model*model, void*stub_func, void *image,
u32*image_off,
boolallow_alloc)
To be a little more specific, the changes in bpf_struct_ops_map_update_elem() could be mostly like this (untested):
ret_image = bpf_struct_ops_prepare_trampoline(tlinks, link,
&st_ops->func_models[i],
*(void **)(st_ops->cfi_stubs + moff),
image, &image_off,
st_map->image_pages_cnt < MAX_TRAMP_IMAGE_PAGES);
if (IS_ERR(ret_image))
goto reset_unlock;
if (image != ret_image) {
image = ret_image;
st_map->image_pages[st_map->image_pages_cnt++] = image;
}Quoted 20 lines
How about pass a struct
bpf_struct_ops_maptobpf_struct_ops_prepare_trampoline(). If the pointer of structbpf_struct_ops_mapis not NULL, try to allocate new pages for the map?For example,
static int
_bpf_struct_ops_prepare_trampoline(structbpf_tramp_links*tlinks,struct
bpf_tramp_link*link,const struct
btf_func_model*model,void
*stub_func, void *image, void*image_end,
structbpf_struct_ops_map*st_map)
{
Quoted 4 lines
...
if (!image || size >
PAGE_SIZE-*image_off) { if (!st_map)
Why only limit to st_map != NULL?
arch_alloc_bpf_trampoline() is also called in bpf_dummy_ops.
If bpf_struct_ops_prepare_trampoline() can do the alloc, it may as well simplify bpf_dummy_ops and just use bpf_struct_ops_prepare_trampoline() to alloc.
Quoted 143 lines
return
-E2BIG;
image =bpf_struct_ops_map_inc_image(st_map); if (IS_ERR(image))
returnPTR_ERR(image);
image_end= image +PAGE_SIZE;
}
...
}int bpf_struct_ops_prepare_trampoline(struct
bpf_tramp_links*tlinks,struct
bpf_tramp_link*link,const struct btf_func_model*model, void
*stub_func, void *image, void*image_end)
{
return_bpf_struct_ops_prepare_trampoline(tlinks, link, model, stub_func, image, image_end, NULL);
}Quoted 121 lines
{
u32 flags =BPF_TRAMP_F_INDIRECT;
void*new_image= NULL;
int size;tlinks[BPF_TRAMP_FENTRY].links[0] = link;
tlinks[BPF_TRAMP_FENTRY].nr_links = 1;if (
model->ret_size> 0)
flags |=BPF_TRAMP_F_RET_FENTRY_RET;size =
arch_bpf_trampoline_size(model, flags, tlinks, NULL); if (size < 0)
returnERR_PTR(size);
if (!image || size >PAGE_SIZE-*image_off) {
int err;if (!allow_alloc)
returnERR_PTR(-E2BIG);err =
bpf_jit_charge_modmem(PAGE_SIZE);
if (err)
returnERR_PTR(err);
new_image= image =arch_alloc_bpf_trampoline(PAGE_SIZE); if (!new_image) {bpf_jit_uncharge_modmem(PAGE_SIZE);
returnERR_PTR(-ENOMEM);
}
*image_off= 0;
}size = arch_prepare_bpf_trampoline(NULL, image +
*image_off, image +PAGE_SIZE,
model, flags, tlinks,stub_func); if (size >= 0) {
*image_off+= size;
return image;
}if (
new_image) {bpf_jit_uncharge_modmem(PAGE_SIZE);arch_free_bpf_trampoline(new_image, PAGE_SIZE); }return
ERR_PTR(size);
}----
pw-bot: cr
Quoted 67 lines
kernel/bpf/bpf_struct_ops.c +2
-3if (err < 0) goto reset_unlock;@@ -672,10 +735,11 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, if (err) goto reset_unlock; } + for (i = 0; i < st_map->image_pages_cnt; i++) + arch_protect_bpf_trampoline(st_map->image_pages[i], PAGE_SIZE); if (st_map->map.map_flags & BPF_F_LINK) { err = 0; - arch_protect_bpf_trampoline(st_map->image, PAGE_SIZE); /* Let bpf_link handle registration & unregistration.* * Pair with smp_load_acquire() during lookup_elem(). @@ -684,7 +748,6 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, goto unlock; } - arch_protect_bpf_trampoline(st_map->image, PAGE_SIZE); err = st_ops->reg(kdata); if (likely(!err)) { /* This refcnt increment on the map here after@@ -707,9 +770,9 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map*map, void *key, * there was a race in registering the struct_ops (under the same name) to * a sub-system through different struct_ops's maps. */ - arch_unprotect_bpf_trampoline(st_map->image, PAGE_SIZE);
reset_unlock:
+bpf_struct_ops_map_free_image(st_map);
bpf_struct_ops_map_put_progs(st_map);
memset(uvalue, 0, map->value_size);
memset(kvalue, 0, map->value_size);kernel/bpf/bpf_struct_ops.c +1
-18@@ -776,10 +839,7 @@ static void __bpf_struct_ops_map_free(struct bpf_map *map) if (st_map->links) bpf_struct_ops_map_put_progs(st_map); bpf_map_area_free(st_map->links); - if (st_map->image) { - arch_free_bpf_trampoline(st_map->image, PAGE_SIZE); - bpf_jit_uncharge_modmem(PAGE_SIZE); - } + bpf_struct_ops_map_free_image(st_map); bpf_map_area_free(st_map->uvalue); bpf_map_area_free(st_map); }@@ -889,20 +949,6 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) st_map->st_ops_desc = st_ops_desc; map = &st_map->map; - ret = bpf_jit_charge_modmem(PAGE_SIZE); - if (ret) - goto errout_free; - - st_map->image = arch_alloc_bpf_trampoline(PAGE_SIZE); - if (!st_map->image) { - /* __bpf_struct_ops_map_free() uses st_map->image as flag - * for "charged or not". In this case, we need to unchange - * here. - */ - bpf_jit_uncharge_modmem(PAGE_SIZE); - ret = -ENOMEM; - goto errout_free; - } st_map->uvalue = bpf_map_area_alloc(vt->size, NUMA_NO_NODE); st_map->links_cnt = btf_type_vlen(t);
st_map->links=
Re: [PATCH bpf-next v2 2/3] bpf: struct_ops supports more than one page for trampolines.
Quoted 99 lines · Martin KaFai Lau
Quoted 77 lines · Kui-Feng Lee
Quoted 74 lines · Martin KaFai Lau
Quoted 58 lines · On 2/21/24 2:59 PM, [email protected] wrote:
kernel/bpf/bpf_struct_ops.c +25
-8@@ -531,10 +567,10 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, const struct btf_type *module_type; const struct btf_member *member; const struct btf_type *t = st_ops_desc->type; + void *image = NULL, *image_end = NULL; struct bpf_tramp_links *tlinks; void *udata, *kdata; int prog_fd, err; - void *image, *image_end; u32 i; if (flags)@@ -573,15 +609,14 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, udata = &uvalue->data; kdata = &kvalue->data; - image = st_map->image; - image_end = st_map->image + PAGE_SIZE; module_type = btf_type_by_id(btf_vmlinux, st_ops_ids[IDX_MODULE_ID]); for_each_member(i, t, member) { const struct btf_type *mtype, *ptype; struct bpf_prog *prog; struct bpf_tramp_link *link; - u32 moff; + u32 moff, tflags; + int tsize; moff = __btf_member_bit_offset(t, member) / 8; ptype = btf_type_resolve_ptr(st_map->btf, member->type, NULL);@@ -653,10 +688,38 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, &bpf_struct_ops_link_lops, prog); st_map->links[i] = &link->link; - err = bpf_struct_ops_prepare_trampoline(tlinks, link, - &st_ops->func_models[i], - *(void **)(st_ops->cfi_stubs + moff), - image, image_end); + tflags = BPF_TRAMP_F_INDIRECT; + if (st_ops->func_models[i].ret_size > 0) + tflags |= BPF_TRAMP_F_RET_FENTRY_RET; + + /* Compute the size of the trampoline */ + tlinks[BPF_TRAMP_FENTRY].links[0] = link; + tlinks[BPF_TRAMP_FENTRY].nr_links = 1; + tsize = arch_bpf_trampoline_size(&st_ops->func_models[i], + tflags, tlinks, NULL); + if (tsize < 0) { + err = tsize; + goto reset_unlock; + } + + /* Allocate pages */ + if (tsize > (unsigned long)image_end - (unsigned long)image) { + if (tsize > PAGE_SIZE) { + err = -E2BIG; + goto reset_unlock; + } + image = bpf_struct_ops_map_inc_image(st_map); + if (IS_ERR(image)) {kernel/bpf/bpf_struct_ops.c static long · line 722
Quoted 10 lines
kernel/bpf/bpf_struct_ops.c +10
-0+ err = PTR_ERR(image); + goto reset_unlock; + } + image_end = image + PAGE_SIZE; + } + + err = arch_prepare_bpf_trampoline(NULL, image, image_end, + &st_ops->func_models[i], + tflags, tlinks, + *(void **)(st_ops->cfi_stubs + moff));I don't prefer to copy the BPF_TRAMP_F_* setting on tflags, tlinks, and the arch_*_trampoline_*() logic from
bpf_struct_ops_prepare_trampoline()which is used by thebpf_dummy_opsfor testing also. Consideringstruct_opssupports kernel module now, in the future, it is better to movebpf_dummy_opsout to thebpf_testmodsomehow and avoid itsbpf_struct_ops_prepare_trampoline()usage. For now, it is still better to keepbpf_struct_ops_prepare_trampoline()to be reusable by both.Have you thought about the earlier suggestion in v1 to do
arch_alloc_bpf_trampoline()inbpf_struct_ops_prepare_trampoline()instead of copying codes frombpf_struct_ops_prepare_trampoline()tobpf_struct_ops_map_update_elem()?Quoted 8 lines
Quoted 8 lines
Something like this (untested code):
void *bpf_struct_ops_prepare_trampoline(struct
bpf_tramp_links*tlinks, structbpf_tramp_link*link,
const structbtf_func_model*model, void*stub_func, void *image,
u32*image_off,
boolallow_alloc)To be a little more specific, the changes in
bpf_struct_ops_map_update_elem()could be mostly like this (untested):
ret_image= bpf_struct_ops_prepare_trampoline(tlinks, link, &st_ops->func_models[i], *(void **)(st_ops->cfi_stubs + moff), image,&image_off,st_map->image_pages_cnt<MAX_TRAMP_IMAGE_PAGES);
if (IS_ERR(ret_image))
gotoreset_unlock;
Quoted 5 lines
if (image !=
ret_image) {
image =ret_image; st_map->image_pages[st_map->image_pages_cnt++] = image; }
What I don't like is the memory management code was in two named functions, bpf_struct_ops_map_free_image() and bpf_struct_ops_map_inc_image().
Now, it falls apart. Allocate in one place, keep accounting in another place, and free yet at the 3rd place.
Quoted 22 lines
Quoted 22 lines
How about pass a struct
bpf_struct_ops_maptobpf_struct_ops_prepare_trampoline(). If the pointer of structbpf_struct_ops_mapis not NULL, try to allocate new pages for the map?For example,
static int
_bpf_struct_ops_prepare_trampoline(structbpf_tramp_links*tlinks,struct
bpf_tramp_link*link,const struct
btf_func_model*model,void
*stub_func, void *image, void*image_end,
structbpf_struct_ops_map*st_map) {...
Quoted 8 lines
Quoted 2 lines
if (!image || size >
PAGE_SIZE-*image_off) { if (!st_map)Why only limit to
st_map!= NULL?
arch_alloc_bpf_trampoline()is also called inbpf_dummy_ops.
Ifbpf_struct_ops_prepare_trampoline()can do the alloc, it may as well simplifybpf_dummy_opsand just usebpf_struct_ops_prepare_trampoline()to alloc.
Yes, it can save a few lines from bpf_dummy_ops. But, bpf_dummy_ops still need to free the memory. And, it doesn't pair alloc and free in the same function. Usually, paring alloc and free in the same function, nearby, or the same module is easier to understand.
[ skip ]
Martin KaFai Lau
reviewerRe: [PATCH bpf-next v2 2/3] bpf: struct_ops supports more than one page for trampolines.
Quoted 108 lines · Kui-Feng Lee
Quoted 98 lines · Martin KaFai Lau
Quoted 77 lines · Kui-Feng Lee
Quoted 74 lines · Martin KaFai Lau
Quoted 58 lines · On 2/21/24 2:59 PM, [email protected] wrote:
kernel/bpf/bpf_struct_ops.c +25
-8@@ -531,10 +567,10 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, const struct btf_type *module_type; const struct btf_member *member; const struct btf_type *t = st_ops_desc->type; + void *image = NULL, *image_end = NULL; struct bpf_tramp_links *tlinks; void *udata, *kdata; int prog_fd, err; - void *image, *image_end; u32 i; if (flags)@@ -573,15 +609,14 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, udata = &uvalue->data; kdata = &kvalue->data; - image = st_map->image; - image_end = st_map->image + PAGE_SIZE; module_type = btf_type_by_id(btf_vmlinux, st_ops_ids[IDX_MODULE_ID]); for_each_member(i, t, member) { const struct btf_type *mtype, *ptype; struct bpf_prog *prog; struct bpf_tramp_link *link; - u32 moff; + u32 moff, tflags; + int tsize; moff = __btf_member_bit_offset(t, member) / 8; ptype = btf_type_resolve_ptr(st_map->btf, member->type, NULL);@@ -653,10 +688,38 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, &bpf_struct_ops_link_lops, prog); st_map->links[i] = &link->link; - err = bpf_struct_ops_prepare_trampoline(tlinks, link, - &st_ops->func_models[i], - *(void **)(st_ops->cfi_stubs + moff), - image, image_end); + tflags = BPF_TRAMP_F_INDIRECT; + if (st_ops->func_models[i].ret_size > 0) + tflags |= BPF_TRAMP_F_RET_FENTRY_RET; + + /* Compute the size of the trampoline */ + tlinks[BPF_TRAMP_FENTRY].links[0] = link; + tlinks[BPF_TRAMP_FENTRY].nr_links = 1; + tsize = arch_bpf_trampoline_size(&st_ops->func_models[i], + tflags, tlinks, NULL); + if (tsize < 0) { + err = tsize; + goto reset_unlock; + } + + /* Allocate pages */ + if (tsize > (unsigned long)image_end - (unsigned long)image) { + if (tsize > PAGE_SIZE) { + err = -E2BIG; + goto reset_unlock; + } + image = bpf_struct_ops_map_inc_image(st_map); + if (IS_ERR(image)) {kernel/bpf/bpf_struct_ops.c static long bpf_struct_ops_map_update_elem(struct · line 722
Quoted 10 lines
kernel/bpf/bpf_struct_ops.c +10
-0+ err = PTR_ERR(image); + goto reset_unlock; + } + image_end = image + PAGE_SIZE; + } + + err = arch_prepare_bpf_trampoline(NULL, image, image_end, + &st_ops->func_models[i], + tflags, tlinks, + *(void **)(st_ops->cfi_stubs + moff));I don't prefer to copy the BPF_TRAMP_F_* setting on tflags, tlinks, and the arch_*_trampoline_*() logic from
bpf_struct_ops_prepare_trampoline()which is used by thebpf_dummy_opsfor testing also. Consideringstruct_opssupports kernel module now, in the future, it is better to movebpf_dummy_opsout to thebpf_testmodsomehow and avoid itsbpf_struct_ops_prepare_trampoline()usage. For now, it is still better to keepbpf_struct_ops_prepare_trampoline()to be reusable by both.Have you thought about the earlier suggestion in v1 to do
arch_alloc_bpf_trampoline()inbpf_struct_ops_prepare_trampoline()instead of copying codes frombpf_struct_ops_prepare_trampoline()tobpf_struct_ops_map_update_elem()?Quoted 8 lines
Quoted 8 lines
Something like this (untested code):
void *bpf_struct_ops_prepare_trampoline(struct
bpf_tramp_links*tlinks, structbpf_tramp_link*link,
const structbtf_func_model*model, void*stub_func, void *image,
u32*image_off,
boolallow_alloc)To be a little more specific, the changes in
bpf_struct_ops_map_update_elem()could be mostly like this (untested):
ret_image= bpf_struct_ops_prepare_trampoline(tlinks, link, &st_ops->func_models[i],
*(void **)(st_ops->cfi_stubs + moff), image,&image_off,st_map->image_pages_cnt<MAX_TRAMP_IMAGE_PAGES); if (IS_ERR(ret_image))
gotoreset_unlock;Quoted 5 lines
if (image !=
ret_image) {
image =ret_image; st_map->image_pages[st_map->image_pages_cnt++] = image; }
Quoted 3 lines
What I don't like is the memory management code was in two named functions,
bpf_struct_ops_map_free_image()andbpf_struct_ops_map_inc_image().
bpf_struct_ops_map_inc_image() is not needed.
Quoted 35 lines
Now, it falls apart. Allocate in one place, keep accounting in another place, and free yet at the 3rd place.
Quoted 22 lines
Quoted 22 lines
How about pass a struct
bpf_struct_ops_maptobpf_struct_ops_prepare_trampoline(). If the pointer of structbpf_struct_ops_mapis not NULL, try to allocate new pages for the map?For example,
static int
_bpf_struct_ops_prepare_trampoline(structbpf_tramp_links*tlinks,struct
bpf_tramp_link*link,const struct
btf_func_model*model,void
*stub_func, void *image, void*image_end,
structbpf_struct_ops_map*st_map) {...
Quoted 8 lines
Quoted 2 lines
if (!image || size >
PAGE_SIZE-*image_off) { if (!st_map)Why only limit to
st_map!= NULL?
arch_alloc_bpf_trampoline()is also called inbpf_dummy_ops.
Ifbpf_struct_ops_prepare_trampoline()can do the alloc, it may as well simplifybpf_dummy_opsand just usebpf_struct_ops_prepare_trampoline()to alloc.
Quoted 4 lines
Yes, it can save a few lines from
bpf_dummy_ops. But,bpf_dummy_opsstill need to free the memory. And, it doesn't pair alloc and free in the same function. Usually, paring alloc and free in the same function, nearby, or the same module is easier to understand.
It is not only about saving a few lines. It just does not make sense to add all this complexity and another "_"bpf_struct_ops_prepare_trampoline() variant to make it conform to the alloc/free pair idea. To be clear, I don't see alloc/free pair is a must have in all cases. There are many situations that non-alloc named function calls multiple kmalloc() in different places and one xyz_free() releases everything. Even alloc/free is really preferred, there has to be a better way or else need to make a trade off.
I suggested the high level ideal on making bpf_struct_ops_prepare_trampoline() to allocate page. It can sure add a bpf_struct_ops_free_trampoline() if you see fit to make it match with bpf_struct_ops_prepare_trampoline() as alloc/free pair, for example,
void bpf_struct_ops_free_trampoline(void *image)
{
bpf_jit_uncharge_modmem(PAGE_SIZE);
arch_free_bpf_trampoline(image, PAGE_SIZE);
}and make bpf_struct_ops_map_free_image() to usebpf_struct_ops_free_trampoline()
static void bpf_struct_ops_map_free_image(struct bpf_struct_ops_map *st_map)
{
int i;
for (i = 0; i < st_map->image_pages_cnt; i++) {
bpf_struct_ops_free_trampoline(st_map->image_pages[i]);
st_map->image_pages[i] = NULL;
}
st_map->image_pages_cnt = 0;
}Then it should work like alloc/free pair.
Re: [PATCH bpf-next v2 2/3] bpf: struct_ops supports more than one page for trampolines.
Quoted 151 lines · Martin KaFai Lau
Quoted 109 lines · Kui-Feng Lee
Quoted 99 lines · Martin KaFai Lau
Quoted 78 lines · Kui-Feng Lee
Quoted 75 lines · Martin KaFai Lau
Quoted 59 lines · On 2/21/24 2:59 PM, [email protected] wrote:
kernel/bpf/bpf_struct_ops.c +25
-8@@ -531,10 +567,10 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, const struct btf_type *module_type; const struct btf_member *member; const struct btf_type *t = st_ops_desc->type; + void *image = NULL, *image_end = NULL; struct bpf_tramp_links *tlinks; void *udata, *kdata; int prog_fd, err; - void *image, *image_end; u32 i; if (flags)@@ -573,15 +609,14 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, udata = &uvalue->data; kdata = &kvalue->data; - image = st_map->image; - image_end = st_map->image + PAGE_SIZE; module_type = btf_type_by_id(btf_vmlinux, st_ops_ids[IDX_MODULE_ID]); for_each_member(i, t, member) { const struct btf_type *mtype, *ptype; struct bpf_prog *prog; struct bpf_tramp_link *link; - u32 moff; + u32 moff, tflags; + int tsize; moff = __btf_member_bit_offset(t, member) / 8; ptype = btf_type_resolve_ptr(st_map->btf, member->type, NULL);@@ -653,10 +688,38 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, &bpf_struct_ops_link_lops, prog); st_map->links[i] = &link->link; - err = bpf_struct_ops_prepare_trampoline(tlinks, link, - &st_ops->func_models[i], - *(void **)(st_ops->cfi_stubs + moff), - image, image_end); + tflags = BPF_TRAMP_F_INDIRECT; + if (st_ops->func_models[i].ret_size > 0) + tflags |= BPF_TRAMP_F_RET_FENTRY_RET; + + /* Compute the size of the trampoline */ + tlinks[BPF_TRAMP_FENTRY].links[0] = link; + tlinks[BPF_TRAMP_FENTRY].nr_links = 1; + tsize = arch_bpf_trampoline_size(&st_ops->func_models[i], + tflags, tlinks, NULL); + if (tsize < 0) { + err = tsize; + goto reset_unlock; + } + + /* Allocate pages */ + if (tsize > (unsigned long)image_end - (unsignedlong )image) { + if (tsize > PAGE_SIZE) { + err = -E2BIG; + goto reset_unlock; + } + image = bpf_struct_ops_map_inc_image(st_map); + if (IS_ERR(image)) {kernel/bpf/bpf_struct_ops.c static long · line 723
Quoted 10 lines
kernel/bpf/bpf_struct_ops.c +10
-0+ err = PTR_ERR(image); + goto reset_unlock; + } + image_end = image + PAGE_SIZE; + } + + err = arch_prepare_bpf_trampoline(NULL, image, image_end, + &st_ops->func_models[i], + tflags, tlinks, + *(void **)(st_ops->cfi_stubs + moff));I don't prefer to copy the BPF_TRAMP_F_* setting on tflags, tlinks, and the arch_*_trampoline_*() logic from
bpf_struct_ops_prepare_trampoline()which is used by thebpf_dummy_opsfor testing also. Consideringstruct_opssupports kernel module now, in the future, it is better to movebpf_dummy_opsout to thebpf_testmodsomehow and avoid itsbpf_struct_ops_prepare_trampoline()usage. For now, it is still better to keepbpf_struct_ops_prepare_trampoline()to be reusable by both.Have you thought about the earlier suggestion in v1 to do
arch_alloc_bpf_trampoline()inbpf_struct_ops_prepare_trampoline()instead of copying codes frombpf_struct_ops_prepare_trampoline()tobpf_struct_ops_map_update_elem()?Quoted 8 lines
Quoted 8 lines
Something like this (untested code):
void *bpf_struct_ops_prepare_trampoline(struct
bpf_tramp_links*tlinks, structbpf_tramp_link*link,
const structbtf_func_model*model, void*stub_func, void *image,
u32*image_off,
boolallow_alloc)To be a little more specific, the changes in
bpf_struct_ops_map_update_elem()could be mostly like this (untested):
ret_image= bpf_struct_ops_prepare_trampoline(tlinks, link, &st_ops->func_models[i],
*(void **)(st_ops->cfi_stubs + moff), image,&image_off,st_map->image_pages_cnt<MAX_TRAMP_IMAGE_PAGES); if (IS_ERR(ret_image))
gotoreset_unlock;Quoted 5 lines
if (image !=
ret_image) {
image =ret_image; st_map->image_pages[st_map->image_pages_cnt++] = image; }Quoted 3 lines
What I don't like is the memory management code was in two named functions,
bpf_struct_ops_map_free_image()andbpf_struct_ops_map_inc_image().
bpf_struct_ops_map_inc_image()is not needed.Quoted 35 lines
Now, it falls apart. Allocate in one place, keep accounting in another place, and free yet at the 3rd place.
Quoted 30 lines
Quoted 20 lines
How about pass a struct
bpf_struct_ops_maptobpf_struct_ops_prepare_trampoline(). If the pointer of structbpf_struct_ops_mapis not NULL, try to allocate new pages for the map?For example,
static int
_bpf_struct_ops_prepare_trampoline(structbpf_tramp_links*tlinks,struct
bpf_tramp_link*link,const struct
btf_func_model*model,void
*stub_func, void *image, void*image_end,
structbpf_struct_ops_map*st_map) {Quoted 4 lines
...
if (!image || size >
PAGE_SIZE-*image_off) { if (!st_map)Why only limit to
st_map!= NULL?
arch_alloc_bpf_trampoline()is also called inbpf_dummy_ops.
Ifbpf_struct_ops_prepare_trampoline()can do the alloc, it may as well simplifybpf_dummy_opsand just usebpf_struct_ops_prepare_trampoline()to alloc.
Quoted 8 lines
Quoted 4 lines
Yes, it can save a few lines from
bpf_dummy_ops. But,bpf_dummy_opsstill need to free the memory. And, it doesn't pair alloc and free in the same function. Usually, paring alloc and free in the same function, nearby, or the same module is easier to understand.It is not only about saving a few lines. It just does not make sense to add all this complexity and another "_"bpf_struct_ops_prepare_trampoline() variant to make it conform to the alloc/free pair idea. To be clear, I don't
To be clear, we are not talking computation or memory complexity here. I consider the complexity in another way. When I look at the code of bpf_dummy_ops, and see it free the memory at the very end of a function. I have to guess who allocate the memory by looking around without a clear sign or hint if we move the allocation to bpf_struct_ops_prepare_trampoline(). It is a source of complexity. Very often, a duplication is much more simple and easy to understand. Especially, when the duplication is in a very well know/recognized pattern. Here will create a unusual way to replace a well recognized one to simplify the code.
My reason of duplicating the code from bpf_struct_ops_prepare_trampoline() was we don't need bpf_struct_ops_prepare_trampoline() in future if we were going to move bpf_dummy_ops out. But, just like you said, we still have bpf_dummy_ops now, so it is a good trade of to move memory allocation into bpf_struct_ops_prepare_trampoline() to avoid the duplication the code about flags and tlinks. But, the trade off we are talking here goes to an opposite way.
By the way, I am not insisting on these tiny details. I am just trying to explain what I don't like here.
Quoted 31 lines
see alloc/free pair is a must have in all cases. There are many situations that non-alloc named function calls multiple
kmalloc()in different places and onexyz_free()releases everything. Even alloc/free is really preferred, there has to be a better way or else need to make a trade off.I suggested the high level ideal on making
bpf_struct_ops_prepare_trampoline()to allocate page. It can sure add abpf_struct_ops_free_trampoline()if you see fit to make it match withbpf_struct_ops_prepare_trampoline()as alloc/free pair, for example,void
bpf_struct_ops_free_trampoline(void *image)
{
bpf_jit_uncharge_modmem(PAGE_SIZE);
arch_free_bpf_trampoline(image,PAGE_SIZE);
}and make
bpf_struct_ops_map_free_image()to usebpf_struct_ops_free_trampoline()static void
bpf_struct_ops_map_free_image(struct bpf_struct_ops_map *st_map){
int i;for (i = 0; i <
st_map->image_pages_cnt; i++) {bpf_struct_ops_free_trampoline(st_map->image_pages[i]); st_map->image_pages[i] = NULL;
}
st_map->image_pages_cnt = 0;
}Then it should work like alloc/free pair.
Re: [PATCH bpf-next v2 2/3] bpf: struct_ops supports more than one page for trampolines.
Quoted 183 lines · Kui-Feng Lee
Quoted 151 lines · Martin KaFai Lau
Quoted 109 lines · Kui-Feng Lee
Quoted 99 lines · Martin KaFai Lau
Quoted 78 lines · Kui-Feng Lee
Quoted 75 lines · Martin KaFai Lau
Quoted 59 lines · On 2/21/24 2:59 PM, [email protected] wrote:
(quoted diff) +25
-8@@ -531,10 +567,10 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, const struct btf_type *module_type; const struct btf_member *member; const struct btf_type *t = st_ops_desc->type; + void *image = NULL, *image_end = NULL; struct bpf_tramp_links *tlinks; void *udata, *kdata; int prog_fd, err; - void *image, *image_end; u32 i; if (flags)@@ -573,15 +609,14 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, udata = &uvalue->data; kdata = &kvalue->data; - image = st_map->image; - image_end = st_map->image + PAGE_SIZE; module_type = btf_type_by_id(btf_vmlinux, st_ops_ids[IDX_MODULE_ID]); for_each_member(i, t, member) { const struct btf_type *mtype, *ptype; struct bpf_prog *prog; struct bpf_tramp_link *link; - u32 moff; + u32 moff, tflags; + int tsize; moff = __btf_member_bit_offset(t, member) / 8; ptype = btf_type_resolve_ptr(st_map->btf, member->type, NULL);@@ -653,10 +688,38 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, &bpf_struct_ops_link_lops, prog); st_map->links[i] = &link->link; - err = bpf_struct_ops_prepare_trampoline(tlinks, link, - &st_ops->func_models[i], - *(void **)(st_ops->cfi_stubs + moff), - image, image_end); + tflags = BPF_TRAMP_F_INDIRECT; + if (st_ops->func_models[i].ret_size > 0) + tflags |= BPF_TRAMP_F_RET_FENTRY_RET; + + /* Compute the size of the trampoline */ + tlinks[BPF_TRAMP_FENTRY].links[0] = link; + tlinks[BPF_TRAMP_FENTRY].nr_links = 1; + tsize = arch_bpf_trampoline_size(&st_ops->func_models[i], + tflags, tlinks, NULL); + if (tsize < 0) { + err = tsize; + goto reset_unlock; + } + + /* Allocate pages */ + if (tsize > (unsigned long)image_end - (unsigned long)image) { + if (tsize > PAGE_SIZE) { + err = -E2BIG; + goto reset_unlock; + } + image = bpf_struct_ops_map_inc_image(st_map); + if (IS_ERR(image)) {Review static long · line 723
Quoted 10 lines
(quoted diff) +10
-0+ err = PTR_ERR(image); + goto reset_unlock; + } + image_end = image + PAGE_SIZE; + } + + err = arch_prepare_bpf_trampoline(NULL, image, image_end, + &st_ops->func_models[i], + tflags, tlinks, + *(void **)(st_ops->cfi_stubs + moff));I don't prefer to copy the BPF_TRAMP_F_* setting on tflags, tlinks, and the arch_*_trampoline_*() logic from
bpf_struct_ops_prepare_trampoline()which is used by thebpf_dummy_opsfor testing also. Consideringstruct_opssupports kernel module now, in the future, it is better to movebpf_dummy_opsout to thebpf_testmodsomehow and avoid itsbpf_struct_ops_prepare_trampoline()usage. For now, it is still better to keepbpf_struct_ops_prepare_trampoline()to be reusable by both.Have you thought about the earlier suggestion in v1 to do
arch_alloc_bpf_trampoline()inbpf_struct_ops_prepare_trampoline()instead of copying codes frombpf_struct_ops_prepare_trampoline()tobpf_struct_ops_map_update_elem()?Quoted 8 lines
Quoted 8 lines
Something like this (untested code):
void *bpf_struct_ops_prepare_trampoline(struct
bpf_tramp_links*tlinks, structbpf_tramp_link*link,
const structbtf_func_model*model, void*stub_func, void *image,
u32*image_off,
boolallow_alloc)To be a little more specific, the changes in
bpf_struct_ops_map_update_elem()could be mostly like this (untested):
ret_image= bpf_struct_ops_prepare_trampoline(tlinks, link, &st_ops->func_models[i],
*(void **)(st_ops->cfi_stubs + moff), image,&image_off,st_map->image_pages_cnt<MAX_TRAMP_IMAGE_PAGES); if (IS_ERR(ret_image))
gotoreset_unlock;Quoted 5 lines
if (image !=
ret_image) {
image =ret_image; st_map->image_pages[st_map->image_pages_cnt++] = image; }Quoted 3 lines
What I don't like is the memory management code was in two named functions,
bpf_struct_ops_map_free_image()andbpf_struct_ops_map_inc_image().
bpf_struct_ops_map_inc_image()is not needed.Quoted 35 lines
Now, it falls apart. Allocate in one place, keep accounting in another place, and free yet at the 3rd place.
Quoted 30 lines
Quoted 20 lines
How about pass a struct
bpf_struct_ops_maptobpf_struct_ops_prepare_trampoline(). If the pointer of structbpf_struct_ops_mapis not NULL, try to allocate new pages for the map?For example,
static int
_bpf_struct_ops_prepare_trampoline(structbpf_tramp_links*tlinks,struct
bpf_tramp_link*link,const struct
btf_func_model*model,void
*stub_func, void *image, void*image_end,
structbpf_struct_ops_map*st_map) {Quoted 4 lines
...
if (!image || size >
PAGE_SIZE-*image_off) { if (!st_map)Why only limit to
st_map!= NULL?
arch_alloc_bpf_trampoline()is also called inbpf_dummy_ops.
Ifbpf_struct_ops_prepare_trampoline()can do the alloc, it may as well simplifybpf_dummy_opsand just usebpf_struct_ops_prepare_trampoline()to alloc.Quoted 8 lines
Quoted 4 lines
Yes, it can save a few lines from
bpf_dummy_ops. But,bpf_dummy_opsstill need to free the memory. And, it doesn't pair alloc and free in the same function. Usually, paring alloc and free in the same function, nearby, or the same module is easier to understand.It is not only about saving a few lines. It just does not make sense to add all this complexity and another "_"bpf_struct_ops_prepare_trampoline() variant to make it conform to the alloc/free pair idea. To be clear, I don't
To be clear, we are not talking computation or memory complexity here. I consider the complexity in another way. When I look at the code of
bpf_dummy_ops, and see it free the memory at the very end of a function. I have to guess who allocate the memory by looking around without a clear sign or hint if we move the allocation tobpf_struct_ops_prepare_trampoline(). It is a source of complexity. Very often, a duplication is much more simple and easy to understand. Especially, when the duplication is in a very well know/recognized pattern. Here will create a unusual way to replace a well recognized one to simplify the code.My reason of duplicating the code from
bpf_struct_ops_prepare_trampoline()was we don't needbpf_struct_ops_prepare_trampoline()in future if we were going to movebpf_dummy_opsout. But, just like you said, we still havebpf_dummy_opsnow, so it is a good trade of to move memory allocation intobpf_struct_ops_prepare_trampoline()to avoid the duplication the code about flags and tlinks. But, the trade off we are talking here goes to an opposite way.
Quoted 2 lines
By the way, I am not insisting on these tiny details. I am just trying to explain what I don't like here.
One thing I forgot to mention is that bpf_dummy_ops has to call bpf_jit_uncharge_modmem(PAGE_SIZE) as well. The other option is to move bpf_jit_charge_modmem() out of bpf_struct_ops_prepare_trampoline(), meaning bpf_struct_ops_map_update_elem() should handle the case that the allocation in bpf_struct_ops_prepare_trampoline() successes, but bpf_jit_charge_modmem() fails.
Quoted 32 lines
Quoted 32 lines
see alloc/free pair is a must have in all cases. There are many situations that non-alloc named function calls multiple
kmalloc()in different places and onexyz_free()releases everything. Even alloc/free is really preferred, there has to be a better way or else need to make a trade off.I suggested the high level ideal on making
bpf_struct_ops_prepare_trampoline()to allocate page. It can sure add abpf_struct_ops_free_trampoline()if you see fit to make it match withbpf_struct_ops_prepare_trampoline()as alloc/free pair, for example,void
bpf_struct_ops_free_trampoline(void *image)
{
bpf_jit_uncharge_modmem(PAGE_SIZE);
arch_free_bpf_trampoline(image,PAGE_SIZE);
}and make
bpf_struct_ops_map_free_image()to usebpf_struct_ops_free_trampoline()static void
bpf_struct_ops_map_free_image(struct bpf_struct_ops_map *st_map){
int i;for (i = 0; i <
st_map->image_pages_cnt; i++) {bpf_struct_ops_free_trampoline(st_map->image_pages[i]); st_map->image_pages[i] = NULL;
}
st_map->image_pages_cnt = 0;
}Then it should work like alloc/free pair.
Martin KaFai Lau
reviewerRe: [PATCH bpf-next v2 2/3] bpf: struct_ops supports more than one page for trampolines.
In reply to Kui-Feng LeeQuoted 7 lines · Kui-Feng Lee
To be clear, we are not talking computation or memory complexity here. I consider the complexity in another way. When I look at the code of
bpf_dummy_ops, and see it free the memory at the very end of a function. I have to guess who allocate the memory by looking around without a clear sign or hint if we move the allocation tobpf_struct_ops_prepare_trampoline(). It is a source of complexity.
It still sounds like a naming perception issue more than a practical code-wise complexity/readability. Rename it to bpf_struct_ops_"s/alloc/prepare/"_trampoline() if it can make it more obvious that it is an alloc function. imo, that function returning a page is a clear sign that it can alloc but I don't mind renaming it if it can help to make it sounds more like alloc and free pair.
Quoted 4 lines
Very often, a duplication is much more simple and easy to understand. Especially, when the duplication is in a very well know/recognized pattern. Here will create a unusual way to replace a well recognized one to simplify the code.
Sorry, I don't agree on this where this patch is duplicating lines of code which is not obvious like setting BPF_TRAMP_F_*. At least I often have to go back to arch_prepare_bpf_trampoline() to understand how it works.
Not copy-and-pasting this piece of codes everywhere is more important than making bpf_dummy_ops looks better.
Quoted 5 lines
My reason of duplicating the code from
bpf_struct_ops_prepare_trampoline()was we don't needbpf_struct_ops_prepare_trampoline()in future if we were going to movebpf_dummy_opsout. But, just like you said, we still havebpf_dummy_ops
Yep, it will be great to move bpf_dummy_ops out but how it can be done and whether it can remove its bpf_struct_ops_prepare_trampoline() usage is still TBD. I think it should be possible. Even it is moved out in the future, bpf_struct_ops_(prepare|alloc)_trampoline() can be keep as is.
Quoted 4 lines
now, so it is a good trade of to move memory allocation into
bpf_struct_ops_prepare_trampoline()to avoid the duplication the code about flags and tlinks. But, the trade off we are talking here goes to an opposite way.
Martin KaFai Lau
reviewerRe: [PATCH bpf-next v2 2/3] bpf: struct_ops supports more than one page for trampolines.
In reply to Kui-Feng LeeQuoted 6 lines · Kui-Feng Lee
One thing I forgot to mention is that
bpf_dummy_opshas to callbpf_jit_uncharge_modmem(PAGE_SIZE)as well. The other option is to movebpf_jit_charge_modmem()out ofbpf_struct_ops_prepare_trampoline(), meaningbpf_struct_ops_map_update_elem()should handle the case that the allocation inbpf_struct_ops_prepare_trampoline()successes, butbpf_jit_charge_modmem()fails.
Keep the charge/uncharge in bpf_struct_ops_prepare_trampoline().
It is fine to have bpf_dummy_ops charge and then uncharge a PAGE_SIZE. There is no need to optimize for bpf_dummy_ops. Use bpf_struct_ops_free_trampoline() in bpf_dummy_ops to uncharge and free.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
void
bpf_struct_ops_free_trampoline(void *image)
{
bpf_jit_uncharge_modmem(PAGE_SIZE);
arch_free_bpf_trampoline(image,PAGE_SIZE);
}
Re: [PATCH bpf-next v2 2/3] bpf: struct_ops supports more than one page for trampolines.
Quoted 11 lines · Martin KaFai Lau
Quoted 6 lines · Kui-Feng Lee
One thing I forgot to mention is that
bpf_dummy_opshas to callbpf_jit_uncharge_modmem(PAGE_SIZE)as well. The other option is to movebpf_jit_charge_modmem()out ofbpf_struct_ops_prepare_trampoline(), meaningbpf_struct_ops_map_update_elem()should handle the case that the allocation inbpf_struct_ops_prepare_trampoline()successes, butbpf_jit_charge_modmem()fails.Keep the charge/uncharge in
bpf_struct_ops_prepare_trampoline().It is fine to have
bpf_dummy_opscharge and then uncharge aPAGE_SIZE. There is no need to optimize forbpf_dummy_ops. Usebpf_struct_ops_free_trampoline()inbpf_dummy_opsto uncharge and free.
Then, I don't get the point here.
I agree with moving the allocation into bpf_struct_ops_prepare_trampoline() to avoid duplication of the code about flags and tlinks. It really simplifies the code with the fact that bpf_dummy_ops is still there. So, I tried to pass a st_map to bpf_struct_ops_prepare_trampoline() to keep page managements code together. But, you said to simplify the code of bpf_dummy_ops by allocating pages in bpf_struct_ops_prepare_trampoline(), do bookkeeping in bpf_struct_ops_map_update_elem(), so bpf_dummy_ops doesn't have to allocate memory. But, we have to move a bpf_jit_uncharge_modmem() to bpf_dummy_ops. For me, this trade-off that include removing an allocation and adding a bpf_jit_uncharge_modmem() make no sense.
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
void
bpf_struct_ops_free_trampoline(void *image)
{
bpf_jit_uncharge_modmem(PAGE_SIZE);
arch_free_bpf_trampoline(image,PAGE_SIZE);
}
Martin KaFai Lau
reviewerRe: [PATCH bpf-next v2 2/3] bpf: struct_ops supports more than one page for trampolines.
Quoted 17 lines · Kui-Feng Lee
Quoted 11 lines · Martin KaFai Lau
Quoted 6 lines · Kui-Feng Lee
One thing I forgot to mention is that
bpf_dummy_opshas to callbpf_jit_uncharge_modmem(PAGE_SIZE)as well. The other option is to movebpf_jit_charge_modmem()out ofbpf_struct_ops_prepare_trampoline(), meaningbpf_struct_ops_map_update_elem()should handle the case that the allocation inbpf_struct_ops_prepare_trampoline()successes, butbpf_jit_charge_modmem()fails.Keep the charge/uncharge in
bpf_struct_ops_prepare_trampoline().It is fine to have
bpf_dummy_opscharge and then uncharge aPAGE_SIZE. There is no need to optimize forbpf_dummy_ops. Usebpf_struct_ops_free_trampoline()inbpf_dummy_opsto uncharge and free.
Quoted 9 lines
Then, I don't get the point here.
I agree with moving the allocation intobpf_struct_ops_prepare_trampoline()to avoid duplication of the code about flags and tlinks. It really simplifies the code with the fact thatbpf_dummy_opsis still there. So, I tried to pass ast_maptobpf_struct_ops_prepare_trampoline()to keep page managements code together. But, you said to simplify the code ofbpf_dummy_opsby allocating pages inbpf_struct_ops_prepare_trampoline(), do bookkeeping inbpf_struct_ops_map_update_elem(), sobpf_dummy_opsdoesn't have to
I don't think I ever mentioned to do book keeping in bpf_struct_ops_map_update_elem(). Have you looked at my earlier code in bpf_struct_ops_prepare_trampoline() which also does the memory charging also?
Quoted 3 lines
allocate memory. But, we have to move a
bpf_jit_uncharge_modmem()tobpf_dummy_ops. For me, this trade-off that include removing an allocation and adding abpf_jit_uncharge_modmem()make no sense.
Which part make no sense? Having bpf_dummy_ops charge/uncharge memory also?
The bpf_dummy_ops() uses the below bpf_struct_ops_free_trampoline() which does uncharge and free. bpf_struct_ops_prepare_trampoline() does charge and alloc. charge/alloc matches with uncharge/free.
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
void
bpf_struct_ops_free_trampoline(void *image)
{
bpf_jit_uncharge_modmem(PAGE_SIZE);
arch_free_bpf_trampoline(image,PAGE_SIZE);
}
Re: [PATCH bpf-next v2 2/3] bpf: struct_ops supports more than one page for trampolines.
Quoted 28 lines · Martin KaFai Lau
Quoted 17 lines · Kui-Feng Lee
Quoted 11 lines · Martin KaFai Lau
Quoted 6 lines · Kui-Feng Lee
One thing I forgot to mention is that
bpf_dummy_opshas to callbpf_jit_uncharge_modmem(PAGE_SIZE)as well. The other option is to movebpf_jit_charge_modmem()out ofbpf_struct_ops_prepare_trampoline(), meaningbpf_struct_ops_map_update_elem()should handle the case that the allocation inbpf_struct_ops_prepare_trampoline()successes, butbpf_jit_charge_modmem()fails.Keep the charge/uncharge in
bpf_struct_ops_prepare_trampoline().It is fine to have
bpf_dummy_opscharge and then uncharge aPAGE_SIZE. There is no need to optimize forbpf_dummy_ops. Usebpf_struct_ops_free_trampoline()inbpf_dummy_opsto uncharge and free.Quoted 9 lines
Then, I don't get the point here.
I agree with moving the allocation intobpf_struct_ops_prepare_trampoline()to avoid duplication of the code about flags and tlinks. It really simplifies the code with the fact thatbpf_dummy_opsis still there. So, I tried to pass ast_maptobpf_struct_ops_prepare_trampoline()to keep page managements code together. But, you said to simplify the code ofbpf_dummy_opsby allocating pages inbpf_struct_ops_prepare_trampoline(), do bookkeeping inbpf_struct_ops_map_update_elem(), sobpf_dummy_opsdoesn't have to
Quoted 1 line
I don't think I ever mentioned to do book keeping in
bpf_struct_ops_map_update_elem(). Have you looked at my earlier code inbpf_struct_ops_prepare_trampoline()which also does the memory charging also?
The bookkeeping that I am saying is about maintaining image_pages and image_pages_cnt.
Quoted 6 lines
Quoted 4 lines
allocate memory. But, we have to move a
bpf_jit_uncharge_modmem()tobpf_dummy_ops. For me, this trade-off that include removing an allocation and adding abpf_jit_uncharge_modmem()make no sense.Which part make no sense? Having
bpf_dummy_opscharge/uncharge memory also?
Simplifying bpf_dummy_ops by removing the duty of allocation but adding the duty of uncharge memory doesn't make sense to me in terms of simplification. Although the lines of code would be similar, it actually makes it more complicated than before.
Quoted 15 lines
The
bpf_dummy_ops()uses the belowbpf_struct_ops_free_trampoline()which does uncharge and free.bpf_struct_ops_prepare_trampoline()does charge and alloc. charge/alloc matches with uncharge/free.Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
void
bpf_struct_ops_free_trampoline(void *image)
{
bpf_jit_uncharge_modmem(PAGE_SIZE);
arch_free_bpf_trampoline(image,PAGE_SIZE);
}
Martin KaFai Lau
reviewerRe: [PATCH bpf-next v2 2/3] bpf: struct_ops supports more than one page for trampolines.
Quoted 31 lines · Kui-Feng Lee
Quoted 28 lines · Martin KaFai Lau
Quoted 26 lines · Kui-Feng Lee
Quoted 11 lines · Martin KaFai Lau
Quoted 6 lines · Kui-Feng Lee
One thing I forgot to mention is that
bpf_dummy_opshas to callbpf_jit_uncharge_modmem(PAGE_SIZE)as well. The other option is to movebpf_jit_charge_modmem()out ofbpf_struct_ops_prepare_trampoline(), meaningbpf_struct_ops_map_update_elem()should handle the case that the allocation inbpf_struct_ops_prepare_trampoline()successes, butbpf_jit_charge_modmem()fails.Keep the charge/uncharge in
bpf_struct_ops_prepare_trampoline().It is fine to have
bpf_dummy_opscharge and then uncharge aPAGE_SIZE. There is no need to optimize forbpf_dummy_ops. Usebpf_struct_ops_free_trampoline()inbpf_dummy_opsto uncharge and free.Then, I don't get the point here.
I agree with moving the allocation intobpf_struct_ops_prepare_trampoline()to avoid duplication of the code about flags and tlinks. It really simplifies the code with the fact thatbpf_dummy_opsis still there. So, I tried to pass ast_maptobpf_struct_ops_prepare_trampoline()to keep page managements code together. But, you said to simplify the code ofbpf_dummy_opsby allocating pages inbpf_struct_ops_prepare_trampoline(), do bookkeeping inbpf_struct_ops_map_update_elem(), sobpf_dummy_opsdoesn't have to
Quoted 4 lines
Quoted 1 line
I don't think I ever mentioned to do book keeping in
bpf_struct_ops_map_update_elem(). Have you looked at my earlier code inbpf_struct_ops_prepare_trampoline()which also does the memory charging also?The bookkeeping that I am saying is about maintaining
image_pagesandimage_pages_cnt.
image_pages and image_pages_cnt are in the st_map (struct_ops map) itself. Why the bpf_struct_ops_map_update_elem() cannot keep track of the pages itself and need "_"bpf_struct_ops_prepare_trampoline() to handle it? It is not like refactoring the image_pages[_cnt] handling to _bpf_struct_ops_prepare_trampoline() and it can be reused by another caller. bpf_struct_ops_map_update_elem() is the only one needing to keep track of its own image_pages and image_pages_cnt.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
allocate memory. But, we have to move a
bpf_jit_uncharge_modmem()tobpf_dummy_ops. For me, this trade-off that include removing an allocation and adding abpf_jit_uncharge_modmem()make no sense.
Quoted 6 lines
Quoted 1 line
Which part make no sense? Having
bpf_dummy_opscharge/uncharge memory also?Simplifying
bpf_dummy_opsby removing the duty of allocation but adding the duty of uncharge memory doesn't make sense to me in terms of simplification. Although the lines of code would be similar, it actually makes it more complicated than before.
The bpf_dummy_ops does not need to call uncharge. It does not need to know the page is charged or not. It uses bpf_struct_ops_prepare_trampoline() to get a prepared trampoline page and bpf_struct_ops_free_trampoline() which does the uncharge+free. It is the alloc/free concept that you have been proposing which fits well here.
The bpf_dummy_ops is doing arch_alloc_bpf_trampoline and arch_free_bpf_trampoline.
The arch_alloc_bpf_trampoline will be gone (-1).
The arch_free_bpf_trampoline will be replaced (+0) by bpf_struct_ops_free_trampoline.
Untested code:
net/bpf/bpf_dummy_struct_ops.c +10 -13
@@ -89,8 +89,8 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr,struct bpf_dummy_ops_test_args *args;struct bpf_tramp_links *tlinks;struct bpf_tramp_link *link =NULL ; +unsigned int op_idx, image_off =0 ;void *image =NULL ; -unsigned int op_idx;int prog_ret;s32 type_id;int err;
@@ -114,12 +114,6 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr, goto out; } - image =arch_alloc_bpf_trampoline (PAGE_SIZE ); -if (!image) { - err = -ENOMEM ; - goto out; - } - link =kzalloc (sizeof (*link),GFP_USER );if (!link) { err = -ENOMEM ;
@@ -130,12 +124,14 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr,bpf_link_init (&link->link,BPF_LINK_TYPE_STRUCT_OPS , &bpf_struct_ops_link_lops, prog); op_idx = prog->expected_attach_type; - err =bpf_struct_ops_prepare_trampoline (tlinks, link, - &st_ops->func_models[op_idx], - &dummy_ops_test_ret_function, - image, image +PAGE_SIZE ); -if (err <0 ) + image =bpf_struct_ops_prepare_trampoline (tlinks, link, + &st_ops->func_models[op_idx], + &dummy_ops_test_ret_function, +NULL , &image_off, true); +if (IS_ERR (image)) { + err =PTR_ERR (image); goto out; + }arch_protect_bpf_trampoline (image,PAGE_SIZE ); prog_ret =dummy_ops_call_op (image, args);
@@ -147,7 +143,8 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr, err = -EFAULT ; out:kfree (args); -arch_free_bpf_trampoline (image,PAGE_SIZE ); +if (!IS_ERR_OR_NULL (image)) +bpf_struct_ops_free_trampoline (image);if (link)bpf_link_put (&link->link);kfree (tlinks);
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
void
bpf_struct_ops_free_trampoline(void *image)
{
bpf_jit_uncharge_modmem(PAGE_SIZE);
arch_free_bpf_trampoline(image,PAGE_SIZE);
}
^^^^^^^^^^^^^^^
To be clear, this bpf_struct_ops_free_trampoline() function that does the uncharge+free.
~~~~~~~~~~~~~~~~
Lets summarize a bit of the thread:
. I think we agree duplicating codes from
bpf_struct_ops_prepare_trampoline() is not good.
. bpf_struct_ops_prepare_trampoline() can allocate page.
. The first concern was there is no alloc/free pairing.
Then adding bpf_struct_ops_free_trampoline was suggested to do the uncharge+free together,
while bpf_struct_ops_prepare_trampoline does the charge+alloc
. The second concern was bpf_struct_ops_prepare_trampoline()
is not obvious that a free(page) needs to be done. It is more like a naming perception since returning a page is already a good signal that needs to be freed. However, it is open for a function name change if it can make the "alloc" part more obvious.
. Another concern was bpf_dummy_ops now needs to remember it
needs to uncharge. bpf_struct_ops_free_trampoline() should address it also since it does uncharge+free.
. Another point brought up was having bpf_struct_ops_prepare_trampoline()
store the allocated page in st_map->image_pages instead of map_update doing it itself which was covered in the first part of this email.
I hope the code pieces have addressed the points brought up above. I have combined these code pieces in a patch
which is only compiler tested. Hope it will make things clearer (first patch):
https://git.kernel.org/pub/scm/linux/kernel/git/martin.lau/bpf-next.git/log/?h=struct_ops.images