bpf: bpf trampoline improvements
cover bpf: bpf trampoline improvements
Patch #1: Fix memleak in bpf trampoline found on our server Patch #2: Improve the trampoline ksym name
Yafang Shao (2):
bpf: Fix memleak due to fentry attach failure
bpf: Show total linked progs cnt instead of selector in trampoline
ksym
include/linux/bpf.h | 1 - kernel/bpf/trampoline.c | 24 +++++++++++++++++------- 2 files changed, 17 insertions(+), 8 deletions(-)
Signature · 1 line
1.8.3.1
PATCH 1/2 bpf: Fix memleak due to fentry attach failure
If it fails to attach fentry, the allocated bpf trampoline image will be left in the system. That can be verified by checking /proc/kallsyms.
This meamleak can be verified by a simple bpf program as follows,
SEC("fentry/trap_init")
int fentry_run()
{
return 0;
}It will fail to attach trap_init because this function is freed after kernel init, and then we can find the trampoline image is left in the system by checking /proc/kallsyms.
$ tail /proc/kallsyms ffffffffc0613000 t bpf_trampoline_6442453466_1 [bpf] ffffffffc06c3000 t bpf_trampoline_6442453466_1 [bpf]
$ bpftool btf dump file /sys/kernel/btf/vmlinux | grep "FUNC 'trap_init'" [2522] FUNC 'trap_init' type_id=119 linkage=static
$ echo $((6442453466 & 0x7fffffff)) 2522
Note that there are two left bpf trampoline images, that is because the libbpf will fallback to raw tracepoint if -EINVAL is returned.
--- kernel/bpf/trampoline.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)
kernel/bpf/trampoline.c +15 -2
@@ -251,6 +251,15 @@ static int register_fentry(struct bpf_trampoline *tr, void *new_addr)return tlinks; } +static void bpf_tramp_image_free (struct bpf_tramp_image *im) +{ +bpf_image_ksym_del (&im->ksym); +bpf_jit_free_exec (im->image); +bpf_jit_uncharge_modmem (PAGE_SIZE ); +percpu_ref_exit (&im->pcref); +kfree (im); +} +static void __bpf_tramp_image_put_deferred (struct work_struct *work) {struct bpf_tramp_image *im;
@@ -438,7 +447,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut &tr->func.model, tr->flags, tlinks, tr->func.addr);if (err <0 ) - goto out; + goto out_free;set_memory_rox ((long )im->image,1 );
@@ -468,7 +477,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut }#endif if (err) - goto out; + goto out_free;if (tr->cur_image)bpf_tramp_image_put (tr->cur_image);
@@ -480,6 +489,10 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut tr->flags = orig_flags;kfree (tlinks);return err; + +out_free: +bpf_tramp_image_free (im); + goto out; }static enum bpf_tramp_prog_type bpf_attach_type_to_tramp (struct bpf_prog *prog)
Signature · 1 line
1.8.3.1
Song Liu
reviewerRe: [PATCH bpf-next 1/2] bpf: Fix memleak due to fentry attach failure
Quoted 11 lines · Yafang Shao
If it fails to attach fentry, the allocated bpf trampoline image will be left in the system. That can be verified by checking
/proc/kallsyms.This meamleak can be verified by a simple bpf program as follows,
SEC("fentry/trap_init") int fentry_run() { return 0; }
Nice trick! We can build some interesting tests with trap_init.
Quoted 11 lines
It will fail to attach
trap_initbecause this function is freed after kernel init, and then we can find the trampoline image is left in the system by checking/proc/kallsyms.$ tail /proc/kallsyms ffffffffc0613000 t bpf_trampoline_6442453466_1 [bpf] ffffffffc06c3000 t bpf_trampoline_6442453466_1 [bpf]$ bpftool btf dump file /sys/kernel/btf/vmlinux | grep "FUNC 'trap_init'" [2522] FUNC 'trap_init' type_id=119 linkage=static
Quoted 7 lines
$ echo $((6442453466 & 0x7fffffff)) 2522Note that there are two left bpf trampoline images, that is because the libbpf will fallback to raw tracepoint if
-EINVALis returned.Signed-off-by Yafang Shao <[email protected]>
I guess we need:
Quoted 10 lines
--- kernel/bpf/trampoline.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)kernel/bpf/trampoline.c +0
-0@@ -251,6 +251,15 @@ static int register_fentry(struct bpf_trampoline *tr, void *new_addr)return tlinks;
kernel/bpf/trampoline.c static int register_fentry(struct bpf_trampoline *tr, void *new_addr) · line 261
Quoted 10 lines
kernel/bpf/trampoline.c +8
-0} +static void bpf_tramp_image_free (struct bpf_tramp_image *im) +{ +bpf_image_ksym_del (&im->ksym); +bpf_jit_free_exec (im->image); +bpf_jit_uncharge_modmem (PAGE_SIZE ); +percpu_ref_exit (&im->pcref); +kfree (im); +}
Can we share some of this function with __bpf_tramp_image_put_deferred?
Thanks,
Song
[...]
Re: [PATCH bpf-next 1/2] bpf: Fix memleak due to fentry attach failure
Quoted 8 lines · Song Liu
Quoted 6 lines · Yafang Shao
If it fails to attach fentry, the allocated bpf trampoline image will be left in the system. That can be verified by checking
/proc/kallsyms.This meamleak can be verified by a simple bpf program as follows,
Quoted 8 lines
Quoted 5 lines
SEC("fentry/trap_init") int fentry_run() { return 0; }Nice trick! We can build some interesting tests with
trap_init.
Good suggestion. I will think about it.
Quoted 14 lines
Quoted 14 lines
It will fail to attach
trap_initbecause this function is freed after kernel init, and then we can find the trampoline image is left in the system by checking/proc/kallsyms.$ tail /proc/kallsyms ffffffffc0613000 t bpf_trampoline_6442453466_1 [bpf] ffffffffc06c3000 t bpf_trampoline_6442453466_1 [bpf]$ bpftool btf dump file /sys/kernel/btf/vmlinux | grep "FUNC 'trap_init'" [2522] FUNC 'trap_init' type_id=119 linkage=static$ echo $((6442453466 & 0x7fffffff)) 2522
Quoted 9 lines
Quoted 4 lines
Note that there are two left bpf trampoline images, that is because the libbpf will fallback to raw tracepoint if
-EINVALis returned.Signed-off-by Yafang Shao <[email protected]>I guess we need:
Fixes e21aa341785c ("bpf: Fix fexit trampoline.")
Thanks for pointing it out. I will add it.
Quoted 12 lines
Quoted 12 lines
--- kernel/bpf/trampoline.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)kernel/bpf/trampoline.c +0
-0@@ -251,6 +251,15 @@ static int register_fentry(struct bpf_trampoline *tr, void *new_addr)return tlinks; }
Quoted 11 lines
kernel/bpf/trampoline.c static int register_fentry(struct bpf_trampoline *tr, void *new_addr) · line 261
Quoted 8 lines
kernel/bpf/trampoline.c +8
-0+static void bpf_tramp_image_free (struct bpf_tramp_image *im) +{ +bpf_image_ksym_del (&im->ksym); +bpf_jit_free_exec (im->image); +bpf_jit_uncharge_modmem (PAGE_SIZE ); +percpu_ref_exit (&im->pcref); +kfree (im); +}Can we share some of this function with
__bpf_tramp_image_put_deferred?
It seems we can introduce a generic helper as follows,
static void __bpf_tramp_image_free(struct bpf_tramp_image *im)
{
bpf_image_ksym_del(&im->ksym);
bpf_jit_free_exec(im->image);
bpf_jit_uncharge_modmem(PAGE_SIZE);
percpu_ref_exit(&im->pcref);}
And then use it in both bpf_tramp_image_free() and __bpf_tramp_image_put_deferred().
WDYT?
Signature · 2 lines
Regards Yafang
Song Liu
reviewerRe: [PATCH bpf-next 1/2] bpf: Fix memleak due to fentry attach failure
Quoted 2 lines · Yafang Shao
On Wed, May 10, 2023 at 1:41 AM Song Liu <[email protected]> wrote:
[...]
Quoted 19 lines
Quoted 11 lines
kernel/bpf/trampoline.c static int register_fentry(struct bpf_trampoline *tr, void *new_addr) · line 261
Quoted 8 lines
kernel/bpf/trampoline.c +8
-0+static void bpf_tramp_image_free (struct bpf_tramp_image *im) +{ +bpf_image_ksym_del (&im->ksym); +bpf_jit_free_exec (im->image); +bpf_jit_uncharge_modmem (PAGE_SIZE ); +percpu_ref_exit (&im->pcref); +kfree (im); +}Can we share some of this function with
__bpf_tramp_image_put_deferred?It seems we can introduce a generic helper as follows,
static void__bpf_tramp_image_free(struct bpf_tramp_image *im){ bpf_image_ksym_del(&im->ksym); bpf_jit_free_exec(im->image); bpf_jit_uncharge_modmem(PAGE_SIZE); percpu_ref_exit(&im->pcref);
Quoted 5 lines
}
And then use it in both
bpf_tramp_image_free()and__bpf_tramp_image_put_deferred().
WDYT?
How about we also use kfree_rcu() in bpf_tramp_image_free()?
Thanks,
Song
Re: [PATCH bpf-next 1/2] bpf: Fix memleak due to fentry attach failure
Quoted 26 lines · Song Liu
Quoted 2 lines · Yafang Shao
On Wed, May 10, 2023 at 1:41 AM Song Liu <[email protected]> wrote:
[...]
Quoted 21 lines
Quoted 11 lines
kernel/bpf/trampoline.c static int register_fentry(struct bpf_trampoline *tr, void *new_addr) · line 261
Quoted 8 lines
kernel/bpf/trampoline.c +8
-0+static void bpf_tramp_image_free (struct bpf_tramp_image *im) +{ +bpf_image_ksym_del (&im->ksym); +bpf_jit_free_exec (im->image); +bpf_jit_uncharge_modmem (PAGE_SIZE ); +percpu_ref_exit (&im->pcref); +kfree (im); +}Can we share some of this function with
__bpf_tramp_image_put_deferred?It seems we can introduce a generic helper as follows,
static void__bpf_tramp_image_free(struct bpf_tramp_image *im){ bpf_image_ksym_del(&im->ksym); bpf_jit_free_exec(im->image); bpf_jit_uncharge_modmem(PAGE_SIZE); percpu_ref_exit(&im->pcref);}
Quoted 6 lines
Quoted 3 lines
And then use it in both
bpf_tramp_image_free()and__bpf_tramp_image_put_deferred().
WDYT?How about we also use
kfree_rcu()inbpf_tramp_image_free()?
Looks good. I will change it.
Signature · 2 lines
Regards Yafang
PATCH 2/2 bpf: Show total linked progs cnt instead of selector in trampoline ksym
In reply to Yafang ShaoAfter commit e21aa341785c ("bpf: Fix fexit trampoline."), the selector is only used to indicate how many times the bpf trampoline image are updated and been displayed in the trampoline ksym name. After the trampoline is freed, the count will start from 0 again.
So the count is a useless value to the user, we'd better
show a more meaningful value like how many progs are linked to this trampoline. After that change, the selector can be removed eventally. If the user want to check whether the bpf trampoline image has been updated or not, the user can also compare the address. Each time the trampoline image is updated, the address will change consequently.
--- include/linux/bpf.h | 1 - kernel/bpf/trampoline.c | 7 ++----- 2 files changed, 2 insertions(+), 6 deletions(-)
include/linux/bpf.h +0 -1
@@ -1125,7 +1125,6 @@ struct bpf_trampoline {int progs_cnt[BPF_TRAMP_MAX ];/* Executable image of trampoline */ struct bpf_tramp_image *cur_image; -u64 selector;struct module *mod; };
kernel/bpf/trampoline.c +2 -5
@@ -410,11 +410,10 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut err =unregister_fentry (tr, tr->cur_image->image);bpf_tramp_image_put (tr->cur_image); tr->cur_image =NULL ; - tr->selector =0 ; goto out; } - im =bpf_tramp_image_alloc (tr->key, tr->selector); + im =bpf_tramp_image_alloc (tr->key, total);if (IS_ERR (im)) { err =PTR_ERR (im); goto out;
@@ -451,8 +450,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutset_memory_rox ((long )im->image,1 ); -WARN_ON (tr->cur_image && tr->selector ==0 ); -WARN_ON (!tr->cur_image && tr->selector); +WARN_ON (tr->cur_image && total ==0 );if (tr->cur_image)/* progs already running at this address */ err =modify_fentry (tr, tr->cur_image->image, im->image, lock_direct_mutex);
@@ -482,7 +480,6 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutif (tr->cur_image)bpf_tramp_image_put (tr->cur_image); tr->cur_image = im; - tr->selector++; out:/* If any error happens, restore previous flags */ if (err)
Signature · 1 line
1.8.3.1
Song Liu
reviewerRe: [PATCH bpf-next 2/2] bpf: Show total linked progs cnt instead of selector in trampoline ksym
Quoted 11 lines · Yafang Shao
After commit
e21aa341785c("bpf: Fix fexit trampoline."), the selector is only used to indicate how many times the bpf trampoline image are updated and been displayed in the trampoline ksym name. After the trampoline is freed, the count will start from 0 again.
So the count is a useless value to the user, we'd better
show a more meaningful value like how many progs are linked to this trampoline. After that change, the selector can be removed eventally. If the user want to check whether the bpf trampoline image has been updated or not, the user can also compare the address. Each time the trampoline image is updated, the address will change consequently.
I wonder whether this will cause confusion to some users. Maybe the saving doesn't worth the churn.
Song
Quoted 58 lines
Signed-off-by Yafang Shao <[email protected]>--- include/linux/bpf.h | 1 - kernel/bpf/trampoline.c | 7 ++----- 2 files changed, 2 insertions(+), 6 deletions(-)include/linux/bpf.h +0
-1@@ -1125,7 +1125,6 @@ struct bpf_trampoline {int progs_cnt[BPF_TRAMP_MAX ];/* Executable image of trampoline */ struct bpf_tramp_image *cur_image; -u64 selector;struct module *mod; };kernel/bpf/trampoline.c +2
-6@@ -410,11 +410,10 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut err =unregister_fentry (tr, tr->cur_image->image);bpf_tramp_image_put (tr->cur_image); tr->cur_image =NULL ; - tr->selector =0 ; goto out; } - im =bpf_tramp_image_alloc (tr->key, tr->selector); + im =bpf_tramp_image_alloc (tr->key, total);if (IS_ERR (im)) { err =PTR_ERR (im); goto out;@@ -451,8 +450,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutset_memory_rox ((long )im->image,1 ); -WARN_ON (tr->cur_image && tr->selector ==0 ); -WARN_ON (!tr->cur_image && tr->selector); +WARN_ON (tr->cur_image && total ==0 );if (tr->cur_image)/* progs already running at this address */ err =modify_fentry (tr, tr->cur_image->image, im->image, lock_direct_mutex);@@ -482,7 +480,6 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutif (tr->cur_image)bpf_tramp_image_put (tr->cur_image); tr->cur_image = im; - tr->selector++; out:/* If any error happens, restore previous flags */ if (err) --1.8.3.1
Re: [PATCH bpf-next 2/2] bpf: Show total linked progs cnt instead of selector in trampoline ksym
Quoted 14 lines · Song Liu
Quoted 11 lines · Yafang Shao
After commit
e21aa341785c("bpf: Fix fexit trampoline."), the selector is only used to indicate how many times the bpf trampoline image are updated and been displayed in the trampoline ksym name. After the trampoline is freed, the count will start from 0 again.
So the count is a useless value to the user, we'd better
show a more meaningful value like how many progs are linked to this trampoline. After that change, the selector can be removed eventally. If the user want to check whether the bpf trampoline image has been updated or not, the user can also compare the address. Each time the trampoline image is updated, the address will change consequently.
Quoted 2 lines
I wonder whether this will cause confusion to some users. Maybe the saving doesn't worth the churn.
The trampoline ksym name as such:ffffffffc06c3000 t bpf_trampoline_6442453466_1 [bpf]
I don't know what the user may use the selector for. It seems that the selector is meaningless. While the cnt of linked progs can really help users, with which the user can easily figure out how many progs are linked to a kernel function.
However the key in the ksym name really confused me before, because its meaning was changed.
In the old kernel(for example, the 4.19), it is
bpf_trampoline_[BTF_ID]
while in the new kernel, it is
bpf_trampoline_[ OBJ_ID | BTF_ID ]_[ SELECTOR ]
But I think that is not a big problem, because the user tools can be changed to (key & 0x7fffffff) to make it backward compatible.
Currently there's no document on the name, so we can choose what we prefer. After we doc it, we have to keep it backward compatible. BTW, I think we should add a document on the name, otherwise the user has to read the kernel code to parse it.
Signature · 2 lines
Regards Yafang
Song Liu
reviewerRe: [PATCH bpf-next 2/2] bpf: Show total linked progs cnt instead of selector in trampoline ksym
Quoted 19 lines · Yafang Shao
Quoted 14 lines · Song Liu
Quoted 11 lines · Yafang Shao
After commit
e21aa341785c("bpf: Fix fexit trampoline."), the selector is only used to indicate how many times the bpf trampoline image are updated and been displayed in the trampoline ksym name. After the trampoline is freed, the count will start from 0 again.
So the count is a useless value to the user, we'd better
show a more meaningful value like how many progs are linked to this trampoline. After that change, the selector can be removed eventally. If the user want to check whether the bpf trampoline image has been updated or not, the user can also compare the address. Each time the trampoline image is updated, the address will change consequently.Quoted 2 lines
I wonder whether this will cause confusion to some users. Maybe the saving doesn't worth the churn.
Quoted 7 lines
The trampoline ksym name as such:
ffffffffc06c3000tbpf_trampoline_6442453466_1[bpf]I don't know what the user may use the selector for. It seems that the selector is meaningless. While the cnt of linked progs can really help users, with which the user can easily figure out how many progs are linked to a kernel function.
Hmm, agreed that the chance to break user space is low. Maybe we can just remove it? IOW, only keep bpf_trampoline_6442453466
Thanks,
Song
Re: [PATCH bpf-next 2/2] bpf: Show total linked progs cnt instead of selector in trampoline ksym
Quoted 24 lines · Song Liu
Quoted 22 lines · Yafang Shao
Quoted 14 lines · Song Liu
Quoted 11 lines · Yafang Shao
After commit
e21aa341785c("bpf: Fix fexit trampoline."), the selector is only used to indicate how many times the bpf trampoline image are updated and been displayed in the trampoline ksym name. After the trampoline is freed, the count will start from 0 again.
So the count is a useless value to the user, we'd better
show a more meaningful value like how many progs are linked to this trampoline. After that change, the selector can be removed eventally. If the user want to check whether the bpf trampoline image has been updated or not, the user can also compare the address. Each time the trampoline image is updated, the address will change consequently.Quoted 2 lines
I wonder whether this will cause confusion to some users. Maybe the saving doesn't worth the churn.
The trampoline ksym name as such:
ffffffffc06c3000tbpf_trampoline_6442453466_1[bpf]
Quoted 8 lines
Quoted 4 lines
I don't know what the user may use the selector for. It seems that the selector is meaningless. While the cnt of linked progs can really help users, with which the user can easily figure out how many progs are linked to a kernel function.
Hmm, agreed that the chance to break user space is low. Maybe we can just remove it? IOW, only keep
bpf_trampoline_6442453466
Agree. I will do it as you suggested.
Signature · 2 lines
Regards Yafang
Re: [PATCH bpf-next 2/2] bpf: Show total linked progs cnt instead of selector in trampoline ksym
Quoted 30 lines · Yafang Shao
Quoted 29 lines · Song Liu
Quoted 26 lines · Yafang Shao
Quoted 14 lines · Song Liu
Quoted 11 lines · Yafang Shao
After commit
e21aa341785c("bpf: Fix fexit trampoline."), the selector is only used to indicate how many times the bpf trampoline image are updated and been displayed in the trampoline ksym name. After the trampoline is freed, the count will start from 0 again.
So the count is a useless value to the user, we'd better
show a more meaningful value like how many progs are linked to this trampoline. After that change, the selector can be removed eventally. If the user want to check whether the bpf trampoline image has been updated or not, the user can also compare the address. Each time the trampoline image is updated, the address will change consequently.Quoted 2 lines
I wonder whether this will cause confusion to some users. Maybe the saving doesn't worth the churn.
The trampoline ksym name as such:
ffffffffc06c3000tbpf_trampoline_6442453466_1[bpf]I don't know what the user may use the selector for. It seems that the selector is meaningless. While the cnt of linked progs can really help users, with which the user can easily figure out how many progs are linked to a kernel function.
Quoted 5 lines
Quoted 3 lines
Hmm, agreed that the chance to break user space is low. Maybe we can just remove it? IOW, only keep
bpf_trampoline_6442453466Agree. I will do it as you suggested.
perf is actually is still trying parse the old name
/* .. and only for trampolines and dispatchers */
if ((sscanf(name, "bpf_trampoline_%lu", &id) == 1) ||
(sscanf(name, "bpf_dispatcher_%s", disp) == 1))
err = process_bpf_image(name, start, data);so this change would actualy fix it ;-)
thanks,
jirka