bpf: bpf trampoline improvements
v2 cover bpf: bpf trampoline improvements
When we run fexit bpf programs (e.g. attaching tcp_recvmsg) on our servers which were running old kernels, some of these servers crashed. Finally we figured out that it was caused by the same issue resolved by commit e21aa341785c ("bpf: Fix fexit trampoline."). After we backported that commit, the crash disappears. However new issues are introduced by that commit. This patchset fixes them.
PATCH #1: Fix a memory leak found on our server
PATCH #2: Remove bpf trampoline selector and also fix the issue in perf
caused by the name change in bpf trampoline
PATCH #3: Show target_{obj,btf}_id when link to a bpf trampoline
v1->v2:
- Reuse the common code between __bpf_tramp_image_put_deferred and bpf_tramp_image_free (Song)
- Add fixes tag in patch #1 (Song)
- Restore the old bpf trampoline name format (Song)
- Jiri pointed out a issue in perf
- Show btf information in the tracing link info
Yafang Shao (3):
bpf: Fix memleak due to fentry attach failure
bpf: Remove bpf trampoline selector
bpf: Show target_{obj,btf}_id in tracing link info
include/linux/bpf.h | 1 - kernel/bpf/syscall.c | 12 ++++++++++-- kernel/bpf/trampoline.c | 32 +++++++++++++++++++------------- tools/bpf/bpftool/link.c | 4 ++++ 4 files changed, 33 insertions(+), 16 deletions(-)
Signature · 1 line
1.8.3.1
PATCH v2 1/3 bpf: Fix memleak due to fentry attach failure
If it fails to attach fentry, the allocated bpf trampoline image will be left in the system. That can be verified by checking /proc/kallsyms.
This meamleak can be verified by a simple bpf program as follows,
SEC("fentry/trap_init")
int fentry_run()
{
return 0;
}It will fail to attach trap_init because this function is freed after kernel init, and then we can find the trampoline image is left in the system by checking /proc/kallsyms.
$ tail /proc/kallsyms ffffffffc0613000 t bpf_trampoline_6442453466_1 [bpf] ffffffffc06c3000 t bpf_trampoline_6442453466_1 [bpf]
$ bpftool btf dump file /sys/kernel/btf/vmlinux | grep "FUNC 'trap_init'" [2522] FUNC 'trap_init' type_id=119 linkage=static
$ echo $((6442453466 & 0x7fffffff)) 2522
Note that there are two left bpf trampoline images, that is because the libbpf will fallback to raw tracepoint if -EINVAL is returned.
--- kernel/bpf/trampoline.c | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-)
kernel/bpf/trampoline.c +15 -6
@@ -251,11 +251,8 @@ static int register_fentry(struct bpf_trampoline *tr, void *new_addr)return tlinks; } -static void __bpf_tramp_image_put_deferred (struct work_struct *work) +static void bpf_tramp_image_free (struct bpf_tramp_image *im) { -struct bpf_tramp_image *im; - - im = container_of(work, struct bpf_tramp_image, work);bpf_image_ksym_del (&im->ksym);bpf_jit_free_exec (im->image);bpf_jit_uncharge_modmem (PAGE_SIZE );
@@ -263,6 +260,14 @@ static void __bpf_tramp_image_put_deferred(struct work_struct *work)kfree_rcu (im, rcu); } +static void __bpf_tramp_image_put_deferred (struct work_struct *work) +{ +struct bpf_tramp_image *im; + + im = container_of(work, struct bpf_tramp_image, work); +bpf_tramp_image_free (im); +} +/* callback, fexit step 3 or fentry step 2 */ static void __bpf_tramp_image_put_rcu (struct rcu_head *rcu) {
@@ -438,7 +443,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut &tr->func.model, tr->flags, tlinks, tr->func.addr);if (err <0 ) - goto out; + goto out_free;set_memory_rox ((long )im->image,1 );
@@ -468,7 +473,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut }#endif if (err) - goto out; + goto out_free;if (tr->cur_image)bpf_tramp_image_put (tr->cur_image);
@@ -480,6 +485,10 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut tr->flags = orig_flags;kfree (tlinks);return err; + +out_free: +bpf_tramp_image_free (im); + goto out; }static enum bpf_tramp_prog_type bpf_attach_type_to_tramp (struct bpf_prog *prog)
Signature · 1 line
1.8.3.1
Re: [PATCH bpf-next v2 1/3] bpf: Fix memleak due to fentry attach failure
Quoted 25 lines · Yafang Shao
If it fails to attach fentry, the allocated bpf trampoline image will be left in the system. That can be verified by checking
/proc/kallsyms.This meamleak can be verified by a simple bpf program as follows,
SEC("fentry/trap_init") int fentry_run() { return 0; }It will fail to attach
trap_initbecause this function is freed after kernel init, and then we can find the trampoline image is left in the system by checking/proc/kallsyms.$ tail /proc/kallsyms ffffffffc0613000 t bpf_trampoline_6442453466_1 [bpf] ffffffffc06c3000 t bpf_trampoline_6442453466_1 [bpf]$ bpftool btf dump file /sys/kernel/btf/vmlinux | grep "FUNC 'trap_init'" [2522] FUNC 'trap_init' type_id=119 linkage=static$ echo $((6442453466 & 0x7fffffff)) 2522
Quoted 7 lines
Note that there are two left bpf trampoline images, that is because the libbpf will fallback to raw tracepoint if
-EINVALis returned.Fixes e21aa341785c ("bpf: Fix fexit trampoline.")Signed-off-by Yafang Shao <[email protected]>Cc Song Liu <[email protected]>Cc Jiri Olsa <[email protected]>
Daniel Borkmann
maintainerRe: [PATCH bpf-next v2 1/3] bpf: Fix memleak due to fentry attach failure
Quoted 29 lines · Song Liu
Quoted 28 lines · Yafang Shao
If it fails to attach fentry, the allocated bpf trampoline image will be left in the system. That can be verified by checking
/proc/kallsyms.This meamleak can be verified by a simple bpf program as follows,
SEC("fentry/trap_init") int fentry_run() { return 0; }It will fail to attach
trap_initbecause this function is freed after kernel init, and then we can find the trampoline image is left in the system by checking/proc/kallsyms.$ tail /proc/kallsyms ffffffffc0613000 t bpf_trampoline_6442453466_1 [bpf] ffffffffc06c3000 t bpf_trampoline_6442453466_1 [bpf]$ bpftool btf dump file /sys/kernel/btf/vmlinux | grep "FUNC 'trap_init'" [2522] FUNC 'trap_init' type_id=119 linkage=static$ echo $((6442453466 & 0x7fffffff)) 2522Note that there are two left bpf trampoline images, that is because the libbpf will fallback to raw tracepoint if
-EINVALis returned.
Quoted 6 lines
Quoted 4 lines
Fixes e21aa341785c ("bpf: Fix fexit trampoline.")Signed-off-by Yafang Shao <[email protected]>Cc Song Liu <[email protected]>Cc Jiri Olsa <[email protected]>Acked-by Song Liu <[email protected]>
Won't this trigger a UAF for the case when progs are already running at this address aka modify_fentry() fails where you would then also hit the goto out_free path? This looks not correct to me.
Thanks,
Daniel
Re: [PATCH bpf-next v2 1/3] bpf: Fix memleak due to fentry attach failure
Quoted 38 lines · Daniel Borkmann
Quoted 33 lines · Song Liu
Quoted 32 lines · Yafang Shao
If it fails to attach fentry, the allocated bpf trampoline image will be left in the system. That can be verified by checking
/proc/kallsyms.This meamleak can be verified by a simple bpf program as follows,
SEC("fentry/trap_init") int fentry_run() { return 0; }It will fail to attach
trap_initbecause this function is freed after kernel init, and then we can find the trampoline image is left in the system by checking/proc/kallsyms.$ tail /proc/kallsyms ffffffffc0613000 t bpf_trampoline_6442453466_1 [bpf] ffffffffc06c3000 t bpf_trampoline_6442453466_1 [bpf]$ bpftool btf dump file /sys/kernel/btf/vmlinux | grep "FUNC 'trap_init'" [2522] FUNC 'trap_init' type_id=119 linkage=static$ echo $((6442453466 & 0x7fffffff)) 2522Note that there are two left bpf trampoline images, that is because the libbpf will fallback to raw tracepoint if
-EINVALis returned.Fixes e21aa341785c ("bpf: Fix fexit trampoline.")Signed-off-by Yafang Shao <[email protected]>Cc Song Liu <[email protected]>Cc Jiri Olsa <[email protected]>
Quoted 5 lines
Quoted 1 line
Acked-by Song Liu <[email protected]>Won't this trigger a UAF for the case when progs are already running at this address aka
modify_fentry()fails where you would then also hit the gotoout_freepath? This looks not correct to me.
I am not following. If modify_fentry() fails, we will not use the new image anywhere, no? Did I miss something?
Thanks,
Song
Daniel Borkmann
maintainerRe: [PATCH bpf-next v2 1/3] bpf: Fix memleak due to fentry attach failure
Quoted 35 lines · Song Liu
Quoted 35 lines · Daniel Borkmann
Quoted 33 lines · Song Liu
Quoted 32 lines · Yafang Shao
If it fails to attach fentry, the allocated bpf trampoline image will be left in the system. That can be verified by checking
/proc/kallsyms.This meamleak can be verified by a simple bpf program as follows,
SEC("fentry/trap_init") int fentry_run() { return 0; }It will fail to attach
trap_initbecause this function is freed after kernel init, and then we can find the trampoline image is left in the system by checking/proc/kallsyms.$ tail /proc/kallsyms ffffffffc0613000 t bpf_trampoline_6442453466_1 [bpf] ffffffffc06c3000 t bpf_trampoline_6442453466_1 [bpf]$ bpftool btf dump file /sys/kernel/btf/vmlinux | grep "FUNC 'trap_init'" [2522] FUNC 'trap_init' type_id=119 linkage=static$ echo $((6442453466 & 0x7fffffff)) 2522Note that there are two left bpf trampoline images, that is because the libbpf will fallback to raw tracepoint if
-EINVALis returned.Fixes e21aa341785c ("bpf: Fix fexit trampoline.")Signed-off-by Yafang Shao <[email protected]>Cc Song Liu <[email protected]>Cc Jiri Olsa <[email protected]>
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Acked-by Song Liu <[email protected]>Won't this trigger a UAF for the case when progs are already running at this address aka
modify_fentry()fails where you would then also hit the gotoout_freepath? This looks not correct to me.I am not following. If
modify_fentry()fails, we will not use the new image anywhere, no? Did I miss something?
Hm, agree, I think I got confused with the again label earlier. Looks ok indeed. Applied, thanks!
PATCH v2 2/3 bpf: Remove bpf trampoline selector
In reply to Yafang ShaoAfter commit e21aa341785c ("bpf: Fix fexit trampoline."), the selector is only used to indicate how many times the bpf trampoline image are updated and been displayed in the trampoline ksym name. After the trampoline is freed, the selector will start from 0 again. So the selector is a useless value to the user. We can remove it.
If the user want to check whether the bpf trampoline image has been updated or not, the user can compare the address. Each time the trampoline image is updated, the address will change consequently.
Jiri pointed out antoher issue that perf is still using the old name "bpf_trampoline_%lu", so this change can fix the issue in perf.
--- include/linux/bpf.h | 1 - kernel/bpf/trampoline.c | 11 ++++------- 2 files changed, 4 insertions(+), 8 deletions(-)
include/linux/bpf.h +0 -1
@@ -1125,7 +1125,6 @@ struct bpf_trampoline {int progs_cnt[BPF_TRAMP_MAX ];/* Executable image of trampoline */ struct bpf_tramp_image *cur_image; -u64 selector;struct module *mod; };
kernel/bpf/trampoline.c +4 -7
@@ -349,7 +349,7 @@ static void bpf_tramp_image_put(struct bpf_tramp_image *im)call_rcu_tasks_trace (&im->rcu, __bpf_tramp_image_put_rcu_tasks); } -static struct bpf_tramp_image *bpf_tramp_image_alloc (u64 key,u32 idx) +static struct bpf_tramp_image *bpf_tramp_image_alloc (u64 key) {struct bpf_tramp_image *im;struct bpf_ksym *ksym;
@@ -376,7 +376,7 @@ static struct bpf_tramp_image *bpf_tramp_image_alloc(u64 key, u32 idx) ksym = &im->ksym;INIT_LIST_HEAD_RCU (&ksym->lnode); -snprintf (ksym->name,KSYM_NAME_LEN ,"bpf_trampoline_%llu_%u" , key, idx); +snprintf (ksym->name,KSYM_NAME_LEN ,"bpf_trampoline_%llu" , key);bpf_image_ksym_add (image, ksym);return im;
@@ -406,11 +406,10 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut err =unregister_fentry (tr, tr->cur_image->image);bpf_tramp_image_put (tr->cur_image); tr->cur_image =NULL ; - tr->selector =0 ; goto out; } - im =bpf_tramp_image_alloc (tr->key, tr->selector); + im =bpf_tramp_image_alloc (tr->key);if (IS_ERR (im)) { err =PTR_ERR (im); goto out;
@@ -447,8 +446,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutset_memory_rox ((long )im->image,1 ); -WARN_ON (tr->cur_image && tr->selector ==0 ); -WARN_ON (!tr->cur_image && tr->selector); +WARN_ON (tr->cur_image && total ==0 );if (tr->cur_image)/* progs already running at this address */ err =modify_fentry (tr, tr->cur_image->image, im->image, lock_direct_mutex);
@@ -478,7 +476,6 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutif (tr->cur_image)bpf_tramp_image_put (tr->cur_image); tr->cur_image = im; - tr->selector++; out:/* If any error happens, restore previous flags */ if (err)
Signature · 1 line
1.8.3.1
Re: [PATCH bpf-next v2 2/3] bpf: Remove bpf trampoline selector
Quoted 10 lines · Yafang Shao
After commit
e21aa341785c("bpf: Fix fexit trampoline."), the selector is only used to indicate how many times the bpf trampoline image are updated and been displayed in the trampoline ksym name. After the trampoline is freed, the selector will start from 0 again. So the selector is a useless value to the user. We can remove it.
If the user want to check whether the bpf trampoline image has been updated or not, the user can compare the address. Each time the trampoline image is updated, the address will change consequently.
Quoted 7 lines
Jiri pointed out antoher issue that perf is still using the old name "bpf_trampoline_%lu", so this change can fix the issue in perf.
Fixes e21aa341785c ("bpf: Fix fexit trampoline.")Signed-off-by Yafang Shao <[email protected]>Cc Song Liu <[email protected]>Cc Jiri Olsa <[email protected]>
Daniel Borkmann
maintainerRe: [PATCH bpf-next v2 2/3] bpf: Remove bpf trampoline selector
Quoted 14 lines · Song Liu
Quoted 13 lines · Yafang Shao
After commit
e21aa341785c("bpf: Fix fexit trampoline."), the selector is only used to indicate how many times the bpf trampoline image are updated and been displayed in the trampoline ksym name. After the trampoline is freed, the selector will start from 0 again. So the selector is a useless value to the user. We can remove it.
If the user want to check whether the bpf trampoline image has been updated or not, the user can compare the address. Each time the trampoline image is updated, the address will change consequently.Jiri pointed out antoher issue that perf is still using the old name "bpf_trampoline_%lu", so this change can fix the issue in perf.
Quoted 6 lines
Quoted 4 lines
Fixes e21aa341785c ("bpf: Fix fexit trampoline.")Signed-off-by Yafang Shao <[email protected]>Cc Song Liu <[email protected]>Cc Jiri Olsa <[email protected]>Acked-by Song Liu <[email protected]>
Lgtm, I took this individual one for now.
PATCH v2 3/3 bpf: Show target_{obj,btf}_id in tracing link info
In reply to Yafang ShaoThe target_btf_id can help us understand which kernel function is linked by a tracing prog. The target_btf_id and target_obj_id have already been exposed to userspace, so we just need to show them.
The result as follows,
tools/bpf/bpftool/bpftool link show
2: tracing prog 13
prog_type tracing attach_type trace_fentry
target_obj_id 1 target_btf_id 13964
pids trace(10673)$ tools/bpf/bpftool/bpftool link show -j
[{"id":2,"type":"tracing","prog_id":13,"prog_type":"tracing","attach_type":"trace_fentry","target_obj_id":1,"target_btf_id":13964,"pids":[{"pid":10673,"comm":"trace"}]}]$ cat /proc/10673/fdinfo/10 pos: 0 flags: 02000000 mnt_id: 15 ino: 2094 link_type: tracing link_id: 2 prog_tag: a04f5eef06a7f555 prog_id: 13 attach_type: 24 target_obj_id: 1 target_btf_id: 13964
$ tail /proc/kallsyms ffffffffc0400fa0 t bpf_prog_a04f5eef06a7f555_fentry_run [bpf] ffffffffc062f000 t bpf_trampoline_6442464908 [bpf]
$ echo $((6442464908 & 0x7fffffff)) $((6442464908 >> 32)) 13964 1
--- kernel/bpf/syscall.c | 12 ++++++++++-- tools/bpf/bpftool/link.c | 4 ++++ 2 files changed, 14 insertions(+), 2 deletions(-)
kernel/bpf/syscall.c +10 -2
@@ -2968,10 +2968,18 @@ static void bpf_tracing_link_show_fdinfo(const struct bpf_link *link, {struct bpf_tracing_link *tr_link = container_of(link, struct bpf_tracing_link, link.link); +u32 target_btf_id; +u32 target_obj_id; +bpf_trampoline_unpack_key (tr_link->trampoline->key, + &target_obj_id, &target_btf_id);seq_printf (seq, -"attach_type:\t%d\n" , - tr_link->attach_type); +"attach_type:\t%d\n" +"target_obj_id:\t%u\n" +"target_btf_id:\t%u\n" , + tr_link->attach_type, + target_obj_id, + target_btf_id); } static int bpf_tracing_link_fill_link_info(const struct bpf_link *link,
tools/bpf/bpftool/link.c +4 -0
@@ -195,6 +195,8 @@ static int show_link_close_json(int fd, struct bpf_link_info *info)show_link_attach_type_json (info->tracing.attach_type, json_wtr); +jsonw_uint_field (json_wtr,"target_obj_id" , info->tracing.target_obj_id); +jsonw_uint_field (json_wtr,"target_btf_id" , info->tracing.target_btf_id);break ;case BPF_LINK_TYPE_CGROUP : jsonw_lluint_field(json_wtr, "cgroup_id",
@@ -375,6 +377,8 @@ static int show_link_close_plain(int fd, struct bpf_link_info *info)printf ("\n\tprog_type %u " , prog_info.type);show_link_attach_type_plain (info->tracing.attach_type); +printf ("\n\ttarget_obj_id %u target_btf_id %u " , + info->tracing.target_obj_id, info->tracing.target_btf_id);break ;case BPF_LINK_TYPE_CGROUP :printf ("\n\tcgroup_id %zu " , (size_t )info->cgroup.cgroup_id);
Signature · 1 line
1.8.3.1
Re: [PATCH bpf-next v2 3/3] bpf: Show target_{obj,btf}_id in tracing link info
Quoted 1 line · Yafang Shao
[...]
Quoted 2 lines
Signed-off-by Yafang Shao <[email protected]>
The change looks good to me, except that we should split the change into two commits.
Also, this doesn't seem to be related to the other two patches. So it is a little weird to add it in v2.
Other than these.
Re: [PATCH bpf-next v2 3/3] bpf: Show target_{obj,btf}_id in tracing link info
Quoted 4 lines · Song Liu
Quoted 1 line · Yafang Shao
Quoted 8 lines
[...]
Quoted 2 lines
Signed-off-by Yafang Shao <[email protected]>The change looks good to me, except that we should split the change into two commits.
Will split it and send them. Thanks for your review.
Quoted 6 lines
Also, this doesn't seem to be related to the other two patches. So it is a little weird to add it in v2.
Other than these.
Acked-by Song Liu <[email protected]>
Signature · 2 lines
Regards Yafang