Skip to contentSkip to conversation

bpf: bpf trampoline improvements

PATCH In review Yafang Shao 4 participants 12 messages May 15, 2023 – May 16, 2023

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v2 cover bpf: bpf trampoline improvements

When we run fexit bpf programs (e.g. attaching tcp_recvmsg) on our servers which were running old kernels, some of these servers crashed. Finally we figured out that it was caused by the same issue resolved by commit e21aa341785c ("bpf: Fix fexit trampoline."). After we backported that commit, the crash disappears. However new issues are introduced by that commit. This patchset fixes them.

PATCH #1: Fix a memory leak found on our server
PATCH #2: Remove bpf trampoline selector and also fix the issue in perf
caused by the name change in bpf trampoline
PATCH #3: Show target_{obj,btf}_id when link to a bpf trampoline

v1->v2:
- Reuse the common code between __bpf_tramp_image_put_deferred and bpf_tramp_image_free (Song)
- Add fixes tag in patch #1 (Song)
- Restore the old bpf trampoline name format (Song)
- Jiri pointed out a issue in perf
- Show btf information in the tracing link info

Yafang Shao (3):
bpf: Fix memleak due to fentry attach failure
bpf: Remove bpf trampoline selector
bpf: Show target_{obj,btf}_id in tracing link info

 include/linux/bpf.h      |  1 -
 kernel/bpf/syscall.c     | 12 ++++++++++--
 kernel/bpf/trampoline.c  | 32 +++++++++++++++++++-------------
 tools/bpf/bpftool/link.c |  4 ++++
 4 files changed, 33 insertions(+), 16 deletions(-)
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH v2 1/3 bpf: Fix memleak due to fentry attach failure

If it fails to attach fentry, the allocated bpf trampoline image will be left in the system. That can be verified by checking /proc/kallsyms.

This meamleak can be verified by a simple bpf program as follows,

  SEC("fentry/trap_init")
  int fentry_run()
  {
      return 0;
  }

It will fail to attach trap_init because this function is freed after kernel init, and then we can find the trampoline image is left in the system by checking /proc/kallsyms.

  $ tail /proc/kallsyms
  ffffffffc0613000 t bpf_trampoline_6442453466_1  [bpf]
  ffffffffc06c3000 t bpf_trampoline_6442453466_1  [bpf]
  $ bpftool btf dump file /sys/kernel/btf/vmlinux | grep "FUNC 'trap_init'"
  [2522] FUNC 'trap_init' type_id=119 linkage=static
  $ echo $((6442453466 & 0x7fffffff))
  2522

Note that there are two left bpf trampoline images, that is because the libbpf will fallback to raw tracepoint if -EINVAL is returned.

Fixes e21aa341785c ("bpf: Fix fexit trampoline.")
Cc Song Liu <[email protected]>
Cc Jiri Olsa <[email protected]>
---
 kernel/bpf/trampoline.c | 21 +++++++++++++++------
 1 file changed, 15 insertions(+), 6 deletions(-)
kernel/bpf/trampoline.c +15 -6
index ac021bc..2a3849c 100644
@@ -251,11 +251,8 @@ static int register_fentry(struct bpf_trampoline *tr, void *new_addr)
 	return tlinks;
 }
 
-static void __bpf_tramp_image_put_deferred(struct work_struct *work)
+static void bpf_tramp_image_free(struct bpf_tramp_image *im)
 {
-	struct bpf_tramp_image *im;
-
-	im = container_of(work, struct bpf_tramp_image, work);
 	bpf_image_ksym_del(&im->ksym);
 	bpf_jit_free_exec(im->image);
 	bpf_jit_uncharge_modmem(PAGE_SIZE);
@@ -263,6 +260,14 @@ static void __bpf_tramp_image_put_deferred(struct work_struct *work)
 	kfree_rcu(im, rcu);
 }
 
+static void __bpf_tramp_image_put_deferred(struct work_struct *work)
+{
+	struct bpf_tramp_image *im;
+
+	im = container_of(work, struct bpf_tramp_image, work);
+	bpf_tramp_image_free(im);
+}
+
 /* callback, fexit step 3 or fentry step 2 */
 static void __bpf_tramp_image_put_rcu(struct rcu_head *rcu)
 {
@@ -438,7 +443,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut
 					  &tr->func.model, tr->flags, tlinks,
 					  tr->func.addr);
 	if (err < 0)
-		goto out;
+		goto out_free;
 
 	set_memory_rox((long)im->image, 1);
 
@@ -468,7 +473,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut
 	}
 #endif
 	if (err)
-		goto out;
+		goto out_free;
 
 	if (tr->cur_image)
 		bpf_tramp_image_put(tr->cur_image);
@@ -480,6 +485,10 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut
 		tr->flags = orig_flags;
 	kfree(tlinks);
 	return err;
+
+out_free:
+	bpf_tramp_image_free(im);
+	goto out;
 }
 
 static enum bpf_tramp_prog_type bpf_attach_type_to_tramp(struct bpf_prog *prog)
Signature · 1 line
1.8.3.1
reviewer Acked-by on v2 1/3
#2 lore raw

Re: [PATCH bpf-next v2 1/3] bpf: Fix memleak due to fentry attach failure

Quoted 25 lines · Yafang Shao

If it fails to attach fentry, the allocated bpf trampoline image will be left in the system. That can be verified by checking /proc/kallsyms.

This meamleak can be verified by a simple bpf program as follows,

  SEC("fentry/trap_init")
  int fentry_run()
  {
      return 0;
  }

It will fail to attach trap_init because this function is freed after kernel init, and then we can find the trampoline image is left in the system by checking /proc/kallsyms.

  $ tail /proc/kallsyms
  ffffffffc0613000 t bpf_trampoline_6442453466_1  [bpf]
  ffffffffc06c3000 t bpf_trampoline_6442453466_1  [bpf]
  $ bpftool btf dump file /sys/kernel/btf/vmlinux | grep "FUNC 'trap_init'"
  [2522] FUNC 'trap_init' type_id=119 linkage=static
  $ echo $((6442453466 & 0x7fffffff))
  2522
Quoted 7 lines

Note that there are two left bpf trampoline images, that is because the libbpf will fallback to raw tracepoint if -EINVAL is returned.

Fixes e21aa341785c ("bpf: Fix fexit trampoline.")
Cc Song Liu <[email protected]>
Cc Jiri Olsa <[email protected]>
maintainer
#3 lore raw

Re: [PATCH bpf-next v2 1/3] bpf: Fix memleak due to fentry attach failure

Quoted 29 lines · Song Liu
Quoted 28 lines · Yafang Shao

If it fails to attach fentry, the allocated bpf trampoline image will be left in the system. That can be verified by checking /proc/kallsyms.

This meamleak can be verified by a simple bpf program as follows,

   SEC("fentry/trap_init")
   int fentry_run()
   {
       return 0;
   }

It will fail to attach trap_init because this function is freed after kernel init, and then we can find the trampoline image is left in the system by checking /proc/kallsyms.

   $ tail /proc/kallsyms
   ffffffffc0613000 t bpf_trampoline_6442453466_1  [bpf]
   ffffffffc06c3000 t bpf_trampoline_6442453466_1  [bpf]
   $ bpftool btf dump file /sys/kernel/btf/vmlinux | grep "FUNC 'trap_init'"
   [2522] FUNC 'trap_init' type_id=119 linkage=static
   $ echo $((6442453466 & 0x7fffffff))
   2522

Note that there are two left bpf trampoline images, that is because the libbpf will fallback to raw tracepoint if -EINVAL is returned.

Quoted 6 lines
Quoted 4 lines
Fixes e21aa341785c ("bpf: Fix fexit trampoline.")
Cc Song Liu <[email protected]>
Cc Jiri Olsa <[email protected]>

Won't this trigger a UAF for the case when progs are already running at this address aka modify_fentry() fails where you would then also hit the goto out_free path? This looks not correct to me.

Thanks,
Daniel

#4 lore raw

Re: [PATCH bpf-next v2 1/3] bpf: Fix memleak due to fentry attach failure

Quoted 38 lines · Daniel Borkmann
Quoted 33 lines · Song Liu
Quoted 32 lines · Yafang Shao

If it fails to attach fentry, the allocated bpf trampoline image will be left in the system. That can be verified by checking /proc/kallsyms.

This meamleak can be verified by a simple bpf program as follows,

  SEC("fentry/trap_init")
  int fentry_run()
  {
      return 0;
  }

It will fail to attach trap_init because this function is freed after kernel init, and then we can find the trampoline image is left in the system by checking /proc/kallsyms.

  $ tail /proc/kallsyms
  ffffffffc0613000 t bpf_trampoline_6442453466_1  [bpf]
  ffffffffc06c3000 t bpf_trampoline_6442453466_1  [bpf]
  $ bpftool btf dump file /sys/kernel/btf/vmlinux | grep "FUNC 'trap_init'"
  [2522] FUNC 'trap_init' type_id=119 linkage=static
  $ echo $((6442453466 & 0x7fffffff))
  2522

Note that there are two left bpf trampoline images, that is because the libbpf will fallback to raw tracepoint if -EINVAL is returned.

Fixes e21aa341785c ("bpf: Fix fexit trampoline.")
Cc Song Liu <[email protected]>
Cc Jiri Olsa <[email protected]>
Quoted 5 lines
Quoted 1 line

Won't this trigger a UAF for the case when progs are already running at this address aka modify_fentry() fails where you would then also hit the goto out_free path? This looks not correct to me.

I am not following. If modify_fentry() fails, we will not use the new image anywhere, no? Did I miss something?

Thanks,
Song

maintainer
#5 lore raw

Re: [PATCH bpf-next v2 1/3] bpf: Fix memleak due to fentry attach failure

Quoted 35 lines · Song Liu
Quoted 35 lines · Daniel Borkmann
Quoted 33 lines · Song Liu
Quoted 32 lines · Yafang Shao

If it fails to attach fentry, the allocated bpf trampoline image will be left in the system. That can be verified by checking /proc/kallsyms.

This meamleak can be verified by a simple bpf program as follows,

   SEC("fentry/trap_init")
   int fentry_run()
   {
       return 0;
   }

It will fail to attach trap_init because this function is freed after kernel init, and then we can find the trampoline image is left in the system by checking /proc/kallsyms.

   $ tail /proc/kallsyms
   ffffffffc0613000 t bpf_trampoline_6442453466_1  [bpf]
   ffffffffc06c3000 t bpf_trampoline_6442453466_1  [bpf]
   $ bpftool btf dump file /sys/kernel/btf/vmlinux | grep "FUNC 'trap_init'"
   [2522] FUNC 'trap_init' type_id=119 linkage=static
   $ echo $((6442453466 & 0x7fffffff))
   2522

Note that there are two left bpf trampoline images, that is because the libbpf will fallback to raw tracepoint if -EINVAL is returned.

Fixes e21aa341785c ("bpf: Fix fexit trampoline.")
Cc Song Liu <[email protected]>
Cc Jiri Olsa <[email protected]>
Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Won't this trigger a UAF for the case when progs are already running at this address aka modify_fentry() fails where you would then also hit the goto out_free path? This looks not correct to me.

I am not following. If modify_fentry() fails, we will not use the new image anywhere, no? Did I miss something?

Hm, agree, I think I got confused with the again label earlier. Looks ok indeed. Applied, thanks!

#6 lore raw

PATCH v2 2/3 bpf: Remove bpf trampoline selector

In reply to Yafang Shao

After commit e21aa341785c ("bpf: Fix fexit trampoline."), the selector is only used to indicate how many times the bpf trampoline image are updated and been displayed in the trampoline ksym name. After the trampoline is freed, the selector will start from 0 again. So the selector is a useless value to the user. We can remove it.
If the user want to check whether the bpf trampoline image has been updated or not, the user can compare the address. Each time the trampoline image is updated, the address will change consequently.

Jiri pointed out antoher issue that perf is still using the old name "bpf_trampoline_%lu", so this change can fix the issue in perf.

Fixes e21aa341785c ("bpf: Fix fexit trampoline.")
Cc Song Liu <[email protected]>
Cc Jiri Olsa <[email protected]>
---
 include/linux/bpf.h     |  1 -
 kernel/bpf/trampoline.c | 11 ++++-------
 2 files changed, 4 insertions(+), 8 deletions(-)
include/linux/bpf.h +0 -1
index 456f33b..36e4b2d 100644
@@ -1125,7 +1125,6 @@ struct bpf_trampoline {
 	int progs_cnt[BPF_TRAMP_MAX];
 	/* Executable image of trampoline */
 	struct bpf_tramp_image *cur_image;
-	u64 selector;
 	struct module *mod;
 };
 
kernel/bpf/trampoline.c +4 -7
index 2a3849c..78acf28 100644
@@ -349,7 +349,7 @@ static void bpf_tramp_image_put(struct bpf_tramp_image *im)
 	call_rcu_tasks_trace(&im->rcu, __bpf_tramp_image_put_rcu_tasks);
 }
 
-static struct bpf_tramp_image *bpf_tramp_image_alloc(u64 key, u32 idx)
+static struct bpf_tramp_image *bpf_tramp_image_alloc(u64 key)
 {
 	struct bpf_tramp_image *im;
 	struct bpf_ksym *ksym;
@@ -376,7 +376,7 @@ static struct bpf_tramp_image *bpf_tramp_image_alloc(u64 key, u32 idx)
 
 	ksym = &im->ksym;
 	INIT_LIST_HEAD_RCU(&ksym->lnode);
-	snprintf(ksym->name, KSYM_NAME_LEN, "bpf_trampoline_%llu_%u", key, idx);
+	snprintf(ksym->name, KSYM_NAME_LEN, "bpf_trampoline_%llu", key);
 	bpf_image_ksym_add(image, ksym);
 	return im;
 
@@ -406,11 +406,10 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut
 		err = unregister_fentry(tr, tr->cur_image->image);
 		bpf_tramp_image_put(tr->cur_image);
 		tr->cur_image = NULL;
-		tr->selector = 0;
 		goto out;
 	}
 
-	im = bpf_tramp_image_alloc(tr->key, tr->selector);
+	im = bpf_tramp_image_alloc(tr->key);
 	if (IS_ERR(im)) {
 		err = PTR_ERR(im);
 		goto out;
@@ -447,8 +446,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut
 
 	set_memory_rox((long)im->image, 1);
 
-	WARN_ON(tr->cur_image && tr->selector == 0);
-	WARN_ON(!tr->cur_image && tr->selector);
+	WARN_ON(tr->cur_image && total == 0);
 	if (tr->cur_image)
 		/* progs already running at this address */
 		err = modify_fentry(tr, tr->cur_image->image, im->image, lock_direct_mutex);
@@ -478,7 +476,6 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut
 	if (tr->cur_image)
 		bpf_tramp_image_put(tr->cur_image);
 	tr->cur_image = im;
-	tr->selector++;
 out:
 	/* If any error happens, restore previous flags */
 	if (err)
Signature · 1 line
1.8.3.1
reviewer Acked-by on v2 2/3
#7 lore raw

Re: [PATCH bpf-next v2 2/3] bpf: Remove bpf trampoline selector

Quoted 10 lines · Yafang Shao

After commit e21aa341785c ("bpf: Fix fexit trampoline."), the selector is only used to indicate how many times the bpf trampoline image are updated and been displayed in the trampoline ksym name. After the trampoline is freed, the selector will start from 0 again. So the selector is a useless value to the user. We can remove it.
If the user want to check whether the bpf trampoline image has been updated or not, the user can compare the address. Each time the trampoline image is updated, the address will change consequently.

Quoted 7 lines

Jiri pointed out antoher issue that perf is still using the old name "bpf_trampoline_%lu", so this change can fix the issue in perf.

Fixes e21aa341785c ("bpf: Fix fexit trampoline.")
Cc Song Liu <[email protected]>
Cc Jiri Olsa <[email protected]>
maintainer
#8 lore raw

Re: [PATCH bpf-next v2 2/3] bpf: Remove bpf trampoline selector

Quoted 14 lines · Song Liu
Quoted 13 lines · Yafang Shao

After commit e21aa341785c ("bpf: Fix fexit trampoline."), the selector is only used to indicate how many times the bpf trampoline image are updated and been displayed in the trampoline ksym name. After the trampoline is freed, the selector will start from 0 again. So the selector is a useless value to the user. We can remove it.
If the user want to check whether the bpf trampoline image has been updated or not, the user can compare the address. Each time the trampoline image is updated, the address will change consequently.

Jiri pointed out antoher issue that perf is still using the old name "bpf_trampoline_%lu", so this change can fix the issue in perf.

Quoted 6 lines
Quoted 4 lines
Fixes e21aa341785c ("bpf: Fix fexit trampoline.")
Cc Song Liu <[email protected]>
Cc Jiri Olsa <[email protected]>

Lgtm, I took this individual one for now.

#9 lore raw

PATCH v2 3/3 bpf: Show target_{obj,btf}_id in tracing link info

In reply to Yafang Shao

The target_btf_id can help us understand which kernel function is linked by a tracing prog. The target_btf_id and target_obj_id have already been exposed to userspace, so we just need to show them.

The result as follows,

tools/bpf/bpftool/bpftool link show
2: tracing prog 13

        prog_type tracing  attach_type trace_fentry
        target_obj_id 1  target_btf_id 13964
        pids trace(10673)
$ tools/bpf/bpftool/bpftool link show -j
[{"id":2,"type":"tracing","prog_id":13,"prog_type":"tracing","attach_type":"trace_fentry","target_obj_id":1,"target_btf_id":13964,"pids":[{"pid":10673,"comm":"trace"}]}]
$ cat /proc/10673/fdinfo/10
pos:    0
flags:  02000000
mnt_id: 15
ino:    2094
link_type:      tracing
link_id:        2
prog_tag:       a04f5eef06a7f555
prog_id:        13
attach_type:    24
target_obj_id:  1
target_btf_id:  13964
$ tail /proc/kallsyms
ffffffffc0400fa0 t bpf_prog_a04f5eef06a7f555_fentry_run [bpf]
ffffffffc062f000 t bpf_trampoline_6442464908    [bpf]
$ echo $((6442464908 & 0x7fffffff)) $((6442464908 >> 32))
13964 1
---
 kernel/bpf/syscall.c     | 12 ++++++++++--
 tools/bpf/bpftool/link.c |  4 ++++
 2 files changed, 14 insertions(+), 2 deletions(-)
kernel/bpf/syscall.c +10 -2
index 909c112..870395a 100644
@@ -2968,10 +2968,18 @@ static void bpf_tracing_link_show_fdinfo(const struct bpf_link *link,
 {
 	struct bpf_tracing_link *tr_link =
 		container_of(link, struct bpf_tracing_link, link.link);
+	u32 target_btf_id;
+	u32 target_obj_id;
 
+	bpf_trampoline_unpack_key(tr_link->trampoline->key,
+							  &target_obj_id, &target_btf_id);
 	seq_printf(seq,
-		   "attach_type:\t%d\n",
-		   tr_link->attach_type);
+		   "attach_type:\t%d\n"
+		   "target_obj_id:\t%u\n"
+		   "target_btf_id:\t%u\n",
+		   tr_link->attach_type,
+		   target_obj_id,
+		   target_btf_id);
 }
 
 static int bpf_tracing_link_fill_link_info(const struct bpf_link *link,
tools/bpf/bpftool/link.c +4 -0
index 243b74e..cfe896f 100644
@@ -195,6 +195,8 @@ static int show_link_close_json(int fd, struct bpf_link_info *info)
 
 		show_link_attach_type_json(info->tracing.attach_type,
 					   json_wtr);
+		jsonw_uint_field(json_wtr, "target_obj_id", info->tracing.target_obj_id);
+		jsonw_uint_field(json_wtr, "target_btf_id", info->tracing.target_btf_id);
 		break;
 	case BPF_LINK_TYPE_CGROUP:
 		jsonw_lluint_field(json_wtr, "cgroup_id",
@@ -375,6 +377,8 @@ static int show_link_close_plain(int fd, struct bpf_link_info *info)
 			printf("\n\tprog_type %u  ", prog_info.type);
 
 		show_link_attach_type_plain(info->tracing.attach_type);
+		printf("\n\ttarget_obj_id %u  target_btf_id %u  ",
+			   info->tracing.target_obj_id, info->tracing.target_btf_id);
 		break;
 	case BPF_LINK_TYPE_CGROUP:
 		printf("\n\tcgroup_id %zu  ", (size_t)info->cgroup.cgroup_id);
Signature · 1 line
1.8.3.1
reviewer Acked-by on v2 3/3
#10 lore raw

Re: [PATCH bpf-next v2 3/3] bpf: Show target_{obj,btf}_id in tracing link info

Quoted 1 line · Yafang Shao

[...]

Quoted 2 lines

The change looks good to me, except that we should split the change into two commits.

Also, this doesn't seem to be related to the other two patches. So it is a little weird to add it in v2.

Other than these.

#11 lore raw

Re: [PATCH bpf-next v2 3/3] bpf: Show target_{obj,btf}_id in tracing link info

Quoted 4 lines · Song Liu
Quoted 1 line · Yafang Shao
Quoted 8 lines

[...]

Quoted 2 lines

The change looks good to me, except that we should split the change into two commits.

Will split it and send them. Thanks for your review.

Quoted 6 lines

Also, this doesn't seem to be related to the other two patches. So it is a little weird to add it in v2.

Other than these.

Signature · 2 lines
Regards
Yafang