Skip to contentSkip to conversation

bpf: Add multi uprobe link

PATCH Superseded Jiri Olsa 5 participants 61 messages Jun 20, 2023 – Jun 26, 2023

Subsystems (8)
Kselftest BPF Tracing BPF selftests +4 more subsystems

v2 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Jiri Olsa

author reviewer
#0 lore raw

v2 cover bpf: Add multi uprobe link

hi,
this patchset is adding support to attach multiple uprobes and usdt probes through new uprobe_multi link.

The current uprobe is attached through the perf event and attaching many uprobes takes a lot of time because of that.

The main reason is that we need to install perf event for each probed function and profile shows perf event installation (perf_install_in_context) as culprit.

The new uprobe_multi link just creates raw uprobes and attaches the bpf program to them without perf event being involved.

In addition to being faster we also save file descriptors. For the current uprobe attach we use extra perf event fd for each probed function. The new link just need one fd that covers all the functions we are attaching to.

v2 changes:
- allow to run sleepable programs [Alexei]
- allow user to specify only single path [Andrii]
- pid filter support [Andrii]
- add 'u[ret]probe.multi.s' section defs [Andrii]
- add acked [Andrii]
- remove WARN_ON_ONCE in bpf_uprobe_multi_cookie [Andrii]
- merge tools uapi change with kernel change [Andrii]
- drop ref_ctr_offset from struct bpf_uprobe [Andrii]
- use generic string parsing instead of specific pattern matching in attach_uprobe_multi [Andrii]

followup changes:
- move elf specific code into separate C object [Andrii]
- will send fill_info support when [3] is merged [Andrii]

Also available at:
https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
uprobe_multi

The support for bpftrace [2] and tetragon [1] is updated to v2 API changes.

thanks,
jirka

[1] https://github.com/cilium/tetragon/pull/936
[2] https://github.com/iovisor/bpftrace/compare/master...olsajiri:bpftrace:uprobe_multi
[3] https://lore.kernel.org/bpf/[email protected]/
[4] https://lore.kernel.org/bpf/[email protected]/
---
Jiri Olsa (24):
bpf: Add multi uprobe link
bpf: Add cookies support for uprobe_multi link
bpf: Add pid filter support for uprobe_multi link
bpf: Add bpf_get_func_ip helper support for uprobe link
libbpf: Add uprobe_multi attach type and link names
libbpf: Add elf symbol iterator
libbpf: Add open_elf/close_elf functions
libbpf: Add elf_find_multi_func_offset function
libbpf: Add elf_find_pattern_func_offset function
libbpf: Add bpf_link_create support for multi uprobes
libbpf: Add bpf_program__attach_uprobe_multi_opts function
libbpf: Add support for u[ret]probe.multi[.s] program sections
libbpf: Add uprobe multi link detection
libbpf: Add uprobe multi link support to bpf_program__attach_usdt selftests/bpf: Add uprobe_multi skel test
selftests/bpf: Add uprobe_multi api test
selftests/bpf: Add uprobe_multi link test
selftests/bpf: Add uprobe_multi test program
selftests/bpf: Add uprobe_multi bench test
selftests/bpf: Add usdt_multi test program
selftests/bpf: Add usdt_multi bench test
selftests/bpf: Add uprobe_multi cookie test
selftests/bpf: Add uprobe_multi pid filter tests
selftests/bpf: Add extra link to uprobe_multi tests

 include/linux/trace_events.h                               |   6 ++
 include/uapi/linux/bpf.h                                   |  16 ++++
 kernel/bpf/syscall.c                                       |  14 +++-
 kernel/trace/bpf_trace.c                                   | 349 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 tools/include/uapi/linux/bpf.h                             |  16 ++++
 tools/lib/bpf/bpf.c                                        |  11 +++
 tools/lib/bpf/bpf.h                                        |  11 ++-
 tools/lib/bpf/libbpf.c                                     | 645 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------------------
 tools/lib/bpf/libbpf.h                                     |  31 +++++++
 tools/lib/bpf/libbpf.map                                   |   1 +
 tools/lib/bpf/libbpf_internal.h                            |   7 ++
 tools/lib/bpf/usdt.c                                       | 120 ++++++++++++++++++++-------
 tools/testing/selftests/bpf/Makefile                       |  10 +++
 tools/testing/selftests/bpf/prog_tests/bpf_cookie.c        |  78 ++++++++++++++++++
 tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c | 449 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tools/testing/selftests/bpf/progs/uprobe_multi.c           | 110 +++++++++++++++++++++++++
 tools/testing/selftests/bpf/progs/uprobe_multi_usdt.c      |  16 ++++
 tools/testing/selftests/bpf/uprobe_multi.c                 |  53 ++++++++++++
 tools/testing/selftests/bpf/usdt_multi.c                   |  24 ++++++
 19 files changed, 1841 insertions(+), 126 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c
 create mode 100644 tools/testing/selftests/bpf/progs/uprobe_multi.c
 create mode 100644 tools/testing/selftests/bpf/progs/uprobe_multi_usdt.c
 create mode 100644 tools/testing/selftests/bpf/uprobe_multi.c
 create mode 100644 tools/testing/selftests/bpf/usdt_multi.c
author reviewer
#1 lore raw

PATCH v2 1/24 bpf: Add multi uprobe link

Adding new multi uprobe link that allows to attach bpf program to multiple uprobes.

Uprobes to attach are specified via new link_create uprobe_multi union:

  struct {
          __u32           flags;
          __u32           cnt;
          __aligned_u64   path;
          __aligned_u64   offsets;
          __aligned_u64   ref_ctr_offsets;
  } uprobe_multi;

Uprobes are defined for single binary specified in path and multiple calling sites specified in offsets array with optional reference counters specified in ref_ctr_offsets array. All specified arrays have length of 'cnt'.

The 'flags' supports single bit for now that marks the uprobe as return probe.

---
 include/linux/trace_events.h   |   6 +
 include/uapi/linux/bpf.h       |  14 ++
 kernel/bpf/syscall.c           |  12 +-
 kernel/trace/bpf_trace.c       | 237 +++++++++++++++++++++++++++++++++
 tools/include/uapi/linux/bpf.h |  14 ++
 5 files changed, 281 insertions(+), 2 deletions(-)
include/linux/trace_events.h +6 -0
index 0e373222a6df..b0db245fc0f5 100644
@@ -749,6 +749,7 @@ int bpf_get_perf_event_info(const struct perf_event *event, u32 *prog_id,
 			    u32 *fd_type, const char **buf,
 			    u64 *probe_offset, u64 *probe_addr);
 int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
+int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
 #else
 static inline unsigned int trace_call_bpf(struct trace_event_call *call, void *ctx)
 {
@@ -795,6 +796,11 @@ bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
 {
 	return -EOPNOTSUPP;
 }
+static inline int
+bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
+{
+	return -EOPNOTSUPP;
+}
 #endif
 
 enum {
include/uapi/linux/bpf.h +14 -0
index a7b5e91dd768..bfbc1246b220 100644
@@ -1035,6 +1035,7 @@ enum bpf_attach_type {
 	BPF_TRACE_KPROBE_MULTI,
 	BPF_LSM_CGROUP,
 	BPF_STRUCT_OPS,
+	BPF_TRACE_UPROBE_MULTI,
 	__MAX_BPF_ATTACH_TYPE
 };
 
@@ -1052,6 +1053,7 @@ enum bpf_link_type {
 	BPF_LINK_TYPE_KPROBE_MULTI = 8,
 	BPF_LINK_TYPE_STRUCT_OPS = 9,
 	BPF_LINK_TYPE_NETFILTER = 10,
+	BPF_LINK_TYPE_UPROBE_MULTI = 11,
 
 	MAX_BPF_LINK_TYPE,
 };
@@ -1169,6 +1171,11 @@ enum bpf_link_type {
  */
 #define BPF_F_KPROBE_MULTI_RETURN	(1U << 0)
 
+/* link_create.uprobe_multi.flags used in LINK_CREATE command for
+ * BPF_TRACE_UPROBE_MULTI attach type to create return probe.
+ */
+#define BPF_F_UPROBE_MULTI_RETURN	(1U << 0)
+
 /* When BPF ldimm64's insn[0].src_reg != 0 then this can have
  * the following extensions:
  *
@@ -1578,6 +1585,13 @@ union bpf_attr {
 				__s32		priority;
 				__u32		flags;
 			} netfilter;
+			struct {
+				__u32		flags;
+				__u32		cnt;
+				__aligned_u64	path;
+				__aligned_u64	offsets;
+				__aligned_u64	ref_ctr_offsets;
+			} uprobe_multi;
 		};
 	} link_create;
 
kernel/bpf/syscall.c +10 -2
index a75c54b6f8a3..a96e46cd407e 100644
@@ -3516,6 +3516,11 @@ static int bpf_prog_attach_check_attach_type(const struct bpf_prog *prog,
 		return prog->enforce_expected_attach_type &&
 			prog->expected_attach_type != attach_type ?
 			-EINVAL : 0;
+	case BPF_PROG_TYPE_KPROBE:
+		if (prog->expected_attach_type == BPF_TRACE_KPROBE_MULTI &&
+		    attach_type != BPF_TRACE_KPROBE_MULTI)
+			return -EINVAL;
+		fallthrough;
 	default:
 		return 0;
 	}
@@ -4681,7 +4686,8 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr)
 		break;
 	case BPF_PROG_TYPE_KPROBE:
 		if (attr->link_create.attach_type != BPF_PERF_EVENT &&
-		    attr->link_create.attach_type != BPF_TRACE_KPROBE_MULTI) {
+		    attr->link_create.attach_type != BPF_TRACE_KPROBE_MULTI &&
+		    attr->link_create.attach_type != BPF_TRACE_UPROBE_MULTI) {
 			ret = -EINVAL;
 			goto out;
 		}
@@ -4748,8 +4754,10 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr)
 	case BPF_PROG_TYPE_KPROBE:
 		if (attr->link_create.attach_type == BPF_PERF_EVENT)
 			ret = bpf_perf_link_attach(attr, prog);
-		else
+		else if (attr->link_create.attach_type == BPF_TRACE_KPROBE_MULTI)
 			ret = bpf_kprobe_multi_link_attach(attr, prog);
+		else if (attr->link_create.attach_type == BPF_TRACE_UPROBE_MULTI)
+			ret = bpf_uprobe_multi_link_attach(attr, prog);
 		break;
 	default:
 		ret = -EINVAL;
kernel/trace/bpf_trace.c +237 -0

246 lines not on this page · raw message

tools/include/uapi/linux/bpf.h +14 -0
index a7b5e91dd768..bfbc1246b220 100644
@@ -1035,6 +1035,7 @@ enum bpf_attach_type {
 	BPF_TRACE_KPROBE_MULTI,
 	BPF_LSM_CGROUP,
 	BPF_STRUCT_OPS,
+	BPF_TRACE_UPROBE_MULTI,
 	__MAX_BPF_ATTACH_TYPE
 };
 
@@ -1052,6 +1053,7 @@ enum bpf_link_type {
 	BPF_LINK_TYPE_KPROBE_MULTI = 8,
 	BPF_LINK_TYPE_STRUCT_OPS = 9,
 	BPF_LINK_TYPE_NETFILTER = 10,
+	BPF_LINK_TYPE_UPROBE_MULTI = 11,
 
 	MAX_BPF_LINK_TYPE,
 };
@@ -1169,6 +1171,11 @@ enum bpf_link_type {
  */
 #define BPF_F_KPROBE_MULTI_RETURN	(1U << 0)
 
+/* link_create.uprobe_multi.flags used in LINK_CREATE command for
+ * BPF_TRACE_UPROBE_MULTI attach type to create return probe.
+ */
+#define BPF_F_UPROBE_MULTI_RETURN	(1U << 0)
+
 /* When BPF ldimm64's insn[0].src_reg != 0 then this can have
  * the following extensions:
  *
@@ -1578,6 +1585,13 @@ union bpf_attr {
 				__s32		priority;
 				__u32		flags;
 			} netfilter;
+			struct {
+				__u32		flags;
+				__u32		cnt;
+				__aligned_u64	path;
+				__aligned_u64	offsets;
+				__aligned_u64	ref_ctr_offsets;
+			} uprobe_multi;
 		};
 	} link_create;
 
Signature · 1 line
2.41.0
#2 lore raw
Quoted 9 lines · Jiri Olsa
kernel/trace/bpf_trace.c +9 -0
+static int uprobe_prog_run(struct bpf_uprobe *uprobe,
+			   unsigned long entry_ip,
+			   struct pt_regs *regs)
+{
+	struct bpf_uprobe_multi_link *link = uprobe->link;
+	struct bpf_uprobe_multi_run_ctx run_ctx = {
+		.entry_ip = entry_ip,
+	};
+	struct bpf_prog *prog = link->link.prog;

kernel/trace/bpf_trace.c static u64 bpf_kprobe_multi_entry_ip(struct bpf_run_ctx *ctx) · line 2990

Quoted 10 lines
kernel/trace/bpf_trace.c +10 -0
+	struct bpf_run_ctx *old_run_ctx;
+	int err = 0;
+
+	might_fault();
+
+	rcu_read_lock_trace();
+	migrate_disable();
+
+	if (unlikely(__this_cpu_inc_return(bpf_prog_active) != 1))
+		goto out;

bpf_prog_run_array_sleepable() doesn't do such things. Such 'proteciton' will actively hurt.
The sleepable prog below will block all kprobes on this cpu. please remove.

Quoted 18 lines
kernel/trace/bpf_trace.c +18 -0
+
+	old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
+
+	if (!prog->aux->sleepable)
+		rcu_read_lock();
+
+	err = bpf_prog_run(link->link.prog, regs);
+
+	if (!prog->aux->sleepable)
+		rcu_read_unlock();
+
+	bpf_reset_run_ctx(old_run_ctx);
+
+out:
+	__this_cpu_dec(bpf_prog_active);
+	migrate_enable();
+	rcu_read_unlock_trace();
+	return err;
#3 lore raw
Quoted 17 lines · Alexei Starovoitov
Quoted 16 lines · Jiri Olsa
kernel/trace/bpf_trace.c +16 -0
+static int uprobe_prog_run(struct bpf_uprobe *uprobe,
+			   unsigned long entry_ip,
+			   struct pt_regs *regs)
+{
+	struct bpf_uprobe_multi_link *link = uprobe->link;
+	struct bpf_uprobe_multi_run_ctx run_ctx = {
+		.entry_ip = entry_ip,
+	};
+	struct bpf_prog *prog = link->link.prog;
+	struct bpf_run_ctx *old_run_ctx;
+	int err = 0;
+
+	might_fault();
+
+	rcu_read_lock_trace();
+	migrate_disable();
Quoted 8 lines

kernel/trace/bpf_trace.c static u64 bpf_kprobe_multi_entry_ip(struct bpf_run_ctx *ctx) · line 2990

Quoted 3 lines
kernel/trace/bpf_trace.c +3 -0
+
+	if (unlikely(__this_cpu_inc_return(bpf_prog_active) != 1))
+		goto out;

bpf_prog_run_array_sleepable() doesn't do such things. Such 'proteciton' will actively hurt.
The sleepable prog below will block all kprobes on this cpu. please remove.

ok makes sense, can't recall the reason why I added it

jirka

Quoted 19 lines
Quoted 19 lines
kernel/trace/bpf_trace.c +18 -0
+
+	old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
+
+	if (!prog->aux->sleepable)
+		rcu_read_lock();
+
+	err = bpf_prog_run(link->link.prog, regs);
+
+	if (!prog->aux->sleepable)
+		rcu_read_unlock();
+
+	bpf_reset_run_ctx(old_run_ctx);
+
+out:
+	__this_cpu_dec(bpf_prog_active);
+	migrate_enable();
+	rcu_read_unlock_trace();
+	return err;
#4 lore raw
In reply to Jiri Olsa
Quoted 32 lines · Jiri Olsa

Adding new multi uprobe link that allows to attach bpf program to multiple uprobes.

Uprobes to attach are specified via new link_create uprobe_multi union:

  struct {
          __u32           flags;
          __u32           cnt;
          __aligned_u64   path;
          __aligned_u64   offsets;
          __aligned_u64   ref_ctr_offsets;
  } uprobe_multi;

Uprobes are defined for single binary specified in path and multiple calling sites specified in offsets array with optional reference counters specified in ref_ctr_offsets array. All specified arrays have length of 'cnt'.

The 'flags' supports single bit for now that marks the uprobe as return probe.

---
 include/linux/trace_events.h   |   6 +
 include/uapi/linux/bpf.h       |  14 ++
 kernel/bpf/syscall.c           |  12 +-
 kernel/trace/bpf_trace.c       | 237 +++++++++++++++++++++++++++++++++
 tools/include/uapi/linux/bpf.h |  14 ++
 5 files changed, 281 insertions(+), 2 deletions(-)

[...]

kernel/bpf/syscall.c static int bpf_prog_attach_check_attach_type(const struct bpf_prog *prog, · line 3521

Quoted 11 lines
kernel/bpf/syscall.c +3 -0
index a75c54b6f8a3..a96e46cd407e 100644
@@ -3516,6 +3516,11 @@ static int bpf_prog_attach_check_attach_type(const struct bpf_prog *prog,
                return prog->enforce_expected_attach_type &&
                        prog->expected_attach_type != attach_type ?
                        -EINVAL : 0;
+       case BPF_PROG_TYPE_KPROBE:
+               if (prog->expected_attach_type == BPF_TRACE_KPROBE_MULTI &&
+                   attach_type != BPF_TRACE_KPROBE_MULTI)

should this be UPROBE_MULTI? this looks like your recent bug fix, which already landed

kernel/bpf/syscall.c static int bpf_prog_attach_check_attach_type(const struct bpf_prog *prog, · line 3523

Quoted 2 lines
kernel/bpf/syscall.c +2 -0
+                       return -EINVAL;
+               fallthrough;

and I replaced this with return 0; ;)

kernel/bpf/syscall.c static int link_create(union bpf_attr *attr, bpfptr_t uattr) · line 4693

Quoted 13 lines
kernel/bpf/syscall.c +2 -1
        default:
                return 0;
        }
@@ -4681,7 +4686,8 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr)
                break;
        case BPF_PROG_TYPE_KPROBE:
                if (attr->link_create.attach_type != BPF_PERF_EVENT &&
-                   attr->link_create.attach_type != BPF_TRACE_KPROBE_MULTI) {
+                   attr->link_create.attach_type != BPF_TRACE_KPROBE_MULTI &&
+                   attr->link_create.attach_type != BPF_TRACE_UPROBE_MULTI) {
                        ret = -EINVAL;
                        goto out;
                }

should this be moved into bpf_prog_attach_check_attach_type() and unify these checks?

Quoted 12 lines
kernel/bpf/syscall.c +3 -1
@@ -4748,8 +4754,10 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr)
        case BPF_PROG_TYPE_KPROBE:
                if (attr->link_create.attach_type == BPF_PERF_EVENT)
                        ret = bpf_perf_link_attach(attr, prog);
-               else
+               else if (attr->link_create.attach_type == BPF_TRACE_KPROBE_MULTI)
                        ret = bpf_kprobe_multi_link_attach(attr, prog);
+               else if (attr->link_create.attach_type == BPF_TRACE_UPROBE_MULTI)
+                       ret = bpf_uprobe_multi_link_attach(attr, prog);
                break;
        default:
                ret = -EINVAL;

[...]

Quoted 39 lines
kernel/bpf/syscall.c +39 -0
+static void bpf_uprobe_unregister(struct path *path, struct bpf_uprobe *uprobes,
+                                 u32 cnt)
+{
+       u32 i;
+
+       for (i = 0; i < cnt; i++) {
+               uprobe_unregister(d_real_inode(path->dentry), uprobes[i].offset,
+                                 &uprobes[i].consumer);
+       }
+}
+
+static void bpf_uprobe_multi_link_release(struct bpf_link *link)
+{
+       struct bpf_uprobe_multi_link *umulti_link;
+
+       umulti_link = container_of(link, struct bpf_uprobe_multi_link, link);
+       bpf_uprobe_unregister(&umulti_link->path, umulti_link->uprobes, umulti_link->cnt);
+       path_put(&umulti_link->path);
+}
+
+static void bpf_uprobe_multi_link_dealloc(struct bpf_link *link)
+{
+       struct bpf_uprobe_multi_link *umulti_link;
+
+       umulti_link = container_of(link, struct bpf_uprobe_multi_link, link);
+       kvfree(umulti_link->uprobes);
+       kfree(umulti_link);
+}
+
+static const struct bpf_link_ops bpf_uprobe_multi_link_lops = {
+       .release = bpf_uprobe_multi_link_release,
+       .dealloc = bpf_uprobe_multi_link_dealloc,
+};
+
+static int uprobe_prog_run(struct bpf_uprobe *uprobe,
+                          unsigned long entry_ip,
+                          struct pt_regs *regs)
+{
+       struct bpf_uprobe_multi_link *link = uprobe->link;

kernel/bpf/syscall.c

Quoted 10 lines
kernel/bpf/syscall.c +10 -0
+       struct bpf_uprobe_multi_run_ctx run_ctx = {
+               .entry_ip = entry_ip,
+       };
+       struct bpf_prog *prog = link->link.prog;
+       struct bpf_run_ctx *old_run_ctx;
+       int err = 0;
+
+       might_fault();
+
+       rcu_read_lock_trace();

we don't need this if uprobe is not sleepable, right? why unconditional then?

Quoted 24 lines
kernel/bpf/syscall.c +24 -0
+       migrate_disable();
+
+       if (unlikely(__this_cpu_inc_return(bpf_prog_active) != 1))
+               goto out;
+
+       old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
+
+       if (!prog->aux->sleepable)
+               rcu_read_lock();
+
+       err = bpf_prog_run(link->link.prog, regs);
+
+       if (!prog->aux->sleepable)
+               rcu_read_unlock();
+
+       bpf_reset_run_ctx(old_run_ctx);
+
+out:
+       __this_cpu_dec(bpf_prog_active);
+       migrate_enable();
+       rcu_read_unlock_trace();
+       return err;
+}
+

[...]

Quoted 6 lines
kernel/bpf/syscall.c +6 -0
+
+       err = kern_path(name, LOOKUP_FOLLOW, &path);
+       kfree(name);
+       if (err)
+               return err;
+

kernel/bpf/syscall.c

Quoted 10 lines
kernel/bpf/syscall.c +10 -0
+       if (!d_is_reg(path.dentry)) {
+               err = -EINVAL;
+               goto error_path_put;
+       }
+
+       err = -ENOMEM;
+
+       link = kzalloc(sizeof(*link), GFP_KERNEL);
+       uprobes = kvcalloc(cnt, sizeof(*uprobes), GFP_KERNEL);
+       ref_ctr_offsets = kvcalloc(cnt, sizeof(*ref_ctr_offsets), GFP_KERNEL);

ref_ctr_offsets is optional, but we'll unconditionally allocate this array?

Quoted 32 lines
kernel/bpf/syscall.c +32 -0
+
+       if (!uprobes || !ref_ctr_offsets || !link)
+               goto error_free;
+
+       for (i = 0; i < cnt; i++) {
+               if (uref_ctr_offsets && __get_user(ref_ctr_offset, uref_ctr_offsets + i)) {
+                       err = -EFAULT;
+                       goto error_free;
+               }
+               if (__get_user(offset, uoffsets + i)) {
+                       err = -EFAULT;
+                       goto error_free;
+               }
+
+               uprobes[i].offset = offset;
+               uprobes[i].link = link;
+
+               if (flags & BPF_F_UPROBE_MULTI_RETURN)
+                       uprobes[i].consumer.ret_handler = uprobe_multi_link_ret_handler;
+               else
+                       uprobes[i].consumer.handler = uprobe_multi_link_handler;
+
+               ref_ctr_offsets[i] = ref_ctr_offset;
+       }
+
+       link->cnt = cnt;
+       link->uprobes = uprobes;
+       link->path = path;
+
+       bpf_link_init(&link->link, BPF_LINK_TYPE_UPROBE_MULTI,
+                     &bpf_uprobe_multi_link_lops, prog);
+

kernel/bpf/syscall.c

Quoted 10 lines
kernel/bpf/syscall.c +10 -0
+       err = bpf_link_prime(&link->link, &link_primer);
+       if (err)
+               goto error_free;
+
+       for (i = 0; i < cnt; i++) {
+               err = uprobe_register_refctr(d_real_inode(link->path.dentry),
+                                            uprobes[i].offset, ref_ctr_offsets[i],
+                                            &uprobes[i].consumer);
+               if (err) {
+                       bpf_uprobe_unregister(&path, uprobes, i);

bpf_link_cleanup() will do this through bpf_uprobe_multi_link_release(), no? So you are double unregistering? Either drop cnt to zero, or just don't do this here? Latter is better, IMO.

Quoted 22 lines
kernel/bpf/syscall.c +22 -0
+                       bpf_link_cleanup(&link_primer);
+                       kvfree(ref_ctr_offsets);
+                       return err;
+               }
+       }
+
+       kvfree(ref_ctr_offsets);
+       return bpf_link_settle(&link_primer);
+
+error_free:
+       kvfree(ref_ctr_offsets);
+       kvfree(uprobes);
+       kfree(link);
+error_path_put:
+       path_put(&path);
+       return err;
+}
+#else /* !CONFIG_UPROBES */
+int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
+{
+       return -EOPNOTSUPP;
+}

[...]

#5 lore raw
Quoted 51 lines · Andrii Nakryiko
Quoted 32 lines · Jiri Olsa

Adding new multi uprobe link that allows to attach bpf program to multiple uprobes.

Uprobes to attach are specified via new link_create uprobe_multi union:

  struct {
          __u32           flags;
          __u32           cnt;
          __aligned_u64   path;
          __aligned_u64   offsets;
          __aligned_u64   ref_ctr_offsets;
  } uprobe_multi;

Uprobes are defined for single binary specified in path and multiple calling sites specified in offsets array with optional reference counters specified in ref_ctr_offsets array. All specified arrays have length of 'cnt'.

The 'flags' supports single bit for now that marks the uprobe as return probe.

---
 include/linux/trace_events.h   |   6 +
 include/uapi/linux/bpf.h       |  14 ++
 kernel/bpf/syscall.c           |  12 +-
 kernel/trace/bpf_trace.c       | 237 +++++++++++++++++++++++++++++++++
 tools/include/uapi/linux/bpf.h |  14 ++
 5 files changed, 281 insertions(+), 2 deletions(-)

[...]

kernel/bpf/syscall.c static int bpf_prog_attach_check_attach_type(const struct bpf_prog *prog, · line 3521

Quoted 11 lines
kernel/bpf/syscall.c +3 -0
index a75c54b6f8a3..a96e46cd407e 100644
@@ -3516,6 +3516,11 @@ static int bpf_prog_attach_check_attach_type(const struct bpf_prog *prog,
                return prog->enforce_expected_attach_type &&
                        prog->expected_attach_type != attach_type ?
                        -EINVAL : 0;
+       case BPF_PROG_TYPE_KPROBE:
+               if (prog->expected_attach_type == BPF_TRACE_KPROBE_MULTI &&
+                   attach_type != BPF_TRACE_KPROBE_MULTI)

should this be UPROBE_MULTI? this looks like your recent bug fix, which already landed

Quoted 4 lines

kernel/bpf/syscall.c static int bpf_prog_attach_check_attach_type(const struct bpf_prog *prog, · line 3523

Quoted 2 lines
kernel/bpf/syscall.c +2 -0
+                       return -EINVAL;
+               fallthrough;

and I replaced this with return 0; ;)

ugh, yes, will fix

Quoted 8 lines
Quoted 8 lines
kernel/bpf/syscall.c +0 -1
        default:
                return 0;
        }
@@ -4681,7 +4686,8 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr)
                break;
        case BPF_PROG_TYPE_KPROBE:
                if (attr->link_create.attach_type != BPF_PERF_EVENT &&
-                   attr->link_create.attach_type != BPF_TRACE_KPROBE_MULTI) {
Quoted 8 lines

kernel/bpf/syscall.c static int link_create(union bpf_attr *attr, bpfptr_t uattr) · line 4693

Quoted 5 lines
kernel/bpf/syscall.c +2 -0
+                   attr->link_create.attach_type != BPF_TRACE_KPROBE_MULTI &&
+                   attr->link_create.attach_type != BPF_TRACE_UPROBE_MULTI) {
                        ret = -EINVAL;
                        goto out;
                }

should this be moved into bpf_prog_attach_check_attach_type() and unify these checks?

ok, perhaps we could move there the whole switch, will check

Quoted 59 lines
Quoted 13 lines
kernel/bpf/syscall.c +3 -1
@@ -4748,8 +4754,10 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr)
        case BPF_PROG_TYPE_KPROBE:
                if (attr->link_create.attach_type == BPF_PERF_EVENT)
                        ret = bpf_perf_link_attach(attr, prog);
-               else
+               else if (attr->link_create.attach_type == BPF_TRACE_KPROBE_MULTI)
                        ret = bpf_kprobe_multi_link_attach(attr, prog);
+               else if (attr->link_create.attach_type == BPF_TRACE_UPROBE_MULTI)
+                       ret = bpf_uprobe_multi_link_attach(attr, prog);
                break;
        default:
                ret = -EINVAL;

[...]

Quoted 43 lines
kernel/bpf/syscall.c +43 -0
+static void bpf_uprobe_unregister(struct path *path, struct bpf_uprobe *uprobes,
+                                 u32 cnt)
+{
+       u32 i;
+
+       for (i = 0; i < cnt; i++) {
+               uprobe_unregister(d_real_inode(path->dentry), uprobes[i].offset,
+                                 &uprobes[i].consumer);
+       }
+}
+
+static void bpf_uprobe_multi_link_release(struct bpf_link *link)
+{
+       struct bpf_uprobe_multi_link *umulti_link;
+
+       umulti_link = container_of(link, struct bpf_uprobe_multi_link, link);
+       bpf_uprobe_unregister(&umulti_link->path, umulti_link->uprobes, umulti_link->cnt);
+       path_put(&umulti_link->path);
+}
+
+static void bpf_uprobe_multi_link_dealloc(struct bpf_link *link)
+{
+       struct bpf_uprobe_multi_link *umulti_link;
+
+       umulti_link = container_of(link, struct bpf_uprobe_multi_link, link);
+       kvfree(umulti_link->uprobes);
+       kfree(umulti_link);
+}
+
+static const struct bpf_link_ops bpf_uprobe_multi_link_lops = {
+       .release = bpf_uprobe_multi_link_release,
+       .dealloc = bpf_uprobe_multi_link_dealloc,
+};
+
+static int uprobe_prog_run(struct bpf_uprobe *uprobe,
+                          unsigned long entry_ip,
+                          struct pt_regs *regs)
+{
+       struct bpf_uprobe_multi_link *link = uprobe->link;
+       struct bpf_uprobe_multi_run_ctx run_ctx = {
+               .entry_ip = entry_ip,
+       };
+       struct bpf_prog *prog = link->link.prog;
Quoted 8 lines

kernel/bpf/syscall.c

Quoted 6 lines
kernel/bpf/syscall.c +6 -0
+       struct bpf_run_ctx *old_run_ctx;
+       int err = 0;
+
+       might_fault();
+
+       rcu_read_lock_trace();

we don't need this if uprobe is not sleepable, right? why unconditional then?

I won't pretend I understand what rcu_read_lock_trace does ;-)

I tried to follow bpf_prog_run_array_sleepable where it's called unconditionally for both sleepable and non-sleepable progs

there are conditional rcu_read_un/lock calls later on

I will check

Quoted 38 lines
Quoted 25 lines
kernel/bpf/syscall.c +24 -0
+       migrate_disable();
+
+       if (unlikely(__this_cpu_inc_return(bpf_prog_active) != 1))
+               goto out;
+
+       old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
+
+       if (!prog->aux->sleepable)
+               rcu_read_lock();
+
+       err = bpf_prog_run(link->link.prog, regs);
+
+       if (!prog->aux->sleepable)
+               rcu_read_unlock();
+
+       bpf_reset_run_ctx(old_run_ctx);
+
+out:
+       __this_cpu_dec(bpf_prog_active);
+       migrate_enable();
+       rcu_read_unlock_trace();
+       return err;
+}
+

[...]

Quoted 10 lines
kernel/bpf/syscall.c +10 -0
+
+       err = kern_path(name, LOOKUP_FOLLOW, &path);
+       kfree(name);
+       if (err)
+               return err;
+
+       if (!d_is_reg(path.dentry)) {
+               err = -EINVAL;
+               goto error_path_put;
+       }
Quoted 8 lines

kernel/bpf/syscall.c

Quoted 6 lines
kernel/bpf/syscall.c +6 -0
+
+       err = -ENOMEM;
+
+       link = kzalloc(sizeof(*link), GFP_KERNEL);
+       uprobes = kvcalloc(cnt, sizeof(*uprobes), GFP_KERNEL);
+       ref_ctr_offsets = kvcalloc(cnt, sizeof(*ref_ctr_offsets), GFP_KERNEL);

ref_ctr_offsets is optional, but we'll unconditionally allocate this array?

true :-\ will add the uref_ctr_offsets check

Quoted 40 lines
Quoted 40 lines
kernel/bpf/syscall.c +39 -0
+
+       if (!uprobes || !ref_ctr_offsets || !link)
+               goto error_free;
+
+       for (i = 0; i < cnt; i++) {
+               if (uref_ctr_offsets && __get_user(ref_ctr_offset, uref_ctr_offsets + i)) {
+                       err = -EFAULT;
+                       goto error_free;
+               }
+               if (__get_user(offset, uoffsets + i)) {
+                       err = -EFAULT;
+                       goto error_free;
+               }
+
+               uprobes[i].offset = offset;
+               uprobes[i].link = link;
+
+               if (flags & BPF_F_UPROBE_MULTI_RETURN)
+                       uprobes[i].consumer.ret_handler = uprobe_multi_link_ret_handler;
+               else
+                       uprobes[i].consumer.handler = uprobe_multi_link_handler;
+
+               ref_ctr_offsets[i] = ref_ctr_offset;
+       }
+
+       link->cnt = cnt;
+       link->uprobes = uprobes;
+       link->path = path;
+
+       bpf_link_init(&link->link, BPF_LINK_TYPE_UPROBE_MULTI,
+                     &bpf_uprobe_multi_link_lops, prog);
+
+       err = bpf_link_prime(&link->link, &link_primer);
+       if (err)
+               goto error_free;
+
+       for (i = 0; i < cnt; i++) {
+               err = uprobe_register_refctr(d_real_inode(link->path.dentry),
+                                            uprobes[i].offset, ref_ctr_offsets[i],
Quoted 8 lines

kernel/bpf/syscall.c

Quoted 3 lines
kernel/bpf/syscall.c +3 -0
+                                            &uprobes[i].consumer);
+               if (err) {
+                       bpf_uprobe_unregister(&path, uprobes, i);

bpf_link_cleanup() will do this through bpf_uprobe_multi_link_release(), no? So you are double unregistering? Either drop cnt to zero, or just don't do this here? Latter is better, IMO.

bpf_link_cleanup path won't call release callback so we have to do that

I think I can add simple selftest to have this path covered

thanks,
jirka

Quoted 25 lines
Quoted 23 lines
kernel/bpf/syscall.c +22 -0
+                       bpf_link_cleanup(&link_primer);
+                       kvfree(ref_ctr_offsets);
+                       return err;
+               }
+       }
+
+       kvfree(ref_ctr_offsets);
+       return bpf_link_settle(&link_primer);
+
+error_free:
+       kvfree(ref_ctr_offsets);
+       kvfree(uprobes);
+       kfree(link);
+error_path_put:
+       path_put(&path);
+       return err;
+}
+#else /* !CONFIG_UPROBES */
+int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
+{
+       return -EOPNOTSUPP;
+}

[...]

#6 lore raw
Quoted 77 lines · Jiri Olsa
Quoted 51 lines · Andrii Nakryiko
Quoted 32 lines · Jiri Olsa

Adding new multi uprobe link that allows to attach bpf program to multiple uprobes.

Uprobes to attach are specified via new link_create uprobe_multi union:

  struct {
          __u32           flags;
          __u32           cnt;
          __aligned_u64   path;
          __aligned_u64   offsets;
          __aligned_u64   ref_ctr_offsets;
  } uprobe_multi;

Uprobes are defined for single binary specified in path and multiple calling sites specified in offsets array with optional reference counters specified in ref_ctr_offsets array. All specified arrays have length of 'cnt'.

The 'flags' supports single bit for now that marks the uprobe as return probe.

---
 include/linux/trace_events.h   |   6 +
 include/uapi/linux/bpf.h       |  14 ++
 kernel/bpf/syscall.c           |  12 +-
 kernel/trace/bpf_trace.c       | 237 +++++++++++++++++++++++++++++++++
 tools/include/uapi/linux/bpf.h |  14 ++
 5 files changed, 281 insertions(+), 2 deletions(-)

[...]

kernel/bpf/syscall.c static int bpf_prog_attach_check_attach_type(const struct bpf_prog *prog, · line 3521

Quoted 11 lines
kernel/bpf/syscall.c +3 -0
index a75c54b6f8a3..a96e46cd407e 100644
@@ -3516,6 +3516,11 @@ static int bpf_prog_attach_check_attach_type(const struct bpf_prog *prog,
                return prog->enforce_expected_attach_type &&
                        prog->expected_attach_type != attach_type ?
                        -EINVAL : 0;
+       case BPF_PROG_TYPE_KPROBE:
+               if (prog->expected_attach_type == BPF_TRACE_KPROBE_MULTI &&
+                   attach_type != BPF_TRACE_KPROBE_MULTI)

should this be UPROBE_MULTI? this looks like your recent bug fix, which already landed

Quoted 4 lines

kernel/bpf/syscall.c static int bpf_prog_attach_check_attach_type(const struct bpf_prog *prog, · line 3523

Quoted 2 lines
kernel/bpf/syscall.c +2 -0
+                       return -EINVAL;
+               fallthrough;

and I replaced this with return 0; ;)

ugh, yes, will fix

Quoted 8 lines
Quoted 8 lines
kernel/bpf/syscall.c +0 -1
        default:
                return 0;
        }
@@ -4681,7 +4686,8 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr)
                break;
        case BPF_PROG_TYPE_KPROBE:
                if (attr->link_create.attach_type != BPF_PERF_EVENT &&
-                   attr->link_create.attach_type != BPF_TRACE_KPROBE_MULTI) {
Quoted 8 lines

kernel/bpf/syscall.c static int link_create(union bpf_attr *attr, bpfptr_t uattr) · line 4693

Quoted 5 lines
kernel/bpf/syscall.c +2 -0
+                   attr->link_create.attach_type != BPF_TRACE_KPROBE_MULTI &&
+                   attr->link_create.attach_type != BPF_TRACE_UPROBE_MULTI) {
                        ret = -EINVAL;
                        goto out;
                }

should this be moved into bpf_prog_attach_check_attach_type() and unify these checks?

Quoted 1 line

ok, perhaps we could move there the whole switch, will check

+1

Quoted 69 lines
Quoted 60 lines
Quoted 14 lines
kernel/bpf/syscall.c +3 -1
@@ -4748,8 +4754,10 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr)
        case BPF_PROG_TYPE_KPROBE:
                if (attr->link_create.attach_type == BPF_PERF_EVENT)
                        ret = bpf_perf_link_attach(attr, prog);
-               else
+               else if (attr->link_create.attach_type == BPF_TRACE_KPROBE_MULTI)
                        ret = bpf_kprobe_multi_link_attach(attr, prog);
+               else if (attr->link_create.attach_type == BPF_TRACE_UPROBE_MULTI)
+                       ret = bpf_uprobe_multi_link_attach(attr, prog);
                break;
        default:
                ret = -EINVAL;

[...]

Quoted 43 lines
kernel/bpf/syscall.c +43 -0
+static void bpf_uprobe_unregister(struct path *path, struct bpf_uprobe *uprobes,
+                                 u32 cnt)
+{
+       u32 i;
+
+       for (i = 0; i < cnt; i++) {
+               uprobe_unregister(d_real_inode(path->dentry), uprobes[i].offset,
+                                 &uprobes[i].consumer);
+       }
+}
+
+static void bpf_uprobe_multi_link_release(struct bpf_link *link)
+{
+       struct bpf_uprobe_multi_link *umulti_link;
+
+       umulti_link = container_of(link, struct bpf_uprobe_multi_link, link);
+       bpf_uprobe_unregister(&umulti_link->path, umulti_link->uprobes, umulti_link->cnt);
+       path_put(&umulti_link->path);
+}
+
+static void bpf_uprobe_multi_link_dealloc(struct bpf_link *link)
+{
+       struct bpf_uprobe_multi_link *umulti_link;
+
+       umulti_link = container_of(link, struct bpf_uprobe_multi_link, link);
+       kvfree(umulti_link->uprobes);
+       kfree(umulti_link);
+}
+
+static const struct bpf_link_ops bpf_uprobe_multi_link_lops = {
+       .release = bpf_uprobe_multi_link_release,
+       .dealloc = bpf_uprobe_multi_link_dealloc,
+};
+
+static int uprobe_prog_run(struct bpf_uprobe *uprobe,
+                          unsigned long entry_ip,
+                          struct pt_regs *regs)
+{
+       struct bpf_uprobe_multi_link *link = uprobe->link;
+       struct bpf_uprobe_multi_run_ctx run_ctx = {
+               .entry_ip = entry_ip,
+       };
+       struct bpf_prog *prog = link->link.prog;
Quoted 8 lines

kernel/bpf/syscall.c

Quoted 6 lines
kernel/bpf/syscall.c +6 -0
+       struct bpf_run_ctx *old_run_ctx;
+       int err = 0;
+
+       might_fault();
+
+       rcu_read_lock_trace();

we don't need this if uprobe is not sleepable, right? why unconditional then?

Quoted 8 lines

I won't pretend I understand what rcu_read_lock_trace does ;-)

I tried to follow bpf_prog_run_array_sleepable where it's called unconditionally for both sleepable and non-sleepable progs

there are conditional rcu_read_un/lock calls later on

I will check

hm... Alexei can chime in here, but given here we actually are trying to run one BPF program (not entire array of them), we do know whether it's going to be sleepable or not. So we can avoid unnecessary rcu_read_{lock,unlock}_trace() calls. rcu_read_lock_trace() is used when there is going to be sleepable BPF program executed to protect BPF maps and other resources from being freed too soon. But if we know that we don't need sleepable, we can avoid that.

Quoted 99 lines
Quoted 39 lines
Quoted 26 lines
kernel/bpf/syscall.c +24 -0
+       migrate_disable();
+
+       if (unlikely(__this_cpu_inc_return(bpf_prog_active) != 1))
+               goto out;
+
+       old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
+
+       if (!prog->aux->sleepable)
+               rcu_read_lock();
+
+       err = bpf_prog_run(link->link.prog, regs);
+
+       if (!prog->aux->sleepable)
+               rcu_read_unlock();
+
+       bpf_reset_run_ctx(old_run_ctx);
+
+out:
+       __this_cpu_dec(bpf_prog_active);
+       migrate_enable();
+       rcu_read_unlock_trace();
+       return err;
+}
+

[...]

Quoted 10 lines
kernel/bpf/syscall.c +10 -0
+
+       err = kern_path(name, LOOKUP_FOLLOW, &path);
+       kfree(name);
+       if (err)
+               return err;
+
+       if (!d_is_reg(path.dentry)) {
+               err = -EINVAL;
+               goto error_path_put;
+       }
Quoted 8 lines

kernel/bpf/syscall.c

Quoted 6 lines
kernel/bpf/syscall.c +6 -0
+
+       err = -ENOMEM;
+
+       link = kzalloc(sizeof(*link), GFP_KERNEL);
+       uprobes = kvcalloc(cnt, sizeof(*uprobes), GFP_KERNEL);
+       ref_ctr_offsets = kvcalloc(cnt, sizeof(*ref_ctr_offsets), GFP_KERNEL);

ref_ctr_offsets is optional, but we'll unconditionally allocate this array?

true :-\ will add the uref_ctr_offsets check

Quoted 40 lines
Quoted 40 lines
kernel/bpf/syscall.c +39 -0
+
+       if (!uprobes || !ref_ctr_offsets || !link)
+               goto error_free;
+
+       for (i = 0; i < cnt; i++) {
+               if (uref_ctr_offsets && __get_user(ref_ctr_offset, uref_ctr_offsets + i)) {
+                       err = -EFAULT;
+                       goto error_free;
+               }
+               if (__get_user(offset, uoffsets + i)) {
+                       err = -EFAULT;
+                       goto error_free;
+               }
+
+               uprobes[i].offset = offset;
+               uprobes[i].link = link;
+
+               if (flags & BPF_F_UPROBE_MULTI_RETURN)
+                       uprobes[i].consumer.ret_handler = uprobe_multi_link_ret_handler;
+               else
+                       uprobes[i].consumer.handler = uprobe_multi_link_handler;
+
+               ref_ctr_offsets[i] = ref_ctr_offset;
+       }
+
+       link->cnt = cnt;
+       link->uprobes = uprobes;
+       link->path = path;
+
+       bpf_link_init(&link->link, BPF_LINK_TYPE_UPROBE_MULTI,
+                     &bpf_uprobe_multi_link_lops, prog);
+
+       err = bpf_link_prime(&link->link, &link_primer);
+       if (err)
+               goto error_free;
+
+       for (i = 0; i < cnt; i++) {
+               err = uprobe_register_refctr(d_real_inode(link->path.dentry),
+                                            uprobes[i].offset, ref_ctr_offsets[i],
Quoted 8 lines

kernel/bpf/syscall.c

Quoted 3 lines
kernel/bpf/syscall.c +3 -0
+                                            &uprobes[i].consumer);
+               if (err) {
+                       bpf_uprobe_unregister(&path, uprobes, i);

bpf_link_cleanup() will do this through bpf_uprobe_multi_link_release(), no? So you are double unregistering? Either drop cnt to zero, or just don't do this here? Latter is better, IMO.

Quoted 1 line

bpf_link_cleanup path won't call release callback so we have to do that

bpf_link_cleanup() does fput(primer->file); which eventually calls release callback, no? I'd add printk and simulate failure just to be sure

Quoted 31 lines

I think I can add simple selftest to have this path covered

thanks,
jirka

Quoted 25 lines
Quoted 23 lines
kernel/bpf/syscall.c +22 -0
+                       bpf_link_cleanup(&link_primer);
+                       kvfree(ref_ctr_offsets);
+                       return err;
+               }
+       }
+
+       kvfree(ref_ctr_offsets);
+       return bpf_link_settle(&link_primer);
+
+error_free:
+       kvfree(ref_ctr_offsets);
+       kvfree(uprobes);
+       kfree(link);
+error_path_put:
+       path_put(&path);
+       return err;
+}
+#else /* !CONFIG_UPROBES */
+int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
+{
+       return -EOPNOTSUPP;
+}

[...]

#7 lore raw
Quoted 29 lines · Andrii Nakryiko
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
kernel/trace/bpf_trace.c +10 -0
+
+static int uprobe_prog_run(struct bpf_uprobe *uprobe,
+                          unsigned long entry_ip,
+                          struct pt_regs *regs)
+{
+       struct bpf_uprobe_multi_link *link = uprobe->link;
+       struct bpf_uprobe_multi_run_ctx run_ctx = {
+               .entry_ip = entry_ip,
+       };
+       struct bpf_prog *prog = link->link.prog;
Quoted 8 lines

kernel/trace/bpf_trace.c static u64 bpf_kprobe_multi_entry_ip(struct bpf_run_ctx *ctx) · line 2986

Quoted 6 lines
kernel/trace/bpf_trace.c +6 -0
+       struct bpf_run_ctx *old_run_ctx;
+       int err = 0;
+
+       might_fault();
+
+       rcu_read_lock_trace();

we don't need this if uprobe is not sleepable, right? why unconditional then?

Quoted 8 lines

I won't pretend I understand what rcu_read_lock_trace does ;-)

I tried to follow bpf_prog_run_array_sleepable where it's called unconditionally for both sleepable and non-sleepable progs

there are conditional rcu_read_un/lock calls later on

I will check

Quoted 7 lines

hm... Alexei can chime in here, but given here we actually are trying to run one BPF program (not entire array of them), we do know whether it's going to be sleepable or not. So we can avoid unnecessary rcu_read_{lock,unlock}_trace() calls. rcu_read_lock_trace() is used when there is going to be sleepable BPF program executed to protect BPF maps and other resources from being freed too soon. But if we know that we don't need sleepable, we can avoid that.

We can add more checks and bool flags to avoid rcu_read_{lock,unlock}_trace(), but it will likely be slower. These calls are very fast.
Simpler and faster to do it unconditionally even when the array doesn't have sleepable progs.
rcu_read_lock() we have to do conditionally, because it won't be ok if sleepable progs are in the array.

#8 lore raw
Quoted 40 lines · Alexei Starovoitov
Quoted 29 lines · Andrii Nakryiko
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
kernel/trace/bpf_trace.c +10 -0
+
+static int uprobe_prog_run(struct bpf_uprobe *uprobe,
+                          unsigned long entry_ip,
+                          struct pt_regs *regs)
+{
+       struct bpf_uprobe_multi_link *link = uprobe->link;
+       struct bpf_uprobe_multi_run_ctx run_ctx = {
+               .entry_ip = entry_ip,
+       };
+       struct bpf_prog *prog = link->link.prog;
Quoted 8 lines

kernel/trace/bpf_trace.c static u64 bpf_kprobe_multi_entry_ip(struct bpf_run_ctx *ctx) · line 2986

Quoted 6 lines
kernel/trace/bpf_trace.c +6 -0
+       struct bpf_run_ctx *old_run_ctx;
+       int err = 0;
+
+       might_fault();
+
+       rcu_read_lock_trace();

we don't need this if uprobe is not sleepable, right? why unconditional then?

Quoted 8 lines

I won't pretend I understand what rcu_read_lock_trace does ;-)

I tried to follow bpf_prog_run_array_sleepable where it's called unconditionally for both sleepable and non-sleepable progs

there are conditional rcu_read_un/lock calls later on

I will check

Quoted 7 lines

hm... Alexei can chime in here, but given here we actually are trying to run one BPF program (not entire array of them), we do know whether it's going to be sleepable or not. So we can avoid unnecessary rcu_read_{lock,unlock}_trace() calls. rcu_read_lock_trace() is used when there is going to be sleepable BPF program executed to protect BPF maps and other resources from being freed too soon. But if we know that we don't need sleepable, we can avoid that.

Quoted 2 lines

We can add more checks and bool flags to avoid rcu_read_{lock,unlock}_trace(), but it will likely be slower. These calls are very fast.

that's ok then. But seeing how we do

rcu_read_lock_trace();
if (!sleepable)

    rcu_read_lock();

it felt like we might as well just do

if (sleepable)

    rcu_read_lock_trace();
else
    rcu_read_lock();

As I mentioned, in this case we have a single bpf_prog, not a bpf_prog_array, so that changes things a bit.

But ultimately, the context switch required for uprobe dwarfs overhead of any of this, presumably, so it's a minor concern.

Quoted 4 lines

Simpler and faster to do it unconditionally even when the array doesn't have sleepable progs.
rcu_read_lock() we have to do conditionally, because it won't be ok if sleepable progs are in the array.

#9 lore raw
Quoted 54 lines · Andrii Nakryiko
Quoted 40 lines · Alexei Starovoitov
Quoted 29 lines · Andrii Nakryiko
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
kernel/trace/bpf_trace.c +10 -0
+
+static int uprobe_prog_run(struct bpf_uprobe *uprobe,
+                          unsigned long entry_ip,
+                          struct pt_regs *regs)
+{
+       struct bpf_uprobe_multi_link *link = uprobe->link;
+       struct bpf_uprobe_multi_run_ctx run_ctx = {
+               .entry_ip = entry_ip,
+       };
+       struct bpf_prog *prog = link->link.prog;
Quoted 8 lines

kernel/trace/bpf_trace.c static u64 bpf_kprobe_multi_entry_ip(struct bpf_run_ctx *ctx) · line 2986

Quoted 6 lines
kernel/trace/bpf_trace.c +6 -0
+       struct bpf_run_ctx *old_run_ctx;
+       int err = 0;
+
+       might_fault();
+
+       rcu_read_lock_trace();

we don't need this if uprobe is not sleepable, right? why unconditional then?

Quoted 8 lines

I won't pretend I understand what rcu_read_lock_trace does ;-)

I tried to follow bpf_prog_run_array_sleepable where it's called unconditionally for both sleepable and non-sleepable progs

there are conditional rcu_read_un/lock calls later on

I will check

Quoted 7 lines

hm... Alexei can chime in here, but given here we actually are trying to run one BPF program (not entire array of them), we do know whether it's going to be sleepable or not. So we can avoid unnecessary rcu_read_{lock,unlock}_trace() calls. rcu_read_lock_trace() is used when there is going to be sleepable BPF program executed to protect BPF maps and other resources from being freed too soon. But if we know that we don't need sleepable, we can avoid that.

Quoted 2 lines

We can add more checks and bool flags to avoid rcu_read_{lock,unlock}_trace(), but it will likely be slower. These calls are very fast.

that's ok then. But seeing how we do

rcu_read_lock_trace();
if (!sleepable)

    rcu_read_lock();

it felt like we might as well just do

Quoted 8 lines

if (sleepable)

    rcu_read_lock_trace();
else
    rcu_read_lock();

As I mentioned, in this case we have a single bpf_prog, not a bpf_prog_array, so that changes things a bit.

Ahh. It's only one prog. I missed that. Above makes sense then. But why is it not an array? We can attach multiple uprobes to the same location. Anyway that can be dealt with later.

#10 lore raw
Quoted 54 lines · Alexei Starovoitov
Quoted 52 lines · Andrii Nakryiko
Quoted 40 lines · Alexei Starovoitov
Quoted 29 lines · Andrii Nakryiko
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
(quoted diff) +10 -0
+
+static int uprobe_prog_run(struct bpf_uprobe *uprobe,
+                          unsigned long entry_ip,
+                          struct pt_regs *regs)
+{
+       struct bpf_uprobe_multi_link *link = uprobe->link;
+       struct bpf_uprobe_multi_run_ctx run_ctx = {
+               .entry_ip = entry_ip,
+       };
+       struct bpf_prog *prog = link->link.prog;
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+       struct bpf_run_ctx *old_run_ctx;
+       int err = 0;
+
+       might_fault();
+
+       rcu_read_lock_trace();

we don't need this if uprobe is not sleepable, right? why unconditional then?

Quoted 8 lines

I won't pretend I understand what rcu_read_lock_trace does ;-)

I tried to follow bpf_prog_run_array_sleepable where it's called unconditionally for both sleepable and non-sleepable progs

there are conditional rcu_read_un/lock calls later on

I will check

Quoted 7 lines

hm... Alexei can chime in here, but given here we actually are trying to run one BPF program (not entire array of them), we do know whether it's going to be sleepable or not. So we can avoid unnecessary rcu_read_{lock,unlock}_trace() calls. rcu_read_lock_trace() is used when there is going to be sleepable BPF program executed to protect BPF maps and other resources from being freed too soon. But if we know that we don't need sleepable, we can avoid that.

Quoted 2 lines

We can add more checks and bool flags to avoid rcu_read_{lock,unlock}_trace(), but it will likely be slower. These calls are very fast.

that's ok then. But seeing how we do

rcu_read_lock_trace();
if (!sleepable)

    rcu_read_lock();
Quoted 6 lines
Quoted 6 lines

it felt like we might as well just do

if (sleepable)

    rcu_read_lock_trace();
else
    rcu_read_lock();

ok

Quoted 8 lines
Quoted 4 lines

As I mentioned, in this case we have a single bpf_prog, not a bpf_prog_array, so that changes things a bit.

Ahh. It's only one prog. I missed that. Above makes sense then. But why is it not an array? We can attach multiple uprobes to the same location. Anyway that can be dealt with later.

I think we could add support for this later if it's needed

jirka

#11 lore raw
In reply to Andrii Nakryiko

On Fri, Jun 23, 2023 at 09:24:22AM -0700, Andrii Nakryiko wrote:

SNIP

Quoted 48 lines
Quoted 48 lines
Quoted 47 lines
Quoted 32 lines
kernel/trace/bpf_trace.c +32 -0
+
+       if (!uprobes || !ref_ctr_offsets || !link)
+               goto error_free;
+
+       for (i = 0; i < cnt; i++) {
+               if (uref_ctr_offsets && __get_user(ref_ctr_offset, uref_ctr_offsets + i)) {
+                       err = -EFAULT;
+                       goto error_free;
+               }
+               if (__get_user(offset, uoffsets + i)) {
+                       err = -EFAULT;
+                       goto error_free;
+               }
+
+               uprobes[i].offset = offset;
+               uprobes[i].link = link;
+
+               if (flags & BPF_F_UPROBE_MULTI_RETURN)
+                       uprobes[i].consumer.ret_handler = uprobe_multi_link_ret_handler;
+               else
+                       uprobes[i].consumer.handler = uprobe_multi_link_handler;
+
+               ref_ctr_offsets[i] = ref_ctr_offset;
+       }
+
+       link->cnt = cnt;
+       link->uprobes = uprobes;
+       link->path = path;
+
+       bpf_link_init(&link->link, BPF_LINK_TYPE_UPROBE_MULTI,
+                     &bpf_uprobe_multi_link_lops, prog);
+

kernel/trace/bpf_trace.c static u64 bpf_kprobe_multi_entry_ip(struct bpf_run_ctx *ctx) · line 3128

Quoted 10 lines
kernel/trace/bpf_trace.c +10 -0
+       err = bpf_link_prime(&link->link, &link_primer);
+       if (err)
+               goto error_free;
+
+       for (i = 0; i < cnt; i++) {
+               err = uprobe_register_refctr(d_real_inode(link->path.dentry),
+                                            uprobes[i].offset, ref_ctr_offsets[i],
+                                            &uprobes[i].consumer);
+               if (err) {
+                       bpf_uprobe_unregister(&path, uprobes, i);

bpf_link_cleanup() will do this through bpf_uprobe_multi_link_release(), no? So you are double unregistering? Either drop cnt to zero, or just don't do this here? Latter is better, IMO.

Quoted 5 lines
Quoted 1 line

bpf_link_cleanup path won't call release callback so we have to do that

bpf_link_cleanup() does fput(primer->file); which eventually calls release callback, no? I'd add printk and simulate failure just to be sure

I recall we had similar discussion for kprobe_multi link ;-)

I'll double check that but I think bpf_link_cleanup calls just dealloc callback not release

jirka

Quoted 6 lines
Quoted 6 lines

I think I can add simple selftest to have this path covered

thanks,
jirka

SNIP

#12 lore raw
Quoted 55 lines · Jiri Olsa

On Fri, Jun 23, 2023 at 09:24:22AM -0700, Andrii Nakryiko wrote:

SNIP

Quoted 50 lines
Quoted 49 lines
Quoted 39 lines
Quoted 39 lines
kernel/trace/bpf_trace.c +39 -0
+
+       if (!uprobes || !ref_ctr_offsets || !link)
+               goto error_free;
+
+       for (i = 0; i < cnt; i++) {
+               if (uref_ctr_offsets && __get_user(ref_ctr_offset, uref_ctr_offsets + i)) {
+                       err = -EFAULT;
+                       goto error_free;
+               }
+               if (__get_user(offset, uoffsets + i)) {
+                       err = -EFAULT;
+                       goto error_free;
+               }
+
+               uprobes[i].offset = offset;
+               uprobes[i].link = link;
+
+               if (flags & BPF_F_UPROBE_MULTI_RETURN)
+                       uprobes[i].consumer.ret_handler = uprobe_multi_link_ret_handler;
+               else
+                       uprobes[i].consumer.handler = uprobe_multi_link_handler;
+
+               ref_ctr_offsets[i] = ref_ctr_offset;
+       }
+
+       link->cnt = cnt;
+       link->uprobes = uprobes;
+       link->path = path;
+
+       bpf_link_init(&link->link, BPF_LINK_TYPE_UPROBE_MULTI,
+                     &bpf_uprobe_multi_link_lops, prog);
+
+       err = bpf_link_prime(&link->link, &link_primer);
+       if (err)
+               goto error_free;
+
+       for (i = 0; i < cnt; i++) {
+               err = uprobe_register_refctr(d_real_inode(link->path.dentry),
+                                            uprobes[i].offset, ref_ctr_offsets[i],
Quoted 8 lines

kernel/trace/bpf_trace.c static u64 bpf_kprobe_multi_entry_ip(struct bpf_run_ctx *ctx) · line 3128

Quoted 3 lines
kernel/trace/bpf_trace.c +3 -0
+                                            &uprobes[i].consumer);
+               if (err) {
+                       bpf_uprobe_unregister(&path, uprobes, i);

bpf_link_cleanup() will do this through bpf_uprobe_multi_link_release(), no? So you are double unregistering? Either drop cnt to zero, or just don't do this here? Latter is better, IMO.

bpf_link_cleanup path won't call release callback so we have to do that

Quoted 8 lines
Quoted 3 lines

bpf_link_cleanup() does fput(primer->file); which eventually calls release callback, no? I'd add printk and simulate failure just to be sure

I recall we had similar discussion for kprobe_multi link ;-)

I'll double check that but I think bpf_link_cleanup calls just dealloc callback not release

Let's document this in comments for bpf_link_cleanup() so we don't have to discuss this again :)

I think you are right, btw. I see that bpf_link_cleanup() sets link->prog to NULL, and bpf_link_free() won't call link->ops->release() if link->prog is NULL.

Tricky, I keep forgetting this. Let's explicitly explain this in a comment.

Quoted 11 lines

jirka

Quoted 6 lines
Quoted 6 lines

I think I can add simple selftest to have this path covered

thanks,
jirka

SNIP

#13 lore raw

On Mon, Jun 26, 2023 at 11:27:25AM -0700, Andrii Nakryiko wrote:

SNIP

Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines

bpf_link_cleanup() will do this through bpf_uprobe_multi_link_release(), no? So you are double unregistering? Either drop cnt to zero, or just don't do this here? Latter is better, IMO.

bpf_link_cleanup path won't call release callback so we have to do that

Quoted 8 lines
Quoted 3 lines

bpf_link_cleanup() does fput(primer->file); which eventually calls release callback, no? I'd add printk and simulate failure just to be sure

I recall we had similar discussion for kprobe_multi link ;-)

I'll double check that but I think bpf_link_cleanup calls just dealloc callback not release

Quoted 8 lines

Let's document this in comments for bpf_link_cleanup() so we don't have to discuss this again :)

I think you are right, btw. I see that bpf_link_cleanup() sets link->prog to NULL, and bpf_link_free() won't call link->ops->release() if link->prog is NULL.

Tricky, I keep forgetting this. Let's explicitly explain this in a comment.

ok, will add the comment

jirka

Quoted 12 lines
Quoted 12 lines

jirka

Quoted 6 lines
Quoted 6 lines

I think I can add simple selftest to have this path covered

thanks,
jirka

SNIP

author reviewer
#14 lore raw

PATCH v2 2/24 bpf: Add cookies support for uprobe_multi link

In reply to Jiri Olsa

Adding support to specify cookies array for uprobe_multi link.

The cookies array share indexes and length with other uprobe_multi arrays (offsets/ref_ctr_offsets).

The cookies[i] value defines cookie for i-the uprobe and will be returned by bpf_get_attach_cookie helper when called from ebpf program hooked to that specific uprobe.

---
 include/uapi/linux/bpf.h       |  1 +
 kernel/bpf/syscall.c           |  2 +-
 kernel/trace/bpf_trace.c       | 48 +++++++++++++++++++++++++++++++---
 tools/include/uapi/linux/bpf.h |  1 +
 4 files changed, 47 insertions(+), 5 deletions(-)
include/uapi/linux/bpf.h +1 -0
index bfbc1246b220..12d4174fce8f 100644
@@ -1591,6 +1591,7 @@ union bpf_attr {
 				__aligned_u64	path;
 				__aligned_u64	offsets;
 				__aligned_u64	ref_ctr_offsets;
+				__aligned_u64	cookies;
 			} uprobe_multi;
 		};
 	} link_create;
kernel/bpf/syscall.c +1 -1
index a96e46cd407e..3ae444898c15 100644
@@ -4651,7 +4651,7 @@ static int bpf_map_do_batch(const union bpf_attr *attr,
 	return err;
 }
 
-#define BPF_LINK_CREATE_LAST_FIELD link_create.kprobe_multi.cookies
+#define BPF_LINK_CREATE_LAST_FIELD link_create.uprobe_multi.cookies
 static int link_create(union bpf_attr *attr, bpfptr_t uattr)
 {
 	enum bpf_prog_type ptype;
kernel/trace/bpf_trace.c +44 -4

124 lines not on this page · raw message

tools/include/uapi/linux/bpf.h +1 -0
index bfbc1246b220..12d4174fce8f 100644
@@ -1591,6 +1591,7 @@ union bpf_attr {
 				__aligned_u64	path;
 				__aligned_u64	offsets;
 				__aligned_u64	ref_ctr_offsets;
+				__aligned_u64	cookies;
 			} uprobe_multi;
 		};
 	} link_create;
Signature · 1 line
2.41.0
#15 lore raw

Re: [PATCHv2 bpf-next 02/24] bpf: Add cookies support for uprobe_multi link

Quoted 19 lines · Jiri Olsa

Adding support to specify cookies array for uprobe_multi link.

The cookies array share indexes and length with other uprobe_multi arrays (offsets/ref_ctr_offsets).

The cookies[i] value defines cookie for i-the uprobe and will be returned by bpf_get_attach_cookie helper when called from ebpf program hooked to that specific uprobe.

---
 include/uapi/linux/bpf.h       |  1 +
 kernel/bpf/syscall.c           |  2 +-
 kernel/trace/bpf_trace.c       | 48 +++++++++++++++++++++++++++++++---
 tools/include/uapi/linux/bpf.h |  1 +
 4 files changed, 47 insertions(+), 5 deletions(-)

[...]

kernel/trace/bpf_trace.c uprobe_multi_link_ret_handler(struct uprobe_consumer *con, unsigned long func, s · line 3053

Quoted 10 lines
kernel/trace/bpf_trace.c +6 -0
@@ -3026,6 +3045,16 @@ uprobe_multi_link_ret_handler(struct uprobe_consumer *con, unsigned long func, s
        return uprobe_prog_run(uprobe, func, regs);
 }
 
+static u64 bpf_uprobe_multi_cookie(struct bpf_run_ctx *ctx)
+{
+       struct bpf_uprobe_multi_run_ctx *run_ctx;
+
+       if (!ctx)
+               return 0;

can't happen, let's crash if it does happen?

Quoted 4 lines
kernel/trace/bpf_trace.c +4 -0
+       run_ctx = container_of(current->bpf_ctx, struct bpf_uprobe_multi_run_ctx, run_ctx);
+       return run_ctx->uprobe->cookie;
+}
+

[...]

#16 lore raw

Re: [PATCHv2 bpf-next 02/24] bpf: Add cookies support for uprobe_multi link

Quoted 27 lines · Andrii Nakryiko
Quoted 19 lines · Jiri Olsa

Adding support to specify cookies array for uprobe_multi link.

The cookies array share indexes and length with other uprobe_multi arrays (offsets/ref_ctr_offsets).

The cookies[i] value defines cookie for i-the uprobe and will be returned by bpf_get_attach_cookie helper when called from ebpf program hooked to that specific uprobe.

---
 include/uapi/linux/bpf.h       |  1 +
 kernel/bpf/syscall.c           |  2 +-
 kernel/trace/bpf_trace.c       | 48 +++++++++++++++++++++++++++++++---
 tools/include/uapi/linux/bpf.h |  1 +
 4 files changed, 47 insertions(+), 5 deletions(-)

[...]

Quoted 4 lines
kernel/trace/bpf_trace.c +0 -0
@@ -3026,6 +3045,16 @@ uprobe_multi_link_ret_handler(struct uprobe_consumer *con, unsigned long func, s
        return uprobe_prog_run(uprobe, func, regs);
 }
 
Quoted 8 lines

kernel/trace/bpf_trace.c uprobe_multi_link_ret_handler(struct uprobe_consumer *con, unsigned long func, s · line 3053

Quoted 6 lines
kernel/trace/bpf_trace.c +6 -0
+static u64 bpf_uprobe_multi_cookie(struct bpf_run_ctx *ctx)
+{
+       struct bpf_uprobe_multi_run_ctx *run_ctx;
+
+       if (!ctx)
+               return 0;

can't happen, let's crash if it does happen?

ok, will remove

jirka

Quoted 8 lines
Quoted 6 lines
kernel/trace/bpf_trace.c +4 -0
+       run_ctx = container_of(current->bpf_ctx, struct bpf_uprobe_multi_run_ctx, run_ctx);
+       return run_ctx->uprobe->cookie;
+}
+

[...]

author reviewer
#17 lore raw

PATCH v2 3/24 bpf: Add pid filter support for uprobe_multi link

In reply to Jiri Olsa

Adding support to specify pid for uprobe_multi link and the uprobes are created only for task with given pid value.

Using the consumer.filter filter callback for that, so the task gets filtered during the uprobe installation.

We still need to check the task during runtime in the uprobe handler, because the handler could get executed if there's another system wide consumer on the same uprobe (thanks Oleg for the insight).

Cc Oleg Nesterov <[email protected]>
---
 include/uapi/linux/bpf.h       |  1 +
 kernel/bpf/syscall.c           |  2 +-
 kernel/trace/bpf_trace.c       | 33 +++++++++++++++++++++++++++++++++
 tools/include/uapi/linux/bpf.h |  1 +
 4 files changed, 36 insertions(+), 1 deletion(-)
include/uapi/linux/bpf.h +1 -0
index 12d4174fce8f..117acca838f1 100644
@@ -1592,6 +1592,7 @@ union bpf_attr {
 				__aligned_u64	offsets;
 				__aligned_u64	ref_ctr_offsets;
 				__aligned_u64	cookies;
+				__u32		pid;
 			} uprobe_multi;
 		};
 	} link_create;
kernel/bpf/syscall.c +1 -1
index 3ae444898c15..b871f0eb8a05 100644
@@ -4651,7 +4651,7 @@ static int bpf_map_do_batch(const union bpf_attr *attr,
 	return err;
 }
 
-#define BPF_LINK_CREATE_LAST_FIELD link_create.uprobe_multi.cookies
+#define BPF_LINK_CREATE_LAST_FIELD link_create.uprobe_multi.pid
 static int link_create(union bpf_attr *attr, bpfptr_t uattr)
 {
 	enum bpf_prog_type ptype;
kernel/trace/bpf_trace.c +33 -0
index f9cd7d283426..5c5b543d7d03 100644
@@ -2945,6 +2945,7 @@ struct bpf_uprobe_multi_link {
 	struct bpf_link link;
 	u32 cnt;
 	struct bpf_uprobe *uprobes;
+	struct task_struct *task;
 };
 
 struct bpf_uprobe_multi_run_ctx {
@@ -2971,6 +2972,8 @@ static void bpf_uprobe_multi_link_release(struct bpf_link *link)
 	umulti_link = container_of(link, struct bpf_uprobe_multi_link, link);
 	bpf_uprobe_unregister(&umulti_link->path, umulti_link->uprobes, umulti_link->cnt);
 	path_put(&umulti_link->path);
+	if (umulti_link->task)
+		put_task_struct(umulti_link->task);
 }
 
 static void bpf_uprobe_multi_link_dealloc(struct bpf_link *link)
@@ -3000,6 +3003,9 @@ static int uprobe_prog_run(struct bpf_uprobe *uprobe,
 	struct bpf_run_ctx *old_run_ctx;
 	int err = 0;
 
+	if (link->task && current != link->task)
+		return 0;
+
 	might_fault();
 
 	rcu_read_lock_trace();
@@ -3027,6 +3033,16 @@ static int uprobe_prog_run(struct bpf_uprobe *uprobe,
 	return err;
 }
 
+static bool
+uprobe_multi_link_filter(struct uprobe_consumer *con, enum uprobe_filter_ctx ctx,
+			 struct mm_struct *mm)
+{
+	struct bpf_uprobe *uprobe;
+
+	uprobe = container_of(con, struct bpf_uprobe, consumer);
+	return uprobe->link->task->mm == mm;
+}
+
 static int
 uprobe_multi_link_handler(struct uprobe_consumer *con, struct pt_regs *regs)
 {
@@ -3064,10 +3080,12 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr
 	struct bpf_link_primer link_primer;
 	struct bpf_uprobe *uprobes = NULL;
 	u64 __user *ucookies, cookie = 0;
+	struct task_struct *task = NULL;
 	void __user *upath;
 	u32 flags, cnt, i;
 	struct path path;
 	char *name;
+	pid_t pid;
 	int err;
 
 	/* no support for 32bit archs yet */
@@ -3110,6 +3128,15 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr
 		goto error_path_put;
 	}
 
+	pid = attr->link_create.uprobe_multi.pid;
+	if (pid) {
+		rcu_read_lock();
+		task = get_pid_task(find_vpid(pid), PIDTYPE_PID);
+		rcu_read_unlock();
+		if (!task)
+			goto error_path_put;
+	}
+
 	err = -ENOMEM;
 
 	link = kzalloc(sizeof(*link), GFP_KERNEL);
@@ -3143,11 +3170,15 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr
 			uprobes[i].consumer.handler = uprobe_multi_link_handler;
 
 		ref_ctr_offsets[i] = ref_ctr_offset;
+
+		if (pid)
+			uprobes[i].consumer.filter = uprobe_multi_link_filter;
 	}
 
 	link->cnt = cnt;
 	link->uprobes = uprobes;
 	link->path = path;
+	link->task = task;
 
 	bpf_link_init(&link->link, BPF_LINK_TYPE_UPROBE_MULTI,
 		      &bpf_uprobe_multi_link_lops, prog);
@@ -3175,6 +3206,8 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr
 	kvfree(ref_ctr_offsets);
 	kvfree(uprobes);
 	kfree(link);
+	if (task)
+		put_task_struct(task);
 error_path_put:
 	path_put(&path);
 	return err;
tools/include/uapi/linux/bpf.h +1 -0
index 12d4174fce8f..117acca838f1 100644
@@ -1592,6 +1592,7 @@ union bpf_attr {
 				__aligned_u64	offsets;
 				__aligned_u64	ref_ctr_offsets;
 				__aligned_u64	cookies;
+				__u32		pid;
 			} uprobe_multi;
 		};
 	} link_create;
Signature · 1 line
2.41.0
Reviewed-by on v2 3/24
#18 lore raw

Re: [PATCHv2 bpf-next 03/24] bpf: Add pid filter support for uprobe_multi link

Quoted 4 lines · Jiri Olsa

We still need to check the task during runtime in the uprobe handler, because the handler could get executed if there's another system wide consumer on the same uprobe (thanks Oleg for the insight).

... or link->task's sub-thread hits this uprobe.

author reviewer
#19 lore raw

PATCH v2 4/24 bpf: Add bpf_get_func_ip helper support for uprobe link

In reply to Jiri Olsa

Adding support for bpf_get_func_ip helper being called from ebpf program attached by uprobe_multi link.

It returns the ip of the uprobe.

---
 kernel/trace/bpf_trace.c | 33 ++++++++++++++++++++++++++++++---
 1 file changed, 30 insertions(+), 3 deletions(-)
kernel/trace/bpf_trace.c +30 -3
index 5c5b543d7d03..dfba0bb479a2 100644
@@ -88,6 +88,7 @@ static u64 bpf_kprobe_multi_cookie(struct bpf_run_ctx *ctx);
 static u64 bpf_kprobe_multi_entry_ip(struct bpf_run_ctx *ctx);
 
 static u64 bpf_uprobe_multi_cookie(struct bpf_run_ctx *ctx);
+static u64 bpf_uprobe_multi_entry_ip(struct bpf_run_ctx *ctx);
 
 /**
  * trace_call_bpf - invoke BPF program
@@ -1091,6 +1092,18 @@ static const struct bpf_func_proto bpf_get_attach_cookie_proto_kmulti = {
 	.arg1_type	= ARG_PTR_TO_CTX,
 };
 
+BPF_CALL_1(bpf_get_func_ip_uprobe_multi, struct pt_regs *, regs)
+{
+	return bpf_uprobe_multi_entry_ip(current->bpf_ctx);
+}
+
+static const struct bpf_func_proto bpf_get_func_ip_proto_uprobe_multi = {
+	.func		= bpf_get_func_ip_uprobe_multi,
+	.gpl_only	= false,
+	.ret_type	= RET_INTEGER,
+	.arg1_type	= ARG_PTR_TO_CTX,
+};
+
 BPF_CALL_1(bpf_get_attach_cookie_uprobe_multi, struct pt_regs *, regs)
 {
 	return bpf_uprobe_multi_cookie(current->bpf_ctx);
@@ -1545,9 +1558,11 @@ kprobe_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
 		return &bpf_override_return_proto;
 #endif
 	case BPF_FUNC_get_func_ip:
-		return prog->expected_attach_type == BPF_TRACE_KPROBE_MULTI ?
-			&bpf_get_func_ip_proto_kprobe_multi :
-			&bpf_get_func_ip_proto_kprobe;
+		if (prog->expected_attach_type == BPF_TRACE_KPROBE_MULTI)
+			return &bpf_get_func_ip_proto_kprobe_multi;
+		if (prog->expected_attach_type == BPF_TRACE_UPROBE_MULTI)
+			return &bpf_get_func_ip_proto_uprobe_multi;
+		return &bpf_get_func_ip_proto_kprobe;
 	case BPF_FUNC_get_attach_cookie:
 		if (prog->expected_attach_type == BPF_TRACE_KPROBE_MULTI)
 			return &bpf_get_attach_cookie_proto_kmulti;
@@ -3061,6 +3076,14 @@ uprobe_multi_link_ret_handler(struct uprobe_consumer *con, unsigned long func, s
 	return uprobe_prog_run(uprobe, func, regs);
 }
 
+static u64 bpf_uprobe_multi_entry_ip(struct bpf_run_ctx *ctx)
+{
+	struct bpf_uprobe_multi_run_ctx *run_ctx;
+
+	run_ctx = container_of(current->bpf_ctx, struct bpf_uprobe_multi_run_ctx, run_ctx);
+	return run_ctx->entry_ip;
+}
+
 static u64 bpf_uprobe_multi_cookie(struct bpf_run_ctx *ctx)
 {
 	struct bpf_uprobe_multi_run_ctx *run_ctx;
@@ -3221,4 +3244,8 @@ static u64 bpf_uprobe_multi_cookie(struct bpf_run_ctx *ctx)
 {
 	return 0;
 }
+static u64 bpf_uprobe_multi_entry_ip(struct bpf_run_ctx *ctx)
+{
+	return 0;
+}
 #endif /* CONFIG_UPROBES */
Signature · 1 line
2.41.0
author reviewer
#20 lore raw

PATCH v2 5/24 libbpf: Add uprobe_multi attach type and link names

In reply to Jiri Olsa

Adding new uprobe_multi attach type and link names, so the functions can resolve the new values.

---
 tools/lib/bpf/libbpf.c | 2 ++
 1 file changed, 2 insertions(+)
tools/lib/bpf/libbpf.c +2 -0
index 47632606b06d..af52188daa80 100644
@@ -117,6 +117,7 @@ static const char * const attach_type_name[] = {
 	[BPF_PERF_EVENT]		= "perf_event",
 	[BPF_TRACE_KPROBE_MULTI]	= "trace_kprobe_multi",
 	[BPF_STRUCT_OPS]		= "struct_ops",
+	[BPF_TRACE_UPROBE_MULTI]	= "trace_uprobe_multi",
 };
 
 static const char * const link_type_name[] = {
@@ -131,6 +132,7 @@ static const char * const link_type_name[] = {
 	[BPF_LINK_TYPE_KPROBE_MULTI]		= "kprobe_multi",
 	[BPF_LINK_TYPE_STRUCT_OPS]		= "struct_ops",
 	[BPF_LINK_TYPE_NETFILTER]		= "netfilter",
+	[BPF_LINK_TYPE_UPROBE_MULTI]		= "uprobe_multi",
 };
 
 static const char * const map_type_name[] = {
Signature · 1 line
2.41.0
author reviewer
#21 lore raw

PATCH v2 6/24 libbpf: Add elf symbol iterator

In reply to Jiri Olsa

Adding elf symbol iterator object (and some functions) that follow open-coded iterator pattern and some functions to ease up iterating elf object symbols.

The idea is to iterate single symbol section with:

struct elf_symbol_iter iter;
struct elf_symbol *sym;

if (elf_symbol_iter_new(&iter, elf, binary_path, SHT_DYNSYM))

        goto error;
  while ((sym = elf_symbol_iter_next(&iter))) {
        ...
  }

I considered opening the elf inside the iterator and iterate all symbol sections, but then it gets more complicated wrt user checks for when the next section is processed.

Plus side is the we don't need 'exit' function, because caller/user is in charge of that.

The returned iterated symbol object from elf_symbol_iter_next function is placed inside the struct elf_symbol_iter, so no extra allocation or argument is needed.

---
 tools/lib/bpf/libbpf.c | 179 ++++++++++++++++++++++++++---------------
 1 file changed, 114 insertions(+), 65 deletions(-)
tools/lib/bpf/libbpf.c +114 -65
index af52188daa80..cdac368c7ce1 100644
@@ -10824,6 +10824,109 @@ static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn)
 	return NULL;
 }
 
+struct elf_symbol {
+	const char *name;
+	unsigned long offset;
+	int bind;
+};
+
+struct elf_symbol_iter {
+	Elf *elf;
+	Elf_Data *symbols;
+	size_t nr_syms;
+	size_t strtabidx;
+	size_t idx;
+	struct elf_symbol sym;
+};
+
+static int elf_symbol_iter_new(struct elf_symbol_iter *iter,
+			       Elf *elf, const char *binary_path,
+			       int sh_type)
+{
+	Elf_Scn *scn = NULL;
+	GElf_Ehdr ehdr;
+	GElf_Shdr sh;
+
+	memset(iter, 0, sizeof(*iter));
+
+	if (!gelf_getehdr(elf, &ehdr)) {
+		pr_warn("elf: failed to get ehdr from %s: %s\n", binary_path, elf_errmsg(-1));
+		return -LIBBPF_ERRNO__FORMAT;
+	}
+
+	scn = elf_find_next_scn_by_type(elf, sh_type, NULL);
+	if (!scn) {
+		pr_debug("elf: failed to find symbol table ELF sections in '%s'\n",
+			 binary_path);
+		return -EINVAL;
+	}
+
+	if (!gelf_getshdr(scn, &sh))
+		return -EINVAL;
+
+	iter->strtabidx = sh.sh_link;
+	iter->symbols = elf_getdata(scn, 0);
+	if (!iter->symbols) {
+		pr_warn("elf: failed to get symbols for symtab section in '%s': %s\n",
+			binary_path, elf_errmsg(-1));
+		return -LIBBPF_ERRNO__FORMAT;
+	}
+	iter->nr_syms = iter->symbols->d_size / sh.sh_entsize;
+	iter->elf = elf;
+	return 0;
+}
+
+static struct elf_symbol *elf_symbol_iter_next(struct elf_symbol_iter *iter)
+{
+	struct elf_symbol *ret = &iter->sym;
+	unsigned long offset = 0;
+	const char *name = NULL;
+	GElf_Shdr sym_sh;
+	Elf_Scn *sym_scn;
+	GElf_Sym sym;
+	size_t idx;
+
+	for (idx = iter->idx; idx < iter->nr_syms; idx++) {
+		if (!gelf_getsym(iter->symbols, idx, &sym))
+			continue;
+		if (GELF_ST_TYPE(sym.st_info) != STT_FUNC)
+			continue;
+		name = elf_strptr(iter->elf, iter->strtabidx, sym.st_name);
+		if (!name)
+			continue;
+
+		/* Transform symbol's virtual address (absolute for
+		 * binaries and relative for shared libs) into file
+		 * offset, which is what kernel is expecting for
+		 * uprobe/uretprobe attachment.
+		 * See Documentation/trace/uprobetracer.rst for more
+		 * details.
+		 * This is done by looking up symbol's containing
+		 * section's header and using iter's virtual address
+		 * (sh_addr) and corresponding file offset (sh_offset)
+		 * to transform sym.st_value (virtual address) into
+		 * desired final file offset.
+		 */
+		sym_scn = elf_getscn(iter->elf, sym.st_shndx);
+		if (!sym_scn)
+			continue;
+		if (!gelf_getshdr(sym_scn, &sym_sh))
+			continue;
+
+		offset = sym.st_value - sym_sh.sh_addr + sym_sh.sh_offset;
+		break;
+	}
+
+	/* we reached the last symbol */
+	if (idx == iter->nr_syms)
+		return NULL;
+	iter->idx = idx + 1;
+	ret->name = name;
+	ret->bind = GELF_ST_BIND(sym.st_info);
+	ret->offset = offset;
+	return ret;
+}
+
 /* Find offset of function name in the provided ELF object. "binary_path" is
  * the path to the ELF binary represented by "elf", and only used for error
  * reporting matters. "name" matches symbol name or name@@LIB for library
@@ -10855,94 +10958,40 @@ static long elf_find_func_offset(Elf *elf, const char *binary_path, const char *
 	 * reported as a warning/error.
 	 */
 	for (i = 0; i < ARRAY_SIZE(sh_types); i++) {
-		size_t nr_syms, strtabidx, idx;
-		Elf_Data *symbols = NULL;
-		Elf_Scn *scn = NULL;
+		struct elf_symbol_iter iter;
+		struct elf_symbol *sym;
 		int last_bind = -1;
-		const char *sname;
-		GElf_Shdr sh;
 
-		scn = elf_find_next_scn_by_type(elf, sh_types[i], NULL);
-		if (!scn) {
-			pr_debug("elf: failed to find symbol table ELF sections in '%s'\n",
-				 binary_path);
+		if (elf_symbol_iter_new(&iter, elf, binary_path, sh_types[i]))
 			continue;
-		}
-		if (!gelf_getshdr(scn, &sh))
-			continue;
-		strtabidx = sh.sh_link;
-		symbols = elf_getdata(scn, 0);
-		if (!symbols) {
-			pr_warn("elf: failed to get symbols for symtab section in '%s': %s\n",
-				binary_path, elf_errmsg(-1));
-			ret = -LIBBPF_ERRNO__FORMAT;
-			goto out;
-		}
-		nr_syms = symbols->d_size / sh.sh_entsize;
-
-		for (idx = 0; idx < nr_syms; idx++) {
-			int curr_bind;
-			GElf_Sym sym;
-			Elf_Scn *sym_scn;
-			GElf_Shdr sym_sh;
-
-			if (!gelf_getsym(symbols, idx, &sym))
-				continue;
-
-			if (GELF_ST_TYPE(sym.st_info) != STT_FUNC)
-				continue;
-
-			sname = elf_strptr(elf, strtabidx, sym.st_name);
-			if (!sname)
-				continue;
-
-			curr_bind = GELF_ST_BIND(sym.st_info);
 
+		while ((sym = elf_symbol_iter_next(&iter))) {
 			/* User can specify func, func@@LIB or func@@LIB_VERSION. */
-			if (strncmp(sname, name, name_len) != 0)
+			if (strncmp(sym->name, name, name_len) != 0)
 				continue;
 			/* ...but we don't want a search for "foo" to match 'foo2" also, so any
 			 * additional characters in sname should be of the form "@@LIB".
 			 */
-			if (!is_name_qualified && sname[name_len] != '\0' && sname[name_len] != '@')
+			if (!is_name_qualified && sym->name[name_len] != '\0' && sym->name[name_len] != '@')
 				continue;
 
 			if (ret >= 0) {
 				/* handle multiple matches */
-				if (last_bind != STB_WEAK && curr_bind != STB_WEAK) {
+				if (last_bind != STB_WEAK && sym->bind != STB_WEAK) {
 					/* Only accept one non-weak bind. */
 					pr_warn("elf: ambiguous match for '%s', '%s' in '%s'\n",
-						sname, name, binary_path);
+						sym->name, name, binary_path);
 					ret = -LIBBPF_ERRNO__FORMAT;
 					goto out;
-				} else if (curr_bind == STB_WEAK) {
+				} else if (sym->bind == STB_WEAK) {
 					/* already have a non-weak bind, and
 					 * this is a weak bind, so ignore.
 					 */
 					continue;
 				}
 			}
-
-			/* Transform symbol's virtual address (absolute for
-			 * binaries and relative for shared libs) into file
-			 * offset, which is what kernel is expecting for
-			 * uprobe/uretprobe attachment.
-			 * See Documentation/trace/uprobetracer.rst for more
-			 * details.
-			 * This is done by looking up symbol's containing
-			 * section's header and using it's virtual address
-			 * (sh_addr) and corresponding file offset (sh_offset)
-			 * to transform sym.st_value (virtual address) into
-			 * desired final file offset.
-			 */
-			sym_scn = elf_getscn(elf, sym.st_shndx);
-			if (!sym_scn)
-				continue;
-			if (!gelf_getshdr(sym_scn, &sym_sh))
-				continue;
-
-			ret = sym.st_value - sym_sh.sh_addr + sym_sh.sh_offset;
-			last_bind = curr_bind;
+			last_bind = sym->bind;
+			ret = sym->offset;
 		}
 		if (ret > 0)
 			break;
Signature · 1 line
2.41.0
#22 lore raw

Re: [PATCHv2 bpf-next 06/24] libbpf: Add elf symbol iterator

Quoted 28 lines · Jiri Olsa

Adding elf symbol iterator object (and some functions) that follow open-coded iterator pattern and some functions to ease up iterating elf object symbols.

The idea is to iterate single symbol section with:

struct elf_symbol_iter iter;
struct elf_symbol *sym;

if (elf_symbol_iter_new(&iter, elf, binary_path, SHT_DYNSYM))

        goto error;
  while ((sym = elf_symbol_iter_next(&iter))) {
        ...
  }

I considered opening the elf inside the iterator and iterate all symbol sections, but then it gets more complicated wrt user checks for when the next section is processed.

Plus side is the we don't need 'exit' function, because caller/user is in charge of that.

The returned iterated symbol object from elf_symbol_iter_next function is placed inside the struct elf_symbol_iter, so no extra allocation or argument is needed.

Quoted 6 lines
---
 tools/lib/bpf/libbpf.c | 179 ++++++++++++++++++++++++++---------------
 1 file changed, 114 insertions(+), 65 deletions(-)

This is great. Left a few nits below. I'm thinkin maybe we should add a separate elf.c file for all these ELF-related helpers and start offloading code from libbpf.c, which got pretty big already. WDYT?

Quoted 5 lines
tools/lib/bpf/libbpf.c +0 -0
index af52188daa80..cdac368c7ce1 100644

tools/lib/bpf/libbpf.c static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn) · line 10833

Quoted 10 lines
tools/lib/bpf/libbpf.c +7 -0
@@ -10824,6 +10824,109 @@ static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn)
        return NULL;
 }
 
+struct elf_symbol {
+       const char *name;
+       unsigned long offset;
+       int bind;
+};
+
+struct elf_symbol_iter {

naming nits: elf_sym and elf_sym_iter? keep it short, keep it cool :)

tools/lib/bpf/libbpf.c static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn) · line 10835

Quoted 2 lines
tools/lib/bpf/libbpf.c +2 -0
+       Elf *elf;
+       Elf_Data *symbols;

syms :-P

tools/lib/bpf/libbpf.c static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn) · line 10838

Quoted 3 lines
tools/lib/bpf/libbpf.c +3 -0
+       size_t nr_syms;
+       size_t strtabidx;
+       size_t idx;

next_sym_idx?

Quoted 45 lines
tools/lib/bpf/libbpf.c +45 -0
+       struct elf_symbol sym;
+};
+
+static int elf_symbol_iter_new(struct elf_symbol_iter *iter,
+                              Elf *elf, const char *binary_path,
+                              int sh_type)
+{
+       Elf_Scn *scn = NULL;
+       GElf_Ehdr ehdr;
+       GElf_Shdr sh;
+
+       memset(iter, 0, sizeof(*iter));
+
+       if (!gelf_getehdr(elf, &ehdr)) {
+               pr_warn("elf: failed to get ehdr from %s: %s\n", binary_path, elf_errmsg(-1));
+               return -LIBBPF_ERRNO__FORMAT;
+       }
+
+       scn = elf_find_next_scn_by_type(elf, sh_type, NULL);
+       if (!scn) {
+               pr_debug("elf: failed to find symbol table ELF sections in '%s'\n",
+                        binary_path);
+               return -EINVAL;
+       }
+
+       if (!gelf_getshdr(scn, &sh))
+               return -EINVAL;
+
+       iter->strtabidx = sh.sh_link;
+       iter->symbols = elf_getdata(scn, 0);
+       if (!iter->symbols) {
+               pr_warn("elf: failed to get symbols for symtab section in '%s': %s\n",
+                       binary_path, elf_errmsg(-1));
+               return -LIBBPF_ERRNO__FORMAT;
+       }
+       iter->nr_syms = iter->symbols->d_size / sh.sh_entsize;
+       iter->elf = elf;
+       return 0;
+}
+
+static struct elf_symbol *elf_symbol_iter_next(struct elf_symbol_iter *iter)
+{
+       struct elf_symbol *ret = &iter->sym;
+       unsigned long offset = 0;
+       const char *name = NULL;

tools/lib/bpf/libbpf.c static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn) · line 10893

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+       GElf_Shdr sym_sh;
+       Elf_Scn *sym_scn;
+       GElf_Sym sym;
+       size_t idx;
+
+       for (idx = iter->idx; idx < iter->nr_syms; idx++) {
+               if (!gelf_getsym(iter->symbols, idx, &sym))
+                       continue;
+               if (GELF_ST_TYPE(sym.st_info) != STT_FUNC)
+                       continue;

it would be more generic if this symbol type filter was a parameter to iterator, instead of hard-coding it?

Quoted 13 lines
tools/lib/bpf/libbpf.c +13 -0
+               name = elf_strptr(iter->elf, iter->strtabidx, sym.st_name);
+               if (!name)
+                       continue;
+
+               /* Transform symbol's virtual address (absolute for
+                * binaries and relative for shared libs) into file
+                * offset, which is what kernel is expecting for
+                * uprobe/uretprobe attachment.
+                * See Documentation/trace/uprobetracer.rst for more
+                * details.
+                * This is done by looking up symbol's containing
+                * section's header and using iter's virtual address
+                * (sh_addr) and corresponding file offset (sh_offset)

tools/lib/bpf/libbpf.c static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn) · line 10916

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+                * to transform sym.st_value (virtual address) into
+                * desired final file offset.
+                */
+               sym_scn = elf_getscn(iter->elf, sym.st_shndx);
+               if (!sym_scn)
+                       continue;
+               if (!gelf_getshdr(sym_scn, &sym_sh))
+                       continue;
+
+               offset = sym.st_value - sym_sh.sh_addr + sym_sh.sh_offset;

I think this part is not really generic "let's iterate ELF symbols", maybe let users of iterator do this? We can have a helper to do translation if we need to do it in few different places.

tools/lib/bpf/libbpf.c static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn) · line 10926

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+               break;
+       }
+
+       /* we reached the last symbol */
+       if (idx == iter->nr_syms)
+               return NULL;
+       iter->idx = idx + 1;
+       ret->name = name;
+       ret->bind = GELF_ST_BIND(sym.st_info);
+       ret->offset = offset;

Why not just return entire GElf_Sym information and let user process it as desired. So basically for each symbol you'll give back its name, GElf_Sym info, and I'd return symbol index as well. That will keep this very generic for future uses.

tools/lib/bpf/libbpf.c static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn) · line 10927

Quoted 1 line
tools/lib/bpf/libbpf.c +1 -0
+       return ret;

I'd structure this a bit different. If we got out of loop, just return NULL. Then inside the for loop, when we found the symbol, fill out ret and return from inside the for loop. I think it's more straightforward.

Quoted 5 lines
tools/lib/bpf/libbpf.c +2 -0
+}
+
 /* Find offset of function name in the provided ELF object. "binary_path" is
  * the path to the ELF binary represented by "elf", and only used for error
  * reporting matters. "name" matches symbol name or name@@LIB for library

[...]

#23 lore raw

Re: [PATCHv2 bpf-next 06/24] libbpf: Add elf symbol iterator

Quoted 36 lines · Andrii Nakryiko
Quoted 28 lines · Jiri Olsa

Adding elf symbol iterator object (and some functions) that follow open-coded iterator pattern and some functions to ease up iterating elf object symbols.

The idea is to iterate single symbol section with:

struct elf_symbol_iter iter;
struct elf_symbol *sym;

if (elf_symbol_iter_new(&iter, elf, binary_path, SHT_DYNSYM))

        goto error;
  while ((sym = elf_symbol_iter_next(&iter))) {
        ...
  }

I considered opening the elf inside the iterator and iterate all symbol sections, but then it gets more complicated wrt user checks for when the next section is processed.

Plus side is the we don't need 'exit' function, because caller/user is in charge of that.

The returned iterated symbol object from elf_symbol_iter_next function is placed inside the struct elf_symbol_iter, so no extra allocation or argument is needed.

Quoted 6 lines
---
 tools/lib/bpf/libbpf.c | 179 ++++++++++++++++++++++++++---------------
 1 file changed, 114 insertions(+), 65 deletions(-)
Quoted 3 lines

This is great. Left a few nits below. I'm thinkin maybe we should add a separate elf.c file for all these ELF-related helpers and start offloading code from libbpf.c, which got pretty big already. WDYT?

yes, I thought doing the move after this is merged might be better, because it's quite big already

Quoted 10 lines
Quoted 10 lines
tools/lib/bpf/libbpf.c +0 -0
index af52188daa80..cdac368c7ce1 100644
@@ -10824,6 +10824,109 @@ static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn)
        return NULL;
 }
 
Quoted 9 lines

tools/lib/bpf/libbpf.c static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn) · line 10833

Quoted 7 lines
tools/lib/bpf/libbpf.c +7 -0
+struct elf_symbol {
+       const char *name;
+       unsigned long offset;
+       int bind;
+};
+
+struct elf_symbol_iter {

naming nits: elf_sym and elf_sym_iter? keep it short, keep it cool :)

ok

Quoted 5 lines

tools/lib/bpf/libbpf.c static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn) · line 10835

Quoted 3 lines
tools/lib/bpf/libbpf.c +2 -0
+       Elf *elf;
+       Elf_Data *symbols;

syms :-P

ook ;-)

Quoted 6 lines

tools/lib/bpf/libbpf.c static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn) · line 10838

Quoted 4 lines
tools/lib/bpf/libbpf.c +3 -0
+       size_t nr_syms;
+       size_t strtabidx;
+       size_t idx;

next_sym_idx?

ok

Quoted 51 lines
Quoted 51 lines
tools/lib/bpf/libbpf.c +50 -0
+       struct elf_symbol sym;
+};
+
+static int elf_symbol_iter_new(struct elf_symbol_iter *iter,
+                              Elf *elf, const char *binary_path,
+                              int sh_type)
+{
+       Elf_Scn *scn = NULL;
+       GElf_Ehdr ehdr;
+       GElf_Shdr sh;
+
+       memset(iter, 0, sizeof(*iter));
+
+       if (!gelf_getehdr(elf, &ehdr)) {
+               pr_warn("elf: failed to get ehdr from %s: %s\n", binary_path, elf_errmsg(-1));
+               return -LIBBPF_ERRNO__FORMAT;
+       }
+
+       scn = elf_find_next_scn_by_type(elf, sh_type, NULL);
+       if (!scn) {
+               pr_debug("elf: failed to find symbol table ELF sections in '%s'\n",
+                        binary_path);
+               return -EINVAL;
+       }
+
+       if (!gelf_getshdr(scn, &sh))
+               return -EINVAL;
+
+       iter->strtabidx = sh.sh_link;
+       iter->symbols = elf_getdata(scn, 0);
+       if (!iter->symbols) {
+               pr_warn("elf: failed to get symbols for symtab section in '%s': %s\n",
+                       binary_path, elf_errmsg(-1));
+               return -LIBBPF_ERRNO__FORMAT;
+       }
+       iter->nr_syms = iter->symbols->d_size / sh.sh_entsize;
+       iter->elf = elf;
+       return 0;
+}
+
+static struct elf_symbol *elf_symbol_iter_next(struct elf_symbol_iter *iter)
+{
+       struct elf_symbol *ret = &iter->sym;
+       unsigned long offset = 0;
+       const char *name = NULL;
+       GElf_Shdr sym_sh;
+       Elf_Scn *sym_scn;
+       GElf_Sym sym;
+       size_t idx;
+
Quoted 8 lines

tools/lib/bpf/libbpf.c static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn) · line 10893

Quoted 5 lines
tools/lib/bpf/libbpf.c +5 -0
+       for (idx = iter->idx; idx < iter->nr_syms; idx++) {
+               if (!gelf_getsym(iter->symbols, idx, &sym))
+                       continue;
+               if (GELF_ST_TYPE(sym.st_info) != STT_FUNC)
+                       continue;

it would be more generic if this symbol type filter was a parameter to iterator, instead of hard-coding it?

ok

Quoted 20 lines
Quoted 20 lines
tools/lib/bpf/libbpf.c +19 -0
+               name = elf_strptr(iter->elf, iter->strtabidx, sym.st_name);
+               if (!name)
+                       continue;
+
+               /* Transform symbol's virtual address (absolute for
+                * binaries and relative for shared libs) into file
+                * offset, which is what kernel is expecting for
+                * uprobe/uretprobe attachment.
+                * See Documentation/trace/uprobetracer.rst for more
+                * details.
+                * This is done by looking up symbol's containing
+                * section's header and using iter's virtual address
+                * (sh_addr) and corresponding file offset (sh_offset)
+                * to transform sym.st_value (virtual address) into
+                * desired final file offset.
+                */
+               sym_scn = elf_getscn(iter->elf, sym.st_shndx);
+               if (!sym_scn)
+                       continue;
Quoted 8 lines

tools/lib/bpf/libbpf.c static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn) · line 10916

Quoted 4 lines
tools/lib/bpf/libbpf.c +4 -0
+               if (!gelf_getshdr(sym_scn, &sym_sh))
+                       continue;
+
+               offset = sym.st_value - sym_sh.sh_addr + sym_sh.sh_offset;

I think this part is not really generic "let's iterate ELF symbols", maybe let users of iterator do this? We can have a helper to do translation if we need to do it in few different places.

yes this will be called in all the places we use the iterator, I'll add the helper for it

Quoted 12 lines

tools/lib/bpf/libbpf.c static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn) · line 10926

Quoted 11 lines
tools/lib/bpf/libbpf.c +10 -0
+               break;
+       }
+
+       /* we reached the last symbol */
+       if (idx == iter->nr_syms)
+               return NULL;
+       iter->idx = idx + 1;
+       ret->name = name;
+       ret->bind = GELF_ST_BIND(sym.st_info);
+       ret->offset = offset;
Quoted 4 lines

Why not just return entire GElf_Sym information and let user process it as desired. So basically for each symbol you'll give back its name, GElf_Sym info, and I'd return symbol index as well. That will keep this very generic for future uses.

ok, so you have other users of this iterator in mind already?

Quoted 7 lines

tools/lib/bpf/libbpf.c static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn) · line 10927

Quoted 2 lines
tools/lib/bpf/libbpf.c +1 -0
+       return ret;

I'd structure this a bit different. If we got out of loop, just return NULL. Then inside the for loop, when we found the symbol, fill out ret and return from inside the for loop. I think it's more straightforward.

ok, will change

thanks,
jirka

Quoted 8 lines
Quoted 6 lines
tools/lib/bpf/libbpf.c +2 -0
+}
+
 /* Find offset of function name in the provided ELF object. "binary_path" is
  * the path to the ELF binary represented by "elf", and only used for error
  * reporting matters. "name" matches symbol name or name@@LIB for library

[...]

#24 lore raw

Re: [PATCHv2 bpf-next 06/24] libbpf: Add elf symbol iterator

Quoted 190 lines · Jiri Olsa
Quoted 36 lines · Andrii Nakryiko
Quoted 28 lines · Jiri Olsa

Adding elf symbol iterator object (and some functions) that follow open-coded iterator pattern and some functions to ease up iterating elf object symbols.

The idea is to iterate single symbol section with:

struct elf_symbol_iter iter;
struct elf_symbol *sym;

if (elf_symbol_iter_new(&iter, elf, binary_path, SHT_DYNSYM))

        goto error;
  while ((sym = elf_symbol_iter_next(&iter))) {
        ...
  }

I considered opening the elf inside the iterator and iterate all symbol sections, but then it gets more complicated wrt user checks for when the next section is processed.

Plus side is the we don't need 'exit' function, because caller/user is in charge of that.

The returned iterated symbol object from elf_symbol_iter_next function is placed inside the struct elf_symbol_iter, so no extra allocation or argument is needed.

Quoted 6 lines
---
 tools/lib/bpf/libbpf.c | 179 ++++++++++++++++++++++++++---------------
 1 file changed, 114 insertions(+), 65 deletions(-)
Quoted 3 lines

This is great. Left a few nits below. I'm thinkin maybe we should add a separate elf.c file for all these ELF-related helpers and start offloading code from libbpf.c, which got pretty big already. WDYT?

yes, I thought doing the move after this is merged might be better, because it's quite big already

Quoted 10 lines
Quoted 10 lines
tools/lib/bpf/libbpf.c +0 -0
index af52188daa80..cdac368c7ce1 100644
@@ -10824,6 +10824,109 @@ static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn)
        return NULL;
 }
 
Quoted 9 lines

tools/lib/bpf/libbpf.c static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn) · line 10833

Quoted 7 lines
tools/lib/bpf/libbpf.c +7 -0
+struct elf_symbol {
+       const char *name;
+       unsigned long offset;
+       int bind;
+};
+
+struct elf_symbol_iter {

naming nits: elf_sym and elf_sym_iter? keep it short, keep it cool :)

ok

Quoted 5 lines

tools/lib/bpf/libbpf.c static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn) · line 10835

Quoted 3 lines
tools/lib/bpf/libbpf.c +2 -0
+       Elf *elf;
+       Elf_Data *symbols;

syms :-P

ook ;-)

Quoted 6 lines

tools/lib/bpf/libbpf.c static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn) · line 10838

Quoted 4 lines
tools/lib/bpf/libbpf.c +3 -0
+       size_t nr_syms;
+       size_t strtabidx;
+       size_t idx;

next_sym_idx?

ok

Quoted 51 lines
Quoted 51 lines
tools/lib/bpf/libbpf.c +50 -0
+       struct elf_symbol sym;
+};
+
+static int elf_symbol_iter_new(struct elf_symbol_iter *iter,
+                              Elf *elf, const char *binary_path,
+                              int sh_type)
+{
+       Elf_Scn *scn = NULL;
+       GElf_Ehdr ehdr;
+       GElf_Shdr sh;
+
+       memset(iter, 0, sizeof(*iter));
+
+       if (!gelf_getehdr(elf, &ehdr)) {
+               pr_warn("elf: failed to get ehdr from %s: %s\n", binary_path, elf_errmsg(-1));
+               return -LIBBPF_ERRNO__FORMAT;
+       }
+
+       scn = elf_find_next_scn_by_type(elf, sh_type, NULL);
+       if (!scn) {
+               pr_debug("elf: failed to find symbol table ELF sections in '%s'\n",
+                        binary_path);
+               return -EINVAL;
+       }
+
+       if (!gelf_getshdr(scn, &sh))
+               return -EINVAL;
+
+       iter->strtabidx = sh.sh_link;
+       iter->symbols = elf_getdata(scn, 0);
+       if (!iter->symbols) {
+               pr_warn("elf: failed to get symbols for symtab section in '%s': %s\n",
+                       binary_path, elf_errmsg(-1));
+               return -LIBBPF_ERRNO__FORMAT;
+       }
+       iter->nr_syms = iter->symbols->d_size / sh.sh_entsize;
+       iter->elf = elf;
+       return 0;
+}
+
+static struct elf_symbol *elf_symbol_iter_next(struct elf_symbol_iter *iter)
+{
+       struct elf_symbol *ret = &iter->sym;
+       unsigned long offset = 0;
+       const char *name = NULL;
+       GElf_Shdr sym_sh;
+       Elf_Scn *sym_scn;
+       GElf_Sym sym;
+       size_t idx;
+
Quoted 8 lines

tools/lib/bpf/libbpf.c static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn) · line 10893

Quoted 5 lines
tools/lib/bpf/libbpf.c +5 -0
+       for (idx = iter->idx; idx < iter->nr_syms; idx++) {
+               if (!gelf_getsym(iter->symbols, idx, &sym))
+                       continue;
+               if (GELF_ST_TYPE(sym.st_info) != STT_FUNC)
+                       continue;

it would be more generic if this symbol type filter was a parameter to iterator, instead of hard-coding it?

ok

Quoted 20 lines
Quoted 20 lines
tools/lib/bpf/libbpf.c +19 -0
+               name = elf_strptr(iter->elf, iter->strtabidx, sym.st_name);
+               if (!name)
+                       continue;
+
+               /* Transform symbol's virtual address (absolute for
+                * binaries and relative for shared libs) into file
+                * offset, which is what kernel is expecting for
+                * uprobe/uretprobe attachment.
+                * See Documentation/trace/uprobetracer.rst for more
+                * details.
+                * This is done by looking up symbol's containing
+                * section's header and using iter's virtual address
+                * (sh_addr) and corresponding file offset (sh_offset)
+                * to transform sym.st_value (virtual address) into
+                * desired final file offset.
+                */
+               sym_scn = elf_getscn(iter->elf, sym.st_shndx);
+               if (!sym_scn)
+                       continue;
Quoted 8 lines

tools/lib/bpf/libbpf.c static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn) · line 10916

Quoted 4 lines
tools/lib/bpf/libbpf.c +4 -0
+               if (!gelf_getshdr(sym_scn, &sym_sh))
+                       continue;
+
+               offset = sym.st_value - sym_sh.sh_addr + sym_sh.sh_offset;

I think this part is not really generic "let's iterate ELF symbols", maybe let users of iterator do this? We can have a helper to do translation if we need to do it in few different places.

yes this will be called in all the places we use the iterator, I'll add the helper for it

Quoted 12 lines
Quoted 11 lines
tools/lib/bpf/libbpf.c +10 -0
+               break;
+       }
+
+       /* we reached the last symbol */
+       if (idx == iter->nr_syms)
+               return NULL;
+       iter->idx = idx + 1;
+       ret->name = name;
+       ret->bind = GELF_ST_BIND(sym.st_info);
+       ret->offset = offset;
Quoted 6 lines
Quoted 4 lines

Why not just return entire GElf_Sym information and let user process it as desired. So basically for each symbol you'll give back its name, GElf_Sym info, and I'd return symbol index as well. That will keep this very generic for future uses.

ok, so you have other users of this iterator in mind already?

well, there is linker.c that also iterates ELF symbols, though that one is assuming Elf64_Sym, so I wouldn't go updating it. So it's more of a general feeling that "ELF symbol iterator" shouldn't assume functions and func_offset translation, it should just return symbols.

Quoted 22 lines
Quoted 8 lines

tools/lib/bpf/libbpf.c static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn) · line 10927

Quoted 3 lines
tools/lib/bpf/libbpf.c +1 -0
+       return ret;

I'd structure this a bit different. If we got out of loop, just return NULL. Then inside the for loop, when we found the symbol, fill out ret and return from inside the for loop. I think it's more straightforward.

ok, will change

thanks,
jirka

Quoted 8 lines
Quoted 6 lines
tools/lib/bpf/libbpf.c +2 -0
+}
+
 /* Find offset of function name in the provided ELF object. "binary_path" is
  * the path to the ELF binary represented by "elf", and only used for error
  * reporting matters. "name" matches symbol name or name@@LIB for library

[...]

#25 lore raw

Re: [PATCHv2 bpf-next 06/24] libbpf: Add elf symbol iterator

In reply to Jiri Olsa
Quoted 38 lines · Jiri Olsa
Quoted 36 lines · Andrii Nakryiko
Quoted 34 lines · Jiri Olsa

Adding elf symbol iterator object (and some functions) that follow open-coded iterator pattern and some functions to ease up iterating elf object symbols.

The idea is to iterate single symbol section with:

struct elf_symbol_iter iter;
struct elf_symbol *sym;

if (elf_symbol_iter_new(&iter, elf, binary_path, SHT_DYNSYM))

        goto error;
  while ((sym = elf_symbol_iter_next(&iter))) {
        ...
  }

I considered opening the elf inside the iterator and iterate all symbol sections, but then it gets more complicated wrt user checks for when the next section is processed.

Plus side is the we don't need 'exit' function, because caller/user is in charge of that.

The returned iterated symbol object from elf_symbol_iter_next function is placed inside the struct elf_symbol_iter, so no extra allocation or argument is needed.

---
 tools/lib/bpf/libbpf.c | 179 ++++++++++++++++++++++++++---------------
 1 file changed, 114 insertions(+), 65 deletions(-)
Quoted 6 lines
Quoted 3 lines

This is great. Left a few nits below. I'm thinkin maybe we should add a separate elf.c file for all these ELF-related helpers and start offloading code from libbpf.c, which got pretty big already. WDYT?

yes, I thought doing the move after this is merged might be better, because it's quite big already

true (and btw, please give me a bit more time to review the rest of patches before posting a new version), but I'm ok with just going straight to elf.c move. Either way it's a lot of +++ and ---, whether it's in the same file or not doesn't matter all that much.

Quoted 19 lines
Quoted 19 lines
Quoted 18 lines
tools/lib/bpf/libbpf.c +7 -0
index af52188daa80..cdac368c7ce1 100644
@@ -10824,6 +10824,109 @@ static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn)
        return NULL;
 }
 
+struct elf_symbol {
+       const char *name;
+       unsigned long offset;
+       int bind;
+};
+
+struct elf_symbol_iter {

[...]

author reviewer
#26 lore raw

PATCH v2 7/24 libbpf: Add open_elf/close_elf functions

In reply to Jiri Olsa

Adding open_elf/close_elf functions and using it in elf_find_func_offset_from_file function. It will be used in following changes to save some code.

---
 tools/lib/bpf/libbpf.c | 62 ++++++++++++++++++++++++++++++------------
 1 file changed, 44 insertions(+), 18 deletions(-)
tools/lib/bpf/libbpf.c +44 -18
index cdac368c7ce1..30d9e3b69114 100644
@@ -10927,6 +10927,45 @@ static struct elf_symbol *elf_symbol_iter_next(struct elf_symbol_iter *iter)
 	return ret;
 }
 
+struct elf_fd {
+	Elf *elf;
+	int fd;
+};
+
+static int open_elf(const char *binary_path, struct elf_fd *elf_fd)
+{
+	char errmsg[STRERR_BUFSIZE];
+	int fd, ret;
+	Elf *elf;
+
+	if (elf_version(EV_CURRENT) == EV_NONE) {
+		pr_warn("failed to init libelf for %s\n", binary_path);
+		return -LIBBPF_ERRNO__LIBELF;
+	}
+	fd = open(binary_path, O_RDONLY | O_CLOEXEC);
+	if (fd < 0) {
+		ret = -errno;
+		pr_warn("failed to open %s: %s\n", binary_path,
+			libbpf_strerror_r(ret, errmsg, sizeof(errmsg)));
+		return ret;
+	}
+	elf = elf_begin(fd, ELF_C_READ_MMAP, NULL);
+	if (!elf) {
+		pr_warn("elf: could not read elf from %s: %s\n", binary_path, elf_errmsg(-1));
+		close(fd);
+		return -LIBBPF_ERRNO__FORMAT;
+	}
+	elf_fd->fd = fd;
+	elf_fd->elf = elf;
+	return 0;
+}
+
+static void close_elf(struct elf_fd *elf_fd)
+{
+	elf_end(elf_fd->elf);
+	close(elf_fd->fd);
+}
+
 /* Find offset of function name in the provided ELF object. "binary_path" is
  * the path to the ELF binary represented by "elf", and only used for error
  * reporting matters. "name" matches symbol name or name@@LIB for library
@@ -11019,28 +11058,15 @@ static long elf_find_func_offset(Elf *elf, const char *binary_path, const char *
  */
 static long elf_find_func_offset_from_file(const char *binary_path, const char *name)
 {
-	char errmsg[STRERR_BUFSIZE];
+	struct elf_fd elf_fd = {};
 	long ret = -ENOENT;
-	Elf *elf;
-	int fd;
 
-	fd = open(binary_path, O_RDONLY | O_CLOEXEC);
-	if (fd < 0) {
-		ret = -errno;
-		pr_warn("failed to open %s: %s\n", binary_path,
-			libbpf_strerror_r(ret, errmsg, sizeof(errmsg)));
+	ret = open_elf(binary_path, &elf_fd);
+	if (ret)
 		return ret;
-	}
-	elf = elf_begin(fd, ELF_C_READ_MMAP, NULL);
-	if (!elf) {
-		pr_warn("elf: could not read elf from %s: %s\n", binary_path, elf_errmsg(-1));
-		close(fd);
-		return -LIBBPF_ERRNO__FORMAT;
-	}
 
-	ret = elf_find_func_offset(elf, binary_path, name);
-	elf_end(elf);
-	close(fd);
+	ret = elf_find_func_offset(elf_fd.elf, binary_path, name);
+	close_elf(&elf_fd);
 	return ret;
 }
 
Signature · 1 line
2.41.0
#27 lore raw

Re: [PATCHv2 bpf-next 07/24] libbpf: Add open_elf/close_elf functions

Quoted 10 lines · Jiri Olsa

Adding open_elf/close_elf functions and using it in elf_find_func_offset_from_file function. It will be used in following changes to save some code.

---
 tools/lib/bpf/libbpf.c | 62 ++++++++++++++++++++++++++++++------------
 1 file changed, 44 insertions(+), 18 deletions(-)

we should definitely move all this into separate elf.c file

Quoted 18 lines
tools/lib/bpf/libbpf.c +10 -0
index cdac368c7ce1..30d9e3b69114 100644
@@ -10927,6 +10927,45 @@ static struct elf_symbol *elf_symbol_iter_next(struct elf_symbol_iter *iter)
        return ret;
 }
 
+struct elf_fd {
+       Elf *elf;
+       int fd;
+};
+
+static int open_elf(const char *binary_path, struct elf_fd *elf_fd)
+{
+       char errmsg[STRERR_BUFSIZE];
+       int fd, ret;
+       Elf *elf;

tools/lib/bpf/libbpf.c static struct elf_symbol *elf_symbol_iter_next(struct elf_symbol_iter *iter) · line 10949

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+
+       if (elf_version(EV_CURRENT) == EV_NONE) {
+               pr_warn("failed to init libelf for %s\n", binary_path);
+               return -LIBBPF_ERRNO__LIBELF;
+       }
+       fd = open(binary_path, O_RDONLY | O_CLOEXEC);
+       if (fd < 0) {
+               ret = -errno;
+               pr_warn("failed to open %s: %s\n", binary_path,
+                       libbpf_strerror_r(ret, errmsg, sizeof(errmsg)));

let's add "elf: " prefix for consistency?

Quoted 59 lines
tools/lib/bpf/libbpf.c +24 -19
+               return ret;
+       }
+       elf = elf_begin(fd, ELF_C_READ_MMAP, NULL);
+       if (!elf) {
+               pr_warn("elf: could not read elf from %s: %s\n", binary_path, elf_errmsg(-1));
+               close(fd);
+               return -LIBBPF_ERRNO__FORMAT;
+       }
+       elf_fd->fd = fd;
+       elf_fd->elf = elf;
+       return 0;
+}
+
+static void close_elf(struct elf_fd *elf_fd)
+{
+       elf_end(elf_fd->elf);
+       close(elf_fd->fd);
+}
+
 /* Find offset of function name in the provided ELF object. "binary_path" is
  * the path to the ELF binary represented by "elf", and only used for error
  * reporting matters. "name" matches symbol name or name@@LIB for library
@@ -11019,28 +11058,15 @@ static long elf_find_func_offset(Elf *elf, const char *binary_path, const char *
  */
 static long elf_find_func_offset_from_file(const char *binary_path, const char *name)
 {
-       char errmsg[STRERR_BUFSIZE];
+       struct elf_fd elf_fd = {};
        long ret = -ENOENT;
-       Elf *elf;
-       int fd;
 
-       fd = open(binary_path, O_RDONLY | O_CLOEXEC);
-       if (fd < 0) {
-               ret = -errno;
-               pr_warn("failed to open %s: %s\n", binary_path,
-                       libbpf_strerror_r(ret, errmsg, sizeof(errmsg)));
+       ret = open_elf(binary_path, &elf_fd);
+       if (ret)
                return ret;
-       }
-       elf = elf_begin(fd, ELF_C_READ_MMAP, NULL);
-       if (!elf) {
-               pr_warn("elf: could not read elf from %s: %s\n", binary_path, elf_errmsg(-1));
-               close(fd);
-               return -LIBBPF_ERRNO__FORMAT;
-       }
 
-       ret = elf_find_func_offset(elf, binary_path, name);
-       elf_end(elf);
-       close(fd);
+       ret = elf_find_func_offset(elf_fd.elf, binary_path, name);
+       close_elf(&elf_fd);
        return ret;
 }
 
--

2.41.0

#28 lore raw

Re: [PATCHv2 bpf-next 07/24] libbpf: Add open_elf/close_elf functions

Quoted 6 lines · Andrii Nakryiko
Quoted 5 lines · Jiri Olsa

Adding open_elf/close_elf functions and using it in elf_find_func_offset_from_file function. It will be used in following changes to save some code.

Quoted 7 lines
Quoted 5 lines
---
 tools/lib/bpf/libbpf.c | 62 ++++++++++++++++++++++++++++++------------
 1 file changed, 44 insertions(+), 18 deletions(-)

we should definitely move all this into separate elf.c file

right, also we could use this in usdt_manager_attach_usdt as well

Quoted 23 lines
Quoted 23 lines
tools/lib/bpf/libbpf.c +14 -0
index cdac368c7ce1..30d9e3b69114 100644
@@ -10927,6 +10927,45 @@ static struct elf_symbol *elf_symbol_iter_next(struct elf_symbol_iter *iter)
        return ret;
 }
 
+struct elf_fd {
+       Elf *elf;
+       int fd;
+};
+
+static int open_elf(const char *binary_path, struct elf_fd *elf_fd)
+{
+       char errmsg[STRERR_BUFSIZE];
+       int fd, ret;
+       Elf *elf;
+
+       if (elf_version(EV_CURRENT) == EV_NONE) {
+               pr_warn("failed to init libelf for %s\n", binary_path);
+               return -LIBBPF_ERRNO__LIBELF;
Quoted 8 lines

tools/lib/bpf/libbpf.c static struct elf_symbol *elf_symbol_iter_next(struct elf_symbol_iter *iter) · line 10949

Quoted 6 lines
tools/lib/bpf/libbpf.c +6 -0
+       }
+       fd = open(binary_path, O_RDONLY | O_CLOEXEC);
+       if (fd < 0) {
+               ret = -errno;
+               pr_warn("failed to open %s: %s\n", binary_path,
+                       libbpf_strerror_r(ret, errmsg, sizeof(errmsg)));

let's add "elf: " prefix for consistency?

ok

jirka

author reviewer
#29 lore raw

PATCH v2 8/24 libbpf: Add elf_find_multi_func_offset function

In reply to Jiri Olsa

Adding elf_find_multi_func_offset function that looks up offsets for symbols specified in syms array argument.

Offsets are returned in allocated array with the 'cnt' size, that needs to be released by the caller.

---
 tools/lib/bpf/libbpf.c          | 112 ++++++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf_internal.h |   2 +
 2 files changed, 114 insertions(+)
tools/lib/bpf/libbpf.c +112 -0
index 30d9e3b69114..1c310b718961 100644
@@ -11053,6 +11053,118 @@ static long elf_find_func_offset(Elf *elf, const char *binary_path, const char *
 	return ret;
 }
 
+struct elf_symbol_offset {
+	const char *name;
+	unsigned long offset;
+	int bind;
+	int idx;
+};
+
+static int cmp_func_offset(const void *_a, const void *_b)
+{
+	const struct elf_symbol_offset *a = _a;
+	const struct elf_symbol_offset *b = _b;
+
+	return strcmp(a->name, b->name);
+}
+
+static int
+__elf_find_multi_func_offset(Elf *elf, const char *binary_path, int cnt,
+			     const char **syms, unsigned long **poffsets)
+{
+	int sh_types[2] = { SHT_DYNSYM, SHT_SYMTAB };
+	struct elf_symbol_offset *func_offs;
+	int err = 0, i, idx, cnt_done = 0;
+	unsigned long *offsets = NULL;
+
+	func_offs = calloc(cnt, sizeof(*func_offs));
+	if (!func_offs)
+		return -ENOMEM;
+
+	for (i = 0; i < cnt; i++) {
+		func_offs[i].name = syms[i];
+		func_offs[i].idx = i;
+	}
+
+	qsort(func_offs, cnt, sizeof(*func_offs), cmp_func_offset);
+
+	for (i = 0; i < ARRAY_SIZE(sh_types); i++) {
+		struct elf_symbol_iter iter;
+		struct elf_symbol *sym;
+
+		if (elf_symbol_iter_new(&iter, elf, binary_path, sh_types[i]))
+			continue;
+
+		while ((sym = elf_symbol_iter_next(&iter))) {
+			struct elf_symbol_offset *fo, tmp = {
+				.name = sym->name,
+			};
+
+			fo = bsearch(&tmp, func_offs, cnt, sizeof(*func_offs),
+				     cmp_func_offset);
+			if (!fo)
+				continue;
+
+			if (fo->offset > 0) {
+				/* same offset, no problem */
+				if (fo->offset == sym->offset)
+					continue;
+				/* handle multiple matches */
+				if (fo->bind != STB_WEAK && sym->bind != STB_WEAK) {
+					/* Only accept one non-weak bind. */
+					pr_warn("elf: ambiguous match for '%s', '%s' in '%s'\n",
+						sym->name, fo->name, binary_path);
+					err = -LIBBPF_ERRNO__FORMAT;
+					goto out;
+				} else if (sym->bind == STB_WEAK) {
+					/* already have a non-weak bind, and
+					 * this is a weak bind, so ignore.
+					 */
+					continue;
+				}
+			}
+			if (!fo->offset)
+				cnt_done++;
+			fo->offset = sym->offset;
+			fo->bind = sym->bind;
+		}
+	}
+
+	if (cnt != cnt_done) {
+		err = -ENOENT;
+		goto out;
+	}
+	offsets = calloc(cnt, sizeof(*offsets));
+	if (!offsets) {
+		err = -ENOMEM;
+		goto out;
+	}
+	for (i = 0; i < cnt; i++) {
+		idx = func_offs[i].idx;
+		offsets[idx] = func_offs[i].offset;
+	}
+
+out:
+	*poffsets = offsets;
+	free(func_offs);
+	return err;
+}
+
+int elf_find_multi_func_offset(const char *binary_path, int cnt,
+			       const char **syms, unsigned long **poffsets)
+{
+	struct elf_fd elf_fd = {};
+	long ret = -ENOENT;
+
+	ret = open_elf(binary_path, &elf_fd);
+	if (ret)
+		return ret;
+
+	ret = __elf_find_multi_func_offset(elf_fd.elf, binary_path, cnt, syms, poffsets);
+	close_elf(&elf_fd);
+	return ret;
+}
+
 /* Find offset of function name in ELF object specified by path. "name" matches
  * symbol name or name@@LIB for library functions.
  */
tools/lib/bpf/libbpf_internal.h +2 -0
index e4d05662a96c..13d5c12fbd0b 100644
@@ -577,4 +577,6 @@ static inline bool is_pow_of_2(size_t x)
 #define PROG_LOAD_ATTEMPTS 5
 int sys_bpf_prog_load(union bpf_attr *attr, unsigned int size, int attempts);
 
+int elf_find_multi_func_offset(const char *binary_path, int cnt,
+			       const char **syms, unsigned long **poffsets);
 #endif /* __LIBBPF_LIBBPF_INTERNAL_H */
Signature · 1 line
2.41.0
#30 lore raw

Re: [PATCHv2 bpf-next 08/24] libbpf: Add elf_find_multi_func_offset function

Quoted 51 lines · Jiri Olsa

Adding elf_find_multi_func_offset function that looks up offsets for symbols specified in syms array argument.

Offsets are returned in allocated array with the 'cnt' size, that needs to be released by the caller.

---
 tools/lib/bpf/libbpf.c          | 112 ++++++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf_internal.h |   2 +
 2 files changed, 114 insertions(+)
tools/lib/bpf/libbpf.c +30 -0
index 30d9e3b69114..1c310b718961 100644
@@ -11053,6 +11053,118 @@ static long elf_find_func_offset(Elf *elf, const char *binary_path, const char *
        return ret;
 }
 
+struct elf_symbol_offset {
+       const char *name;
+       unsigned long offset;
+       int bind;
+       int idx;
+};
+
+static int cmp_func_offset(const void *_a, const void *_b)
+{
+       const struct elf_symbol_offset *a = _a;
+       const struct elf_symbol_offset *b = _b;
+
+       return strcmp(a->name, b->name);
+}
+
+static int
+__elf_find_multi_func_offset(Elf *elf, const char *binary_path, int cnt,
+                            const char **syms, unsigned long **poffsets)
+{
+       int sh_types[2] = { SHT_DYNSYM, SHT_SYMTAB };
+       struct elf_symbol_offset *func_offs;
+       int err = 0, i, idx, cnt_done = 0;
+       unsigned long *offsets = NULL;
+
+       func_offs = calloc(cnt, sizeof(*func_offs));
+       if (!func_offs)
+               return -ENOMEM;
+
+       for (i = 0; i < cnt; i++) {
+               func_offs[i].name = syms[i];

tools/lib/bpf/libbpf.c static long elf_find_func_offset(Elf *elf, const char *binary_path, const char * · line 11095

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+               func_offs[i].idx = i;
+       }
+
+       qsort(func_offs, cnt, sizeof(*func_offs), cmp_func_offset);
+
+       for (i = 0; i < ARRAY_SIZE(sh_types); i++) {
+               struct elf_symbol_iter iter;
+               struct elf_symbol *sym;
+
+               if (elf_symbol_iter_new(&iter, elf, binary_path, sh_types[i]))

a bit lax handling of initialization errors here, let's be a bit more strict here?

Quoted 32 lines
tools/lib/bpf/libbpf.c +32 -0
+                       continue;
+
+               while ((sym = elf_symbol_iter_next(&iter))) {
+                       struct elf_symbol_offset *fo, tmp = {
+                               .name = sym->name,
+                       };
+
+                       fo = bsearch(&tmp, func_offs, cnt, sizeof(*func_offs),
+                                    cmp_func_offset);
+                       if (!fo)
+                               continue;
+
+                       if (fo->offset > 0) {
+                               /* same offset, no problem */
+                               if (fo->offset == sym->offset)
+                                       continue;
+                               /* handle multiple matches */
+                               if (fo->bind != STB_WEAK && sym->bind != STB_WEAK) {
+                                       /* Only accept one non-weak bind. */
+                                       pr_warn("elf: ambiguous match for '%s', '%s' in '%s'\n",
+                                               sym->name, fo->name, binary_path);
+                                       err = -LIBBPF_ERRNO__FORMAT;
+                                       goto out;
+                               } else if (sym->bind == STB_WEAK) {
+                                       /* already have a non-weak bind, and
+                                        * this is a weak bind, so ignore.
+                                        */
+                                       continue;
+                               }
+                       }
+                       if (!fo->offset)
+                               cnt_done++;

tools/lib/bpf/libbpf.c static long elf_find_func_offset(Elf *elf, const char *binary_path, const char * · line 11137

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+                       fo->offset = sym->offset;
+                       fo->bind = sym->bind;
+               }
+       }
+
+       if (cnt != cnt_done) {
+               err = -ENOENT;
+               goto out;
+       }
+       offsets = calloc(cnt, sizeof(*offsets));

you can allocate it at the very beginning and fill it out based on fo->idx, there is no need to store offset in elf_symbol_offset

Quoted 9 lines
tools/lib/bpf/libbpf.c +9 -0
+       if (!offsets) {
+               err = -ENOMEM;
+               goto out;
+       }
+       for (i = 0; i < cnt; i++) {
+               idx = func_offs[i].idx;
+               offsets[idx] = func_offs[i].offset;
+       }
+

tools/lib/bpf/libbpf.c static long elf_find_func_offset(Elf *elf, const char *binary_path, const char * · line 11156

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+out:
+       *poffsets = offsets;
+       free(func_offs);
+       return err;
+}
+
+int elf_find_multi_func_offset(const char *binary_path, int cnt,
+                              const char **syms, unsigned long **poffsets)
+{
+       struct elf_fd elf_fd = {};

do you need to initialize this struct?

tools/lib/bpf/libbpf.c static long elf_find_func_offset(Elf *elf, const char *binary_path, const char * · line 11157

Quoted 1 line
tools/lib/bpf/libbpf.c +1 -0
+       long ret = -ENOENT;

same here, you always override ret, so no need to init it?

tools/lib/bpf/libbpf.c static long elf_find_func_offset(Elf *elf, const char *binary_path, const char * · line 11163

Quoted 6 lines
tools/lib/bpf/libbpf.c +6 -0
+
+       ret = open_elf(binary_path, &elf_fd);
+       if (ret)
+               return ret;
+
+       ret = __elf_find_multi_func_offset(elf_fd.elf, binary_path, cnt, syms, poffsets);

is there a point of having elf_find_multi_func_offset and __elf_find_multi_func_offset separately? can you please combine?

Quoted 21 lines
tools/lib/bpf/libbpf.c +4 -0
+       close_elf(&elf_fd);
+       return ret;
+}
+
 /* Find offset of function name in ELF object specified by path. "name" matches
  * symbol name or name@@LIB for library functions.
  */
tools/lib/bpf/libbpf_internal.h +2 -1
index e4d05662a96c..13d5c12fbd0b 100644
@@ -577,4 +577,6 @@ static inline bool is_pow_of_2(size_t x)
 #define PROG_LOAD_ATTEMPTS 5
 int sys_bpf_prog_load(union bpf_attr *attr, unsigned int size, int attempts);
 
+int elf_find_multi_func_offset(const char *binary_path, int cnt,
+                              const char **syms, unsigned long **poffsets);
 #endif /* __LIBBPF_LIBBPF_INTERNAL_H */
--

2.41.0

#31 lore raw

Re: [PATCHv2 bpf-next 08/24] libbpf: Add elf_find_multi_func_offset function

Quoted 57 lines · Andrii Nakryiko
Quoted 56 lines · Jiri Olsa

Adding elf_find_multi_func_offset function that looks up offsets for symbols specified in syms array argument.

Offsets are returned in allocated array with the 'cnt' size, that needs to be released by the caller.

---
 tools/lib/bpf/libbpf.c          | 112 ++++++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf_internal.h |   2 +
 2 files changed, 114 insertions(+)
tools/lib/bpf/libbpf.c +35 -0
index 30d9e3b69114..1c310b718961 100644
@@ -11053,6 +11053,118 @@ static long elf_find_func_offset(Elf *elf, const char *binary_path, const char *
        return ret;
 }
 
+struct elf_symbol_offset {
+       const char *name;
+       unsigned long offset;
+       int bind;
+       int idx;
+};
+
+static int cmp_func_offset(const void *_a, const void *_b)
+{
+       const struct elf_symbol_offset *a = _a;
+       const struct elf_symbol_offset *b = _b;
+
+       return strcmp(a->name, b->name);
+}
+
+static int
+__elf_find_multi_func_offset(Elf *elf, const char *binary_path, int cnt,
+                            const char **syms, unsigned long **poffsets)
+{
+       int sh_types[2] = { SHT_DYNSYM, SHT_SYMTAB };
+       struct elf_symbol_offset *func_offs;
+       int err = 0, i, idx, cnt_done = 0;
+       unsigned long *offsets = NULL;
+
+       func_offs = calloc(cnt, sizeof(*func_offs));
+       if (!func_offs)
+               return -ENOMEM;
+
+       for (i = 0; i < cnt; i++) {
+               func_offs[i].name = syms[i];
+               func_offs[i].idx = i;
+       }
+
+       qsort(func_offs, cnt, sizeof(*func_offs), cmp_func_offset);
+
Quoted 8 lines

tools/lib/bpf/libbpf.c static long elf_find_func_offset(Elf *elf, const char *binary_path, const char * · line 11095

Quoted 5 lines
tools/lib/bpf/libbpf.c +5 -0
+       for (i = 0; i < ARRAY_SIZE(sh_types); i++) {
+               struct elf_symbol_iter iter;
+               struct elf_symbol *sym;
+
+               if (elf_symbol_iter_new(&iter, elf, binary_path, sh_types[i]))

a bit lax handling of initialization errors here, let's be a bit more strict here?

right, I'll have elf_symbol_iter_new return -ENOENT in case there's no 'sh_types[i]' section type (and we want to try next section type) and fail otherwise

Quoted 38 lines
Quoted 38 lines
tools/lib/bpf/libbpf.c +37 -0
+                       continue;
+
+               while ((sym = elf_symbol_iter_next(&iter))) {
+                       struct elf_symbol_offset *fo, tmp = {
+                               .name = sym->name,
+                       };
+
+                       fo = bsearch(&tmp, func_offs, cnt, sizeof(*func_offs),
+                                    cmp_func_offset);
+                       if (!fo)
+                               continue;
+
+                       if (fo->offset > 0) {
+                               /* same offset, no problem */
+                               if (fo->offset == sym->offset)
+                                       continue;
+                               /* handle multiple matches */
+                               if (fo->bind != STB_WEAK && sym->bind != STB_WEAK) {
+                                       /* Only accept one non-weak bind. */
+                                       pr_warn("elf: ambiguous match for '%s', '%s' in '%s'\n",
+                                               sym->name, fo->name, binary_path);
+                                       err = -LIBBPF_ERRNO__FORMAT;
+                                       goto out;
+                               } else if (sym->bind == STB_WEAK) {
+                                       /* already have a non-weak bind, and
+                                        * this is a weak bind, so ignore.
+                                        */
+                                       continue;
+                               }
+                       }
+                       if (!fo->offset)
+                               cnt_done++;
+                       fo->offset = sym->offset;
+                       fo->bind = sym->bind;
+               }
+       }
+
Quoted 8 lines

tools/lib/bpf/libbpf.c static long elf_find_func_offset(Elf *elf, const char *binary_path, const char * · line 11137

Quoted 5 lines
tools/lib/bpf/libbpf.c +5 -0
+       if (cnt != cnt_done) {
+               err = -ENOENT;
+               goto out;
+       }
+       offsets = calloc(cnt, sizeof(*offsets));

you can allocate it at the very beginning and fill it out based on fo->idx, there is no need to store offset in elf_symbol_offset

true, will fix

Quoted 14 lines
Quoted 14 lines
tools/lib/bpf/libbpf.c +13 -0
+       if (!offsets) {
+               err = -ENOMEM;
+               goto out;
+       }
+       for (i = 0; i < cnt; i++) {
+               idx = func_offs[i].idx;
+               offsets[idx] = func_offs[i].offset;
+       }
+
+out:
+       *poffsets = offsets;
+       free(func_offs);
+       return err;
Quoted 8 lines

tools/lib/bpf/libbpf.c static long elf_find_func_offset(Elf *elf, const char *binary_path, const char * · line 11156

Quoted 6 lines
tools/lib/bpf/libbpf.c +6 -0
+}
+
+int elf_find_multi_func_offset(const char *binary_path, int cnt,
+                              const char **syms, unsigned long **poffsets)
+{
+       struct elf_fd elf_fd = {};

do you need to initialize this struct?

I recall getting gcc warn/error, which I can't see anymore ;-) will remove

Quoted 4 lines

tools/lib/bpf/libbpf.c static long elf_find_func_offset(Elf *elf, const char *binary_path, const char * · line 11157

Quoted 2 lines
tools/lib/bpf/libbpf.c +1 -0
+       long ret = -ENOENT;

same here, you always override ret, so no need to init it?

yes

Quoted 10 lines

tools/lib/bpf/libbpf.c static long elf_find_func_offset(Elf *elf, const char *binary_path, const char * · line 11163

Quoted 7 lines
tools/lib/bpf/libbpf.c +6 -0
+
+       ret = open_elf(binary_path, &elf_fd);
+       if (ret)
+               return ret;
+
+       ret = __elf_find_multi_func_offset(elf_fd.elf, binary_path, cnt, syms, poffsets);

is there a point of having elf_find_multi_func_offset and __elf_find_multi_func_offset separately? can you please combine?

ok

thanks,
jirka

author reviewer
#32 lore raw

PATCH v2 9/24 libbpf: Add elf_find_pattern_func_offset function

In reply to Jiri Olsa

Adding elf_find_pattern_func_offset function that looks up offsets for symbols specified by pattern argument.

The 'pattern' argument allows wildcards (*?' supported).

Offsets are returned in allocated array together with its size and needs to be released by the caller.

---
 tools/lib/bpf/libbpf.c          | 78 +++++++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf_internal.h |  3 ++
 2 files changed, 81 insertions(+)
tools/lib/bpf/libbpf.c +78 -0
index 1c310b718961..3e5c88caf5d5 100644
@@ -11165,6 +11165,84 @@ int elf_find_multi_func_offset(const char *binary_path, int cnt,
 	return ret;
 }
 
+static int
+__elf_find_pattern_func_offset(Elf *elf, const char *binary_path, const char *pattern,
+			       const char ***pnames, unsigned long **poffsets, size_t *pcnt)
+{
+	int sh_types[2] = { SHT_DYNSYM, SHT_SYMTAB };
+	struct elf_symbol_offset *func_offs = NULL;
+	unsigned long *offsets = NULL;
+	const char **names = NULL;
+	size_t func_offs_cnt = 0;
+	size_t func_offs_cap = 0;
+	int err = 0, i;
+
+	for (i = 0; i < ARRAY_SIZE(sh_types); i++) {
+		struct elf_symbol_iter iter;
+		struct elf_symbol *sym;
+
+		if (elf_symbol_iter_new(&iter, elf, binary_path, sh_types[i]))
+			continue;
+
+		while ((sym = elf_symbol_iter_next(&iter))) {
+			if (!glob_match(sym->name, pattern))
+				continue;
+
+			err = libbpf_ensure_mem((void **) &func_offs, &func_offs_cap,
+						sizeof(*func_offs), func_offs_cnt + 1);
+			if (err)
+				goto out;
+
+			func_offs[func_offs_cnt].offset = sym->offset;
+			func_offs[func_offs_cnt].name = strdup(sym->name);
+			func_offs_cnt++;
+		}
+
+		/* If we found anything in the first symbol section,
+		 * do not search others to avoid duplicates.
+		 */
+		if (func_offs_cnt)
+			break;
+	}
+
+	offsets = calloc(func_offs_cnt, sizeof(*offsets));
+	names = calloc(func_offs_cnt, sizeof(*names));
+	if (!offsets || !names) {
+		free(offsets);
+		free(names);
+		err = -ENOMEM;
+		goto out;
+	}
+
+	for (i = 0; i < func_offs_cnt; i++) {
+		offsets[i] = func_offs[i].offset;
+		names[i] = func_offs[i].name;
+	}
+
+	*pnames = names;
+	*poffsets = offsets;
+	*pcnt = func_offs_cnt;
+out:
+	free(func_offs);
+	return err;
+}
+
+int elf_find_pattern_func_offset(const char *binary_path, const char *pattern,
+				 const char ***pnames, unsigned long **poffsets,
+				 size_t *pcnt)
+{
+	struct elf_fd elf_fd = {};
+	long ret = -ENOENT;
+
+	ret = open_elf(binary_path, &elf_fd);
+	if (ret)
+		return ret;
+
+	ret = __elf_find_pattern_func_offset(elf_fd.elf, binary_path, pattern, pnames, poffsets, pcnt);
+	close_elf(&elf_fd);
+	return ret;
+}
+
 /* Find offset of function name in ELF object specified by path. "name" matches
  * symbol name or name@@LIB for library functions.
  */
tools/lib/bpf/libbpf_internal.h +3 -0
index 13d5c12fbd0b..22b0834e7fe1 100644
@@ -579,4 +579,7 @@ int sys_bpf_prog_load(union bpf_attr *attr, unsigned int size, int attempts);
 
 int elf_find_multi_func_offset(const char *binary_path, int cnt,
 			       const char **syms, unsigned long **poffsets);
+int elf_find_pattern_func_offset(const char *binary_path, const char *pattern,
+				 const char ***pnames, unsigned long **poffsets,
+				 size_t *pcnt);
 #endif /* __LIBBPF_LIBBPF_INTERNAL_H */
Signature · 1 line
2.41.0
#33 lore raw

Re: [PATCHv2 bpf-next 09/24] libbpf: Add elf_find_pattern_func_offset function

Quoted 31 lines · Jiri Olsa

Adding elf_find_pattern_func_offset function that looks up offsets for symbols specified by pattern argument.

The 'pattern' argument allows wildcards (*?' supported).

Offsets are returned in allocated array together with its size and needs to be released by the caller.

---
 tools/lib/bpf/libbpf.c          | 78 +++++++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf_internal.h |  3 ++
 2 files changed, 81 insertions(+)
tools/lib/bpf/libbpf.c +8 -0
index 1c310b718961..3e5c88caf5d5 100644
@@ -11165,6 +11165,84 @@ int elf_find_multi_func_offset(const char *binary_path, int cnt,
        return ret;
 }
 
+static int
+__elf_find_pattern_func_offset(Elf *elf, const char *binary_path, const char *pattern,
+                              const char ***pnames, unsigned long **poffsets, size_t *pcnt)
+{
+       int sh_types[2] = { SHT_DYNSYM, SHT_SYMTAB };
+       struct elf_symbol_offset *func_offs = NULL;
+       unsigned long *offsets = NULL;
+       const char **names = NULL;

tools/lib/bpf/libbpf.c int elf_find_multi_func_offset(const char *binary_path, int cnt, · line 11185

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+       size_t func_offs_cnt = 0;
+       size_t func_offs_cap = 0;
+       int err = 0, i;
+
+       for (i = 0; i < ARRAY_SIZE(sh_types); i++) {
+               struct elf_symbol_iter iter;
+               struct elf_symbol *sym;
+
+               if (elf_symbol_iter_new(&iter, elf, binary_path, sh_types[i]))
+                       continue;

same as in the previous patch, error handling?

tools/lib/bpf/libbpf.c int elf_find_multi_func_offset(const char *binary_path, int cnt, · line 11197

Quoted 12 lines
tools/lib/bpf/libbpf.c +12 -0
+
+               while ((sym = elf_symbol_iter_next(&iter))) {
+                       if (!glob_match(sym->name, pattern))
+                               continue;
+
+                       err = libbpf_ensure_mem((void **) &func_offs, &func_offs_cap,
+                                               sizeof(*func_offs), func_offs_cnt + 1);
+                       if (err)
+                               goto out;
+
+                       func_offs[func_offs_cnt].offset = sym->offset;
+                       func_offs[func_offs_cnt].name = strdup(sym->name);

check for NULL?

and I'm actually unsure why you need to reuse elf_symbol_offset struct here? You just need names and offsets in two separate array, so just do that?..

Quoted 12 lines
tools/lib/bpf/libbpf.c +12 -0
+                       func_offs_cnt++;
+               }
+
+               /* If we found anything in the first symbol section,
+                * do not search others to avoid duplicates.
+                */
+               if (func_offs_cnt)
+                       break;
+       }
+
+       offsets = calloc(func_offs_cnt, sizeof(*offsets));
+       names = calloc(func_offs_cnt, sizeof(*names));

tools/lib/bpf/libbpf.c int elf_find_multi_func_offset(const char *binary_path, int cnt, · line 11219

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+       if (!offsets || !names) {
+               free(offsets);
+               free(names);
+               err = -ENOMEM;
+               goto out;
+       }
+
+       for (i = 0; i < func_offs_cnt; i++) {
+               offsets[i] = func_offs[i].offset;
+               names[i] = func_offs[i].name;

see above, why not fill these out right away during elf symbols iteration?

Quoted 12 lines
tools/lib/bpf/libbpf.c +12 -0
+       }
+
+       *pnames = names;
+       *poffsets = offsets;
+       *pcnt = func_offs_cnt;
+out:
+       free(func_offs);
+       return err;
+}
+
+int elf_find_pattern_func_offset(const char *binary_path, const char *pattern,
+                                const char ***pnames, unsigned long **poffsets,

tools/lib/bpf/libbpf.c int elf_find_multi_func_offset(const char *binary_path, int cnt, · line 11241

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+                                size_t *pcnt)
+{
+       struct elf_fd elf_fd = {};
+       long ret = -ENOENT;
+
+       ret = open_elf(binary_path, &elf_fd);
+       if (ret)
+               return ret;
+
+       ret = __elf_find_pattern_func_offset(elf_fd.elf, binary_path, pattern, pnames, poffsets, pcnt);

I don't really like these underscored functions, elf_find_pattern_func_offset() already has to do goto out for clean ups, this close_elf() thing is just another resource to clean up there, I don't see much reason to separate open_elf/close_elf in this case

Quoted 22 lines
tools/lib/bpf/libbpf.c +4 -0
+       close_elf(&elf_fd);
+       return ret;
+}
+
 /* Find offset of function name in ELF object specified by path. "name" matches
  * symbol name or name@@LIB for library functions.
  */
tools/lib/bpf/libbpf_internal.h +3 -1
index 13d5c12fbd0b..22b0834e7fe1 100644
@@ -579,4 +579,7 @@ int sys_bpf_prog_load(union bpf_attr *attr, unsigned int size, int attempts);
 
 int elf_find_multi_func_offset(const char *binary_path, int cnt,
                               const char **syms, unsigned long **poffsets);
+int elf_find_pattern_func_offset(const char *binary_path, const char *pattern,
+                                const char ***pnames, unsigned long **poffsets,
+                                size_t *pcnt);
 #endif /* __LIBBPF_LIBBPF_INTERNAL_H */
--

2.41.0

#34 lore raw

Re: [PATCHv2 bpf-next 09/24] libbpf: Add elf_find_pattern_func_offset function

Quoted 36 lines · Andrii Nakryiko
Quoted 35 lines · Jiri Olsa

Adding elf_find_pattern_func_offset function that looks up offsets for symbols specified by pattern argument.

The 'pattern' argument allows wildcards (*?' supported).

Offsets are returned in allocated array together with its size and needs to be released by the caller.

---
 tools/lib/bpf/libbpf.c          | 78 +++++++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf_internal.h |  3 ++
 2 files changed, 81 insertions(+)
tools/lib/bpf/libbpf.c +12 -0
index 1c310b718961..3e5c88caf5d5 100644
@@ -11165,6 +11165,84 @@ int elf_find_multi_func_offset(const char *binary_path, int cnt,
        return ret;
 }
 
+static int
+__elf_find_pattern_func_offset(Elf *elf, const char *binary_path, const char *pattern,
+                              const char ***pnames, unsigned long **poffsets, size_t *pcnt)
+{
+       int sh_types[2] = { SHT_DYNSYM, SHT_SYMTAB };
+       struct elf_symbol_offset *func_offs = NULL;
+       unsigned long *offsets = NULL;
+       const char **names = NULL;
+       size_t func_offs_cnt = 0;
+       size_t func_offs_cap = 0;
+       int err = 0, i;
+
Quoted 8 lines

tools/lib/bpf/libbpf.c int elf_find_multi_func_offset(const char *binary_path, int cnt, · line 11185

Quoted 6 lines
tools/lib/bpf/libbpf.c +6 -0
+       for (i = 0; i < ARRAY_SIZE(sh_types); i++) {
+               struct elf_symbol_iter iter;
+               struct elf_symbol *sym;
+
+               if (elf_symbol_iter_new(&iter, elf, binary_path, sh_types[i]))
+                       continue;

same as in the previous patch, error handling?

ok, same solution as in my previous answer

Quoted 16 lines

tools/lib/bpf/libbpf.c int elf_find_multi_func_offset(const char *binary_path, int cnt, · line 11197

Quoted 13 lines
tools/lib/bpf/libbpf.c +12 -0
+
+               while ((sym = elf_symbol_iter_next(&iter))) {
+                       if (!glob_match(sym->name, pattern))
+                               continue;
+
+                       err = libbpf_ensure_mem((void **) &func_offs, &func_offs_cap,
+                                               sizeof(*func_offs), func_offs_cnt + 1);
+                       if (err)
+                               goto out;
+
+                       func_offs[func_offs_cnt].offset = sym->offset;
+                       func_offs[func_offs_cnt].name = strdup(sym->name);

check for NULL?

Quoted 3 lines

and I'm actually unsure why you need to reuse elf_symbol_offset struct here? You just need names and offsets in two separate array, so just do that?..

I guess it seemed easier to do just single libbpf_ensure_mem call, then doing two and maintain cap/cnt for both arrays.. but right, I guess there will be good solution without the extra array

Quoted 17 lines
Quoted 17 lines
tools/lib/bpf/libbpf.c +16 -0
+                       func_offs_cnt++;
+               }
+
+               /* If we found anything in the first symbol section,
+                * do not search others to avoid duplicates.
+                */
+               if (func_offs_cnt)
+                       break;
+       }
+
+       offsets = calloc(func_offs_cnt, sizeof(*offsets));
+       names = calloc(func_offs_cnt, sizeof(*names));
+       if (!offsets || !names) {
+               free(offsets);
+               free(names);
+               err = -ENOMEM;
Quoted 8 lines

tools/lib/bpf/libbpf.c int elf_find_multi_func_offset(const char *binary_path, int cnt, · line 11219

Quoted 6 lines
tools/lib/bpf/libbpf.c +6 -0
+               goto out;
+       }
+
+       for (i = 0; i < func_offs_cnt; i++) {
+               offsets[i] = func_offs[i].offset;
+               names[i] = func_offs[i].name;

see above, why not fill these out right away during elf symbols iteration?

ok

Quoted 21 lines
Quoted 21 lines
tools/lib/bpf/libbpf.c +20 -0
+       }
+
+       *pnames = names;
+       *poffsets = offsets;
+       *pcnt = func_offs_cnt;
+out:
+       free(func_offs);
+       return err;
+}
+
+int elf_find_pattern_func_offset(const char *binary_path, const char *pattern,
+                                const char ***pnames, unsigned long **poffsets,
+                                size_t *pcnt)
+{
+       struct elf_fd elf_fd = {};
+       long ret = -ENOENT;
+
+       ret = open_elf(binary_path, &elf_fd);
+       if (ret)
+               return ret;
Quoted 8 lines

tools/lib/bpf/libbpf.c int elf_find_multi_func_offset(const char *binary_path, int cnt, · line 11241

Quoted 2 lines
tools/lib/bpf/libbpf.c +2 -0
+
+       ret = __elf_find_pattern_func_offset(elf_fd.elf, binary_path, pattern, pnames, poffsets, pcnt);

I don't really like these underscored functions, elf_find_pattern_func_offset() already has to do goto out for clean ups, this close_elf() thing is just another resource to clean up there, I don't see much reason to separate open_elf/close_elf in this case

I think that happened because I originally did the 'struct elf_fd' cleanup as last patch and only then moved it up the git log, so I did not realize elf_find_pattern_func_offset could be single function, will change

thanks,
jirka

Quoted 24 lines
Quoted 24 lines
tools/lib/bpf/libbpf.c +4 -0
+       close_elf(&elf_fd);
+       return ret;
+}
+
 /* Find offset of function name in ELF object specified by path. "name" matches
  * symbol name or name@@LIB for library functions.
  */
tools/lib/bpf/libbpf_internal.h +3 -1
index 13d5c12fbd0b..22b0834e7fe1 100644
@@ -579,4 +579,7 @@ int sys_bpf_prog_load(union bpf_attr *attr, unsigned int size, int attempts);
 
 int elf_find_multi_func_offset(const char *binary_path, int cnt,
                               const char **syms, unsigned long **poffsets);
+int elf_find_pattern_func_offset(const char *binary_path, const char *pattern,
+                                const char ***pnames, unsigned long **poffsets,
+                                size_t *pcnt);
 #endif /* __LIBBPF_LIBBPF_INTERNAL_H */
--

2.41.0

author reviewer
#35 lore raw

PATCH v2 10/24 libbpf: Add bpf_link_create support for multi uprobes

In reply to Jiri Olsa

Adding new uprobe_multi struct to bpf_link_create_opts object to pass multiple uprobe data to link_create attr uapi.

---
 tools/lib/bpf/bpf.c | 11 +++++++++++
 tools/lib/bpf/bpf.h | 11 ++++++++++-
 2 files changed, 21 insertions(+), 1 deletion(-)
tools/lib/bpf/bpf.c +11 -0
index ed86b37d8024..0fd35c91f50c 100644
@@ -733,6 +733,17 @@ int bpf_link_create(int prog_fd, int target_fd,
 		if (!OPTS_ZEROED(opts, kprobe_multi))
 			return libbpf_err(-EINVAL);
 		break;
+	case BPF_TRACE_UPROBE_MULTI:
+		attr.link_create.uprobe_multi.flags = OPTS_GET(opts, uprobe_multi.flags, 0);
+		attr.link_create.uprobe_multi.cnt = OPTS_GET(opts, uprobe_multi.cnt, 0);
+		attr.link_create.uprobe_multi.path = ptr_to_u64(OPTS_GET(opts, uprobe_multi.path, 0));
+		attr.link_create.uprobe_multi.offsets = ptr_to_u64(OPTS_GET(opts, uprobe_multi.offsets, 0));
+		attr.link_create.uprobe_multi.ref_ctr_offsets = ptr_to_u64(OPTS_GET(opts, uprobe_multi.ref_ctr_offsets, 0));
+		attr.link_create.uprobe_multi.cookies = ptr_to_u64(OPTS_GET(opts, uprobe_multi.cookies, 0));
+		attr.link_create.uprobe_multi.pid = OPTS_GET(opts, uprobe_multi.pid, 0);
+		if (!OPTS_ZEROED(opts, uprobe_multi))
+			return libbpf_err(-EINVAL);
+		break;
 	case BPF_TRACE_FENTRY:
 	case BPF_TRACE_FEXIT:
 	case BPF_MODIFY_RETURN:
tools/lib/bpf/bpf.h +10 -1
index 9aa0ee473754..82979b4f2769 100644
@@ -346,13 +346,22 @@ struct bpf_link_create_opts {
 			const unsigned long *addrs;
 			const __u64 *cookies;
 		} kprobe_multi;
+		struct {
+			__u32 flags;
+			__u32 cnt;
+			const char *path;
+			const unsigned long *offsets;
+			const unsigned long *ref_ctr_offsets;
+			const __u64 *cookies;
+			__u32 pid;
+		} uprobe_multi;
 		struct {
 			__u64 cookie;
 		} tracing;
 	};
 	size_t :0;
 };
-#define bpf_link_create_opts__last_field kprobe_multi.cookies
+#define bpf_link_create_opts__last_field uprobe_multi.pid
 
 LIBBPF_API int bpf_link_create(int prog_fd, int target_fd,
 			       enum bpf_attach_type attach_type,
Signature · 1 line
2.41.0
Acked-by on v2 10/24
#36 lore raw

Re: [PATCHv2 bpf-next 10/24] libbpf: Add bpf_link_create support for multi uprobes

Quoted 10 lines · Jiri Olsa

Adding new uprobe_multi struct to bpf_link_create_opts object to pass multiple uprobe data to link_create attr uapi.

---
 tools/lib/bpf/bpf.c | 11 +++++++++++
 tools/lib/bpf/bpf.h | 11 ++++++++++-
 2 files changed, 21 insertions(+), 1 deletion(-)

LGTM

Quoted 53 lines
tools/lib/bpf/bpf.c +11 -0
index ed86b37d8024..0fd35c91f50c 100644
@@ -733,6 +733,17 @@ int bpf_link_create(int prog_fd, int target_fd,
                if (!OPTS_ZEROED(opts, kprobe_multi))
                        return libbpf_err(-EINVAL);
                break;
+       case BPF_TRACE_UPROBE_MULTI:
+               attr.link_create.uprobe_multi.flags = OPTS_GET(opts, uprobe_multi.flags, 0);
+               attr.link_create.uprobe_multi.cnt = OPTS_GET(opts, uprobe_multi.cnt, 0);
+               attr.link_create.uprobe_multi.path = ptr_to_u64(OPTS_GET(opts, uprobe_multi.path, 0));
+               attr.link_create.uprobe_multi.offsets = ptr_to_u64(OPTS_GET(opts, uprobe_multi.offsets, 0));
+               attr.link_create.uprobe_multi.ref_ctr_offsets = ptr_to_u64(OPTS_GET(opts, uprobe_multi.ref_ctr_offsets, 0));
+               attr.link_create.uprobe_multi.cookies = ptr_to_u64(OPTS_GET(opts, uprobe_multi.cookies, 0));
+               attr.link_create.uprobe_multi.pid = OPTS_GET(opts, uprobe_multi.pid, 0);
+               if (!OPTS_ZEROED(opts, uprobe_multi))
+                       return libbpf_err(-EINVAL);
+               break;
        case BPF_TRACE_FENTRY:
        case BPF_TRACE_FEXIT:
        case BPF_MODIFY_RETURN:
tools/lib/bpf/bpf.h +10 -2
index 9aa0ee473754..82979b4f2769 100644
@@ -346,13 +346,22 @@ struct bpf_link_create_opts {
                        const unsigned long *addrs;
                        const __u64 *cookies;
                } kprobe_multi;
+               struct {
+                       __u32 flags;
+                       __u32 cnt;
+                       const char *path;
+                       const unsigned long *offsets;
+                       const unsigned long *ref_ctr_offsets;
+                       const __u64 *cookies;
+                       __u32 pid;
+               } uprobe_multi;
                struct {
                        __u64 cookie;
                } tracing;
        };
        size_t :0;
 };
-#define bpf_link_create_opts__last_field kprobe_multi.cookies
+#define bpf_link_create_opts__last_field uprobe_multi.pid
 
 LIBBPF_API int bpf_link_create(int prog_fd, int target_fd,
                               enum bpf_attach_type attach_type,
--

2.41.0

author reviewer
#37 lore raw

PATCH v2 11/24 libbpf: Add bpf_program__attach_uprobe_multi_opts function

In reply to Jiri Olsa

Adding bpf_program__attach_uprobe_multi_opts function that allows to attach multiple uprobes with uprobe_multi link.

The user can specify uprobes with direct arguments:

binary_path/func_pattern

or with struct bpf_uprobe_multi_opts opts argument fields:

const char *path;
const char **syms;
const unsigned long *offsets;
const unsigned long *ref_ctr_offsets;

User can specify 3 mutually exclusive set of incputs:

1) use only binary_path/func_pattern aruments

2) use only opts argument with allowed combinations of:
path/offsets/ref_ctr_offsets/cookies/cnt

3) use binary_path with allowed combinations of:
syms/offsets/ref_ctr_offsets/cookies/cnt

Any other usage results in error.

---
 tools/lib/bpf/libbpf.c   | 131 +++++++++++++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf.h   |  31 +++++++++
 tools/lib/bpf/libbpf.map |   1 +
 3 files changed, 163 insertions(+)
tools/lib/bpf/libbpf.c +131 -0
index 3e5c88caf5d5..d972cea4c658 100644
@@ -11402,6 +11402,137 @@ static int resolve_full_path(const char *file, char *result, size_t result_sz)
 	return -ENOENT;
 }
 
+struct bpf_link *
+bpf_program__attach_uprobe_multi_opts(const struct bpf_program *prog,
+				      pid_t pid,
+				      const char *binary_path,
+				      const char *func_pattern,
+				      const struct bpf_uprobe_multi_opts *opts)
+{
+	const unsigned long *ref_ctr_offsets = NULL, *offsets = NULL;
+	LIBBPF_OPTS(bpf_link_create_opts, lopts);
+	unsigned long *resolved_offsets = NULL;
+	const char **resolved_symbols = NULL;
+	int err = 0, link_fd, prog_fd;
+	struct bpf_link *link = NULL;
+	char errmsg[STRERR_BUFSIZE];
+	const char *path, **syms;
+	char full_path[PATH_MAX];
+	const __u64 *cookies;
+	bool has_pattern;
+	bool retprobe;
+	size_t cnt;
+
+	if (!OPTS_VALID(opts, bpf_uprobe_multi_opts))
+		return libbpf_err_ptr(-EINVAL);
+
+	path = OPTS_GET(opts, path, NULL);
+	syms = OPTS_GET(opts, syms, NULL);
+	offsets = OPTS_GET(opts, offsets, NULL);
+	ref_ctr_offsets = OPTS_GET(opts, ref_ctr_offsets, NULL);
+	cookies = OPTS_GET(opts, cookies, NULL);
+	cnt = OPTS_GET(opts, cnt, 0);
+
+	/*
+	 * User can specify 3 mutually exclusive set of incputs:
+	 *
+	 * 1) use only binary_path/func_pattern aruments
+	 *
+	 * 2) use only opts argument with allowed combinations of:
+	 *    path/offsets/ref_ctr_offsets/cookies/cnt
+	 *
+	 * 3) use binary_path with allowed combinations of:
+	 *    syms/offsets/ref_ctr_offsets/cookies/cnt
+	 *
+	 * Any other usage results in error.
+	 */
+
+	if (!binary_path && !func_pattern && !cnt)
+		return libbpf_err_ptr(-EINVAL);
+	if (func_pattern && !binary_path)
+		return libbpf_err_ptr(-EINVAL);
+
+	has_pattern = binary_path && func_pattern;
+
+	if (has_pattern) {
+		if (path || syms || offsets || ref_ctr_offsets || cookies || cnt)
+			return libbpf_err_ptr(-EINVAL);
+	} else {
+		if (!cnt)
+			return libbpf_err_ptr(-EINVAL);
+		if (!!path == !!binary_path)
+			return libbpf_err_ptr(-EINVAL);
+		if (!!syms == !!offsets)
+			return libbpf_err_ptr(-EINVAL);
+		if (path && syms)
+			return libbpf_err_ptr(-EINVAL);
+	}
+
+	if (has_pattern) {
+		if (!strchr(binary_path, '/')) {
+			err = resolve_full_path(binary_path, full_path, sizeof(full_path));
+			if (err) {
+				pr_warn("prog '%s': failed to resolve full path for '%s': %d\n",
+					prog->name, binary_path, err);
+				return libbpf_err_ptr(err);
+			}
+			binary_path = full_path;
+		}
+
+		err = elf_find_pattern_func_offset(binary_path, func_pattern,
+						   &resolved_symbols, &resolved_offsets,
+						   &cnt);
+		if (err < 0)
+			return libbpf_err_ptr(err);
+		offsets = resolved_offsets;
+	} else if (syms) {
+		err = elf_find_multi_func_offset(binary_path, cnt, syms, &resolved_offsets);
+		if (err < 0)
+			return libbpf_err_ptr(err);
+		offsets = resolved_offsets;
+	}
+
+	retprobe = OPTS_GET(opts, retprobe, false);
+
+	lopts.uprobe_multi.path = path ?: binary_path;
+	lopts.uprobe_multi.offsets = offsets;
+	lopts.uprobe_multi.ref_ctr_offsets = ref_ctr_offsets;
+	lopts.uprobe_multi.cookies = cookies;
+	lopts.uprobe_multi.cnt = cnt;
+	lopts.uprobe_multi.flags = retprobe ? BPF_F_UPROBE_MULTI_RETURN : 0;
+
+	if (pid == 0)
+		pid = getpid();
+	if (pid > 0)
+		lopts.uprobe_multi.pid = pid;
+
+	link = calloc(1, sizeof(*link));
+	if (!link) {
+		err = -ENOMEM;
+		goto error;
+	}
+	link->detach = &bpf_link__detach_fd;
+
+	prog_fd = bpf_program__fd(prog);
+	link_fd = bpf_link_create(prog_fd, 0, BPF_TRACE_UPROBE_MULTI, &lopts);
+	if (link_fd < 0) {
+		err = -errno;
+		pr_warn("prog '%s': failed to attach: %s\n",
+			prog->name, libbpf_strerror_r(err, errmsg, sizeof(errmsg)));
+		goto error;
+	}
+	link->fd = link_fd;
+	free(resolved_offsets);
+	free(resolved_symbols);
+	return link;
+
+error:
+	free(resolved_offsets);
+	free(resolved_symbols);
+	free(link);
+	return libbpf_err_ptr(err);
+}
+
 LIBBPF_API struct bpf_link *
 bpf_program__attach_uprobe_opts(const struct bpf_program *prog, pid_t pid,
 				const char *binary_path, size_t func_offset,
tools/lib/bpf/libbpf.h +31 -0
index 754da73c643b..b6ff7d69a1d7 100644
@@ -529,6 +529,37 @@ bpf_program__attach_kprobe_multi_opts(const struct bpf_program *prog,
 				      const char *pattern,
 				      const struct bpf_kprobe_multi_opts *opts);
 
+struct bpf_uprobe_multi_opts {
+	/* size of this struct, for forward/backward compatibility */
+	size_t sz;
+	/* path to attach */
+	const char *path;
+	/* array of function symbols to attach */
+	const char **syms;
+	/* array of function addresses to attach */
+	const unsigned long *offsets;
+	/* array of refctr offsets to attach */
+	const unsigned long *ref_ctr_offsets;
+	/* array of user-provided values fetchable through bpf_get_attach_cookie */
+	const __u64 *cookies;
+	/* number of elements in syms/addrs/cookies arrays */
+	size_t cnt;
+	/* create return uprobes */
+	bool retprobe;
+	/* pid filter */
+	int pid;
+	size_t :0;
+};
+
+#define bpf_uprobe_multi_opts__last_field pid
+
+LIBBPF_API struct bpf_link *
+bpf_program__attach_uprobe_multi_opts(const struct bpf_program *prog,
+				      pid_t pid,
+				      const char *binary_path,
+				      const char *func_pattern,
+				      const struct bpf_uprobe_multi_opts *opts);
+
 struct bpf_ksyscall_opts {
 	/* size of this struct, for forward/backward compatibility */
 	size_t sz;
tools/lib/bpf/libbpf.map +1 -0
index 7521a2fb7626..81558ef1bc38 100644
@@ -390,6 +390,7 @@ LIBBPF_1.2.0 {
 		bpf_link_get_info_by_fd;
 		bpf_map_get_info_by_fd;
 		bpf_prog_get_info_by_fd;
+		bpf_program__attach_uprobe_multi_opts;
 } LIBBPF_1.1.0;
 
 LIBBPF_1.3.0 {
Signature · 1 line
2.41.0
#38 lore raw

Re: [PATCHv2 bpf-next 11/24] libbpf: Add bpf_program__attach_uprobe_multi_opts function

Quoted 8 lines · Jiri Olsa

Adding bpf_program__attach_uprobe_multi_opts function that allows to attach multiple uprobes with uprobe_multi link.

The user can specify uprobes with direct arguments:

binary_path/func_pattern

Quoted 8 lines

or with struct bpf_uprobe_multi_opts opts argument fields:

const char *path;
const char **syms;
const unsigned long *offsets;
const unsigned long *ref_ctr_offsets;

User can specify 3 mutually exclusive set of incputs:

typo: inputs

Quoted 2 lines

1) use only binary_path/func_pattern aruments

typo: arguments

Quoted 7 lines

2) use only opts argument with allowed combinations of:
path/offsets/ref_ctr_offsets/cookies/cnt

3) use binary_path with allowed combinations of:
syms/offsets/ref_ctr_offsets/cookies/cnt

why do we need this duplication of binary_path and opts.path? same for pid and opts.pid?

Quoted 14 lines

Any other usage results in error.

---
 tools/lib/bpf/libbpf.c   | 131 +++++++++++++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf.h   |  31 +++++++++
 tools/lib/bpf/libbpf.map |   1 +
 3 files changed, 163 insertions(+)
tools/lib/bpf/libbpf.c +0 -0
index 3e5c88caf5d5..d972cea4c658 100644

tools/lib/bpf/libbpf.c static int resolve_full_path(const char *file, char *result, size_t result_sz) · line 11406

Quoted 5 lines
tools/lib/bpf/libbpf.c +2 -0
@@ -11402,6 +11402,137 @@ static int resolve_full_path(const char *file, char *result, size_t result_sz)
        return -ENOENT;
 }
 
+struct bpf_link *
+bpf_program__attach_uprobe_multi_opts(const struct bpf_program *prog,

let's call it just bpf_program__attach_uprobe_multi()

we should have done that with bpf_program__attach_kprobe_multi_opts() as well. Generally, if the only variant of API is the one with opts, then there is no point in adding "_opts" suffix to the API name, IMO

Quoted 23 lines
tools/lib/bpf/libbpf.c +23 -0
+                                     pid_t pid,
+                                     const char *binary_path,
+                                     const char *func_pattern,
+                                     const struct bpf_uprobe_multi_opts *opts)
+{
+       const unsigned long *ref_ctr_offsets = NULL, *offsets = NULL;
+       LIBBPF_OPTS(bpf_link_create_opts, lopts);
+       unsigned long *resolved_offsets = NULL;
+       const char **resolved_symbols = NULL;
+       int err = 0, link_fd, prog_fd;
+       struct bpf_link *link = NULL;
+       char errmsg[STRERR_BUFSIZE];
+       const char *path, **syms;
+       char full_path[PATH_MAX];
+       const __u64 *cookies;
+       bool has_pattern;
+       bool retprobe;
+       size_t cnt;
+
+       if (!OPTS_VALID(opts, bpf_uprobe_multi_opts))
+               return libbpf_err_ptr(-EINVAL);
+
+       path = OPTS_GET(opts, path, NULL);

tools/lib/bpf/libbpf.c static int resolve_full_path(const char *file, char *result, size_t result_sz) · line 11439

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+       syms = OPTS_GET(opts, syms, NULL);
+       offsets = OPTS_GET(opts, offsets, NULL);
+       ref_ctr_offsets = OPTS_GET(opts, ref_ctr_offsets, NULL);
+       cookies = OPTS_GET(opts, cookies, NULL);
+       cnt = OPTS_GET(opts, cnt, 0);
+
+       /*
+        * User can specify 3 mutually exclusive set of incputs:
+        *
+        * 1) use only binary_path/func_pattern aruments

same typos

tools/lib/bpf/libbpf.c static int resolve_full_path(const char *file, char *result, size_t result_sz) · line 11449

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+        *
+        * 2) use only opts argument with allowed combinations of:
+        *    path/offsets/ref_ctr_offsets/cookies/cnt
+        *
+        * 3) use binary_path with allowed combinations of:
+        *    syms/offsets/ref_ctr_offsets/cookies/cnt
+        *
+        * Any other usage results in error.
+        */
+

Looking at this part (sorry, I already trimmed reply before realizing I want to comment on this):

+ err = elf_find_pattern_func_offset(binary_path, func_pattern,
+ &resolved_symbols,
&resolved_offsets,
+ &cnt);
+ if (err < 0)
+ return libbpf_err_ptr(err);
+ offsets = resolved_offsets;
+ } else if (syms) {
+ err = elf_find_multi_func_offset(binary_path, cnt,
syms, &resolved_offsets);

Few thoughts.

First, we are not using resolved_symbols returned from elf_find_pattern_func_offset(), do we? If not, let's drop it.

Second, I'm not a big fan of chosen naming. Maybe something like elf_resolve_{pattern,syms}_offsets?

[...]

Quoted 15 lines
tools/lib/bpf/libbpf.c +7 -0
+error:
+       free(resolved_offsets);
+       free(resolved_symbols);
+       free(link);
+       return libbpf_err_ptr(err);
+}
+
 LIBBPF_API struct bpf_link *
 bpf_program__attach_uprobe_opts(const struct bpf_program *prog, pid_t pid,
                                const char *binary_path, size_t func_offset,
tools/lib/bpf/libbpf.h +0 -0
index 754da73c643b..b6ff7d69a1d7 100644

tools/lib/bpf/libbpf.h bpf_program__attach_kprobe_multi_opts(const struct bpf_program *prog, · line 536

Quoted 8 lines
tools/lib/bpf/libbpf.h +5 -0
@@ -529,6 +529,37 @@ bpf_program__attach_kprobe_multi_opts(const struct bpf_program *prog,
                                      const char *pattern,
                                      const struct bpf_kprobe_multi_opts *opts);
 
+struct bpf_uprobe_multi_opts {
+       /* size of this struct, for forward/backward compatibility */
+       size_t sz;
+       /* path to attach */
+       const char *path;

how is this different from binary_path? not clear why we need to paths

Quoted 4 lines
tools/lib/bpf/libbpf.h +4 -0
+       /* array of function symbols to attach */
+       const char **syms;
+       /* array of function addresses to attach */
+       const unsigned long *offsets;

tools/lib/bpf/libbpf.h bpf_program__attach_kprobe_multi_opts(const struct bpf_program *prog, · line 550

Quoted 10 lines
tools/lib/bpf/libbpf.h +10 -0
+       /* array of refctr offsets to attach */
+       const unsigned long *ref_ctr_offsets;
+       /* array of user-provided values fetchable through bpf_get_attach_cookie */
+       const __u64 *cookies;
+       /* number of elements in syms/addrs/cookies arrays */
+       size_t cnt;
+       /* create return uprobes */
+       bool retprobe;
+       /* pid filter */
+       int pid;

another duplicate, we already pass pid argument

Quoted 20 lines
tools/lib/bpf/libbpf.h +12 -0
+       size_t :0;
+};
+
+#define bpf_uprobe_multi_opts__last_field pid
+
+LIBBPF_API struct bpf_link *
+bpf_program__attach_uprobe_multi_opts(const struct bpf_program *prog,
+                                     pid_t pid,
+                                     const char *binary_path,
+                                     const char *func_pattern,
+                                     const struct bpf_uprobe_multi_opts *opts);
+
 struct bpf_ksyscall_opts {
        /* size of this struct, for forward/backward compatibility */
        size_t sz;
tools/lib/bpf/libbpf.map +0 -0
index 7521a2fb7626..81558ef1bc38 100644

tools/lib/bpf/libbpf.map LIBBPF_1.2.0 { · line 393

Quoted 4 lines
tools/lib/bpf/libbpf.map +1 -0
@@ -390,6 +390,7 @@ LIBBPF_1.2.0 {
                bpf_link_get_info_by_fd;
                bpf_map_get_info_by_fd;
                bpf_prog_get_info_by_fd;
+               bpf_program__attach_uprobe_multi_opts;

should be in 1.3.0?

Quoted 8 lines
tools/lib/bpf/libbpf.map +0 -1
 } LIBBPF_1.1.0;
 
 LIBBPF_1.3.0 {
 
 
--

2.41.0

#39 lore raw

Re: [PATCHv2 bpf-next 11/24] libbpf: Add bpf_program__attach_uprobe_multi_opts function

Quoted 16 lines · Andrii Nakryiko
Quoted 15 lines · Jiri Olsa

Adding bpf_program__attach_uprobe_multi_opts function that allows to attach multiple uprobes with uprobe_multi link.

The user can specify uprobes with direct arguments:

binary_path/func_pattern

or with struct bpf_uprobe_multi_opts opts argument fields:

const char *path;
const char **syms;
const unsigned long *offsets;
const unsigned long *ref_ctr_offsets;

Quoted 8 lines
Quoted 1 line

User can specify 3 mutually exclusive set of incputs:

typo: inputs

Quoted 2 lines

1) use only binary_path/func_pattern aruments

typo: arguments

ok

Quoted 11 lines
Quoted 8 lines

2) use only opts argument with allowed combinations of:
path/offsets/ref_ctr_offsets/cookies/cnt

3) use binary_path with allowed combinations of:
syms/offsets/ref_ctr_offsets/cookies/cnt

why do we need this duplication of binary_path and opts.path? same for pid and opts.pid?

right, will remove opts.path|pid

Quoted 18 lines
Quoted 18 lines

Any other usage results in error.

---
 tools/lib/bpf/libbpf.c   | 131 +++++++++++++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf.h   |  31 +++++++++
 tools/lib/bpf/libbpf.map |   1 +
 3 files changed, 163 insertions(+)
tools/lib/bpf/libbpf.c +0 -0
index 3e5c88caf5d5..d972cea4c658 100644
@@ -11402,6 +11402,137 @@ static int resolve_full_path(const char *file, char *result, size_t result_sz)
        return -ENOENT;
 }
 
Quoted 8 lines

tools/lib/bpf/libbpf.c static int resolve_full_path(const char *file, char *result, size_t result_sz) · line 11406

Quoted 2 lines
tools/lib/bpf/libbpf.c +2 -0
+struct bpf_link *
+bpf_program__attach_uprobe_multi_opts(const struct bpf_program *prog,

let's call it just bpf_program__attach_uprobe_multi()

we should have done that with bpf_program__attach_kprobe_multi_opts() as well. Generally, if the only variant of API is the one with opts, then there is no point in adding "_opts" suffix to the API name, IMO

ok

Quoted 65 lines
Quoted 25 lines
tools/lib/bpf/libbpf.c +23 -0
+                                     pid_t pid,
+                                     const char *binary_path,
+                                     const char *func_pattern,
+                                     const struct bpf_uprobe_multi_opts *opts)
+{
+       const unsigned long *ref_ctr_offsets = NULL, *offsets = NULL;
+       LIBBPF_OPTS(bpf_link_create_opts, lopts);
+       unsigned long *resolved_offsets = NULL;
+       const char **resolved_symbols = NULL;
+       int err = 0, link_fd, prog_fd;
+       struct bpf_link *link = NULL;
+       char errmsg[STRERR_BUFSIZE];
+       const char *path, **syms;
+       char full_path[PATH_MAX];
+       const __u64 *cookies;
+       bool has_pattern;
+       bool retprobe;
+       size_t cnt;
+
+       if (!OPTS_VALID(opts, bpf_uprobe_multi_opts))
+               return libbpf_err_ptr(-EINVAL);
+
+       path = OPTS_GET(opts, path, NULL);

tools/lib/bpf/libbpf.c static int resolve_full_path(const char *file, char *result, size_t result_sz) · line 11439

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+       syms = OPTS_GET(opts, syms, NULL);
+       offsets = OPTS_GET(opts, offsets, NULL);
+       ref_ctr_offsets = OPTS_GET(opts, ref_ctr_offsets, NULL);
+       cookies = OPTS_GET(opts, cookies, NULL);
+       cnt = OPTS_GET(opts, cnt, 0);
+
+       /*
+        * User can specify 3 mutually exclusive set of incputs:
+        *
+        * 1) use only binary_path/func_pattern aruments

same typos

tools/lib/bpf/libbpf.c static int resolve_full_path(const char *file, char *result, size_t result_sz) · line 11449

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+        *
+        * 2) use only opts argument with allowed combinations of:
+        *    path/offsets/ref_ctr_offsets/cookies/cnt
+        *
+        * 3) use binary_path with allowed combinations of:
+        *    syms/offsets/ref_ctr_offsets/cookies/cnt
+        *
+        * Any other usage results in error.
+        */
+

Looking at this part (sorry, I already trimmed reply before realizing I want to comment on this):

(quoted diff) +2 -0
+               err = elf_find_pattern_func_offset(binary_path, func_pattern,
+                                                  &resolved_symbols,

&resolved_offsets,

tools/lib/bpf/libbpf.c +6 -0
+                                                  &cnt);
+               if (err < 0)
+                       return libbpf_err_ptr(err);
+               offsets = resolved_offsets;
+       } else if (syms) {
+               err = elf_find_multi_func_offset(binary_path, cnt,

syms, &resolved_offsets);

Quoted 4 lines

Few thoughts.

First, we are not using resolved_symbols returned from elf_find_pattern_func_offset(), do we? If not, let's drop it.

I think I used that in bpftrace with the RFC version, where I wanted to display found symbols.. but there's no use for it in libbpf for now.. will remove it

Quoted 3 lines

Second, I'm not a big fan of chosen naming. Maybe something like elf_resolve_{pattern,syms}_offsets?

sure

SNIP

Quoted 13 lines
Quoted 13 lines
tools/lib/bpf/libbpf.c +7 -0
+LIBBPF_API struct bpf_link *
+bpf_program__attach_uprobe_multi_opts(const struct bpf_program *prog,
+                                     pid_t pid,
+                                     const char *binary_path,
+                                     const char *func_pattern,
+                                     const struct bpf_uprobe_multi_opts *opts);
+
 struct bpf_ksyscall_opts {
        /* size of this struct, for forward/backward compatibility */
        size_t sz;
tools/lib/bpf/libbpf.map +0 -1
index 7521a2fb7626..81558ef1bc38 100644
--- a/tools/lib/bpf/libbpf.map
Quoted 8 lines

tools/lib/bpf/libbpf.map LIBBPF_1.2.0 { · line 393

Quoted 6 lines

+++ b/tools/lib/bpf/libbpf.map

tools/lib/bpf/libbpf.map +1 -0
@@ -390,6 +390,7 @@ LIBBPF_1.2.0 {
                bpf_link_get_info_by_fd;
                bpf_map_get_info_by_fd;
                bpf_prog_get_info_by_fd;
+               bpf_program__attach_uprobe_multi_opts;

should be in 1.3.0?

yes, will fix

thanks,
jirka

author reviewer
#40 lore raw

PATCH v2 12/24 libbpf: Add support for u[ret]probe.multi[.s] program sections

In reply to Jiri Olsa

Adding support for several uprobe_multi program sections to allow auto attach of multi_uprobe programs.

---
 tools/lib/bpf/libbpf.c | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
tools/lib/bpf/libbpf.c +40 -0
index d972cea4c658..e42080258ec7 100644
@@ -8652,6 +8652,7 @@ static int attach_tp(const struct bpf_program *prog, long cookie, struct bpf_lin
 static int attach_raw_tp(const struct bpf_program *prog, long cookie, struct bpf_link **link);
 static int attach_trace(const struct bpf_program *prog, long cookie, struct bpf_link **link);
 static int attach_kprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link);
+static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link);
 static int attach_lsm(const struct bpf_program *prog, long cookie, struct bpf_link **link);
 static int attach_iter(const struct bpf_program *prog, long cookie, struct bpf_link **link);
 
@@ -8667,6 +8668,10 @@ static const struct bpf_sec_def section_defs[] = {
 	SEC_DEF("uretprobe.s+",		KPROBE, 0, SEC_SLEEPABLE, attach_uprobe),
 	SEC_DEF("kprobe.multi+",	KPROBE,	BPF_TRACE_KPROBE_MULTI, SEC_NONE, attach_kprobe_multi),
 	SEC_DEF("kretprobe.multi+",	KPROBE,	BPF_TRACE_KPROBE_MULTI, SEC_NONE, attach_kprobe_multi),
+	SEC_DEF("uprobe.multi+",	KPROBE,	BPF_TRACE_UPROBE_MULTI, SEC_NONE, attach_uprobe_multi),
+	SEC_DEF("uretprobe.multi+",	KPROBE,	BPF_TRACE_UPROBE_MULTI, SEC_NONE, attach_uprobe_multi),
+	SEC_DEF("uprobe.multi.s+",	KPROBE,	BPF_TRACE_UPROBE_MULTI, SEC_SLEEPABLE, attach_uprobe_multi),
+	SEC_DEF("uretprobe.multi.s+",	KPROBE,	BPF_TRACE_UPROBE_MULTI, SEC_SLEEPABLE, attach_uprobe_multi),
 	SEC_DEF("ksyscall+",		KPROBE,	0, SEC_NONE, attach_ksyscall),
 	SEC_DEF("kretsyscall+",		KPROBE, 0, SEC_NONE, attach_ksyscall),
 	SEC_DEF("usdt+",		KPROBE,	0, SEC_NONE, attach_usdt),
@@ -10728,6 +10733,41 @@ static int attach_kprobe_multi(const struct bpf_program *prog, long cookie, stru
 	return libbpf_get_error(*link);
 }
 
+static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link)
+{
+	char *probe_type = NULL, *binary_path = NULL, *func_name = NULL;
+	LIBBPF_OPTS(bpf_uprobe_multi_opts, opts);
+	int n, ret = -EINVAL;
+
+	*link = NULL;
+
+	n = sscanf(prog->sec_name, "%m[^/]/%m[^:]:%ms",
+		   &probe_type, &binary_path, &func_name);
+	switch (n) {
+	case 1:
+		/* handle SEC("u[ret]probe") - format is valid, but auto-attach is impossible. */
+		ret = 0;
+		break;
+	case 2:
+		pr_warn("prog '%s': section '%s' missing ':function[+offset]' specification\n",
+			prog->name, prog->sec_name);
+		break;
+	case 3:
+		opts.retprobe = strcmp(probe_type, "uretprobe.multi");
+		*link = bpf_program__attach_uprobe_multi_opts(prog, -1, binary_path, func_name, &opts);
+		ret = libbpf_get_error(*link);
+		break;
+	default:
+		pr_warn("prog '%s': invalid format of section definition '%s'\n", prog->name,
+			prog->sec_name);
+		break;
+	}
+	free(probe_type);
+	free(binary_path);
+	free(func_name);
+	return ret;
+}
+
 static void gen_uprobe_legacy_event_name(char *buf, size_t buf_sz,
 					 const char *binary_path, uint64_t offset)
 {
Signature · 1 line
2.41.0
#41 lore raw

Re: [PATCHv2 bpf-next 12/24] libbpf: Add support for u[ret]probe.multi[.s] program sections

Quoted 43 lines · Jiri Olsa

Adding support for several uprobe_multi program sections to allow auto attach of multi_uprobe programs.

---
 tools/lib/bpf/libbpf.c | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
tools/lib/bpf/libbpf.c +12 -0
index d972cea4c658..e42080258ec7 100644
@@ -8652,6 +8652,7 @@ static int attach_tp(const struct bpf_program *prog, long cookie, struct bpf_lin
 static int attach_raw_tp(const struct bpf_program *prog, long cookie, struct bpf_link **link);
 static int attach_trace(const struct bpf_program *prog, long cookie, struct bpf_link **link);
 static int attach_kprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link);
+static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link);
 static int attach_lsm(const struct bpf_program *prog, long cookie, struct bpf_link **link);
 static int attach_iter(const struct bpf_program *prog, long cookie, struct bpf_link **link);
 
@@ -8667,6 +8668,10 @@ static const struct bpf_sec_def section_defs[] = {
        SEC_DEF("uretprobe.s+",         KPROBE, 0, SEC_SLEEPABLE, attach_uprobe),
        SEC_DEF("kprobe.multi+",        KPROBE, BPF_TRACE_KPROBE_MULTI, SEC_NONE, attach_kprobe_multi),
        SEC_DEF("kretprobe.multi+",     KPROBE, BPF_TRACE_KPROBE_MULTI, SEC_NONE, attach_kprobe_multi),
+       SEC_DEF("uprobe.multi+",        KPROBE, BPF_TRACE_UPROBE_MULTI, SEC_NONE, attach_uprobe_multi),
+       SEC_DEF("uretprobe.multi+",     KPROBE, BPF_TRACE_UPROBE_MULTI, SEC_NONE, attach_uprobe_multi),
+       SEC_DEF("uprobe.multi.s+",      KPROBE, BPF_TRACE_UPROBE_MULTI, SEC_SLEEPABLE, attach_uprobe_multi),
+       SEC_DEF("uretprobe.multi.s+",   KPROBE, BPF_TRACE_UPROBE_MULTI, SEC_SLEEPABLE, attach_uprobe_multi),
        SEC_DEF("ksyscall+",            KPROBE, 0, SEC_NONE, attach_ksyscall),
        SEC_DEF("kretsyscall+",         KPROBE, 0, SEC_NONE, attach_ksyscall),
        SEC_DEF("usdt+",                KPROBE, 0, SEC_NONE, attach_usdt),
@@ -10728,6 +10733,41 @@ static int attach_kprobe_multi(const struct bpf_program *prog, long cookie, stru
        return libbpf_get_error(*link);
 }
 
+static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link)
+{
+       char *probe_type = NULL, *binary_path = NULL, *func_name = NULL;
+       LIBBPF_OPTS(bpf_uprobe_multi_opts, opts);
+       int n, ret = -EINVAL;
+
+       *link = NULL;

tools/lib/bpf/libbpf.c static int attach_kprobe_multi(const struct bpf_program *prog, long cookie, stru · line 10752

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+
+       n = sscanf(prog->sec_name, "%m[^/]/%m[^:]:%ms",
+                  &probe_type, &binary_path, &func_name);
+       switch (n) {
+       case 1:
+               /* handle SEC("u[ret]probe") - format is valid, but auto-attach is impossible. */
+               ret = 0;
+               break;
+       case 2:
+               pr_warn("prog '%s': section '%s' missing ':function[+offset]' specification\n",

message copy/pasted? We don't support +offset part, and it's not "function", but "pattern" or something along those lines?

tools/lib/bpf/libbpf.c static int attach_kprobe_multi(const struct bpf_program *prog, long cookie, stru · line 10756

Quoted 4 lines
tools/lib/bpf/libbpf.c +4 -0
+                       prog->name, prog->sec_name);
+               break;
+       case 3:
+               opts.retprobe = strcmp(probe_type, "uretprobe.multi");

== 0

Quoted 20 lines
tools/lib/bpf/libbpf.c +14 -1
+               *link = bpf_program__attach_uprobe_multi_opts(prog, -1, binary_path, func_name, &opts);
+               ret = libbpf_get_error(*link);
+               break;
+       default:
+               pr_warn("prog '%s': invalid format of section definition '%s'\n", prog->name,
+                       prog->sec_name);
+               break;
+       }
+       free(probe_type);
+       free(binary_path);
+       free(func_name);
+       return ret;
+}
+
 static void gen_uprobe_legacy_event_name(char *buf, size_t buf_sz,
                                         const char *binary_path, uint64_t offset)
 {
--

2.41.0

#42 lore raw

Re: [PATCHv2 bpf-next 12/24] libbpf: Add support for u[ret]probe.multi[.s] program sections

Quoted 49 lines · Andrii Nakryiko
Quoted 48 lines · Jiri Olsa

Adding support for several uprobe_multi program sections to allow auto attach of multi_uprobe programs.

---
 tools/lib/bpf/libbpf.c | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
tools/lib/bpf/libbpf.c +17 -0
index d972cea4c658..e42080258ec7 100644
@@ -8652,6 +8652,7 @@ static int attach_tp(const struct bpf_program *prog, long cookie, struct bpf_lin
 static int attach_raw_tp(const struct bpf_program *prog, long cookie, struct bpf_link **link);
 static int attach_trace(const struct bpf_program *prog, long cookie, struct bpf_link **link);
 static int attach_kprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link);
+static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link);
 static int attach_lsm(const struct bpf_program *prog, long cookie, struct bpf_link **link);
 static int attach_iter(const struct bpf_program *prog, long cookie, struct bpf_link **link);
 
@@ -8667,6 +8668,10 @@ static const struct bpf_sec_def section_defs[] = {
        SEC_DEF("uretprobe.s+",         KPROBE, 0, SEC_SLEEPABLE, attach_uprobe),
        SEC_DEF("kprobe.multi+",        KPROBE, BPF_TRACE_KPROBE_MULTI, SEC_NONE, attach_kprobe_multi),
        SEC_DEF("kretprobe.multi+",     KPROBE, BPF_TRACE_KPROBE_MULTI, SEC_NONE, attach_kprobe_multi),
+       SEC_DEF("uprobe.multi+",        KPROBE, BPF_TRACE_UPROBE_MULTI, SEC_NONE, attach_uprobe_multi),
+       SEC_DEF("uretprobe.multi+",     KPROBE, BPF_TRACE_UPROBE_MULTI, SEC_NONE, attach_uprobe_multi),
+       SEC_DEF("uprobe.multi.s+",      KPROBE, BPF_TRACE_UPROBE_MULTI, SEC_SLEEPABLE, attach_uprobe_multi),
+       SEC_DEF("uretprobe.multi.s+",   KPROBE, BPF_TRACE_UPROBE_MULTI, SEC_SLEEPABLE, attach_uprobe_multi),
        SEC_DEF("ksyscall+",            KPROBE, 0, SEC_NONE, attach_ksyscall),
        SEC_DEF("kretsyscall+",         KPROBE, 0, SEC_NONE, attach_ksyscall),
        SEC_DEF("usdt+",                KPROBE, 0, SEC_NONE, attach_usdt),
@@ -10728,6 +10733,41 @@ static int attach_kprobe_multi(const struct bpf_program *prog, long cookie, stru
        return libbpf_get_error(*link);
 }
 
+static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link)
+{
+       char *probe_type = NULL, *binary_path = NULL, *func_name = NULL;
+       LIBBPF_OPTS(bpf_uprobe_multi_opts, opts);
+       int n, ret = -EINVAL;
+
+       *link = NULL;
+
+       n = sscanf(prog->sec_name, "%m[^/]/%m[^:]:%ms",
+                  &probe_type, &binary_path, &func_name);
+       switch (n) {
+       case 1:
Quoted 8 lines

tools/lib/bpf/libbpf.c static int attach_kprobe_multi(const struct bpf_program *prog, long cookie, stru · line 10752

Quoted 5 lines
tools/lib/bpf/libbpf.c +5 -0
+               /* handle SEC("u[ret]probe") - format is valid, but auto-attach is impossible. */
+               ret = 0;
+               break;
+       case 2:
+               pr_warn("prog '%s': section '%s' missing ':function[+offset]' specification\n",

message copy/pasted? We don't support +offset part, and it's not "function", but "pattern" or something along those lines?

yes, will fix

Quoted 7 lines

tools/lib/bpf/libbpf.c static int attach_kprobe_multi(const struct bpf_program *prog, long cookie, stru · line 10756

Quoted 5 lines
tools/lib/bpf/libbpf.c +4 -0
+                       prog->name, prog->sec_name);
+               break;
+       case 3:
+               opts.retprobe = strcmp(probe_type, "uretprobe.multi");

== 0

ok, thanks

jirka

Quoted 22 lines
Quoted 22 lines
tools/lib/bpf/libbpf.c +14 -1
+               *link = bpf_program__attach_uprobe_multi_opts(prog, -1, binary_path, func_name, &opts);
+               ret = libbpf_get_error(*link);
+               break;
+       default:
+               pr_warn("prog '%s': invalid format of section definition '%s'\n", prog->name,
+                       prog->sec_name);
+               break;
+       }
+       free(probe_type);
+       free(binary_path);
+       free(func_name);
+       return ret;
+}
+
 static void gen_uprobe_legacy_event_name(char *buf, size_t buf_sz,
                                         const char *binary_path, uint64_t offset)
 {
--

2.41.0

author reviewer
#43 lore raw

PATCH v2 13/24 libbpf: Add uprobe multi link detection

In reply to Jiri Olsa

Adding uprobe-multi link detection. It will be used later in bpf_program__attach_usdt function to check and use uprobe_multi link over standard uprobe links.

---
 tools/lib/bpf/libbpf.c          | 29 +++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf_internal.h |  2 ++
 2 files changed, 31 insertions(+)
tools/lib/bpf/libbpf.c +29 -0
index e42080258ec7..3d570898459e 100644
@@ -4815,6 +4815,32 @@ static int probe_perf_link(void)
 	return link_fd < 0 && err == -EBADF;
 }
 
+static int probe_uprobe_multi_link(void)
+{
+	struct bpf_insn insns[] = {
+		BPF_MOV64_IMM(BPF_REG_0, 0),
+		BPF_EXIT_INSN(),
+	};
+	int prog_fd, link_fd, err;
+
+	prog_fd = bpf_prog_load(BPF_PROG_TYPE_KPROBE, NULL, "GPL",
+				insns, ARRAY_SIZE(insns), NULL);
+	if (prog_fd < 0)
+		return -errno;
+
+	/* No need to specify attach function. If the link is not supported
+	 * we will get -EOPNOTSUPP error before any other check is performed.
+	 */
+	link_fd = bpf_link_create(prog_fd, -1, BPF_TRACE_UPROBE_MULTI, NULL);
+	err = -errno; /* close() can clobber errno */
+
+	if (link_fd >= 0)
+		close(link_fd);
+	close(prog_fd);
+
+	return link_fd < 0 && err != -EOPNOTSUPP;
+}
+
 static int probe_kern_bpf_cookie(void)
 {
 	struct bpf_insn insns[] = {
@@ -4911,6 +4937,9 @@ static struct kern_feature_desc {
 	[FEAT_SYSCALL_WRAPPER] = {
 		"Kernel using syscall wrapper", probe_kern_syscall_wrapper,
 	},
+	[FEAT_UPROBE_LINK] = {
+		"BPF uprobe multi link support", probe_uprobe_multi_link,
+	},
 };
 
 bool kernel_supports(const struct bpf_object *obj, enum kern_feature_id feat_id)
tools/lib/bpf/libbpf_internal.h +2 -0
index 22b0834e7fe1..a257eb81af25 100644
@@ -354,6 +354,8 @@ enum kern_feature_id {
 	FEAT_BTF_ENUM64,
 	/* Kernel uses syscall wrapper (CONFIG_ARCH_HAS_SYSCALL_WRAPPER) */
 	FEAT_SYSCALL_WRAPPER,
+	/* BPF uprobe_multi link support */
+	FEAT_UPROBE_LINK,
 	__FEAT_CNT,
 };
 
Signature · 1 line
2.41.0
#44 lore raw

Re: [PATCHv2 bpf-next 13/24] libbpf: Add uprobe multi link detection

Quoted 19 lines · Jiri Olsa

Adding uprobe-multi link detection. It will be used later in bpf_program__attach_usdt function to check and use uprobe_multi link over standard uprobe links.

---
 tools/lib/bpf/libbpf.c          | 29 +++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf_internal.h |  2 ++
 2 files changed, 31 insertions(+)
tools/lib/bpf/libbpf.c +0 -0
index e42080258ec7..3d570898459e 100644
@@ -4815,6 +4815,32 @@ static int probe_perf_link(void)
        return link_fd < 0 && err == -EBADF;
 }
 

tools/lib/bpf/libbpf.c static int probe_perf_link(void) · line 4827

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+static int probe_uprobe_multi_link(void)
+{
+       struct bpf_insn insns[] = {
+               BPF_MOV64_IMM(BPF_REG_0, 0),
+               BPF_EXIT_INSN(),
+       };
+       int prog_fd, link_fd, err;
+
+       prog_fd = bpf_prog_load(BPF_PROG_TYPE_KPROBE, NULL, "GPL",
+                               insns, ARRAY_SIZE(insns), NULL);

I thought we needed to specify expected_attach_type (BPF_TRACE_UPROBE_MULTI)?

tools/lib/bpf/libbpf.c static int probe_perf_link(void) · line 4832

Quoted 5 lines
tools/lib/bpf/libbpf.c +5 -0
+       if (prog_fd < 0)
+               return -errno;
+
+       /* No need to specify attach function. If the link is not supported
+        * we will get -EOPNOTSUPP error before any other check is performed.

what will actually return this -EOPNOTSUPP? I couldn't find this in the code quickly, can you please point me where?

Quoted 29 lines
tools/lib/bpf/libbpf.c +14 -0
+        */
+       link_fd = bpf_link_create(prog_fd, -1, BPF_TRACE_UPROBE_MULTI, NULL);
+       err = -errno; /* close() can clobber errno */
+
+       if (link_fd >= 0)
+               close(link_fd);
+       close(prog_fd);
+
+       return link_fd < 0 && err != -EOPNOTSUPP;
+}
+
 static int probe_kern_bpf_cookie(void)
 {
        struct bpf_insn insns[] = {
@@ -4911,6 +4937,9 @@ static struct kern_feature_desc {
        [FEAT_SYSCALL_WRAPPER] = {
                "Kernel using syscall wrapper", probe_kern_syscall_wrapper,
        },
+       [FEAT_UPROBE_LINK] = {
+               "BPF uprobe multi link support", probe_uprobe_multi_link,
+       },
 };
 
 bool kernel_supports(const struct bpf_object *obj, enum kern_feature_id feat_id)
tools/lib/bpf/libbpf_internal.h +0 -0
index 22b0834e7fe1..a257eb81af25 100644

tools/lib/bpf/libbpf_internal.h enum kern_feature_id { · line 358

Quoted 5 lines
tools/lib/bpf/libbpf_internal.h +2 -0
@@ -354,6 +354,8 @@ enum kern_feature_id {
        FEAT_BTF_ENUM64,
        /* Kernel uses syscall wrapper (CONFIG_ARCH_HAS_SYSCALL_WRAPPER) */
        FEAT_SYSCALL_WRAPPER,
+       /* BPF uprobe_multi link support */
+       FEAT_UPROBE_LINK,

UPROBE_MULTI_LINK, we might have non-multi link in the future as well

Quoted 8 lines
tools/lib/bpf/libbpf_internal.h +0 -1
        __FEAT_CNT,
 };
 
 
 
--

2.41.0

#45 lore raw

Re: [PATCHv2 bpf-next 13/24] libbpf: Add uprobe multi link detection

Quoted 20 lines · Andrii Nakryiko
Quoted 19 lines · Jiri Olsa

Adding uprobe-multi link detection. It will be used later in bpf_program__attach_usdt function to check and use uprobe_multi link over standard uprobe links.

---
 tools/lib/bpf/libbpf.c          | 29 +++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf_internal.h |  2 ++
 2 files changed, 31 insertions(+)
tools/lib/bpf/libbpf.c +0 -0
index e42080258ec7..3d570898459e 100644
@@ -4815,6 +4815,32 @@ static int probe_perf_link(void)
        return link_fd < 0 && err == -EBADF;
 }
 
Quoted 12 lines

tools/lib/bpf/libbpf.c static int probe_perf_link(void) · line 4827

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+static int probe_uprobe_multi_link(void)
+{
+       struct bpf_insn insns[] = {
+               BPF_MOV64_IMM(BPF_REG_0, 0),
+               BPF_EXIT_INSN(),
+       };
+       int prog_fd, link_fd, err;
+
+       prog_fd = bpf_prog_load(BPF_PROG_TYPE_KPROBE, NULL, "GPL",
+                               insns, ARRAY_SIZE(insns), NULL);

I thought we needed to specify expected_attach_type (BPF_TRACE_UPROBE_MULTI)?

hm it should.. I guess it worked because of the KPROBE/UPROBE typo you found in the patch earlier, will check

Quoted 9 lines

tools/lib/bpf/libbpf.c static int probe_perf_link(void) · line 4832

Quoted 6 lines
tools/lib/bpf/libbpf.c +5 -0
+       if (prog_fd < 0)
+               return -errno;
+
+       /* No need to specify attach function. If the link is not supported
+        * we will get -EOPNOTSUPP error before any other check is performed.

what will actually return this -EOPNOTSUPP? I couldn't find this in the code quickly, can you please point me where?

	#else /* !CONFIG_UPROBES */
	int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
	{
		return -EOPNOTSUPP;
	}
Quoted 24 lines
Quoted 24 lines
tools/lib/bpf/libbpf.c +14 -0
+        */
+       link_fd = bpf_link_create(prog_fd, -1, BPF_TRACE_UPROBE_MULTI, NULL);
+       err = -errno; /* close() can clobber errno */
+
+       if (link_fd >= 0)
+               close(link_fd);
+       close(prog_fd);
+
+       return link_fd < 0 && err != -EOPNOTSUPP;
+}
+
 static int probe_kern_bpf_cookie(void)
 {
        struct bpf_insn insns[] = {
@@ -4911,6 +4937,9 @@ static struct kern_feature_desc {
        [FEAT_SYSCALL_WRAPPER] = {
                "Kernel using syscall wrapper", probe_kern_syscall_wrapper,
        },
+       [FEAT_UPROBE_LINK] = {
+               "BPF uprobe multi link support", probe_uprobe_multi_link,
+       },
 };
 
Quoted 13 lines

tools/lib/bpf/libbpf_internal.h enum kern_feature_id { · line 358

Quoted 11 lines
tools/lib/bpf/libbpf.c +0 -0
 bool kernel_supports(const struct bpf_object *obj, enum kern_feature_id feat_id)
tools/lib/bpf/libbpf_internal.h +2 -0
index 22b0834e7fe1..a257eb81af25 100644
@@ -354,6 +354,8 @@ enum kern_feature_id {
        FEAT_BTF_ENUM64,
        /* Kernel uses syscall wrapper (CONFIG_ARCH_HAS_SYSCALL_WRAPPER) */
        FEAT_SYSCALL_WRAPPER,
+       /* BPF uprobe_multi link support */
+       FEAT_UPROBE_LINK,

UPROBE_MULTI_LINK, we might have non-multi link in the future as well

ok

thanks,
jirka

#46 lore raw

Re: [PATCHv2 bpf-next 13/24] libbpf: Add uprobe multi link detection

Quoted 45 lines · Jiri Olsa
Quoted 20 lines · Andrii Nakryiko
Quoted 19 lines · Jiri Olsa

Adding uprobe-multi link detection. It will be used later in bpf_program__attach_usdt function to check and use uprobe_multi link over standard uprobe links.

---
 tools/lib/bpf/libbpf.c          | 29 +++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf_internal.h |  2 ++
 2 files changed, 31 insertions(+)
tools/lib/bpf/libbpf.c +0 -0
index e42080258ec7..3d570898459e 100644
@@ -4815,6 +4815,32 @@ static int probe_perf_link(void)
        return link_fd < 0 && err == -EBADF;
 }
 
Quoted 12 lines

tools/lib/bpf/libbpf.c static int probe_perf_link(void) · line 4827

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+static int probe_uprobe_multi_link(void)
+{
+       struct bpf_insn insns[] = {
+               BPF_MOV64_IMM(BPF_REG_0, 0),
+               BPF_EXIT_INSN(),
+       };
+       int prog_fd, link_fd, err;
+
+       prog_fd = bpf_prog_load(BPF_PROG_TYPE_KPROBE, NULL, "GPL",
+                               insns, ARRAY_SIZE(insns), NULL);

I thought we needed to specify expected_attach_type (BPF_TRACE_UPROBE_MULTI)?

hm it should.. I guess it worked because of the KPROBE/UPROBE typo you found in the patch earlier, will check

Quoted 7 lines
Quoted 6 lines
tools/lib/bpf/libbpf.c +5 -0
+       if (prog_fd < 0)
+               return -errno;
+
+       /* No need to specify attach function. If the link is not supported
+        * we will get -EOPNOTSUPP error before any other check is performed.
Quoted 8 lines
Quoted 2 lines

what will actually return this -EOPNOTSUPP? I couldn't find this in the code quickly, can you please point me where?

        #else /* !CONFIG_UPROBES */
        int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
        {
                return -EOPNOTSUPP;
        }

that's a new code in new kernel that doesn't have CONFIG_UPROBES. What about old kernels? They will return -EINVAL, no? Which we will assume means "yep, multi-uprobe BPF link is supported", which would be wrong. Or am I missing something?

Quoted 43 lines
Quoted 25 lines
Quoted 25 lines
tools/lib/bpf/libbpf.c +14 -0
+        */
+       link_fd = bpf_link_create(prog_fd, -1, BPF_TRACE_UPROBE_MULTI, NULL);
+       err = -errno; /* close() can clobber errno */
+
+       if (link_fd >= 0)
+               close(link_fd);
+       close(prog_fd);
+
+       return link_fd < 0 && err != -EOPNOTSUPP;
+}
+
 static int probe_kern_bpf_cookie(void)
 {
        struct bpf_insn insns[] = {
@@ -4911,6 +4937,9 @@ static struct kern_feature_desc {
        [FEAT_SYSCALL_WRAPPER] = {
                "Kernel using syscall wrapper", probe_kern_syscall_wrapper,
        },
+       [FEAT_UPROBE_LINK] = {
+               "BPF uprobe multi link support", probe_uprobe_multi_link,
+       },
 };
 
Quoted 13 lines

tools/lib/bpf/libbpf_internal.h enum kern_feature_id { · line 358

Quoted 11 lines
tools/lib/bpf/libbpf.c +0 -0
 bool kernel_supports(const struct bpf_object *obj, enum kern_feature_id feat_id)
tools/lib/bpf/libbpf_internal.h +2 -0
index 22b0834e7fe1..a257eb81af25 100644
@@ -354,6 +354,8 @@ enum kern_feature_id {
        FEAT_BTF_ENUM64,
        /* Kernel uses syscall wrapper (CONFIG_ARCH_HAS_SYSCALL_WRAPPER) */
        FEAT_SYSCALL_WRAPPER,
+       /* BPF uprobe_multi link support */
+       FEAT_UPROBE_LINK,

UPROBE_MULTI_LINK, we might have non-multi link in the future as well

ok

thanks,
jirka

#47 lore raw

Re: [PATCHv2 bpf-next 13/24] libbpf: Add uprobe multi link detection

On Mon, Jun 26, 2023 at 11:21:40AM -0700, Andrii Nakryiko wrote:

SNIP

Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
tools/lib/bpf/libbpf.c +5 -0
+       if (prog_fd < 0)
+               return -errno;
+
+       /* No need to specify attach function. If the link is not supported
+        * we will get -EOPNOTSUPP error before any other check is performed.
Quoted 8 lines
Quoted 2 lines

what will actually return this -EOPNOTSUPP? I couldn't find this in the code quickly, can you please point me where?

        #else /* !CONFIG_UPROBES */
        int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
        {
                return -EOPNOTSUPP;
        }
Quoted 4 lines

that's a new code in new kernel that doesn't have CONFIG_UPROBES. What about old kernels? They will return -EINVAL, no? Which we will assume means "yep, multi-uprobe BPF link is supported", which would be wrong. Or am I missing something?

ah nope, I'm missing old kernel case.. will check and fix that

jirka

Quoted 44 lines
Quoted 44 lines
Quoted 26 lines
Quoted 26 lines
tools/lib/bpf/libbpf.c +14 -0
+        */
+       link_fd = bpf_link_create(prog_fd, -1, BPF_TRACE_UPROBE_MULTI, NULL);
+       err = -errno; /* close() can clobber errno */
+
+       if (link_fd >= 0)
+               close(link_fd);
+       close(prog_fd);
+
+       return link_fd < 0 && err != -EOPNOTSUPP;
+}
+
 static int probe_kern_bpf_cookie(void)
 {
        struct bpf_insn insns[] = {
@@ -4911,6 +4937,9 @@ static struct kern_feature_desc {
        [FEAT_SYSCALL_WRAPPER] = {
                "Kernel using syscall wrapper", probe_kern_syscall_wrapper,
        },
+       [FEAT_UPROBE_LINK] = {
+               "BPF uprobe multi link support", probe_uprobe_multi_link,
+       },
 };
 
Quoted 13 lines

tools/lib/bpf/libbpf_internal.h enum kern_feature_id { · line 358

Quoted 11 lines
tools/lib/bpf/libbpf.c +0 -0
 bool kernel_supports(const struct bpf_object *obj, enum kern_feature_id feat_id)
tools/lib/bpf/libbpf_internal.h +2 -0
index 22b0834e7fe1..a257eb81af25 100644
@@ -354,6 +354,8 @@ enum kern_feature_id {
        FEAT_BTF_ENUM64,
        /* Kernel uses syscall wrapper (CONFIG_ARCH_HAS_SYSCALL_WRAPPER) */
        FEAT_SYSCALL_WRAPPER,
+       /* BPF uprobe_multi link support */
+       FEAT_UPROBE_LINK,

UPROBE_MULTI_LINK, we might have non-multi link in the future as well

ok

thanks,
jirka

author reviewer
#48 lore raw

PATCH v2 14/24 libbpf: Add uprobe multi link support to bpf_program__attach_usdt

In reply to Jiri Olsa

Adding support for usdt_manager_attach_usdt to use uprobe_multi link to attach to usdt probes.

The uprobe_multi support is detected before the usdt program is loaded and its expected_attach_type is set accordingly.

If uprobe_multi support is detected the usdt_manager_attach_usdt gathers uprobes info and calls bpf_program__attach_uprobe_opts to create all needed uprobes.

If uprobe_multi support is not detected the old behaviour stays.

---
 tools/lib/bpf/libbpf.c |  12 ++++-
 tools/lib/bpf/usdt.c   | 120 ++++++++++++++++++++++++++++++-----------
 2 files changed, 99 insertions(+), 33 deletions(-)
tools/lib/bpf/libbpf.c +10 -2
index 3d570898459e..9c7a67c5cbe8 100644
@@ -363,6 +363,8 @@ enum sec_def_flags {
 	SEC_SLEEPABLE = 8,
 	/* BPF program support non-linear XDP buffer */
 	SEC_XDP_FRAGS = 16,
+	/* Setup proper attach type for usdt probes. */
+	SEC_USDT = 32,
 };
 
 struct bpf_sec_def {
@@ -6799,6 +6801,10 @@ static int libbpf_prepare_prog_load(struct bpf_program *prog,
 	if (prog->type == BPF_PROG_TYPE_XDP && (def & SEC_XDP_FRAGS))
 		opts->prog_flags |= BPF_F_XDP_HAS_FRAGS;
 
+	/* special check for usdt to use uprobe_multi link */
+	if ((def & SEC_USDT) && kernel_supports(NULL, FEAT_UPROBE_LINK))
+		prog->expected_attach_type = BPF_TRACE_UPROBE_MULTI;
+
 	if ((def & SEC_ATTACH_BTF) && !prog->attach_btf_id) {
 		int btf_obj_fd = 0, btf_type_id = 0, err;
 		const char *attach_name;
@@ -6867,7 +6873,6 @@ static int bpf_object_load_prog(struct bpf_object *obj, struct bpf_program *prog
 	if (!insns || !insns_cnt)
 		return -EINVAL;
 
-	load_attr.expected_attach_type = prog->expected_attach_type;
 	if (kernel_supports(obj, FEAT_PROG_NAME))
 		prog_name = prog->name;
 	load_attr.attach_prog_fd = prog->attach_prog_fd;
@@ -6903,6 +6908,9 @@ static int bpf_object_load_prog(struct bpf_object *obj, struct bpf_program *prog
 		insns_cnt = prog->insns_cnt;
 	}
 
+	/* allow prog_prepare_load_fn to change expected_attach_type */
+	load_attr.expected_attach_type = prog->expected_attach_type;
+
 	if (obj->gen_loader) {
 		bpf_gen__prog_load(obj->gen_loader, prog->type, prog->name,
 				   license, insns, insns_cnt, &load_attr,
@@ -8703,7 +8711,7 @@ static const struct bpf_sec_def section_defs[] = {
 	SEC_DEF("uretprobe.multi.s+",	KPROBE,	BPF_TRACE_UPROBE_MULTI, SEC_SLEEPABLE, attach_uprobe_multi),
 	SEC_DEF("ksyscall+",		KPROBE,	0, SEC_NONE, attach_ksyscall),
 	SEC_DEF("kretsyscall+",		KPROBE, 0, SEC_NONE, attach_ksyscall),
-	SEC_DEF("usdt+",		KPROBE,	0, SEC_NONE, attach_usdt),
+	SEC_DEF("usdt+",		KPROBE,	0, SEC_USDT, attach_usdt),
 	SEC_DEF("tc",			SCHED_CLS, 0, SEC_NONE),
 	SEC_DEF("classifier",		SCHED_CLS, 0, SEC_NONE),
 	SEC_DEF("action",		SCHED_ACT, 0, SEC_NONE),
tools/lib/bpf/usdt.c +89 -31

170 lines not on this page · raw message

Signature · 1 line
2.41.0
#49 lore raw

Re: [PATCHv2 bpf-next 14/24] libbpf: Add uprobe multi link support to bpf_program__attach_usdt

Quoted 33 lines · Jiri Olsa

Adding support for usdt_manager_attach_usdt to use uprobe_multi link to attach to usdt probes.

The uprobe_multi support is detected before the usdt program is loaded and its expected_attach_type is set accordingly.

If uprobe_multi support is detected the usdt_manager_attach_usdt gathers uprobes info and calls bpf_program__attach_uprobe_opts to create all needed uprobes.

If uprobe_multi support is not detected the old behaviour stays.

---
 tools/lib/bpf/libbpf.c |  12 ++++-
 tools/lib/bpf/usdt.c   | 120 ++++++++++++++++++++++++++++++-----------
 2 files changed, 99 insertions(+), 33 deletions(-)
tools/lib/bpf/libbpf.c +2 -0
index 3d570898459e..9c7a67c5cbe8 100644
@@ -363,6 +363,8 @@ enum sec_def_flags {
        SEC_SLEEPABLE = 8,
        /* BPF program support non-linear XDP buffer */
        SEC_XDP_FRAGS = 16,
+       /* Setup proper attach type for usdt probes. */
+       SEC_USDT = 32,
 };
 
 struct bpf_sec_def {

tools/lib/bpf/libbpf.c static int libbpf_prepare_prog_load(struct bpf_program *prog, · line 6806

Quoted 6 lines
tools/lib/bpf/libbpf.c +3 -0
@@ -6799,6 +6801,10 @@ static int libbpf_prepare_prog_load(struct bpf_program *prog,
        if (prog->type == BPF_PROG_TYPE_XDP && (def & SEC_XDP_FRAGS))
                opts->prog_flags |= BPF_F_XDP_HAS_FRAGS;
 
+       /* special check for usdt to use uprobe_multi link */
+       if ((def & SEC_USDT) && kernel_supports(NULL, FEAT_UPROBE_LINK))
+               prog->expected_attach_type = BPF_TRACE_UPROBE_MULTI;

this is quite ugly. I think KPROBE programs do not have enforcement for expected_attach_type during BPF_PROG_LOAD, so we can set BPF_TRACE_UPROBE_MULTI unconditionally, right?

Quoted 23 lines
tools/lib/bpf/libbpf.c +4 -1
+
        if ((def & SEC_ATTACH_BTF) && !prog->attach_btf_id) {
                int btf_obj_fd = 0, btf_type_id = 0, err;
                const char *attach_name;
@@ -6867,7 +6873,6 @@ static int bpf_object_load_prog(struct bpf_object *obj, struct bpf_program *prog
        if (!insns || !insns_cnt)
                return -EINVAL;
 
-       load_attr.expected_attach_type = prog->expected_attach_type;
        if (kernel_supports(obj, FEAT_PROG_NAME))
                prog_name = prog->name;
        load_attr.attach_prog_fd = prog->attach_prog_fd;
@@ -6903,6 +6908,9 @@ static int bpf_object_load_prog(struct bpf_object *obj, struct bpf_program *prog
                insns_cnt = prog->insns_cnt;
        }
 
+       /* allow prog_prepare_load_fn to change expected_attach_type */
+       load_attr.expected_attach_type = prog->expected_attach_type;
+
        if (obj->gen_loader) {
                bpf_gen__prog_load(obj->gen_loader, prog->type, prog->name,
                                   license, insns, insns_cnt, &load_attr,

tools/lib/bpf/libbpf.c static const struct bpf_sec_def section_defs[] = { · line 8714

Quoted 5 lines
tools/lib/bpf/libbpf.c +1 -1
@@ -8703,7 +8711,7 @@ static const struct bpf_sec_def section_defs[] = {
        SEC_DEF("uretprobe.multi.s+",   KPROBE, BPF_TRACE_UPROBE_MULTI, SEC_SLEEPABLE, attach_uprobe_multi),
        SEC_DEF("ksyscall+",            KPROBE, 0, SEC_NONE, attach_ksyscall),
        SEC_DEF("kretsyscall+",         KPROBE, 0, SEC_NONE, attach_ksyscall),
-       SEC_DEF("usdt+",                KPROBE, 0, SEC_NONE, attach_usdt),
+       SEC_DEF("usdt+",                KPROBE, 0, SEC_USDT, attach_usdt),

btw, given you are touching USDT stuff, can you please also add sleepable USDT (usdt.s+) ?

Quoted 9 lines
tools/lib/bpf/libbpf.c +0 -0
        SEC_DEF("tc",                   SCHED_CLS, 0, SEC_NONE),
        SEC_DEF("classifier",           SCHED_CLS, 0, SEC_NONE),
        SEC_DEF("action",               SCHED_ACT, 0, SEC_NONE),
tools/lib/bpf/usdt.c +0 -0
index f1a141555f08..33f0a2b4cc1c 100644
@@ -808,6 +808,16 @@ struct bpf_link_usdt {
                long abs_ip;

tools/lib/bpf/usdt.c struct bpf_link_usdt { · line 818

Quoted 10 lines
tools/lib/bpf/usdt.c +8 -0
                struct bpf_link *link;
        } *uprobes;
+
+       bool has_uprobe_multi;
+
+       struct {
+               char *path;
+               unsigned long *offsets;
+               unsigned long *ref_ctr_offsets;
+               __u64 *cookies;

you shouldn't need to persist this, this can be allocated and freed inside usdt_manager_attach_usdt(), you only need the link pointer

Quoted 30 lines
tools/lib/bpf/usdt.c +10 -13
+               struct bpf_link *link;
+       } uprobe_multi;
 };
 
 static int bpf_link_usdt_detach(struct bpf_link *link)
@@ -816,19 +826,23 @@ static int bpf_link_usdt_detach(struct bpf_link *link)
        struct usdt_manager *man = usdt_link->usdt_man;
        int i;
 
-       for (i = 0; i < usdt_link->uprobe_cnt; i++) {
-               /* detach underlying uprobe link */
-               bpf_link__destroy(usdt_link->uprobes[i].link);
-               /* there is no need to update specs map because it will be
-                * unconditionally overwritten on subsequent USDT attaches,
-                * but if BPF cookies are not used we need to remove entry
-                * from ip_to_spec_id map, otherwise we'll run into false
-                * conflicting IP errors
-                */
-               if (!man->has_bpf_cookie) {
-                       /* not much we can do about errors here */
-                       (void)bpf_map_delete_elem(bpf_map__fd(man->ip_to_spec_id_map),
-                                                 &usdt_link->uprobes[i].abs_ip);
+       if (usdt_link->has_uprobe_multi) {
+               bpf_link__destroy(usdt_link->uprobe_multi.link);
+       } else {
+               for (i = 0; i < usdt_link->uprobe_cnt; i++) {
+                       /* detach underlying uprobe link */
+                       bpf_link__destroy(usdt_link->uprobes[i].link);
+                       /* there is no need to update specs map because it will be
+                        * unconditionally overwritten on subsequent USDT attaches,

tools/lib/bpf/usdt.c static int bpf_link_usdt_detach(struct bpf_link *link) · line 846

Quoted 10 lines
tools/lib/bpf/usdt.c +9 -0
+                        * but if BPF cookies are not used we need to remove entry
+                        * from ip_to_spec_id map, otherwise we'll run into false
+                        * conflicting IP errors
+                        */
+                       if (!man->has_bpf_cookie) {
+                               /* not much we can do about errors here */
+                               (void)bpf_map_delete_elem(bpf_map__fd(man->ip_to_spec_id_map),
+                                                         &usdt_link->uprobes[i].abs_ip);
+                       }
                }

you can avoid shifting all this by keeping uprobe_cnt to zero

bpf_link__destory(usdt_link->uprobe_multi.link) will work fine for NULL

so just do both clean ups sequentially, knowing that only one of them will actually do anything

Quoted 8 lines
tools/lib/bpf/usdt.c +0 -2
        }
 
@@ -868,9 +882,15 @@ static void bpf_link_usdt_dealloc(struct bpf_link *link)
 {
        struct bpf_link_usdt *usdt_link = container_of(link, struct bpf_link_usdt, link);
 
-       free(usdt_link->spec_ids);
-       free(usdt_link->uprobes);

tools/lib/bpf/usdt.c static void bpf_link_usdt_dealloc(struct bpf_link *link) · line 893

Quoted 10 lines
tools/lib/bpf/usdt.c +9 -1
-       free(usdt_link);
+       if (usdt_link->has_uprobe_multi) {
+               free(usdt_link->uprobe_multi.offsets);
+               free(usdt_link->uprobe_multi.ref_ctr_offsets);
+               free(usdt_link->uprobe_multi.cookies);
+       } else {
+               free(usdt_link->spec_ids);
+               free(usdt_link->uprobes);
+               free(usdt_link);
+       }

similar to the above, just do *all* the clean up unconditionally and rely on free() handling NULLs just fine

Quoted 18 lines
tools/lib/bpf/usdt.c +2 -0
 }
 
 static size_t specs_hash_fn(long key, void *ctx)
@@ -943,11 +963,13 @@ struct bpf_link *usdt_manager_attach_usdt(struct usdt_manager *man, const struct
                                          const char *usdt_provider, const char *usdt_name,
                                          __u64 usdt_cookie)
 {
+       LIBBPF_OPTS(bpf_uprobe_multi_opts, opts_multi);
        int i, fd, err, spec_map_fd, ip_map_fd;
        LIBBPF_OPTS(bpf_uprobe_opts, opts);
        struct hashmap *specs_hash = NULL;
        struct bpf_link_usdt *link = NULL;
        struct usdt_target *targets = NULL;
+       struct bpf_link *uprobe_link;
        size_t target_cnt;
        Elf *elf;
 

tools/lib/bpf/usdt.c struct bpf_link *usdt_manager_attach_usdt(struct usdt_manager *man, const struct · line 1028

Quoted 4 lines
tools/lib/bpf/usdt.c +1 -0
@@ -1003,16 +1025,29 @@ struct bpf_link *usdt_manager_attach_usdt(struct usdt_manager *man, const struct
        link->usdt_man = man;
        link->link.detach = &bpf_link_usdt_detach;
        link->link.dealloc = &bpf_link_usdt_dealloc;
+       link->has_uprobe_multi = bpf_program__expected_attach_type(prog) == BPF_TRACE_UPROBE_MULTI;

just use kernel_supports(), it's cleaner (and result is cached, so it's not less efficient)

Quoted 24 lines
tools/lib/bpf/usdt.c +17 -4
-       link->uprobes = calloc(target_cnt, sizeof(*link->uprobes));
-       if (!link->uprobes) {
-               err = -ENOMEM;
-               goto err_out;
+       if (link->has_uprobe_multi) {
+               link->uprobe_multi.offsets = calloc(target_cnt, sizeof(*link->uprobe_multi.offsets));
+               link->uprobe_multi.ref_ctr_offsets = calloc(target_cnt, sizeof(*link->uprobe_multi.ref_ctr_offsets));
+               link->uprobe_multi.cookies = calloc(target_cnt, sizeof(*link->uprobe_multi.cookies));
+
+               if (!link->uprobe_multi.offsets ||
+                   !link->uprobe_multi.ref_ctr_offsets ||
+                   !link->uprobe_multi.cookies) {
+                       err = -ENOMEM;
+                       goto err_out;
+               }
+       } else {
+               link->uprobes = calloc(target_cnt, sizeof(*link->uprobes));
+               if (!link->uprobes) {
+                       err = -ENOMEM;
+                       goto err_out;
+               }
        }
 

[...]

#50 lore raw

Re: [PATCHv2 bpf-next 14/24] libbpf: Add uprobe multi link support to bpf_program__attach_usdt

Quoted 37 lines · Andrii Nakryiko
Quoted 36 lines · Jiri Olsa

Adding support for usdt_manager_attach_usdt to use uprobe_multi link to attach to usdt probes.

The uprobe_multi support is detected before the usdt program is loaded and its expected_attach_type is set accordingly.

If uprobe_multi support is detected the usdt_manager_attach_usdt gathers uprobes info and calls bpf_program__attach_uprobe_opts to create all needed uprobes.

If uprobe_multi support is not detected the old behaviour stays.

---
 tools/lib/bpf/libbpf.c |  12 ++++-
 tools/lib/bpf/usdt.c   | 120 ++++++++++++++++++++++++++++++-----------
 2 files changed, 99 insertions(+), 33 deletions(-)
tools/lib/bpf/libbpf.c +2 -0
index 3d570898459e..9c7a67c5cbe8 100644
@@ -363,6 +363,8 @@ enum sec_def_flags {
        SEC_SLEEPABLE = 8,
        /* BPF program support non-linear XDP buffer */
        SEC_XDP_FRAGS = 16,
+       /* Setup proper attach type for usdt probes. */
+       SEC_USDT = 32,
 };
 
 struct bpf_sec_def {
@@ -6799,6 +6801,10 @@ static int libbpf_prepare_prog_load(struct bpf_program *prog,
        if (prog->type == BPF_PROG_TYPE_XDP && (def & SEC_XDP_FRAGS))
                opts->prog_flags |= BPF_F_XDP_HAS_FRAGS;
 
Quoted 7 lines

tools/lib/bpf/libbpf.c static int libbpf_prepare_prog_load(struct bpf_program *prog, · line 6806

Quoted 3 lines
tools/lib/bpf/libbpf.c +3 -0
+       /* special check for usdt to use uprobe_multi link */
+       if ((def & SEC_USDT) && kernel_supports(NULL, FEAT_UPROBE_LINK))
+               prog->expected_attach_type = BPF_TRACE_UPROBE_MULTI;

this is quite ugly. I think KPROBE programs do not have enforcement for expected_attach_type during BPF_PROG_LOAD, so we can set BPF_TRACE_UPROBE_MULTI unconditionally, right?

t doesn't but we are adding one earlier to bpf_prog_attach_check_attach_type (the one with U/K typo):

+ case BPF_PROG_TYPE_KPROBE:
+ if (prog->expected_attach_type == BPF_TRACE_UPROBE_MULTI &&
+ attach_type != BPF_TRACE_UPROBE_MULTI)
+ return -EINVAL;

for same reasons we did the check for kprobe_multi in:

db8eae6bc5c7 bpf: Force kprobe multi expected_attach_type for kprobe_multi link

Quoted 17 lines
Quoted 17 lines
tools/lib/bpf/libbpf.c +1 -1
+
        if ((def & SEC_ATTACH_BTF) && !prog->attach_btf_id) {
                int btf_obj_fd = 0, btf_type_id = 0, err;
                const char *attach_name;
@@ -6867,7 +6873,6 @@ static int bpf_object_load_prog(struct bpf_object *obj, struct bpf_program *prog
        if (!insns || !insns_cnt)
                return -EINVAL;
 
-       load_attr.expected_attach_type = prog->expected_attach_type;
        if (kernel_supports(obj, FEAT_PROG_NAME))
                prog_name = prog->name;
        load_attr.attach_prog_fd = prog->attach_prog_fd;
@@ -6903,6 +6908,9 @@ static int bpf_object_load_prog(struct bpf_object *obj, struct bpf_program *prog
                insns_cnt = prog->insns_cnt;
        }
 
Quoted 15 lines

tools/lib/bpf/libbpf.c static const struct bpf_sec_def section_defs[] = { · line 8714

Quoted 12 lines
tools/lib/bpf/libbpf.c +4 -1
+       /* allow prog_prepare_load_fn to change expected_attach_type */
+       load_attr.expected_attach_type = prog->expected_attach_type;
+
        if (obj->gen_loader) {
                bpf_gen__prog_load(obj->gen_loader, prog->type, prog->name,
                                   license, insns, insns_cnt, &load_attr,
@@ -8703,7 +8711,7 @@ static const struct bpf_sec_def section_defs[] = {
        SEC_DEF("uretprobe.multi.s+",   KPROBE, BPF_TRACE_UPROBE_MULTI, SEC_SLEEPABLE, attach_uprobe_multi),
        SEC_DEF("ksyscall+",            KPROBE, 0, SEC_NONE, attach_ksyscall),
        SEC_DEF("kretsyscall+",         KPROBE, 0, SEC_NONE, attach_ksyscall),
-       SEC_DEF("usdt+",                KPROBE, 0, SEC_NONE, attach_usdt),
+       SEC_DEF("usdt+",                KPROBE, 0, SEC_USDT, attach_usdt),

btw, given you are touching USDT stuff, can you please also add sleepable USDT (usdt.s+) ?

ok

Quoted 16 lines
Quoted 16 lines
tools/lib/bpf/libbpf.c +0 -0
        SEC_DEF("tc",                   SCHED_CLS, 0, SEC_NONE),
        SEC_DEF("classifier",           SCHED_CLS, 0, SEC_NONE),
        SEC_DEF("action",               SCHED_ACT, 0, SEC_NONE),
tools/lib/bpf/usdt.c +3 -0
index f1a141555f08..33f0a2b4cc1c 100644
@@ -808,6 +808,16 @@ struct bpf_link_usdt {
                long abs_ip;
                struct bpf_link *link;
        } *uprobes;
+
+       bool has_uprobe_multi;
+
Quoted 8 lines

tools/lib/bpf/usdt.c struct bpf_link_usdt { · line 818

Quoted 5 lines
tools/lib/bpf/usdt.c +5 -0
+       struct {
+               char *path;
+               unsigned long *offsets;
+               unsigned long *ref_ctr_offsets;
+               __u64 *cookies;

you shouldn't need to persist this, this can be allocated and freed inside usdt_manager_attach_usdt(), you only need the link pointer

aah right, that can go away, nice

Quoted 45 lines
Quoted 32 lines
tools/lib/bpf/usdt.c +10 -13
+               struct bpf_link *link;
+       } uprobe_multi;
 };
 
 static int bpf_link_usdt_detach(struct bpf_link *link)
@@ -816,19 +826,23 @@ static int bpf_link_usdt_detach(struct bpf_link *link)
        struct usdt_manager *man = usdt_link->usdt_man;
        int i;
 
-       for (i = 0; i < usdt_link->uprobe_cnt; i++) {
-               /* detach underlying uprobe link */
-               bpf_link__destroy(usdt_link->uprobes[i].link);
-               /* there is no need to update specs map because it will be
-                * unconditionally overwritten on subsequent USDT attaches,
-                * but if BPF cookies are not used we need to remove entry
-                * from ip_to_spec_id map, otherwise we'll run into false
-                * conflicting IP errors
-                */
-               if (!man->has_bpf_cookie) {
-                       /* not much we can do about errors here */
-                       (void)bpf_map_delete_elem(bpf_map__fd(man->ip_to_spec_id_map),
-                                                 &usdt_link->uprobes[i].abs_ip);
+       if (usdt_link->has_uprobe_multi) {
+               bpf_link__destroy(usdt_link->uprobe_multi.link);
+       } else {
+               for (i = 0; i < usdt_link->uprobe_cnt; i++) {
+                       /* detach underlying uprobe link */
+                       bpf_link__destroy(usdt_link->uprobes[i].link);
+                       /* there is no need to update specs map because it will be
+                        * unconditionally overwritten on subsequent USDT attaches,

tools/lib/bpf/usdt.c static int bpf_link_usdt_detach(struct bpf_link *link) · line 846

Quoted 10 lines
tools/lib/bpf/usdt.c +9 -0
+                        * but if BPF cookies are not used we need to remove entry
+                        * from ip_to_spec_id map, otherwise we'll run into false
+                        * conflicting IP errors
+                        */
+                       if (!man->has_bpf_cookie) {
+                               /* not much we can do about errors here */
+                               (void)bpf_map_delete_elem(bpf_map__fd(man->ip_to_spec_id_map),
+                                                         &usdt_link->uprobes[i].abs_ip);
+                       }
                }

you can avoid shifting all this by keeping uprobe_cnt to zero

Quoted 4 lines

bpf_link__destory(usdt_link->uprobe_multi.link) will work fine for NULL

so just do both clean ups sequentially, knowing that only one of them will actually do anything

ok, good idea

Quoted 7 lines
Quoted 7 lines
tools/lib/bpf/usdt.c +0 -0
        }
 
@@ -868,9 +882,15 @@ static void bpf_link_usdt_dealloc(struct bpf_link *link)
 {
        struct bpf_link_usdt *usdt_link = container_of(link, struct bpf_link_usdt, link);
 
Quoted 15 lines

tools/lib/bpf/usdt.c static void bpf_link_usdt_dealloc(struct bpf_link *link) · line 893

Quoted 12 lines
tools/lib/bpf/usdt.c +9 -3
-       free(usdt_link->spec_ids);
-       free(usdt_link->uprobes);
-       free(usdt_link);
+       if (usdt_link->has_uprobe_multi) {
+               free(usdt_link->uprobe_multi.offsets);
+               free(usdt_link->uprobe_multi.ref_ctr_offsets);
+               free(usdt_link->uprobe_multi.cookies);
+       } else {
+               free(usdt_link->spec_ids);
+               free(usdt_link->uprobes);
+               free(usdt_link);
+       }

similar to the above, just do *all* the clean up unconditionally and rely on free() handling NULLs just fine

ok

Quoted 18 lines
Quoted 18 lines
tools/lib/bpf/usdt.c +2 -0
 }
 
 static size_t specs_hash_fn(long key, void *ctx)
@@ -943,11 +963,13 @@ struct bpf_link *usdt_manager_attach_usdt(struct usdt_manager *man, const struct
                                          const char *usdt_provider, const char *usdt_name,
                                          __u64 usdt_cookie)
 {
+       LIBBPF_OPTS(bpf_uprobe_multi_opts, opts_multi);
        int i, fd, err, spec_map_fd, ip_map_fd;
        LIBBPF_OPTS(bpf_uprobe_opts, opts);
        struct hashmap *specs_hash = NULL;
        struct bpf_link_usdt *link = NULL;
        struct usdt_target *targets = NULL;
+       struct bpf_link *uprobe_link;
        size_t target_cnt;
        Elf *elf;
 
Quoted 8 lines

tools/lib/bpf/usdt.c struct bpf_link *usdt_manager_attach_usdt(struct usdt_manager *man, const struct · line 1028

Quoted 5 lines
tools/lib/bpf/usdt.c +1 -0
@@ -1003,16 +1025,29 @@ struct bpf_link *usdt_manager_attach_usdt(struct usdt_manager *man, const struct
        link->usdt_man = man;
        link->link.detach = &bpf_link_usdt_detach;
        link->link.dealloc = &bpf_link_usdt_dealloc;
+       link->has_uprobe_multi = bpf_program__expected_attach_type(prog) == BPF_TRACE_UPROBE_MULTI;

just use kernel_supports(), it's cleaner (and result is cached, so it's not less efficient)

ok

thanks,
jirka

author reviewer
#51 lore raw

PATCH v2 15/24 selftests/bpf: Add uprobe_multi skel test

In reply to Jiri Olsa

Adding uprobe_multi test for skeleton load/attach functions, to test skeleton auto attach for uprobe_multi link.

Test that bpf_get_func_ip works properly for uprobe_multi attachment.

---
 .../bpf/prog_tests/uprobe_multi_test.c        | 76 ++++++++++++++++
 .../selftests/bpf/progs/uprobe_multi.c        | 91 +++++++++++++++++++
 2 files changed, 167 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c
 create mode 100644 tools/testing/selftests/bpf/progs/uprobe_multi.c
tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c new file +76 -0
new file mode 100644
index 000000000000..5cd1116bbb62
@@ -0,0 +1,76 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <unistd.h>
+#include <test_progs.h>
+#include "uprobe_multi.skel.h"
+
+static char test_data[] = "test_data";
+
+noinline void uprobe_multi_func_1(void)
+{
+	asm volatile ("");
+}
+
+noinline void uprobe_multi_func_2(void)
+{
+	asm volatile ("");
+}
+
+noinline void uprobe_multi_func_3(void)
+{
+	asm volatile ("");
+}
+
+static void uprobe_multi_test_run(struct uprobe_multi *skel)
+{
+	skel->bss->uprobe_multi_func_1_addr = (__u64) uprobe_multi_func_1;
+	skel->bss->uprobe_multi_func_2_addr = (__u64) uprobe_multi_func_2;
+	skel->bss->uprobe_multi_func_3_addr = (__u64) uprobe_multi_func_3;
+
+	skel->bss->user_ptr = test_data;
+	skel->bss->pid = getpid();
+
+	/* trigger all probes */
+	uprobe_multi_func_1();
+	uprobe_multi_func_2();
+	uprobe_multi_func_3();
+
+	/*
+	 * There are 2 entry and 2 exit probe called for each uprobe_multi_func_[123]
+	 * function and each slepable probe (6) increments uprobe_multi_sleep_result.
+	 */
+	ASSERT_EQ(skel->bss->uprobe_multi_func_1_result, 2, "uprobe_multi_func_1_result");
+	ASSERT_EQ(skel->bss->uprobe_multi_func_2_result, 2, "uprobe_multi_func_2_result");
+	ASSERT_EQ(skel->bss->uprobe_multi_func_3_result, 2, "uprobe_multi_func_3_result");
+
+	ASSERT_EQ(skel->bss->uretprobe_multi_func_1_result, 2, "uretprobe_multi_func_1_result");
+	ASSERT_EQ(skel->bss->uretprobe_multi_func_2_result, 2, "uretprobe_multi_func_2_result");
+	ASSERT_EQ(skel->bss->uretprobe_multi_func_3_result, 2, "uretprobe_multi_func_3_result");
+
+	ASSERT_EQ(skel->bss->uprobe_multi_sleep_result, 6, "uprobe_multi_sleep_result");
+}
+
+static void test_skel_api(void)
+{
+	struct uprobe_multi *skel = NULL;
+	int err;
+
+	skel = uprobe_multi__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "uprobe_multi__open_and_load"))
+		goto cleanup;
+
+	err = uprobe_multi__attach(skel);
+	if (!ASSERT_OK(err, "uprobe_multi__attach"))
+		goto cleanup;
+
+	uprobe_multi_test_run(skel);
+
+cleanup:
+	uprobe_multi__destroy(skel);
+}
+
+void test_uprobe_multi_test(void)
+{
+	if (test__start_subtest("skel_api"))
+		test_skel_api();
+}
tools/testing/selftests/bpf/progs/uprobe_multi.c new file +91 -0

91 lines not on this page · raw message

Signature · 1 line
2.41.0
author reviewer
#52 lore raw

PATCH v2 16/24 selftests/bpf: Add uprobe_multi api test

In reply to Jiri Olsa

Adding uprobe_multi test for bpf_program__attach_uprobe_multi_opts attach function.

Testing attachment using glob patterns and via bpf_uprobe_multi_opts paths/syms fields.

---
 .../bpf/prog_tests/uprobe_multi_test.c        | 71 +++++++++++++++++++
 1 file changed, 71 insertions(+)
tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c +71 -0
index 5cd1116bbb62..28b8e8d451fb 100644
@@ -69,8 +69,79 @@ static void test_skel_api(void)
 	uprobe_multi__destroy(skel);
 }
 
+static void
+test_attach_api(const char *binary, const char *pattern, struct bpf_uprobe_multi_opts *opts)
+{
+	struct bpf_link *link1 = NULL, *link2 = NULL;
+	struct bpf_link *link3 = NULL, *link4 = NULL;
+	struct uprobe_multi *skel = NULL;
+
+	skel = uprobe_multi__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "uprobe_multi__open_and_load"))
+		goto cleanup;
+
+	opts->retprobe = false;
+	link1 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uprobe, -1,
+						      binary, pattern, opts);
+	if (!ASSERT_OK_PTR(link1, "bpf_program__attach_uprobe_multi_opts"))
+		goto cleanup;
+
+	opts->retprobe = true;
+	link2 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uretprobe, -1,
+						      binary, pattern, opts);
+	if (!ASSERT_OK_PTR(link2, "bpf_program__attach_uprobe_multi_opts_retprobe"))
+		goto cleanup;
+
+	opts->retprobe = false;
+	link3 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uprobe_sleep, -1,
+						      binary, pattern, opts);
+	if (!ASSERT_OK_PTR(link1, "bpf_program__attach_uprobe_multi_opts"))
+		goto cleanup;
+
+	opts->retprobe = true;
+	link4 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uretprobe_sleep, -1,
+						      binary, pattern, opts);
+	if (!ASSERT_OK_PTR(link2, "bpf_program__attach_uprobe_multi_opts_retprobe"))
+		goto cleanup;
+
+	uprobe_multi_test_run(skel);
+
+cleanup:
+	bpf_link__destroy(link4);
+	bpf_link__destroy(link3);
+	bpf_link__destroy(link2);
+	bpf_link__destroy(link1);
+	uprobe_multi__destroy(skel);
+}
+
+static void test_attach_api_pattern(void)
+{
+	LIBBPF_OPTS(bpf_uprobe_multi_opts, opts);
+
+	test_attach_api("/proc/self/exe", "uprobe_multi_func_*", &opts);
+	test_attach_api("/proc/self/exe", "uprobe_multi_func_?", &opts);
+}
+
+static void test_attach_api_syms(void)
+{
+	LIBBPF_OPTS(bpf_uprobe_multi_opts, opts);
+	const char *syms[3] = {
+		"uprobe_multi_func_1",
+		"uprobe_multi_func_2",
+		"uprobe_multi_func_3",
+	};
+
+	opts.syms = syms;
+	opts.cnt = ARRAY_SIZE(syms);
+	test_attach_api("/proc/self/exe", NULL, &opts);
+}
+
 void test_uprobe_multi_test(void)
 {
 	if (test__start_subtest("skel_api"))
 		test_skel_api();
+	if (test__start_subtest("attach_api_pattern"))
+		test_attach_api_pattern();
+	if (test__start_subtest("attach_api_syms"))
+		test_attach_api_syms();
 }
Signature · 1 line
2.41.0
author reviewer
#53 lore raw

PATCH v2 17/24 selftests/bpf: Add uprobe_multi link test

In reply to Jiri Olsa

Adding uprobe_multi test for bpf_link_create attach function.

Testing attachment using the struct bpf_link_create_opts.

---
 .../bpf/prog_tests/uprobe_multi_test.c        | 68 +++++++++++++++++++
 1 file changed, 68 insertions(+)
tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c +68 -0
index 28b8e8d451fb..fb5247a86559 100644
@@ -3,6 +3,7 @@
 #include <unistd.h>
 #include <test_progs.h>
 #include "uprobe_multi.skel.h"
+#include "bpf/libbpf_internal.h"
 
 static char test_data[] = "test_data";
 
@@ -136,6 +137,71 @@ static void test_attach_api_syms(void)
 	test_attach_api("/proc/self/exe", NULL, &opts);
 }
 
+static void test_link_api(void)
+{
+	int prog_fd, link1_fd = -1, link2_fd = -1, link3_fd = -1, link4_fd = -1;
+	LIBBPF_OPTS(bpf_link_create_opts, opts);
+	const char *path = "/proc/self/exe";
+	struct uprobe_multi *skel = NULL;
+	unsigned long *offsets = NULL;
+	const char *syms[3] = {
+		"uprobe_multi_func_1",
+		"uprobe_multi_func_2",
+		"uprobe_multi_func_3",
+	};
+	int err;
+
+	err = elf_find_multi_func_offset(path, 3, syms, (unsigned long **) &offsets);
+	if (!ASSERT_OK(err, "elf_find_multi_func_offset"))
+		return;
+
+	opts.uprobe_multi.path = path;
+	opts.uprobe_multi.offsets = offsets;
+	opts.uprobe_multi.cnt = ARRAY_SIZE(syms);
+
+	skel = uprobe_multi__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "uprobe_multi__open_and_load"))
+		goto cleanup;
+
+	opts.kprobe_multi.flags = 0;
+	prog_fd = bpf_program__fd(skel->progs.test_uprobe);
+	link1_fd = bpf_link_create(prog_fd, 0, BPF_TRACE_UPROBE_MULTI, &opts);
+	if (!ASSERT_GE(link1_fd, 0, "link1_fd"))
+		goto cleanup;
+
+	opts.kprobe_multi.flags = BPF_F_UPROBE_MULTI_RETURN;
+	prog_fd = bpf_program__fd(skel->progs.test_uretprobe);
+	link2_fd = bpf_link_create(prog_fd, 0, BPF_TRACE_UPROBE_MULTI, &opts);
+	if (!ASSERT_GE(link2_fd, 0, "link2_fd"))
+		goto cleanup;
+
+	opts.kprobe_multi.flags = 0;
+	prog_fd = bpf_program__fd(skel->progs.test_uprobe_sleep);
+	link3_fd = bpf_link_create(prog_fd, 0, BPF_TRACE_UPROBE_MULTI, &opts);
+	if (!ASSERT_GE(link1_fd, 0, "link3_fd"))
+		goto cleanup;
+
+	opts.kprobe_multi.flags = BPF_F_UPROBE_MULTI_RETURN;
+	prog_fd = bpf_program__fd(skel->progs.test_uretprobe_sleep);
+	link4_fd = bpf_link_create(prog_fd, 0, BPF_TRACE_UPROBE_MULTI, &opts);
+	if (!ASSERT_GE(link2_fd, 0, "link4_fd"))
+		goto cleanup;
+	uprobe_multi_test_run(skel);
+
+cleanup:
+	if (link1_fd >= 0)
+		close(link1_fd);
+	if (link2_fd >= 0)
+		close(link2_fd);
+	if (link3_fd >= 0)
+		close(link3_fd);
+	if (link4_fd >= 0)
+		close(link4_fd);
+
+	uprobe_multi__destroy(skel);
+	free(offsets);
+}
+
 void test_uprobe_multi_test(void)
 {
 	if (test__start_subtest("skel_api"))
@@ -144,4 +210,6 @@ void test_uprobe_multi_test(void)
 		test_attach_api_pattern();
 	if (test__start_subtest("attach_api_syms"))
 		test_attach_api_syms();
+	if (test__start_subtest("link_api"))
+		test_link_api();
 }
Signature · 1 line
2.41.0
author reviewer
#54 lore raw

PATCH v2 18/24 selftests/bpf: Add uprobe_multi test program

In reply to Jiri Olsa

Adding uprobe_multi test program that defines 50k uprobe_multi_func_* functions and will serve as attach point for uprobe_multi bench test in following patch.

---
 tools/testing/selftests/bpf/Makefile       |  5 ++
 tools/testing/selftests/bpf/uprobe_multi.c | 53 ++++++++++++++++++++++
 2 files changed, 58 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/uprobe_multi.c
tools/testing/selftests/bpf/Makefile +5 -0
index 538df8fb8c42..26ecdd117f71 100644
@@ -567,6 +567,7 @@ TRUNNER_EXTRA_FILES := $(OUTPUT)/urandom_read $(OUTPUT)/bpf_testmod.ko	\
 		       $(OUTPUT)/liburandom_read.so			\
 		       $(OUTPUT)/xdp_synproxy				\
 		       $(OUTPUT)/sign-file				\
+		       $(OUTPUT)/uprobe_multi				\
 		       ima_setup.sh 					\
 		       verify_sig_setup.sh				\
 		       $(wildcard progs/btf_dump_test_case_*.c)		\
@@ -670,6 +671,10 @@ $(OUTPUT)/veristat: $(OUTPUT)/veristat.o
 	$(call msg,BINARY,,$@)
 	$(Q)$(CC) $(CFLAGS) $(LDFLAGS) $(filter %.a %.o,$^) $(LDLIBS) -o $@
 
+$(OUTPUT)/uprobe_multi: uprobe_multi.c
+	$(call msg,BINARY,,$@)
+	$(Q)$(CC) $(CFLAGS) $(LDFLAGS) $^ $(LDLIBS) -o $@
+
 EXTRA_CLEAN := $(TEST_CUSTOM_PROGS) $(SCRATCH_DIR) $(HOST_SCRATCH_DIR)	\
 	prog_tests/tests.h map_tests/tests.h verifier/tests.h		\
 	feature bpftool							\
tools/testing/selftests/bpf/uprobe_multi.c new file +53 -0

53 lines not on this page · raw message

Signature · 1 line
2.41.0
author reviewer
#55 lore raw

PATCH v2 19/24 selftests/bpf: Add uprobe_multi bench test

In reply to Jiri Olsa

Adding test that attaches 50k uprobes in uprobe_multi binary.

After the attach is done we run the binary and make sure we get proper amount of hits.

---
 .../bpf/prog_tests/uprobe_multi_test.c        | 56 +++++++++++++++++++
 .../selftests/bpf/progs/uprobe_multi.c        |  9 +++
 2 files changed, 65 insertions(+)
tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c +56 -0
index fb5247a86559..9a34c754f1ad 100644
@@ -202,6 +202,60 @@ static void test_link_api(void)
 	free(offsets);
 }
 
+static inline __u64 get_time_ns(void)
+{
+	struct timespec t;
+
+	clock_gettime(CLOCK_MONOTONIC, &t);
+	return (__u64) t.tv_sec * 1000000000 + t.tv_nsec;
+}
+
+static void test_bench_attach_uprobe(void)
+{
+	long attach_start_ns, attach_end_ns;
+	long detach_start_ns, detach_end_ns;
+	double attach_delta, detach_delta;
+	struct uprobe_multi *skel = NULL;
+	struct bpf_program *prog;
+	int err;
+
+	skel = uprobe_multi__open();
+	if (!ASSERT_OK_PTR(skel, "uprobe_multi__open"))
+		goto cleanup;
+
+	bpf_object__for_each_program(prog, skel->obj)
+		bpf_program__set_autoload(prog, false);
+
+	bpf_program__set_autoload(skel->progs.test_uprobe_bench, true);
+
+	err = uprobe_multi__load(skel);
+	if (!ASSERT_EQ(err, 0, "uprobe_multi__load"))
+		goto cleanup;
+
+	attach_start_ns = get_time_ns();
+
+	err = uprobe_multi__attach(skel);
+	if (!ASSERT_OK(err, "uprobe_multi__attach"))
+		goto cleanup;
+
+	attach_end_ns = get_time_ns();
+
+	system("./uprobe_multi");
+
+	ASSERT_EQ(skel->bss->count, 50000, "uprobes_count");
+
+cleanup:
+	detach_start_ns = get_time_ns();
+	uprobe_multi__destroy(skel);
+	detach_end_ns = get_time_ns();
+
+	attach_delta = (attach_end_ns - attach_start_ns) / 1000000000.0;
+	detach_delta = (detach_end_ns - detach_start_ns) / 1000000000.0;
+
+	printf("%s: attached in %7.3lfs\n", __func__, attach_delta);
+	printf("%s: detached in %7.3lfs\n", __func__, detach_delta);
+}
+
 void test_uprobe_multi_test(void)
 {
 	if (test__start_subtest("skel_api"))
@@ -212,4 +266,6 @@ void test_uprobe_multi_test(void)
 		test_attach_api_syms();
 	if (test__start_subtest("link_api"))
 		test_link_api();
+	if (test__start_subtest("bench_uprobe"))
+		test_bench_attach_uprobe();
 }
tools/testing/selftests/bpf/progs/uprobe_multi.c +9 -0
index 1eeb9b7b9cad..cd73139dc881 100644
@@ -89,3 +89,12 @@ int test_uretprobe_sleep(struct pt_regs *ctx)
 	uprobe_multi_check(ctx, true, true);
 	return 0;
 }
+
+int count;
+
+SEC("?uprobe.multi/./uprobe_multi:uprobe_multi_func_*")
+int test_uprobe_bench(struct pt_regs *ctx)
+{
+	count++;
+	return 0;
+}
Signature · 1 line
2.41.0
author reviewer
#56 lore raw

PATCH v2 20/24 selftests/bpf: Add usdt_multi test program

In reply to Jiri Olsa

Adding usdt_multi test program that defines 50k usdts and will serve as attach point for uprobe_multi usdt bench test in following patch.

---
 tools/testing/selftests/bpf/Makefile     |  5 +++++
 tools/testing/selftests/bpf/usdt_multi.c | 24 ++++++++++++++++++++++++
 2 files changed, 29 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/usdt_multi.c
tools/testing/selftests/bpf/Makefile +5 -0
index 26ecdd117f71..135b909f7bc9 100644
@@ -568,6 +568,7 @@ TRUNNER_EXTRA_FILES := $(OUTPUT)/urandom_read $(OUTPUT)/bpf_testmod.ko	\
 		       $(OUTPUT)/xdp_synproxy				\
 		       $(OUTPUT)/sign-file				\
 		       $(OUTPUT)/uprobe_multi				\
+		       $(OUTPUT)/usdt_multi				\
 		       ima_setup.sh 					\
 		       verify_sig_setup.sh				\
 		       $(wildcard progs/btf_dump_test_case_*.c)		\
@@ -675,6 +676,10 @@ $(OUTPUT)/uprobe_multi: uprobe_multi.c
 	$(call msg,BINARY,,$@)
 	$(Q)$(CC) $(CFLAGS) $(LDFLAGS) $^ $(LDLIBS) -o $@
 
+$(OUTPUT)/usdt_multi: usdt_multi.c
+	$(call msg,BINARY,,$@)
+	$(Q)$(CC) $(CFLAGS) $(LDFLAGS) $^ $(LDLIBS) -o $@
+
 EXTRA_CLEAN := $(TEST_CUSTOM_PROGS) $(SCRATCH_DIR) $(HOST_SCRATCH_DIR)	\
 	prog_tests/tests.h map_tests/tests.h verifier/tests.h		\
 	feature bpftool							\
tools/testing/selftests/bpf/usdt_multi.c new file +24 -0
new file mode 100644
index 000000000000..fedf856bad2b
@@ -0,0 +1,24 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <sdt.h>
+
+#define PROBE STAP_PROBE(test, usdt);
+
+#define PROBE10    PROBE PROBE PROBE PROBE PROBE \
+		   PROBE PROBE PROBE PROBE PROBE
+#define PROBE100   PROBE10 PROBE10 PROBE10 PROBE10 PROBE10 \
+		   PROBE10 PROBE10 PROBE10 PROBE10 PROBE10
+#define PROBE1000  PROBE100 PROBE100 PROBE100 PROBE100 PROBE100 \
+		   PROBE100 PROBE100 PROBE100 PROBE100 PROBE100
+#define PROBE10000 PROBE1000 PROBE1000 PROBE1000 PROBE1000 PROBE1000 \
+		   PROBE1000 PROBE1000 PROBE1000 PROBE1000 PROBE1000
+
+int main(void)
+{
+	PROBE10000
+	PROBE10000
+	PROBE10000
+	PROBE10000
+	PROBE10000
+	return 0;
+}
Signature · 1 line
2.41.0
author reviewer
#57 lore raw

PATCH v2 21/24 selftests/bpf: Add usdt_multi bench test

In reply to Jiri Olsa

Adding test that attaches 50k usdt probes in usdt_multi binary.

After the attach is done we run the binary and make sure we get proper amount of hits.

With current uprobes:

  # perf stat --null ./test_progs -n 254/6
  #254/6   uprobe_multi_test/bench_usdt:OK
  #254     uprobe_multi_test:OK
  Summary: 1/1 PASSED, 0 SKIPPED, 0 FAILED

Performance counter stats for './test_progs -n 254/6':

      1353.659680562 seconds time elapsed

With uprobe_multi link:

  # perf stat --null ./test_progs -n 254/6
  #254/6   uprobe_multi_test/bench_usdt:OK
  #254     uprobe_multi_test:OK
  Summary: 1/1 PASSED, 0 SKIPPED, 0 FAILED

Performance counter stats for './test_progs -n 254/6':

         0.322046364 seconds time elapsed
---
 .../bpf/prog_tests/uprobe_multi_test.c        | 50 +++++++++++++++++++
 .../selftests/bpf/progs/uprobe_multi_usdt.c   | 16 ++++++
 2 files changed, 66 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/uprobe_multi_usdt.c
tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c +50 -0
index 9a34c754f1ad..a9d5a1773407 100644
@@ -4,6 +4,7 @@
 #include <test_progs.h>
 #include "uprobe_multi.skel.h"
 #include "bpf/libbpf_internal.h"
+#include "uprobe_multi_usdt.skel.h"
 
 static char test_data[] = "test_data";
 
@@ -256,6 +257,53 @@ static void test_bench_attach_uprobe(void)
 	printf("%s: detached in %7.3lfs\n", __func__, detach_delta);
 }
 
+static void test_bench_attach_usdt(void)
+{
+	struct uprobe_multi_usdt *skel = NULL;
+	long attach_start_ns, attach_end_ns;
+	long detach_start_ns, detach_end_ns;
+	double attach_delta, detach_delta;
+	struct bpf_program *prog;
+	int err;
+
+	skel = uprobe_multi_usdt__open();
+	if (!ASSERT_OK_PTR(skel, "uprobe_multi__open"))
+		goto cleanup;
+
+	bpf_object__for_each_program(prog, skel->obj)
+		bpf_program__set_autoload(prog, false);
+
+	bpf_program__set_autoload(skel->progs.usdt0, true);
+
+	err = uprobe_multi_usdt__load(skel);
+	if (!ASSERT_EQ(err, 0, "uprobe_multi_usdt__load"))
+		goto cleanup;
+
+	attach_start_ns = get_time_ns();
+
+	skel->links.usdt0 = bpf_program__attach_usdt(skel->progs.usdt0, -1, "./usdt_multi",
+						     "test", "usdt", NULL);
+	if (!ASSERT_OK_PTR(skel->links.usdt0, "bpf_program__attach_usdt"))
+		goto cleanup;
+
+	attach_end_ns = get_time_ns();
+
+	system("./usdt_multi");
+
+	ASSERT_EQ(skel->bss->count, 50000, "usdt_count");
+
+cleanup:
+	detach_start_ns = get_time_ns();
+	uprobe_multi_usdt__destroy(skel);
+	detach_end_ns = get_time_ns();
+
+	attach_delta = (attach_end_ns - attach_start_ns) / 1000000000.0;
+	detach_delta = (detach_end_ns - detach_start_ns) / 1000000000.0;
+
+	printf("%s: attached in %7.3lfs\n", __func__, attach_delta);
+	printf("%s: detached in %7.3lfs\n", __func__, detach_delta);
+}
+
 void test_uprobe_multi_test(void)
 {
 	if (test__start_subtest("skel_api"))
@@ -268,4 +316,6 @@ void test_uprobe_multi_test(void)
 		test_link_api();
 	if (test__start_subtest("bench_uprobe"))
 		test_bench_attach_uprobe();
+	if (test__start_subtest("bench_usdt"))
+		test_bench_attach_usdt();
 }
tools/testing/selftests/bpf/progs/uprobe_multi_usdt.c new file +16 -0
new file mode 100644
index 000000000000..9e1c33d0bd2f
@@ -0,0 +1,16 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/usdt.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+int count;
+
+SEC("usdt")
+int usdt0(struct pt_regs *ctx)
+{
+	count++;
+	return 0;
+}
Signature · 1 line
2.41.0
author reviewer
#58 lore raw

PATCH v2 22/24 selftests/bpf: Add uprobe_multi cookie test

In reply to Jiri Olsa

Adding test for cookies setup/retrieval in uprobe_link uprobes and making sure bpf_get_attach_cookie works properly.

---
 .../selftests/bpf/prog_tests/bpf_cookie.c     | 78 +++++++++++++++++++
 1 file changed, 78 insertions(+)
tools/testing/selftests/bpf/prog_tests/bpf_cookie.c +78 -0
index 26b2d1bffdfd..47f692a64cef 100644
@@ -11,6 +11,7 @@
 #include <bpf/btf.h>
 #include "test_bpf_cookie.skel.h"
 #include "kprobe_multi.skel.h"
+#include "uprobe_multi.skel.h"
 
 /* uprobe attach point */
 static noinline void trigger_func(void)
@@ -239,6 +240,81 @@ static void kprobe_multi_attach_api_subtest(void)
 	bpf_link__destroy(link1);
 	kprobe_multi__destroy(skel);
 }
+
+/* defined in prog_tests/uprobe_multi_test.c */
+void uprobe_multi_func_1(void);
+void uprobe_multi_func_2(void);
+void uprobe_multi_func_3(void);
+
+static void uprobe_multi_test_run(struct uprobe_multi *skel)
+{
+	skel->bss->uprobe_multi_func_1_addr = (__u64) uprobe_multi_func_1;
+	skel->bss->uprobe_multi_func_2_addr = (__u64) uprobe_multi_func_2;
+	skel->bss->uprobe_multi_func_3_addr = (__u64) uprobe_multi_func_3;
+
+	skel->bss->pid = getpid();
+	skel->bss->test_cookie = true;
+
+	uprobe_multi_func_1();
+	uprobe_multi_func_2();
+	uprobe_multi_func_3();
+
+	ASSERT_EQ(skel->bss->uprobe_multi_func_1_result, 1, "uprobe_multi_func_1_result");
+	ASSERT_EQ(skel->bss->uprobe_multi_func_2_result, 1, "uprobe_multi_func_2_result");
+	ASSERT_EQ(skel->bss->uprobe_multi_func_3_result, 1, "uprobe_multi_func_3_result");
+
+	ASSERT_EQ(skel->bss->uretprobe_multi_func_1_result, 1, "uretprobe_multi_func_1_result");
+	ASSERT_EQ(skel->bss->uretprobe_multi_func_2_result, 1, "uretprobe_multi_func_2_result");
+	ASSERT_EQ(skel->bss->uretprobe_multi_func_3_result, 1, "uretprobe_multi_func_3_result");
+}
+
+static void uprobe_multi_attach_api_subtest(void)
+{
+	struct bpf_link *link1 = NULL, *link2 = NULL;
+	struct uprobe_multi *skel = NULL;
+	LIBBPF_OPTS(bpf_uprobe_multi_opts, opts);
+	const char *syms[3] = {
+		"uprobe_multi_func_1",
+		"uprobe_multi_func_2",
+		"uprobe_multi_func_3",
+	};
+	__u64 cookies[3];
+
+	cookies[0] = 3; /* uprobe_multi_func_1 */
+	cookies[1] = 1; /* uprobe_multi_func_2 */
+	cookies[2] = 2; /* uprobe_multi_func_3 */
+
+	opts.syms = syms;
+	opts.cnt = ARRAY_SIZE(syms);
+	opts.cookies = &cookies[0];
+
+	skel = uprobe_multi__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "uprobe_multi"))
+		goto cleanup;
+
+	link1 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uprobe, -1,
+						      "/proc/self/exe", NULL, &opts);
+	if (!ASSERT_OK_PTR(link1, "bpf_program__attach_uprobe_multi_opts"))
+		goto cleanup;
+
+	cookies[0] = 2; /* uprobe_multi_func_1 */
+	cookies[1] = 3; /* uprobe_multi_func_2 */
+	cookies[2] = 1; /* uprobe_multi_func_3 */
+
+	opts.retprobe = true;
+	link2 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uretprobe, -1,
+						      "/proc/self/exe", NULL, &opts);
+	if (!ASSERT_OK_PTR(link2, "bpf_program__attach_uprobe_multi_opts_retprobe"))
+		goto cleanup;
+
+	uprobe_multi_test_run(skel);
+
+cleanup:
+	bpf_link__destroy(link2);
+	bpf_link__destroy(link1);
+	uprobe_multi__destroy(skel);
+}
+
 static void uprobe_subtest(struct test_bpf_cookie *skel)
 {
 	DECLARE_LIBBPF_OPTS(bpf_uprobe_opts, opts);
@@ -515,6 +591,8 @@ void test_bpf_cookie(void)
 		kprobe_multi_attach_api_subtest();
 	if (test__start_subtest("uprobe"))
 		uprobe_subtest(skel);
+	if (test__start_subtest("multi_uprobe_attach_api"))
+		uprobe_multi_attach_api_subtest();
 	if (test__start_subtest("tracepoint"))
 		tp_subtest(skel);
 	if (test__start_subtest("perf_event"))
Signature · 1 line
2.41.0
author reviewer
#59 lore raw

PATCH v2 23/24 selftests/bpf: Add uprobe_multi pid filter tests

In reply to Jiri Olsa

Running api and link tests also with pid filter and checking the probe gets executed only for specific pid.

Spawning extra process to trigger attached uprobes and checking we get correct counts from executed programs.

---
 .../bpf/prog_tests/uprobe_multi_test.c        | 131 ++++++++++++++++--
 .../selftests/bpf/progs/uprobe_multi.c        |   6 +-
 2 files changed, 125 insertions(+), 12 deletions(-)
tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c +120 -11
index a9d5a1773407..efa7b5d49940 100644
@@ -23,14 +23,86 @@ noinline void uprobe_multi_func_3(void)
 	asm volatile ("");
 }
 
-static void uprobe_multi_test_run(struct uprobe_multi *skel)
+struct child {
+	int go[2];
+	int pid;
+};
+
+static void release_child(struct child *child)
+{
+	int child_status;
+
+	if (!child)
+		return;
+	close(child->go[1]);
+	close(child->go[0]);
+	if (child->pid > 0)
+		waitpid(child->pid, &child_status, 0);
+}
+
+static void kick_child(struct child *child)
+{
+	char c = 1;
+
+	if (child) {
+		write(child->go[1], &c, 1);
+		release_child(child);
+	}
+	fflush(NULL);
+}
+
+static struct child *spawn_child(void)
+{
+	static struct child child;
+	int err;
+	int c;
+
+	/* pid filter */
+	if (!ASSERT_OK(pipe(child.go), "pipe"))
+		return NULL;
+
+	child.pid = fork();
+	if (child.pid < 0) {
+		release_child(&child);
+		return NULL;
+	}
+
+	/* child */
+	if (child.pid == 0) {
+		close(child.go[1]);
+		fflush(NULL);
+		/* wait for parent's kick */
+		err = read(child.go[0], &c, 1);
+		if (!ASSERT_EQ(err, 1, "child_read_pipe"))
+			exit(err);
+
+		uprobe_multi_func_1();
+		uprobe_multi_func_2();
+		uprobe_multi_func_3();
+
+		exit(errno);
+	}
+
+	return &child;
+}
+
+static void uprobe_multi_test_run(struct uprobe_multi *skel, struct child *child)
 {
 	skel->bss->uprobe_multi_func_1_addr = (__u64) uprobe_multi_func_1;
 	skel->bss->uprobe_multi_func_2_addr = (__u64) uprobe_multi_func_2;
 	skel->bss->uprobe_multi_func_3_addr = (__u64) uprobe_multi_func_3;
 
 	skel->bss->user_ptr = test_data;
-	skel->bss->pid = getpid();
+
+	/*
+	 * Disable pid check in bpf program if we are pid filter test,
+	 * because the probe should be executed only by child->pid
+	 * passed at the probe attach.
+	 */
+	skel->bss->pid = child ? 0 : getpid();
+
+	if (child)
+		kick_child(child);
 
 	/* trigger all probes */
 	uprobe_multi_func_1();
@@ -50,6 +122,9 @@ static void uprobe_multi_test_run(struct uprobe_multi *skel)
 	ASSERT_EQ(skel->bss->uretprobe_multi_func_3_result, 2, "uretprobe_multi_func_3_result");
 
 	ASSERT_EQ(skel->bss->uprobe_multi_sleep_result, 6, "uprobe_multi_sleep_result");
+
+	if (child)
+		ASSERT_EQ(skel->bss->child_pid, child->pid, "uprobe_multi_child_pid");
 }
 
 static void test_skel_api(void)
@@ -65,17 +140,19 @@ static void test_skel_api(void)
 	if (!ASSERT_OK(err, "uprobe_multi__attach"))
 		goto cleanup;
 
-	uprobe_multi_test_run(skel);
+	uprobe_multi_test_run(skel, NULL);
 
 cleanup:
 	uprobe_multi__destroy(skel);
 }
 
 static void
-test_attach_api(const char *binary, const char *pattern, struct bpf_uprobe_multi_opts *opts)
+__test_attach_api(const char *binary, const char *pattern, struct bpf_uprobe_multi_opts *opts,
+		  struct child *child)
 {
 	struct bpf_link *link1 = NULL, *link2 = NULL;
 	struct bpf_link *link3 = NULL, *link4 = NULL;
+	pid_t pid = child ? child->pid : -1;
 	struct uprobe_multi *skel = NULL;
 
 	skel = uprobe_multi__open_and_load();
@@ -83,30 +160,30 @@ test_attach_api(const char *binary, const char *pattern, struct bpf_uprobe_multi
 		goto cleanup;
 
 	opts->retprobe = false;
-	link1 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uprobe, -1,
+	link1 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uprobe, pid,
 						      binary, pattern, opts);
 	if (!ASSERT_OK_PTR(link1, "bpf_program__attach_uprobe_multi_opts"))
 		goto cleanup;
 
 	opts->retprobe = true;
-	link2 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uretprobe, -1,
+	link2 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uretprobe, pid,
 						      binary, pattern, opts);
 	if (!ASSERT_OK_PTR(link2, "bpf_program__attach_uprobe_multi_opts_retprobe"))
 		goto cleanup;
 
 	opts->retprobe = false;
-	link3 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uprobe_sleep, -1,
+	link3 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uprobe_sleep, pid,
 						      binary, pattern, opts);
 	if (!ASSERT_OK_PTR(link1, "bpf_program__attach_uprobe_multi_opts"))
 		goto cleanup;
 
 	opts->retprobe = true;
-	link4 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uretprobe_sleep, -1,
+	link4 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uretprobe_sleep, pid,
 						      binary, pattern, opts);
 	if (!ASSERT_OK_PTR(link2, "bpf_program__attach_uprobe_multi_opts_retprobe"))
 		goto cleanup;
 
-	uprobe_multi_test_run(skel);
+	uprobe_multi_test_run(skel, child);
 
 cleanup:
 	bpf_link__destroy(link4);
@@ -116,6 +193,22 @@ test_attach_api(const char *binary, const char *pattern, struct bpf_uprobe_multi
 	uprobe_multi__destroy(skel);
 }
 
+static void
+test_attach_api(const char *binary, const char *pattern, struct bpf_uprobe_multi_opts *opts)
+{
+	struct child *child;
+
+	/* no pid filter */
+	__test_attach_api(binary, pattern, opts, NULL);
+
+	/* pid filter */
+	child = spawn_child();
+	if (!child)
+		return;
+
+	__test_attach_api(binary, pattern, opts, child);
+}
+
 static void test_attach_api_pattern(void)
 {
 	LIBBPF_OPTS(bpf_uprobe_multi_opts, opts);
@@ -138,7 +231,7 @@ static void test_attach_api_syms(void)
 	test_attach_api("/proc/self/exe", NULL, &opts);
 }
 
-static void test_link_api(void)
+static void __test_link_api(struct child *child)
 {
 	int prog_fd, link1_fd = -1, link2_fd = -1, link3_fd = -1, link4_fd = -1;
 	LIBBPF_OPTS(bpf_link_create_opts, opts);
@@ -159,6 +252,7 @@ static void test_link_api(void)
 	opts.uprobe_multi.path = path;
 	opts.uprobe_multi.offsets = offsets;
 	opts.uprobe_multi.cnt = ARRAY_SIZE(syms);
+	opts.uprobe_multi.pid = child ? child->pid : 0;
 
 	skel = uprobe_multi__open_and_load();
 	if (!ASSERT_OK_PTR(skel, "uprobe_multi__open_and_load"))
@@ -187,7 +281,7 @@ static void test_link_api(void)
 	link4_fd = bpf_link_create(prog_fd, 0, BPF_TRACE_UPROBE_MULTI, &opts);
 	if (!ASSERT_GE(link2_fd, 0, "link4_fd"))
 		goto cleanup;
-	uprobe_multi_test_run(skel);
+	uprobe_multi_test_run(skel, child);
 
 cleanup:
 	if (link1_fd >= 0)
@@ -203,6 +297,21 @@ static void test_link_api(void)
 	free(offsets);
 }
 
+void test_link_api(void)
+{
+	struct child *child;
+
+	/* no pid filter */
+	__test_link_api(NULL);
+
+	/* pid filter */
+	child = spawn_child();
+	if (!child)
+		return;
+
+	__test_link_api(child);
+}
+
 static inline __u64 get_time_ns(void)
 {
 	struct timespec t;
tools/testing/selftests/bpf/progs/uprobe_multi.c +5 -1
index cd73139dc881..1086312b5d1e 100644
@@ -21,6 +21,8 @@ __u64 uretprobe_multi_func_3_result = 0;
 __u64 uprobe_multi_sleep_result = 0;
 
 int pid = 0;
+int child_pid = 0;
+
 bool test_cookie = false;
 void *user_ptr = 0;
 
@@ -34,7 +36,9 @@ static __always_inline bool verify_sleepable_user_copy(void)
 
 static void uprobe_multi_check(void *ctx, bool is_return, bool is_sleep)
 {
-	if (bpf_get_current_pid_tgid() >> 32 != pid)
+	child_pid = bpf_get_current_pid_tgid() >> 32;
+
+	if (pid && child_pid != pid)
 		return;
 
 	__u64 cookie = test_cookie ? bpf_get_attach_cookie(ctx) : 0;
Signature · 1 line
2.41.0
author reviewer
#60 lore raw

PATCH v2 24/24 selftests/bpf: Add extra link to uprobe_multi tests

In reply to Jiri Olsa

Attaching extra program to same functions system wide for api and link tests.

This way we can test the pid filter works properly when there's extra system wide consumer on the same uprobe that will trigger the original uprobe handler.

We expect to have the same counts as before.

---
 .../bpf/prog_tests/uprobe_multi_test.c        | 19 +++++++++++++++++++
 .../selftests/bpf/progs/uprobe_multi.c        |  6 ++++++
 2 files changed, 25 insertions(+)
tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c +19 -0
index efa7b5d49940..9057a228c12d 100644
@@ -153,6 +153,7 @@ __test_attach_api(const char *binary, const char *pattern, struct bpf_uprobe_mul
 	struct bpf_link *link1 = NULL, *link2 = NULL;
 	struct bpf_link *link3 = NULL, *link4 = NULL;
 	pid_t pid = child ? child->pid : -1;
+	struct bpf_link *link_extra = NULL;
 	struct uprobe_multi *skel = NULL;
 
 	skel = uprobe_multi__open_and_load();
@@ -183,9 +184,16 @@ __test_attach_api(const char *binary, const char *pattern, struct bpf_uprobe_mul
 	if (!ASSERT_OK_PTR(link2, "bpf_program__attach_uprobe_multi_opts_retprobe"))
 		goto cleanup;
 
+	opts->retprobe = false;
+	link_extra = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uprobe_extra, -1,
+						      binary, pattern, opts);
+	if (!ASSERT_OK_PTR(link_extra, "bpf_program__attach_uprobe_multi_opts"))
+		goto cleanup;
+
 	uprobe_multi_test_run(skel, child);
 
 cleanup:
+	bpf_link__destroy(link_extra);
 	bpf_link__destroy(link4);
 	bpf_link__destroy(link3);
 	bpf_link__destroy(link2);
@@ -243,6 +251,7 @@ static void __test_link_api(struct child *child)
 		"uprobe_multi_func_2",
 		"uprobe_multi_func_3",
 	};
+	int link_extra_fd = -1;
 	int err;
 
 	err = elf_find_multi_func_offset(path, 3, syms, (unsigned long **) &offsets);
@@ -281,6 +290,14 @@ static void __test_link_api(struct child *child)
 	link4_fd = bpf_link_create(prog_fd, 0, BPF_TRACE_UPROBE_MULTI, &opts);
 	if (!ASSERT_GE(link2_fd, 0, "link4_fd"))
 		goto cleanup;
+
+	opts.kprobe_multi.flags = 0;
+	opts.uprobe_multi.pid = 0;
+	prog_fd = bpf_program__fd(skel->progs.test_uprobe_extra);
+	link_extra_fd = bpf_link_create(prog_fd, 0, BPF_TRACE_UPROBE_MULTI, &opts);
+	if (!ASSERT_GE(link_extra_fd, 0, "link_extra_fd"))
+		goto cleanup;
+
 	uprobe_multi_test_run(skel, child);
 
 cleanup:
@@ -292,6 +309,8 @@ static void __test_link_api(struct child *child)
 		close(link3_fd);
 	if (link4_fd >= 0)
 		close(link4_fd);
+	if (link_extra_fd >= 0)
+		close(link_extra_fd);
 
 	uprobe_multi__destroy(skel);
 	free(offsets);
tools/testing/selftests/bpf/progs/uprobe_multi.c +6 -0
index 1086312b5d1e..8ee957670303 100644
@@ -102,3 +102,9 @@ int test_uprobe_bench(struct pt_regs *ctx)
 	count++;
 	return 0;
 }
+
+SEC("uprobe.multi//proc/self/exe:uprobe_multi_func_*")
+int test_uprobe_extra(struct pt_regs *ctx)
+{
+	return 0;
+}
Signature · 1 line
2.41.0