Skip to contentSkip to conversation

libbpf: Add support for dynamic tracepoints

RFC PATCH In review Yafang Shao 2 participants 6 messages Jan 12, 2025 – Jan 16, 2025

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

v2 cover libbpf: Add support for dynamic tracepoints

The primary goal of this change is to enable tracing of inlined kernel functions with BPF programs.

Dynamic tracepoints can be created using tools like perf-probe, debugfs, or similar utilities. For example:

  $ perf probe -a 'tcp_listendrop sk'
  $ ls /sys/kernel/debug/tracing/events/probe/tcp_listendrop/
  enable  filter  format  hist  id  trigger

Here, tcp_listendrop() is an example of an inlined kernel function.

While these dynamic tracepoints are functional, they cannot be easily attached to BPF programs. For instance, attempting to use them with bpftrace results in the following error:

  $ bpftrace -l 'tracepoint:probe:*'
  tracepoint:probe:tcp_listendrop
  $ bpftrace -e 'tracepoint:probe:tcp_listendrop {print(comm)}'
  Attaching 1 probe...
  ioctl(PERF_EVENT_IOC_SET_BPF): Invalid argument
  ERROR: Error attaching probe: tracepoint:probe:tcp_listendrop

The issue lies in how these dynamic tracepoints are implemented: despite being exposed as tracepoints, they remain kprobe events internally. As a result, loading them as a tracepoint program fails. Instead, they must be loaded as kprobe programs.

This change introduces support for such use cases in libbpf by adding a new section: SEC("kprobe/SUBSYSTEM/PROBE")

- Future work
Extend support for dynamic tracepoints in bpftrace.

Changes:
v1->v2:
- Use a new SEC("kprobe/SUBSYSTEM/PROBE") instead (Jiri)

v1: https://lore.kernel.org/bpf/[email protected]/

Yafang Shao (2):
libbpf: Add support for dynamic tracepoint selftests/bpf: Add selftest for dynamic tracepoint

 tools/lib/bpf/libbpf.c                        | 29 ++++++++-
 .../bpf/prog_tests/test_dynamic_tp.c          | 64 +++++++++++++++++++
 .../testing/selftests/bpf/progs/dynamic_tp.c  | 27 ++++++++
 3 files changed, 119 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/test_dynamic_tp.c
 create mode 100644 tools/testing/selftests/bpf/progs/dynamic_tp.c
Signature · 1 line
2.43.5
#1 lore raw

RFC PATCH v2 1/2 libbpf: Add support for dynamic tracepoint

Dynamic tracepoints can be created using debugfs, perf or similar tools. For example:

  $ perf probe -a 'tcp_listendrop sk'

This command creates a new tracepoint under debugfs:

  $ ls /sys/kernel/debug/tracing/events/probe/tcp_listendrop/
  enable  filter  format  hist  id  trigger

Notably, the probed function tcp_listendrop() is an inlined kernel function.

Although this dynamic tracepoint appears as a tracepoint, it is internally implemented as a kprobe. Therefore, if we want to attach a bpf prog to it, the bpf prog must be loaded as a kprobe prog.

The primary motivation for adding support for dynamic tracepoints is to simplify tracing of inlined kernel functions using BPF tools, such as bpftrace. By leveraging tools like perf, users can create a dynamic tracepoint for an inlined kernel function and then attach a BPF program to it.

To achieve this, a new section, SEC("kprobe/SUBSYSTEM/PROBE"), has been introduced.

---
 tools/lib/bpf/libbpf.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)
tools/lib/bpf/libbpf.c +28 -1
index 66173ddb5a2d..23ea9272491b 100644
@@ -11600,11 +11600,34 @@ bpf_program__attach_kprobe_multi_opts(const struct bpf_program *prog,
 	return libbpf_err_ptr(err);
 }
 
+/* A dynamic tracepoint: "kprobe/SUBSYSTEM/PROBE" */
+static int attach_dynamic_tracepoint(const struct bpf_program *prog, const char *func_name,
+				     struct bpf_link **link)
+{
+	char *tp_subsys, *tp_name;
+
+	tp_subsys = strdup(func_name);
+	if (!tp_subsys)
+		return -ENOMEM;
+
+	tp_name = strchr(tp_subsys, '/');
+	if (!tp_name) {
+		free(tp_subsys);
+		return -EINVAL;
+	}
+
+	*tp_name = '\0';
+	tp_name++;
+	*link = bpf_program__attach_tracepoint(prog, tp_subsys, tp_name);
+	free(tp_subsys);
+	return libbpf_get_error(*link);
+}
+
 static int attach_kprobe(const struct bpf_program *prog, long cookie, struct bpf_link **link)
 {
 	DECLARE_LIBBPF_OPTS(bpf_kprobe_opts, opts);
+	const char *func_name, *dynamic_tp;
 	unsigned long offset = 0;
-	const char *func_name;
 	char *func;
 	int n;
 
@@ -11620,6 +11643,10 @@ static int attach_kprobe(const struct bpf_program *prog, long cookie, struct bpf
 	else
 		func_name = prog->sec_name + sizeof("kprobe/") - 1;
 
+	dynamic_tp = strchr(func_name, '/');
+	if (dynamic_tp)
+		return attach_dynamic_tracepoint(prog, func_name, link);
+
 	n = sscanf(func_name, "%m[a-zA-Z0-9_.]+%li", &func, &offset);
 	if (n < 1) {
 		pr_warn("kprobe name is invalid: %s\n", func_name);
Signature · 1 line
2.43.5
#2 lore raw

RFC PATCH v2 2/2 selftests/bpf: Add selftest for dynamic tracepoint

In reply to Yafang Shao

The result is as follows,

  $ tools/testing/selftests/bpf/test_progs --name=dynamic_tp
  #85      dynamic_tp:OK
  Summary: 1/0 PASSED, 0 SKIPPED, 0 FAILED
---
 .../bpf/prog_tests/test_dynamic_tp.c          | 64 +++++++++++++++++++
 .../testing/selftests/bpf/progs/dynamic_tp.c  | 27 ++++++++
 2 files changed, 91 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/test_dynamic_tp.c
 create mode 100644 tools/testing/selftests/bpf/progs/dynamic_tp.c
tools/testing/selftests/bpf/prog_tests/test_dynamic_tp.c new file +64 -0
new file mode 100644
index 000000000000..c205e0c8e3e3
@@ -0,0 +1,64 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include <test_progs.h>
+#include <bpf/btf.h>
+#include <bpf/bpf.h>
+
+#include "dynamic_tp.skel.h"
+
+int dynamic_tp(const char *cmd)
+{
+	const char *kprobe_file = "/sys/kernel/debug/tracing/kprobe_events";
+	ssize_t bytes_written;
+	int fd, err;
+
+	fd = open(kprobe_file, O_WRONLY | O_APPEND);
+	if (!ASSERT_GE(fd, 0, "open kprobe_events"))
+		return -1;
+
+	bytes_written = write(fd, cmd, strlen(cmd));
+	if (!ASSERT_GT(bytes_written, 0, "write kprobe_events")) {
+		close(fd);
+		return -1;
+	}
+
+	err = close(fd);
+	if (!ASSERT_OK(err, "close kprobe_events"))
+		return -1;
+	return 0;
+}
+
+void test_dynamic_tp(void)
+{
+	struct dynamic_tp *skel;
+	pid_t child_pid;
+	int status, err;
+
+	/* create a dynamic tracepoint */
+	err = dynamic_tp("p:my_dynamic_tp kernel_clone");
+	if (!ASSERT_OK(err, "create dynamic tp"))
+		return;
+
+	skel = dynamic_tp__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "load progs"))
+		goto remove_tp;
+	skel->bss->pid = getpid();
+	err = dynamic_tp__attach(skel);
+	if (!ASSERT_OK(err, "attach progs"))
+		goto cleanup;
+
+	/* trigger the dynamic tracepoint */
+	child_pid = fork();
+	if (!ASSERT_GT(child_pid, -1, "child_pid"))
+		goto cleanup;
+	if (child_pid == 0)
+		_exit(0);
+	waitpid(child_pid, &status, 0);
+
+	ASSERT_EQ(skel->bss->result, 1, "result");
+
+cleanup:
+	dynamic_tp__destroy(skel);
+remove_tp:
+	dynamic_tp("-:my_dynamic_tp kernel_clone");
+}
tools/testing/selftests/bpf/progs/dynamic_tp.c new file +27 -0
new file mode 100644
index 000000000000..d3be37c220f3
@@ -0,0 +1,27 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+
+char _license[] SEC("license") = "GPL";
+
+#define MAX_STACK_TRACE_DEPTH 32
+unsigned long entries[MAX_STACK_TRACE_DEPTH] = {};
+#define SIZE_OF_ULONG (sizeof(unsigned long))
+
+int result, pid;
+
+SEC("kprobe/kprobes/my_dynamic_tp")
+int dynamic_tp(struct pt_regs *ctx)
+{
+	int ret;
+
+	ret = bpf_get_stack(ctx, entries, MAX_STACK_TRACE_DEPTH * SIZE_OF_ULONG, 0);
+	if (ret < 0) {
+		result = -1;
+		return ret;
+	}
+	if (bpf_get_current_pid_tgid() >> 32 == pid)
+		result = 1;
+	return 0;
+}
Signature · 1 line
2.43.5
#3 lore raw
In reply to Yafang Shao
Quoted 30 lines · Yafang Shao

The primary goal of this change is to enable tracing of inlined kernel functions with BPF programs.

Dynamic tracepoints can be created using tools like perf-probe, debugfs, or similar utilities. For example:

  $ perf probe -a 'tcp_listendrop sk'
  $ ls /sys/kernel/debug/tracing/events/probe/tcp_listendrop/
  enable  filter  format  hist  id  trigger

Here, tcp_listendrop() is an example of an inlined kernel function.

While these dynamic tracepoints are functional, they cannot be easily attached to BPF programs. For instance, attempting to use them with bpftrace results in the following error:

  $ bpftrace -l 'tracepoint:probe:*'
  tracepoint:probe:tcp_listendrop
  $ bpftrace -e 'tracepoint:probe:tcp_listendrop {print(comm)}'
  Attaching 1 probe...
  ioctl(PERF_EVENT_IOC_SET_BPF): Invalid argument
  ERROR: Error attaching probe: tracepoint:probe:tcp_listendrop

The issue lies in how these dynamic tracepoints are implemented: despite being exposed as tracepoints, they remain kprobe events internally. As a result, loading them as a tracepoint program fails. Instead, they must be loaded as kprobe programs.

Quoted 5 lines

This change introduces support for such use cases in libbpf by adding a new section: SEC("kprobe/SUBSYSTEM/PROBE")

- Future work
Extend support for dynamic tracepoints in bpftrace.

Seems like the primary motivation is bpftrace support, so let's start there. I'm hesitant to include this in libbpf right now. The whole SEC("kprobe") that calls "attach_tracepoint()" underneath makes me uncomfortable.

You can still attach to such dynamic probe today with pure libbpf (e.g., if bpftrace needs to do this, for example) by creating perf_event FD from tracefs' id, and then using bpf_program__attach_perf_event_opts() to attach to it. It will be on the user to use either tracepoint or kprobe BPF program for such attachment.

Yes, this doesn't work "declaratively" with a nice SEC("...") syntax, but at least it's doable programmatically, and that's what matters for bpftrace.

Quoted 21 lines

Changes:
v1->v2:
- Use a new SEC("kprobe/SUBSYSTEM/PROBE") instead (Jiri)

v1: https://lore.kernel.org/bpf/[email protected]/

Yafang Shao (2):
libbpf: Add support for dynamic tracepoint selftests/bpf: Add selftest for dynamic tracepoint

 tools/lib/bpf/libbpf.c                        | 29 ++++++++-
 .../bpf/prog_tests/test_dynamic_tp.c          | 64 +++++++++++++++++++
 .../testing/selftests/bpf/progs/dynamic_tp.c  | 27 ++++++++
 3 files changed, 119 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/test_dynamic_tp.c
 create mode 100644 tools/testing/selftests/bpf/progs/dynamic_tp.c

--
2.43.5

#4 lore raw
Quoted 32 lines · Andrii Nakryiko
Quoted 30 lines · Yafang Shao

The primary goal of this change is to enable tracing of inlined kernel functions with BPF programs.

Dynamic tracepoints can be created using tools like perf-probe, debugfs, or similar utilities. For example:

  $ perf probe -a 'tcp_listendrop sk'
  $ ls /sys/kernel/debug/tracing/events/probe/tcp_listendrop/
  enable  filter  format  hist  id  trigger

Here, tcp_listendrop() is an example of an inlined kernel function.

While these dynamic tracepoints are functional, they cannot be easily attached to BPF programs. For instance, attempting to use them with bpftrace results in the following error:

  $ bpftrace -l 'tracepoint:probe:*'
  tracepoint:probe:tcp_listendrop
  $ bpftrace -e 'tracepoint:probe:tcp_listendrop {print(comm)}'
  Attaching 1 probe...
  ioctl(PERF_EVENT_IOC_SET_BPF): Invalid argument
  ERROR: Error attaching probe: tracepoint:probe:tcp_listendrop

The issue lies in how these dynamic tracepoints are implemented: despite being exposed as tracepoints, they remain kprobe events internally. As a result, loading them as a tracepoint program fails. Instead, they must be loaded as kprobe programs.

Quoted 8 lines
Quoted 5 lines

This change introduces support for such use cases in libbpf by adding a new section: SEC("kprobe/SUBSYSTEM/PROBE")

- Future work
Extend support for dynamic tracepoints in bpftrace.

Seems like the primary motivation is bpftrace support, so let's start there.

I believe we should extend support for this feature in BCC as well. Since this is a common and widely applicable feature, wouldn't it make sense to include it directly in libbpf?

Quoted 3 lines

I'm hesitant to include this in libbpf right now. The whole SEC("kprobe") that calls "attach_tracepoint()" underneath makes me uncomfortable.

I reused the bpf_program__attach_tracepoint() function, but if we examine its implementation closely, we can see that it actually creates a DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts); rather than a bpf_tracepoint_opts.

If naming is a significant concern, it wouldn’t be difficult to reimplement the same logic with a more appropriate name, such as attach_kprobe_based_tracepoint().

Quoted 7 lines

You can still attach to such dynamic probe today with pure libbpf (e.g., if bpftrace needs to do this, for example) by creating perf_event FD from tracefs' id, and then using bpf_program__attach_perf_event_opts() to attach to it. It will be on the user to use either tracepoint or kprobe BPF program for such attachment.

You're right—we can manually attach it in the tools, but if we can make it auto-attachable in libbpf, why not take advantage of that and simplify the process?

Quoted 4 lines

Yes, this doesn't work "declaratively" with a nice SEC("...") syntax, but at least it's doable programmatically, and that's what matters for bpftrace.

--
Regards
Yafang

#5 lore raw
Quoted 52 lines · Yafang Shao
Quoted 32 lines · Andrii Nakryiko
Quoted 30 lines · Yafang Shao

The primary goal of this change is to enable tracing of inlined kernel functions with BPF programs.

Dynamic tracepoints can be created using tools like perf-probe, debugfs, or similar utilities. For example:

  $ perf probe -a 'tcp_listendrop sk'
  $ ls /sys/kernel/debug/tracing/events/probe/tcp_listendrop/
  enable  filter  format  hist  id  trigger

Here, tcp_listendrop() is an example of an inlined kernel function.

While these dynamic tracepoints are functional, they cannot be easily attached to BPF programs. For instance, attempting to use them with bpftrace results in the following error:

  $ bpftrace -l 'tracepoint:probe:*'
  tracepoint:probe:tcp_listendrop
  $ bpftrace -e 'tracepoint:probe:tcp_listendrop {print(comm)}'
  Attaching 1 probe...
  ioctl(PERF_EVENT_IOC_SET_BPF): Invalid argument
  ERROR: Error attaching probe: tracepoint:probe:tcp_listendrop

The issue lies in how these dynamic tracepoints are implemented: despite being exposed as tracepoints, they remain kprobe events internally. As a result, loading them as a tracepoint program fails. Instead, they must be loaded as kprobe programs.

Quoted 8 lines
Quoted 5 lines

This change introduces support for such use cases in libbpf by adding a new section: SEC("kprobe/SUBSYSTEM/PROBE")

- Future work
Extend support for dynamic tracepoints in bpftrace.

Seems like the primary motivation is bpftrace support, so let's start there.

I believe we should extend support for this feature in BCC as well. Since this is a common and widely applicable feature, wouldn't it make sense to include it directly in libbpf?

Quoted 3 lines

I'm hesitant to include this in libbpf right now. The whole SEC("kprobe") that calls "attach_tracepoint()" underneath makes me uncomfortable.

Quoted 8 lines

I reused the bpf_program__attach_tracepoint() function, but if we examine its implementation closely, we can see that it actually creates a DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts); rather than a bpf_tracepoint_opts.

If naming is a significant concern, it wouldn’t be difficult to reimplement the same logic with a more appropriate name, such as attach_kprobe_based_tracepoint().

"kprobe based tracepoint" is exactly the thing that makes me unwilling to include it right now. Let's put this on hold for a little bit and see how bpftrace users use it first, we can always add something to libbpf later.

Quoted 23 lines
Quoted 8 lines

You can still attach to such dynamic probe today with pure libbpf (e.g., if bpftrace needs to do this, for example) by creating perf_event FD from tracefs' id, and then using bpf_program__attach_perf_event_opts() to attach to it. It will be on the user to use either tracepoint or kprobe BPF program for such attachment.

You're right—we can manually attach it in the tools, but if we can make it auto-attachable in libbpf, why not take advantage of that and simplify the process?

Quoted 4 lines

Yes, this doesn't work "declaratively" with a nice SEC("...") syntax, but at least it's doable programmatically, and that's what matters for bpftrace.

--
Regards
Yafang