libbpf: Add support for dynamic tracepoints
Subsystems (6)
v2 cover libbpf: Add support for dynamic tracepoints
The primary goal of this change is to enable tracing of inlined kernel functions with BPF programs.
Dynamic tracepoints can be created using tools like perf-probe, debugfs, or similar utilities. For example:
$ perf probe -a 'tcp_listendrop sk' $ ls /sys/kernel/debug/tracing/events/probe/tcp_listendrop/ enable filter format hist id trigger
Here, tcp_listendrop() is an example of an inlined kernel function.
While these dynamic tracepoints are functional, they cannot be easily attached to BPF programs. For instance, attempting to use them with bpftrace results in the following error:
$ bpftrace -l 'tracepoint:probe:*' tracepoint:probe:tcp_listendrop
$ bpftrace -e 'tracepoint:probe:tcp_listendrop {print(comm)}'
Attaching 1 probe...
ioctl(PERF_EVENT_IOC_SET_BPF): Invalid argument
ERROR: Error attaching probe: tracepoint:probe:tcp_listendropThe issue lies in how these dynamic tracepoints are implemented: despite being exposed as tracepoints, they remain kprobe events internally. As a result, loading them as a tracepoint program fails. Instead, they must be loaded as kprobe programs.
This change introduces support for such use cases in libbpf by adding a new section: SEC("kprobe/SUBSYSTEM/PROBE")
- Future work
Extend support for dynamic tracepoints in bpftrace.
Changes:v1->v2:
- Use a new SEC("kprobe/SUBSYSTEM/PROBE") instead (Jiri)
v1: https://lore.kernel.org/bpf/[email protected]/
Yafang Shao (2):
libbpf: Add support for dynamic tracepoint selftests/bpf: Add selftest for dynamic tracepoint
tools/lib/bpf/libbpf.c | 29 ++++++++- .../bpf/prog_tests/test_dynamic_tp.c | 64 +++++++++++++++++++ .../testing/selftests/bpf/progs/dynamic_tp.c | 27 ++++++++ 3 files changed, 119 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/test_dynamic_tp.c create mode 100644 tools/testing/selftests/bpf/progs/dynamic_tp.c
Signature · 1 line
2.43.5
RFC PATCH v2 1/2 libbpf: Add support for dynamic tracepoint
Dynamic tracepoints can be created using debugfs, perf or similar tools. For example:
$ perf probe -a 'tcp_listendrop sk'
This command creates a new tracepoint under debugfs:
$ ls /sys/kernel/debug/tracing/events/probe/tcp_listendrop/ enable filter format hist id trigger
Notably, the probed function tcp_listendrop() is an inlined kernel function.
Although this dynamic tracepoint appears as a tracepoint, it is internally implemented as a kprobe. Therefore, if we want to attach a bpf prog to it, the bpf prog must be loaded as a kprobe prog.
The primary motivation for adding support for dynamic tracepoints is to simplify tracing of inlined kernel functions using BPF tools, such as bpftrace. By leveraging tools like perf, users can create a dynamic tracepoint for an inlined kernel function and then attach a BPF program to it.
To achieve this, a new section, SEC("kprobe/SUBSYSTEM/PROBE"), has been introduced.
--- tools/lib/bpf/libbpf.c | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-)
tools/lib/bpf/libbpf.c +28 -1
@@ -11600,11 +11600,34 @@ bpf_program__attach_kprobe_multi_opts(const struct bpf_program *prog,return libbpf_err_ptr (err); } +/* A dynamic tracepoint: "kprobe/SUBSYSTEM/PROBE" */ +static int attach_dynamic_tracepoint (const struct bpf_program *prog,const char *func_name, +struct bpf_link **link) +{ +char *tp_subsys, *tp_name; + + tp_subsys =strdup (func_name); +if (!tp_subsys) +return -ENOMEM ; + + tp_name =strchr (tp_subsys,'/' ); +if (!tp_name) { +free (tp_subsys); +return -EINVAL ; + } + + *tp_name ='\0' ; + tp_name++; + *link =bpf_program__attach_tracepoint (prog, tp_subsys, tp_name); +free (tp_subsys); +return libbpf_get_error (*link); +} +static int attach_kprobe (const struct bpf_program *prog,long cookie,struct bpf_link **link) {DECLARE_LIBBPF_OPTS (bpf_kprobe_opts, opts); +const char *func_name, *dynamic_tp;unsigned long offset =0 ; -const char *func_name;char *func;int n;
@@ -11620,6 +11643,10 @@ static int attach_kprobe(const struct bpf_program *prog, long cookie, struct bpfelse func_name = prog->sec_name +sizeof ("kprobe/" ) -1 ; + dynamic_tp =strchr (func_name,'/' ); +if (dynamic_tp) +return attach_dynamic_tracepoint (prog, func_name, link); + n =sscanf (func_name,"%m[a-zA-Z0-9_.]+%li" , &func, &offset);if (n <1 ) {pr_warn ("kprobe name is invalid: %s\n" , func_name);
Signature · 1 line
2.43.5
RFC PATCH v2 2/2 selftests/bpf: Add selftest for dynamic tracepoint
In reply to Yafang ShaoThe result is as follows,
$ tools/testing/selftests/bpf/test_progs --name=dynamic_tp #85 dynamic_tp:OK Summary: 1/0 PASSED, 0 SKIPPED, 0 FAILED
--- .../bpf/prog_tests/test_dynamic_tp.c | 64 +++++++++++++++++++ .../testing/selftests/bpf/progs/dynamic_tp.c | 27 ++++++++ 2 files changed, 91 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/test_dynamic_tp.c create mode 100644 tools/testing/selftests/bpf/progs/dynamic_tp.c
tools/testing/selftests/bpf/prog_tests/test_dynamic_tp.c new file +64 -0
@@ -0,0 +1,64 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include <test_progs.h> +#include <bpf/btf.h> +#include <bpf/bpf.h> + +#include "dynamic_tp.skel.h" + +int dynamic_tp (const char *cmd) +{ +const char *kprobe_file ="/sys/kernel/debug/tracing/kprobe_events" ; +ssize_t bytes_written; +int fd, err; + + fd =open (kprobe_file,O_WRONLY |O_APPEND ); +if (!ASSERT_GE (fd,0 ,"open kprobe_events" )) +return -1 ; + + bytes_written =write (fd, cmd,strlen (cmd)); +if (!ASSERT_GT (bytes_written,0 ,"write kprobe_events" )) { +close (fd); +return -1 ; + } + + err =close (fd); +if (!ASSERT_OK (err,"close kprobe_events" )) +return -1 ; +return 0 ; +} + +void test_dynamic_tp (void ) +{ +struct dynamic_tp *skel; +pid_t child_pid; +int status, err; + +/* create a dynamic tracepoint */ + err =dynamic_tp ("p:my_dynamic_tp kernel_clone" ); +if (!ASSERT_OK (err,"create dynamic tp" )) +return ; + + skel =dynamic_tp__open_and_load (); +if (!ASSERT_OK_PTR (skel,"load progs" )) + goto remove_tp; + skel->bss->pid =getpid (); + err =dynamic_tp__attach (skel); +if (!ASSERT_OK (err,"attach progs" )) + goto cleanup; + +/* trigger the dynamic tracepoint */ + child_pid =fork (); +if (!ASSERT_GT (child_pid,-1 ,"child_pid" )) + goto cleanup; +if (child_pid ==0 ) +_exit (0 ); +waitpid (child_pid, &status,0 ); + +ASSERT_EQ (skel->bss->result,1 ,"result" ); + +cleanup: +dynamic_tp__destroy (skel); +remove_tp: +dynamic_tp ("-:my_dynamic_tp kernel_clone" ); +}
tools/testing/selftests/bpf/progs/dynamic_tp.c new file +27 -0
@@ -0,0 +1,27 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> + +char _license[] SEC("license") = "GPL"; + +#define MAX_STACK_TRACE_DEPTH 32 +unsigned long entries[MAX_STACK_TRACE_DEPTH ] = {}; +#define SIZE_OF_ULONG (sizeof(unsigned long)) + +int result, pid; + +SEC("kprobe/kprobes/my_dynamic_tp") +int dynamic_tp (struct pt_regs *ctx) +{ +int ret; + + ret =bpf_get_stack (ctx, entries,MAX_STACK_TRACE_DEPTH *SIZE_OF_ULONG ,0 ); +if (ret <0 ) { + result =-1 ; +return ret; + } +if (bpf_get_current_pid_tgid () >>32 == pid) + result =1 ; +return 0 ; +}
Signature · 1 line
2.43.5
Quoted 30 lines · Yafang Shao
The primary goal of this change is to enable tracing of inlined kernel functions with BPF programs.
Dynamic tracepoints can be created using tools like perf-probe, debugfs, or similar utilities. For example:
$ perf probe -a 'tcp_listendrop sk' $ ls /sys/kernel/debug/tracing/events/probe/tcp_listendrop/ enable filter format hist id triggerHere,
tcp_listendrop()is an example of an inlined kernel function.While these dynamic tracepoints are functional, they cannot be easily attached to BPF programs. For instance, attempting to use them with bpftrace results in the following error:
$ bpftrace -l 'tracepoint:probe:*' tracepoint:probe:tcp_listendrop$ bpftrace -e 'tracepoint:probe:tcp_listendrop {print(comm)}' Attaching 1 probe... ioctl(PERF_EVENT_IOC_SET_BPF): Invalid argument ERROR: Error attaching probe: tracepoint:probe:tcp_listendropThe issue lies in how these dynamic tracepoints are implemented: despite being exposed as tracepoints, they remain kprobe events internally. As a result, loading them as a tracepoint program fails. Instead, they must be loaded as kprobe programs.
Quoted 5 lines
This change introduces support for such use cases in libbpf by adding a new section:
SEC("kprobe/SUBSYSTEM/PROBE")- Future work
Extend support for dynamic tracepoints in bpftrace.
Seems like the primary motivation is bpftrace support, so let's start there. I'm hesitant to include this in libbpf right now. The whole SEC("kprobe") that calls "attach_tracepoint()" underneath makes me uncomfortable.
You can still attach to such dynamic probe today with pure libbpf (e.g., if bpftrace needs to do this, for example) by creating perf_event FD from tracefs' id, and then using bpf_program__attach_perf_event_opts() to attach to it. It will be on the user to use either tracepoint or kprobe BPF program for such attachment.
Yes, this doesn't work "declaratively" with a nice SEC("...") syntax, but at least it's doable programmatically, and that's what matters for bpftrace.
Quoted 21 lines
Changes:
v1->v2:
- Use a newSEC("kprobe/SUBSYSTEM/PROBE")instead (Jiri)v1: https://lore.kernel.org/bpf/[email protected]/
Yafang Shao (2):
libbpf: Add support for dynamic tracepoint selftests/bpf: Add selftest for dynamic tracepointtools/lib/bpf/libbpf.c | 29 ++++++++- .../bpf/prog_tests/test_dynamic_tp.c | 64 +++++++++++++++++++ .../testing/selftests/bpf/progs/dynamic_tp.c | 27 ++++++++ 3 files changed, 119 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/test_dynamic_tp.c create mode 100644 tools/testing/selftests/bpf/progs/dynamic_tp.c--
2.43.5
Quoted 32 lines · Andrii Nakryiko
Quoted 30 lines · Yafang Shao
The primary goal of this change is to enable tracing of inlined kernel functions with BPF programs.
Dynamic tracepoints can be created using tools like perf-probe, debugfs, or similar utilities. For example:
$ perf probe -a 'tcp_listendrop sk' $ ls /sys/kernel/debug/tracing/events/probe/tcp_listendrop/ enable filter format hist id triggerHere,
tcp_listendrop()is an example of an inlined kernel function.While these dynamic tracepoints are functional, they cannot be easily attached to BPF programs. For instance, attempting to use them with bpftrace results in the following error:
$ bpftrace -l 'tracepoint:probe:*' tracepoint:probe:tcp_listendrop$ bpftrace -e 'tracepoint:probe:tcp_listendrop {print(comm)}' Attaching 1 probe... ioctl(PERF_EVENT_IOC_SET_BPF): Invalid argument ERROR: Error attaching probe: tracepoint:probe:tcp_listendropThe issue lies in how these dynamic tracepoints are implemented: despite being exposed as tracepoints, they remain kprobe events internally. As a result, loading them as a tracepoint program fails. Instead, they must be loaded as kprobe programs.
Quoted 8 lines
Quoted 5 lines
This change introduces support for such use cases in libbpf by adding a new section:
SEC("kprobe/SUBSYSTEM/PROBE")- Future work
Extend support for dynamic tracepoints in bpftrace.Seems like the primary motivation is bpftrace support, so let's start there.
I believe we should extend support for this feature in BCC as well. Since this is a common and widely applicable feature, wouldn't it make sense to include it directly in libbpf?
Quoted 3 lines
I'm hesitant to include this in libbpf right now. The whole
SEC("kprobe")that calls "attach_tracepoint()" underneath makes me uncomfortable.
I reused the bpf_program__attach_tracepoint() function, but if we examine its implementation closely, we can see that it actually creates a DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts); rather than a bpf_tracepoint_opts.
If naming is a significant concern, it wouldn’t be difficult to reimplement the same logic with a more appropriate name, such as attach_kprobe_based_tracepoint().
Quoted 7 lines
You can still attach to such dynamic probe today with pure libbpf (e.g., if bpftrace needs to do this, for example) by creating
perf_eventFD from tracefs' id, and then usingbpf_program__attach_perf_event_opts()to attach to it. It will be on the user to use either tracepoint or kprobe BPF program for such attachment.
You're right—we can manually attach it in the tools, but if we can make it auto-attachable in libbpf, why not take advantage of that and simplify the process?
Quoted 4 lines
Yes, this doesn't work "declaratively" with a nice
SEC("...")syntax, but at least it's doable programmatically, and that's what matters for bpftrace.
--
Regards
Yafang
Quoted 52 lines · Yafang Shao
Quoted 32 lines · Andrii Nakryiko
Quoted 30 lines · Yafang Shao
The primary goal of this change is to enable tracing of inlined kernel functions with BPF programs.
Dynamic tracepoints can be created using tools like perf-probe, debugfs, or similar utilities. For example:
$ perf probe -a 'tcp_listendrop sk' $ ls /sys/kernel/debug/tracing/events/probe/tcp_listendrop/ enable filter format hist id triggerHere,
tcp_listendrop()is an example of an inlined kernel function.While these dynamic tracepoints are functional, they cannot be easily attached to BPF programs. For instance, attempting to use them with bpftrace results in the following error:
$ bpftrace -l 'tracepoint:probe:*' tracepoint:probe:tcp_listendrop$ bpftrace -e 'tracepoint:probe:tcp_listendrop {print(comm)}' Attaching 1 probe... ioctl(PERF_EVENT_IOC_SET_BPF): Invalid argument ERROR: Error attaching probe: tracepoint:probe:tcp_listendropThe issue lies in how these dynamic tracepoints are implemented: despite being exposed as tracepoints, they remain kprobe events internally. As a result, loading them as a tracepoint program fails. Instead, they must be loaded as kprobe programs.
Quoted 8 lines
Quoted 5 lines
This change introduces support for such use cases in libbpf by adding a new section:
SEC("kprobe/SUBSYSTEM/PROBE")- Future work
Extend support for dynamic tracepoints in bpftrace.Seems like the primary motivation is bpftrace support, so let's start there.
I believe we should extend support for this feature in BCC as well. Since this is a common and widely applicable feature, wouldn't it make sense to include it directly in libbpf?
Quoted 3 lines
I'm hesitant to include this in libbpf right now. The whole
SEC("kprobe")that calls "attach_tracepoint()" underneath makes me uncomfortable.
Quoted 8 lines
I reused the
bpf_program__attach_tracepoint()function, but if we examine its implementation closely, we can see that it actually creates aDECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts); rather than abpf_tracepoint_opts.If naming is a significant concern, it wouldn’t be difficult to reimplement the same logic with a more appropriate name, such as
attach_kprobe_based_tracepoint().
"kprobe based tracepoint" is exactly the thing that makes me unwilling to include it right now. Let's put this on hold for a little bit and see how bpftrace users use it first, we can always add something to libbpf later.
Quoted 23 lines
Quoted 8 lines
You can still attach to such dynamic probe today with pure libbpf (e.g., if bpftrace needs to do this, for example) by creating
perf_eventFD from tracefs' id, and then usingbpf_program__attach_perf_event_opts()to attach to it. It will be on the user to use either tracepoint or kprobe BPF program for such attachment.You're right—we can manually attach it in the tools, but if we can make it auto-attachable in libbpf, why not take advantage of that and simplify the process?
Quoted 4 lines
Yes, this doesn't work "declaratively" with a nice
SEC("...")syntax, but at least it's doable programmatically, and that's what matters for bpftrace.--
Regards
Yafang