libbpf Userspace Runtime-Defined Tracing (URDT)
Subsystems (4)
cover libbpf Userspace Runtime-Defined Tracing (URDT)
Adding userspace tracepoints in other languages like python and go is a very useful for observability. libstapsdt [1] and language bindings like python-stapsdt [2] that rely on it use a clever scheme of emulating static (USDT) userspace tracepoints at runtime. This involves (as I understand it):
- fabricating a shared library
- annotating it with ELF notes that describe its tracepoints
- dlopen()ing it and calling the appropriate probe fire function to trigger probe firing.
bcc already supports this mechanism (the examples in [2] use bcc to list/trigger the tracepoints), so it seems like it would be a good candidate for adding support to libbpf.
However, before doing that, it's worth considering if there are simpler ways to support runtime probe firing. This small series demonstrates a simple method based on USDT probes added to libbpf itself.
The suggested solution comprises 3 parts
1. functions to fire dynamic probes are added to libbpf itself bpf_urdt__probeN(), where N is the number of probe arguemnts. A sample usage would be
bpf_urdt__probe3("myprovider", "myprobe", 1, 2, 3);Under the hood these correspond to USDT probes with an additional argument for uniquely identifying the probe (a hash of provider/probe name).
2. we attach to the appropriate USDT probe for the specified number of arguments urdt/probe0 for none, urdt/probe1 for 1, etc. We utilize the high-order 32 bits of the attach cookie to store the hash of the provider/probe name.
3. when urdt/probeN fires, the BPF_URDT() macro (which is similar to BPF_USDT()) checks if the hash passed in (identifying provider/probe) matches the attach cookie high-order 32 bits; if not it must be a firing for a different dynamic probe and we exit early.
Auto-attach support is also added, for example the following would add a dynamic probe for provider:myprobe:
SEC("udrt/libbpf.so:2:myprovider:myprobe")
int BPF_URDT(myprobe, int arg1, char *arg2)
{
...
}
(Note the "2" above specifies the number of arguments to the probe, otherwise it is identical to USDT).
The above program can then be triggered by a call to
BPF_URDT_PROBE2("myprovider", "myprobe", 1, "hi");
The useful thing about this is that by attaching to libbpf.so (and firing probes using that library) we can get system-wide dynamic probe firing. It is also easy to fire a dynamic probe - no setup is required.
More examples of auto and manual attach can be found in the selftests (patch 2).
If this approach appears to be worth pursing, we could also look at adding support to libstapsdt for it.
Alan Maguire (2):
libbpf: add support for Userspace Runtime Dynamic Tracing (URDT) selftests/bpf: add tests for Userspace Runtime Defined Tracepoints
(URDT)
tools/lib/bpf/Build | 2 +- tools/lib/bpf/Makefile | 2 +- tools/lib/bpf/libbpf.c | 94 ++++++++++ tools/lib/bpf/libbpf.h | 94 ++++++++++ tools/lib/bpf/libbpf.map | 13 ++ tools/lib/bpf/libbpf_internal.h | 2 + tools/lib/bpf/urdt.bpf.h | 103 +++++++++++ tools/lib/bpf/urdt.c | 145 +++++++++++++++ tools/testing/selftests/bpf/Makefile | 2 +- tools/testing/selftests/bpf/prog_tests/urdt.c | 173 ++++++++++++++++++ tools/testing/selftests/bpf/progs/test_urdt.c | 100 ++++++++++ .../selftests/bpf/progs/test_urdt_shared.c | 59 ++++++ 12 files changed, 786 insertions(+), 3 deletions(-) create mode 100644 tools/lib/bpf/urdt.bpf.h create mode 100644 tools/lib/bpf/urdt.c create mode 100644 tools/testing/selftests/bpf/prog_tests/urdt.c create mode 100644 tools/testing/selftests/bpf/progs/test_urdt.c create mode 100644 tools/testing/selftests/bpf/progs/test_urdt_shared.c
Signature · 1 line
2.39.3
RFC 1/2 libbpf: add support for Userspace Runtime Dynamic Tracing (URDT)
Userspace Static Defined Tracing (USDT) provides a means to add tracepoints to userspace code; these are marked by a no-op and described in a .notes.stapsdt section, and libbpf provides mechanisms to support tracing USDT via uprobes and the USDT manager.
However, in order to be able to add tracepoints to userspace code in other languages like python or go, a library call (that can then be wrapped in the runtime-specific glue code) is required. One example of how such runtime probes are supported is libstapsdt [1]; it creates a shared library and annotations on the fly in order to emulate USDT. BCC supports this mechanism, and one option would be to add similar support to libbpf. However, it would involve retrieving the ELF notes from the dynamically-created library, which at first glance looked tricky.
The approach here is simpler; essentially we bootstrap from USDT to supporting runtime probes. USDT probes marking runtime probe firing are added to libbpf itself, and these are passed the probe arguments along with a hash of provider/probe names. For example, to fire a URDT probe, the user simply calls:
BPF_URDT_PROBE3("myprovider", "myprobe", arg1, arg2, arg3);There is no need to declare such probes in advance.
Under the hood, an associated USDT probe then fires, and this is the triggering mechanism for URDT. All that is left to do on the BPF side then is
1. ensure that the probe firing is the one we are tracing; this is done by matching the hash of provider/probe passed into the USDT probe with the cookie associated with the URDT attachment. The high-order 32 bits are used for this; the rest is still available to the user to set as a per-attachment cookie value. A mismatch here triggers early exit.
2. map from USDT -> URDT arguments. urdt.bpf.h provides means to do this; it is simply a matter of ignoring the last USDT hash argument.
BPF_URDT() is a macro similar to BPF_USDT() that handles probe matching, argument assignment etc.
From the BPF side, a URDT BPF program will look like this:
SEC("urdt:/path:2:myprovider:myprobe")
int BPF_URDT(myprobe, int arg1, const char *arg2)
{
...
}Note that prior to provider probe, "2" is specified to tell auto-attach that we will attach to the equivalent 2-argument URDT probe (which is a 3-argument USDT probe under the hood). The path should be the binary that calls the firing functions itself (if libbpf is statically linked like test_progs), or the path to libbpf.so (if libbpf is dynamically linked), since we want to attach to the USDT probe firings in libbpf itself. Future work could add this URDT firing mechanism to libstapsdt also.
It appears to have a few advantages over the current libstapsdt method:
- it supports global runtime probes. As I understand it, the on-the-fly library creation will be specific to the process that triggers the probe in libstapsdt, whereas with the above approach we can instrument the libbpf shared library such that firings will appear system-wide.
- it is a bit simpler to fire probes; no prior setup is required
However, it may be preferable to add support for the existing libstapsdt approach to libbpf. This RFC is just intended to start that discussion.
[1] https://github.com/linux-usdt/libstapsdt
--- tools/lib/bpf/Build | 2 +- tools/lib/bpf/Makefile | 2 +- tools/lib/bpf/libbpf.c | 94 +++++++++++++++++++++ tools/lib/bpf/libbpf.h | 94 +++++++++++++++++++++ tools/lib/bpf/libbpf.map | 13 +++ tools/lib/bpf/libbpf_internal.h | 2 + tools/lib/bpf/urdt.bpf.h | 103 +++++++++++++++++++++++ tools/lib/bpf/urdt.c | 145 ++++++++++++++++++++++++++++++++ 8 files changed, 453 insertions(+), 2 deletions(-) create mode 100644 tools/lib/bpf/urdt.bpf.h create mode 100644 tools/lib/bpf/urdt.c
tools/lib/bpf/Build +1 -1
@@ -1,4 +1,4 @@ libbpf-y := libbpf.o bpf.o nlattr.o btf.o libbpf_errno.o str_error.o \ netlink.o bpf_prog_linfo.o libbpf_probes.o hashmap.o \ btf_dump.o ringbuf.o strset.o linker.o gen_loader.o relo_core.o \ - usdt.o zip.o elf.o features.o + usdt.o urdt.o zip.o elf.o features.o
tools/lib/bpf/Makefile +1 -1
@@ -239,7 +239,7 @@ install_lib: all_cmd SRC_HDRS := bpf.h libbpf.h btf.h libbpf_common.h libbpf_legacy.h \ bpf_helpers.h bpf_tracing.h bpf_endian.h bpf_core_read.h \ - skel_internal.h libbpf_version.h usdt.bpf.h + skel_internal.h libbpf_version.h usdt.bpf.h urdt.bpf.h GEN_HDRS := $(BPF_GENERATED) INSTALL_PFX := $(DESTDIR)$(prefix)/include/bpf
tools/lib/bpf/libbpf.c +94 -0
112 lines not on this page · raw message
tools/lib/bpf/libbpf.h +94 -0
100 lines not on this page · raw message
tools/lib/bpf/libbpf.map +13 -0
@@ -412,4 +412,17 @@ LIBBPF_1.3.0 { LIBBPF_1.4.0 { bpf_token_create; + bpf_program__attach_urdt; + bpf_urdt__probe0; + bpf_urdt__probe1; + bpf_urdt__probe2; + bpf_urdt__probe3; + bpf_urdt__probe4; + bpf_urdt__probe5; + bpf_urdt__probe6; + bpf_urdt__probe7; + bpf_urdt__probe8; + bpf_urdt__probe9; + bpf_urdt__probe10; + bpf_urdt__probe11; } LIBBPF_1.3.0;
tools/lib/bpf/libbpf_internal.h +2 -0
@@ -624,6 +624,8 @@ struct bpf_link * usdt_manager_attach_usdt(struct usdt_manager *man, const char *usdt_provider, const char *usdt_name, __u64 usdt_cookie); +unsigned int urdt_probe_hash (const char *provider,const char *probe); +static inline bool is_pow_of_2 (size_t x) {return x && (x & (x -1 )) ==0 ;
tools/lib/bpf/urdt.bpf.h new file +103 -0
103 lines not on this page · raw message
tools/lib/bpf/urdt.c new file +145 -0
145 lines not on this page · raw message
Signature · 1 line
2.39.3
RFC 2/2 selftests/bpf: add tests for Userspace Runtime Defined Tracepoints (URDT)
In reply to Alan MaguireAdd tests that verify operation of URDT probes for both the statically and dynamically-linked libbpf cases.
Ensure probe attach and auto-attach succeed, and argument counts, cookies and argument values match expectations.
--- tools/testing/selftests/bpf/Makefile | 2 +- tools/testing/selftests/bpf/prog_tests/urdt.c | 173 ++++++++++++++++++ tools/testing/selftests/bpf/progs/test_urdt.c | 100 ++++++++++ .../selftests/bpf/progs/test_urdt_shared.c | 59 ++++++ 4 files changed, 333 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/urdt.c create mode 100644 tools/testing/selftests/bpf/progs/test_urdt.c create mode 100644 tools/testing/selftests/bpf/progs/test_urdt_shared.c
tools/testing/selftests/bpf/Makefile +1 -1
@@ -39,7 +39,7 @@ CFLAGS += -g $(OPT_FLAGS) -rdynamic \ -I$(CURDIR) -I$(INCLUDE_DIR) -I$(GENDIR) -I$(LIBDIR) \ -I$(TOOLSINCDIR) -I$(APIDIR) -I$(OUTPUT) LDFLAGS += $(SAN_LDFLAGS) -LDLIBS += $(LIBELF_LIBS) -lz -lrt -lpthread +LDLIBS += $(LIBELF_LIBS) -lz -lrt -lpthread -ldl ifneq ($(LLVM),) # Silence some warnings when compiled with clang
tools/testing/selftests/bpf/prog_tests/urdt.c new file +173 -0
173 lines not on this page · raw message
tools/testing/selftests/bpf/progs/test_urdt.c new file +100 -0
100 lines not on this page · raw message
tools/testing/selftests/bpf/progs/test_urdt_shared.c new file +59 -0
59 lines not on this page · raw message
Signature · 1 line
2.39.3
Hi Alan,
Quoted 63 lines · Alan Maguire
Adding userspace tracepoints in other languages like python and go is a very useful for observability. libstapsdt [1] and language bindings like python-stapsdt [2] that rely on it use a clever scheme of emulating static (USDT) userspace tracepoints at runtime. This involves (as I understand it):
- fabricating a shared library
- annotating it with ELF notes that describe its tracepoints
- dlopen()ing it and calling the appropriate probe fire function to trigger probe firing.bcc already supports this mechanism (the examples in [2] use bcc to list/trigger the tracepoints), so it seems like it would be a good candidate for adding support to libbpf.
However, before doing that, it's worth considering if there are simpler ways to support runtime probe firing. This small series demonstrates a simple method based on USDT probes added to libbpf itself.
The suggested solution comprises 3 parts
1. functions to fire dynamic probes are added to libbpf itself
bpf_urdt__probeN(), where N is the number of probe arguemnts. A sample usage would bebpf_urdt__probe3("myprovider", "myprobe", 1, 2, 3);Under the hood these correspond to USDT probes with an additional argument for uniquely identifying the probe (a hash of provider/probe name).
2. we attach to the appropriate USDT probe for the specified number of arguments urdt/probe0 for none, urdt/probe1 for 1, etc. We utilize the high-order 32 bits of the attach cookie to store the hash of the provider/probe name.
3. when urdt/probeN fires, the
BPF_URDT()macro (which is similar toBPF_USDT()) checks if the hash passed in (identifying provider/probe) matches the attach cookie high-order 32 bits; if not it must be a firing for a different dynamic probe and we exit early.Auto-attach support is also added, for example the following would add a dynamic probe for provider:myprobe:
SEC("udrt/libbpf.so:2:myprovider:myprobe")
intBPF_URDT(myprobe, int arg1, char *arg2)
{
...
}(Note the "2" above specifies the number of arguments to the probe, otherwise it is identical to USDT).
The above program can then be triggered by a call to
BPF_URDT_PROBE2("myprovider", "myprobe", 1, "hi");The useful thing about this is that by attaching to libbpf.so (and firing probes using that library) we can get system-wide dynamic probe firing. It is also easy to fire a dynamic probe - no setup is required.
Quoted 5 lines
More examples of auto and manual attach can be found in the selftests (patch 2).
If this approach appears to be worth pursing, we could also look at adding support to libstapsdt for it.
This is quite interesting, thanks for the RFC. I hope to take a closer look at it this week.
At a high level, it looks like you're basically defining a scheme for well-known USDT probes, right? Since, not all languages enjoy linking against C (looking at you golang...), perhaps it would make sense to codify the scheme in a "spec". Probably just located in Documentation/ or something. That way there can be independent implementations.
This is something that would be nice to support in bpftrace as well. I'm sure other tracers would probably find use as well.
Thanks,
Daniel
Quoted 72 lines · Daniel Xu
Hi Alan,
Quoted 63 lines · Alan Maguire
Adding userspace tracepoints in other languages like python and go is a very useful for observability. libstapsdt [1] and language bindings like python-stapsdt [2] that rely on it use a clever scheme of emulating static (USDT) userspace tracepoints at runtime. This involves (as I understand it):
- fabricating a shared library
- annotating it with ELF notes that describe its tracepoints
- dlopen()ing it and calling the appropriate probe fire function to trigger probe firing.bcc already supports this mechanism (the examples in [2] use bcc to list/trigger the tracepoints), so it seems like it would be a good candidate for adding support to libbpf.
However, before doing that, it's worth considering if there are simpler ways to support runtime probe firing. This small series demonstrates a simple method based on USDT probes added to libbpf itself.
The suggested solution comprises 3 parts
1. functions to fire dynamic probes are added to libbpf itself
bpf_urdt__probeN(), where N is the number of probe arguemnts. A sample usage would bebpf_urdt__probe3("myprovider", "myprobe", 1, 2, 3);Under the hood these correspond to USDT probes with an additional argument for uniquely identifying the probe (a hash of provider/probe name).
2. we attach to the appropriate USDT probe for the specified number of arguments urdt/probe0 for none, urdt/probe1 for 1, etc. We utilize the high-order 32 bits of the attach cookie to store the hash of the provider/probe name.
3. when urdt/probeN fires, the
BPF_URDT()macro (which is similar toBPF_USDT()) checks if the hash passed in (identifying provider/probe) matches the attach cookie high-order 32 bits; if not it must be a firing for a different dynamic probe and we exit early.Auto-attach support is also added, for example the following would add a dynamic probe for provider:myprobe:
SEC("udrt/libbpf.so:2:myprovider:myprobe")
intBPF_URDT(myprobe, int arg1, char *arg2)
{
...
}(Note the "2" above specifies the number of arguments to the probe, otherwise it is identical to USDT).
The above program can then be triggered by a call to
BPF_URDT_PROBE2("myprovider", "myprobe", 1, "hi");The useful thing about this is that by attaching to libbpf.so (and firing probes using that library) we can get system-wide dynamic probe firing. It is also easy to fire a dynamic probe - no setup is required.
Quoted 5 lines
More examples of auto and manual attach can be found in the selftests (patch 2).
If this approach appears to be worth pursing, we could also look at adding support to libstapsdt for it.
Quoted 9 lines
This is quite interesting, thanks for the RFC. I hope to take a closer look at it this week.
At a high level, it looks like you're basically defining a scheme for well-known USDT probes, right? Since, not all languages enjoy linking against C (looking at you golang...), perhaps it would make sense to codify the scheme in a "spec". Probably just located in
Documentation/or something. That way there can be independent implementations.
That's a great idea. If tracers or libraries like libstapsdt wanted to define their own probe firings, we could add option fields to the urdt attach function to specify the USDT provider/probe names for attachment for runtime probe emulation. If another entity was specifying USDT probes like this, it would also need to pass through a hash value, so that triple could potentially be added to the urdt attach options; doing that would give other entities like libstapsdt a free hand to define their own trigger mechanisms, and not have to link to libbpf. That said, the benefits of a common scheme are that libbpf does not have to figure out which USDT probes to use in advance, so having defaults that are documented makes sense. It's especially useful for auto-attach by SEC() name.
Quoted 3 lines
This is something that would be nice to support in bpftrace as well. I'm sure other tracers would probably find use as well.
Yeah, I was aiming for as simple as possible to make it easy for tracers to adopt. The aim is to try and facilitate wider support for runtime probes, so I'm curious if other folks have run into issues in that area, or have suggestions.
Thanks!
Alan
Quoted 2 lines
Thanks,
Daniel
Quoted 1 line · Alan Maguire
On 31/01/2024 17:06, Daniel Xu wrote:
[..]
Quoted 9 lines
Quoted 4 lines
This is something that would be nice to support in bpftrace as well. I'm sure other tracers would probably find use as well.
Yeah, I was aiming for as simple as possible to make it easy for tracers to adopt. The aim is to try and facilitate wider support for runtime probes, so I'm curious if other folks have run into issues in that area, or have suggestions.
I think user_events is a thing: https://docs.kernel.org/trace/user_events.html
Might be good to take a look at that and see how URDT compares.
Quoted 20 lines · Alan Maguire
Adding userspace tracepoints in other languages like python and go is a very useful for observability. libstapsdt [1] and language bindings like python-stapsdt [2] that rely on it use a clever scheme of emulating static (USDT) userspace tracepoints at runtime. This involves (as I understand it):
- fabricating a shared library
- annotating it with ELF notes that describe its tracepoints
- dlopen()ing it and calling the appropriate probe fire function to trigger probe firing.bcc already supports this mechanism (the examples in [2] use bcc to list/trigger the tracepoints), so it seems like it would be a good candidate for adding support to libbpf.
However, before doing that, it's worth considering if there are simpler ways to support runtime probe firing. This small series demonstrates a simple method based on USDT probes added to libbpf itself.
[snip]
Quoted 11 lines
The useful thing about this is that by attaching to libbpf.so (and firing probes using that library) we can get system-wide dynamic probe firing. It is also easy to fire a dynamic probe - no setup is required.
More examples of auto and manual attach can be found in the selftests (patch 2).
If this approach appears to be worth pursing, we could also look at adding support to libstapsdt for it.
Proof-of-concept libstapsdt support has been built and tested; consumers of libstapsdt continue to work the same way but with URDT support, we can trace dynamic events system-wide. See
https://github.com/linux-usdt/libstapsdt/compare/main...alan-maguire:libstapsdt:urdt
Quoted 24 lines
Alan Maguire (2):
libbpf: add support for Userspace Runtime Dynamic Tracing (URDT) selftests/bpf: add tests for Userspace Runtime Defined Tracepoints(URDT)tools/lib/bpf/Build | 2 +- tools/lib/bpf/Makefile | 2 +- tools/lib/bpf/libbpf.c | 94 ++++++++++ tools/lib/bpf/libbpf.h | 94 ++++++++++ tools/lib/bpf/libbpf.map | 13 ++ tools/lib/bpf/libbpf_internal.h | 2 + tools/lib/bpf/urdt.bpf.h | 103 +++++++++++ tools/lib/bpf/urdt.c | 145 +++++++++++++++ tools/testing/selftests/bpf/Makefile | 2 +- tools/testing/selftests/bpf/prog_tests/urdt.c | 173 ++++++++++++++++++ tools/testing/selftests/bpf/progs/test_urdt.c | 100 ++++++++++ .../selftests/bpf/progs/test_urdt_shared.c | 59 ++++++ 12 files changed, 786 insertions(+), 3 deletions(-) create mode 100644 tools/lib/bpf/urdt.bpf.h create mode 100644 tools/lib/bpf/urdt.c create mode 100644 tools/testing/selftests/bpf/prog_tests/urdt.c create mode 100644 tools/testing/selftests/bpf/progs/test_urdt.c create mode 100644 tools/testing/selftests/bpf/progs/test_urdt_shared.c
Quoted 37 lines · Alan Maguire
Adding userspace tracepoints in other languages like python and go is a very useful for observability. libstapsdt [1] and language bindings like python-stapsdt [2] that rely on it use a clever scheme of emulating static (USDT) userspace tracepoints at runtime. This involves (as I understand it):
- fabricating a shared library
- annotating it with ELF notes that describe its tracepoints
- dlopen()ing it and calling the appropriate probe fire function to trigger probe firing.bcc already supports this mechanism (the examples in [2] use bcc to list/trigger the tracepoints), so it seems like it would be a good candidate for adding support to libbpf.
However, before doing that, it's worth considering if there are simpler ways to support runtime probe firing. This small series demonstrates a simple method based on USDT probes added to libbpf itself.
The suggested solution comprises 3 parts
1. functions to fire dynamic probes are added to libbpf itself
bpf_urdt__probeN(), where N is the number of probe arguemnts. A sample usage would bebpf_urdt__probe3("myprovider", "myprobe", 1, 2, 3);Under the hood these correspond to USDT probes with an additional argument for uniquely identifying the probe (a hash of provider/probe name).
2. we attach to the appropriate USDT probe for the specified number of arguments urdt/probe0 for none, urdt/probe1 for 1, etc. We utilize the high-order 32 bits of the attach cookie to store the hash of the provider/probe name.
Quoted 5 lines
3. when urdt/probeN fires, the
BPF_URDT()macro (which is similar toBPF_USDT()) checks if the hash passed in (identifying provider/probe) matches the attach cookie high-order 32 bits; if not it must be a firing for a different dynamic probe and we exit early.
I'm sorry Alan, but I don't see this being added to libbpf. This is nothing else than USDT with a bunch of extra conventions bolted on. And those conventions might not work for many environments. It is completely arbitrary that libbpf is a) assumed to be a dynamic library and b) provides USDT hooks that will be triggered. Just because it will be libbpf that will be used to trace those USDT hooks doesn't mean that libbpf has to define those hooks. Just because libbpf can trace USDTs it doesn't mean that libbpf should provide those STAP_PROBEx() macros to define and trigger USDTs within some application. Applications that define USDTs and applications that attach to those USDTs are completely separate and independent. Same here, there might be an overlap in some cases, but conceptually it's two separate sides of the solution.
Overall, this is definitely a useful overall approach, to have a single system-wide .so library that can be attached to trace some USDTs, and we've explored this approach internally at Meta as well. But I don't believe it should be part of libbpf. From libbpf's standpoint it's just a standard USDT probe to attach to.
Quoted 55 lines
Auto-attach support is also added, for example the following would add a dynamic probe for provider:myprobe:
SEC("udrt/libbpf.so:2:myprovider:myprobe")
intBPF_URDT(myprobe, int arg1, char *arg2)
{
...
}(Note the "2" above specifies the number of arguments to the probe, otherwise it is identical to USDT).
The above program can then be triggered by a call to
BPF_URDT_PROBE2("myprovider", "myprobe", 1, "hi");The useful thing about this is that by attaching to libbpf.so (and firing probes using that library) we can get system-wide dynamic probe firing. It is also easy to fire a dynamic probe - no setup is required.
More examples of auto and manual attach can be found in the selftests (patch 2).
If this approach appears to be worth pursing, we could also look at adding support to libstapsdt for it.
Alan Maguire (2):
libbpf: add support for Userspace Runtime Dynamic Tracing (URDT) selftests/bpf: add tests for Userspace Runtime Defined Tracepoints(URDT)tools/lib/bpf/Build | 2 +- tools/lib/bpf/Makefile | 2 +- tools/lib/bpf/libbpf.c | 94 ++++++++++ tools/lib/bpf/libbpf.h | 94 ++++++++++ tools/lib/bpf/libbpf.map | 13 ++ tools/lib/bpf/libbpf_internal.h | 2 + tools/lib/bpf/urdt.bpf.h | 103 +++++++++++ tools/lib/bpf/urdt.c | 145 +++++++++++++++ tools/testing/selftests/bpf/Makefile | 2 +- tools/testing/selftests/bpf/prog_tests/urdt.c | 173 ++++++++++++++++++ tools/testing/selftests/bpf/progs/test_urdt.c | 100 ++++++++++ .../selftests/bpf/progs/test_urdt_shared.c | 59 ++++++ 12 files changed, 786 insertions(+), 3 deletions(-) create mode 100644 tools/lib/bpf/urdt.bpf.h create mode 100644 tools/lib/bpf/urdt.c create mode 100644 tools/testing/selftests/bpf/prog_tests/urdt.c create mode 100644 tools/testing/selftests/bpf/progs/test_urdt.c create mode 100644 tools/testing/selftests/bpf/progs/test_urdt_shared.c--
2.39.3
Quoted 44 lines · Andrii Nakryiko
Quoted 37 lines · Alan Maguire
Adding userspace tracepoints in other languages like python and go is a very useful for observability. libstapsdt [1] and language bindings like python-stapsdt [2] that rely on it use a clever scheme of emulating static (USDT) userspace tracepoints at runtime. This involves (as I understand it):
- fabricating a shared library
- annotating it with ELF notes that describe its tracepoints
- dlopen()ing it and calling the appropriate probe fire function to trigger probe firing.bcc already supports this mechanism (the examples in [2] use bcc to list/trigger the tracepoints), so it seems like it would be a good candidate for adding support to libbpf.
However, before doing that, it's worth considering if there are simpler ways to support runtime probe firing. This small series demonstrates a simple method based on USDT probes added to libbpf itself.
The suggested solution comprises 3 parts
1. functions to fire dynamic probes are added to libbpf itself
bpf_urdt__probeN(), where N is the number of probe arguemnts. A sample usage would bebpf_urdt__probe3("myprovider", "myprobe", 1, 2, 3);Under the hood these correspond to USDT probes with an additional argument for uniquely identifying the probe (a hash of provider/probe name).
2. we attach to the appropriate USDT probe for the specified number of arguments urdt/probe0 for none, urdt/probe1 for 1, etc. We utilize the high-order 32 bits of the attach cookie to store the hash of the provider/probe name.
Quoted 5 lines
3. when urdt/probeN fires, the
BPF_URDT()macro (which is similar toBPF_USDT()) checks if the hash passed in (identifying provider/probe) matches the attach cookie high-order 32 bits; if not it must be a firing for a different dynamic probe and we exit early.
Quoted 7 lines
I'm sorry Alan, but I don't see this being added to libbpf. This is nothing else than USDT with a bunch of extra conventions bolted on. And those conventions might not work for many environments. It is completely arbitrary that libbpf is a) assumed to be a dynamic library and b) provides USDT hooks that will be triggered. Just because it will be libbpf that will be used to trace those USDT hooks doesn't mean that libbpf has to define those hooks.
Right - that came up with the discussion with Daniel also. Adding probes in libbpf was just a means of providing a method of last resort for runtime probe firing, it's not strictly necessary.
Quoted 8 lines
Just because libbpf can
trace USDTs it doesn't mean that libbpf should provide thoseSTAP_PROBEx()macros to define and trigger USDTs within some application. Applications that define USDTs and applications that attach to those USDTs are completely separate and independent. Same here, there might be an overlap in some cases, but conceptually it's two separate sides of the solution.
I think the point is though that USDT got its start by establishing a shared set of conventions between the to-be-traced side and the tracer, and built upon existing uprobe support to make that work. Is there a similar approach that we could apply for dynamic probes? libbpf isn't necessarily the right vehicle for establishing those conventions and I'm far from wedded to the specifics of this approach, but I do think it's a question we should explore a bit.
Quoted 7 lines
Overall, this is definitely a useful overall approach, to have a single system-wide .so library that can be attached to trace some USDTs, and we've explored this approach internally at Meta as well. But I don't believe it should be part of libbpf. From libbpf's standpoint it's just a standard USDT probe to attach to.
For now we can pursue the approach of adding a static probe - triggered when a dynamic probe firing is requested - to libstapsdt, and this would give us libbpf support via USDT tracing of libstapsdt.
Quoted 56 lines
Quoted 55 lines
Auto-attach support is also added, for example the following would add a dynamic probe for provider:myprobe:
SEC("udrt/libbpf.so:2:myprovider:myprobe")
intBPF_URDT(myprobe, int arg1, char *arg2)
{
...
}(Note the "2" above specifies the number of arguments to the probe, otherwise it is identical to USDT).
The above program can then be triggered by a call to
BPF_URDT_PROBE2("myprovider", "myprobe", 1, "hi");The useful thing about this is that by attaching to libbpf.so (and firing probes using that library) we can get system-wide dynamic probe firing. It is also easy to fire a dynamic probe - no setup is required.
More examples of auto and manual attach can be found in the selftests (patch 2).
If this approach appears to be worth pursing, we could also look at adding support to libstapsdt for it.
Alan Maguire (2):
libbpf: add support for Userspace Runtime Dynamic Tracing (URDT) selftests/bpf: add tests for Userspace Runtime Defined Tracepoints(URDT)tools/lib/bpf/Build | 2 +- tools/lib/bpf/Makefile | 2 +- tools/lib/bpf/libbpf.c | 94 ++++++++++ tools/lib/bpf/libbpf.h | 94 ++++++++++ tools/lib/bpf/libbpf.map | 13 ++ tools/lib/bpf/libbpf_internal.h | 2 + tools/lib/bpf/urdt.bpf.h | 103 +++++++++++ tools/lib/bpf/urdt.c | 145 +++++++++++++++ tools/testing/selftests/bpf/Makefile | 2 +- tools/testing/selftests/bpf/prog_tests/urdt.c | 173 ++++++++++++++++++ tools/testing/selftests/bpf/progs/test_urdt.c | 100 ++++++++++ .../selftests/bpf/progs/test_urdt_shared.c | 59 ++++++ 12 files changed, 786 insertions(+), 3 deletions(-) create mode 100644 tools/lib/bpf/urdt.bpf.h create mode 100644 tools/lib/bpf/urdt.c create mode 100644 tools/testing/selftests/bpf/prog_tests/urdt.c create mode 100644 tools/testing/selftests/bpf/progs/test_urdt.c create mode 100644 tools/testing/selftests/bpf/progs/test_urdt_shared.c--
2.39.3
Quoted 54 lines · Alan Maguire
Quoted 44 lines · Andrii Nakryiko
Quoted 37 lines · Alan Maguire
Adding userspace tracepoints in other languages like python and go is a very useful for observability. libstapsdt [1] and language bindings like python-stapsdt [2] that rely on it use a clever scheme of emulating static (USDT) userspace tracepoints at runtime. This involves (as I understand it):
- fabricating a shared library
- annotating it with ELF notes that describe its tracepoints
- dlopen()ing it and calling the appropriate probe fire function to trigger probe firing.bcc already supports this mechanism (the examples in [2] use bcc to list/trigger the tracepoints), so it seems like it would be a good candidate for adding support to libbpf.
However, before doing that, it's worth considering if there are simpler ways to support runtime probe firing. This small series demonstrates a simple method based on USDT probes added to libbpf itself.
The suggested solution comprises 3 parts
1. functions to fire dynamic probes are added to libbpf itself
bpf_urdt__probeN(), where N is the number of probe arguemnts. A sample usage would bebpf_urdt__probe3("myprovider", "myprobe", 1, 2, 3);Under the hood these correspond to USDT probes with an additional argument for uniquely identifying the probe (a hash of provider/probe name).
2. we attach to the appropriate USDT probe for the specified number of arguments urdt/probe0 for none, urdt/probe1 for 1, etc. We utilize the high-order 32 bits of the attach cookie to store the hash of the provider/probe name.
Quoted 5 lines
3. when urdt/probeN fires, the
BPF_URDT()macro (which is similar toBPF_USDT()) checks if the hash passed in (identifying provider/probe) matches the attach cookie high-order 32 bits; if not it must be a firing for a different dynamic probe and we exit early.Quoted 7 lines
I'm sorry Alan, but I don't see this being added to libbpf. This is nothing else than USDT with a bunch of extra conventions bolted on. And those conventions might not work for many environments. It is completely arbitrary that libbpf is a) assumed to be a dynamic library and b) provides USDT hooks that will be triggered. Just because it will be libbpf that will be used to trace those USDT hooks doesn't mean that libbpf has to define those hooks.
Quoted 4 lines
Right - that came up with the discussion with Daniel also. Adding probes in libbpf was just a means of providing a method of last resort for runtime probe firing, it's not strictly necessary.
Ok, glad we agree on not putting this into libbpf.
Quoted 9 lines
Quoted 8 lines
Just because libbpf can
trace USDTs it doesn't mean that libbpf should provide thoseSTAP_PROBEx()macros to define and trigger USDTs within some application. Applications that define USDTs and applications that attach to those USDTs are completely separate and independent. Same here, there might be an overlap in some cases, but conceptually it's two separate sides of the solution.
Quoted 7 lines
I think the point is though that USDT got its start by establishing a shared set of conventions between the to-be-traced side and the tracer, and built upon existing uprobe support to make that work. Is there a similar approach that we could apply for dynamic probes? libbpf isn't necessarily the right vehicle for establishing those conventions and I'm far from wedded to the specifics of this approach, but I do think it's a question we should explore a bit.
I'm a bit skeptical about "standardizing" this approach so early. Let's see how and whether this gets used in practice widely enough, before adding SEC("urdt") as a standard feature into libbpf.
Quoted 9 lines
Quoted 8 lines
Overall, this is definitely a useful overall approach, to have a single system-wide .so library that can be attached to trace some USDTs, and we've explored this approach internally at Meta as well. But I don't believe it should be part of libbpf. From libbpf's standpoint it's just a standard USDT probe to attach to.
Quoted 3 lines
For now we can pursue the approach of adding a static probe - triggered when a dynamic probe firing is requested - to libstapsdt, and this would give us libbpf support via USDT tracing of libstapsdt.
Exactly. And I suspect big companies like Meta, Google and whatnot might have their own small system-wide shared library, developed and maintained internally, deployed using whatever deployment strategy they see fit, with their own USDT name and convention about naming and namespacing these dynamic USDTs, etc, etc.
As I said, the idea is sound and neat, I'm just not sure that assuming libstapsdt (or whatever other predefined library) is going to work in practice.
If anything, maybe making the kernel provide something like this as a standard functionality is the right way to go. Something along the VDSO lines, which doesn't trigger a context switch, but can be centralized through the kernel? Have you considered such options?
Quoted 57 lines
Quoted 57 lines
Quoted 56 lines
Auto-attach support is also added, for example the following would add a dynamic probe for provider:myprobe:
SEC("udrt/libbpf.so:2:myprovider:myprobe")
intBPF_URDT(myprobe, int arg1, char *arg2)
{
...
}(Note the "2" above specifies the number of arguments to the probe, otherwise it is identical to USDT).
The above program can then be triggered by a call to
BPF_URDT_PROBE2("myprovider", "myprobe", 1, "hi");The useful thing about this is that by attaching to libbpf.so (and firing probes using that library) we can get system-wide dynamic probe firing. It is also easy to fire a dynamic probe - no setup is required.
More examples of auto and manual attach can be found in the selftests (patch 2).
If this approach appears to be worth pursing, we could also look at adding support to libstapsdt for it.
Alan Maguire (2):
libbpf: add support for Userspace Runtime Dynamic Tracing (URDT) selftests/bpf: add tests for Userspace Runtime Defined Tracepoints(URDT)tools/lib/bpf/Build | 2 +- tools/lib/bpf/Makefile | 2 +- tools/lib/bpf/libbpf.c | 94 ++++++++++ tools/lib/bpf/libbpf.h | 94 ++++++++++ tools/lib/bpf/libbpf.map | 13 ++ tools/lib/bpf/libbpf_internal.h | 2 + tools/lib/bpf/urdt.bpf.h | 103 +++++++++++ tools/lib/bpf/urdt.c | 145 +++++++++++++++ tools/testing/selftests/bpf/Makefile | 2 +- tools/testing/selftests/bpf/prog_tests/urdt.c | 173 ++++++++++++++++++ tools/testing/selftests/bpf/progs/test_urdt.c | 100 ++++++++++ .../selftests/bpf/progs/test_urdt_shared.c | 59 ++++++ 12 files changed, 786 insertions(+), 3 deletions(-) create mode 100644 tools/lib/bpf/urdt.bpf.h create mode 100644 tools/lib/bpf/urdt.c create mode 100644 tools/testing/selftests/bpf/prog_tests/urdt.c create mode 100644 tools/testing/selftests/bpf/progs/test_urdt.c create mode 100644 tools/testing/selftests/bpf/progs/test_urdt_shared.c--
2.39.3