libbpf: Make optimized uprobes backward compatible
Subsystems (4)
Jiri Olsa
reviewerv5 cover libbpf: Make optimized uprobes backward compatible
hi,
we can currently optimize uprobes on top of nop5 instructions, so application can define USDT_NOP to nop5 and use USDT macro to define optimized usdt probes.
This works fine on new kernels, but could have performance penalty on older kernels, that do not have the support to optimize and to emulate nop5 instruction.
This patchset adds support to workaround the performance penalty on older kernels that do not support uprobe optimization, please see detailed description in patch 2.
v1: https://lore.kernel.org/bpf/[email protected]/
v2: https://lore.kernel.org/bpf/[email protected]/
v3: https://lore.kernel.org/bpf/[email protected]/T/#t
v4: https://lore.kernel.org/bpf/[email protected]/
v5 changes:
- keep nop_combo on stack and levae buf uninitialized in has_nop_combo function [David]
v4 changes:
- rebased on latest bpf-next/master
- use pread for nop combo read [Andrii]
- renamed usdt triger benchmark names [Andrii]
- added more ip address checks to tests [Andrii]
v3 changes:
- fix __x86_64 define and other typos [CI]
- add missing '?' to usdt trigger program [CI]
v2 changes:
- after more investigation we realized there are some versions of bpftrace and stap that does not work with solution suggested in version 1, so we decided to switch to following solution:
- change USDT macro [1] emits nop,nop5 instructions combo by default
- libbpf detects nop,nop5 instructions combo for USDT probe, if there is and if uprobe syscall is detected libbpf installs usdt probe on top of nop5 instruction to get it optimized
- added usdt trigger benchmarks [Andrii]
- several small fixes on uprobe syscall detection, tests and other places [Andrii]
- true usdt.h source [1] updated [Andrii]
- compile usdt_* objects unconditionally [Andrii]
thanks,
jirka
[1] https://github.com/libbpf/usdt
---
Jiri Olsa (5):
libbpf: Add uprobe syscall feature detection
libbpf: Add support to detect nop,nop5 instructions combo for usdt probe selftests/bpf: Emit nop,nop5 instructions combo for x86_64 arch selftests/bpf: Add test for checking correct nop of optimized usdt selftests/bpf: Add usdt trigger bench
tools/lib/bpf/features.c | 24 ++++++++++++++++++++++++ tools/lib/bpf/libbpf_internal.h | 2 ++ tools/lib/bpf/usdt.c | 47 +++++++++++++++++++++++++++++++++++++++++++---- tools/testing/selftests/bpf/.gitignore | 2 ++ tools/testing/selftests/bpf/Makefile | 5 ++++- tools/testing/selftests/bpf/bench.c | 4 ++++ tools/testing/selftests/bpf/benchs/bench_trigger.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ tools/testing/selftests/bpf/benchs/run_bench_uprobes.sh | 2 +- tools/testing/selftests/bpf/prog_tests/usdt.c | 92 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ tools/testing/selftests/bpf/progs/test_usdt.c | 12 ++++++++++++ tools/testing/selftests/bpf/progs/trigger_bench.c | 10 +++++++++- tools/testing/selftests/bpf/usdt.h | 2 ++ tools/testing/selftests/bpf/usdt_1.c | 18 ++++++++++++++++++ tools/testing/selftests/bpf/usdt_2.c | 16 ++++++++++++++++ 14 files changed, 289 insertions(+), 7 deletions(-) create mode 100644 tools/testing/selftests/bpf/usdt_1.c create mode 100644 tools/testing/selftests/bpf/usdt_2.c
Jiri Olsa
reviewerPATCH v5 1/5 libbpf: Add uprobe syscall feature detection
Adding uprobe syscall feature detection that will be used in following changes.
--- tools/lib/bpf/features.c | 24 ++++++++++++++++++++++++ tools/lib/bpf/libbpf_internal.h | 2 ++ 2 files changed, 26 insertions(+)
tools/lib/bpf/features.c +24 -0
@@ -568,6 +568,27 @@ static int probe_ldimm64_full_range_off(int token_fd)return 1 ; } +#ifdef __x86_64__ + +#ifndef __NR_uprobe +#define __NR_uprobe 336 +#endif + +static int probe_uprobe_syscall (int token_fd) +{ +/* +* If kernel supports uprobe() syscall, it will return -ENXIO when called +* from the outside of a kernel-generated uprobe trampoline. +*/ +return syscall (__NR_uprobe) <0 && errno ==ENXIO ; +} +#else +static int probe_uprobe_syscall (int token_fd) +{ +return 0 ; +} +#endif +typedef int (*feature_probe_fn )(int /* token_fd */ );static struct kern_feature_cache feature_cache;
@@ -646,6 +667,9 @@ static struct kern_feature_desc { [FEAT_LDIMM64_FULL_RANGE_OFF ] = {"full range LDIMM64 support" , probe_ldimm64_full_range_off, }, + [FEAT_UPROBE_SYSCALL ] = { +"kernel supports uprobe syscall" , probe_uprobe_syscall, + }, };bool feat_supported (struct kern_feature_cache *cache,enum kern_feature_id feat_id)
tools/lib/bpf/libbpf_internal.h +2 -0
@@ -394,6 +394,8 @@ enum kern_feature_id {FEAT_BTF_QMARK_DATASEC ,/* Kernel supports LDIMM64 imm offsets past 512 MiB. */ FEAT_LDIMM64_FULL_RANGE_OFF , +/* Kernel supports uprobe syscall */ +FEAT_UPROBE_SYSCALL , __FEAT_CNT, };
Signature · 1 line
2.53.0
Jiri Olsa
reviewerPATCH v5 2/5 libbpf: Add support to detect nop,nop5 instructions combo for usdt probe
In reply to Jiri OlsaAdding support to detect nop,nop5 instructions combo for usdt probe by checking on probe's following nop5 instruction.
When the nop,nop5 combo is detected together with uprobe syscall, we can place the probe on top of nop5 and get it optimized.
--- tools/lib/bpf/usdt.c | 47 ++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 43 insertions(+), 4 deletions(-)
tools/lib/bpf/usdt.c +43 -4
@@ -262,6 +262,7 @@ struct usdt_manager {bool has_bpf_cookie;bool has_sema_refcnt;bool has_uprobe_multi; +bool has_uprobe_syscall; };struct usdt_manager *usdt_manager_new (struct bpf_object *obj)
@@ -301,6 +302,13 @@ struct usdt_manager *usdt_manager_new(struct bpf_object *obj)* usdt probes. */ man->has_uprobe_multi =kernel_supports (obj,FEAT_UPROBE_MULTI_LINK ); + +/* +* Detect kernel support for uprobe() syscall, it's presence means we can +* take advantage of faster nop5 uprobe handling. +* Added in: 56101b69c919 ("uprobes/x86: Add uprobe syscall to speed up uprobe") +*/ + man->has_uprobe_syscall =kernel_supports (obj,FEAT_UPROBE_SYSCALL );return man; }
@@ -585,13 +593,34 @@ static int parse_usdt_note(GElf_Nhdr *nhdr, const char *data, size_t name_off,static int parse_usdt_spec (struct usdt_spec *spec,const struct usdt_note *note,__u64 usdt_cookie); -static int collect_usdt_targets (struct usdt_manager *man,Elf *elf,const char *path,pid_t pid, -const char *usdt_provider,const char *usdt_name,__u64 usdt_cookie, -struct usdt_target **out_targets,size_t *out_target_cnt) +#if defined(__x86_64__) +static bool has_nop_combo (int fd,long off) +{ +unsigned char nop_combo[6 ] = { +0x90 ,0x0f ,0x1f ,0x44 ,0x00 ,0x00 /* nop,nop5 */ + }; +unsigned char buf[6 ]; + +if (pread (fd, buf,6 , off) !=6 ) +return false; +return memcmp (buf, nop_combo,6 ) ==0 ; +} +#else +static bool has_nop_combo (int fd,long off) +{ +return false; +} +#endif + +static int collect_usdt_targets (struct usdt_manager *man,struct elf_fd *elf_fd,const char *path, +pid_t pid,const char *usdt_provider,const char *usdt_name, +__u64 usdt_cookie,struct usdt_target **out_targets, +size_t *out_target_cnt) {size_t off, name_off, desc_off, seg_cnt =0 , vma_seg_cnt =0 , target_cnt =0 ;struct elf_seg *segs =NULL , *vma_segs =NULL ;struct usdt_target *targets =NULL , *target; +Elf *elf = elf_fd->elf;long base_addr =0 ;Elf_Scn *notes_scn, *base_scn;GElf_Shdr base_shdr, notes_shdr;
@@ -784,6 +813,16 @@ static int collect_usdt_targets(struct usdt_manager *man, Elf *elf, const char * target = &targets[target_cnt];memset (target,0 ,sizeof (*target)); +/* +* We have uprobe syscall and usdt with nop,nop5 instructions combo, +* so we can place the uprobe directly on nop5 (+1) and get this probe +* optimized. +*/ +if (man->has_uprobe_syscall &&has_nop_combo (elf_fd->fd, usdt_rel_ip)) { + usdt_abs_ip++; + usdt_rel_ip++; + } + target->abs_ip = usdt_abs_ip; target->rel_ip = usdt_rel_ip; target->sema_off = usdt_sema_off;
@@ -998,7 +1037,7 @@ struct bpf_link *usdt_manager_attach_usdt(struct usdt_manager *man, const struct/* discover USDT in given binary, optionally limiting * activations to a given PID, if pid > 0 */ - err =collect_usdt_targets (man, elf_fd.elf, path, pid, usdt_provider, usdt_name, + err =collect_usdt_targets (man, &elf_fd, path, pid, usdt_provider, usdt_name, usdt_cookie, &targets, &target_cnt);if (err <=0 ) { err = (err ==0 ) ? -ENOENT : err;
Signature · 1 line
2.53.0
Jiri Olsa
reviewerPATCH v5 3/5 selftests/bpf: Emit nop,nop5 instructions combo for x86_64 arch
In reply to Jiri OlsaSyncing latest usdt.h change [1].
Now that we have nop5 optimization support in kernel, let's emit nop,nop5 for usdt probe. We leave it up to the library to use desirable nop instruction.
[1] https://github.com/libbpf/usdt/commit/c9865d158984fb2b73e3cbbdcdfb4f583ad36a73
--- tools/testing/selftests/bpf/usdt.h | 2 ++ 1 file changed, 2 insertions(+)
tools/testing/selftests/bpf/usdt.h +2 -0
@@ -312,6 +312,8 @@ struct usdt_sema { volatile unsigned short active; };#ifndef USDT_NOP #if defined(__ia64__) || defined(__s390__) || defined(__s390x__)#define USDT_NOP nop 0 +#elif defined(__x86_64__) +#define USDT_NOP .byte 0x90, 0x0f, 0x1f, 0x44, 0x00, 0x0/* nop, nop5 */ #else #define USDT_NOP nop#endif
Signature · 1 line
2.53.0
Jiri Olsa
reviewerPATCH v5 4/5 selftests/bpf: Add test for checking correct nop of optimized usdt
In reply to Jiri OlsaAdding test that attaches bpf program on usdt probe in 2 scenarios;
- attach program on top of usdt_1, which is single nop instruction, so the probe stays on nop instruction and is not optimized.
- attach program on top of usdt_2 which is probe defined on top of nop,nop5 combo, so the probe is placed on top of nop5 and is optimized.
--- tools/testing/selftests/bpf/.gitignore | 2 + tools/testing/selftests/bpf/Makefile | 3 +- tools/testing/selftests/bpf/prog_tests/usdt.c | 92 +++++++++++++++++++ tools/testing/selftests/bpf/progs/test_usdt.c | 12 +++ tools/testing/selftests/bpf/usdt_1.c | 18 ++++ tools/testing/selftests/bpf/usdt_2.c | 16 ++++ 6 files changed, 142 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/usdt_1.c create mode 100644 tools/testing/selftests/bpf/usdt_2.c
tools/testing/selftests/bpf/.gitignore +2 -0
@@ -47,3 +47,5 @@ verification_cert.h *.BTF *.BTF_ids *.BTF.base +usdt_1 +usdt_2
tools/testing/selftests/bpf/Makefile +2 -1
@@ -749,7 +749,8 @@ TRUNNER_EXTRA_SOURCES := test_progs.c \ $(VERIFY_SIG_HDR) \ flow_dissector_load.h \ ip_check_defrag_frags.h \ - bpftool_helpers.c + bpftool_helpers.c \ + usdt_1.c usdt_2.c TRUNNER_LIB_SOURCES := find_bit.c TRUNNER_EXTRA_FILES := $(OUTPUT)/urandom_read \ $(OUTPUT)/liburandom_read.so \
tools/testing/selftests/bpf/prog_tests/usdt.c +92 -0
104 lines not on this page · raw message
tools/testing/selftests/bpf/progs/test_usdt.c +12 -0
@@ -138,4 +138,16 @@ int usdt_sib(struct pt_regs *ctx)return 0 ; } +#ifdef __TARGET_ARCH_x86 +int executed; +unsigned long expected_ip; + +SEC("usdt") +int usdt_executed (struct pt_regs *ctx) +{ +if (expected_ip == ctx->ip) + executed++; +return 0 ; +} +#endif char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/usdt_1.c new file +18 -0
@@ -0,0 +1,18 @@ +// SPDX-License-Identifier: GPL-2.0 + +#if defined(__x86_64__) + +/* +* Include usdt.h with defined USDT_NOP macro to use single +* nop instruction. +*/ +#define USDT_NOP .byte 0x90 +#include "usdt.h" + +__attribute__((aligned (16 ))) +void usdt_1 (void ) +{ +USDT (optimized_attach, usdt_1); +} + +#endif
tools/testing/selftests/bpf/usdt_2.c new file +16 -0
@@ -0,0 +1,16 @@ +// SPDX-License-Identifier: GPL-2.0 + +#if defined(__x86_64__) + +/* +* Include usdt.h with default nop,nop5 instructions combo. +*/ +#include "usdt.h" + +__attribute__((aligned (16 ))) +void usdt_2 (void ) +{ +USDT (optimized_attach, usdt_2); +} + +#endif
Signature · 1 line
2.53.0
Jiri Olsa
reviewerPATCH v5 5/5 selftests/bpf: Add usdt trigger bench
In reply to Jiri OlsaAdding usdt trigger bench for usdt:
trig-usdt-nop - usdt on top of nop1 instruction
trig-usdt-nop5 - usdt on top of nop1/nop5 combo
Adding it to benchs/run_bench_uprobes.sh script.
Example run on x86_64 kernel with uprobe syscall:
# ./benchs/run_bench_uprobes.sh usermode-count : 152.507 ± 0.098M/s syscall-count : 14.309 ± 0.093M/s uprobe-nop : 3.190 ± 0.012M/s uprobe-push : 3.057 ± 0.004M/s uprobe-ret : 1.095 ± 0.009M/s uprobe-nop5 : 7.305 ± 0.034M/s uretprobe-nop : 2.175 ± 0.005M/s uretprobe-push : 2.109 ± 0.003M/s uretprobe-ret : 0.945 ± 0.002M/s uretprobe-nop5 : 3.530 ± 0.006M/s usdt-nop : 3.235 ± 0.008M/s <-- added usdt-nop5 : 7.511 ± 0.045M/s <-- added
--- tools/testing/selftests/bpf/Makefile | 2 + tools/testing/selftests/bpf/bench.c | 4 ++ .../selftests/bpf/benchs/bench_trigger.c | 60 +++++++++++++++++++ .../selftests/bpf/benchs/run_bench_uprobes.sh | 2 +- .../selftests/bpf/progs/trigger_bench.c | 10 +++- 5 files changed, 76 insertions(+), 2 deletions(-)
tools/testing/selftests/bpf/Makefile +2 -0
@@ -874,6 +874,8 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \ $(OUTPUT)/bench_bpf_crypto.o \ $(OUTPUT)/bench_sockmap.o \ $(OUTPUT)/bench_lpm_trie_map.o \ + $(OUTPUT)/usdt_1.o \ + $(OUTPUT)/usdt_2.o \ # $(call msg,BINARY,,$@) $(Q)$(CC) $(CFLAGS) $(LDFLAGS) $(filter %.a %.o,$^) $(LDLIBS) -o $@
tools/testing/selftests/bpf/bench.c +4 -0
@@ -541,6 +541,8 @@ extern const struct bench bench_trig_uprobe_nop5;extern const struct bench bench_trig_uretprobe_nop5;extern const struct bench bench_trig_uprobe_multi_nop5;extern const struct bench bench_trig_uretprobe_multi_nop5; +extern const struct bench bench_trig_usdt_nop; +extern const struct bench bench_trig_usdt_nop5;#endif extern const struct bench bench_rb_libbpf;
@@ -617,6 +619,8 @@ static const struct bench *benchs[] = { &bench_trig_uretprobe_nop5, &bench_trig_uprobe_multi_nop5, &bench_trig_uretprobe_multi_nop5, + &bench_trig_usdt_nop, + &bench_trig_usdt_nop5, #endif/* ringbuf/perfbuf benchmarks */ &bench_rb_libbpf,
tools/testing/selftests/bpf/benchs/bench_trigger.c +60 -0
76 lines not on this page · raw message
tools/testing/selftests/bpf/benchs/run_bench_uprobes.sh +1 -1
@@ -2,7 +2,7 @@ set -eufo pipefail -for i in usermode-count syscall-count {uprobe,uretprobe}-{nop,push,ret,nop5} +for i in usermode-count syscall-count {uprobe,uretprobe}-{nop,push,ret,nop5} usdt-nop usdt-nop5 do summary=$(sudo ./bench -w2 -d5 -a trig-$i | tail -n1 | cut -d'(' -f1 | cut -d' ' -f3-) printf "%-15s: %s\n" $i "$summary"
tools/testing/selftests/bpf/progs/trigger_bench.c +9 -1
@@ -1,10 +1,11 @@// SPDX-License-Identifier: GPL-2.0 // Copyright (c) 2020 Facebook -#include <linux/bpf.h> +#include "vmlinux.h" #include <asm/unistd.h> #include <bpf/bpf_helpers.h> #include <bpf/bpf_tracing.h> #include "bpf_misc.h" +#include "bpf/usdt.bpf.h" char _license[] SEC("license") = "GPL";
@@ -180,3 +181,10 @@ int bench_trigger_rawtp(void *ctx)handle (ctx);return 0 ; } + +SEC("?usdt") +int bench_trigger_usdt (void *ctx) +{ +inc_counter (); +return 0 ; +}
Signature · 1 line
2.53.0
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Quoted 10 lines · On Tue, 24 Feb 2026 11:39:10 +0100 you wrote:
hi,
we can currently optimize uprobes on top of nop5 instructions, so application can defineUSDT_NOPto nop5 and use USDT macro to define optimized usdt probes.This works fine on new kernels, but could have performance penalty on older kernels, that do not have the support to optimize and to emulate nop5 instruction.
[...]
Here is the summary with links:
- [PATCHv5,bpf-next,1/5] libbpf: Add uprobe syscall feature detection
https://git.kernel.org/bpf/bpf-next/c/05738da0efa1 - [PATCHv5,bpf-next,2/5] libbpf: Add support to detect nop,nop5 instructions combo for usdt probe https://git.kernel.org/bpf/bpf-next/c/41a5c7df4466 - [PATCHv5,bpf-next,3/5] selftests/bpf: Emit nop,nop5 instructions combo for x86_64 arch https://git.kernel.org/bpf/bpf-next/c/0c178e9debc8 - [PATCHv5,bpf-next,4/5] selftests/bpf: Add test for checking correct nop of optimized usdt https://git.kernel.org/bpf/bpf-next/c/304841967c55 - [PATCHv5,bpf-next,5/5] selftests/bpf: Add usdt trigger bench https://git.kernel.org/bpf/bpf-next/c/0c4fc6bd6105
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html