Usage of kfuncs in tracepoints
Hi
I'm trying to use bpf_strstr in a program that's running on kernel 6.18.2, but I'm getting the following error on load:
Quoted 1 line
failed to find BTF for extern '
bpf_strstr' [53] section: -2
A minimal reproducer for this is:
```
extern int bpf_strstr(const char *s1__ign, const char *s2__ign);
SEC("tracepoint/syscalls/sys_enter_sendto")
int trace_sendto_entry(struct trace_event_raw_sys_enter *ctx)
{
char buf[128];
int pos = bpf_strstr(buf, "A");
}
```My kernel was initially built with CONFIG_DEBUG_INFO_BTF=n, rebuilding with CONFIG_DEBUG_INFO_BTF=y did not change the error.
I've only tried this with a stripped kernel image, which is 4MiB larger than the image with no BTF info.
Running bpftool btf dump on the stripped iamge does show the kfunc:
``` $ bpftool btf dump file ~/git/linux-6.18.2/vmlinux | grep strstr [26877] FUNC 'bpf_strstr' type_id=26855 linkage=static [60337] FUNC 'strstr' type_id=60336 linkage=static ```
I'm running this program in a virtual machine with a custom init; running through strace shows a failed load of /proc/version_signature, but I assume a fallback to uname.
```
faccessat2(AT_FDCWD, "/proc/version_signature", R_OK, AT_EACCESS) = -1 ENOENT (No such file or directory)
uname({sysname="Linux", nodename="(none)", ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3fe6630000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3fe662f000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3fe662e000
munmap(0x7f3fe662e000, 4096) = 0
munmap(0x7f3fe6630000, 8192 <unfinished ...>
nanosleep({tv_sec=0, tv_nsec=10000000} <unfinished ...>
futex(0x7f3fe6a44858, FUTEX_WAKE_PRIVATE, 1) = 1
munmap(0x7f3fe6839000, 16384 <unfinished ...>
sendto(3, "\1", 1, MSG_NOSIGNAL, NULL, 0 <unfinished ...>
futex(0x7f3fe6a43b70, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
sendto(3, "@", 1, MSG_NOSIGNAL, NULL, 0) = 1
sendto(3, "libbpf: failed to find BTF for e"..., 64, MSG_NOSIGNAL, NULL, 0) = 64
close(3)
```Maybe there's another dependency I'm not aware of for kfuncs?
I'm not sure what I'm doing wrong, can you point me in the right direction?
David
Quoted 11 lines · David
Hi
I'm trying to use
bpf_strstrin a program that's running on kernel 6.18.2, but I'm getting the following error on load:Quoted 1 line
failed to find BTF for extern '
bpf_strstr' [53] section: -2A minimal reproducer for this is:
```
extern intbpf_strstr(const char *s1__ign, const char *s2__ign);
I think you need to add "__ksym __weak";" here i.e.
extern int bpf_strstr(const char *s1__ign, const char *s2__ign) __ksym __weak;
If these aren't already defined you'll need:
#ifndef __ksym
#define __ksym __attribute__((section(".ksyms")))
#endif
#ifndef __weak
#define __weak __attribute__((weak))
#endif
Most of the examples use a bpftool-generated vmlinux.h which has kfunc declarations of that form; if you want to use a generated vmlinux.h (created from the running kernel's BTF) yourself you'd need to
bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h
Anyway, hopefully the ksym/weak attributes should be enough to get things working.
Alan
Quoted 52 lines
SEC("tracepoint/syscalls/sys_enter_sendto")
inttrace_sendto_entry(struct trace_event_raw_sys_enter *ctx){
char buf[128];
int pos =bpf_strstr(buf, "A");
}``` My kernel was initially built with CONFIG_DEBUG_INFO_BTF=n, rebuilding with CONFIG_DEBUG_INFO_BTF=y did not change the error. I've only tried this with a stripped kernel image, which is 4MiB larger than the image with no BTF info. Running `bpftool btf dump` on the stripped iamge does show the kfunc: ``` $ bpftool btf dump file ~/git/linux-6.18.2/vmlinux | grep strstr [26877] FUNC 'bpf_strstr' type_id=26855 linkage=static [60337] FUNC 'strstr' type_id=60336 linkage=static ```I'm running this program in a virtual machine with a custom init; running through strace shows a failed load of
/proc/version_signature, but I assume a fallback touname.``` faccessat2(AT_FDCWD, "/proc/version_signature", R_OK, AT_EACCESS) = -1 ENOENT (No such file or directory) uname({sysname="Linux", nodename="(none)", ...}) = 0 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3fe6630000 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3fe662f000 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3fe662e000 munmap(0x7f3fe662e000, 4096) = 0 munmap(0x7f3fe6630000, 8192 <unfinished ...> nanosleep({tv_sec=0, tv_nsec=10000000} <unfinished ...> futex(0x7f3fe6a44858, FUTEX_WAKE_PRIVATE, 1) = 1 munmap(0x7f3fe6839000, 16384 <unfinished ...> sendto(3, "\1", 1, MSG_NOSIGNAL, NULL, 0 <unfinished ...> futex(0x7f3fe6a43b70, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...> sendto(3, "@", 1, MSG_NOSIGNAL, NULL, 0) = 1 sendto(3, "libbpf: failed to find BTF for e"..., 64, MSG_NOSIGNAL, NULL, 0) = 64 close(3) ```Maybe there's another dependency I'm not aware of for kfuncs?
I'm not sure what I'm doing wrong, can you point me in the right direction?
David
Quoted 1 line · Alan Maguire
Most of the examples use a bpftool-generated
vmlinux.h
I am using a generated header, but not seeing bpf_strstr in the output:
``` $ bpftool btf dump file ~/git/linux-6.18.2/vmlinux.unstripped | grep strstr [42254] FUNC 'strstr' type_id=42253 linkage=static [51220] FUNC 'bpf_strstr' type_id=51198 linkage=static $ bpftool btf dump file ~/git/linux-6.18.2/vmlinux.unstripped format c | grep -c strstr 0 ```
I am generating my header with an older bpftool:
``` $ bpftool --version bpftool v7.4.0 using libbpf v1.4 features: ```
But I'm using libbpf v1.6.2 on my custom loader.
Quoted 1 line
I think you need to add "
__ksym__weak";" here i.e.
This change let me load the program, however, libbpf cannot find a kernel image at /sys/kernel/btf/vmlinux, because /sys/kernel/btf is not populated on my system.
My kernel _is_ built with CONFIG_DEBUG_INFO_BTF=y, is there something else I need to do to get this path populated?
Because this path is missing, libbpf reports:
``` kernel BTF is missing at '/sys/kernel/btf/vmlinux', was CONFIG_DEBUG_INFO_BTF enabled? ```
But I see from strace that it tries a few fallback paths.
In the meantime, I copied my kernel into /boot/vmlinux-6.18.2 so libbpf can find it, but now the loader says
``` calling kernel function is not supported without CONFIG_DEBUG_INFO_BTF ```
with strace showing a return value of ENOTSUPP + error 524:
```
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_TRACEPOINT, insn_cnt=6, insns=0x7f6ab777c230, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(6, 18, 2), prog_flags=0, prog_name="trace_sendto
_en", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=8, func_info_rec_size=8, func_info=0x7f6ab777d280, func_info_cnt=1, line_info_rec_size=16, line_info=0x7f6ab777ed40, line_info_cnt=2, attach_btf_id=0, a
ttach_prog_fd=0, core_relo_cnt=0, fd_array=NULL, core_relos=NULL, core_relo_rec_size=0, log_true_size=0, prog_token_fd=0, fd_array_cnt=0}, 152) = -1 ENOTSUPP (Unknown error 524)
```Can I not use bpf_strstr on a tracepoint? To validate, I tried a raw_tp but had the same result.
David
Quoted 4 lines · David
Quoted 1 line · Alan Maguire
I think you need to add "
__ksym__weak";" here i.e.
Quoted 8 lines
This change let me load the program, however, libbpf cannot find a kernel image at
/sys/kernel/btf/vmlinux, because/sys/kernel/btfis not populated on my system.My kernel _is_ built with
CONFIG_DEBUG_INFO_BTF=y, is there something else I need to do
to get this path populated?
Did you try rebuilding from scratch (with make clean) after enabling CONFIG_DEBUG_INFO_BTF? If the sources were already built without debug info, they will not be automatically rebuilt just by adding the config option.
Quoted 8 lines
Because this path is missing, libbpf reports:
``` kernel BTF is missing at '/sys/kernel/btf/vmlinux', was CONFIG_DEBUG_INFO_BTF enabled? ```
Quoted 8 lines
But I see from strace that it tries a few fallback paths.
In the meantime, I copied my kernel into/boot/vmlinux-6.18.2so libbpf can find it, but
now the loader says``` calling kernel function is not supported without CONFIG_DEBUG_INFO_BTF ```
While libbpf may try other fallback paths to find BTF, using kfuncs requires the kernel to find that kfunc in BTF and kernel will only use the system BTF (the one from /sys/kernel/vmlinux/btf).
Quoted 3 lines
Can I not use
bpf_strstron a tracepoint? To validate, I tried araw_tpbut
had the same result.
There shouldn't be any issue using bpf_strstr from tracepoints (or any other program type).
Viktor
Quoted 9 lines · Viktor Malik
Quoted 8 lines · David
Quoted 1 line · Alan Maguire
I think you need to add "
__ksym__weak";" here i.e.This change let me load the program, however, libbpf cannot find a kernel image at
/sys/kernel/btf/vmlinux, because/sys/kernel/btfis not populated on my system.
Quoted 8 lines
Quoted 3 lines
My kernel _is_ built with
CONFIG_DEBUG_INFO_BTF=y, is there something else I need to do
to get this path populated?Did you try rebuilding from scratch (with
make clean) after enablingCONFIG_DEBUG_INFO_BTF? If the sources were already built without debug info, they will not be automatically rebuilt just by adding the config option.
I thought the change took place because make vmlinux took a while, but after a clean build, it does work.
Quoted 12 lines
Quoted 12 lines
Because this path is missing, libbpf reports:
``` kernel BTF is missing at '/sys/kernel/btf/vmlinux', was CONFIG_DEBUG_INFO_BTF enabled? ```But I see from strace that it tries a few fallback paths.
In the meantime, I copied my kernel into/boot/vmlinux-6.18.2so libbpf can find it, but
now the loader says
Quoted 7 lines
Quoted 3 lines
``` calling kernel function is not supported without CONFIG_DEBUG_INFO_BTF ```While libbpf may try other fallback paths to find BTF, using kfuncs requires the kernel to find that kfunc in BTF and kernel will only use the system BTF (the one from
/sys/kernel/vmlinux/btf).
This is good to know, thanks. I've removed the /boot/ file from my system.
Quoted 8 lines
Quoted 3 lines
Can I not use
bpf_strstron a tracepoint? To validate, I tried araw_tpbut
had the same result.There shouldn't be any issue using
bpf_strstrfrom tracepoints (or any other program type).Viktor
Do you happen to know how to generate the kfunc headers with bpftool? Even a new bpftool build from the newest commit, ad5d76e5c6b622e5ed05fecfa68029bae949d408, does not generate headers:
``` $ ./bpftool btf dump file ~/git/linux-6.18.2/vmlinux | grep strstr [28376] FUNC 'bpf_strstr' type_id=28354 linkage=static [60023] FUNC 'strstr' type_id=60022 linkage=static $ ./bpftool btf dump file ~/git/linux-6.18.2/vmlinux format c | grep -c strstr 0 ```
Could this be related to my host's kernel being older than 6.18?
For now, I'll generate my files from kernel/bpf/helpers.c, but it'd be great if I could use bpftool.
David
Quoted 62 lines · David
Quoted 9 lines · Viktor Malik
Quoted 8 lines · David
Quoted 1 line · Alan Maguire
I think you need to add "
__ksym__weak";" here i.e.This change let me load the program, however, libbpf cannot find a kernel image at
/sys/kernel/btf/vmlinux, because/sys/kernel/btfis not populated on my system.Quoted 8 lines
Quoted 3 lines
My kernel _is_ built with
CONFIG_DEBUG_INFO_BTF=y, is there something else I need to do
to get this path populated?Did you try rebuilding from scratch (with
make clean) after enablingCONFIG_DEBUG_INFO_BTF? If the sources were already built without debug info, they will not be automatically rebuilt just by adding the config option.I thought the change took place because
make vmlinuxtook a while, but after a clean build, it does work.Quoted 12 lines
Quoted 12 lines
Because this path is missing, libbpf reports:
``` kernel BTF is missing at '/sys/kernel/btf/vmlinux', was CONFIG_DEBUG_INFO_BTF enabled? ```But I see from strace that it tries a few fallback paths.
In the meantime, I copied my kernel into/boot/vmlinux-6.18.2so libbpf can find it, but
now the loader saysQuoted 7 lines
Quoted 3 lines
``` calling kernel function is not supported without CONFIG_DEBUG_INFO_BTF ```While libbpf may try other fallback paths to find BTF, using kfuncs requires the kernel to find that kfunc in BTF and kernel will only use the system BTF (the one from
/sys/kernel/vmlinux/btf).This is good to know, thanks. I've removed the
/boot/file from my system.Quoted 8 lines
Quoted 3 lines
Can I not use
bpf_strstron a tracepoint? To validate, I tried araw_tpbut
had the same result.There shouldn't be any issue using
bpf_strstrfrom tracepoints (or any other program type).Viktor
Do you happen to know how to generate the kfunc headers with bpftool? Even a new bpftool build from the newest commit,
ad5d76e5c6b622e5ed05fecfa68029bae949d408, does not generate headers:``` $ ./bpftool btf dump file ~/git/linux-6.18.2/vmlinux | grep strstr [28376] FUNC 'bpf_strstr' type_id=28354 linkage=static [60023] FUNC 'strstr' type_id=60022 linkage=static $ ./bpftool btf dump file ~/git/linux-6.18.2/vmlinux format c | grep -c strstr 0 ```
Quoted 4 lines
Could this be related to my host's kernel being older than 6.18?
For now, I'll generate my files from
kernel/bpf/helpers.c, but it'd be great if I could use bpftool.
hi,
do you use the latest pahole ?
jirka
Quoted 63 lines · Jiri Olsa
Quoted 62 lines · David
Quoted 9 lines · Viktor Malik
Quoted 8 lines · David
Quoted 1 line · Alan Maguire
I think you need to add "
__ksym__weak";" here i.e.This change let me load the program, however, libbpf cannot find a kernel image at
/sys/kernel/btf/vmlinux, because/sys/kernel/btfis not populated on my system.Quoted 8 lines
Quoted 3 lines
My kernel _is_ built with
CONFIG_DEBUG_INFO_BTF=y, is there something else I need to do
to get this path populated?Did you try rebuilding from scratch (with
make clean) after enablingCONFIG_DEBUG_INFO_BTF? If the sources were already built without debug info, they will not be automatically rebuilt just by adding the config option.I thought the change took place because
make vmlinuxtook a while, but after a clean build, it does work.Quoted 12 lines
Quoted 12 lines
Because this path is missing, libbpf reports:
``` kernel BTF is missing at '/sys/kernel/btf/vmlinux', was CONFIG_DEBUG_INFO_BTF enabled? ```But I see from strace that it tries a few fallback paths.
In the meantime, I copied my kernel into/boot/vmlinux-6.18.2so libbpf can find it, but
now the loader saysQuoted 7 lines
Quoted 3 lines
``` calling kernel function is not supported without CONFIG_DEBUG_INFO_BTF ```While libbpf may try other fallback paths to find BTF, using kfuncs requires the kernel to find that kfunc in BTF and kernel will only use the system BTF (the one from
/sys/kernel/vmlinux/btf).This is good to know, thanks. I've removed the
/boot/file from my system.Quoted 8 lines
Quoted 3 lines
Can I not use
bpf_strstron a tracepoint? To validate, I tried araw_tpbut
had the same result.There shouldn't be any issue using
bpf_strstrfrom tracepoints (or any other program type).Viktor
Do you happen to know how to generate the kfunc headers with bpftool? Even a new bpftool build from the newest commit,
ad5d76e5c6b622e5ed05fecfa68029bae949d408, does not generate headers:``` $ ./bpftool btf dump file ~/git/linux-6.18.2/vmlinux | grep strstr [28376] FUNC 'bpf_strstr' type_id=28354 linkage=static [60023] FUNC 'strstr' type_id=60022 linkage=static $ ./bpftool btf dump file ~/git/linux-6.18.2/vmlinux format c | grep -c strstr 0 ```
Quoted 6 lines
Quoted 4 lines
Could this be related to my host's kernel being older than 6.18?
For now, I'll generate my files from
kernel/bpf/helpers.c, but it'd be great if I could use bpftool.hi,
do you use the latest pahole ?
I was using v1.25, using master tip (d1dda58ffac121b10a87d2738f3b931847e29acb), which reports as v1.31, made it work.
I needed newer binaries for both bpftool and pahole for the headers to be fully generated.
Thanks for the help
Quoted 2 lines
jirka