Skip to contentSkip to conversation

Usage of kfuncs in tracepoints

DISCUSSION David 4 participants 7 messages Jan 12 – Jan 13 BPF

lore
Share & tools
lore

David

author
#0 lore raw

Hi

I'm trying to use bpf_strstr in a program that's running on kernel 6.18.2, but I'm getting the following error on load:

Quoted 1 line

failed to find BTF for extern 'bpf_strstr' [53] section: -2

A minimal reproducer for this is:

```
extern int bpf_strstr(const char *s1__ign, const char *s2__ign);
SEC("tracepoint/syscalls/sys_enter_sendto")
int trace_sendto_entry(struct trace_event_raw_sys_enter *ctx)
{
      char buf[128];
      int pos = bpf_strstr(buf, "A");
}
```

My kernel was initially built with CONFIG_DEBUG_INFO_BTF=n, rebuilding with CONFIG_DEBUG_INFO_BTF=y did not change the error.

I've only tried this with a stripped kernel image, which is 4MiB larger than the image with no BTF info.

Running bpftool btf dump on the stripped iamge does show the kfunc:

```
$ bpftool btf dump file ~/git/linux-6.18.2/vmlinux | grep strstr
[26877] FUNC 'bpf_strstr' type_id=26855 linkage=static
[60337] FUNC 'strstr' type_id=60336 linkage=static
```

I'm running this program in a virtual machine with a custom init; running through strace shows a failed load of /proc/version_signature, but I assume a fallback to uname.

```
faccessat2(AT_FDCWD, "/proc/version_signature", R_OK, AT_EACCESS) = -1 ENOENT (No such file or directory)
uname({sysname="Linux", nodename="(none)", ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3fe6630000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3fe662f000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3fe662e000
munmap(0x7f3fe662e000, 4096) = 0
munmap(0x7f3fe6630000, 8192 <unfinished ...>
nanosleep({tv_sec=0, tv_nsec=10000000} <unfinished ...>
futex(0x7f3fe6a44858, FUTEX_WAKE_PRIVATE, 1) = 1
munmap(0x7f3fe6839000, 16384 <unfinished ...>
sendto(3, "\1", 1, MSG_NOSIGNAL, NULL, 0 <unfinished ...>
futex(0x7f3fe6a43b70, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
sendto(3, "@", 1, MSG_NOSIGNAL, NULL, 0) = 1
sendto(3, "libbpf: failed to find BTF for e"..., 64, MSG_NOSIGNAL, NULL, 0) = 64
close(3)
```

Maybe there's another dependency I'm not aware of for kfuncs?

I'm not sure what I'm doing wrong, can you point me in the right direction?

David

#1 lore raw
Quoted 11 lines · David

Hi

I'm trying to use bpf_strstr in a program that's running on kernel 6.18.2, but I'm getting the following error on load:

Quoted 1 line

failed to find BTF for extern 'bpf_strstr' [53] section: -2

A minimal reproducer for this is:

```
extern int bpf_strstr(const char *s1__ign, const char *s2__ign);

I think you need to add "__ksym __weak";" here i.e.

extern int bpf_strstr(const char *s1__ign, const char *s2__ign) __ksym __weak;

If these aren't already defined you'll need:

#ifndef __ksym
#define __ksym __attribute__((section(".ksyms")))
#endif

#ifndef __weak
#define __weak __attribute__((weak))
#endif

Most of the examples use a bpftool-generated vmlinux.h which has kfunc declarations of that form; if you want to use a generated vmlinux.h (created from the running kernel's BTF) yourself you'd need to

bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

Anyway, hopefully the ksym/weak attributes should be enough to get things working.

Alan

Quoted 52 lines

SEC("tracepoint/syscalls/sys_enter_sendto")
int trace_sendto_entry(struct trace_event_raw_sys_enter *ctx) {
      char buf[128];
      int pos = bpf_strstr(buf, "A");
}

```

My kernel was initially built with CONFIG_DEBUG_INFO_BTF=n, rebuilding with
CONFIG_DEBUG_INFO_BTF=y did not change the error.

I've only tried this with a stripped kernel image, which is 4MiB larger than
the image with no BTF info.

Running `bpftool btf dump` on the stripped iamge does show the kfunc:

```
$ bpftool btf dump file ~/git/linux-6.18.2/vmlinux | grep strstr
[26877] FUNC 'bpf_strstr' type_id=26855 linkage=static
[60337] FUNC 'strstr' type_id=60336 linkage=static
```

I'm running this program in a virtual machine with a custom init; running through strace shows a failed load of /proc/version_signature, but I assume a fallback to uname.

```
faccessat2(AT_FDCWD, "/proc/version_signature", R_OK, AT_EACCESS) = -1 ENOENT (No such file or directory)
uname({sysname="Linux", nodename="(none)", ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3fe6630000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3fe662f000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3fe662e000
munmap(0x7f3fe662e000, 4096) = 0
munmap(0x7f3fe6630000, 8192 <unfinished ...>
nanosleep({tv_sec=0, tv_nsec=10000000} <unfinished ...>
futex(0x7f3fe6a44858, FUTEX_WAKE_PRIVATE, 1) = 1
munmap(0x7f3fe6839000, 16384 <unfinished ...>
sendto(3, "\1", 1, MSG_NOSIGNAL, NULL, 0 <unfinished ...>
futex(0x7f3fe6a43b70, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
sendto(3, "@", 1, MSG_NOSIGNAL, NULL, 0) = 1
sendto(3, "libbpf: failed to find BTF for e"..., 64, MSG_NOSIGNAL, NULL, 0) = 64
close(3)
```

Maybe there's another dependency I'm not aware of for kfuncs?

I'm not sure what I'm doing wrong, can you point me in the right direction?

David

author
#2 lore raw
Quoted 1 line · Alan Maguire

Most of the examples use a bpftool-generated vmlinux.h

I am using a generated header, but not seeing bpf_strstr in the output:

```

$ bpftool btf dump file ~/git/linux-6.18.2/vmlinux.unstripped | grep strstr
[42254] FUNC 'strstr' type_id=42253 linkage=static
[51220] FUNC 'bpf_strstr' type_id=51198 linkage=static
$ bpftool btf dump file ~/git/linux-6.18.2/vmlinux.unstripped format c  | grep -c strstr
0
```

I am generating my header with an older bpftool:

```
$ bpftool --version
bpftool v7.4.0
using libbpf v1.4
features:
```

But I'm using libbpf v1.6.2 on my custom loader.

Quoted 1 line

I think you need to add "__ksym __weak";" here i.e.

This change let me load the program, however, libbpf cannot find a kernel image at /sys/kernel/btf/vmlinux, because /sys/kernel/btf is not populated on my system.

My kernel _is_ built with CONFIG_DEBUG_INFO_BTF=y, is there something else I need to do to get this path populated?

Because this path is missing, libbpf reports:

```
kernel BTF is missing at '/sys/kernel/btf/vmlinux', was CONFIG_DEBUG_INFO_BTF enabled?
```

But I see from strace that it tries a few fallback paths.
In the meantime, I copied my kernel into /boot/vmlinux-6.18.2 so libbpf can find it, but now the loader says

```
calling kernel function is not supported without CONFIG_DEBUG_INFO_BTF
```

with strace showing a return value of ENOTSUPP + error 524:

```
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_TRACEPOINT, insn_cnt=6, insns=0x7f6ab777c230, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(6, 18, 2), prog_flags=0, prog_name="trace_sendto
_en", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=8, func_info_rec_size=8, func_info=0x7f6ab777d280, func_info_cnt=1, line_info_rec_size=16, line_info=0x7f6ab777ed40, line_info_cnt=2, attach_btf_id=0, a
ttach_prog_fd=0, core_relo_cnt=0, fd_array=NULL, core_relos=NULL, core_relo_rec_size=0, log_true_size=0, prog_token_fd=0, fd_array_cnt=0}, 152) = -1 ENOTSUPP (Unknown error 524)
```

Can I not use bpf_strstr on a tracepoint? To validate, I tried a raw_tp but had the same result.

David

#3 lore raw
Quoted 4 lines · David
Quoted 1 line · Alan Maguire

I think you need to add "__ksym __weak";" here i.e.

Quoted 8 lines

This change let me load the program, however, libbpf cannot find a kernel image at
/sys/kernel/btf/vmlinux, because /sys/kernel/btf is not populated on my system.

My kernel _is_ built with CONFIG_DEBUG_INFO_BTF=y, is there something else I need to do
to get this path populated?

Did you try rebuilding from scratch (with make clean) after enabling CONFIG_DEBUG_INFO_BTF? If the sources were already built without debug info, they will not be automatically rebuilt just by adding the config option.

Quoted 8 lines

Because this path is missing, libbpf reports:

```
kernel BTF is missing at '/sys/kernel/btf/vmlinux', was 
CONFIG_DEBUG_INFO_BTF enabled?
```
Quoted 8 lines

But I see from strace that it tries a few fallback paths.
In the meantime, I copied my kernel into /boot/vmlinux-6.18.2 so libbpf can find it, but
now the loader says

```
calling kernel function is not supported without CONFIG_DEBUG_INFO_BTF
```

While libbpf may try other fallback paths to find BTF, using kfuncs requires the kernel to find that kfunc in BTF and kernel will only use the system BTF (the one from /sys/kernel/vmlinux/btf).

Quoted 3 lines

Can I not use bpf_strstr on a tracepoint? To validate, I tried a raw_tp but
had the same result.

There shouldn't be any issue using bpf_strstr from tracepoints (or any other program type).

Viktor

author
#4 lore raw
Quoted 9 lines · Viktor Malik
Quoted 8 lines · David
Quoted 1 line · Alan Maguire

I think you need to add "__ksym __weak";" here i.e.

This change let me load the program, however, libbpf cannot find a kernel image at
/sys/kernel/btf/vmlinux, because /sys/kernel/btf is not populated on my system.

Quoted 8 lines
Quoted 3 lines

My kernel _is_ built with CONFIG_DEBUG_INFO_BTF=y, is there something else I need to do
to get this path populated?

Did you try rebuilding from scratch (with make clean) after enabling CONFIG_DEBUG_INFO_BTF? If the sources were already built without debug info, they will not be automatically rebuilt just by adding the config option.

I thought the change took place because make vmlinux took a while, but after a clean build, it does work.

Quoted 12 lines
Quoted 12 lines

Because this path is missing, libbpf reports:

```
kernel BTF is missing at '/sys/kernel/btf/vmlinux', was
CONFIG_DEBUG_INFO_BTF enabled?
```

But I see from strace that it tries a few fallback paths.
In the meantime, I copied my kernel into /boot/vmlinux-6.18.2 so libbpf can find it, but
now the loader says

Quoted 7 lines
Quoted 3 lines
```
calling kernel function is not supported without CONFIG_DEBUG_INFO_BTF
```

While libbpf may try other fallback paths to find BTF, using kfuncs requires the kernel to find that kfunc in BTF and kernel will only use the system BTF (the one from /sys/kernel/vmlinux/btf).

This is good to know, thanks. I've removed the /boot/ file from my system.

Quoted 8 lines
Quoted 3 lines

Can I not use bpf_strstr on a tracepoint? To validate, I tried a raw_tp but
had the same result.

There shouldn't be any issue using bpf_strstr from tracepoints (or any other program type).

Viktor

Do you happen to know how to generate the kfunc headers with bpftool? Even a new bpftool build from the newest commit, ad5d76e5c6b622e5ed05fecfa68029bae949d408, does not generate headers:

```
$ ./bpftool btf dump file ~/git/linux-6.18.2/vmlinux | grep strstr
[28376] FUNC 'bpf_strstr' type_id=28354 linkage=static
[60023] FUNC 'strstr' type_id=60022 linkage=static
$ ./bpftool btf dump file ~/git/linux-6.18.2/vmlinux format c  | grep -c strstr
0
```

Could this be related to my host's kernel being older than 6.18?

For now, I'll generate my files from kernel/bpf/helpers.c, but it'd be great if I could use bpftool.

David

#5 lore raw
Quoted 62 lines · David
Quoted 9 lines · Viktor Malik
Quoted 8 lines · David
Quoted 1 line · Alan Maguire

I think you need to add "__ksym __weak";" here i.e.

This change let me load the program, however, libbpf cannot find a kernel image at
/sys/kernel/btf/vmlinux, because /sys/kernel/btf is not populated on my system.

Quoted 8 lines
Quoted 3 lines

My kernel _is_ built with CONFIG_DEBUG_INFO_BTF=y, is there something else I need to do
to get this path populated?

Did you try rebuilding from scratch (with make clean) after enabling CONFIG_DEBUG_INFO_BTF? If the sources were already built without debug info, they will not be automatically rebuilt just by adding the config option.

I thought the change took place because make vmlinux took a while, but after a clean build, it does work.

Quoted 12 lines
Quoted 12 lines

Because this path is missing, libbpf reports:

```
kernel BTF is missing at '/sys/kernel/btf/vmlinux', was
CONFIG_DEBUG_INFO_BTF enabled?
```

But I see from strace that it tries a few fallback paths.
In the meantime, I copied my kernel into /boot/vmlinux-6.18.2 so libbpf can find it, but
now the loader says

Quoted 7 lines
Quoted 3 lines
```
calling kernel function is not supported without CONFIG_DEBUG_INFO_BTF
```

While libbpf may try other fallback paths to find BTF, using kfuncs requires the kernel to find that kfunc in BTF and kernel will only use the system BTF (the one from /sys/kernel/vmlinux/btf).

This is good to know, thanks. I've removed the /boot/ file from my system.

Quoted 8 lines
Quoted 3 lines

Can I not use bpf_strstr on a tracepoint? To validate, I tried a raw_tp but
had the same result.

There shouldn't be any issue using bpf_strstr from tracepoints (or any other program type).

Viktor

Do you happen to know how to generate the kfunc headers with bpftool? Even a new bpftool build from the newest commit, ad5d76e5c6b622e5ed05fecfa68029bae949d408, does not generate headers:

```
$ ./bpftool btf dump file ~/git/linux-6.18.2/vmlinux | grep strstr
[28376] FUNC 'bpf_strstr' type_id=28354 linkage=static
[60023] FUNC 'strstr' type_id=60022 linkage=static
$ ./bpftool btf dump file ~/git/linux-6.18.2/vmlinux format c  | grep -c
strstr
0
```
Quoted 4 lines

Could this be related to my host's kernel being older than 6.18?

For now, I'll generate my files from kernel/bpf/helpers.c, but it'd be great if I could use bpftool.

hi,
do you use the latest pahole ?

jirka

author
#6 lore raw
Quoted 63 lines · Jiri Olsa
Quoted 62 lines · David
Quoted 9 lines · Viktor Malik
Quoted 8 lines · David
Quoted 1 line · Alan Maguire

I think you need to add "__ksym __weak";" here i.e.

This change let me load the program, however, libbpf cannot find a kernel image at
/sys/kernel/btf/vmlinux, because /sys/kernel/btf is not populated on my system.

Quoted 8 lines
Quoted 3 lines

My kernel _is_ built with CONFIG_DEBUG_INFO_BTF=y, is there something else I need to do
to get this path populated?

Did you try rebuilding from scratch (with make clean) after enabling CONFIG_DEBUG_INFO_BTF? If the sources were already built without debug info, they will not be automatically rebuilt just by adding the config option.

I thought the change took place because make vmlinux took a while, but after a clean build, it does work.

Quoted 12 lines
Quoted 12 lines

Because this path is missing, libbpf reports:

```
kernel BTF is missing at '/sys/kernel/btf/vmlinux', was
CONFIG_DEBUG_INFO_BTF enabled?
```

But I see from strace that it tries a few fallback paths.
In the meantime, I copied my kernel into /boot/vmlinux-6.18.2 so libbpf can find it, but
now the loader says

Quoted 7 lines
Quoted 3 lines
```
calling kernel function is not supported without CONFIG_DEBUG_INFO_BTF
```

While libbpf may try other fallback paths to find BTF, using kfuncs requires the kernel to find that kfunc in BTF and kernel will only use the system BTF (the one from /sys/kernel/vmlinux/btf).

This is good to know, thanks. I've removed the /boot/ file from my system.

Quoted 8 lines
Quoted 3 lines

Can I not use bpf_strstr on a tracepoint? To validate, I tried a raw_tp but
had the same result.

There shouldn't be any issue using bpf_strstr from tracepoints (or any other program type).

Viktor

Do you happen to know how to generate the kfunc headers with bpftool? Even a new bpftool build from the newest commit, ad5d76e5c6b622e5ed05fecfa68029bae949d408, does not generate headers:

```
$ ./bpftool btf dump file ~/git/linux-6.18.2/vmlinux | grep strstr
[28376] FUNC 'bpf_strstr' type_id=28354 linkage=static
[60023] FUNC 'strstr' type_id=60022 linkage=static
$ ./bpftool btf dump file ~/git/linux-6.18.2/vmlinux format c  | grep -c
strstr
0
```
Quoted 6 lines
Quoted 4 lines

Could this be related to my host's kernel being older than 6.18?

For now, I'll generate my files from kernel/bpf/helpers.c, but it'd be great if I could use bpftool.

hi,
do you use the latest pahole ?

I was using v1.25, using master tip (d1dda58ffac121b10a87d2738f3b931847e29acb), which reports as v1.31, made it work.

I needed newer binaries for both bpftool and pahole for the headers to be fully generated.

Thanks for the help

Quoted 2 lines

jirka