Skip to contentSkip to conversation

BTF loading failing on perf

DISCUSSION Arnaldo Carvalho de Melo 3 participants 7 messages Jun 6, 2025 – Jun 10, 2025

Subsystems (2)
lore
Share & tools
lore

Arnaldo Carvalho de Melo

author Tested-by Alan Maguire Acked-by Andrii Nakryiko
#0 lore raw
root@number:~# perf trace -e openat --max-events=1
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV
     0.000 ( 0.016 ms): ptyxis-agent/4375 openat(dfd: CWD, filename: "/proc/6593/cmdline", flags: RDONLY|CLOEXEC) = 13
root@number:~#

openat(AT_FDCWD, "/sys/kernel/btf/vmlinux", O_RDONLY) = 258 mmap(NULL, 6519699, PROT_READ, MAP_PRIVATE, 258, 0) = -1 ENODEV (No such device)
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV

root@number:~# ls -la /sys/kernel/btf/vmlinux 
-r--r--r--. 1 root root 6519699 Jun  6 12:19 /sys/kernel/btf/vmlinux
root@number:~# uname -a
Linux number 6.14.9-300.fc42.x86_64 #1 SMP PREEMPT_DYNAMIC Thu May 29 14:27:53 UTC 2025 x86_64 GNU/Linux
root@number:~# head /etc/os-release 
NAME="Fedora Linux"
VERSION="42 (Workstation Edition)"
RELEASE_TYPE=stable
ID=fedora
VERSION_ID=42
VERSION_CODENAME=""
PLATFORM_ID="platform:f42"
PRETTY_NAME="Fedora Linux 42 (Workstation Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
root@number:~# rpm -q glibc-devel
package glibc-devel is not installed
root@number:~# rpm -q glibc
glibc-2.41-5.fc42.x86_64
root@number:~# 

Reverting the patch below "cures" the problem.

⬢ [acme@toolbx perf-tools]$ git log --oneline -10 tools/lib/bpf/
370118ff875244d4 (HEAD -> perf-tools) Revert "libbpf: Use mmap to parse vmlinux BTF from sysfs" 90b83efa6701656e Merge tag 'bpf-next-6.16' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next
3c0421c93ce4ff0f libbpf: Use mmap to parse vmlinux BTF from sysfs
4e29128a9acec2a6 libbpf/btf: Fix string handling to support multi-split BTF d0445d7dd3fd9b15 libbpf: Check bpf_map_skeleton link for NULL
fd5fd538a1f4b34c libbpf: Use proper errno value in nlattr
62e23f183839c3d7 libbpf: Improve BTF dedup handling of "identical" BTF types 41d4ce6df3f49453 bpf: Fix uninitialized values in BPF_{CORE,PROBE}_READ 358b1c0f56ebb699 libbpf: Use proper errno value in linker
8e64c387c942229c libbpf: Add identical pointer detection to btf_dedup_is_equiv()
⬢ [acme@toolbx perf-tools]$

root@number:~# perf -v
perf version 6.15.g370118ff8752
root@number:~# perf trace -e connect --max-events=5
     0.000 ( 0.021 ms): DNS Res~ver #4/7932 connect(fd: 102, uservaddr: { .family: LOCAL, path: /run/systemd/r }, addrlen: 42) = 0
     0.304 ( 0.013 ms): systemd-resolv/1420 connect(fd: 25, uservaddr: { .family: INET, port: 53, addr: 186.208.78.194 }, addrlen: 16) = 0
     0.371 ( 0.002 ms): systemd-resolv/1420 connect(fd: 26, uservaddr: { .family: INET, port: 53, addr: 186.208.78.194 }, addrlen: 16) = 0
     0.591 ( 0.009 ms): DNS Res~ver #2/7523 connect(fd: 104, uservaddr: { .family: LOCAL, path: /run/systemd/r }, addrlen: 42) = 0
     0.731 ( 0.006 ms): systemd-resolv/1420 connect(fd: 28, uservaddr: { .family: INET, port: 53, addr: 186.208.78.194 }, addrlen: 16) = 0
root@number:~# 

Ideas?

- Arnaldo

⬢ [acme@toolbx perf-tools]$ git bisect good 3c0421c93ce4ff0f5f2612666122c34fc941d569 is the first bad commit commit 3c0421c93ce4ff0f5f2612666122c34fc941d569

Author: Lorenz Bauer <[email protected]>
Date:   Tue May 20 14:01:19 2025 +0100

libbpf: Use mmap to parse vmlinux BTF from sysfs

Teach libbpf to use mmap when parsing vmlinux BTF from /sys. We don't apply this to fall-back paths on the regular file system because there is no way to ensure that modifications underlying the MAP_PRIVATE mapping are not visible to the process.

 tools/lib/bpf/btf.c | 89 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------
 1 file changed, 71 insertions(+), 18 deletions(-)

⬢ [acme@toolbx perf-tools]$

Tested-by Alan Maguire Reviewed-by Shakeel Butt
#1 lore raw

Hi Arnaldo,

Quoted 9 lines · Arnaldo Carvalho de Melo
root@number:~# perf trace -e openat --max-events=1
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV
     0.000 ( 0.016 ms): ptyxis-agent/4375 openat(dfd: CWD, filename: "/proc/6593/cmdline", flags: RDONLY|CLOEXEC) = 13
root@number:~#

openat(AT_FDCWD, "/sys/kernel/btf/vmlinux", O_RDONLY) = 258 mmap(NULL, 6519699, PROT_READ, MAP_PRIVATE, 258, 0) = -1 ENODEV (No such device)
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV

Have you included the commit below in the kernel side?

commit a539e2a6d51d1c12d89eec149ccc72ec561639bc

Author: Lorenz Bauer <[email protected]>
Date:   Tue May 20 14:01:17 2025 +0100

btf: Allow mmap of vmlinux btf

User space needs access to kernel BTF for many modern features of BPF. Right now each process needs to read the BTF blob either in pieces or as a whole. Allow mmaping the sysfs file so that processes can directly access the memory allocated for it in the kernel.

remap_pfn_range is used instead of vm_insert_page due to aarch64 compatibility issues.

Thanks,
Leo

#2 lore raw
Quoted 10 lines · Leo Yan

Hi Arnaldo,

Quoted 6 lines · Arnaldo Carvalho de Melo
root@number:~# perf trace -e openat --max-events=1
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV
     0.000 ( 0.016 ms): ptyxis-agent/4375 openat(dfd: CWD, filename: "/proc/6593/cmdline", flags: RDONLY|CLOEXEC) = 13
root@number:~#
Quoted 5 lines
Quoted 3 lines

openat(AT_FDCWD, "/sys/kernel/btf/vmlinux", O_RDONLY) = 258 mmap(NULL, 6519699, PROT_READ, MAP_PRIVATE, 258, 0) = -1 ENODEV (No such device)
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV

Have you included the commit below in the kernel side?

It doesn't matter, libbpf should silently fallback to non-mmap() way, and it clearly doesn't.

We need something like this:

tools/lib/bpf/btf.c +0 -0
index f1d495dc66bb..37682908cb0f 100644
@@ -1384,12 +1384,12 @@ static struct btf *btf_parse_raw_mmap(const

char *path, struct btf *base_btf)

        fd = open(path, O_RDONLY);
        if (fd < 0)

- return libbpf_err_ptr(-errno);
+ return ERR_PTR(-errno);

        if (fstat(fd, &st) < 0) {
                err = -errno;
                close(fd);
-               return libbpf_err_ptr(err);
+               return ERR_PTR(err);
        }

        data = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
@@ -1397,7 +1397,7 @@ static struct btf *btf_parse_raw_mmap(const char
*path, struct btf *base_btf)
        close(fd);

        if (data == MAP_FAILED)

- return libbpf_err_ptr(err);
+ return ERR_PTR(err);

        btf = btf_new(data, st.st_size, base_btf, true);
        if (IS_ERR(btf))

libbpf_err_ptr() should be used for user-facing API functions, they return NULL on error and set errno, so checking for IS_ERR() is wrong here.

Lorenz, can you please test and send a proper fix ASAP?

Quoted 23 lines

commit a539e2a6d51d1c12d89eec149ccc72ec561639bc

Author: Lorenz Bauer <[email protected]>
Date:   Tue May 20 14:01:17 2025 +0100

btf: Allow mmap of vmlinux btf

User space needs access to kernel BTF for many modern features of BPF. Right now each process needs to read the BTF blob either in pieces or as a whole. Allow mmaping the sysfs file so that processes can directly access the memory allocated for it in the kernel.

remap_pfn_range is used instead of vm_insert_page due to aarch64 compatibility issues.

Thanks,
Leo

author Reviewed-by Tested-by
#3 lore raw
Quoted 8 lines · Andrii Nakryiko
Quoted 7 lines · Leo Yan
Quoted 6 lines · Arnaldo Carvalho de Melo
root@number:~# perf trace -e openat --max-events=1
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV
     0.000 ( 0.016 ms): ptyxis-agent/4375 openat(dfd: CWD, filename: "/proc/6593/cmdline", flags: RDONLY|CLOEXEC) = 13
root@number:~#
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

openat(AT_FDCWD, "/sys/kernel/btf/vmlinux", O_RDONLY) = 258 mmap(NULL, 6519699, PROT_READ, MAP_PRIVATE, 258, 0) = -1 ENODEV (No such device)
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV

Have you included the commit below in the kernel side?

It doesn't matter, libbpf should silently fallback to non-mmap() way,

Right, it has to work with older kernels, etc.

Quoted 31 lines

and it clearly doesn't.

We need something like this:
+++ b/tools/lib/bpf/btf.c

(quoted diff) +2 -2
@@ -1384,12 +1384,12 @@ static struct btf *btf_parse_raw_mmap(const
 char *path, struct btf *base_btf)
 
        fd = open(path, O_RDONLY);
        if (fd < 0)
-               return libbpf_err_ptr(-errno);
+               return ERR_PTR(-errno);
 
        if (fstat(fd, &st) < 0) {
                err = -errno;
                close(fd);
-               return libbpf_err_ptr(err);
+               return ERR_PTR(err);
        }
 
        data = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
(quoted diff) +1 -1
@@ -1397,7 +1397,7 @@ static struct btf *btf_parse_raw_mmap(const char
 *path, struct btf *base_btf)
        close(fd);
 
        if (data == MAP_FAILED)
-               return libbpf_err_ptr(err);
+               return ERR_PTR(err);
 
        btf = btf_new(data, st.st_size, base_btf, true);
        if (IS_ERR(btf))
Quoted 3 lines

libbpf_err_ptr() should be used for user-facing API functions, they return NULL on error and set errno, so checking for IS_ERR() is wrong here.

And the only user of the above function is:

                btf = btf_parse_raw_mmap(sysfs_btf_path, NULL);
                if (IS_ERR(btf))
                        btf = btf__parse(sysfs_btf_path, NULL);

That expects ERR_PTR() to then use IS_ERR().

I think this could be automated with something like coccinnele(sp)?

Anyway, I tested the patch above and it seems to fix the issue, so:

- Arnaldo

Quoted 25 lines

Lorenz, can you please test and send a proper fix ASAP?

Quoted 23 lines

commit a539e2a6d51d1c12d89eec149ccc72ec561639bc

Author: Lorenz Bauer <[email protected]>
Date:   Tue May 20 14:01:17 2025 +0100

btf: Allow mmap of vmlinux btf

User space needs access to kernel BTF for many modern features of BPF. Right now each process needs to read the BTF blob either in pieces or as a whole. Allow mmaping the sysfs file so that processes can directly access the memory allocated for it in the kernel.

remap_pfn_range is used instead of vm_insert_page due to aarch64 compatibility issues.

Thanks,
Leo

#4 lore raw
Quoted 63 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Andrii Nakryiko
Quoted 7 lines · Leo Yan
Quoted 6 lines · Arnaldo Carvalho de Melo
root@number:~# perf trace -e openat --max-events=1
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV
     0.000 ( 0.016 ms): ptyxis-agent/4375 openat(dfd: CWD, filename: "/proc/6593/cmdline", flags: RDONLY|CLOEXEC) = 13
root@number:~#
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

openat(AT_FDCWD, "/sys/kernel/btf/vmlinux", O_RDONLY) = 258 mmap(NULL, 6519699, PROT_READ, MAP_PRIVATE, 258, 0) = -1 ENODEV (No such device)
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV

Have you included the commit below in the kernel side?

It doesn't matter, libbpf should silently fallback to non-mmap() way,

Right, it has to work with older kernels, etc.

Quoted 31 lines

and it clearly doesn't.

We need something like this:
+++ b/tools/lib/bpf/btf.c

(quoted diff) +2 -2
@@ -1384,12 +1384,12 @@ static struct btf *btf_parse_raw_mmap(const
 char *path, struct btf *base_btf)
 
        fd = open(path, O_RDONLY);
        if (fd < 0)
-               return libbpf_err_ptr(-errno);
+               return ERR_PTR(-errno);
 
        if (fstat(fd, &st) < 0) {
                err = -errno;
                close(fd);
-               return libbpf_err_ptr(err);
+               return ERR_PTR(err);
        }
 
        data = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
(quoted diff) +1 -1
@@ -1397,7 +1397,7 @@ static struct btf *btf_parse_raw_mmap(const char
 *path, struct btf *base_btf)
        close(fd);
 
        if (data == MAP_FAILED)
-               return libbpf_err_ptr(err);
+               return ERR_PTR(err);
 
        btf = btf_new(data, st.st_size, base_btf, true);
        if (IS_ERR(btf))
Quoted 3 lines

libbpf_err_ptr() should be used for user-facing API functions, they return NULL on error and set errno, so checking for IS_ERR() is wrong here.

And the only user of the above function is:

                btf = btf_parse_raw_mmap(sysfs_btf_path, NULL);
                if (IS_ERR(btf))
                        btf = btf__parse(sysfs_btf_path, NULL);

That expects ERR_PTR() to then use IS_ERR().

Quoted 8 lines

I think this could be automated with something like coccinnele(sp)?

Anyway, I tested the patch above and it seems to fix the issue, so:

Just sent a quick fix to not leave this waiting over the weekend. Thanks for catching and reporting, Arnaldo!

Quoted 28 lines

- Arnaldo

Quoted 25 lines

Lorenz, can you please test and send a proper fix ASAP?

Quoted 23 lines

commit a539e2a6d51d1c12d89eec149ccc72ec561639bc

Author: Lorenz Bauer <[email protected]>
Date:   Tue May 20 14:01:17 2025 +0100

btf: Allow mmap of vmlinux btf

User space needs access to kernel BTF for many modern features of BPF. Right now each process needs to read the BTF blob either in pieces or as a whole. Allow mmaping the sysfs file so that processes can directly access the memory allocated for it in the kernel.

remap_pfn_range is used instead of vm_insert_page due to aarch64 compatibility issues.

Thanks,
Leo

#5 lore raw
In reply to Arnaldo Carvalho de Melo
Quoted 62 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Andrii Nakryiko
Quoted 7 lines · Leo Yan
Quoted 6 lines · Arnaldo Carvalho de Melo
root@number:~# perf trace -e openat --max-events=1
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV
     0.000 ( 0.016 ms): ptyxis-agent/4375 openat(dfd: CWD, filename: "/proc/6593/cmdline", flags: RDONLY|CLOEXEC) = 13
root@number:~#
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

openat(AT_FDCWD, "/sys/kernel/btf/vmlinux", O_RDONLY) = 258 mmap(NULL, 6519699, PROT_READ, MAP_PRIVATE, 258, 0) = -1 ENODEV (No such device)
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV

Have you included the commit below in the kernel side?

It doesn't matter, libbpf should silently fallback to non-mmap() way,

Right, it has to work with older kernels, etc.

Quoted 31 lines

and it clearly doesn't.

We need something like this:
+++ b/tools/lib/bpf/btf.c

(quoted diff) +2 -2
@@ -1384,12 +1384,12 @@ static struct btf *btf_parse_raw_mmap(const
 char *path, struct btf *base_btf)
 
        fd = open(path, O_RDONLY);
        if (fd < 0)
-               return libbpf_err_ptr(-errno);
+               return ERR_PTR(-errno);
 
        if (fstat(fd, &st) < 0) {
                err = -errno;
                close(fd);
-               return libbpf_err_ptr(err);
+               return ERR_PTR(err);
        }
 
        data = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
(quoted diff) +1 -1
@@ -1397,7 +1397,7 @@ static struct btf *btf_parse_raw_mmap(const char
 *path, struct btf *base_btf)
        close(fd);
 
        if (data == MAP_FAILED)
-               return libbpf_err_ptr(err);
+               return ERR_PTR(err);
 
        btf = btf_new(data, st.st_size, base_btf, true);
        if (IS_ERR(btf))
Quoted 3 lines

libbpf_err_ptr() should be used for user-facing API functions, they return NULL on error and set errno, so checking for IS_ERR() is wrong here.

And the only user of the above function is:

                btf = btf_parse_raw_mmap(sysfs_btf_path, NULL);
                if (IS_ERR(btf))
                        btf = btf__parse(sysfs_btf_path, NULL);

That expects ERR_PTR() to then use IS_ERR().

Quoted 7 lines

I think this could be automated with something like coccinnele(sp)?

Anyway, I tested the patch above and it seems to fix the issue, so:

Was this fixed/merged?

- Arnaldo

#6 lore raw
Quoted 66 lines · Arnaldo Carvalho de Melo
Quoted 63 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Andrii Nakryiko
Quoted 7 lines · Leo Yan
Quoted 6 lines · Arnaldo Carvalho de Melo
root@number:~# perf trace -e openat --max-events=1
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV
     0.000 ( 0.016 ms): ptyxis-agent/4375 openat(dfd: CWD, filename: "/proc/6593/cmdline", flags: RDONLY|CLOEXEC) = 13
root@number:~#
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

openat(AT_FDCWD, "/sys/kernel/btf/vmlinux", O_RDONLY) = 258 mmap(NULL, 6519699, PROT_READ, MAP_PRIVATE, 258, 0) = -1 ENODEV (No such device)
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV

Have you included the commit below in the kernel side?

It doesn't matter, libbpf should silently fallback to non-mmap() way,

Right, it has to work with older kernels, etc.

Quoted 31 lines

and it clearly doesn't.

We need something like this:
+++ b/tools/lib/bpf/btf.c

(quoted diff) +2 -2
@@ -1384,12 +1384,12 @@ static struct btf *btf_parse_raw_mmap(const
 char *path, struct btf *base_btf)
 
        fd = open(path, O_RDONLY);
        if (fd < 0)
-               return libbpf_err_ptr(-errno);
+               return ERR_PTR(-errno);
 
        if (fstat(fd, &st) < 0) {
                err = -errno;
                close(fd);
-               return libbpf_err_ptr(err);
+               return ERR_PTR(err);
        }
 
        data = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
(quoted diff) +1 -1
@@ -1397,7 +1397,7 @@ static struct btf *btf_parse_raw_mmap(const char
 *path, struct btf *base_btf)
        close(fd);
 
        if (data == MAP_FAILED)
-               return libbpf_err_ptr(err);
+               return ERR_PTR(err);
 
        btf = btf_new(data, st.st_size, base_btf, true);
        if (IS_ERR(btf))
Quoted 3 lines

libbpf_err_ptr() should be used for user-facing API functions, they return NULL on error and set errno, so checking for IS_ERR() is wrong here.

And the only user of the above function is:

                btf = btf_parse_raw_mmap(sysfs_btf_path, NULL);
                if (IS_ERR(btf))
                        btf = btf__parse(sysfs_btf_path, NULL);

That expects ERR_PTR() to then use IS_ERR().

I think this could be automated with something like coccinnele(sp)?

Quoted 7 lines
Quoted 5 lines

Anyway, I tested the patch above and it seems to fix the issue, so:

Was this fixed/merged?

yes, this was applied to bpf tree (not bpf-next)

Quoted 2 lines

- Arnaldo