BTF loading failing on perf
root@number:~# perf trace -e openat --max-events=1
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV
0.000 ( 0.016 ms): ptyxis-agent/4375 openat(dfd: CWD, filename: "/proc/6593/cmdline", flags: RDONLY|CLOEXEC) = 13
root@number:~#openat(AT_FDCWD, "/sys/kernel/btf/vmlinux", O_RDONLY) = 258 mmap(NULL, 6519699, PROT_READ, MAP_PRIVATE, 258, 0) = -1 ENODEV (No such device)
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV
root@number:~# ls -la /sys/kernel/btf/vmlinux -r--r--r--. 1 root root 6519699 Jun 6 12:19 /sys/kernel/btf/vmlinux root@number:~# uname -a Linux number 6.14.9-300.fc42.x86_64 #1 SMP PREEMPT_DYNAMIC Thu May 29 14:27:53 UTC 2025 x86_64 GNU/Linux root@number:~# head /etc/os-release NAME="Fedora Linux" VERSION="42 (Workstation Edition)" RELEASE_TYPE=stable ID=fedora VERSION_ID=42 VERSION_CODENAME="" PLATFORM_ID="platform:f42" PRETTY_NAME="Fedora Linux 42 (Workstation Edition)" ANSI_COLOR="0;38;2;60;110;180" LOGO=fedora-logo-icon root@number:~# rpm -q glibc-devel package glibc-devel is not installed root@number:~# rpm -q glibc glibc-2.41-5.fc42.x86_64 root@number:~#
Reverting the patch below "cures" the problem.
⬢ [acme@toolbx perf-tools]$ git log --oneline -10 tools/lib/bpf/370118ff875244d4 (HEAD -> perf-tools) Revert "libbpf: Use mmap to parse vmlinux BTF from sysfs" 90b83efa6701656e Merge tag 'bpf-next-6.16' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next3c0421c93ce4ff0f libbpf: Use mmap to parse vmlinux BTF from sysfs4e29128a9acec2a6 libbpf/btf: Fix string handling to support multi-split BTF d0445d7dd3fd9b15 libbpf: Check bpf_map_skeleton link for NULLfd5fd538a1f4b34c libbpf: Use proper errno value in nlattr62e23f183839c3d7 libbpf: Improve BTF dedup handling of "identical" BTF types 41d4ce6df3f49453 bpf: Fix uninitialized values in BPF_{CORE,PROBE}_READ 358b1c0f56ebb699 libbpf: Use proper errno value in linker8e64c387c942229c libbpf: Add identical pointer detection to btf_dedup_is_equiv()
⬢ [acme@toolbx perf-tools]$
root@number:~# perf -v
perf version 6.15.g370118ff8752
root@number:~# perf trace -e connect --max-events=5
0.000 ( 0.021 ms): DNS Res~ver #4/7932 connect(fd: 102, uservaddr: { .family: LOCAL, path: /run/systemd/r }, addrlen: 42) = 0
0.304 ( 0.013 ms): systemd-resolv/1420 connect(fd: 25, uservaddr: { .family: INET, port: 53, addr: 186.208.78.194 }, addrlen: 16) = 0
0.371 ( 0.002 ms): systemd-resolv/1420 connect(fd: 26, uservaddr: { .family: INET, port: 53, addr: 186.208.78.194 }, addrlen: 16) = 0
0.591 ( 0.009 ms): DNS Res~ver #2/7523 connect(fd: 104, uservaddr: { .family: LOCAL, path: /run/systemd/r }, addrlen: 42) = 0
0.731 ( 0.006 ms): systemd-resolv/1420 connect(fd: 28, uservaddr: { .family: INET, port: 53, addr: 186.208.78.194 }, addrlen: 16) = 0
root@number:~# Ideas?
- Arnaldo
⬢ [acme@toolbx perf-tools]$ git bisect good 3c0421c93ce4ff0f5f2612666122c34fc941d569 is the first bad commit commit 3c0421c93ce4ff0f5f2612666122c34fc941d569
Author: Lorenz Bauer <[email protected]> Date: Tue May 20 14:01:19 2025 +0100
libbpf: Use mmap to parse vmlinux BTF from sysfs
Teach libbpf to use mmap when parsing vmlinux BTF from /sys. We don't apply this to fall-back paths on the regular file system because there is no way to ensure that modifications underlying the MAP_PRIVATE mapping are not visible to the process.
tools/lib/bpf/btf.c | 89 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------ 1 file changed, 71 insertions(+), 18 deletions(-)
⬢ [acme@toolbx perf-tools]$
Hi Arnaldo,
Quoted 9 lines · Arnaldo Carvalho de Melo
root@number:~# perf trace -e openat --max-events=1 libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV 0.000 ( 0.016 ms): ptyxis-agent/4375 openat(dfd: CWD, filename: "/proc/6593/cmdline", flags: RDONLY|CLOEXEC) = 13 root@number:~#
openat(AT_FDCWD, "/sys/kernel/btf/vmlinux", O_RDONLY)= 258mmap(NULL, 6519699, PROT_READ, MAP_PRIVATE, 258, 0)= -1 ENODEV (No such device)
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux':-ENODEV
Have you included the commit below in the kernel side?
commit a539e2a6d51d1c12d89eec149ccc72ec561639bc
Author: Lorenz Bauer <[email protected]> Date: Tue May 20 14:01:17 2025 +0100
btf: Allow mmap of vmlinux btf
User space needs access to kernel BTF for many modern features of BPF. Right now each process needs to read the BTF blob either in pieces or as a whole. Allow mmaping the sysfs file so that processes can directly access the memory allocated for it in the kernel.
remap_pfn_range is used instead of vm_insert_page due to aarch64 compatibility issues.
Thanks,
Leo
Quoted 10 lines · Leo Yan
Hi Arnaldo,
Quoted 6 lines · Arnaldo Carvalho de Melo
root@number:~# perf trace -e openat --max-events=1 libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV 0.000 ( 0.016 ms): ptyxis-agent/4375 openat(dfd: CWD, filename: "/proc/6593/cmdline", flags: RDONLY|CLOEXEC) = 13 root@number:~#
Quoted 5 lines
Quoted 3 lines
openat(AT_FDCWD, "/sys/kernel/btf/vmlinux", O_RDONLY)= 258mmap(NULL, 6519699, PROT_READ, MAP_PRIVATE, 258, 0)= -1 ENODEV (No such device)
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux':-ENODEVHave you included the commit below in the kernel side?
It doesn't matter, libbpf should silently fallback to non-mmap() way, and it clearly doesn't.
We need something like this:
tools/lib/bpf/btf.c +0 -0
@@ -1384,12 +1384,12 @@ static struct btf *btf_parse_raw_mmap(const
char *path, struct btf *base_btf)
fd = open(path, O_RDONLY);
if (fd < 0)- return libbpf_err_ptr(-errno);
+ return ERR_PTR(-errno);
if (fstat(fd, &st) < 0) {
err = -errno;
close(fd);
- return libbpf_err_ptr(err);
+ return ERR_PTR(err);
}
data = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
@@ -1397,7 +1397,7 @@ static struct btf *btf_parse_raw_mmap(const char
*path, struct btf *base_btf)
close(fd);
if (data == MAP_FAILED)- return libbpf_err_ptr(err);
+ return ERR_PTR(err);
btf = btf_new(data, st.st_size, base_btf, true);
if (IS_ERR(btf))libbpf_err_ptr() should be used for user-facing API functions, they return NULL on error and set errno, so checking for IS_ERR() is wrong here.
Lorenz, can you please test and send a proper fix ASAP?
Quoted 23 lines
commit
a539e2a6d51d1c12d89eec149ccc72ec561639bcAuthor: Lorenz Bauer <[email protected]> Date: Tue May 20 14:01:17 2025 +0100btf: Allow mmap of vmlinux btf
User space needs access to kernel BTF for many modern features of BPF. Right now each process needs to read the BTF blob either in pieces or as a whole. Allow mmaping the sysfs file so that processes can directly access the memory allocated for it in the kernel.
remap_pfn_rangeis used instead ofvm_insert_pagedue to aarch64 compatibility issues.Signed-off-by Lorenz Bauer <[email protected]>Signed-off-by Andrii Nakryiko <[email protected]>Tested-by Alan Maguire <[email protected]>Reviewed-by Shakeel Butt <[email protected]>Thanks,
Leo
Quoted 8 lines · Andrii Nakryiko
Quoted 7 lines · Leo Yan
Quoted 6 lines · Arnaldo Carvalho de Melo
root@number:~# perf trace -e openat --max-events=1 libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV 0.000 ( 0.016 ms): ptyxis-agent/4375 openat(dfd: CWD, filename: "/proc/6593/cmdline", flags: RDONLY|CLOEXEC) = 13 root@number:~#
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
openat(AT_FDCWD, "/sys/kernel/btf/vmlinux", O_RDONLY)= 258mmap(NULL, 6519699, PROT_READ, MAP_PRIVATE, 258, 0)= -1 ENODEV (No such device)
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux':-ENODEVHave you included the commit below in the kernel side?
It doesn't matter, libbpf should silently fallback to non-mmap() way,
Right, it has to work with older kernels, etc.
Quoted 31 lines
and it clearly doesn't.
We need something like this:
+++b/tools/lib/bpf/btf.c(quoted diff) +2
-2@@ -1384,12 +1384,12 @@ static struct btf *btf_parse_raw_mmap(const char *path, struct btf *base_btf) fd = open(path, O_RDONLY); if (fd < 0) - return libbpf_err_ptr(-errno); + return ERR_PTR(-errno); if (fstat(fd, &st) < 0) { err = -errno; close(fd); - return libbpf_err_ptr(err); + return ERR_PTR(err); }data = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);(quoted diff) +1
-1@@ -1397,7 +1397,7 @@ static struct btf *btf_parse_raw_mmap(const char *path, struct btf *base_btf) close(fd); if (data == MAP_FAILED) - return libbpf_err_ptr(err); + return ERR_PTR(err); btf = btf_new(data, st.st_size, base_btf, true);if (IS_ERR(btf))
Quoted 3 lines
libbpf_err_ptr()should be used for user-facing API functions, they return NULL on error and set errno, so checking forIS_ERR()is wrong here.
And the only user of the above function is:
btf = btf_parse_raw_mmap(sysfs_btf_path, NULL);
if (IS_ERR(btf))
btf = btf__parse(sysfs_btf_path, NULL);That expects ERR_PTR() to then use IS_ERR().
I think this could be automated with something like coccinnele(sp)?
Anyway, I tested the patch above and it seems to fix the issue, so:
- Arnaldo
Quoted 25 lines
Lorenz, can you please test and send a proper fix ASAP?
Quoted 23 lines
commit
a539e2a6d51d1c12d89eec149ccc72ec561639bcAuthor: Lorenz Bauer <[email protected]> Date: Tue May 20 14:01:17 2025 +0100btf: Allow mmap of vmlinux btf
User space needs access to kernel BTF for many modern features of BPF. Right now each process needs to read the BTF blob either in pieces or as a whole. Allow mmaping the sysfs file so that processes can directly access the memory allocated for it in the kernel.
remap_pfn_rangeis used instead ofvm_insert_pagedue to aarch64 compatibility issues.Signed-off-by Lorenz Bauer <[email protected]>Signed-off-by Andrii Nakryiko <[email protected]>Tested-by Alan Maguire <[email protected]>Reviewed-by Shakeel Butt <[email protected]>Thanks,
Leo
Quoted 63 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Andrii Nakryiko
Quoted 7 lines · Leo Yan
Quoted 6 lines · Arnaldo Carvalho de Melo
root@number:~# perf trace -e openat --max-events=1 libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV 0.000 ( 0.016 ms): ptyxis-agent/4375 openat(dfd: CWD, filename: "/proc/6593/cmdline", flags: RDONLY|CLOEXEC) = 13 root@number:~#Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
openat(AT_FDCWD, "/sys/kernel/btf/vmlinux", O_RDONLY)= 258mmap(NULL, 6519699, PROT_READ, MAP_PRIVATE, 258, 0)= -1 ENODEV (No such device)
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux':-ENODEVHave you included the commit below in the kernel side?
It doesn't matter, libbpf should silently fallback to non-mmap() way,
Right, it has to work with older kernels, etc.
Quoted 31 lines
and it clearly doesn't.
We need something like this:
+++b/tools/lib/bpf/btf.c(quoted diff) +2
-2@@ -1384,12 +1384,12 @@ static struct btf *btf_parse_raw_mmap(const char *path, struct btf *base_btf) fd = open(path, O_RDONLY); if (fd < 0) - return libbpf_err_ptr(-errno); + return ERR_PTR(-errno); if (fstat(fd, &st) < 0) { err = -errno; close(fd); - return libbpf_err_ptr(err); + return ERR_PTR(err); }data = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);(quoted diff) +1
-1@@ -1397,7 +1397,7 @@ static struct btf *btf_parse_raw_mmap(const char *path, struct btf *base_btf) close(fd); if (data == MAP_FAILED) - return libbpf_err_ptr(err); + return ERR_PTR(err); btf = btf_new(data, st.st_size, base_btf, true);if (IS_ERR(btf))Quoted 3 lines
libbpf_err_ptr()should be used for user-facing API functions, they return NULL on error and set errno, so checking forIS_ERR()is wrong here.And the only user of the above function is:
btf = btf_parse_raw_mmap(sysfs_btf_path, NULL); if (IS_ERR(btf)) btf = btf__parse(sysfs_btf_path, NULL);That expects
ERR_PTR()to then useIS_ERR().
Quoted 8 lines
I think this could be automated with something like coccinnele(sp)?
Anyway, I tested the patch above and it seems to fix the issue, so:
Reported-by Arnaldo Carvalho de Melo <[email protected]>Reviewed-by Arnaldo Carvalho de Melo <[email protected]>
Just sent a quick fix to not leave this waiting over the weekend. Thanks for catching and reporting, Arnaldo!
Quoted 28 lines
- Arnaldo
Quoted 25 lines
Lorenz, can you please test and send a proper fix ASAP?
Quoted 23 lines
commit
a539e2a6d51d1c12d89eec149ccc72ec561639bcAuthor: Lorenz Bauer <[email protected]> Date: Tue May 20 14:01:17 2025 +0100btf: Allow mmap of vmlinux btf
User space needs access to kernel BTF for many modern features of BPF. Right now each process needs to read the BTF blob either in pieces or as a whole. Allow mmaping the sysfs file so that processes can directly access the memory allocated for it in the kernel.
remap_pfn_rangeis used instead ofvm_insert_pagedue to aarch64 compatibility issues.Signed-off-by Lorenz Bauer <[email protected]>Signed-off-by Andrii Nakryiko <[email protected]>Tested-by Alan Maguire <[email protected]>Reviewed-by Shakeel Butt <[email protected]>Thanks,
Leo
Quoted 62 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Andrii Nakryiko
Quoted 7 lines · Leo Yan
Quoted 6 lines · Arnaldo Carvalho de Melo
root@number:~# perf trace -e openat --max-events=1 libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV 0.000 ( 0.016 ms): ptyxis-agent/4375 openat(dfd: CWD, filename: "/proc/6593/cmdline", flags: RDONLY|CLOEXEC) = 13 root@number:~#Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
openat(AT_FDCWD, "/sys/kernel/btf/vmlinux", O_RDONLY)= 258mmap(NULL, 6519699, PROT_READ, MAP_PRIVATE, 258, 0)= -1 ENODEV (No such device)
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux':-ENODEVHave you included the commit below in the kernel side?
It doesn't matter, libbpf should silently fallback to non-mmap() way,
Right, it has to work with older kernels, etc.
Quoted 31 lines
and it clearly doesn't.
We need something like this:
+++b/tools/lib/bpf/btf.c(quoted diff) +2
-2@@ -1384,12 +1384,12 @@ static struct btf *btf_parse_raw_mmap(const char *path, struct btf *base_btf) fd = open(path, O_RDONLY); if (fd < 0) - return libbpf_err_ptr(-errno); + return ERR_PTR(-errno); if (fstat(fd, &st) < 0) { err = -errno; close(fd); - return libbpf_err_ptr(err); + return ERR_PTR(err); }data = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);(quoted diff) +1
-1@@ -1397,7 +1397,7 @@ static struct btf *btf_parse_raw_mmap(const char *path, struct btf *base_btf) close(fd); if (data == MAP_FAILED) - return libbpf_err_ptr(err); + return ERR_PTR(err); btf = btf_new(data, st.st_size, base_btf, true);if (IS_ERR(btf))Quoted 3 lines
libbpf_err_ptr()should be used for user-facing API functions, they return NULL on error and set errno, so checking forIS_ERR()is wrong here.And the only user of the above function is:
btf = btf_parse_raw_mmap(sysfs_btf_path, NULL); if (IS_ERR(btf)) btf = btf__parse(sysfs_btf_path, NULL);That expects
ERR_PTR()to then useIS_ERR().
Quoted 7 lines
I think this could be automated with something like coccinnele(sp)?
Anyway, I tested the patch above and it seems to fix the issue, so:
Reported-by Arnaldo Carvalho de Melo <[email protected]>Reviewed-by Arnaldo Carvalho de Melo <[email protected]>
Was this fixed/merged?
- Arnaldo
Quoted 66 lines · Arnaldo Carvalho de Melo
Quoted 63 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Andrii Nakryiko
Quoted 7 lines · Leo Yan
Quoted 6 lines · Arnaldo Carvalho de Melo
root@number:~# perf trace -e openat --max-events=1 libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV 0.000 ( 0.016 ms): ptyxis-agent/4375 openat(dfd: CWD, filename: "/proc/6593/cmdline", flags: RDONLY|CLOEXEC) = 13 root@number:~#Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
openat(AT_FDCWD, "/sys/kernel/btf/vmlinux", O_RDONLY)= 258mmap(NULL, 6519699, PROT_READ, MAP_PRIVATE, 258, 0)= -1 ENODEV (No such device)
libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux':-ENODEVHave you included the commit below in the kernel side?
It doesn't matter, libbpf should silently fallback to non-mmap() way,
Right, it has to work with older kernels, etc.
Quoted 31 lines
and it clearly doesn't.
We need something like this:
+++b/tools/lib/bpf/btf.c(quoted diff) +2
-2@@ -1384,12 +1384,12 @@ static struct btf *btf_parse_raw_mmap(const char *path, struct btf *base_btf) fd = open(path, O_RDONLY); if (fd < 0) - return libbpf_err_ptr(-errno); + return ERR_PTR(-errno); if (fstat(fd, &st) < 0) { err = -errno; close(fd); - return libbpf_err_ptr(err); + return ERR_PTR(err); }data = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);(quoted diff) +1
-1@@ -1397,7 +1397,7 @@ static struct btf *btf_parse_raw_mmap(const char *path, struct btf *base_btf) close(fd); if (data == MAP_FAILED) - return libbpf_err_ptr(err); + return ERR_PTR(err); btf = btf_new(data, st.st_size, base_btf, true);if (IS_ERR(btf))Quoted 3 lines
libbpf_err_ptr()should be used for user-facing API functions, they return NULL on error and set errno, so checking forIS_ERR()is wrong here.And the only user of the above function is:
btf = btf_parse_raw_mmap(sysfs_btf_path, NULL); if (IS_ERR(btf)) btf = btf__parse(sysfs_btf_path, NULL);That expects
ERR_PTR()to then useIS_ERR().I think this could be automated with something like coccinnele(sp)?
Quoted 7 lines
Quoted 5 lines
Anyway, I tested the patch above and it seems to fix the issue, so:
Reported-by Arnaldo Carvalho de Melo <[email protected]>Reviewed-by Arnaldo Carvalho de Melo <[email protected]>Was this fixed/merged?
yes, this was applied to bpf tree (not bpf-next)
Quoted 2 lines
- Arnaldo