perf build: Require at least clang 16.0.6 to build BPF skeletons
Arnaldo Carvalho de Melo
maintainerPATCH 1/1 perf build: Require at least clang 16.0.6 to build BPF skeletons
Howard reported problems using perf features that use BPF:
perf $ clang -v
Debian clang version 15.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
But it works with:
perf $ clang -v
Debian clang version 16.0.6 (15~deb12u1)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8) = 8 (kworker/0:0-eve)
perf $So lets make that the required version, if you happen to have a slightly older version where this work, please report so that we can adjust the minimum required version.
--- tools/perf/Makefile.config | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/Makefile.config +2 -2
@@ -705,8 +705,8 @@ ifeq ($(BUILD_BPF_SKEL),1) BUILD_BPF_SKEL := 0 else CLANG_VERSION := $(shell $(CLANG) --version | head -1 | sed 's/.*clang version \([[:digit:]]\+.[[:digit:]]\+.[[:digit:]]\+\).*/\1/g') - ifeq ($(call version-lt3,$(CLANG_VERSION),12.0.1),1) - $(warning Warning: Disabled BPF skeletons as reliable BTF generation needs at least $(CLANG) version 12.0.1) + ifeq ($(call version-lt3,$(CLANG_VERSION),16.0.6),1) + $(warning Warning: Disabled BPF skeletons as at least $(CLANG) version 16.0.6 is reported to be a working setup with the current of BPF based perf features) BUILD_BPF_SKEL := 0 endif endif
Signature · 1 line
2.46.0
James Clark
reviewerQuoted 31 lines · Arnaldo Carvalho de Melo
Howard reported problems using perf features that use BPF:
perf $ clang -v
Debian clang version 15.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/12Selected GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/12Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0But it works with:
perf $ clang -v Debian clang version 16.0.6 (15~deb12u1) Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /bin Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64 Selected multilib: .;@m64 perf $ ./perf trace -e write --max-events=1 0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8) = 8 (kworker/0:0-eve) perf $
Quoted 4 lines
So lets make that the required version, if you happen to have a slightly older version where this work, please report so that we can adjust the minimum required version.
I wasn't able to reproduce the issue with either of these versions. But I suppose it could be an issue with only 15.0.6.
$ clang -v Ubuntu clang version 15.0.7 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/11 Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64 Selected multilib: .;@m64
Ubuntu clang version 14.0.0-1ubuntu1.1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/11 Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64
Selected multilib: .;@m64
Arnaldo Carvalho de Melo
maintainerQuoted 32 lines · James Clark
Quoted 31 lines · Arnaldo Carvalho de Melo
Howard reported problems using perf features that use BPF:
perf $ clang -v
Debian clang version 15.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/12Selected GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/12Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0But it works with:
perf $ clang -v Debian clang version 16.0.6 (15~deb12u1) Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /bin Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64 Selected multilib: .;@m64 perf $ ./perf trace -e write --max-events=1 0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8) = 8 (kworker/0:0-eve) perf $
Quoted 6 lines
Quoted 3 lines
So lets make that the required version, if you happen to have a slightly older version where this work, please report so that we can adjust the minimum required version.
I wasn't able to reproduce the issue with either of these versions. But I suppose it could be an issue with only 15.0.6.
Interesting, that complicates things, probably the best way then is to try to build it, if it fails, mention that 15.0.6 is known to be problematic and suggest working versions?
- Arnaldo
Quoted 21 lines
$ clang -v Ubuntu clang version 15.0.7 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/11 Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64 Selected multilib: .;@m64Ubuntu clang version 14.0.0-1ubuntu1.1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir:/usr/bin
Found candidate GCC installation:/usr/bin/../lib/gcc/x86_64-linux-gnu/11Found candidate GCC installation:/usr/bin/../lib/gcc/x86_64-linux-gnu/12Selected GCC installation:/usr/bin/../lib/gcc/x86_64-linux-gnu/12Candidate multilib: .;@m64
Selected multilib: .;@m64
James Clark
reviewerQuoted 37 lines · Arnaldo Carvalho de Melo
Quoted 35 lines · James Clark
Quoted 34 lines · Arnaldo Carvalho de Melo
Howard reported problems using perf features that use BPF:
perf $ clang -v
Debian clang version 15.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/12Selected GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/12Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0But it works with:
perf $ clang -v Debian clang version 16.0.6 (15~deb12u1) Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /bin Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64 Selected multilib: .;@m64 perf $ ./perf trace -e write --max-events=1 0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8) = 8 (kworker/0:0-eve) perf $So lets make that the required version, if you happen to have a slightly older version where this work, please report so that we can adjust the minimum required version.
Quoted 9 lines
Quoted 2 lines
I wasn't able to reproduce the issue with either of these versions. But I suppose it could be an issue with only 15.0.6.
Interesting, that complicates things, probably the best way then is to try to build it, if it fails, mention that 15.0.6 is known to be problematic and suggest working versions?
- Arnaldo
I'll try to build it and see. But Ubuntu 22 only has clang 11 to 15 so making 16 the minimum could be an issue.
James Clark
reviewerQuoted 37 lines · Arnaldo Carvalho de Melo
Quoted 35 lines · James Clark
Quoted 34 lines · Arnaldo Carvalho de Melo
Howard reported problems using perf features that use BPF:
perf $ clang -v
Debian clang version 15.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/12Selected GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/12Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0But it works with:
perf $ clang -v Debian clang version 16.0.6 (15~deb12u1) Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /bin Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64 Selected multilib: .;@m64 perf $ ./perf trace -e write --max-events=1 0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8) = 8 (kworker/0:0-eve) perf $So lets make that the required version, if you happen to have a slightly older version where this work, please report so that we can adjust the minimum required version.
Quoted 8 lines
Quoted 2 lines
I wasn't able to reproduce the issue with either of these versions. But I suppose it could be an issue with only 15.0.6.
Interesting, that complicates things, probably the best way then is to try to build it, if it fails, mention that 15.0.6 is known to be problematic and suggest working versions?
- Arnaldo
I still wasn't able to reproduce it with 15.0.6. And I double checked with V=1 that the build was using the right clang. I suppose it could be a build configuration issue, or maybe with a different kernel version?
$ uname --kernel-release 6.8.0-76060800daily20240311-generic
$ ../../llvm-project/build/bin/clang -v clang version 15.0.6 (https://github.com/llvm/llvm-project.git 088f33605d8a61ff519c580a71b1dd57d16a03f8) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /home/james/workspace/linux/linux/../../llvm-project/build/bin Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11 Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64 Selected multilib: .;@m64
$ git log commit 003265bb6f028d7bcd7cbd92d6ba2b4e26382796 (perf-tools-next/perf-tools-next)
$ make O=../build/local/ CLANG=../../llvm-project/build/bin/clang -C \ tools/perf
$ perf trace -e write --max-events=1
0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
0x7fffa102d9b0, count: 8) = 8Hello James and Arnaldo,
Quoted 54 lines · Arnaldo Carvalho de Melo
Quoted 50 lines · James Clark
Quoted 45 lines · Arnaldo Carvalho de Melo
Quoted 32 lines · James Clark
Quoted 31 lines · Arnaldo Carvalho de Melo
Howard reported problems using perf features that use BPF:
perf $ clang -v
Debian clang version 15.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/12Selected GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/12Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0But it works with:
perf $ clang -v Debian clang version 16.0.6 (15~deb12u1) Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /bin Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64 Selected multilib: .;@m64 perf $ ./perf trace -e write --max-events=1 0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8) = 8 (kworker/0:0-eve) perf $Quoted 6 lines
Quoted 3 lines
So lets make that the required version, if you happen to have a slightly older version where this work, please report so that we can adjust the minimum required version.
I wasn't able to reproduce the issue with either of these versions. But I suppose it could be an issue with only 15.0.6.
Interesting, that complicates things, probably the best way then is to try to build it, if it fails, mention that 15.0.6 is known to be problematic and suggest working versions?
- Arnaldo
Quoted 37 lines
Quoted 3 lines
I still wasn't able to reproduce it with 15.0.6. And I double checked with V=1 that the build was using the right clang. I suppose it could be a build configuration issue, or maybe with a different kernel version?
Howard? If not reproducible we can revert it.
- Arnaldo
Quoted 27 lines
$ uname --kernel-release 6.8.0-76060800daily20240311-generic$ ../../llvm-project/build/bin/clang -v clang version 15.0.6 (https://github.com/llvm/llvm-project.git 088f33605d8a61ff519c580a71b1dd57d16a03f8) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /home/james/workspace/linux/linux/../../llvm-project/build/bin Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11 Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64 Selected multilib: .;@m64$ git log commit 003265bb6f028d7bcd7cbd92d6ba2b4e26382796 (perf-tools-next/perf-tools-next)$ make O=../build/local/ CLANG=../../llvm-project/build/bin/clang -C \ tools/perf$ perf trace -e write --max-events=10.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf: 0x7fffa102d9b0, count: 8) = 8
Thanks for the effort! But this output seems to be unaugmented? I think you are testing the perf/perf-tools-next branch that's slightly older (perf mem: Fix the wrong reference in parse_record_events() 003265bb6f028d7bcd7cbd92d6ba2b4e26382796), and I think that doesn't have the new perf trace feature yet...
That's why the perf trace output looks like:
Quoted 4 lines
Quoted 4 lines
$ perf trace -e write --max-events=10.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf: 0x7fffa102d9b0, count: 8)
With the new buffer augmentation it should look like this:
perf $ ./perf trace -e write --max-events=1
0.000 ( 0.011 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,
count: 8) =
The new perf trace can be found on
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/
perf-tools-next
perf $ git log
commit 1de5b5dcb8353f36581c963df2d359a5f151a0be (HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next)
Also, if you want to test the clang, you may want to checkout to the commit before the clang-16 guard (otherwise you can't test different clang version).
perf $ git log --oneline1de5b5dcb835 (HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next) perf
trace: Mark the 'head' arg in the set_robust_list syscall as coming from user space0c1019e3463b perf trace: Mark the 'rseq' arg in the rseq syscall as coming from user spaceedf3ce0ed38e perf env: Find correct branch counter info on hybrid 9953807c9e01 perf evlist: Print hint for groupeb9b9a6f5ab3 tools: Drop nonsensical -O689c0a55e550e perf pmu: To info add event_type_descf08cc258431d perf evsel: Add accessor for tool_event 925320737ae2 perf pmus: Fake PMU clean upd3d5c1a00fcd perf list: Avoid potential out of bounds memory read 4ae354d73a8e perf help: Fix a typo ("bellow")74298dd8acb8 perf ftrace: Detect whether ftrace is enabled on system 83420d5f5863 perf test shell probe_vfs_getname: Remove extraneous '=' from probe line number regex9327f0ecad48 perf build: Require at least clang 16.0.6 to build BPF skeletons. **********[[[[[[[[[[[[!!!!(This is the clang guard)]]]]]]]]]]*****4c1af9bf97eb perf trace: If a syscall arg is marked as 'const', assume it is coming _from_ userspacee37b315c17df perf parse-events: Remove duplicated include in parse-events.c
My machine is debian 12, and I cannot build perf trace with the new augmentation feature on clang-13, clang-14, and clang-15.
perf $ uname -r
6.11.0-061100rc7-generic
perf $ git log
commit 4c1af9bf97eb56d069421c3233ce61608458d5c8 (HEAD)
perf $ make CLANG=/bin/clang-13 && ./perf trace -e write --max-events=1
...
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter(struct syscall_enter_args *args) @ augmented_raw_syscalls.bpf.c:509
0: (bf) r7 = r1 ; R1=ctx() R7_w=ctx()
perf $ make CLANG=/bin/clang-14 && ./perf trace -e write --max-events=1
...
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter(struct syscall_enter_args *args) @ augmented_raw_syscalls.bpf.c:509
perf $ make CLANG=/bin/clang-15 && ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
perf $ make CLANG=/bin/clang-16 && ./perf trace -e write --max-events=1
0.000 ( 0.010 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,
count: 8) =
Anyway thank you so much, I'll try to reproduce it on Ubuntu, like you mentioned.
Thanks,
Howard
Quoted 1 line
Quoted 1 line
Hello,
I'm able to reproduce it on Ubuntu 24.
perf $ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
perf $ uname -r
6.11.0-061100-generic
perf $ git log
commit 4c1af9bf97eb56d069421c3233ce61608458d5c8 (HEAD)
Author: Arnaldo Carvalho de Melo <[email protected]> Date: Tue Sep 10 13:54:23 2024 -0300
perf trace: If a syscall arg is marked as 'const', assume it is
coming _from_ userspace
perf $ clang -v
Ubuntu clang version 15.0.7
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter_rename(struct syscall_enter_args *args) @ augmented_raw_syscalls.bpf.c:275
With clang-16:
perf $ clang -v
Ubuntu clang version 16.0.6 (23ubuntu4)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
0.000 ( 0.021 ms): sudo/4741 write(fd: 8, buf:
\27[?25l\27[37m\27[48;5;96m\27[H[trace] , count: 205) =
Quoted 2 lines
I'll try to build it and see. But Ubuntu 22 only has clang 11 to 15 so making 16 the minimum could be an issue.
Yes, I think disabling any distro that doesn't come with clang 16 (released in 2023) is not a good idea. I'll try to tame the BPF verifier.
Thanks,
Howard
Quoted 199 lines · Howard Chu
Hello James and Arnaldo,
Quoted 54 lines · Arnaldo Carvalho de Melo
Quoted 50 lines · James Clark
Quoted 45 lines · Arnaldo Carvalho de Melo
Quoted 32 lines · James Clark
Quoted 31 lines · Arnaldo Carvalho de Melo
Howard reported problems using perf features that use BPF:
perf $ clang -v
Debian clang version 15.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/12Selected GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/12Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0But it works with:
perf $ clang -v Debian clang version 16.0.6 (15~deb12u1) Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /bin Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64 Selected multilib: .;@m64 perf $ ./perf trace -e write --max-events=1 0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8) = 8 (kworker/0:0-eve) perf $Quoted 6 lines
Quoted 3 lines
So lets make that the required version, if you happen to have a slightly older version where this work, please report so that we can adjust the minimum required version.
I wasn't able to reproduce the issue with either of these versions. But I suppose it could be an issue with only 15.0.6.
Interesting, that complicates things, probably the best way then is to try to build it, if it fails, mention that 15.0.6 is known to be problematic and suggest working versions?
- Arnaldo
Quoted 37 lines
Quoted 3 lines
I still wasn't able to reproduce it with 15.0.6. And I double checked with V=1 that the build was using the right clang. I suppose it could be a build configuration issue, or maybe with a different kernel version?
Howard? If not reproducible we can revert it.
- Arnaldo
Quoted 27 lines
$ uname --kernel-release 6.8.0-76060800daily20240311-generic$ ../../llvm-project/build/bin/clang -v clang version 15.0.6 (https://github.com/llvm/llvm-project.git 088f33605d8a61ff519c580a71b1dd57d16a03f8) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /home/james/workspace/linux/linux/../../llvm-project/build/bin Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11 Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64 Selected multilib: .;@m64$ git log commit 003265bb6f028d7bcd7cbd92d6ba2b4e26382796 (perf-tools-next/perf-tools-next)$ make O=../build/local/ CLANG=../../llvm-project/build/bin/clang -C \ tools/perf$ perf trace -e write --max-events=10.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf: 0x7fffa102d9b0, count: 8) = 8Thanks for the effort! But this output seems to be unaugmented? I think you are testing the perf/perf-tools-next branch that's slightly older (perf mem: Fix the wrong reference in
parse_record_events()003265bb6f028d7bcd7cbd92d6ba2b4e26382796), and I think that doesn't have the new perf trace feature yet...That's why the perf trace output looks like:
Quoted 4 lines
Quoted 4 lines
$ perf trace -e write --max-events=10.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf: 0x7fffa102d9b0, count: 8)With the new buffer augmentation it should look like this:
perf $ ./perf trace -e write --max-events=1
0.000 ( 0.011 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,count: 8) =
The new perf trace can be found on
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/
perf-tools-nextperf $ git log
commit1de5b5dcb8353f36581c963df2d359a5f151a0be(HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next)Also, if you want to test the clang, you may want to checkout to the commit before the clang-16 guard (otherwise you can't test different clang version).
perf $ git log --oneline
1de5b5dcb835(HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next) perf
trace: Mark the 'head' arg in theset_robust_listsyscall as coming from user space0c1019e3463bperf trace: Mark the 'rseq' arg in the rseq syscall as coming from user spaceedf3ce0ed38eperf env: Find correct branch counter info on hybrid9953807c9e01perf evlist: Print hint for groupeb9b9a6f5ab3tools: Drop nonsensical -O689c0a55e550eperf pmu: To info addevent_type_descf08cc258431dperf evsel: Add accessor fortool_event925320737ae2perf pmus: Fake PMU clean upd3d5c1a00fcdperf list: Avoid potential out of bounds memory read4ae354d73a8eperf help: Fix a typo ("bellow")74298dd8acb8perf ftrace: Detect whether ftrace is enabled on system83420d5f5863perf test shellprobe_vfs_getname: Remove extraneous '=' from probe line number regex9327f0ecad48perf build: Require at least clang 16.0.6 to build BPF skeletons. **********[[[[[[[[[[[[!!!!(This is the clang guard)]]]]]]]]]]*****4c1af9bf97ebperf trace: If a syscall arg is marked as 'const', assume it is coming _from_ userspacee37b315c17dfperf parse-events: Remove duplicated include inparse-events.cMy machine is debian 12, and I cannot build perf trace with the new augmentation feature on clang-13, clang-14, and clang-15.
perf $ uname -r
6.11.0-061100rc7-genericperf $ git log
commit4c1af9bf97eb56d069421c3233ce61608458d5c8(HEAD)perf $ make CLANG=/bin/clang-13 && ./perf trace -e write --max-events=1
...
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; intsys_enter(struct syscall_enter_args *args)@ augmented_raw_syscalls.bpf.c:509
0: (bf) r7 = r1 ; R1=ctx() R7_w=ctx()perf $ make CLANG=/bin/clang-14 && ./perf trace -e write --max-events=1
...
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; intsys_enter(struct syscall_enter_args *args)@ augmented_raw_syscalls.bpf.c:509perf $ make CLANG=/bin/clang-15 && ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0perf $ make CLANG=/bin/clang-16 && ./perf trace -e write --max-events=1
0.000 ( 0.010 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,count: 8) =
Anyway thank you so much, I'll try to reproduce it on Ubuntu, like you mentioned.
Thanks,
HowardQuoted 1 line
Quoted 1 line
Hello,
I got some good news:
Not being able to pass the BPF verifier is solely (or bi-solely) caused by these two functions:
SEC("tp/syscalls/sys_enter_rename")
int sys_enter_rename(struct syscall_enter_args *args)
SEC("tp/syscalls/sys_enter_renameat2")
int sys_enter_renameat2(struct syscall_enter_args *args)
The problem with it is double-string augmentation, but I haven't figured out how to solve it. Can you please take a look please, Arnaldo? Now I just commented these two BPF functions and perf trace runs no problem, built by clang that's older than clang 16, for example clang 15.0.7.
perf $ clang -v
Ubuntu clang version 15.0.7
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
0.000 ( 0.008 ms): gmain/2173 write(fd: 4, buf: \1\0\0\0\0\0\0\0,
count: 8) =
But plot twist, it won't build on clang-14
perf $ clang -v
Ubuntu clang version 14.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter(struct syscall_enter_args *args) @ augmented_raw_syscalls.bpf.c:509
0: (bf) r7 = r1 ; R1=ctx() R7_w=ctx()
This time the problem is my code in BTF general collector:
; payload_offset += written; @ augmented_raw_syscalls.bpf.c:497
I'll resolve this.
Thanks,
Howard
Quoted 276 lines · Howard Chu
Hello,
I'm able to reproduce it on Ubuntu 24.
perf $ cat
/etc/os-release
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=noble
LOGO=ubuntu-logoperf $ uname -r
6.11.0-061100-genericperf $ git log
commit4c1af9bf97eb56d069421c3233ce61608458d5c8(HEAD)Author: Arnaldo Carvalho de Melo <[email protected]> Date: Tue Sep 10 13:54:23 2024 -0300perf trace: If a syscall arg is marked as 'const', assume it is
coming _from_ userspaceperf $ clang -v
Ubuntu clang version 15.0.7
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Selected GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Candidate multilib: .;@m64
Selected multilib: .;@m64perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; intsys_enter_rename(struct syscall_enter_args *args)@ augmented_raw_syscalls.bpf.c:275With clang-16:
perf $ clang -v
Ubuntu clang version 16.0.6 (23ubuntu4)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Selected GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Candidate multilib: .;@m64
Selected multilib: .;@m64perf $ ./perf trace -e write --max-events=1
0.000 ( 0.021 ms): sudo/4741 write(fd: 8, buf:\27[?25l\27[37m\27[48;5;96m\27[H[trace] , count: 205) =
Quoted 2 lines
I'll try to build it and see. But Ubuntu 22 only has clang 11 to 15 so making 16 the minimum could be an issue.
Yes, I think disabling any distro that doesn't come with clang 16 (released in 2023) is not a good idea. I'll try to tame the BPF verifier.
Thanks,
HowardQuoted 199 lines · Howard Chu
Hello James and Arnaldo,
Quoted 54 lines · Arnaldo Carvalho de Melo
Quoted 50 lines · James Clark
Quoted 45 lines · Arnaldo Carvalho de Melo
Quoted 32 lines · James Clark
Quoted 31 lines · Arnaldo Carvalho de Melo
Howard reported problems using perf features that use BPF:
perf $ clang -v
Debian clang version 15.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/12Selected GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/12Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0But it works with:
perf $ clang -v Debian clang version 16.0.6 (15~deb12u1) Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /bin Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64 Selected multilib: .;@m64 perf $ ./perf trace -e write --max-events=1 0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8) = 8 (kworker/0:0-eve) perf $Quoted 6 lines
Quoted 3 lines
So lets make that the required version, if you happen to have a slightly older version where this work, please report so that we can adjust the minimum required version.
I wasn't able to reproduce the issue with either of these versions. But I suppose it could be an issue with only 15.0.6.
Interesting, that complicates things, probably the best way then is to try to build it, if it fails, mention that 15.0.6 is known to be problematic and suggest working versions?
- Arnaldo
Quoted 37 lines
Quoted 3 lines
I still wasn't able to reproduce it with 15.0.6. And I double checked with V=1 that the build was using the right clang. I suppose it could be a build configuration issue, or maybe with a different kernel version?
Howard? If not reproducible we can revert it.
- Arnaldo
Quoted 27 lines
$ uname --kernel-release 6.8.0-76060800daily20240311-generic$ ../../llvm-project/build/bin/clang -v clang version 15.0.6 (https://github.com/llvm/llvm-project.git 088f33605d8a61ff519c580a71b1dd57d16a03f8) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /home/james/workspace/linux/linux/../../llvm-project/build/bin Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11 Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64 Selected multilib: .;@m64$ git log commit 003265bb6f028d7bcd7cbd92d6ba2b4e26382796 (perf-tools-next/perf-tools-next)$ make O=../build/local/ CLANG=../../llvm-project/build/bin/clang -C \ tools/perf$ perf trace -e write --max-events=10.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf: 0x7fffa102d9b0, count: 8) = 8Thanks for the effort! But this output seems to be unaugmented? I think you are testing the perf/perf-tools-next branch that's slightly older (perf mem: Fix the wrong reference in
parse_record_events()003265bb6f028d7bcd7cbd92d6ba2b4e26382796), and I think that doesn't have the new perf trace feature yet...That's why the perf trace output looks like:
Quoted 4 lines
Quoted 4 lines
$ perf trace -e write --max-events=10.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf: 0x7fffa102d9b0, count: 8)With the new buffer augmentation it should look like this:
perf $ ./perf trace -e write --max-events=1
0.000 ( 0.011 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,count: 8) =
The new perf trace can be found on
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/
perf-tools-nextperf $ git log
commit1de5b5dcb8353f36581c963df2d359a5f151a0be(HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next)Also, if you want to test the clang, you may want to checkout to the commit before the clang-16 guard (otherwise you can't test different clang version).
perf $ git log --oneline
1de5b5dcb835(HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next) perf
trace: Mark the 'head' arg in theset_robust_listsyscall as coming from user space0c1019e3463bperf trace: Mark the 'rseq' arg in the rseq syscall as coming from user spaceedf3ce0ed38eperf env: Find correct branch counter info on hybrid9953807c9e01perf evlist: Print hint for groupeb9b9a6f5ab3tools: Drop nonsensical -O689c0a55e550eperf pmu: To info addevent_type_descf08cc258431dperf evsel: Add accessor fortool_event925320737ae2perf pmus: Fake PMU clean upd3d5c1a00fcdperf list: Avoid potential out of bounds memory read4ae354d73a8eperf help: Fix a typo ("bellow")74298dd8acb8perf ftrace: Detect whether ftrace is enabled on system83420d5f5863perf test shellprobe_vfs_getname: Remove extraneous '=' from probe line number regex9327f0ecad48perf build: Require at least clang 16.0.6 to build BPF skeletons. **********[[[[[[[[[[[[!!!!(This is the clang guard)]]]]]]]]]]*****4c1af9bf97ebperf trace: If a syscall arg is marked as 'const', assume it is coming _from_ userspacee37b315c17dfperf parse-events: Remove duplicated include inparse-events.cMy machine is debian 12, and I cannot build perf trace with the new augmentation feature on clang-13, clang-14, and clang-15.
perf $ uname -r
6.11.0-061100rc7-genericperf $ git log
commit4c1af9bf97eb56d069421c3233ce61608458d5c8(HEAD)perf $ make CLANG=/bin/clang-13 && ./perf trace -e write --max-events=1
...
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; intsys_enter(struct syscall_enter_args *args)@ augmented_raw_syscalls.bpf.c:509
0: (bf) r7 = r1 ; R1=ctx() R7_w=ctx()perf $ make CLANG=/bin/clang-14 && ./perf trace -e write --max-events=1
...
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; intsys_enter(struct syscall_enter_args *args)@ augmented_raw_syscalls.bpf.c:509perf $ make CLANG=/bin/clang-15 && ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0perf $ make CLANG=/bin/clang-16 && ./perf trace -e write --max-events=1
0.000 ( 0.010 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,count: 8) =
Anyway thank you so much, I'll try to reproduce it on Ubuntu, like you mentioned.
Thanks,
HowardQuoted 1 line
Quoted 1 line
Quoted 10 lines · Howard Chu
Hello,
I got some good news:
Not being able to pass the BPF verifier is solely (or bi-solely) caused by these two functions:
SEC("tp/syscalls/sys_enter_rename")
intsys_enter_rename(struct syscall_enter_args *args)
Quoted 9 lines
SEC("tp/syscalls/sys_enter_renameat2")
intsys_enter_renameat2(struct syscall_enter_args *args)The problem with it is double-string augmentation, but I haven't figured out how to solve it. Can you please take a look please, Arnaldo? Now I just commented these two BPF functions and perf trace runs no problem, built by clang that's older than clang 16, for example clang 15.0.7.
hi Howard,
I could be wrong - and I don't know this code at all - but I _think_ the problem is here:
oldpath_len = augmented_arg__read_str(&augmented_args->arg,
oldpath_arg, sizeof(augmented_args->arg.value));
augmented_args->arg.size = PERF_ALIGN(oldpath_len + 1, sizeof(u64));
len += augmented_args->arg.size;
struct augmented_arg *arg2 = (void *)&augmented_args->arg.value+ augmented_args->arg.size;
newpath_len = augmented_arg__read_str(arg2, newpath_arg,sizeof(augmented_args->arg.value));
Notice that these strings share the augmented_args->arg.value for storage, but in the second case (reading arg2) we have added augmented_args->arg.size to the offset where we store the second string. However at the same time we have also told augmented_arg__read_str() that it has
sizeof(augmented_args->arg.value)
...to work with as arg_len, where in reality we should have specified
sizeof(augmented_args->arg.value) - augmented_args->arg.size
...and ensured that after doing that subtraction, there is still space to work with (i.e. the above is > 0).
I _think_ the verifier is likely complaining that we can write past the end of augmented_args->arg.value , does that sound right?
Fixing the arg_len parameter for cases where we record more than one augmented string by reducing arg_len available should solve this if so. Hope this helps,
Alan
Quoted 318 lines
perf $ clang -v
Ubuntu clang version 15.0.7
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Selected GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=10.000 ( 0.008 ms): gmain/2173 write(fd: 4, buf: \1\0\0\0\0\0\0\0,count: 8) =
But plot twist, it won't build on clang-14
perf $ clang -v
Ubuntu clang version 14.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Selected GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; intsys_enter(struct syscall_enter_args *args)@ augmented_raw_syscalls.bpf.c:509
0: (bf) r7 = r1 ; R1=ctx() R7_w=ctx()This time the problem is my code in BTF general collector:
;
payload_offset+= written; @ augmented_raw_syscalls.bpf.c:497I'll resolve this.
Thanks,
HowardQuoted 276 lines · Howard Chu
Hello,
I'm able to reproduce it on Ubuntu 24.
perf $ cat
/etc/os-release
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=noble
LOGO=ubuntu-logoperf $ uname -r
6.11.0-061100-genericperf $ git log
commit4c1af9bf97eb56d069421c3233ce61608458d5c8(HEAD)Author: Arnaldo Carvalho de Melo <[email protected]> Date: Tue Sep 10 13:54:23 2024 -0300perf trace: If a syscall arg is marked as 'const', assume it is
coming _from_ userspaceperf $ clang -v
Ubuntu clang version 15.0.7
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Selected GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Candidate multilib: .;@m64
Selected multilib: .;@m64perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; intsys_enter_rename(struct syscall_enter_args *args)@ augmented_raw_syscalls.bpf.c:275With clang-16:
perf $ clang -v
Ubuntu clang version 16.0.6 (23ubuntu4)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Selected GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Candidate multilib: .;@m64
Selected multilib: .;@m64perf $ ./perf trace -e write --max-events=1
0.000 ( 0.021 ms): sudo/4741 write(fd: 8, buf:\27[?25l\27[37m\27[48;5;96m\27[H[trace] , count: 205) =
Quoted 2 lines
I'll try to build it and see. But Ubuntu 22 only has clang 11 to 15 so making 16 the minimum could be an issue.
Yes, I think disabling any distro that doesn't come with clang 16 (released in 2023) is not a good idea. I'll try to tame the BPF verifier.
Thanks,
HowardQuoted 199 lines · Howard Chu
Hello James and Arnaldo,
Quoted 54 lines · Arnaldo Carvalho de Melo
Quoted 50 lines · James Clark
Quoted 45 lines · Arnaldo Carvalho de Melo
Quoted 32 lines · James Clark
Quoted 31 lines · Arnaldo Carvalho de Melo
Howard reported problems using perf features that use BPF:
perf $ clang -v
Debian clang version 15.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/12Selected GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/12Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0But it works with:
perf $ clang -v Debian clang version 16.0.6 (15~deb12u1) Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /bin Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64 Selected multilib: .;@m64 perf $ ./perf trace -e write --max-events=1 0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8) = 8 (kworker/0:0-eve) perf $Quoted 6 lines
Quoted 3 lines
So lets make that the required version, if you happen to have a slightly older version where this work, please report so that we can adjust the minimum required version.
I wasn't able to reproduce the issue with either of these versions. But I suppose it could be an issue with only 15.0.6.
Interesting, that complicates things, probably the best way then is to try to build it, if it fails, mention that 15.0.6 is known to be problematic and suggest working versions?
- Arnaldo
Quoted 37 lines
Quoted 3 lines
I still wasn't able to reproduce it with 15.0.6. And I double checked with V=1 that the build was using the right clang. I suppose it could be a build configuration issue, or maybe with a different kernel version?
Howard? If not reproducible we can revert it.
- Arnaldo
Quoted 27 lines
$ uname --kernel-release 6.8.0-76060800daily20240311-generic$ ../../llvm-project/build/bin/clang -v clang version 15.0.6 (https://github.com/llvm/llvm-project.git 088f33605d8a61ff519c580a71b1dd57d16a03f8) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /home/james/workspace/linux/linux/../../llvm-project/build/bin Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11 Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64 Selected multilib: .;@m64$ git log commit 003265bb6f028d7bcd7cbd92d6ba2b4e26382796 (perf-tools-next/perf-tools-next)$ make O=../build/local/ CLANG=../../llvm-project/build/bin/clang -C \ tools/perf$ perf trace -e write --max-events=10.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf: 0x7fffa102d9b0, count: 8) = 8Thanks for the effort! But this output seems to be unaugmented? I think you are testing the perf/perf-tools-next branch that's slightly older (perf mem: Fix the wrong reference in
parse_record_events()003265bb6f028d7bcd7cbd92d6ba2b4e26382796), and I think that doesn't have the new perf trace feature yet...That's why the perf trace output looks like:
Quoted 4 lines
Quoted 4 lines
$ perf trace -e write --max-events=10.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf: 0x7fffa102d9b0, count: 8)With the new buffer augmentation it should look like this:
perf $ ./perf trace -e write --max-events=1
0.000 ( 0.011 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,count: 8) =
The new perf trace can be found on
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/
perf-tools-nextperf $ git log
commit1de5b5dcb8353f36581c963df2d359a5f151a0be(HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next)Also, if you want to test the clang, you may want to checkout to the commit before the clang-16 guard (otherwise you can't test different clang version).
perf $ git log --oneline
1de5b5dcb835(HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next) perf
trace: Mark the 'head' arg in theset_robust_listsyscall as coming from user space0c1019e3463bperf trace: Mark the 'rseq' arg in the rseq syscall as coming from user spaceedf3ce0ed38eperf env: Find correct branch counter info on hybrid9953807c9e01perf evlist: Print hint for groupeb9b9a6f5ab3tools: Drop nonsensical -O689c0a55e550eperf pmu: To info addevent_type_descf08cc258431dperf evsel: Add accessor fortool_event925320737ae2perf pmus: Fake PMU clean upd3d5c1a00fcdperf list: Avoid potential out of bounds memory read4ae354d73a8eperf help: Fix a typo ("bellow")74298dd8acb8perf ftrace: Detect whether ftrace is enabled on system83420d5f5863perf test shellprobe_vfs_getname: Remove extraneous '=' from probe line number regex9327f0ecad48perf build: Require at least clang 16.0.6 to build BPF skeletons. **********[[[[[[[[[[[[!!!!(This is the clang guard)]]]]]]]]]]*****4c1af9bf97ebperf trace: If a syscall arg is marked as 'const', assume it is coming _from_ userspacee37b315c17dfperf parse-events: Remove duplicated include inparse-events.cMy machine is debian 12, and I cannot build perf trace with the new augmentation feature on clang-13, clang-14, and clang-15.
perf $ uname -r
6.11.0-061100rc7-genericperf $ git log
commit4c1af9bf97eb56d069421c3233ce61608458d5c8(HEAD)perf $ make CLANG=/bin/clang-13 && ./perf trace -e write --max-events=1
...
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; intsys_enter(struct syscall_enter_args *args)@ augmented_raw_syscalls.bpf.c:509
0: (bf) r7 = r1 ; R1=ctx() R7_w=ctx()perf $ make CLANG=/bin/clang-14 && ./perf trace -e write --max-events=1
...
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; intsys_enter(struct syscall_enter_args *args)@ augmented_raw_syscalls.bpf.c:509perf $ make CLANG=/bin/clang-15 && ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0perf $ make CLANG=/bin/clang-16 && ./perf trace -e write --max-events=1
0.000 ( 0.010 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,count: 8) =
Anyway thank you so much, I'll try to reproduce it on Ubuntu, like you mentioned.
Thanks,
HowardQuoted 1 line
Quoted 1 line
Hello Alan,
Quoted 55 lines · Alan Maguire
Quoted 10 lines · Howard Chu
Hello,
I got some good news:
Not being able to pass the BPF verifier is solely (or bi-solely) caused by these two functions:
SEC("tp/syscalls/sys_enter_rename")
intsys_enter_rename(struct syscall_enter_args *args)Quoted 9 lines
SEC("tp/syscalls/sys_enter_renameat2")
intsys_enter_renameat2(struct syscall_enter_args *args)The problem with it is double-string augmentation, but I haven't figured out how to solve it. Can you please take a look please, Arnaldo? Now I just commented these two BPF functions and perf trace runs no problem, built by clang that's older than clang 16, for example clang 15.0.7.
hi Howard,
I could be wrong - and I don't know this code at all - but I _think_ the problem is here:
oldpath_len = augmented_arg__read_str(&augmented_args->arg, oldpath_arg, sizeof(augmented_args->arg.value)); augmented_args->arg.size = PERF_ALIGN(oldpath_len + 1, sizeof(u64)); len += augmented_args->arg.size; struct augmented_arg *arg2 = (void *)&augmented_args->arg.value+
augmented_args->arg.size;
newpath_len= augmented_arg__read_str(arg2,newpath_arg,sizeof(augmented_args->arg.value));Notice that these strings share the
augmented_args->arg.valuefor storage, but in the second case (reading arg2) we have addedaugmented_args->arg.sizeto the offset where we store the second string. However at the same time we have also toldaugmented_arg__read_str()that it has
sizeof(augmented_args->arg.value)...to work with as
arg_len, where in reality we should have specified
sizeof(augmented_args->arg.value)-augmented_args->arg.size...and ensured that after doing that subtraction, there is still space to work with (i.e. the above is > 0).
Quoted 6 lines
I _think_ the verifier is likely complaining that we can write past the end of
augmented_args->arg.value, does that sound right?Fixing the
arg_lenparameter for cases where we record more than one augmented string by reducingarg_lenavailable should solve this if so. Hope this helps,
Thanks for the help, but unfortunately it didn't work... But that's a valid analysis, I had the same idea as you, but I haven't been able to solve it. I'll give it more tries.
Thanks,
Howard
Quoted 322 lines
Alan
Quoted 318 lines
perf $ clang -v
Ubuntu clang version 15.0.7
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Selected GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=10.000 ( 0.008 ms): gmain/2173 write(fd: 4, buf: \1\0\0\0\0\0\0\0,count: 8) =
But plot twist, it won't build on clang-14
perf $ clang -v
Ubuntu clang version 14.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Selected GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; intsys_enter(struct syscall_enter_args *args)@ augmented_raw_syscalls.bpf.c:509
0: (bf) r7 = r1 ; R1=ctx() R7_w=ctx()This time the problem is my code in BTF general collector:
;
payload_offset+= written; @ augmented_raw_syscalls.bpf.c:497I'll resolve this.
Thanks,
HowardQuoted 276 lines · Howard Chu
Hello,
I'm able to reproduce it on Ubuntu 24.
perf $ cat
/etc/os-release
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=noble
LOGO=ubuntu-logoperf $ uname -r
6.11.0-061100-genericperf $ git log
commit4c1af9bf97eb56d069421c3233ce61608458d5c8(HEAD)Author: Arnaldo Carvalho de Melo <[email protected]> Date: Tue Sep 10 13:54:23 2024 -0300perf trace: If a syscall arg is marked as 'const', assume it is
coming _from_ userspaceperf $ clang -v
Ubuntu clang version 15.0.7
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Selected GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Candidate multilib: .;@m64
Selected multilib: .;@m64perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; intsys_enter_rename(struct syscall_enter_args *args)@ augmented_raw_syscalls.bpf.c:275With clang-16:
perf $ clang -v
Ubuntu clang version 16.0.6 (23ubuntu4)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Selected GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Candidate multilib: .;@m64
Selected multilib: .;@m64perf $ ./perf trace -e write --max-events=1
0.000 ( 0.021 ms): sudo/4741 write(fd: 8, buf:\27[?25l\27[37m\27[48;5;96m\27[H[trace] , count: 205) =
Quoted 2 lines
I'll try to build it and see. But Ubuntu 22 only has clang 11 to 15 so making 16 the minimum could be an issue.
Yes, I think disabling any distro that doesn't come with clang 16 (released in 2023) is not a good idea. I'll try to tame the BPF verifier.
Thanks,
HowardQuoted 199 lines · Howard Chu
Hello James and Arnaldo,
Quoted 54 lines · Arnaldo Carvalho de Melo
Quoted 50 lines · James Clark
Quoted 45 lines · Arnaldo Carvalho de Melo
Quoted 32 lines · James Clark
Quoted 31 lines · Arnaldo Carvalho de Melo
Howard reported problems using perf features that use BPF:
perf $ clang -v
Debian clang version 15.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/12Selected GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/12Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0But it works with:
perf $ clang -v Debian clang version 16.0.6 (15~deb12u1) Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /bin Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64 Selected multilib: .;@m64 perf $ ./perf trace -e write --max-events=1 0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8) = 8 (kworker/0:0-eve) perf $Quoted 6 lines
Quoted 3 lines
So lets make that the required version, if you happen to have a slightly older version where this work, please report so that we can adjust the minimum required version.
I wasn't able to reproduce the issue with either of these versions. But I suppose it could be an issue with only 15.0.6.
Interesting, that complicates things, probably the best way then is to try to build it, if it fails, mention that 15.0.6 is known to be problematic and suggest working versions?
- Arnaldo
Quoted 37 lines
Quoted 3 lines
I still wasn't able to reproduce it with 15.0.6. And I double checked with V=1 that the build was using the right clang. I suppose it could be a build configuration issue, or maybe with a different kernel version?
Howard? If not reproducible we can revert it.
- Arnaldo
Quoted 27 lines
$ uname --kernel-release 6.8.0-76060800daily20240311-generic$ ../../llvm-project/build/bin/clang -v clang version 15.0.6 (https://github.com/llvm/llvm-project.git 088f33605d8a61ff519c580a71b1dd57d16a03f8) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /home/james/workspace/linux/linux/../../llvm-project/build/bin Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11 Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64 Selected multilib: .;@m64$ git log commit 003265bb6f028d7bcd7cbd92d6ba2b4e26382796 (perf-tools-next/perf-tools-next)$ make O=../build/local/ CLANG=../../llvm-project/build/bin/clang -C \ tools/perf$ perf trace -e write --max-events=10.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf: 0x7fffa102d9b0, count: 8) = 8Thanks for the effort! But this output seems to be unaugmented? I think you are testing the perf/perf-tools-next branch that's slightly older (perf mem: Fix the wrong reference in
parse_record_events()003265bb6f028d7bcd7cbd92d6ba2b4e26382796), and I think that doesn't have the new perf trace feature yet...That's why the perf trace output looks like:
Quoted 4 lines
Quoted 4 lines
$ perf trace -e write --max-events=10.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf: 0x7fffa102d9b0, count: 8)With the new buffer augmentation it should look like this:
perf $ ./perf trace -e write --max-events=1
0.000 ( 0.011 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,count: 8) =
The new perf trace can be found on
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/
perf-tools-nextperf $ git log
commit1de5b5dcb8353f36581c963df2d359a5f151a0be(HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next)Also, if you want to test the clang, you may want to checkout to the commit before the clang-16 guard (otherwise you can't test different clang version).
perf $ git log --oneline
1de5b5dcb835(HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next) perf
trace: Mark the 'head' arg in theset_robust_listsyscall as coming from user space0c1019e3463bperf trace: Mark the 'rseq' arg in the rseq syscall as coming from user spaceedf3ce0ed38eperf env: Find correct branch counter info on hybrid9953807c9e01perf evlist: Print hint for groupeb9b9a6f5ab3tools: Drop nonsensical -O689c0a55e550eperf pmu: To info addevent_type_descf08cc258431dperf evsel: Add accessor fortool_event925320737ae2perf pmus: Fake PMU clean upd3d5c1a00fcdperf list: Avoid potential out of bounds memory read4ae354d73a8eperf help: Fix a typo ("bellow")74298dd8acb8perf ftrace: Detect whether ftrace is enabled on system83420d5f5863perf test shellprobe_vfs_getname: Remove extraneous '=' from probe line number regex9327f0ecad48perf build: Require at least clang 16.0.6 to build BPF skeletons. **********[[[[[[[[[[[[!!!!(This is the clang guard)]]]]]]]]]]*****4c1af9bf97ebperf trace: If a syscall arg is marked as 'const', assume it is coming _from_ userspacee37b315c17dfperf parse-events: Remove duplicated include inparse-events.cMy machine is debian 12, and I cannot build perf trace with the new augmentation feature on clang-13, clang-14, and clang-15.
perf $ uname -r
6.11.0-061100rc7-genericperf $ git log
commit4c1af9bf97eb56d069421c3233ce61608458d5c8(HEAD)perf $ make CLANG=/bin/clang-13 && ./perf trace -e write --max-events=1
...
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; intsys_enter(struct syscall_enter_args *args)@ augmented_raw_syscalls.bpf.c:509
0: (bf) r7 = r1 ; R1=ctx() R7_w=ctx()perf $ make CLANG=/bin/clang-14 && ./perf trace -e write --max-events=1
...
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; intsys_enter(struct syscall_enter_args *args)@ augmented_raw_syscalls.bpf.c:509perf $ make CLANG=/bin/clang-15 && ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0perf $ make CLANG=/bin/clang-16 && ./perf trace -e write --max-events=1
0.000 ( 0.010 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,count: 8) =
Anyway thank you so much, I'll try to reproduce it on Ubuntu, like you mentioned.
Thanks,
HowardQuoted 1 line
Quoted 1 line
Hi Alan, Arnaldo and James,
This problem was solved in [PATCH 0/2] perf trace: Fix support for the new BPF feature in clang 12 (Link:
https://lore.kernel.org/linux-perf-users/[email protected]/T/#t),
sorry I forgot to cc you two.
Alan's thought was correct. Thank you so much! :)
Thanks,
Howard
Quoted 396 lines · Howard Chu
Hello Alan,
Quoted 55 lines · Alan Maguire
Quoted 10 lines · Howard Chu
Hello,
I got some good news:
Not being able to pass the BPF verifier is solely (or bi-solely) caused by these two functions:
SEC("tp/syscalls/sys_enter_rename")
intsys_enter_rename(struct syscall_enter_args *args)Quoted 9 lines
SEC("tp/syscalls/sys_enter_renameat2")
intsys_enter_renameat2(struct syscall_enter_args *args)The problem with it is double-string augmentation, but I haven't figured out how to solve it. Can you please take a look please, Arnaldo? Now I just commented these two BPF functions and perf trace runs no problem, built by clang that's older than clang 16, for example clang 15.0.7.
hi Howard,
I could be wrong - and I don't know this code at all - but I _think_ the problem is here:
oldpath_len = augmented_arg__read_str(&augmented_args->arg, oldpath_arg, sizeof(augmented_args->arg.value)); augmented_args->arg.size = PERF_ALIGN(oldpath_len + 1, sizeof(u64)); len += augmented_args->arg.size; struct augmented_arg *arg2 = (void *)&augmented_args->arg.value+
augmented_args->arg.size;
newpath_len= augmented_arg__read_str(arg2,newpath_arg,sizeof(augmented_args->arg.value));Notice that these strings share the
augmented_args->arg.valuefor storage, but in the second case (reading arg2) we have addedaugmented_args->arg.sizeto the offset where we store the second string. However at the same time we have also toldaugmented_arg__read_str()that it has
sizeof(augmented_args->arg.value)...to work with as
arg_len, where in reality we should have specified
sizeof(augmented_args->arg.value)-augmented_args->arg.size...and ensured that after doing that subtraction, there is still space to work with (i.e. the above is > 0).
Quoted 6 lines
I _think_ the verifier is likely complaining that we can write past the end of
augmented_args->arg.value, does that sound right?Fixing the
arg_lenparameter for cases where we record more than one augmented string by reducingarg_lenavailable should solve this if so. Hope this helps,Thanks for the help, but unfortunately it didn't work... But that's a valid analysis, I had the same idea as you, but I haven't been able to solve it. I'll give it more tries.
Thanks,
HowardQuoted 322 lines
Alan
Quoted 318 lines
perf $ clang -v
Ubuntu clang version 15.0.7
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Selected GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=10.000 ( 0.008 ms): gmain/2173 write(fd: 4, buf: \1\0\0\0\0\0\0\0,count: 8) =
But plot twist, it won't build on clang-14
perf $ clang -v
Ubuntu clang version 14.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Selected GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; intsys_enter(struct syscall_enter_args *args)@ augmented_raw_syscalls.bpf.c:509
0: (bf) r7 = r1 ; R1=ctx() R7_w=ctx()This time the problem is my code in BTF general collector:
;
payload_offset+= written; @ augmented_raw_syscalls.bpf.c:497I'll resolve this.
Thanks,
HowardQuoted 276 lines · Howard Chu
Hello,
I'm able to reproduce it on Ubuntu 24.
perf $ cat
/etc/os-release
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=noble
LOGO=ubuntu-logoperf $ uname -r
6.11.0-061100-genericperf $ git log
commit4c1af9bf97eb56d069421c3233ce61608458d5c8(HEAD)Author: Arnaldo Carvalho de Melo <[email protected]> Date: Tue Sep 10 13:54:23 2024 -0300perf trace: If a syscall arg is marked as 'const', assume it is
coming _from_ userspaceperf $ clang -v
Ubuntu clang version 15.0.7
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Selected GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Candidate multilib: .;@m64
Selected multilib: .;@m64perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; intsys_enter_rename(struct syscall_enter_args *args)@ augmented_raw_syscalls.bpf.c:275With clang-16:
perf $ clang -v
Ubuntu clang version 16.0.6 (23ubuntu4)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Selected GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/13Candidate multilib: .;@m64
Selected multilib: .;@m64perf $ ./perf trace -e write --max-events=1
0.000 ( 0.021 ms): sudo/4741 write(fd: 8, buf:\27[?25l\27[37m\27[48;5;96m\27[H[trace] , count: 205) =
Quoted 2 lines
I'll try to build it and see. But Ubuntu 22 only has clang 11 to 15 so making 16 the minimum could be an issue.
Yes, I think disabling any distro that doesn't come with clang 16 (released in 2023) is not a good idea. I'll try to tame the BPF verifier.
Thanks,
HowardQuoted 199 lines · Howard Chu
Hello James and Arnaldo,
Quoted 54 lines · Arnaldo Carvalho de Melo
Quoted 50 lines · James Clark
Quoted 45 lines · Arnaldo Carvalho de Melo
Quoted 32 lines · James Clark
Quoted 31 lines · Arnaldo Carvalho de Melo
Howard reported problems using perf features that use BPF:
perf $ clang -v
Debian clang version 15.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/12Selected GCC installation:/bin/../lib/gcc/x86_64-linux-gnu/12Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0But it works with:
perf $ clang -v Debian clang version 16.0.6 (15~deb12u1) Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /bin Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64 Selected multilib: .;@m64 perf $ ./perf trace -e write --max-events=1 0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8) = 8 (kworker/0:0-eve) perf $Quoted 6 lines
Quoted 3 lines
So lets make that the required version, if you happen to have a slightly older version where this work, please report so that we can adjust the minimum required version.
I wasn't able to reproduce the issue with either of these versions. But I suppose it could be an issue with only 15.0.6.
Interesting, that complicates things, probably the best way then is to try to build it, if it fails, mention that 15.0.6 is known to be problematic and suggest working versions?
- Arnaldo
Quoted 37 lines
Quoted 3 lines
I still wasn't able to reproduce it with 15.0.6. And I double checked with V=1 that the build was using the right clang. I suppose it could be a build configuration issue, or maybe with a different kernel version?
Howard? If not reproducible we can revert it.
- Arnaldo
Quoted 27 lines
$ uname --kernel-release 6.8.0-76060800daily20240311-generic$ ../../llvm-project/build/bin/clang -v clang version 15.0.6 (https://github.com/llvm/llvm-project.git 088f33605d8a61ff519c580a71b1dd57d16a03f8) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /home/james/workspace/linux/linux/../../llvm-project/build/bin Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11 Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64 Selected multilib: .;@m64$ git log commit 003265bb6f028d7bcd7cbd92d6ba2b4e26382796 (perf-tools-next/perf-tools-next)$ make O=../build/local/ CLANG=../../llvm-project/build/bin/clang -C \ tools/perf$ perf trace -e write --max-events=10.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf: 0x7fffa102d9b0, count: 8) = 8Thanks for the effort! But this output seems to be unaugmented? I think you are testing the perf/perf-tools-next branch that's slightly older (perf mem: Fix the wrong reference in
parse_record_events()003265bb6f028d7bcd7cbd92d6ba2b4e26382796), and I think that doesn't have the new perf trace feature yet...That's why the perf trace output looks like:
Quoted 4 lines
Quoted 4 lines
$ perf trace -e write --max-events=10.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf: 0x7fffa102d9b0, count: 8)With the new buffer augmentation it should look like this:
perf $ ./perf trace -e write --max-events=1
0.000 ( 0.011 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,count: 8) =
The new perf trace can be found on
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/
perf-tools-nextperf $ git log
commit1de5b5dcb8353f36581c963df2d359a5f151a0be(HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next)Also, if you want to test the clang, you may want to checkout to the commit before the clang-16 guard (otherwise you can't test different clang version).
perf $ git log --oneline
1de5b5dcb835(HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next) perf
trace: Mark the 'head' arg in theset_robust_listsyscall as coming from user space0c1019e3463bperf trace: Mark the 'rseq' arg in the rseq syscall as coming from user spaceedf3ce0ed38eperf env: Find correct branch counter info on hybrid9953807c9e01perf evlist: Print hint for groupeb9b9a6f5ab3tools: Drop nonsensical -O689c0a55e550eperf pmu: To info addevent_type_descf08cc258431dperf evsel: Add accessor fortool_event925320737ae2perf pmus: Fake PMU clean upd3d5c1a00fcdperf list: Avoid potential out of bounds memory read4ae354d73a8eperf help: Fix a typo ("bellow")74298dd8acb8perf ftrace: Detect whether ftrace is enabled on system83420d5f5863perf test shellprobe_vfs_getname: Remove extraneous '=' from probe line number regex9327f0ecad48perf build: Require at least clang 16.0.6 to build BPF skeletons. **********[[[[[[[[[[[[!!!!(This is the clang guard)]]]]]]]]]]*****4c1af9bf97ebperf trace: If a syscall arg is marked as 'const', assume it is coming _from_ userspacee37b315c17dfperf parse-events: Remove duplicated include inparse-events.cMy machine is debian 12, and I cannot build perf trace with the new augmentation feature on clang-13, clang-14, and clang-15.
perf $ uname -r
6.11.0-061100rc7-genericperf $ git log
commit4c1af9bf97eb56d069421c3233ce61608458d5c8(HEAD)perf $ make CLANG=/bin/clang-13 && ./perf trace -e write --max-events=1
...
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; intsys_enter(struct syscall_enter_args *args)@ augmented_raw_syscalls.bpf.c:509
0: (bf) r7 = r1 ; R1=ctx() R7_w=ctx()perf $ make CLANG=/bin/clang-14 && ./perf trace -e write --max-events=1
...
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; intsys_enter(struct syscall_enter_args *args)@ augmented_raw_syscalls.bpf.c:509perf $ make CLANG=/bin/clang-15 && ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0perf $ make CLANG=/bin/clang-16 && ./perf trace -e write --max-events=1
0.000 ( 0.010 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,count: 8) =
Anyway thank you so much, I'll try to reproduce it on Ubuntu, like you mentioned.
Thanks,
HowardQuoted 1 line
Quoted 1 line
Namhyung Kim
maintainerQuoted 8 lines · Howard Chu
Hi Alan, Arnaldo and James,
This problem was solved in [PATCH 0/2] perf trace: Fix support for the new BPF feature in clang 12 (Link:
https://lore.kernel.org/linux-perf-users/[email protected]/T/#t),
sorry I forgot to cc you two.Alan's thought was correct. Thank you so much! :)
It'd be great if any of you can test this change. Now I only have machines with clang 16.
Thanks,
Namhyung
Quoted 14 lines · Arnaldo Carvalho de Melo
Quoted 12 lines · Namhyung Kim
Quoted 5 lines · Howard Chu
Hi Alan, Arnaldo and James,
This problem was solved in [PATCH 0/2] perf trace: Fix support for the new BPF feature in clang 12 (Link:
https://lore.kernel.org/linux-perf-users/[email protected]/T/#t
),Quoted 3 lines
sorry I forgot to cc you two.
Alan's thought was correct. Thank you so much! :)
Quoted 14 lines
Quoted 3 lines
It'd be great if any of you can test this change. Now I only have machines with clang 16.
I'll test this tomorrow.
- Arnaldo
Quoted 5 lines
Thanks,
Namhyung
Tested with clang 15:
$ sudo perf trace -e write --max-events=100 -- echo hello
0.000 ( 0.014 ms): echo/834165 write(fd: 1, buf: hello\10, count: 6)
=
Unrelated to this change, I noticed that with older clangs or with BUILD_BPF_SKEL=0 that commit b257fac12f38 ("perf trace: Pretty print buffer data") changes the buffer address to print nothing, and the '6' return value is missing. Not sure if this was intended or not:
$ sudo perf trace -e write --max-events=100 -- echo hello
Before:
0.000 ( 0.009 ms): echo/772951 write(fd: 1, buf: 0x58c415257440,
count: 6) = 6
After:
0.000 ( 0.009 ms): echo/760370 write(fd: 1, buf: , count: 6)
=Thanks
James
James Clark
reviewerQuoted 32 lines · James Clark
Quoted 14 lines · Arnaldo Carvalho de Melo
Quoted 12 lines · Namhyung Kim
Quoted 5 lines · Howard Chu
Hi Alan, Arnaldo and James,
This problem was solved in [PATCH 0/2] perf trace: Fix support for the new BPF feature in clang 12 (Link:
https://lore.kernel.org/linux-perf-users/20241007051414.2995674-1- [email protected]/T/#t ),
Quoted 3 lines
sorry I forgot to cc you two.
Alan's thought was correct. Thank you so much! :)
Quoted 14 lines
Quoted 3 lines
It'd be great if any of you can test this change. Now I only have machines with clang 16.
I'll test this tomorrow.
- Arnaldo
Quoted 5 lines
Thanks,
Namhyung
Quoted 9 lines
Tested with clang 15:
$ sudo perf trace -e write --max-events=100 -- echo hello0.000 ( 0.014 ms): echo/834165
write(fd: 1, buf: hello\10, count: 6)=Tested-by James Clark <[email protected]>
Should have left this tag on the actual patch. Will re-post it there.
Namhyung Kim
maintainerQuoted 40 lines · James Clark
Quoted 37 lines · James Clark
Quoted 15 lines · Arnaldo Carvalho de Melo
Quoted 13 lines · Namhyung Kim
Quoted 5 lines · Howard Chu
Hi Alan, Arnaldo and James,
This problem was solved in [PATCH 0/2] perf trace: Fix support for the new BPF feature in clang 12 (Link:
https://lore.kernel.org/linux-perf-users/20241007051414.2995674-1-
[email protected]/T/#t
),Quoted 3 lines
sorry I forgot to cc you two.
Alan's thought was correct. Thank you so much! :)
Quoted 14 lines
Quoted 3 lines
It'd be great if any of you can test this change. Now I only have machines with clang 16.
I'll test this tomorrow.
- Arnaldo
Quoted 5 lines
Thanks,
NamhyungTested with clang 15:
$ sudo perf trace -e write --max-events=100 -- echo hello
Quoted 7 lines
Quoted 5 lines
0.000 ( 0.014 ms): echo/834165
write(fd: 1, buf: hello\10, count: 6)=Tested-by James Clark <[email protected]>Should have left this tag on the actual patch. Will re-post it there.
Right, thanks for taking care of this!
Namhyung
Arnaldo Carvalho de Melo
maintainerQuoted 48 lines · James Clark
Quoted 14 lines · Arnaldo Carvalho de Melo
Quoted 12 lines · Namhyung Kim
Quoted 5 lines · Howard Chu
Hi Alan, Arnaldo and James,
This problem was solved in [PATCH 0/2] perf trace: Fix support for the new BPF feature in clang 12 (Link:
https://lore.kernel.org/linux-perf-users/[email protected]/T/#t
),Quoted 3 lines
sorry I forgot to cc you two.
Alan's thought was correct. Thank you so much! :)
Quoted 14 lines
Quoted 3 lines
It'd be great if any of you can test this change. Now I only have machines with clang 16.
I'll test this tomorrow.
- Arnaldo
Quoted 5 lines
Thanks,
NamhyungTested with clang 15:
$ sudo perf trace -e write --max-events=100 -- echo hello0.000 ( 0.014 ms): echo/834165
write(fd: 1, buf: hello\10, count: 6)=Tested-by James Clark <[email protected]>Unrelated to this change, I noticed that with older clangs or with BUILD_BPF_SKEL=0 that commit
b257fac12f38("perf trace: Pretty print buffer data") changes the buffer address to print nothing, and the '6' return value is missing. Not sure if this was intended or not:
Quoted 9 lines
$ sudo perf trace -e write --max-events=100 -- echo helloBefore:
0.000 ( 0.009 ms): echo/772951 write(fd: 1, buf:0x58c415257440,
count: 6) = 6After:
0.000 ( 0.009 ms): echo/760370 write(fd: 1, buf: , count: 6)
I noticed this with fedora:40, and bisected it to:
⬢[acme@toolbox perf-tools]$ git bisect good b257fac12f38d7f503b932313d704cee21092350 is the first bad commit commit b257fac12f38d7f503b932313d704cee21092350
Author: Howard Chu <[email protected]> Date: Sun Aug 25 00:33:19 2024 +0800
perf trace: Pretty print buffer data
Define TRACE_AUG_MAX_BUF in trace_augment.h data, which is the maximum buffer size we can augment. BPF will include this header too.
Print buffer in a way that's different than just printing a string, we print all the control characters in \digits (such as \0 for null, and \10 for newline, LF).
For character that has a bigger value than 127, we print the digits instead of the character itself as well.
Committer notes:
Simplified the buffer scnprintf to avoid using multiple buffers as discussed in the patch review thread.
We can't really all 'buf' args to SCA_BUF as we're collecting so far just on the sys_enter path, so we would be printing the previous 'read' arg buffer contents, not what the kernel puts there.
So instead of:
static int syscall_fmt__cmp(const void *name, const void *fmtp)
@@ -1987,8 +1989,6 @@ syscall_arg_fmt__init_array(struct syscall_arg_fmt *arg, struct tep_format_field
- else if (strstr(field->type, "char *") && strstr(field->name, "buf"))
- arg->scnprintf = SCA_BUF;
Do:
static const struct syscall_fmt syscall_fmts[] = {
+ { .name = "write", .errpid = true,
+ .arg = { [1] = { .scnprintf = SCA_BUF /* buf */, from_user = true, }, }, },
Just did a:
⬢[acme@toolbox perf-tools]$ git checkout b257fac12f38d7f503b932313d704cee21092350 Updating files: 100% (12326/12326), done.
Note: switching to 'b257fac12f38d7f503b932313d704cee21092350'.
You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch. <SNIP>
HEAD is now at b257fac12f38d7f5 perf trace: Pretty print buffer
make clean + rebuild and the returns are gone, if I do:
⬢[acme@toolbox perf-tools]$ git checkout b257fac12f38d7f503b932313d704cee21092350^ Previous HEAD position was b257fac12f38d7f5 perf trace: Pretty print buffer data HEAD is now at cb32035214b9a09d perf trace: Pretty print struct data ⬢[acme@toolbox perf-tools]$
clean + rebuild instead I get those returns back:
Now staring at it...
- Arnaldo