Skip to contentSkip to conversation

perf build: Require at least clang 16.0.6 to build BPF skeletons

PATCH In review Arnaldo Carvalho de Melo 5 participants 16 messages Sep 11, 2024 – Oct 11, 2024 2 messages not in the archive perf events

lore
Share & tools
lore

Arnaldo Carvalho de Melo

author maintainer
#0 lore raw

PATCH 1/1 perf build: Require at least clang 16.0.6 to build BPF skeletons

Howard reported problems using perf features that use BPF:

perf $ clang -v
Debian clang version 15.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0

But it works with:

  perf $ clang -v
  Debian clang version 16.0.6 (15~deb12u1)
  Target: x86_64-pc-linux-gnu
  Thread model: posix
  InstalledDir: /bin
  Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
  Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
  Candidate multilib: .;@m64
  Selected multilib: .;@m64
  perf $ ./perf trace -e write --max-events=1
       0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8)                         = 8 (kworker/0:0-eve)
  perf $

So lets make that the required version, if you happen to have a slightly older version where this work, please report so that we can adjust the minimum required version.

Cc Ian Rogers <[email protected]>
Cc Adrian Hunter <[email protected]>
Cc Alan Maguire <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Kan Liang <[email protected]>
Cc Namhyung Kim <[email protected]>
---
 tools/perf/Makefile.config | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/Makefile.config +2 -2
index 37e3eee2986e580a..4c1f75e326e2c1d2 100644
@@ -705,8 +705,8 @@ ifeq ($(BUILD_BPF_SKEL),1)
     BUILD_BPF_SKEL := 0
   else
     CLANG_VERSION := $(shell $(CLANG) --version | head -1 | sed 's/.*clang version \([[:digit:]]\+.[[:digit:]]\+.[[:digit:]]\+\).*/\1/g')
-    ifeq ($(call version-lt3,$(CLANG_VERSION),12.0.1),1)
-      $(warning Warning: Disabled BPF skeletons as reliable BTF generation needs at least $(CLANG) version 12.0.1)
+    ifeq ($(call version-lt3,$(CLANG_VERSION),16.0.6),1)
+      $(warning Warning: Disabled BPF skeletons as at least $(CLANG) version 16.0.6 is reported to be a working setup with the current of BPF based perf features)
       BUILD_BPF_SKEL := 0
     endif
   endif
Signature · 1 line
2.46.0
reviewer
#1 lore raw
Quoted 31 lines · Arnaldo Carvalho de Melo

Howard reported problems using perf features that use BPF:

perf $ clang -v
Debian clang version 15.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0

But it works with:

   perf $ clang -v
   Debian clang version 16.0.6 (15~deb12u1)
   Target: x86_64-pc-linux-gnu
   Thread model: posix
   InstalledDir: /bin
   Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
   Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
   Candidate multilib: .;@m64
   Selected multilib: .;@m64
   perf $ ./perf trace -e write --max-events=1
        0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8)                         = 8 (kworker/0:0-eve)
   perf $
Quoted 4 lines

So lets make that the required version, if you happen to have a slightly older version where this work, please report so that we can adjust the minimum required version.

I wasn't able to reproduce the issue with either of these versions. But I suppose it could be an issue with only 15.0.6.

$ clang -v
Ubuntu clang version 15.0.7
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/11
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12
Candidate multilib: .;@m64
Selected multilib: .;@m64

Ubuntu clang version 14.0.0-1ubuntu1.1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/11 Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64
Selected multilib: .;@m64

author maintainer
#2 lore raw
Quoted 32 lines · James Clark
Quoted 31 lines · Arnaldo Carvalho de Melo

Howard reported problems using perf features that use BPF:

perf $ clang -v
Debian clang version 15.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0

But it works with:

   perf $ clang -v
   Debian clang version 16.0.6 (15~deb12u1)
   Target: x86_64-pc-linux-gnu
   Thread model: posix
   InstalledDir: /bin
   Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
   Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
   Candidate multilib: .;@m64
   Selected multilib: .;@m64
   perf $ ./perf trace -e write --max-events=1
        0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8)                         = 8 (kworker/0:0-eve)
   perf $
Quoted 6 lines
Quoted 3 lines

So lets make that the required version, if you happen to have a slightly older version where this work, please report so that we can adjust the minimum required version.

I wasn't able to reproduce the issue with either of these versions. But I suppose it could be an issue with only 15.0.6.

Interesting, that complicates things, probably the best way then is to try to build it, if it fails, mention that 15.0.6 is known to be problematic and suggest working versions?

- Arnaldo

Quoted 21 lines
$ clang -v
Ubuntu clang version 15.0.7
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/11
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12
Candidate multilib: .;@m64
Selected multilib: .;@m64

Ubuntu clang version 14.0.0-1ubuntu1.1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/11 Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64
Selected multilib: .;@m64

reviewer
#3 lore raw
Quoted 37 lines · Arnaldo Carvalho de Melo
Quoted 35 lines · James Clark
Quoted 34 lines · Arnaldo Carvalho de Melo

Howard reported problems using perf features that use BPF:

perf $ clang -v
Debian clang version 15.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0

But it works with:

    perf $ clang -v
    Debian clang version 16.0.6 (15~deb12u1)
    Target: x86_64-pc-linux-gnu
    Thread model: posix
    InstalledDir: /bin
    Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
    Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
    Candidate multilib: .;@m64
    Selected multilib: .;@m64
    perf $ ./perf trace -e write --max-events=1
         0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8)                         = 8 (kworker/0:0-eve)
    perf $

So lets make that the required version, if you happen to have a slightly older version where this work, please report so that we can adjust the minimum required version.

Quoted 9 lines
Quoted 2 lines

I wasn't able to reproduce the issue with either of these versions. But I suppose it could be an issue with only 15.0.6.

Interesting, that complicates things, probably the best way then is to try to build it, if it fails, mention that 15.0.6 is known to be problematic and suggest working versions?

- Arnaldo

I'll try to build it and see. But Ubuntu 22 only has clang 11 to 15 so making 16 the minimum could be an issue.

reviewer
#4 lore raw
In reply to Arnaldo Carvalho de Melo
Quoted 37 lines · Arnaldo Carvalho de Melo
Quoted 35 lines · James Clark
Quoted 34 lines · Arnaldo Carvalho de Melo

Howard reported problems using perf features that use BPF:

perf $ clang -v
Debian clang version 15.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0

But it works with:

    perf $ clang -v
    Debian clang version 16.0.6 (15~deb12u1)
    Target: x86_64-pc-linux-gnu
    Thread model: posix
    InstalledDir: /bin
    Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
    Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
    Candidate multilib: .;@m64
    Selected multilib: .;@m64
    perf $ ./perf trace -e write --max-events=1
         0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8)                         = 8 (kworker/0:0-eve)
    perf $

So lets make that the required version, if you happen to have a slightly older version where this work, please report so that we can adjust the minimum required version.

Quoted 8 lines
Quoted 2 lines

I wasn't able to reproduce the issue with either of these versions. But I suppose it could be an issue with only 15.0.6.

Interesting, that complicates things, probably the best way then is to try to build it, if it fails, mention that 15.0.6 is known to be problematic and suggest working versions?

- Arnaldo

I still wasn't able to reproduce it with 15.0.6. And I double checked with V=1 that the build was using the right clang. I suppose it could be a build configuration issue, or maybe with a different kernel version?

$  uname --kernel-release
6.8.0-76060800daily20240311-generic
$ ../../llvm-project/build/bin/clang -v
clang version 15.0.6 (https://github.com/llvm/llvm-project.git 088f33605d8a61ff519c580a71b1dd57d16a03f8)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/james/workspace/linux/linux/../../llvm-project/build/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
Candidate multilib: .;@m64
Selected multilib: .;@m64
$ git log
commit 003265bb6f028d7bcd7cbd92d6ba2b4e26382796 (perf-tools-next/perf-tools-next)
$ make O=../build/local/ CLANG=../../llvm-project/build/bin/clang -C \
  tools/perf
$ perf trace -e write --max-events=1
     0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
       0x7fffa102d9b0, count: 8)                           = 8

A message that is not in the archive (sent off-list or not yet received): CA+JHD936J-q0-7LANQ3aW2G-PEmFP8PnXQ-TF-AMs9MtrCqfew@mail.gmail.com

#6 lore raw

Hello James and Arnaldo,

Quoted 54 lines · Arnaldo Carvalho de Melo
Quoted 50 lines · James Clark
Quoted 45 lines · Arnaldo Carvalho de Melo
Quoted 32 lines · James Clark
Quoted 31 lines · Arnaldo Carvalho de Melo

Howard reported problems using perf features that use BPF:

perf $ clang -v
Debian clang version 15.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0

But it works with:

    perf $ clang -v
    Debian clang version 16.0.6 (15~deb12u1)
    Target: x86_64-pc-linux-gnu
    Thread model: posix
    InstalledDir: /bin
    Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
    Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
    Candidate multilib: .;@m64
    Selected multilib: .;@m64
    perf $ ./perf trace -e write --max-events=1
         0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8)                         = 8 (kworker/0:0-eve)
    perf $
Quoted 6 lines
Quoted 3 lines

So lets make that the required version, if you happen to have a slightly older version where this work, please report so that we can adjust the minimum required version.

I wasn't able to reproduce the issue with either of these versions. But I suppose it could be an issue with only 15.0.6.

Interesting, that complicates things, probably the best way then is to try to build it, if it fails, mention that 15.0.6 is known to be problematic and suggest working versions?

- Arnaldo

Quoted 37 lines
Quoted 3 lines

I still wasn't able to reproduce it with 15.0.6. And I double checked with V=1 that the build was using the right clang. I suppose it could be a build configuration issue, or maybe with a different kernel version?

Howard? If not reproducible we can revert it.

- Arnaldo

Quoted 27 lines
$  uname --kernel-release
6.8.0-76060800daily20240311-generic
$ ../../llvm-project/build/bin/clang -v
clang version 15.0.6 (https://github.com/llvm/llvm-project.git
088f33605d8a61ff519c580a71b1dd57d16a03f8)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/james/workspace/linux/linux/../../llvm-project/build/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
Candidate multilib: .;@m64
Selected multilib: .;@m64
$ git log
commit 003265bb6f028d7bcd7cbd92d6ba2b4e26382796
(perf-tools-next/perf-tools-next)
$ make O=../build/local/ CLANG=../../llvm-project/build/bin/clang -C \
   tools/perf
$ perf trace -e write --max-events=1
      0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
        0x7fffa102d9b0, count: 8)                           = 8

Thanks for the effort! But this output seems to be unaugmented? I think you are testing the perf/perf-tools-next branch that's slightly older (perf mem: Fix the wrong reference in parse_record_events() 003265bb6f028d7bcd7cbd92d6ba2b4e26382796), and I think that doesn't have the new perf trace feature yet...

That's why the perf trace output looks like:

Quoted 4 lines
Quoted 4 lines
$ perf trace -e write --max-events=1
      0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
        0x7fffa102d9b0, count: 8)

With the new buffer augmentation it should look like this:

perf $ ./perf trace -e write --max-events=1

     0.000 ( 0.011 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,

count: 8) =

The new perf trace can be found on
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/
perf-tools-next

perf $ git log
commit 1de5b5dcb8353f36581c963df2d359a5f151a0be (HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next)

Also, if you want to test the clang, you may want to checkout to the commit before the clang-16 guard (otherwise you can't test different clang version).

perf $ git log --oneline
1de5b5dcb835 (HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next) perf
trace: Mark the 'head' arg in the set_robust_list syscall as coming from user space
0c1019e3463b perf trace: Mark the 'rseq' arg in the rseq syscall as coming from user space
edf3ce0ed38e perf env: Find correct branch counter info on hybrid 9953807c9e01 perf evlist: Print hint for group
eb9b9a6f5ab3 tools: Drop nonsensical -O6
89c0a55e550e perf pmu: To info add event_type_desc
f08cc258431d perf evsel: Add accessor for tool_event 925320737ae2 perf pmus: Fake PMU clean up
d3d5c1a00fcd perf list: Avoid potential out of bounds memory read 4ae354d73a8e perf help: Fix a typo ("bellow")
74298dd8acb8 perf ftrace: Detect whether ftrace is enabled on system 83420d5f5863 perf test shell probe_vfs_getname: Remove extraneous '=' from probe line number regex
9327f0ecad48 perf build: Require at least clang 16.0.6 to build BPF skeletons. **********[[[[[[[[[[[[!!!!(This is the clang guard)]]]]]]]]]]*****
4c1af9bf97eb perf trace: If a syscall arg is marked as 'const', assume it is coming _from_ userspace
e37b315c17df perf parse-events: Remove duplicated include in parse-events.c

My machine is debian 12, and I cannot build perf trace with the new augmentation feature on clang-13, clang-14, and clang-15.

perf $ uname -r
6.11.0-061100rc7-generic

perf $ git log
commit 4c1af9bf97eb56d069421c3233ce61608458d5c8 (HEAD)

perf $ make CLANG=/bin/clang-13 && ./perf trace -e write --max-events=1
...
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter(struct syscall_enter_args *args) @ augmented_raw_syscalls.bpf.c:509
0: (bf) r7 = r1 ; R1=ctx() R7_w=ctx()

perf $ make CLANG=/bin/clang-14 && ./perf trace -e write --max-events=1
...
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter(struct syscall_enter_args *args) @ augmented_raw_syscalls.bpf.c:509

perf $ make CLANG=/bin/clang-15 && ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0

perf $ make CLANG=/bin/clang-16 && ./perf trace -e write --max-events=1

     0.000 ( 0.010 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,

count: 8) =

Anyway thank you so much, I'll try to reproduce it on Ubuntu, like you mentioned.

Thanks,
Howard

Quoted 1 line
Quoted 1 line
#7 lore raw

Hello,

I'm able to reproduce it on Ubuntu 24.

perf $ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

perf $ uname -r
6.11.0-061100-generic

perf $ git log
commit 4c1af9bf97eb56d069421c3233ce61608458d5c8 (HEAD)

Author: Arnaldo Carvalho de Melo <[email protected]>
Date:   Tue Sep 10 13:54:23 2024 -0300

perf trace: If a syscall arg is marked as 'const', assume it is
coming _from_ userspace

perf $ clang -v
Ubuntu clang version 15.0.7
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Candidate multilib: .;@m64
Selected multilib: .;@m64

perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter_rename(struct syscall_enter_args *args) @ augmented_raw_syscalls.bpf.c:275

With clang-16:

perf $ clang -v
Ubuntu clang version 16.0.6 (23ubuntu4)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Candidate multilib: .;@m64
Selected multilib: .;@m64

perf $ ./perf trace -e write --max-events=1

     0.000 ( 0.021 ms): sudo/4741 write(fd: 8, buf:

\27[?25l\27[37m\27[48;5;96m\27[H[trace] , count: 205) =

Quoted 2 lines

I'll try to build it and see. But Ubuntu 22 only has clang 11 to 15 so making 16 the minimum could be an issue.

Yes, I think disabling any distro that doesn't come with clang 16 (released in 2023) is not a good idea. I'll try to tame the BPF verifier.

Thanks,
Howard

Quoted 199 lines · Howard Chu

Hello James and Arnaldo,

Quoted 54 lines · Arnaldo Carvalho de Melo
Quoted 50 lines · James Clark
Quoted 45 lines · Arnaldo Carvalho de Melo
Quoted 32 lines · James Clark
Quoted 31 lines · Arnaldo Carvalho de Melo

Howard reported problems using perf features that use BPF:

perf $ clang -v
Debian clang version 15.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0

But it works with:

    perf $ clang -v
    Debian clang version 16.0.6 (15~deb12u1)
    Target: x86_64-pc-linux-gnu
    Thread model: posix
    InstalledDir: /bin
    Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
    Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
    Candidate multilib: .;@m64
    Selected multilib: .;@m64
    perf $ ./perf trace -e write --max-events=1
         0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8)                         = 8 (kworker/0:0-eve)
    perf $
Quoted 6 lines
Quoted 3 lines

So lets make that the required version, if you happen to have a slightly older version where this work, please report so that we can adjust the minimum required version.

I wasn't able to reproduce the issue with either of these versions. But I suppose it could be an issue with only 15.0.6.

Interesting, that complicates things, probably the best way then is to try to build it, if it fails, mention that 15.0.6 is known to be problematic and suggest working versions?

- Arnaldo

Quoted 37 lines
Quoted 3 lines

I still wasn't able to reproduce it with 15.0.6. And I double checked with V=1 that the build was using the right clang. I suppose it could be a build configuration issue, or maybe with a different kernel version?

Howard? If not reproducible we can revert it.

- Arnaldo

Quoted 27 lines
$  uname --kernel-release
6.8.0-76060800daily20240311-generic
$ ../../llvm-project/build/bin/clang -v
clang version 15.0.6 (https://github.com/llvm/llvm-project.git
088f33605d8a61ff519c580a71b1dd57d16a03f8)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/james/workspace/linux/linux/../../llvm-project/build/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
Candidate multilib: .;@m64
Selected multilib: .;@m64
$ git log
commit 003265bb6f028d7bcd7cbd92d6ba2b4e26382796
(perf-tools-next/perf-tools-next)
$ make O=../build/local/ CLANG=../../llvm-project/build/bin/clang -C \
   tools/perf
$ perf trace -e write --max-events=1
      0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
        0x7fffa102d9b0, count: 8)                           = 8

Thanks for the effort! But this output seems to be unaugmented? I think you are testing the perf/perf-tools-next branch that's slightly older (perf mem: Fix the wrong reference in parse_record_events() 003265bb6f028d7bcd7cbd92d6ba2b4e26382796), and I think that doesn't have the new perf trace feature yet...

That's why the perf trace output looks like:

Quoted 4 lines
Quoted 4 lines
$ perf trace -e write --max-events=1
      0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
        0x7fffa102d9b0, count: 8)

With the new buffer augmentation it should look like this:

perf $ ./perf trace -e write --max-events=1

     0.000 ( 0.011 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,

count: 8) =

The new perf trace can be found on
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/
perf-tools-next

perf $ git log
commit 1de5b5dcb8353f36581c963df2d359a5f151a0be (HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next)

Also, if you want to test the clang, you may want to checkout to the commit before the clang-16 guard (otherwise you can't test different clang version).

perf $ git log --oneline
1de5b5dcb835 (HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next) perf
trace: Mark the 'head' arg in the set_robust_list syscall as coming from user space
0c1019e3463b perf trace: Mark the 'rseq' arg in the rseq syscall as coming from user space
edf3ce0ed38e perf env: Find correct branch counter info on hybrid 9953807c9e01 perf evlist: Print hint for group
eb9b9a6f5ab3 tools: Drop nonsensical -O6
89c0a55e550e perf pmu: To info add event_type_desc
f08cc258431d perf evsel: Add accessor for tool_event 925320737ae2 perf pmus: Fake PMU clean up
d3d5c1a00fcd perf list: Avoid potential out of bounds memory read 4ae354d73a8e perf help: Fix a typo ("bellow")
74298dd8acb8 perf ftrace: Detect whether ftrace is enabled on system 83420d5f5863 perf test shell probe_vfs_getname: Remove extraneous '=' from probe line number regex
9327f0ecad48 perf build: Require at least clang 16.0.6 to build BPF skeletons. **********[[[[[[[[[[[[!!!!(This is the clang guard)]]]]]]]]]]*****
4c1af9bf97eb perf trace: If a syscall arg is marked as 'const', assume it is coming _from_ userspace
e37b315c17df perf parse-events: Remove duplicated include in parse-events.c

My machine is debian 12, and I cannot build perf trace with the new augmentation feature on clang-13, clang-14, and clang-15.

perf $ uname -r
6.11.0-061100rc7-generic

perf $ git log
commit 4c1af9bf97eb56d069421c3233ce61608458d5c8 (HEAD)

perf $ make CLANG=/bin/clang-13 && ./perf trace -e write --max-events=1
...
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter(struct syscall_enter_args *args) @ augmented_raw_syscalls.bpf.c:509
0: (bf) r7 = r1 ; R1=ctx() R7_w=ctx()

perf $ make CLANG=/bin/clang-14 && ./perf trace -e write --max-events=1
...
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter(struct syscall_enter_args *args) @ augmented_raw_syscalls.bpf.c:509

perf $ make CLANG=/bin/clang-15 && ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0

perf $ make CLANG=/bin/clang-16 && ./perf trace -e write --max-events=1

     0.000 ( 0.010 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,

count: 8) =

Anyway thank you so much, I'll try to reproduce it on Ubuntu, like you mentioned.

Thanks,
Howard

Quoted 1 line
Quoted 1 line
#8 lore raw

Hello,

I got some good news:

Not being able to pass the BPF verifier is solely (or bi-solely) caused by these two functions:

SEC("tp/syscalls/sys_enter_rename")
int sys_enter_rename(struct syscall_enter_args *args)

SEC("tp/syscalls/sys_enter_renameat2")
int sys_enter_renameat2(struct syscall_enter_args *args)

The problem with it is double-string augmentation, but I haven't figured out how to solve it. Can you please take a look please, Arnaldo? Now I just commented these two BPF functions and perf trace runs no problem, built by clang that's older than clang 16, for example clang 15.0.7.

perf $ clang -v
Ubuntu clang version 15.0.7
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1

     0.000 ( 0.008 ms): gmain/2173 write(fd: 4, buf: \1\0\0\0\0\0\0\0,

count: 8) =

But plot twist, it won't build on clang-14

perf $ clang -v
Ubuntu clang version 14.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter(struct syscall_enter_args *args) @ augmented_raw_syscalls.bpf.c:509
0: (bf) r7 = r1 ; R1=ctx() R7_w=ctx()

This time the problem is my code in BTF general collector:

; payload_offset += written; @ augmented_raw_syscalls.bpf.c:497

I'll resolve this.

Thanks,
Howard

Quoted 276 lines · Howard Chu

Hello,

I'm able to reproduce it on Ubuntu 24.

perf $ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

perf $ uname -r
6.11.0-061100-generic

perf $ git log
commit 4c1af9bf97eb56d069421c3233ce61608458d5c8 (HEAD)

Author: Arnaldo Carvalho de Melo <[email protected]>
Date:   Tue Sep 10 13:54:23 2024 -0300

perf trace: If a syscall arg is marked as 'const', assume it is
coming _from_ userspace

perf $ clang -v
Ubuntu clang version 15.0.7
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Candidate multilib: .;@m64
Selected multilib: .;@m64

perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter_rename(struct syscall_enter_args *args) @ augmented_raw_syscalls.bpf.c:275

With clang-16:

perf $ clang -v
Ubuntu clang version 16.0.6 (23ubuntu4)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Candidate multilib: .;@m64
Selected multilib: .;@m64

perf $ ./perf trace -e write --max-events=1

     0.000 ( 0.021 ms): sudo/4741 write(fd: 8, buf:

\27[?25l\27[37m\27[48;5;96m\27[H[trace] , count: 205) =

Quoted 2 lines

I'll try to build it and see. But Ubuntu 22 only has clang 11 to 15 so making 16 the minimum could be an issue.

Yes, I think disabling any distro that doesn't come with clang 16 (released in 2023) is not a good idea. I'll try to tame the BPF verifier.

Thanks,
Howard

Quoted 199 lines · Howard Chu

Hello James and Arnaldo,

Quoted 54 lines · Arnaldo Carvalho de Melo
Quoted 50 lines · James Clark
Quoted 45 lines · Arnaldo Carvalho de Melo
Quoted 32 lines · James Clark
Quoted 31 lines · Arnaldo Carvalho de Melo

Howard reported problems using perf features that use BPF:

perf $ clang -v
Debian clang version 15.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0

But it works with:

    perf $ clang -v
    Debian clang version 16.0.6 (15~deb12u1)
    Target: x86_64-pc-linux-gnu
    Thread model: posix
    InstalledDir: /bin
    Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
    Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
    Candidate multilib: .;@m64
    Selected multilib: .;@m64
    perf $ ./perf trace -e write --max-events=1
         0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8)                         = 8 (kworker/0:0-eve)
    perf $
Quoted 6 lines
Quoted 3 lines

So lets make that the required version, if you happen to have a slightly older version where this work, please report so that we can adjust the minimum required version.

I wasn't able to reproduce the issue with either of these versions. But I suppose it could be an issue with only 15.0.6.

Interesting, that complicates things, probably the best way then is to try to build it, if it fails, mention that 15.0.6 is known to be problematic and suggest working versions?

- Arnaldo

Quoted 37 lines
Quoted 3 lines

I still wasn't able to reproduce it with 15.0.6. And I double checked with V=1 that the build was using the right clang. I suppose it could be a build configuration issue, or maybe with a different kernel version?

Howard? If not reproducible we can revert it.

- Arnaldo

Quoted 27 lines
$  uname --kernel-release
6.8.0-76060800daily20240311-generic
$ ../../llvm-project/build/bin/clang -v
clang version 15.0.6 (https://github.com/llvm/llvm-project.git
088f33605d8a61ff519c580a71b1dd57d16a03f8)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/james/workspace/linux/linux/../../llvm-project/build/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
Candidate multilib: .;@m64
Selected multilib: .;@m64
$ git log
commit 003265bb6f028d7bcd7cbd92d6ba2b4e26382796
(perf-tools-next/perf-tools-next)
$ make O=../build/local/ CLANG=../../llvm-project/build/bin/clang -C \
   tools/perf
$ perf trace -e write --max-events=1
      0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
        0x7fffa102d9b0, count: 8)                           = 8

Thanks for the effort! But this output seems to be unaugmented? I think you are testing the perf/perf-tools-next branch that's slightly older (perf mem: Fix the wrong reference in parse_record_events() 003265bb6f028d7bcd7cbd92d6ba2b4e26382796), and I think that doesn't have the new perf trace feature yet...

That's why the perf trace output looks like:

Quoted 4 lines
Quoted 4 lines
$ perf trace -e write --max-events=1
      0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
        0x7fffa102d9b0, count: 8)

With the new buffer augmentation it should look like this:

perf $ ./perf trace -e write --max-events=1

     0.000 ( 0.011 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,

count: 8) =

The new perf trace can be found on
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/
perf-tools-next

perf $ git log
commit 1de5b5dcb8353f36581c963df2d359a5f151a0be (HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next)

Also, if you want to test the clang, you may want to checkout to the commit before the clang-16 guard (otherwise you can't test different clang version).

perf $ git log --oneline
1de5b5dcb835 (HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next) perf
trace: Mark the 'head' arg in the set_robust_list syscall as coming from user space
0c1019e3463b perf trace: Mark the 'rseq' arg in the rseq syscall as coming from user space
edf3ce0ed38e perf env: Find correct branch counter info on hybrid 9953807c9e01 perf evlist: Print hint for group
eb9b9a6f5ab3 tools: Drop nonsensical -O6
89c0a55e550e perf pmu: To info add event_type_desc
f08cc258431d perf evsel: Add accessor for tool_event 925320737ae2 perf pmus: Fake PMU clean up
d3d5c1a00fcd perf list: Avoid potential out of bounds memory read 4ae354d73a8e perf help: Fix a typo ("bellow")
74298dd8acb8 perf ftrace: Detect whether ftrace is enabled on system 83420d5f5863 perf test shell probe_vfs_getname: Remove extraneous '=' from probe line number regex
9327f0ecad48 perf build: Require at least clang 16.0.6 to build BPF skeletons. **********[[[[[[[[[[[[!!!!(This is the clang guard)]]]]]]]]]]*****
4c1af9bf97eb perf trace: If a syscall arg is marked as 'const', assume it is coming _from_ userspace
e37b315c17df perf parse-events: Remove duplicated include in parse-events.c

My machine is debian 12, and I cannot build perf trace with the new augmentation feature on clang-13, clang-14, and clang-15.

perf $ uname -r
6.11.0-061100rc7-generic

perf $ git log
commit 4c1af9bf97eb56d069421c3233ce61608458d5c8 (HEAD)

perf $ make CLANG=/bin/clang-13 && ./perf trace -e write --max-events=1
...
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter(struct syscall_enter_args *args) @ augmented_raw_syscalls.bpf.c:509
0: (bf) r7 = r1 ; R1=ctx() R7_w=ctx()

perf $ make CLANG=/bin/clang-14 && ./perf trace -e write --max-events=1
...
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter(struct syscall_enter_args *args) @ augmented_raw_syscalls.bpf.c:509

perf $ make CLANG=/bin/clang-15 && ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0

perf $ make CLANG=/bin/clang-16 && ./perf trace -e write --max-events=1

     0.000 ( 0.010 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,

count: 8) =

Anyway thank you so much, I'll try to reproduce it on Ubuntu, like you mentioned.

Thanks,
Howard

Quoted 1 line
Quoted 1 line
#9 lore raw
Quoted 10 lines · Howard Chu

Hello,

I got some good news:

Not being able to pass the BPF verifier is solely (or bi-solely) caused by these two functions:

SEC("tp/syscalls/sys_enter_rename")
int sys_enter_rename(struct syscall_enter_args *args)

Quoted 9 lines

SEC("tp/syscalls/sys_enter_renameat2")
int sys_enter_renameat2(struct syscall_enter_args *args)

The problem with it is double-string augmentation, but I haven't figured out how to solve it. Can you please take a look please, Arnaldo? Now I just commented these two BPF functions and perf trace runs no problem, built by clang that's older than clang 16, for example clang 15.0.7.

hi Howard,

I could be wrong - and I don't know this code at all - but I _think_ the problem is here:

	oldpath_len = augmented_arg__read_str(&augmented_args->arg,
oldpath_arg, sizeof(augmented_args->arg.value));
        augmented_args->arg.size = PERF_ALIGN(oldpath_len + 1, sizeof(u64));
        len += augmented_args->arg.size;

        struct augmented_arg *arg2 = (void *)&augmented_args->arg.value

+ augmented_args->arg.size;

newpath_len = augmented_arg__read_str(arg2, newpath_arg,
sizeof(augmented_args->arg.value));

Notice that these strings share the augmented_args->arg.value for storage, but in the second case (reading arg2) we have added augmented_args->arg.size to the offset where we store the second string. However at the same time we have also told augmented_arg__read_str() that it has

sizeof(augmented_args->arg.value)

...to work with as arg_len, where in reality we should have specified

sizeof(augmented_args->arg.value) - augmented_args->arg.size

...and ensured that after doing that subtraction, there is still space to work with (i.e. the above is > 0).

I _think_ the verifier is likely complaining that we can write past the end of augmented_args->arg.value , does that sound right?

Fixing the arg_len parameter for cases where we record more than one augmented string by reducing arg_len available should solve this if so. Hope this helps,

Alan

Quoted 318 lines

perf $ clang -v
Ubuntu clang version 15.0.7
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1

     0.000 ( 0.008 ms): gmain/2173 write(fd: 4, buf: \1\0\0\0\0\0\0\0,

count: 8) =

But plot twist, it won't build on clang-14

perf $ clang -v
Ubuntu clang version 14.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter(struct syscall_enter_args *args) @ augmented_raw_syscalls.bpf.c:509
0: (bf) r7 = r1 ; R1=ctx() R7_w=ctx()

This time the problem is my code in BTF general collector:

; payload_offset += written; @ augmented_raw_syscalls.bpf.c:497

I'll resolve this.

Thanks,
Howard

Quoted 276 lines · Howard Chu

Hello,

I'm able to reproduce it on Ubuntu 24.

perf $ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

perf $ uname -r
6.11.0-061100-generic

perf $ git log
commit 4c1af9bf97eb56d069421c3233ce61608458d5c8 (HEAD)

Author: Arnaldo Carvalho de Melo <[email protected]>
Date:   Tue Sep 10 13:54:23 2024 -0300

perf trace: If a syscall arg is marked as 'const', assume it is
coming _from_ userspace

perf $ clang -v
Ubuntu clang version 15.0.7
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Candidate multilib: .;@m64
Selected multilib: .;@m64

perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter_rename(struct syscall_enter_args *args) @ augmented_raw_syscalls.bpf.c:275

With clang-16:

perf $ clang -v
Ubuntu clang version 16.0.6 (23ubuntu4)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Candidate multilib: .;@m64
Selected multilib: .;@m64

perf $ ./perf trace -e write --max-events=1

     0.000 ( 0.021 ms): sudo/4741 write(fd: 8, buf:

\27[?25l\27[37m\27[48;5;96m\27[H[trace] , count: 205) =

Quoted 2 lines

I'll try to build it and see. But Ubuntu 22 only has clang 11 to 15 so making 16 the minimum could be an issue.

Yes, I think disabling any distro that doesn't come with clang 16 (released in 2023) is not a good idea. I'll try to tame the BPF verifier.

Thanks,
Howard

Quoted 199 lines · Howard Chu

Hello James and Arnaldo,

Quoted 54 lines · Arnaldo Carvalho de Melo
Quoted 50 lines · James Clark
Quoted 45 lines · Arnaldo Carvalho de Melo
Quoted 32 lines · James Clark
Quoted 31 lines · Arnaldo Carvalho de Melo

Howard reported problems using perf features that use BPF:

perf $ clang -v
Debian clang version 15.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0

But it works with:

    perf $ clang -v
    Debian clang version 16.0.6 (15~deb12u1)
    Target: x86_64-pc-linux-gnu
    Thread model: posix
    InstalledDir: /bin
    Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
    Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
    Candidate multilib: .;@m64
    Selected multilib: .;@m64
    perf $ ./perf trace -e write --max-events=1
         0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8)                         = 8 (kworker/0:0-eve)
    perf $
Quoted 6 lines
Quoted 3 lines

So lets make that the required version, if you happen to have a slightly older version where this work, please report so that we can adjust the minimum required version.

I wasn't able to reproduce the issue with either of these versions. But I suppose it could be an issue with only 15.0.6.

Interesting, that complicates things, probably the best way then is to try to build it, if it fails, mention that 15.0.6 is known to be problematic and suggest working versions?

- Arnaldo

Quoted 37 lines
Quoted 3 lines

I still wasn't able to reproduce it with 15.0.6. And I double checked with V=1 that the build was using the right clang. I suppose it could be a build configuration issue, or maybe with a different kernel version?

Howard? If not reproducible we can revert it.

- Arnaldo

Quoted 27 lines
$  uname --kernel-release
6.8.0-76060800daily20240311-generic
$ ../../llvm-project/build/bin/clang -v
clang version 15.0.6 (https://github.com/llvm/llvm-project.git
088f33605d8a61ff519c580a71b1dd57d16a03f8)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/james/workspace/linux/linux/../../llvm-project/build/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
Candidate multilib: .;@m64
Selected multilib: .;@m64
$ git log
commit 003265bb6f028d7bcd7cbd92d6ba2b4e26382796
(perf-tools-next/perf-tools-next)
$ make O=../build/local/ CLANG=../../llvm-project/build/bin/clang -C \
   tools/perf
$ perf trace -e write --max-events=1
      0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
        0x7fffa102d9b0, count: 8)                           = 8

Thanks for the effort! But this output seems to be unaugmented? I think you are testing the perf/perf-tools-next branch that's slightly older (perf mem: Fix the wrong reference in parse_record_events() 003265bb6f028d7bcd7cbd92d6ba2b4e26382796), and I think that doesn't have the new perf trace feature yet...

That's why the perf trace output looks like:

Quoted 4 lines
Quoted 4 lines
$ perf trace -e write --max-events=1
      0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
        0x7fffa102d9b0, count: 8)

With the new buffer augmentation it should look like this:

perf $ ./perf trace -e write --max-events=1

     0.000 ( 0.011 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,

count: 8) =

The new perf trace can be found on
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/
perf-tools-next

perf $ git log
commit 1de5b5dcb8353f36581c963df2d359a5f151a0be (HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next)

Also, if you want to test the clang, you may want to checkout to the commit before the clang-16 guard (otherwise you can't test different clang version).

perf $ git log --oneline
1de5b5dcb835 (HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next) perf
trace: Mark the 'head' arg in the set_robust_list syscall as coming from user space
0c1019e3463b perf trace: Mark the 'rseq' arg in the rseq syscall as coming from user space
edf3ce0ed38e perf env: Find correct branch counter info on hybrid 9953807c9e01 perf evlist: Print hint for group
eb9b9a6f5ab3 tools: Drop nonsensical -O6
89c0a55e550e perf pmu: To info add event_type_desc
f08cc258431d perf evsel: Add accessor for tool_event 925320737ae2 perf pmus: Fake PMU clean up
d3d5c1a00fcd perf list: Avoid potential out of bounds memory read 4ae354d73a8e perf help: Fix a typo ("bellow")
74298dd8acb8 perf ftrace: Detect whether ftrace is enabled on system 83420d5f5863 perf test shell probe_vfs_getname: Remove extraneous '=' from probe line number regex
9327f0ecad48 perf build: Require at least clang 16.0.6 to build BPF skeletons. **********[[[[[[[[[[[[!!!!(This is the clang guard)]]]]]]]]]]*****
4c1af9bf97eb perf trace: If a syscall arg is marked as 'const', assume it is coming _from_ userspace
e37b315c17df perf parse-events: Remove duplicated include in parse-events.c

My machine is debian 12, and I cannot build perf trace with the new augmentation feature on clang-13, clang-14, and clang-15.

perf $ uname -r
6.11.0-061100rc7-generic

perf $ git log
commit 4c1af9bf97eb56d069421c3233ce61608458d5c8 (HEAD)

perf $ make CLANG=/bin/clang-13 && ./perf trace -e write --max-events=1
...
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter(struct syscall_enter_args *args) @ augmented_raw_syscalls.bpf.c:509
0: (bf) r7 = r1 ; R1=ctx() R7_w=ctx()

perf $ make CLANG=/bin/clang-14 && ./perf trace -e write --max-events=1
...
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter(struct syscall_enter_args *args) @ augmented_raw_syscalls.bpf.c:509

perf $ make CLANG=/bin/clang-15 && ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0

perf $ make CLANG=/bin/clang-16 && ./perf trace -e write --max-events=1

     0.000 ( 0.010 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,

count: 8) =

Anyway thank you so much, I'll try to reproduce it on Ubuntu, like you mentioned.

Thanks,
Howard

Quoted 1 line
Quoted 1 line
#10 lore raw

Hello Alan,

Quoted 55 lines · Alan Maguire
Quoted 10 lines · Howard Chu

Hello,

I got some good news:

Not being able to pass the BPF verifier is solely (or bi-solely) caused by these two functions:

SEC("tp/syscalls/sys_enter_rename")
int sys_enter_rename(struct syscall_enter_args *args)

Quoted 9 lines

SEC("tp/syscalls/sys_enter_renameat2")
int sys_enter_renameat2(struct syscall_enter_args *args)

The problem with it is double-string augmentation, but I haven't figured out how to solve it. Can you please take a look please, Arnaldo? Now I just commented these two BPF functions and perf trace runs no problem, built by clang that's older than clang 16, for example clang 15.0.7.

hi Howard,

I could be wrong - and I don't know this code at all - but I _think_ the problem is here:

        oldpath_len = augmented_arg__read_str(&augmented_args->arg,
oldpath_arg, sizeof(augmented_args->arg.value));
        augmented_args->arg.size = PERF_ALIGN(oldpath_len + 1, sizeof(u64));
        len += augmented_args->arg.size;

        struct augmented_arg *arg2 = (void *)&augmented_args->arg.value

+ augmented_args->arg.size;

newpath_len = augmented_arg__read_str(arg2, newpath_arg,
sizeof(augmented_args->arg.value));

Notice that these strings share the augmented_args->arg.value for storage, but in the second case (reading arg2) we have added augmented_args->arg.size to the offset where we store the second string. However at the same time we have also told augmented_arg__read_str() that it has

sizeof(augmented_args->arg.value)

...to work with as arg_len, where in reality we should have specified

sizeof(augmented_args->arg.value) - augmented_args->arg.size

...and ensured that after doing that subtraction, there is still space to work with (i.e. the above is > 0).

Quoted 6 lines

I _think_ the verifier is likely complaining that we can write past the end of augmented_args->arg.value , does that sound right?

Fixing the arg_len parameter for cases where we record more than one augmented string by reducing arg_len available should solve this if so. Hope this helps,

Thanks for the help, but unfortunately it didn't work... But that's a valid analysis, I had the same idea as you, but I haven't been able to solve it. I'll give it more tries.

Thanks,
Howard

Quoted 322 lines

Alan

Quoted 318 lines

perf $ clang -v
Ubuntu clang version 15.0.7
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1

     0.000 ( 0.008 ms): gmain/2173 write(fd: 4, buf: \1\0\0\0\0\0\0\0,

count: 8) =

But plot twist, it won't build on clang-14

perf $ clang -v
Ubuntu clang version 14.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter(struct syscall_enter_args *args) @ augmented_raw_syscalls.bpf.c:509
0: (bf) r7 = r1 ; R1=ctx() R7_w=ctx()

This time the problem is my code in BTF general collector:

; payload_offset += written; @ augmented_raw_syscalls.bpf.c:497

I'll resolve this.

Thanks,
Howard

Quoted 276 lines · Howard Chu

Hello,

I'm able to reproduce it on Ubuntu 24.

perf $ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

perf $ uname -r
6.11.0-061100-generic

perf $ git log
commit 4c1af9bf97eb56d069421c3233ce61608458d5c8 (HEAD)

Author: Arnaldo Carvalho de Melo <[email protected]>
Date:   Tue Sep 10 13:54:23 2024 -0300

perf trace: If a syscall arg is marked as 'const', assume it is
coming _from_ userspace

perf $ clang -v
Ubuntu clang version 15.0.7
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Candidate multilib: .;@m64
Selected multilib: .;@m64

perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter_rename(struct syscall_enter_args *args) @ augmented_raw_syscalls.bpf.c:275

With clang-16:

perf $ clang -v
Ubuntu clang version 16.0.6 (23ubuntu4)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Candidate multilib: .;@m64
Selected multilib: .;@m64

perf $ ./perf trace -e write --max-events=1

     0.000 ( 0.021 ms): sudo/4741 write(fd: 8, buf:

\27[?25l\27[37m\27[48;5;96m\27[H[trace] , count: 205) =

Quoted 2 lines

I'll try to build it and see. But Ubuntu 22 only has clang 11 to 15 so making 16 the minimum could be an issue.

Yes, I think disabling any distro that doesn't come with clang 16 (released in 2023) is not a good idea. I'll try to tame the BPF verifier.

Thanks,
Howard

Quoted 199 lines · Howard Chu

Hello James and Arnaldo,

Quoted 54 lines · Arnaldo Carvalho de Melo
Quoted 50 lines · James Clark
Quoted 45 lines · Arnaldo Carvalho de Melo
Quoted 32 lines · James Clark
Quoted 31 lines · Arnaldo Carvalho de Melo

Howard reported problems using perf features that use BPF:

perf $ clang -v
Debian clang version 15.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0

But it works with:

    perf $ clang -v
    Debian clang version 16.0.6 (15~deb12u1)
    Target: x86_64-pc-linux-gnu
    Thread model: posix
    InstalledDir: /bin
    Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
    Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
    Candidate multilib: .;@m64
    Selected multilib: .;@m64
    perf $ ./perf trace -e write --max-events=1
         0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8)                         = 8 (kworker/0:0-eve)
    perf $
Quoted 6 lines
Quoted 3 lines

So lets make that the required version, if you happen to have a slightly older version where this work, please report so that we can adjust the minimum required version.

I wasn't able to reproduce the issue with either of these versions. But I suppose it could be an issue with only 15.0.6.

Interesting, that complicates things, probably the best way then is to try to build it, if it fails, mention that 15.0.6 is known to be problematic and suggest working versions?

- Arnaldo

Quoted 37 lines
Quoted 3 lines

I still wasn't able to reproduce it with 15.0.6. And I double checked with V=1 that the build was using the right clang. I suppose it could be a build configuration issue, or maybe with a different kernel version?

Howard? If not reproducible we can revert it.

- Arnaldo

Quoted 27 lines
$  uname --kernel-release
6.8.0-76060800daily20240311-generic
$ ../../llvm-project/build/bin/clang -v
clang version 15.0.6 (https://github.com/llvm/llvm-project.git
088f33605d8a61ff519c580a71b1dd57d16a03f8)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/james/workspace/linux/linux/../../llvm-project/build/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
Candidate multilib: .;@m64
Selected multilib: .;@m64
$ git log
commit 003265bb6f028d7bcd7cbd92d6ba2b4e26382796
(perf-tools-next/perf-tools-next)
$ make O=../build/local/ CLANG=../../llvm-project/build/bin/clang -C \
   tools/perf
$ perf trace -e write --max-events=1
      0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
        0x7fffa102d9b0, count: 8)                           = 8

Thanks for the effort! But this output seems to be unaugmented? I think you are testing the perf/perf-tools-next branch that's slightly older (perf mem: Fix the wrong reference in parse_record_events() 003265bb6f028d7bcd7cbd92d6ba2b4e26382796), and I think that doesn't have the new perf trace feature yet...

That's why the perf trace output looks like:

Quoted 4 lines
Quoted 4 lines
$ perf trace -e write --max-events=1
      0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
        0x7fffa102d9b0, count: 8)

With the new buffer augmentation it should look like this:

perf $ ./perf trace -e write --max-events=1

     0.000 ( 0.011 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,

count: 8) =

The new perf trace can be found on
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/
perf-tools-next

perf $ git log
commit 1de5b5dcb8353f36581c963df2d359a5f151a0be (HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next)

Also, if you want to test the clang, you may want to checkout to the commit before the clang-16 guard (otherwise you can't test different clang version).

perf $ git log --oneline
1de5b5dcb835 (HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next) perf
trace: Mark the 'head' arg in the set_robust_list syscall as coming from user space
0c1019e3463b perf trace: Mark the 'rseq' arg in the rseq syscall as coming from user space
edf3ce0ed38e perf env: Find correct branch counter info on hybrid 9953807c9e01 perf evlist: Print hint for group
eb9b9a6f5ab3 tools: Drop nonsensical -O6
89c0a55e550e perf pmu: To info add event_type_desc
f08cc258431d perf evsel: Add accessor for tool_event 925320737ae2 perf pmus: Fake PMU clean up
d3d5c1a00fcd perf list: Avoid potential out of bounds memory read 4ae354d73a8e perf help: Fix a typo ("bellow")
74298dd8acb8 perf ftrace: Detect whether ftrace is enabled on system 83420d5f5863 perf test shell probe_vfs_getname: Remove extraneous '=' from probe line number regex
9327f0ecad48 perf build: Require at least clang 16.0.6 to build BPF skeletons. **********[[[[[[[[[[[[!!!!(This is the clang guard)]]]]]]]]]]*****
4c1af9bf97eb perf trace: If a syscall arg is marked as 'const', assume it is coming _from_ userspace
e37b315c17df perf parse-events: Remove duplicated include in parse-events.c

My machine is debian 12, and I cannot build perf trace with the new augmentation feature on clang-13, clang-14, and clang-15.

perf $ uname -r
6.11.0-061100rc7-generic

perf $ git log
commit 4c1af9bf97eb56d069421c3233ce61608458d5c8 (HEAD)

perf $ make CLANG=/bin/clang-13 && ./perf trace -e write --max-events=1
...
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter(struct syscall_enter_args *args) @ augmented_raw_syscalls.bpf.c:509
0: (bf) r7 = r1 ; R1=ctx() R7_w=ctx()

perf $ make CLANG=/bin/clang-14 && ./perf trace -e write --max-events=1
...
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter(struct syscall_enter_args *args) @ augmented_raw_syscalls.bpf.c:509

perf $ make CLANG=/bin/clang-15 && ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0

perf $ make CLANG=/bin/clang-16 && ./perf trace -e write --max-events=1

     0.000 ( 0.010 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,

count: 8) =

Anyway thank you so much, I'll try to reproduce it on Ubuntu, like you mentioned.

Thanks,
Howard

Quoted 1 line
Quoted 1 line
#11 lore raw

Hi Alan, Arnaldo and James,

This problem was solved in [PATCH 0/2] perf trace: Fix support for the new BPF feature in clang 12 (Link:
https://lore.kernel.org/linux-perf-users/[email protected]/T/#t),
sorry I forgot to cc you two.

Alan's thought was correct. Thank you so much! :)

Thanks,
Howard

Quoted 396 lines · Howard Chu

Hello Alan,

Quoted 55 lines · Alan Maguire
Quoted 10 lines · Howard Chu

Hello,

I got some good news:

Not being able to pass the BPF verifier is solely (or bi-solely) caused by these two functions:

SEC("tp/syscalls/sys_enter_rename")
int sys_enter_rename(struct syscall_enter_args *args)

Quoted 9 lines

SEC("tp/syscalls/sys_enter_renameat2")
int sys_enter_renameat2(struct syscall_enter_args *args)

The problem with it is double-string augmentation, but I haven't figured out how to solve it. Can you please take a look please, Arnaldo? Now I just commented these two BPF functions and perf trace runs no problem, built by clang that's older than clang 16, for example clang 15.0.7.

hi Howard,

I could be wrong - and I don't know this code at all - but I _think_ the problem is here:

        oldpath_len = augmented_arg__read_str(&augmented_args->arg,
oldpath_arg, sizeof(augmented_args->arg.value));
        augmented_args->arg.size = PERF_ALIGN(oldpath_len + 1, sizeof(u64));
        len += augmented_args->arg.size;

        struct augmented_arg *arg2 = (void *)&augmented_args->arg.value

+ augmented_args->arg.size;

newpath_len = augmented_arg__read_str(arg2, newpath_arg,
sizeof(augmented_args->arg.value));

Notice that these strings share the augmented_args->arg.value for storage, but in the second case (reading arg2) we have added augmented_args->arg.size to the offset where we store the second string. However at the same time we have also told augmented_arg__read_str() that it has

sizeof(augmented_args->arg.value)

...to work with as arg_len, where in reality we should have specified

sizeof(augmented_args->arg.value) - augmented_args->arg.size

...and ensured that after doing that subtraction, there is still space to work with (i.e. the above is > 0).

Quoted 6 lines

I _think_ the verifier is likely complaining that we can write past the end of augmented_args->arg.value , does that sound right?

Fixing the arg_len parameter for cases where we record more than one augmented string by reducing arg_len available should solve this if so. Hope this helps,

Thanks for the help, but unfortunately it didn't work... But that's a valid analysis, I had the same idea as you, but I haven't been able to solve it. I'll give it more tries.

Thanks,
Howard

Quoted 322 lines

Alan

Quoted 318 lines

perf $ clang -v
Ubuntu clang version 15.0.7
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1

     0.000 ( 0.008 ms): gmain/2173 write(fd: 4, buf: \1\0\0\0\0\0\0\0,

count: 8) =

But plot twist, it won't build on clang-14

perf $ clang -v
Ubuntu clang version 14.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter(struct syscall_enter_args *args) @ augmented_raw_syscalls.bpf.c:509
0: (bf) r7 = r1 ; R1=ctx() R7_w=ctx()

This time the problem is my code in BTF general collector:

; payload_offset += written; @ augmented_raw_syscalls.bpf.c:497

I'll resolve this.

Thanks,
Howard

Quoted 276 lines · Howard Chu

Hello,

I'm able to reproduce it on Ubuntu 24.

perf $ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

perf $ uname -r
6.11.0-061100-generic

perf $ git log
commit 4c1af9bf97eb56d069421c3233ce61608458d5c8 (HEAD)

Author: Arnaldo Carvalho de Melo <[email protected]>
Date:   Tue Sep 10 13:54:23 2024 -0300

perf trace: If a syscall arg is marked as 'const', assume it is
coming _from_ userspace

perf $ clang -v
Ubuntu clang version 15.0.7
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Candidate multilib: .;@m64
Selected multilib: .;@m64

perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter_rename(struct syscall_enter_args *args) @ augmented_raw_syscalls.bpf.c:275

With clang-16:

perf $ clang -v
Ubuntu clang version 16.0.6 (23ubuntu4)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13 Candidate multilib: .;@m64
Selected multilib: .;@m64

perf $ ./perf trace -e write --max-events=1

     0.000 ( 0.021 ms): sudo/4741 write(fd: 8, buf:

\27[?25l\27[37m\27[48;5;96m\27[H[trace] , count: 205) =

Quoted 2 lines

I'll try to build it and see. But Ubuntu 22 only has clang 11 to 15 so making 16 the minimum could be an issue.

Yes, I think disabling any distro that doesn't come with clang 16 (released in 2023) is not a good idea. I'll try to tame the BPF verifier.

Thanks,
Howard

Quoted 199 lines · Howard Chu

Hello James and Arnaldo,

Quoted 54 lines · Arnaldo Carvalho de Melo
Quoted 50 lines · James Clark
Quoted 45 lines · Arnaldo Carvalho de Melo
Quoted 32 lines · James Clark
Quoted 31 lines · Arnaldo Carvalho de Melo

Howard reported problems using perf features that use BPF:

perf $ clang -v
Debian clang version 15.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12 Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0

But it works with:

    perf $ clang -v
    Debian clang version 16.0.6 (15~deb12u1)
    Target: x86_64-pc-linux-gnu
    Thread model: posix
    InstalledDir: /bin
    Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
    Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
    Candidate multilib: .;@m64
    Selected multilib: .;@m64
    perf $ ./perf trace -e write --max-events=1
         0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8)                         = 8 (kworker/0:0-eve)
    perf $
Quoted 6 lines
Quoted 3 lines

So lets make that the required version, if you happen to have a slightly older version where this work, please report so that we can adjust the minimum required version.

I wasn't able to reproduce the issue with either of these versions. But I suppose it could be an issue with only 15.0.6.

Interesting, that complicates things, probably the best way then is to try to build it, if it fails, mention that 15.0.6 is known to be problematic and suggest working versions?

- Arnaldo

Quoted 37 lines
Quoted 3 lines

I still wasn't able to reproduce it with 15.0.6. And I double checked with V=1 that the build was using the right clang. I suppose it could be a build configuration issue, or maybe with a different kernel version?

Howard? If not reproducible we can revert it.

- Arnaldo

Quoted 27 lines
$  uname --kernel-release
6.8.0-76060800daily20240311-generic
$ ../../llvm-project/build/bin/clang -v
clang version 15.0.6 (https://github.com/llvm/llvm-project.git
088f33605d8a61ff519c580a71b1dd57d16a03f8)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/james/workspace/linux/linux/../../llvm-project/build/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
Candidate multilib: .;@m64
Selected multilib: .;@m64
$ git log
commit 003265bb6f028d7bcd7cbd92d6ba2b4e26382796
(perf-tools-next/perf-tools-next)
$ make O=../build/local/ CLANG=../../llvm-project/build/bin/clang -C \
   tools/perf
$ perf trace -e write --max-events=1
      0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
        0x7fffa102d9b0, count: 8)                           = 8

Thanks for the effort! But this output seems to be unaugmented? I think you are testing the perf/perf-tools-next branch that's slightly older (perf mem: Fix the wrong reference in parse_record_events() 003265bb6f028d7bcd7cbd92d6ba2b4e26382796), and I think that doesn't have the new perf trace feature yet...

That's why the perf trace output looks like:

Quoted 4 lines
Quoted 4 lines
$ perf trace -e write --max-events=1
      0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
        0x7fffa102d9b0, count: 8)

With the new buffer augmentation it should look like this:

perf $ ./perf trace -e write --max-events=1

     0.000 ( 0.011 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,

count: 8) =

The new perf trace can be found on
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/
perf-tools-next

perf $ git log
commit 1de5b5dcb8353f36581c963df2d359a5f151a0be (HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next)

Also, if you want to test the clang, you may want to checkout to the commit before the clang-16 guard (otherwise you can't test different clang version).

perf $ git log --oneline
1de5b5dcb835 (HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next) perf
trace: Mark the 'head' arg in the set_robust_list syscall as coming from user space
0c1019e3463b perf trace: Mark the 'rseq' arg in the rseq syscall as coming from user space
edf3ce0ed38e perf env: Find correct branch counter info on hybrid 9953807c9e01 perf evlist: Print hint for group
eb9b9a6f5ab3 tools: Drop nonsensical -O6
89c0a55e550e perf pmu: To info add event_type_desc
f08cc258431d perf evsel: Add accessor for tool_event 925320737ae2 perf pmus: Fake PMU clean up
d3d5c1a00fcd perf list: Avoid potential out of bounds memory read 4ae354d73a8e perf help: Fix a typo ("bellow")
74298dd8acb8 perf ftrace: Detect whether ftrace is enabled on system 83420d5f5863 perf test shell probe_vfs_getname: Remove extraneous '=' from probe line number regex
9327f0ecad48 perf build: Require at least clang 16.0.6 to build BPF skeletons. **********[[[[[[[[[[[[!!!!(This is the clang guard)]]]]]]]]]]*****
4c1af9bf97eb perf trace: If a syscall arg is marked as 'const', assume it is coming _from_ userspace
e37b315c17df perf parse-events: Remove duplicated include in parse-events.c

My machine is debian 12, and I cannot build perf trace with the new augmentation feature on clang-13, clang-14, and clang-15.

perf $ uname -r
6.11.0-061100rc7-generic

perf $ git log
commit 4c1af9bf97eb56d069421c3233ce61608458d5c8 (HEAD)

perf $ make CLANG=/bin/clang-13 && ./perf trace -e write --max-events=1
...
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter(struct syscall_enter_args *args) @ augmented_raw_syscalls.bpf.c:509
0: (bf) r7 = r1 ; R1=ctx() R7_w=ctx()

perf $ make CLANG=/bin/clang-14 && ./perf trace -e write --max-events=1
...
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter(struct syscall_enter_args *args) @ augmented_raw_syscalls.bpf.c:509

perf $ make CLANG=/bin/clang-15 && ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0

perf $ make CLANG=/bin/clang-16 && ./perf trace -e write --max-events=1

     0.000 ( 0.010 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,

count: 8) =

Anyway thank you so much, I'll try to reproduce it on Ubuntu, like you mentioned.

Thanks,
Howard

Quoted 1 line
Quoted 1 line
maintainer
#12 lore raw
Quoted 8 lines · Howard Chu

Hi Alan, Arnaldo and James,

This problem was solved in [PATCH 0/2] perf trace: Fix support for the new BPF feature in clang 12 (Link:
https://lore.kernel.org/linux-perf-users/[email protected]/T/#t),
sorry I forgot to cc you two.

Alan's thought was correct. Thank you so much! :)

It'd be great if any of you can test this change. Now I only have machines with clang 16.

Thanks,
Namhyung

A message that is not in the archive (sent off-list or not yet received): CA+JHD93JgJL_4GJFcFUNu-FpNfFOoyDRJ7QuvO82M8G1EwM5pQ@mail.gmail.com

reviewer Tested-by
#14 lore raw
Quoted 14 lines · Arnaldo Carvalho de Melo
Quoted 12 lines · Namhyung Kim
Quoted 5 lines · Howard Chu

Hi Alan, Arnaldo and James,

This problem was solved in [PATCH 0/2] perf trace: Fix support for the new BPF feature in clang 12 (Link:

https://lore.kernel.org/linux-perf-users/[email protected]/T/#t
),

Quoted 3 lines

sorry I forgot to cc you two.

Alan's thought was correct. Thank you so much! :)

Quoted 14 lines
Quoted 3 lines

It'd be great if any of you can test this change. Now I only have machines with clang 16.

I'll test this tomorrow.

- Arnaldo

Quoted 5 lines

Thanks,
Namhyung

Tested with clang 15:

$ sudo perf trace -e write --max-events=100 -- echo hello

0.000 ( 0.014 ms): echo/834165 write(fd: 1, buf: hello\10, count: 6)

                                           =

Unrelated to this change, I noticed that with older clangs or with BUILD_BPF_SKEL=0 that commit b257fac12f38 ("perf trace: Pretty print buffer data") changes the buffer address to print nothing, and the '6' return value is missing. Not sure if this was intended or not:

 $ sudo perf trace -e write --max-events=100 -- echo hello

Before:
0.000 ( 0.009 ms): echo/772951 write(fd: 1, buf: 0x58c415257440,
count: 6) = 6

After:

     0.000 ( 0.009 ms): echo/760370 write(fd: 1, buf: , count: 6) 

                                            =

Thanks
James

reviewer
#15 lore raw
Quoted 32 lines · James Clark
Quoted 14 lines · Arnaldo Carvalho de Melo
Quoted 12 lines · Namhyung Kim
Quoted 5 lines · Howard Chu

Hi Alan, Arnaldo and James,

This problem was solved in [PATCH 0/2] perf trace: Fix support for the new BPF feature in clang 12 (Link:

https://lore.kernel.org/linux-perf-users/20241007051414.2995674-1- [email protected]/T/#t ),

Quoted 3 lines

sorry I forgot to cc you two.

Alan's thought was correct. Thank you so much! :)

Quoted 14 lines
Quoted 3 lines

It'd be great if any of you can test this change.  Now I only have machines with clang 16.

I'll test this tomorrow.

- Arnaldo

Quoted 5 lines

Thanks,
Namhyung

Quoted 9 lines

Tested with clang 15:

$ sudo perf trace -e write --max-events=100 -- echo hello

  0.000 ( 0.014 ms): echo/834165 write(fd: 1, buf: hello\10, count: 6) =

Should have left this tag on the actual patch. Will re-post it there.

maintainer
#16 lore raw
Quoted 40 lines · James Clark
Quoted 37 lines · James Clark
Quoted 15 lines · Arnaldo Carvalho de Melo
Quoted 13 lines · Namhyung Kim
Quoted 5 lines · Howard Chu

Hi Alan, Arnaldo and James,

This problem was solved in [PATCH 0/2] perf trace: Fix support for the new BPF feature in clang 12 (Link:

https://lore.kernel.org/linux-perf-users/20241007051414.2995674-1-
[email protected]/T/#t
),

Quoted 3 lines

sorry I forgot to cc you two.

Alan's thought was correct. Thank you so much! :)

Quoted 14 lines
Quoted 3 lines

It'd be great if any of you can test this change.  Now I only have machines with clang 16.

I'll test this tomorrow.

- Arnaldo

Quoted 5 lines

Thanks,
Namhyung

Tested with clang 15:

$ sudo perf trace -e write --max-events=100 -- echo hello
Quoted 7 lines
Quoted 5 lines

  0.000 ( 0.014 ms): echo/834165 write(fd: 1, buf: hello\10, count: 6) =

Should have left this tag on the actual patch. Will re-post it there.

Right, thanks for taking care of this!
Namhyung

author maintainer
#17 lore raw
In reply to James Clark
Quoted 48 lines · James Clark
Quoted 14 lines · Arnaldo Carvalho de Melo
Quoted 12 lines · Namhyung Kim
Quoted 5 lines · Howard Chu

Hi Alan, Arnaldo and James,

This problem was solved in [PATCH 0/2] perf trace: Fix support for the new BPF feature in clang 12 (Link:

https://lore.kernel.org/linux-perf-users/[email protected]/T/#t
),

Quoted 3 lines

sorry I forgot to cc you two.

Alan's thought was correct. Thank you so much! :)

Quoted 14 lines
Quoted 3 lines

It'd be great if any of you can test this change. Now I only have machines with clang 16.

I'll test this tomorrow.

- Arnaldo

Quoted 5 lines

Thanks,
Namhyung

Tested with clang 15:

$ sudo perf trace -e write --max-events=100 -- echo hello

0.000 ( 0.014 ms): echo/834165 write(fd: 1, buf: hello\10, count: 6)

                                           =

Unrelated to this change, I noticed that with older clangs or with BUILD_BPF_SKEL=0 that commit b257fac12f38 ("perf trace: Pretty print buffer data") changes the buffer address to print nothing, and the '6' return value is missing. Not sure if this was intended or not:

Quoted 9 lines
 $ sudo perf trace -e write --max-events=100 -- echo hello

Before:
0.000 ( 0.009 ms): echo/772951 write(fd: 1, buf: 0x58c415257440,
count: 6) = 6

After:

     0.000 ( 0.009 ms): echo/760370 write(fd: 1, buf: , count: 6)

I noticed this with fedora:40, and bisected it to:

⬢[acme@toolbox perf-tools]$ git bisect good b257fac12f38d7f503b932313d704cee21092350 is the first bad commit commit b257fac12f38d7f503b932313d704cee21092350

Author: Howard Chu <[email protected]>
Date:   Sun Aug 25 00:33:19 2024 +0800

perf trace: Pretty print buffer data

Define TRACE_AUG_MAX_BUF in trace_augment.h data, which is the maximum buffer size we can augment. BPF will include this header too.

Print buffer in a way that's different than just printing a string, we print all the control characters in \digits (such as \0 for null, and \10 for newline, LF).

For character that has a bigger value than 127, we print the digits instead of the character itself as well.

Committer notes:

Simplified the buffer scnprintf to avoid using multiple buffers as discussed in the patch review thread.

We can't really all 'buf' args to SCA_BUF as we're collecting so far just on the sys_enter path, so we would be printing the previous 'read' arg buffer contents, not what the kernel puts there.

So instead of:
static int syscall_fmt__cmp(const void *name, const void *fmtp)
@@ -1987,8 +1989,6 @@ syscall_arg_fmt__init_array(struct syscall_arg_fmt *arg, struct tep_format_field
- else if (strstr(field->type, "char *") && strstr(field->name, "buf"))
- arg->scnprintf = SCA_BUF;

Do:

static const struct syscall_fmt syscall_fmts[] = {
+ { .name = "write", .errpid = true,
+ .arg = { [1] = { .scnprintf = SCA_BUF /* buf */, from_user = true, }, }, },

Just did a:

⬢[acme@toolbox perf-tools]$ git checkout b257fac12f38d7f503b932313d704cee21092350 Updating files: 100% (12326/12326), done.
Note: switching to 'b257fac12f38d7f503b932313d704cee21092350'.

You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch. <SNIP>
HEAD is now at b257fac12f38d7f5 perf trace: Pretty print buffer

make clean + rebuild and the returns are gone, if I do:

⬢[acme@toolbox perf-tools]$ git checkout b257fac12f38d7f503b932313d704cee21092350^ Previous HEAD position was b257fac12f38d7f5 perf trace: Pretty print buffer data HEAD is now at cb32035214b9a09d perf trace: Pretty print struct data ⬢[acme@toolbox perf-tools]$

clean + rebuild instead I get those returns back:

Now staring at it...

- Arnaldo