Skip to contentSkip to conversation

perf trace: Add more tests for BTF-augmented perf trace

PATCH Applied Howard Chu 3 participants 16 messages Dec 15, 2024 – Feb 4, 2025 perf events

lore
Share & tools
lore

Howard Chu

author
#0 lore raw

v4 cover perf trace: Add more tests for BTF-augmented perf trace

Changes in v4:
- Use if -f to check the existence of vmlinux BTF, and exit if it doesn't, so trace_test_string will not overwrite $err, and keep running the test.

Changes in v3:
- Add vmlinux BTF check, and skip the tests if it doesn't exist

v1, v2:

The previous version of the perf trace BTF general augmentation tests didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo <[email protected]> for pointing this out), this version uses bash instead of POSIX shell to pass Shellcheck.

This patch series also adds documentation for the new option --force-btf, which is used in the tests.

Howard Chu (2):
perf trace: Add tests for BTF general augmentation perf docs: Add documentation for --force-btf option

 tools/perf/Documentation/perf-trace.txt     |  5 ++
 tools/perf/tests/shell/trace_btf_general.sh | 93 +++++++++++++++++++++
 2 files changed, 98 insertions(+)
 create mode 100755 tools/perf/tests/shell/trace_btf_general.sh
Signature · 1 line
2.43.0
author
#1 lore raw

PATCH v4 1/2 perf trace: Add tests for BTF general augmentation

Currently, we only have perf trace augmentation tests for enum arguments. This patch adds tests for more general syscall arguments, such as struct pointers, strings, and buffers.

These tests utilize the perf config system to configure the perf trace output, as suggested by Arnaldo Carvalho de Melo <[email protected]>

---
 tools/perf/tests/shell/trace_btf_general.sh | 93 +++++++++++++++++++++
 1 file changed, 93 insertions(+)
 create mode 100755 tools/perf/tests/shell/trace_btf_general.sh
tools/perf/tests/shell/trace_btf_general.sh new file +93 -0
new file mode 100755
index 000000000000..005c93e6c770
@@ -0,0 +1,93 @@
+#!/bin/bash
+# perf trace BTF general tests
+# SPDX-License-Identifier: GPL-2.0
+
+err=0
+set -e
+
+. "$(dirname $0)"/lib/probe.sh
+
+file1=$(mktemp /tmp/file1_XXXX)
+file2=$(echo $file1 | sed 's/file1/file2/g')
+
+buffer="buffer content"
+perf_config_tmp=$(mktemp /tmp/.perfconfig_XXXXX)
+
+trap cleanup EXIT TERM INT HUP
+
+check_vmlinux() {
+  echo "Checking if vmlinux BTF exists"
+  if [ ! -f /sys/kernel/btf/vmlinux ]
+  then
+    echo "Skipped due to missing vmlinux BTF"
+    return 2
+  fi
+  return 0
+}
+
+trace_test_string() {
+  echo "Testing perf trace's string augmentation"
+  if ! perf trace -e renameat* --max-events=1 -- mv ${file1} ${file2} 2>&1 | \
+    grep -q -E "^mv/[0-9]+ renameat(2)?\(.*, \"${file1}\", .*, \"${file2}\", .*\) += +[0-9]+$"
+  then
+    echo "String augmentation test failed"
+    err=1
+  fi
+}
+
+trace_test_buffer() {
+  echo "Testing perf trace's buffer augmentation"
+  # echo will insert a newline (\10) at the end of the buffer
+  if ! perf trace -e write --max-events=1 -- echo "${buffer}" 2>&1 | \
+    grep -q -E "^echo/[0-9]+ write\([0-9]+, ${buffer}.*, [0-9]+\) += +[0-9]+$"
+  then
+    echo "Buffer augmentation test failed"
+    err=1
+  fi
+}
+
+trace_test_struct_btf() {
+  echo "Testing perf trace's struct augmentation"
+  if ! perf trace -e clock_nanosleep --force-btf --max-events=1 -- sleep 1 2>&1 | \
+    grep -q -E "^sleep/[0-9]+ clock_nanosleep\(0, 0, \{1,\}, 0x[0-9a-f]+\) += +[0-9]+$"
+  then
+    echo "BTF struct augmentation test failed"
+    err=1
+  fi
+}
+
+cleanup() {
+  rm -rf ${file1} ${file2} ${perf_config_tmp}
+}
+
+trap_cleanup() {
+  echo "Unexpected signal in ${FUNCNAME[1]}"
+  cleanup
+  exit 1
+}
+
+# don't overwrite user's perf config
+trace_config() {
+  export PERF_CONFIG=${perf_config_tmp}
+  perf config trace.show_arg_names=false trace.show_duration=false \
+    trace.show_timestamp=false trace.args_alignment=0
+}
+
+skip_if_no_perf_trace || exit 2
+check_vmlinux || exit 2
+
+trace_config
+
+trace_test_string
+
+if [ $err = 0 ]; then
+  trace_test_buffer
+fi
+
+if [ $err = 0 ]; then
+  trace_test_struct_btf
+fi
+
+cleanup
+
+exit $err
Signature · 1 line
2.43.0
author
#2 lore raw

PATCH v4 2/2 perf docs: Add documentation for --force-btf option

In reply to Howard Chu

The --force-btf option is intended for debugging purposes and is currently undocumented. Add documentation for it.

---
 tools/perf/Documentation/perf-trace.txt | 5 +++++
 1 file changed, 5 insertions(+)
tools/perf/Documentation/perf-trace.txt +5 -0
index 6e0cc50bbc13..fb3d2af33844 100644
@@ -241,6 +241,11 @@ the thread executes on the designated CPUs. Default is to monitor all CPUs.
 	printing using the existing 'perf trace' syscall arg beautifiers to map integer
 	arguments to strings (pid to comm, syscall id to syscall name, etc).
 
+--force-btf::
+	Use btf_dump to pretty print syscall argument data, instead of using hand-crafted pretty
+	printers. This option is intended for testing BTF integration in perf trace. btf_dump-based
+	pretty-printing serves as a fallback to hand-crafted pretty printers, as the latter can
+	better pretty-print integer flags and struct pointers.
 
 PAGEFAULTS
 ----------
Signature · 1 line
2.43.0
maintainer Applied
#3 lore raw

Re: [PATCH v4 2/2] perf docs: Add documentation for --force-btf option

Quoted 13 lines · Howard Chu

The --force-btf option is intended for debugging purposes and is currently undocumented. Add documentation for it.

---
 tools/perf/Documentation/perf-trace.txt | 5 +++++
 1 file changed, 5 insertions(+)
tools/perf/Documentation/perf-trace.txt +0 -0
index 6e0cc50bbc13..fb3d2af33844 100644

tools/perf/Documentation/perf-trace.txt the thread executes on the designated CPUs. Default is to monitor all CPUs. · line 248

Quoted 8 lines
tools/perf/Documentation/perf-trace.txt +5 -0
@@ -241,6 +241,11 @@ the thread executes on the designated CPUs. Default is to monitor all CPUs.
 	printing using the existing 'perf trace' syscall arg beautifiers to map integer
 	arguments to strings (pid to comm, syscall id to syscall name, etc).
 
+--force-btf::
+	Use btf_dump to pretty print syscall argument data, instead of using hand-crafted pretty
+	printers. This option is intended for testing BTF integration in perf trace. btf_dump-based
+	pretty-printing serves as a fallback to hand-crafted pretty printers, as the latter can
+	better pretty-print integer flags and struct pointers.

Applied, but for the casual reader the "btf_dump" reference is vague, it would be interesting to mention that it is part of libbpf and uses only what is available in the BTF format.

Then talking about the "hand-crafted" pretty printers, it is not all the time "hand crafted", but automated by all those shell scripts that convert kernel source code that doesn't get converted into BTF info (defines) to create tables that then get associated with those syscall args or even struct members.

But that is something for a followup paches, I'm applying it as is as it improves the current documentation.

- Arnaldo

maintainer Acked-by on the v4 series
#4 lore raw
In reply to Howard Chu
Quoted 15 lines · Howard Chu

Changes in v4:
- Use if -f to check the existence of vmlinux BTF, and exit if it doesn't, so trace_test_string will not overwrite $err, and keep running the test.

Changes in v3:
- Add vmlinux BTF check, and skip the tests if it doesn't exist

v1, v2:

The previous version of the perf trace BTF general augmentation tests didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo <[email protected]> for pointing this out), this version uses bash instead of POSIX shell to pass Shellcheck.

Quoted 8 lines

This patch series also adds documentation for the new option --force-btf, which is used in the tests.

Howard Chu (2):
perf trace: Add tests for BTF general augmentation perf docs: Add documentation for --force-btf option

Thanks,
Namhyung

Quoted 9 lines
 tools/perf/Documentation/perf-trace.txt     |  5 ++
 tools/perf/tests/shell/trace_btf_general.sh | 93 +++++++++++++++++++++
 2 files changed, 98 insertions(+)
 create mode 100755 tools/perf/tests/shell/trace_btf_general.sh
Signature · 1 line
2.43.0
maintainer Applied
#5 lore raw
Quoted 19 lines · Namhyung Kim
Quoted 18 lines · Howard Chu

Changes in v4:
- Use if -f to check the existence of vmlinux BTF, and exit if it doesn't, so trace_test_string will not overwrite $err, and keep running the test.

Changes in v3:
- Add vmlinux BTF check, and skip the tests if it doesn't exist

v1, v2:

The previous version of the perf trace BTF general augmentation tests didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo <[email protected]> for pointing this out), this version uses bash instead of POSIX shell to pass Shellcheck.

This patch series also adds documentation for the new option --force-btf, which is used in the tests.

Quoted 7 lines
Quoted 5 lines

Howard Chu (2):
perf trace: Add tests for BTF general augmentation perf docs: Add documentation for --force-btf option

Thanks, applied to perf-tools-next,

- Arnaldo

#6 lore raw
Quoted 22 lines · Arnaldo Carvalho de Melo
Quoted 21 lines · Namhyung Kim
Quoted 20 lines · Howard Chu

Changes in v4:
- Use if -f to check the existence of vmlinux BTF, and exit if it doesn't, so trace_test_string will not overwrite $err, and keep running the test.

Changes in v3:
- Add vmlinux BTF check, and skip the tests if it doesn't exist

v1, v2:

The previous version of the perf trace BTF general augmentation tests didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo <[email protected]> for pointing this out), this version uses bash instead of POSIX shell to pass Shellcheck.

This patch series also adds documentation for the new option --force-btf, which is used in the tests.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Howard Chu (2):
perf trace: Add tests for BTF general augmentation perf docs: Add documentation for --force-btf option

Thanks, applied to perf-tools-next,

It fails when running in parallel mode, sometimes:

109: perf trace BTF general tests : FAILED!

Then:

root@number:~# perf stat --null -r 10 perf test "BTF general"
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : FAILED!
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : FAILED!
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok

Performance counter stats for 'perf test BTF general' (10 runs):

             2.148 +- 0.293 seconds time elapsed  ( +- 13.63% )

root@number:~#

So its not just when running in paralell, anyway, its merged, we can go on from what we got there.

- Arnaldo

author
#7 lore raw

Hello Arnaldo,

Quoted 52 lines · Arnaldo Carvalho de Melo
Quoted 22 lines · Arnaldo Carvalho de Melo
Quoted 21 lines · Namhyung Kim
Quoted 20 lines · Howard Chu

Changes in v4:
- Use if -f to check the existence of vmlinux BTF, and exit if it doesn't, so trace_test_string will not overwrite $err, and keep running the test.

Changes in v3:
- Add vmlinux BTF check, and skip the tests if it doesn't exist

v1, v2:

The previous version of the perf trace BTF general augmentation tests didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo <[email protected]> for pointing this out), this version uses bash instead of POSIX shell to pass Shellcheck.

This patch series also adds documentation for the new option --force-btf, which is used in the tests.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Howard Chu (2):
perf trace: Add tests for BTF general augmentation perf docs: Add documentation for --force-btf option

Thanks, applied to perf-tools-next,

It fails when running in parallel mode, sometimes:

109: perf trace BTF general tests : FAILED!

Then:

root@number:~# perf stat --null -r 10 perf test "BTF general"
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : FAILED!
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : FAILED!
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok

Performance counter stats for 'perf test BTF general' (10 runs):

Quoted 7 lines
             2.148 +- 0.293 seconds time elapsed  ( +- 13.63% )

root@number:~#

So its not just when running in paralell, anyway, its merged, we can go on from what we got there.

Thanks, I will resolve this issue and send v5 along with the documentation changes you suggested.

Thanks,
Howard

Quoted 2 lines

- Arnaldo

#8 lore raw
Quoted 7 lines · Howard Chu
Quoted 3 lines · Arnaldo Carvalho de Melo

On Mon, Dec 23, 2024 at 04:02:43PM -0300, Arnaldo Carvalho de Melo wrote:
So its not just when running in paralell, anyway, its merged, we can go on from what we got there.

Thanks, I will resolve this issue and send v5 along with the documentation changes you suggested.

I got v4 merged meanwhile, we can go on from there, with further patches polishihg those aspects.

Thanks,

- Arnaldo

author
#9 lore raw
In reply to Arnaldo Carvalho de Melo

Hello Arnaldo,

Quoted 52 lines · Arnaldo Carvalho de Melo
Quoted 22 lines · Arnaldo Carvalho de Melo
Quoted 21 lines · Namhyung Kim
Quoted 20 lines · Howard Chu

Changes in v4:
- Use if -f to check the existence of vmlinux BTF, and exit if it doesn't, so trace_test_string will not overwrite $err, and keep running the test.

Changes in v3:
- Add vmlinux BTF check, and skip the tests if it doesn't exist

v1, v2:

The previous version of the perf trace BTF general augmentation tests didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo <[email protected]> for pointing this out), this version uses bash instead of POSIX shell to pass Shellcheck.

This patch series also adds documentation for the new option --force-btf, which is used in the tests.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Howard Chu (2):
perf trace: Add tests for BTF general augmentation perf docs: Add documentation for --force-btf option

Thanks, applied to perf-tools-next,

It fails when running in parallel mode, sometimes:

109: perf trace BTF general tests : FAILED!

Then:

root@number:~# perf stat --null -r 10 perf test "BTF general"
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : FAILED!
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : FAILED!
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok

Performance counter stats for 'perf test BTF general' (10 runs):

Quoted 7 lines
             2.148 +- 0.293 seconds time elapsed  ( +- 13.63% )

root@number:~#

So its not just when running in paralell, anyway, its merged, we can go on from what we got there.

It never fails on my machine, I think the reason is my machine is not fully-loaded. Can you please run

```
perf stat --null -r 10 perf test -vv "BTF general"
```

And provide the output?

Thanks,
Howard

Quoted 2 lines

- Arnaldo

#10 lore raw
Quoted 64 lines · Howard Chu

Hello Arnaldo,

Quoted 52 lines · Arnaldo Carvalho de Melo
Quoted 22 lines · Arnaldo Carvalho de Melo
Quoted 21 lines · Namhyung Kim
Quoted 20 lines · Howard Chu

Changes in v4:
- Use if -f to check the existence of vmlinux BTF, and exit if it doesn't, so trace_test_string will not overwrite $err, and keep running the test.

Changes in v3:
- Add vmlinux BTF check, and skip the tests if it doesn't exist

v1, v2:

The previous version of the perf trace BTF general augmentation tests didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo <[email protected]> for pointing this out), this version uses bash instead of POSIX shell to pass Shellcheck.

This patch series also adds documentation for the new option --force-btf, which is used in the tests.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Howard Chu (2):
perf trace: Add tests for BTF general augmentation perf docs: Add documentation for --force-btf option

Thanks, applied to perf-tools-next,

It fails when running in parallel mode, sometimes:

109: perf trace BTF general tests : FAILED!

Then:

root@number:~# perf stat --null -r 10 perf test "BTF general"
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : FAILED!
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : FAILED!
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok

Performance counter stats for 'perf test BTF general' (10 runs):

Quoted 7 lines
             2.148 +- 0.293 seconds time elapsed  ( +- 13.63% )

root@number:~#

So its not just when running in paralell, anyway, its merged, we can go on from what we got there.

Quoted 6 lines

It never fails on my machine, I think the reason is my machine is not fully-loaded. Can you please run

```
perf stat --null -r 10 perf test -vv "BTF general"
```

And provide the output?

root@number:~# grep -m1 'model name' /proc/cpuinfo 
model name	: Intel(R) Core(TM) i7-14700K
root@number:~#
root@number:~# perf stat --null -r 10 perf test -vv "BTF general"
106: perf trace BTF general tests:
--- start ---
test child forked, pid 47824
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
String augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests                                    : FAILED!
106: perf trace BTF general tests:
--- start ---
test child forked, pid 47842
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
String augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests                                    : FAILED!
106: perf trace BTF general tests:
--- start ---
test child forked, pid 47860
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
String augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests                                    : FAILED!
106: perf trace BTF general tests:
--- start ---
test child forked, pid 47878
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
String augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests                                    : FAILED!
106: perf trace BTF general tests:
--- start ---
test child forked, pid 47896
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
Testing perf trace's buffer augmentation
Buffer augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests                                    : FAILED!
106: perf trace BTF general tests:
--- start ---
test child forked, pid 47917
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
Testing perf trace's buffer augmentation
Buffer augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests                                    : FAILED!
106: perf trace BTF general tests:
--- start ---
test child forked, pid 47938
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
Testing perf trace's buffer augmentation
Testing perf trace's struct augmentation
---- end(0) ----
106: perf trace BTF general tests                                    : Ok
106: perf trace BTF general tests:
--- start ---
test child forked, pid 47962
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
String augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests                                    : FAILED!
106: perf trace BTF general tests:
--- start ---
test child forked, pid 47980
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
Testing perf trace's buffer augmentation
Testing perf trace's struct augmentation
---- end(0) ----
106: perf trace BTF general tests                                    : Ok
106: perf trace BTF general tests:
--- start ---
test child forked, pid 48004
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
Testing perf trace's buffer augmentation
Testing perf trace's struct augmentation
---- end(0) ----
106: perf trace BTF general tests                                    : Ok

Performance counter stats for 'perf test -vv BTF general' (10 runs):

             1.166 +- 0.272 seconds time elapsed  ( +- 23.34% )

root@number:~#

maintainer
#11 lore raw
Quoted 86 lines · Arnaldo Carvalho de Melo
Quoted 64 lines · Howard Chu

Hello Arnaldo,

Quoted 52 lines · Arnaldo Carvalho de Melo
Quoted 22 lines · Arnaldo Carvalho de Melo
Quoted 21 lines · Namhyung Kim
Quoted 20 lines · Howard Chu

Changes in v4:
- Use if -f to check the existence of vmlinux BTF, and exit if it doesn't, so trace_test_string will not overwrite $err, and keep running the test.

Changes in v3:
- Add vmlinux BTF check, and skip the tests if it doesn't exist

v1, v2:

The previous version of the perf trace BTF general augmentation tests didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo <[email protected]> for pointing this out), this version uses bash instead of POSIX shell to pass Shellcheck.

This patch series also adds documentation for the new option --force-btf, which is used in the tests.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Howard Chu (2):
perf trace: Add tests for BTF general augmentation perf docs: Add documentation for --force-btf option

Thanks, applied to perf-tools-next,

It fails when running in parallel mode, sometimes:

109: perf trace BTF general tests : FAILED!

Then:

root@number:~# perf stat --null -r 10 perf test "BTF general"
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : FAILED!
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : FAILED!
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok

Performance counter stats for 'perf test BTF general' (10 runs):

Quoted 7 lines
             2.148 +- 0.293 seconds time elapsed  ( +- 13.63% )

root@number:~#

So its not just when running in paralell, anyway, its merged, we can go on from what we got there.

Quoted 6 lines

It never fails on my machine, I think the reason is my machine is not fully-loaded. Can you please run

```
perf stat --null -r 10 perf test -vv "BTF general"
```

And provide the output?

root@number:~# grep -m1 'model name' /proc/cpuinfo 
model name	: Intel(R) Core(TM) i7-14700K
root@number:~#
root@number:~# perf stat --null -r 10 perf test -vv "BTF general"
106: perf trace BTF general tests:
--- start ---
test child forked, pid 47824
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
String augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests                                    : FAILED!
106: perf trace BTF general tests:
Quoted 8 lines
--- start ---
test child forked, pid 47842
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
String augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests                                    : FAILED!
...

I also see the failure on not fully-loaded machines (both Intel and AMD).

  $ sudo ./perf test -v BTF
  --- start ---
  test child forked, pid 525025
  Checking if vmlinux BTF exists
  Testing perf trace's string augmentation
  String augmentation test failed
  ---- end(-1) ----
  106: perf trace BTF general tests                                    : FAILED!

Thanks,
Namhyung

author
#12 lore raw

Hello Namhyung,

Quoted 109 lines · Namhyung Kim
Quoted 86 lines · Arnaldo Carvalho de Melo
Quoted 64 lines · Howard Chu

Hello Arnaldo,

Quoted 52 lines · Arnaldo Carvalho de Melo
Quoted 22 lines · Arnaldo Carvalho de Melo
Quoted 21 lines · Namhyung Kim
Quoted 20 lines · Howard Chu

Changes in v4:
- Use if -f to check the existence of vmlinux BTF, and exit if it doesn't, so trace_test_string will not overwrite $err, and keep running the test.

Changes in v3:
- Add vmlinux BTF check, and skip the tests if it doesn't exist

v1, v2:

The previous version of the perf trace BTF general augmentation tests didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo <[email protected]> for pointing this out), this version uses bash instead of POSIX shell to pass Shellcheck.

This patch series also adds documentation for the new option --force-btf, which is used in the tests.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Howard Chu (2):
perf trace: Add tests for BTF general augmentation perf docs: Add documentation for --force-btf option

Thanks, applied to perf-tools-next,

It fails when running in parallel mode, sometimes:

109: perf trace BTF general tests : FAILED!

Then:

root@number:~# perf stat --null -r 10 perf test "BTF general"
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : FAILED!
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : FAILED!
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok

Performance counter stats for 'perf test BTF general' (10 runs):

Quoted 7 lines
             2.148 +- 0.293 seconds time elapsed  ( +- 13.63% )

root@number:~#

So its not just when running in paralell, anyway, its merged, we can go on from what we got there.

Quoted 6 lines

It never fails on my machine, I think the reason is my machine is not fully-loaded. Can you please run

```
perf stat --null -r 10 perf test -vv "BTF general"
```

And provide the output?

root@number:~# grep -m1 'model name' /proc/cpuinfo
model name    : Intel(R) Core(TM) i7-14700K
root@number:~#
root@number:~# perf stat --null -r 10 perf test -vv "BTF general"
106: perf trace BTF general tests:
--- start ---
test child forked, pid 47824
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
String augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests                                    : FAILED!
106: perf trace BTF general tests:
Quoted 8 lines
--- start ---
test child forked, pid 47842
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
String augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests                                    : FAILED!
...

I also see the failure on not fully-loaded machines (both Intel and AMD).

  $ sudo ./perf test -v BTF
  --- start ---
  test child forked, pid 525025
  Checking if vmlinux BTF exists
  Testing perf trace's string augmentation
  String augmentation test failed
  ---- end(-1) ----
  106: perf trace BTF general tests                                    : FAILED!
Quoted 3 lines

Thanks,
Namhyung

Just wondering, did you and Arnaldo apply this patch:
https://lore.kernel.org/linux-perf-users/[email protected]/

This seems to solve the problem for me...

Thanks,
Howard

maintainer
#13 lore raw
Quoted 112 lines · Howard Chu

Hello Namhyung,

Quoted 109 lines · Namhyung Kim
Quoted 86 lines · Arnaldo Carvalho de Melo
Quoted 64 lines · Howard Chu

Hello Arnaldo,

Quoted 52 lines · Arnaldo Carvalho de Melo
Quoted 22 lines · Arnaldo Carvalho de Melo
Quoted 21 lines · Namhyung Kim
Quoted 20 lines · Howard Chu

Changes in v4:
- Use if -f to check the existence of vmlinux BTF, and exit if it doesn't, so trace_test_string will not overwrite $err, and keep running the test.

Changes in v3:
- Add vmlinux BTF check, and skip the tests if it doesn't exist

v1, v2:

The previous version of the perf trace BTF general augmentation tests didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo <[email protected]> for pointing this out), this version uses bash instead of POSIX shell to pass Shellcheck.

This patch series also adds documentation for the new option --force-btf, which is used in the tests.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Howard Chu (2):
perf trace: Add tests for BTF general augmentation perf docs: Add documentation for --force-btf option

Thanks, applied to perf-tools-next,

It fails when running in parallel mode, sometimes:

109: perf trace BTF general tests : FAILED!

Then:

root@number:~# perf stat --null -r 10 perf test "BTF general"
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : FAILED!
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : FAILED!
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok

Performance counter stats for 'perf test BTF general' (10 runs):

Quoted 7 lines
             2.148 +- 0.293 seconds time elapsed  ( +- 13.63% )

root@number:~#

So its not just when running in paralell, anyway, its merged, we can go on from what we got there.

Quoted 6 lines

It never fails on my machine, I think the reason is my machine is not fully-loaded. Can you please run

```
perf stat --null -r 10 perf test -vv "BTF general"
```

And provide the output?

root@number:~# grep -m1 'model name' /proc/cpuinfo
model name    : Intel(R) Core(TM) i7-14700K
root@number:~#
root@number:~# perf stat --null -r 10 perf test -vv "BTF general"
106: perf trace BTF general tests:
--- start ---
test child forked, pid 47824
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
String augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests                                    : FAILED!
106: perf trace BTF general tests:
Quoted 8 lines
--- start ---
test child forked, pid 47842
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
String augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests                                    : FAILED!
...

I also see the failure on not fully-loaded machines (both Intel and AMD).

  $ sudo ./perf test -v BTF
  --- start ---
  test child forked, pid 525025
  Checking if vmlinux BTF exists
  Testing perf trace's string augmentation
  String augmentation test failed
  ---- end(-1) ----
  106: perf trace BTF general tests                                    : FAILED!
Quoted 6 lines
Quoted 3 lines

Thanks,
Namhyung

Just wondering, did you and Arnaldo apply this patch:
https://lore.kernel.org/linux-perf-users/[email protected]/

I don't think we applied it, will add to perf-tools.

Quoted 2 lines

This seems to solve the problem for me...

But I'm not sure if it fixes the problem. IIUC the patch fixes the loading problem. And for some reason, I don't see the problem any more. But I remember this sometimes happened in the past.

Thanks,
Namhyung

author
#14 lore raw

Hello Namhyung,

Sorry for somehow missing this reply...

Quoted 121 lines · Namhyung Kim
Quoted 112 lines · Howard Chu

Hello Namhyung,

Quoted 109 lines · Namhyung Kim
Quoted 86 lines · Arnaldo Carvalho de Melo
Quoted 64 lines · Howard Chu

Hello Arnaldo,

Quoted 52 lines · Arnaldo Carvalho de Melo
Quoted 22 lines · Arnaldo Carvalho de Melo
Quoted 21 lines · Namhyung Kim
Quoted 20 lines · Howard Chu

Changes in v4:
- Use if -f to check the existence of vmlinux BTF, and exit if it doesn't, so trace_test_string will not overwrite $err, and keep running the test.

Changes in v3:
- Add vmlinux BTF check, and skip the tests if it doesn't exist

v1, v2:

The previous version of the perf trace BTF general augmentation tests didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo <[email protected]> for pointing this out), this version uses bash instead of POSIX shell to pass Shellcheck.

This patch series also adds documentation for the new option --force-btf, which is used in the tests.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Howard Chu (2):
perf trace: Add tests for BTF general augmentation perf docs: Add documentation for --force-btf option

Thanks, applied to perf-tools-next,

It fails when running in parallel mode, sometimes:

109: perf trace BTF general tests : FAILED!

Then:

root@number:~# perf stat --null -r 10 perf test "BTF general"
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : FAILED!
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : FAILED!
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok

Performance counter stats for 'perf test BTF general' (10 runs):

Quoted 7 lines
             2.148 +- 0.293 seconds time elapsed  ( +- 13.63% )

root@number:~#

So its not just when running in paralell, anyway, its merged, we can go on from what we got there.

Quoted 6 lines

It never fails on my machine, I think the reason is my machine is not fully-loaded. Can you please run

```
perf stat --null -r 10 perf test -vv "BTF general"
```

And provide the output?

root@number:~# grep -m1 'model name' /proc/cpuinfo
model name    : Intel(R) Core(TM) i7-14700K
root@number:~#
root@number:~# perf stat --null -r 10 perf test -vv "BTF general"
106: perf trace BTF general tests:
--- start ---
test child forked, pid 47824
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
String augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests                                    : FAILED!
106: perf trace BTF general tests:
Quoted 8 lines
--- start ---
test child forked, pid 47842
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
String augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests                                    : FAILED!
...

I also see the failure on not fully-loaded machines (both Intel and AMD).

  $ sudo ./perf test -v BTF
  --- start ---
  test child forked, pid 525025
  Checking if vmlinux BTF exists
  Testing perf trace's string augmentation
  String augmentation test failed
  ---- end(-1) ----
  106: perf trace BTF general tests                                    : FAILED!
Quoted 6 lines
Quoted 3 lines

Thanks,
Namhyung

Just wondering, did you and Arnaldo apply this patch:
https://lore.kernel.org/linux-perf-users/[email protected]/

Quoted 7 lines

I don't think we applied it, will add to perf-tools.

Quoted 2 lines

This seems to solve the problem for me...

But I'm not sure if it fixes the problem. IIUC the patch fixes the loading problem.

Yes. However, if the machine is unable to load the BPF program, the error message will be:

perf $ sudo ./perf test general -v
--- start ---
test child forked, pid 173990
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
String augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests : FAILED!

Which is indistinguishable from an actual string augmentation error that has a string mismatch...

Quoted 2 lines

And for some reason, I don't see the problem any more. But I remember this sometimes happened in the past.

Do you mean that perf trace breaks occasionally? This sounds urgent. Could you please provide me with your kernel version and Clang version? Thank you very much! :)

Thanks,
Howard

Quoted 4 lines

Thanks,
Namhyung

maintainer
#15 lore raw
Quoted 146 lines · Howard Chu

Hello Namhyung,

Sorry for somehow missing this reply...

Quoted 121 lines · Namhyung Kim
Quoted 112 lines · Howard Chu

Hello Namhyung,

Quoted 109 lines · Namhyung Kim
Quoted 86 lines · Arnaldo Carvalho de Melo
Quoted 64 lines · Howard Chu

Hello Arnaldo,

Quoted 52 lines · Arnaldo Carvalho de Melo
Quoted 22 lines · Arnaldo Carvalho de Melo
Quoted 21 lines · Namhyung Kim
Quoted 20 lines · Howard Chu

Changes in v4:
- Use if -f to check the existence of vmlinux BTF, and exit if it doesn't, so trace_test_string will not overwrite $err, and keep running the test.

Changes in v3:
- Add vmlinux BTF check, and skip the tests if it doesn't exist

v1, v2:

The previous version of the perf trace BTF general augmentation tests didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo <[email protected]> for pointing this out), this version uses bash instead of POSIX shell to pass Shellcheck.

This patch series also adds documentation for the new option --force-btf, which is used in the tests.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Howard Chu (2):
perf trace: Add tests for BTF general augmentation perf docs: Add documentation for --force-btf option

Thanks, applied to perf-tools-next,

It fails when running in parallel mode, sometimes:

109: perf trace BTF general tests : FAILED!

Then:

root@number:~# perf stat --null -r 10 perf test "BTF general"
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : FAILED!
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : FAILED!
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok

Performance counter stats for 'perf test BTF general' (10 runs):

Quoted 7 lines
             2.148 +- 0.293 seconds time elapsed  ( +- 13.63% )

root@number:~#

So its not just when running in paralell, anyway, its merged, we can go on from what we got there.

Quoted 6 lines

It never fails on my machine, I think the reason is my machine is not fully-loaded. Can you please run

```
perf stat --null -r 10 perf test -vv "BTF general"
```

And provide the output?

root@number:~# grep -m1 'model name' /proc/cpuinfo
model name    : Intel(R) Core(TM) i7-14700K
root@number:~#
root@number:~# perf stat --null -r 10 perf test -vv "BTF general"
106: perf trace BTF general tests:
--- start ---
test child forked, pid 47824
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
String augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests                                    : FAILED!
106: perf trace BTF general tests:
Quoted 8 lines
--- start ---
test child forked, pid 47842
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
String augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests                                    : FAILED!
...

I also see the failure on not fully-loaded machines (both Intel and AMD).

  $ sudo ./perf test -v BTF
  --- start ---
  test child forked, pid 525025
  Checking if vmlinux BTF exists
  Testing perf trace's string augmentation
  String augmentation test failed
  ---- end(-1) ----
  106: perf trace BTF general tests                                    : FAILED!
Quoted 6 lines
Quoted 3 lines

Thanks,
Namhyung

Just wondering, did you and Arnaldo apply this patch:
https://lore.kernel.org/linux-perf-users/[email protected]/

Quoted 7 lines

I don't think we applied it, will add to perf-tools.

Quoted 2 lines

This seems to solve the problem for me...

But I'm not sure if it fixes the problem. IIUC the patch fixes the loading problem.

Yes. However, if the machine is unable to load the BPF program, the error message will be:

perf $ sudo ./perf test general -v
--- start ---
test child forked, pid 173990
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
String augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests : FAILED!

Quoted 2 lines

Which is indistinguishable from an actual string augmentation error that has a string mismatch...

Ok, that's not good.

Quoted 7 lines
Quoted 3 lines

And for some reason, I don't see the problem any more. But I remember this sometimes happened in the past.

Do you mean that perf trace breaks occasionally? This sounds urgent. Could you please provide me with your kernel version and Clang version? Thank you very much! :)

Yep, sometimes it works and sometimes it failed to load the BPF.

The kernel version of this laptop is v6.10.11 and clang version is 16.0.6.

Thanks,
Namhyung