perf trace: Add more tests for BTF-augmented perf trace
v4 cover perf trace: Add more tests for BTF-augmented perf trace
Changes in v4:
- Use if -f to check the existence of vmlinux BTF, and exit if it doesn't, so trace_test_string will not overwrite $err, and keep running the test.
Changes in v3:
- Add vmlinux BTF check, and skip the tests if it doesn't exist
v1, v2:
The previous version of the perf trace BTF general augmentation tests didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo <[email protected]> for pointing this out), this version uses bash instead of POSIX shell to pass Shellcheck.
This patch series also adds documentation for the new option --force-btf, which is used in the tests.
Howard Chu (2):
perf trace: Add tests for BTF general augmentation perf docs: Add documentation for --force-btf option
tools/perf/Documentation/perf-trace.txt | 5 ++ tools/perf/tests/shell/trace_btf_general.sh | 93 +++++++++++++++++++++ 2 files changed, 98 insertions(+) create mode 100755 tools/perf/tests/shell/trace_btf_general.sh
Signature · 1 line
2.43.0
PATCH v4 1/2 perf trace: Add tests for BTF general augmentation
Currently, we only have perf trace augmentation tests for enum arguments. This patch adds tests for more general syscall arguments, such as struct pointers, strings, and buffers.
These tests utilize the perf config system to configure the perf trace output, as suggested by Arnaldo Carvalho de Melo <[email protected]>
--- tools/perf/tests/shell/trace_btf_general.sh | 93 +++++++++++++++++++++ 1 file changed, 93 insertions(+) create mode 100755 tools/perf/tests/shell/trace_btf_general.sh
tools/perf/tests/shell/trace_btf_general.sh new file +93 -0
@@ -0,0 +1,93 @@ +#!/bin/bash +# perf trace BTF general tests +# SPDX-License-Identifier: GPL-2.0 + +err=0 +set -e + +. "$(dirname $0)"/lib/probe.sh + +file1=$(mktemp /tmp/file1_XXXX) +file2=$(echo $file1 | sed 's/file1/file2/g') + +buffer="buffer content" +perf_config_tmp=$(mktemp /tmp/.perfconfig_XXXXX) + +trap cleanup EXIT TERM INT HUP + +check_vmlinux() { + echo "Checking if vmlinux BTF exists" + if [ ! -f /sys/kernel/btf/vmlinux ] + then + echo "Skipped due to missing vmlinux BTF" + return 2 + fi + return 0 +} + +trace_test_string() { + echo "Testing perf trace's string augmentation" + if ! perf trace -e renameat* --max-events=1 -- mv ${file1} ${file2} 2>&1 | \ + grep -q -E "^mv/[0-9]+ renameat(2)?\(.*, \"${file1}\", .*, \"${file2}\", .*\) += +[0-9]+$" + then + echo "String augmentation test failed" + err=1 + fi +} + +trace_test_buffer() { + echo "Testing perf trace's buffer augmentation" + # echo will insert a newline (\10) at the end of the buffer + if ! perf trace -e write --max-events=1 -- echo "${buffer}" 2>&1 | \ + grep -q -E "^echo/[0-9]+ write\([0-9]+, ${buffer}.*, [0-9]+\) += +[0-9]+$" + then + echo "Buffer augmentation test failed" + err=1 + fi +} + +trace_test_struct_btf() { + echo "Testing perf trace's struct augmentation" + if ! perf trace -e clock_nanosleep --force-btf --max-events=1 -- sleep 1 2>&1 | \ + grep -q -E "^sleep/[0-9]+ clock_nanosleep\(0, 0, \{1,\}, 0x[0-9a-f]+\) += +[0-9]+$" + then + echo "BTF struct augmentation test failed" + err=1 + fi +} + +cleanup() { + rm -rf ${file1} ${file2} ${perf_config_tmp} +} + +trap_cleanup() { + echo "Unexpected signal in ${FUNCNAME[1]}" + cleanup + exit 1 +} + +# don't overwrite user's perf config +trace_config() { + export PERF_CONFIG=${perf_config_tmp} + perf config trace.show_arg_names=false trace.show_duration=false \ + trace.show_timestamp=false trace.args_alignment=0 +} + +skip_if_no_perf_trace || exit 2 +check_vmlinux || exit 2 + +trace_config + +trace_test_string + +if [ $err = 0 ]; then + trace_test_buffer +fi + +if [ $err = 0 ]; then + trace_test_struct_btf +fi + +cleanup + +exit $err
Signature · 1 line
2.43.0
PATCH v4 2/2 perf docs: Add documentation for --force-btf option
In reply to Howard ChuThe --force-btf option is intended for debugging purposes and is currently undocumented. Add documentation for it.
--- tools/perf/Documentation/perf-trace.txt | 5 +++++ 1 file changed, 5 insertions(+)
tools/perf/Documentation/perf-trace.txt +5 -0
@@ -241,6 +241,11 @@ the thread executes on the designated CPUs. Default is to monitor all CPUs. printing using the existing 'perf trace' syscall arg beautifiers to map integer arguments to strings (pid to comm, syscall id to syscall name, etc). +--force-btf:: + Use btf_dump to pretty print syscall argument data, instead of using hand-crafted pretty + printers. This option is intended for testing BTF integration in perf trace. btf_dump-based + pretty-printing serves as a fallback to hand-crafted pretty printers, as the latter can + better pretty-print integer flags and struct pointers. PAGEFAULTS ----------
Signature · 1 line
2.43.0
Re: [PATCH v4 2/2] perf docs: Add documentation for --force-btf option
Quoted 13 lines · Howard Chu
The --force-btf option is intended for debugging purposes and is currently undocumented. Add documentation for it.
Signed-off-by Howard Chu <[email protected]>--- tools/perf/Documentation/perf-trace.txt | 5 +++++ 1 file changed, 5 insertions(+)tools/perf/Documentation/perf-trace.txt +0
-0
tools/perf/Documentation/perf-trace.txt the thread executes on the designated CPUs. Default is to monitor all CPUs. · line 248
Quoted 8 lines
tools/perf/Documentation/perf-trace.txt +5
-0@@ -241,6 +241,11 @@ the thread executes on the designated CPUs. Default is to monitor all CPUs. printing using the existing 'perf trace' syscall arg beautifiers to map integer arguments to strings (pid to comm, syscall id to syscall name, etc). +--force-btf:: + Use btf_dump to pretty print syscall argument data, instead of using hand-crafted pretty + printers. This option is intended for testing BTF integration in perf trace. btf_dump-based + pretty-printing serves as a fallback to hand-crafted pretty printers, as the latter can + better pretty-print integer flags and struct pointers.
Applied, but for the casual reader the "btf_dump" reference is vague, it would be interesting to mention that it is part of libbpf and uses only what is available in the BTF format.
Then talking about the "hand-crafted" pretty printers, it is not all the time "hand crafted", but automated by all those shell scripts that convert kernel source code that doesn't get converted into BTF info (defines) to create tables that then get associated with those syscall args or even struct members.
But that is something for a followup paches, I'm applying it as is as it improves the current documentation.
- Arnaldo
Quoted 15 lines · Howard Chu
Changes in v4:
- Use if -f to check the existence of vmlinux BTF, and exit if it doesn't, sotrace_test_stringwill not overwrite $err, and keep running the test.Changes in v3:
- Add vmlinux BTF check, and skip the tests if it doesn't existv1, v2:
The previous version of the perf trace BTF general augmentation tests didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo <[email protected]> for pointing this out), this version uses bash instead of POSIX shell to pass Shellcheck.
Quoted 8 lines
This patch series also adds documentation for the new option --force-btf, which is used in the tests.
Howard Chu (2):
perf trace: Add tests for BTF general augmentation perf docs: Add documentation for --force-btf option
Thanks,
Namhyung
Quoted 9 lines
tools/perf/Documentation/perf-trace.txt | 5 ++ tools/perf/tests/shell/trace_btf_general.sh | 93 +++++++++++++++++++++ 2 files changed, 98 insertions(+) create mode 100755 tools/perf/tests/shell/trace_btf_general.shSignature · 1 line
2.43.0
Quoted 19 lines · Namhyung Kim
Quoted 18 lines · Howard Chu
Changes in v4:
- Use if -f to check the existence of vmlinux BTF, and exit if it doesn't, sotrace_test_stringwill not overwrite $err, and keep running the test.Changes in v3:
- Add vmlinux BTF check, and skip the tests if it doesn't existv1, v2:
The previous version of the perf trace BTF general augmentation tests didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo <[email protected]> for pointing this out), this version uses bash instead of POSIX shell to pass Shellcheck.
This patch series also adds documentation for the new option --force-btf, which is used in the tests.
Quoted 7 lines
Quoted 5 lines
Howard Chu (2):
perf trace: Add tests for BTF general augmentation perf docs: Add documentation for --force-btf optionAcked-by Namhyung Kim <[email protected]>
Thanks, applied to perf-tools-next,
- Arnaldo
Arnaldo Carvalho de Melo
maintainerQuoted 22 lines · Arnaldo Carvalho de Melo
Quoted 21 lines · Namhyung Kim
Quoted 20 lines · Howard Chu
Changes in v4:
- Use if -f to check the existence of vmlinux BTF, and exit if it doesn't, sotrace_test_stringwill not overwrite $err, and keep running the test.Changes in v3:
- Add vmlinux BTF check, and skip the tests if it doesn't existv1, v2:
The previous version of the perf trace BTF general augmentation tests didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo <[email protected]> for pointing this out), this version uses bash instead of POSIX shell to pass Shellcheck.
This patch series also adds documentation for the new option --force-btf, which is used in the tests.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Howard Chu (2):
perf trace: Add tests for BTF general augmentation perf docs: Add documentation for --force-btf optionAcked-by Namhyung Kim <[email protected]>Thanks, applied to perf-tools-next,
It fails when running in parallel mode, sometimes:
109: perf trace BTF general tests : FAILED!
Then:
root@number:~# perf stat --null -r 10 perf test "BTF general" 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : FAILED! 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : FAILED! 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok
Performance counter stats for 'perf test BTF general' (10 runs):
2.148 +- 0.293 seconds time elapsed ( +- 13.63% )
root@number:~#
So its not just when running in paralell, anyway, its merged, we can go on from what we got there.
- Arnaldo
Hello Arnaldo,
Quoted 52 lines · Arnaldo Carvalho de Melo
Quoted 22 lines · Arnaldo Carvalho de Melo
Quoted 21 lines · Namhyung Kim
Quoted 20 lines · Howard Chu
Changes in v4:
- Use if -f to check the existence of vmlinux BTF, and exit if it doesn't, sotrace_test_stringwill not overwrite $err, and keep running the test.Changes in v3:
- Add vmlinux BTF check, and skip the tests if it doesn't existv1, v2:
The previous version of the perf trace BTF general augmentation tests didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo <[email protected]> for pointing this out), this version uses bash instead of POSIX shell to pass Shellcheck.
This patch series also adds documentation for the new option --force-btf, which is used in the tests.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Howard Chu (2):
perf trace: Add tests for BTF general augmentation perf docs: Add documentation for --force-btf optionAcked-by Namhyung Kim <[email protected]>Thanks, applied to perf-tools-next,
It fails when running in parallel mode, sometimes:
109: perf trace BTF general tests : FAILED!
Then:
root@number:~# perf stat --null -r 10 perf test "BTF general" 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : FAILED! 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : FAILED! 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : OkPerformance counter stats for 'perf test BTF general' (10 runs):
Quoted 7 lines
2.148 +- 0.293 seconds time elapsed ( +- 13.63% )root@number:~#
So its not just when running in paralell, anyway, its merged, we can go on from what we got there.
Thanks, I will resolve this issue and send v5 along with the documentation changes you suggested.
Thanks,
Howard
Quoted 2 lines
- Arnaldo
Arnaldo Carvalho de Melo
maintainerQuoted 7 lines · Howard Chu
Quoted 3 lines · Arnaldo Carvalho de Melo
On Mon, Dec 23, 2024 at 04:02:43PM -0300, Arnaldo Carvalho de Melo wrote:
So its not just when running in paralell, anyway, its merged, we can go on from what we got there.Thanks, I will resolve this issue and send v5 along with the documentation changes you suggested.
I got v4 merged meanwhile, we can go on from there, with further patches polishihg those aspects.
Thanks,
- Arnaldo
Hello Arnaldo,
Quoted 52 lines · Arnaldo Carvalho de Melo
Quoted 22 lines · Arnaldo Carvalho de Melo
Quoted 21 lines · Namhyung Kim
Quoted 20 lines · Howard Chu
Changes in v4:
- Use if -f to check the existence of vmlinux BTF, and exit if it doesn't, sotrace_test_stringwill not overwrite $err, and keep running the test.Changes in v3:
- Add vmlinux BTF check, and skip the tests if it doesn't existv1, v2:
The previous version of the perf trace BTF general augmentation tests didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo <[email protected]> for pointing this out), this version uses bash instead of POSIX shell to pass Shellcheck.
This patch series also adds documentation for the new option --force-btf, which is used in the tests.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Howard Chu (2):
perf trace: Add tests for BTF general augmentation perf docs: Add documentation for --force-btf optionAcked-by Namhyung Kim <[email protected]>Thanks, applied to perf-tools-next,
It fails when running in parallel mode, sometimes:
109: perf trace BTF general tests : FAILED!
Then:
root@number:~# perf stat --null -r 10 perf test "BTF general" 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : FAILED! 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : FAILED! 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : OkPerformance counter stats for 'perf test BTF general' (10 runs):
Quoted 7 lines
2.148 +- 0.293 seconds time elapsed ( +- 13.63% )root@number:~#
So its not just when running in paralell, anyway, its merged, we can go on from what we got there.
It never fails on my machine, I think the reason is my machine is not fully-loaded. Can you please run
``` perf stat --null -r 10 perf test -vv "BTF general" ```
And provide the output?
Thanks,
Howard
Quoted 2 lines
- Arnaldo
Arnaldo Carvalho de Melo
maintainerQuoted 64 lines · Howard Chu
Hello Arnaldo,
Quoted 52 lines · Arnaldo Carvalho de Melo
Quoted 22 lines · Arnaldo Carvalho de Melo
Quoted 21 lines · Namhyung Kim
Quoted 20 lines · Howard Chu
Changes in v4:
- Use if -f to check the existence of vmlinux BTF, and exit if it doesn't, sotrace_test_stringwill not overwrite $err, and keep running the test.Changes in v3:
- Add vmlinux BTF check, and skip the tests if it doesn't existv1, v2:
The previous version of the perf trace BTF general augmentation tests didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo <[email protected]> for pointing this out), this version uses bash instead of POSIX shell to pass Shellcheck.
This patch series also adds documentation for the new option --force-btf, which is used in the tests.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Howard Chu (2):
perf trace: Add tests for BTF general augmentation perf docs: Add documentation for --force-btf optionAcked-by Namhyung Kim <[email protected]>Thanks, applied to perf-tools-next,
It fails when running in parallel mode, sometimes:
109: perf trace BTF general tests : FAILED!
Then:
root@number:~# perf stat --null -r 10 perf test "BTF general" 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : FAILED! 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : FAILED! 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : OkPerformance counter stats for 'perf test BTF general' (10 runs):
Quoted 7 lines
2.148 +- 0.293 seconds time elapsed ( +- 13.63% )root@number:~#
So its not just when running in paralell, anyway, its merged, we can go on from what we got there.
Quoted 6 lines
It never fails on my machine, I think the reason is my machine is not fully-loaded. Can you please run
``` perf stat --null -r 10 perf test -vv "BTF general" ```And provide the output?
root@number:~# grep -m1 'model name' /proc/cpuinfo model name : Intel(R) Core(TM) i7-14700K root@number:~#
root@number:~# perf stat --null -r 10 perf test -vv "BTF general" 106: perf trace BTF general tests: --- start --- test child forked, pid 47824 Checking if vmlinux BTF exists Testing perf trace's string augmentation String augmentation test failed ---- end(-1) ---- 106: perf trace BTF general tests : FAILED! 106: perf trace BTF general tests: --- start --- test child forked, pid 47842 Checking if vmlinux BTF exists Testing perf trace's string augmentation String augmentation test failed ---- end(-1) ---- 106: perf trace BTF general tests : FAILED! 106: perf trace BTF general tests: --- start --- test child forked, pid 47860 Checking if vmlinux BTF exists Testing perf trace's string augmentation String augmentation test failed ---- end(-1) ---- 106: perf trace BTF general tests : FAILED! 106: perf trace BTF general tests: --- start --- test child forked, pid 47878 Checking if vmlinux BTF exists Testing perf trace's string augmentation String augmentation test failed ---- end(-1) ---- 106: perf trace BTF general tests : FAILED! 106: perf trace BTF general tests: --- start --- test child forked, pid 47896 Checking if vmlinux BTF exists Testing perf trace's string augmentation Testing perf trace's buffer augmentation Buffer augmentation test failed ---- end(-1) ---- 106: perf trace BTF general tests : FAILED! 106: perf trace BTF general tests: --- start --- test child forked, pid 47917 Checking if vmlinux BTF exists Testing perf trace's string augmentation Testing perf trace's buffer augmentation Buffer augmentation test failed ---- end(-1) ---- 106: perf trace BTF general tests : FAILED! 106: perf trace BTF general tests: --- start --- test child forked, pid 47938 Checking if vmlinux BTF exists Testing perf trace's string augmentation Testing perf trace's buffer augmentation Testing perf trace's struct augmentation ---- end(0) ---- 106: perf trace BTF general tests : Ok 106: perf trace BTF general tests: --- start --- test child forked, pid 47962 Checking if vmlinux BTF exists Testing perf trace's string augmentation String augmentation test failed ---- end(-1) ---- 106: perf trace BTF general tests : FAILED! 106: perf trace BTF general tests: --- start --- test child forked, pid 47980 Checking if vmlinux BTF exists Testing perf trace's string augmentation Testing perf trace's buffer augmentation Testing perf trace's struct augmentation ---- end(0) ---- 106: perf trace BTF general tests : Ok 106: perf trace BTF general tests: --- start --- test child forked, pid 48004 Checking if vmlinux BTF exists Testing perf trace's string augmentation Testing perf trace's buffer augmentation Testing perf trace's struct augmentation ---- end(0) ---- 106: perf trace BTF general tests : Ok
Performance counter stats for 'perf test -vv BTF general' (10 runs):
1.166 +- 0.272 seconds time elapsed ( +- 23.34% )
root@number:~#
Namhyung Kim
maintainerQuoted 86 lines · Arnaldo Carvalho de Melo
Quoted 64 lines · Howard Chu
Hello Arnaldo,
Quoted 52 lines · Arnaldo Carvalho de Melo
Quoted 22 lines · Arnaldo Carvalho de Melo
Quoted 21 lines · Namhyung Kim
Quoted 20 lines · Howard Chu
Changes in v4:
- Use if -f to check the existence of vmlinux BTF, and exit if it doesn't, sotrace_test_stringwill not overwrite $err, and keep running the test.Changes in v3:
- Add vmlinux BTF check, and skip the tests if it doesn't existv1, v2:
The previous version of the perf trace BTF general augmentation tests didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo <[email protected]> for pointing this out), this version uses bash instead of POSIX shell to pass Shellcheck.
This patch series also adds documentation for the new option --force-btf, which is used in the tests.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Howard Chu (2):
perf trace: Add tests for BTF general augmentation perf docs: Add documentation for --force-btf optionAcked-by Namhyung Kim <[email protected]>Thanks, applied to perf-tools-next,
It fails when running in parallel mode, sometimes:
109: perf trace BTF general tests : FAILED!
Then:
root@number:~# perf stat --null -r 10 perf test "BTF general" 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : FAILED! 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : FAILED! 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : OkPerformance counter stats for 'perf test BTF general' (10 runs):
Quoted 7 lines
2.148 +- 0.293 seconds time elapsed ( +- 13.63% )root@number:~#
So its not just when running in paralell, anyway, its merged, we can go on from what we got there.
Quoted 6 lines
It never fails on my machine, I think the reason is my machine is not fully-loaded. Can you please run
``` perf stat --null -r 10 perf test -vv "BTF general" ```And provide the output?
root@number:~# grep -m1 'model name' /proc/cpuinfo model name : Intel(R) Core(TM) i7-14700K root@number:~#root@number:~# perf stat --null -r 10 perf test -vv "BTF general" 106: perf trace BTF general tests: --- start --- test child forked, pid 47824 Checking if vmlinux BTF exists Testing perf trace's string augmentation String augmentation test failed ---- end(-1) ---- 106: perf trace BTF general tests : FAILED! 106: perf trace BTF general tests:
Quoted 8 lines
--- start --- test child forked, pid 47842 Checking if vmlinux BTF exists Testing perf trace's string augmentation String augmentation test failed ---- end(-1) ---- 106: perf trace BTF general tests : FAILED! ...
I also see the failure on not fully-loaded machines (both Intel and AMD).
$ sudo ./perf test -v BTF --- start --- test child forked, pid 525025 Checking if vmlinux BTF exists Testing perf trace's string augmentation String augmentation test failed ---- end(-1) ---- 106: perf trace BTF general tests : FAILED!
Thanks,
Namhyung
Hello Namhyung,
Quoted 109 lines · Namhyung Kim
Quoted 86 lines · Arnaldo Carvalho de Melo
Quoted 64 lines · Howard Chu
Hello Arnaldo,
Quoted 52 lines · Arnaldo Carvalho de Melo
Quoted 22 lines · Arnaldo Carvalho de Melo
Quoted 21 lines · Namhyung Kim
Quoted 20 lines · Howard Chu
Changes in v4:
- Use if -f to check the existence of vmlinux BTF, and exit if it doesn't, sotrace_test_stringwill not overwrite $err, and keep running the test.Changes in v3:
- Add vmlinux BTF check, and skip the tests if it doesn't existv1, v2:
The previous version of the perf trace BTF general augmentation tests didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo <[email protected]> for pointing this out), this version uses bash instead of POSIX shell to pass Shellcheck.
This patch series also adds documentation for the new option --force-btf, which is used in the tests.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Howard Chu (2):
perf trace: Add tests for BTF general augmentation perf docs: Add documentation for --force-btf optionAcked-by Namhyung Kim <[email protected]>Thanks, applied to perf-tools-next,
It fails when running in parallel mode, sometimes:
109: perf trace BTF general tests : FAILED!
Then:
root@number:~# perf stat --null -r 10 perf test "BTF general" 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : FAILED! 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : FAILED! 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : OkPerformance counter stats for 'perf test BTF general' (10 runs):
Quoted 7 lines
2.148 +- 0.293 seconds time elapsed ( +- 13.63% )root@number:~#
So its not just when running in paralell, anyway, its merged, we can go on from what we got there.
Quoted 6 lines
It never fails on my machine, I think the reason is my machine is not fully-loaded. Can you please run
``` perf stat --null -r 10 perf test -vv "BTF general" ```And provide the output?
root@number:~# grep -m1 'model name' /proc/cpuinfo model name : Intel(R) Core(TM) i7-14700K root@number:~#root@number:~# perf stat --null -r 10 perf test -vv "BTF general" 106: perf trace BTF general tests: --- start --- test child forked, pid 47824 Checking if vmlinux BTF exists Testing perf trace's string augmentation String augmentation test failed ---- end(-1) ---- 106: perf trace BTF general tests : FAILED! 106: perf trace BTF general tests:Quoted 8 lines
--- start --- test child forked, pid 47842 Checking if vmlinux BTF exists Testing perf trace's string augmentation String augmentation test failed ---- end(-1) ---- 106: perf trace BTF general tests : FAILED! ...I also see the failure on not fully-loaded machines (both Intel and AMD).
$ sudo ./perf test -v BTF --- start --- test child forked, pid 525025 Checking if vmlinux BTF exists Testing perf trace's string augmentation String augmentation test failed ---- end(-1) ---- 106: perf trace BTF general tests : FAILED!
Quoted 3 lines
Thanks,
Namhyung
Just wondering, did you and Arnaldo apply this patch:
https://lore.kernel.org/linux-perf-users/[email protected]/
This seems to solve the problem for me...
Thanks,
Howard
Namhyung Kim
maintainerQuoted 112 lines · Howard Chu
Hello Namhyung,
Quoted 109 lines · Namhyung Kim
Quoted 86 lines · Arnaldo Carvalho de Melo
Quoted 64 lines · Howard Chu
Hello Arnaldo,
Quoted 52 lines · Arnaldo Carvalho de Melo
Quoted 22 lines · Arnaldo Carvalho de Melo
Quoted 21 lines · Namhyung Kim
Quoted 20 lines · Howard Chu
Changes in v4:
- Use if -f to check the existence of vmlinux BTF, and exit if it doesn't, sotrace_test_stringwill not overwrite $err, and keep running the test.Changes in v3:
- Add vmlinux BTF check, and skip the tests if it doesn't existv1, v2:
The previous version of the perf trace BTF general augmentation tests didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo <[email protected]> for pointing this out), this version uses bash instead of POSIX shell to pass Shellcheck.
This patch series also adds documentation for the new option --force-btf, which is used in the tests.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Howard Chu (2):
perf trace: Add tests for BTF general augmentation perf docs: Add documentation for --force-btf optionAcked-by Namhyung Kim <[email protected]>Thanks, applied to perf-tools-next,
It fails when running in parallel mode, sometimes:
109: perf trace BTF general tests : FAILED!
Then:
root@number:~# perf stat --null -r 10 perf test "BTF general" 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : FAILED! 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : FAILED! 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : OkPerformance counter stats for 'perf test BTF general' (10 runs):
Quoted 7 lines
2.148 +- 0.293 seconds time elapsed ( +- 13.63% )root@number:~#
So its not just when running in paralell, anyway, its merged, we can go on from what we got there.
Quoted 6 lines
It never fails on my machine, I think the reason is my machine is not fully-loaded. Can you please run
``` perf stat --null -r 10 perf test -vv "BTF general" ```And provide the output?
root@number:~# grep -m1 'model name' /proc/cpuinfo model name : Intel(R) Core(TM) i7-14700K root@number:~#root@number:~# perf stat --null -r 10 perf test -vv "BTF general" 106: perf trace BTF general tests: --- start --- test child forked, pid 47824 Checking if vmlinux BTF exists Testing perf trace's string augmentation String augmentation test failed ---- end(-1) ---- 106: perf trace BTF general tests : FAILED! 106: perf trace BTF general tests:Quoted 8 lines
--- start --- test child forked, pid 47842 Checking if vmlinux BTF exists Testing perf trace's string augmentation String augmentation test failed ---- end(-1) ---- 106: perf trace BTF general tests : FAILED! ...I also see the failure on not fully-loaded machines (both Intel and AMD).
$ sudo ./perf test -v BTF --- start --- test child forked, pid 525025 Checking if vmlinux BTF exists Testing perf trace's string augmentation String augmentation test failed ---- end(-1) ---- 106: perf trace BTF general tests : FAILED!
Quoted 6 lines
Quoted 3 lines
Thanks,
NamhyungJust wondering, did you and Arnaldo apply this patch:
https://lore.kernel.org/linux-perf-users/[email protected]/
I don't think we applied it, will add to perf-tools.
Quoted 2 lines
This seems to solve the problem for me...
But I'm not sure if it fixes the problem. IIUC the patch fixes the loading problem. And for some reason, I don't see the problem any more. But I remember this sometimes happened in the past.
Thanks,
Namhyung
Hello Namhyung,
Sorry for somehow missing this reply...
Quoted 121 lines · Namhyung Kim
Quoted 112 lines · Howard Chu
Hello Namhyung,
Quoted 109 lines · Namhyung Kim
Quoted 86 lines · Arnaldo Carvalho de Melo
Quoted 64 lines · Howard Chu
Hello Arnaldo,
Quoted 52 lines · Arnaldo Carvalho de Melo
Quoted 22 lines · Arnaldo Carvalho de Melo
Quoted 21 lines · Namhyung Kim
Quoted 20 lines · Howard Chu
Changes in v4:
- Use if -f to check the existence of vmlinux BTF, and exit if it doesn't, sotrace_test_stringwill not overwrite $err, and keep running the test.Changes in v3:
- Add vmlinux BTF check, and skip the tests if it doesn't existv1, v2:
The previous version of the perf trace BTF general augmentation tests didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo <[email protected]> for pointing this out), this version uses bash instead of POSIX shell to pass Shellcheck.
This patch series also adds documentation for the new option --force-btf, which is used in the tests.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Howard Chu (2):
perf trace: Add tests for BTF general augmentation perf docs: Add documentation for --force-btf optionAcked-by Namhyung Kim <[email protected]>Thanks, applied to perf-tools-next,
It fails when running in parallel mode, sometimes:
109: perf trace BTF general tests : FAILED!
Then:
root@number:~# perf stat --null -r 10 perf test "BTF general" 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : FAILED! 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : FAILED! 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : OkPerformance counter stats for 'perf test BTF general' (10 runs):
Quoted 7 lines
2.148 +- 0.293 seconds time elapsed ( +- 13.63% )root@number:~#
So its not just when running in paralell, anyway, its merged, we can go on from what we got there.
Quoted 6 lines
It never fails on my machine, I think the reason is my machine is not fully-loaded. Can you please run
``` perf stat --null -r 10 perf test -vv "BTF general" ```And provide the output?
root@number:~# grep -m1 'model name' /proc/cpuinfo model name : Intel(R) Core(TM) i7-14700K root@number:~#root@number:~# perf stat --null -r 10 perf test -vv "BTF general" 106: perf trace BTF general tests: --- start --- test child forked, pid 47824 Checking if vmlinux BTF exists Testing perf trace's string augmentation String augmentation test failed ---- end(-1) ---- 106: perf trace BTF general tests : FAILED! 106: perf trace BTF general tests:Quoted 8 lines
--- start --- test child forked, pid 47842 Checking if vmlinux BTF exists Testing perf trace's string augmentation String augmentation test failed ---- end(-1) ---- 106: perf trace BTF general tests : FAILED! ...I also see the failure on not fully-loaded machines (both Intel and AMD).
$ sudo ./perf test -v BTF --- start --- test child forked, pid 525025 Checking if vmlinux BTF exists Testing perf trace's string augmentation String augmentation test failed ---- end(-1) ---- 106: perf trace BTF general tests : FAILED!Quoted 6 lines
Quoted 3 lines
Thanks,
NamhyungJust wondering, did you and Arnaldo apply this patch:
https://lore.kernel.org/linux-perf-users/[email protected]/
Quoted 7 lines
I don't think we applied it, will add to perf-tools.
Quoted 2 lines
This seems to solve the problem for me...
But I'm not sure if it fixes the problem. IIUC the patch fixes the loading problem.
Yes. However, if the machine is unable to load the BPF program, the error message will be:
perf $ sudo ./perf test general -v
--- start ---
test child forked, pid 173990
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
String augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests : FAILED!
Which is indistinguishable from an actual string augmentation error that has a string mismatch...
Quoted 2 lines
And for some reason, I don't see the problem any more. But I remember this sometimes happened in the past.
Do you mean that perf trace breaks occasionally? This sounds urgent. Could you please provide me with your kernel version and Clang version? Thank you very much! :)
Thanks,
Howard
Quoted 4 lines
Thanks,
Namhyung
Namhyung Kim
maintainerQuoted 146 lines · Howard Chu
Hello Namhyung,
Sorry for somehow missing this reply...
Quoted 121 lines · Namhyung Kim
Quoted 112 lines · Howard Chu
Hello Namhyung,
Quoted 109 lines · Namhyung Kim
Quoted 86 lines · Arnaldo Carvalho de Melo
Quoted 64 lines · Howard Chu
Hello Arnaldo,
Quoted 52 lines · Arnaldo Carvalho de Melo
Quoted 22 lines · Arnaldo Carvalho de Melo
Quoted 21 lines · Namhyung Kim
Quoted 20 lines · Howard Chu
Changes in v4:
- Use if -f to check the existence of vmlinux BTF, and exit if it doesn't, sotrace_test_stringwill not overwrite $err, and keep running the test.Changes in v3:
- Add vmlinux BTF check, and skip the tests if it doesn't existv1, v2:
The previous version of the perf trace BTF general augmentation tests didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo <[email protected]> for pointing this out), this version uses bash instead of POSIX shell to pass Shellcheck.
This patch series also adds documentation for the new option --force-btf, which is used in the tests.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Howard Chu (2):
perf trace: Add tests for BTF general augmentation perf docs: Add documentation for --force-btf optionAcked-by Namhyung Kim <[email protected]>Thanks, applied to perf-tools-next,
It fails when running in parallel mode, sometimes:
109: perf trace BTF general tests : FAILED!
Then:
root@number:~# perf stat --null -r 10 perf test "BTF general" 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : FAILED! 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : FAILED! 109: perf trace BTF general tests : Ok 109: perf trace BTF general tests : OkPerformance counter stats for 'perf test BTF general' (10 runs):
Quoted 7 lines
2.148 +- 0.293 seconds time elapsed ( +- 13.63% )root@number:~#
So its not just when running in paralell, anyway, its merged, we can go on from what we got there.
Quoted 6 lines
It never fails on my machine, I think the reason is my machine is not fully-loaded. Can you please run
``` perf stat --null -r 10 perf test -vv "BTF general" ```And provide the output?
root@number:~# grep -m1 'model name' /proc/cpuinfo model name : Intel(R) Core(TM) i7-14700K root@number:~#root@number:~# perf stat --null -r 10 perf test -vv "BTF general" 106: perf trace BTF general tests: --- start --- test child forked, pid 47824 Checking if vmlinux BTF exists Testing perf trace's string augmentation String augmentation test failed ---- end(-1) ---- 106: perf trace BTF general tests : FAILED! 106: perf trace BTF general tests:Quoted 8 lines
--- start --- test child forked, pid 47842 Checking if vmlinux BTF exists Testing perf trace's string augmentation String augmentation test failed ---- end(-1) ---- 106: perf trace BTF general tests : FAILED! ...I also see the failure on not fully-loaded machines (both Intel and AMD).
$ sudo ./perf test -v BTF --- start --- test child forked, pid 525025 Checking if vmlinux BTF exists Testing perf trace's string augmentation String augmentation test failed ---- end(-1) ---- 106: perf trace BTF general tests : FAILED!Quoted 6 lines
Quoted 3 lines
Thanks,
NamhyungJust wondering, did you and Arnaldo apply this patch:
https://lore.kernel.org/linux-perf-users/[email protected]/Quoted 7 lines
I don't think we applied it, will add to perf-tools.
Quoted 2 lines
This seems to solve the problem for me...
But I'm not sure if it fixes the problem. IIUC the patch fixes the loading problem.
Yes. However, if the machine is unable to load the BPF program, the error message will be:
perf $ sudo ./perf test general -v
--- start ---
test child forked, pid 173990
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
String augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests : FAILED!
Quoted 2 lines
Which is indistinguishable from an actual string augmentation error that has a string mismatch...
Ok, that's not good.
Quoted 7 lines
Quoted 3 lines
And for some reason, I don't see the problem any more. But I remember this sometimes happened in the past.
Do you mean that perf trace breaks occasionally? This sounds urgent. Could you please provide me with your kernel version and Clang version? Thank you very much! :)
Yep, sometimes it works and sometimes it failed to load the BPF.
The kernel version of this laptop is v6.10.11 and clang version is 16.0.6.
Thanks,
Namhyung