perf tests: Reduce inexplicable test failures
Adrian Hunter
reviewercover perf tests: Reduce inexplicable test failures
Hi
Here are some small changes to help reduce inexplicable perf test failures.
Regards
Adrian
Adrian Hunter
reviewerPATCH 1/8 perf header: Fix segfault on build_mem_topology() error path
Do not increase the node count unless a node has been successfully read, because it can lead to a segfault if an error occurs.
For example, if perf exceeds the open file limit in memory_node__read(), which, on a test system, could be made to happen by setting the file limit to exactly 32:
Before:
$ ulimit -n 32 $ perf mem record --all-user -- sleep 1 [ perf record: Woken up 1 times to write data ] failed: can't open memory sysfs data perf: Segmentation fault Obtained 14 stack frames. perf(sighandler_dump_stack+0x48) [0x55f4b1f59558] /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f4ba1c42520] /lib/x86_64-linux-gnu/libc.so.6(free+0x1e) [0x7f4ba1ca53fe] perf(+0x178ff4) [0x55f4b1f48ff4] perf(+0x179a70) [0x55f4b1f49a70] perf(+0x17ef5d) [0x55f4b1f4ef5d] perf(+0x85c0b) [0x55f4b1e55c0b] perf(cmd_record+0xe1d) [0x55f4b1e5920d] perf(cmd_mem+0xc96) [0x55f4b1e80e56] perf(+0x130460) [0x55f4b1f00460] perf(main+0x689) [0x55f4b1e427d9] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f4ba1c29d90] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f4ba1c29e40] perf(_start+0x25) [0x55f4b1e42a25] Segmentation fault (core dumped) $
After:
$ ulimit -n 32 $ perf mem record --all-user -- sleep 1 [ perf record: Woken up 1 times to write data ] failed: can't open memory sysfs data [ perf record: Captured and wrote 0.005 MB perf.data (11 samples) ] $
--- tools/perf/util/header.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
tools/perf/util/header.c +3 -1
@@ -1444,7 +1444,9 @@ static int build_mem_topology(struct memory_node **nodesp, u64 *cntp) nodes = new_nodes; size +=4 ; } - ret =memory_node__read (&nodes[cnt++], idx); + ret =memory_node__read (&nodes[cnt], idx); +if (!ret) + cnt +=1 ; } out:closedir (dir);
Signature · 1 line
2.34.1
Adrian Hunter
reviewerPATCH 2/8 perf tests lib: Add perf_has_symbol.sh
In reply to Adrian HunterSome shell tests depend on finding symbols for perf itself, and fail if perf has been stripped and no debug object is available. Add helper functions to check if perf has a needed symbol. This is preparation for amending the tests themselves to be skipped if a needed symbol is not found.
The functions make use of the "Symbols" test which reads and checks symbols from a dso, perf itself by default. Note the "Symbols" test will find symbols using the same method as other perf tests, including, for example, looking in the buildid cache.
An alternative would be to prevent the needed symbols from being stripped, which seems to work with gcc's externally_visible attribute, but that attribute is not supported by clang.
Another alternative would be to use option -Wl,-E (which is already used when perf is built with perl support) which causes the linker to add all (global) symbols to the dynamic symbol table. Then the required symbols need only be made global in scope to avoid being strippable. However that goes beyond what is needed.
--- tools/perf/tests/shell/lib/perf_has_symbol.sh | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 tools/perf/tests/shell/lib/perf_has_symbol.sh
tools/perf/tests/shell/lib/perf_has_symbol.sh new file +21 -0
@@ -0,0 +1,21 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 + +perf_has_symbol() +{ + if perf test -vv "Symbols" 2>&1 | grep "[[:space:]]$1$"; then + echo "perf does have symbol '$1'" + return 0 + fi + echo "perf does not have symbol '$1'" + return 1 +} + +skip_test_missing_symbol() +{ + if ! perf_has_symbol "$1" ; then + echo "perf is missing symbols - skipping test" + exit 2 + fi + return 0 +}
Signature · 1 line
2.34.1
Adrian Hunter
reviewerPATCH 3/8 perf tests: Skip pipe test if noploop symbol is missing
In reply to Adrian Hunterperf pipe recording and injection test depends on finding symbol noploop in perf, and fails if perf has been stripped and no debug object is available. In that case, skip the test instead.
Example:
Before:
$ strip tools/perf/perf
$ tools/perf/perf buildid-cache -p `realpath tools/perf/perf`
$ tools/perf/perf test -v pipe
86: perf pipe recording and injection test :
--- start ---
test child forked, pid 47734
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.000 MB - ]
47741 47741 -1 |perf
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.000 MB - ]
cannot find noploop function in pipe #1
test child finished with -1
---- end ----
perf pipe recording and injection test: FAILED!After:
$ tools/perf/perf test -v pipe 86: perf pipe recording and injection test : --- start --- test child forked, pid 48996 perf does not have symbol 'noploop' perf is missing symbols - skipping test test child finished with -2 ---- end ---- perf pipe recording and injection test: Skip
--- tools/perf/tests/shell/pipe_test.sh | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
tools/perf/tests/shell/pipe_test.sh +8 -1
@@ -2,10 +2,17 @@ # perf pipe recording and injection test # SPDX-License-Identifier: GPL-2.0 +shelldir=$(dirname "$0") +# shellcheck source=lib/perf_has_symbol.sh +. "${shelldir}"/lib/perf_has_symbol.sh + +sym="noploop" + +skip_test_missing_symbol ${sym} + data=$(mktemp /tmp/perf.data.XXXXXX) prog="perf test -w noploop" task="perf" -sym="noploop" if ! perf record -e task-clock:u -o - ${prog} | perf report -i - --task | grep ${task}; then echo "cannot find the test file in the perf report"
Signature · 1 line
2.34.1
Adrian Hunter
reviewerPATCH 4/8 perf tests: Skip record test if test_loop symbol is missing
In reply to Adrian Hunterperf record test depends on finding symbol test_loop in perf, and fails if perf has been stripped and no debug object is available. In that case, skip the test instead.
Example:
Note, building with perl support adds option -Wl,-E which causes the linker to add all (global) symbols to the dynamic symbol table. So the test_loop symbol, being global, does not get stripped unless NO_LIBPERL=1
Before:
$ make NO_LIBPERL=1 -C tools/perf >/dev/null 2>&1 $ strip tools/perf/perf $ tools/perf/perf buildid-cache -p `realpath tools/perf/perf` $ tools/perf/perf test -v 'record tests' 91: perf record tests : --- start --- test child forked, pid 118750 Basic --per-thread mode test Per-thread record [Failed missing output] Register capture test Register capture test [Success] Basic --system-wide mode test System-wide record [Skipped not supported] Basic target workload test Workload record [Failed missing output] test child finished with -1 ---- end ---- perf record tests: FAILED!
After:
$ tools/perf/perf test -v 'record tests' 91: perf record tests : --- start --- test child forked, pid 120025 perf does not have symbol 'test_loop' perf is missing symbols - skipping test test child finished with -2 ---- end ---- perf record tests: Skip
--- tools/perf/tests/shell/record.sh | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
tools/perf/tests/shell/record.sh +7 -1
@@ -8,10 +8,16 @@ shelldir=$(dirname "$0") # shellcheck source=lib/waiting.sh . "${shelldir}"/lib/waiting.sh +# shellcheck source=lib/perf_has_symbol.sh +. "${shelldir}"/lib/perf_has_symbol.sh + +testsym="test_loop" + +skip_test_missing_symbol ${testsym} + err=0 perfdata=$(mktemp /tmp/__perf_test.perf.data.XXXXX) testprog="perf test -w thloop" -testsym="test_loop" cleanup() { rm -rf "${perfdata}"
Signature · 1 line
2.34.1
Adrian Hunter
reviewerPATCH 5/8 perf tests: Skip Arm64 callgraphs test if leafloop symbol is missing
In reply to Adrian HunterThe test "Check Arm64 callgraphs are complete in fp mode" depends on finding symbol leafloop in perf, and fails if perf has been stripped and no debug object is available. In that case, skip the test instead.
--- tools/perf/tests/shell/test_arm_callgraph_fp.sh | 6 ++++++ 1 file changed, 6 insertions(+)
tools/perf/tests/shell/test_arm_callgraph_fp.sh +6 -0
@@ -2,8 +2,14 @@ # Check Arm64 callgraphs are complete in fp mode # SPDX-License-Identifier: GPL-2.0 +shelldir=$(dirname "$0") +# shellcheck source=lib/perf_has_symbol.sh +. "${shelldir}"/lib/perf_has_symbol.sh + lscpu | grep -q "aarch64" || exit 2 +skip_test_missing_symbol leafloop + PERF_DATA=$(mktemp /tmp/__perf_test.perf.data.XXXXX) TEST_PROGRAM="perf test -w leafloop"
Signature · 1 line
2.34.1
Adrian Hunter
reviewerPATCH 6/8 perf tests: Skip branch stack sampling test if brstack_bench symbol is missing
In reply to Adrian HunterThe test "Check branch stack sampling" depends on finding symbol brstack_bench (and several others) in perf, and fails if perf has been stripped and no debug object is available. In that case, skip the test instead.
Example:
Before:
$ strip tools/perf/perf $ tools/perf/perf buildid-cache -p `realpath tools/perf/perf` $ tools/perf/perf test -v 'branch stack sampling' 112: Check branch stack sampling : --- start --- test child forked, pid 123741 Testing user branch stack sampling + grep -E -m1 ^brstack_bench\+[^ ]*/brstack_foo\+[^ ]*/IND_CALL/.*$ /tmp/__perf_test.program.5Dz1U/perf.script + cleanup + rm -rf /tmp/__perf_test.program.5Dz1U test child finished with -1 ---- end ---- Check branch stack sampling: FAILED!
After:
$ tools/perf/perf test -v 'branch stack sampling' 112: Check branch stack sampling : --- start --- test child forked, pid 125157 perf does not have symbol 'brstack_bench' perf is missing symbols - skipping test test child finished with -2 ---- end ---- Check branch stack sampling: Skip
--- tools/perf/tests/shell/test_brstack.sh | 6 ++++++ 1 file changed, 6 insertions(+)
tools/perf/tests/shell/test_brstack.sh +6 -0
@@ -4,6 +4,10 @@ # SPDX-License-Identifier: GPL-2.0 # German Gomez <[email protected]>, 2022 +shelldir=$(dirname "$0") +# shellcheck source=lib/perf_has_symbol.sh +. "${shelldir}"/lib/perf_has_symbol.sh + # skip the test if the hardware doesn't support branch stack sampling # and if the architecture doesn't support filter types: any,save_type,u if ! perf record -o- --no-buildid --branch-filter any,save_type,u -- true > /dev/null 2>&1 ; then
@@ -11,6 +15,8 @@ if ! perf record -o- --no-buildid --branch-filter any,save_type,u -- true > /dev exit 2 fi +skip_test_missing_symbol brstack_bench + TMPDIR=$(mktemp -d /tmp/__perf_test.program.XXXXX) TESTPROG="perf test -w brstack"
Signature · 1 line
2.34.1
Adrian Hunter
reviewerPATCH 7/8 perf tests: Make data symbol test wait for perf to start
In reply to Adrian HunterThe perf data symbol test waits 1 second for perf to run and collect data, which may be too little if perf takes a long time to start up, which has been noticed on systems with many CPUs. Use existing wait_for_perf_to_start helper to wait for perf to start.
--- tools/perf/tests/shell/test_data_symbol.sh | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
tools/perf/tests/shell/test_data_symbol.sh +9 -2
@@ -4,6 +4,10 @@ # SPDX-License-Identifier: GPL-2.0 # Leo Yan <[email protected]>, 2022 +shelldir=$(dirname "$0") +# shellcheck source=lib/waiting.sh +. "${shelldir}"/lib/waiting.sh + skip_if_no_mem_event() { perf mem record -e list 2>&1 | grep -E -q 'available' && return 0 return 2
@@ -13,6 +17,7 @@ skip_if_no_mem_event || exit 2 TEST_PROGRAM="perf test -w datasym" PERF_DATA=$(mktemp /tmp/__perf_test.perf.data.XXXXX) +ERR_FILE=$(mktemp /tmp/__perf_test.stderr.XXXXX) check_result() { # The memory report format is as below:
@@ -50,13 +55,15 @@ echo "Recording workload..." # specific CPU and test in per-CPU mode. is_amd=$(grep -E -c 'vendor_id.*AuthenticAMD' /proc/cpuinfo) if (($is_amd >= 1)); then - perf mem record -o ${PERF_DATA} -C 0 -- taskset -c 0 $TEST_PROGRAM & + perf mem record -vvv -o ${PERF_DATA} -C 0 -- taskset -c 0 $TEST_PROGRAM 2>"${ERR_FILE}" & else - perf mem record --all-user -o ${PERF_DATA} -- $TEST_PROGRAM & + perf mem record -vvv --all-user -o ${PERF_DATA} -- $TEST_PROGRAM 2>"${ERR_FILE}" & fi PERFPID=$! +wait_for_perf_to_start ${PERFPID} "${ERR_FILE}" + sleep 1 kill $PERFPID
Signature · 1 line
2.34.1
Adrian Hunter
reviewerPATCH 8/8 perf tests: Skip data symbol test if buf1 symbol is missing
In reply to Adrian Hunterperf data symbol test depends on finding symbol buf1 in perf, and fails if perf has been stripped and no debug object is available. In that case, skip the test instead.
Example:
Before:
$ strip tools/perf/perf $ tools/perf/perf buildid-cache -p `realpath tools/perf/perf` $ tools/perf/perf test -v 'data symbol' 113: Test data symbol : --- start --- test child forked, pid 125646 Recording workload... [ perf record: Woken up 3 times to write data ] [ perf record: Captured and wrote 0.577 MB /tmp/__perf_test.perf.data.Jhbdp (7794 samples) ] Cleaning up files... test child finished with -1 ---- end ---- Test data symbol: FAILED!
After:
$ tools/perf/perf test -v 'data symbol' 113: Test data symbol : --- start --- test child forked, pid 125747 perf does not have symbol 'buf1' perf is missing symbols - skipping test test child finished with -2 ---- end ---- Test data symbol: Skip
--- tools/perf/tests/shell/test_data_symbol.sh | 5 +++++ 1 file changed, 5 insertions(+)
tools/perf/tests/shell/test_data_symbol.sh +5 -0
@@ -8,6 +8,9 @@ shelldir=$(dirname "$0") # shellcheck source=lib/waiting.sh . "${shelldir}"/lib/waiting.sh +# shellcheck source=lib/perf_has_symbol.sh +. "${shelldir}"/lib/perf_has_symbol.sh + skip_if_no_mem_event() { perf mem record -e list 2>&1 | grep -E -q 'available' && return 0 return 2
@@ -15,6 +18,8 @@ skip_if_no_mem_event() { skip_if_no_mem_event || exit 2 +skip_test_missing_symbol buf1 + TEST_PROGRAM="perf test -w datasym" PERF_DATA=$(mktemp /tmp/__perf_test.perf.data.XXXXX) ERR_FILE=$(mktemp /tmp/__perf_test.stderr.XXXXX)
Signature · 1 line
2.34.1
Arnaldo Carvalho de Melo
maintainerEm Thu, Nov 23, 2023 at 09:58:40AM +0200, Adrian Hunter escreveu:
Quoted 1 line
Here are some small changes to help reduce inexplicable perf test failures.
Looks ok, applying to perf-tools-next, had to remove a patch adding a new test, by Kan Liang, then reapply it on top of your series, git managed to sort it out:
https://lore.kernel.org/r/[email protected]
- Arnaldo
Quoted 10 lines · Arnaldo Carvalho de Melo
Em Thu, Nov 23, 2023 at 09:58:40AM +0200, Adrian Hunter escreveu:
Quoted 1 line
Here are some small changes to help reduce inexplicable perf test failures.
Looks ok, applying to perf-tools-next, had to remove a patch adding a new test, by Kan Liang, then reapply it on top of your series, git managed to sort it out:
Hi Arnaldo,
I didn’t find this in either of these places:
https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/log/?h=perf-tools-next
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/log/?h=perf-tools-next
Sorry if I missed checking correct location. Can you please point to git link ?
Thanks
Athira
Quoted 5 lines
https://lore.kernel.org/r/[email protected]
- Arnaldo
Arnaldo Carvalho de Melo
maintainerEm Mon, Nov 27, 2023 at 12:49:13PM +0530, Athira Rajeev escreveu:
Quoted 11 lines
Quoted 10 lines · Arnaldo Carvalho de Melo
Em Thu, Nov 23, 2023 at 09:58:40AM +0200, Adrian Hunter escreveu:
Quoted 1 line
Here are some small changes to help reduce inexplicable perf test failures.
Looks ok, applying to perf-tools-next, had to remove a patch adding a new test, by Kan Liang, then reapply it on top of your series, git managed to sort it out:
Quoted 5 lines
Hi Arnaldo,
I didn’t find this in either of these places:
https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/log/?h=perf-tools-next
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/log/?h=perf-tools-next
I was still testing it on the containers setup, it is at:
https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/log/?h=tmp.perf-tools-next
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/log/?h=tmp.perf-tools-next
And will move to the perf-tools-next branches soon, I just collected some more acked-by for some patches in the tmp branch and will push the updated branch.
- Arnaldo
Quoted 10 lines
Sorry if I missed checking correct location. Can you please point to git link ?
Thanks
AthiraQuoted 5 lines
https://lore.kernel.org/r/[email protected]
- Arnaldo
Signature · 1 line
- Arnaldo
Arnaldo Carvalho de Melo
maintainerEm Mon, Nov 27, 2023 at 10:23:45AM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 6 lines
Em Mon, Nov 27, 2023 at 12:49:13PM +0530, Athira Rajeev escreveu:
Quoted 4 lines
Quoted 1 line · Arnaldo Carvalho de Melo
I didn’t find this in either of these places:
https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/log/?h=perf-tools-next
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/log/?h=perf-tools-next
Quoted 8 lines
I was still testing it on the containers setup, it is at:
https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/log/?h=tmp.perf-tools-next
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/log/?h=tmp.perf-tools-nextAnd will move to the perf-tools-next branches soon, I just collected some more acked-by for some patches in the tmp branch and will push the updated branch.
Done.
- Arnaldo
Quoted 7 lines · Adrian Hunter
Hi
Here are some small changes to help reduce inexplicable perf test failures.
Regards
Adrian
Thanks for doing this!
Thanks,
Ian