Skip to contentSkip to conversation

perf tests: Reduce inexplicable test failures

PATCH In review Adrian Hunter 4 participants 14 messages Nov 23, 2023 – Nov 27, 2023 perf events

lore
Share & tools
lore
author reviewer
#1 lore raw

PATCH 1/8 perf header: Fix segfault on build_mem_topology() error path

In reply to Adrian Hunter

Do not increase the node count unless a node has been successfully read, because it can lead to a segfault if an error occurs.

For example, if perf exceeds the open file limit in memory_node__read(), which, on a test system, could be made to happen by setting the file limit to exactly 32:

Before:

  $ ulimit -n 32
  $ perf mem record --all-user -- sleep 1
  [ perf record: Woken up 1 times to write data ]
  failed: can't open memory sysfs data
  perf: Segmentation fault
  Obtained 14 stack frames.
  perf(sighandler_dump_stack+0x48) [0x55f4b1f59558]
  /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f4ba1c42520]
  /lib/x86_64-linux-gnu/libc.so.6(free+0x1e) [0x7f4ba1ca53fe]
  perf(+0x178ff4) [0x55f4b1f48ff4]
  perf(+0x179a70) [0x55f4b1f49a70]
  perf(+0x17ef5d) [0x55f4b1f4ef5d]
  perf(+0x85c0b) [0x55f4b1e55c0b]
  perf(cmd_record+0xe1d) [0x55f4b1e5920d]
  perf(cmd_mem+0xc96) [0x55f4b1e80e56]
  perf(+0x130460) [0x55f4b1f00460]
  perf(main+0x689) [0x55f4b1e427d9]
  /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f4ba1c29d90]
  /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f4ba1c29e40]
  perf(_start+0x25) [0x55f4b1e42a25]
  Segmentation fault (core dumped)
  $

After:

  $ ulimit -n 32
  $ perf mem record --all-user -- sleep 1
  [ perf record: Woken up 1 times to write data ]
  failed: can't open memory sysfs data
  [ perf record: Captured and wrote 0.005 MB perf.data (11 samples) ]
  $
Fixes f8e502b9d1b3 ("perf header: Ensure bitmaps are freed")
---
 tools/perf/util/header.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
tools/perf/util/header.c +3 -1
index 1c687b5789c0..08cc2febabde 100644
@@ -1444,7 +1444,9 @@ static int build_mem_topology(struct memory_node **nodesp, u64 *cntp)
 			nodes = new_nodes;
 			size += 4;
 		}
-		ret = memory_node__read(&nodes[cnt++], idx);
+		ret = memory_node__read(&nodes[cnt], idx);
+		if (!ret)
+			cnt += 1;
 	}
 out:
 	closedir(dir);
Signature · 1 line
2.34.1

Adrian Hunter

author reviewer
#0 lore raw

cover perf tests: Reduce inexplicable test failures

Hi

Here are some small changes to help reduce inexplicable perf test failures.

Regards
Adrian

author reviewer
#2 lore raw

PATCH 2/8 perf tests lib: Add perf_has_symbol.sh

Some shell tests depend on finding symbols for perf itself, and fail if perf has been stripped and no debug object is available. Add helper functions to check if perf has a needed symbol. This is preparation for amending the tests themselves to be skipped if a needed symbol is not found.

The functions make use of the "Symbols" test which reads and checks symbols from a dso, perf itself by default. Note the "Symbols" test will find symbols using the same method as other perf tests, including, for example, looking in the buildid cache.

An alternative would be to prevent the needed symbols from being stripped, which seems to work with gcc's externally_visible attribute, but that attribute is not supported by clang.

Another alternative would be to use option -Wl,-E (which is already used when perf is built with perl support) which causes the linker to add all (global) symbols to the dynamic symbol table. Then the required symbols need only be made global in scope to avoid being strippable. However that goes beyond what is needed.

---
 tools/perf/tests/shell/lib/perf_has_symbol.sh | 21 +++++++++++++++++++
 1 file changed, 21 insertions(+)
 create mode 100644 tools/perf/tests/shell/lib/perf_has_symbol.sh
tools/perf/tests/shell/lib/perf_has_symbol.sh new file +21 -0
new file mode 100644
index 000000000000..5d59c32ae3e7
@@ -0,0 +1,21 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+
+perf_has_symbol()
+{
+	if perf test -vv "Symbols" 2>&1 | grep "[[:space:]]$1$"; then
+		echo "perf does have symbol '$1'"
+		return 0
+	fi
+	echo "perf does not have symbol '$1'"
+	return 1
+}
+
+skip_test_missing_symbol()
+{
+	if ! perf_has_symbol "$1" ; then
+		echo "perf is missing symbols - skipping test"
+		exit 2
+	fi
+	return 0
+}
Signature · 1 line
2.34.1
author reviewer
#3 lore raw

PATCH 3/8 perf tests: Skip pipe test if noploop symbol is missing

In reply to Adrian Hunter

perf pipe recording and injection test depends on finding symbol noploop in perf, and fails if perf has been stripped and no debug object is available. In that case, skip the test instead.

Example:

Before:

  $ strip tools/perf/perf
  $ tools/perf/perf buildid-cache -p `realpath tools/perf/perf`
  $ tools/perf/perf test -v pipe
   86: perf pipe recording and injection test                          :
  --- start ---
  test child forked, pid 47734
  [ perf record: Woken up 1 times to write data ]
  [ perf record: Captured and wrote 0.000 MB - ]
       47741    47741       -1 |perf
  [ perf record: Woken up 1 times to write data ]
  [ perf record: Captured and wrote 0.000 MB - ]
  cannot find noploop function in pipe #1
  test child finished with -1
  ---- end ----
  perf pipe recording and injection test: FAILED!

After:

  $ tools/perf/perf test -v pipe
   86: perf pipe recording and injection test                          :
  --- start ---
  test child forked, pid 48996
  perf does not have symbol 'noploop'
  perf is missing symbols - skipping test
  test child finished with -2
  ---- end ----
  perf pipe recording and injection test: Skip
---
 tools/perf/tests/shell/pipe_test.sh | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
tools/perf/tests/shell/pipe_test.sh +8 -1
index 8dd115dd35a7..a78d35d2cff0 100755
@@ -2,10 +2,17 @@
 # perf pipe recording and injection test
 # SPDX-License-Identifier: GPL-2.0
 
+shelldir=$(dirname "$0")
+# shellcheck source=lib/perf_has_symbol.sh
+. "${shelldir}"/lib/perf_has_symbol.sh
+
+sym="noploop"
+
+skip_test_missing_symbol ${sym}
+
 data=$(mktemp /tmp/perf.data.XXXXXX)
 prog="perf test -w noploop"
 task="perf"
-sym="noploop"
 
 if ! perf record -e task-clock:u -o - ${prog} | perf report -i - --task | grep ${task}; then
 	echo "cannot find the test file in the perf report"
Signature · 1 line
2.34.1
author reviewer
#4 lore raw

PATCH 4/8 perf tests: Skip record test if test_loop symbol is missing

In reply to Adrian Hunter

perf record test depends on finding symbol test_loop in perf, and fails if perf has been stripped and no debug object is available. In that case, skip the test instead.

Example:

Note, building with perl support adds option -Wl,-E which causes the linker to add all (global) symbols to the dynamic symbol table. So the test_loop symbol, being global, does not get stripped unless NO_LIBPERL=1

Before:

  $ make NO_LIBPERL=1 -C tools/perf >/dev/null 2>&1
  $ strip tools/perf/perf
  $ tools/perf/perf buildid-cache -p `realpath tools/perf/perf`
  $ tools/perf/perf test -v 'record tests'
   91: perf record tests                                               :
  --- start ---
  test child forked, pid 118750
  Basic --per-thread mode test
  Per-thread record [Failed missing output]
  Register capture test
  Register capture test [Success]
  Basic --system-wide mode test
  System-wide record [Skipped not supported]
  Basic target workload test
  Workload record [Failed missing output]
  test child finished with -1
  ---- end ----
  perf record tests: FAILED!

After:

  $ tools/perf/perf test -v 'record tests'
   91: perf record tests                                               :
  --- start ---
  test child forked, pid 120025
  perf does not have symbol 'test_loop'
  perf is missing symbols - skipping test
  test child finished with -2
  ---- end ----
  perf record tests: Skip
---
 tools/perf/tests/shell/record.sh | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
tools/perf/tests/shell/record.sh +7 -1
index 29443b8e8876..1838b76e2282 100755
@@ -8,10 +8,16 @@ shelldir=$(dirname "$0")
 # shellcheck source=lib/waiting.sh
 . "${shelldir}"/lib/waiting.sh
 
+# shellcheck source=lib/perf_has_symbol.sh
+. "${shelldir}"/lib/perf_has_symbol.sh
+
+testsym="test_loop"
+
+skip_test_missing_symbol ${testsym}
+
 err=0
 perfdata=$(mktemp /tmp/__perf_test.perf.data.XXXXX)
 testprog="perf test -w thloop"
-testsym="test_loop"
 
 cleanup() {
   rm -rf "${perfdata}"
Signature · 1 line
2.34.1
author reviewer
#5 lore raw

PATCH 5/8 perf tests: Skip Arm64 callgraphs test if leafloop symbol is missing

In reply to Adrian Hunter

The test "Check Arm64 callgraphs are complete in fp mode" depends on finding symbol leafloop in perf, and fails if perf has been stripped and no debug object is available. In that case, skip the test instead.

---
 tools/perf/tests/shell/test_arm_callgraph_fp.sh | 6 ++++++
 1 file changed, 6 insertions(+)
tools/perf/tests/shell/test_arm_callgraph_fp.sh +6 -0
index 66dfdfdad553..e342e6c8aa50 100755
@@ -2,8 +2,14 @@
 # Check Arm64 callgraphs are complete in fp mode
 # SPDX-License-Identifier: GPL-2.0
 
+shelldir=$(dirname "$0")
+# shellcheck source=lib/perf_has_symbol.sh
+. "${shelldir}"/lib/perf_has_symbol.sh
+
 lscpu | grep -q "aarch64" || exit 2
 
+skip_test_missing_symbol leafloop
+
 PERF_DATA=$(mktemp /tmp/__perf_test.perf.data.XXXXX)
 TEST_PROGRAM="perf test -w leafloop"
 
Signature · 1 line
2.34.1
author reviewer
#6 lore raw

PATCH 6/8 perf tests: Skip branch stack sampling test if brstack_bench symbol is missing

In reply to Adrian Hunter

The test "Check branch stack sampling" depends on finding symbol brstack_bench (and several others) in perf, and fails if perf has been stripped and no debug object is available. In that case, skip the test instead.

Example:

Before:

  $ strip tools/perf/perf
  $ tools/perf/perf buildid-cache -p `realpath tools/perf/perf`
  $ tools/perf/perf test -v 'branch stack sampling'
  112: Check branch stack sampling                                     :
  --- start ---
  test child forked, pid 123741
  Testing user branch stack sampling
  + grep -E -m1 ^brstack_bench\+[^ ]*/brstack_foo\+[^ ]*/IND_CALL/.*$ /tmp/__perf_test.program.5Dz1U/perf.script
  + cleanup
  + rm -rf /tmp/__perf_test.program.5Dz1U
  test child finished with -1
  ---- end ----
  Check branch stack sampling: FAILED!

After:

  $ tools/perf/perf test -v 'branch stack sampling'
  112: Check branch stack sampling                                     :
  --- start ---
  test child forked, pid 125157
  perf does not have symbol 'brstack_bench'
  perf is missing symbols - skipping test
  test child finished with -2
  ---- end ----
  Check branch stack sampling: Skip
---
 tools/perf/tests/shell/test_brstack.sh | 6 ++++++
 1 file changed, 6 insertions(+)
tools/perf/tests/shell/test_brstack.sh +6 -0
index 09908d71c994..5f14d0cb013f 100755
@@ -4,6 +4,10 @@
 # SPDX-License-Identifier: GPL-2.0
 # German Gomez <[email protected]>, 2022
 
+shelldir=$(dirname "$0")
+# shellcheck source=lib/perf_has_symbol.sh
+. "${shelldir}"/lib/perf_has_symbol.sh
+
 # skip the test if the hardware doesn't support branch stack sampling
 # and if the architecture doesn't support filter types: any,save_type,u
 if ! perf record -o- --no-buildid --branch-filter any,save_type,u -- true > /dev/null 2>&1 ; then
@@ -11,6 +15,8 @@ if ! perf record -o- --no-buildid --branch-filter any,save_type,u -- true > /dev
 	exit 2
 fi
 
+skip_test_missing_symbol brstack_bench
+
 TMPDIR=$(mktemp -d /tmp/__perf_test.program.XXXXX)
 TESTPROG="perf test -w brstack"
 
Signature · 1 line
2.34.1
author reviewer
#7 lore raw

PATCH 7/8 perf tests: Make data symbol test wait for perf to start

In reply to Adrian Hunter

The perf data symbol test waits 1 second for perf to run and collect data, which may be too little if perf takes a long time to start up, which has been noticed on systems with many CPUs. Use existing wait_for_perf_to_start helper to wait for perf to start.

---
 tools/perf/tests/shell/test_data_symbol.sh | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
tools/perf/tests/shell/test_data_symbol.sh +9 -2
index 69bb6fe86c50..e50e54e94f6f 100755
@@ -4,6 +4,10 @@
 # SPDX-License-Identifier: GPL-2.0
 # Leo Yan <[email protected]>, 2022
 
+shelldir=$(dirname "$0")
+# shellcheck source=lib/waiting.sh
+. "${shelldir}"/lib/waiting.sh
+
 skip_if_no_mem_event() {
 	perf mem record -e list 2>&1 | grep -E -q 'available' && return 0
 	return 2
@@ -13,6 +17,7 @@ skip_if_no_mem_event || exit 2
 
 TEST_PROGRAM="perf test -w datasym"
 PERF_DATA=$(mktemp /tmp/__perf_test.perf.data.XXXXX)
+ERR_FILE=$(mktemp /tmp/__perf_test.stderr.XXXXX)
 
 check_result() {
 	# The memory report format is as below:
@@ -50,13 +55,15 @@ echo "Recording workload..."
 # specific CPU and test in per-CPU mode.
 is_amd=$(grep -E -c 'vendor_id.*AuthenticAMD' /proc/cpuinfo)
 if (($is_amd >= 1)); then
-	perf mem record -o ${PERF_DATA} -C 0 -- taskset -c 0 $TEST_PROGRAM &
+	perf mem record -vvv -o ${PERF_DATA} -C 0 -- taskset -c 0 $TEST_PROGRAM 2>"${ERR_FILE}" &
 else
-	perf mem record --all-user -o ${PERF_DATA} -- $TEST_PROGRAM &
+	perf mem record -vvv --all-user -o ${PERF_DATA} -- $TEST_PROGRAM 2>"${ERR_FILE}" &
 fi
 
 PERFPID=$!
 
+wait_for_perf_to_start ${PERFPID} "${ERR_FILE}"
+
 sleep 1
 
 kill $PERFPID
Signature · 1 line
2.34.1
author reviewer
#8 lore raw

PATCH 8/8 perf tests: Skip data symbol test if buf1 symbol is missing

In reply to Adrian Hunter

perf data symbol test depends on finding symbol buf1 in perf, and fails if perf has been stripped and no debug object is available. In that case, skip the test instead.

Example:

Before:

  $ strip tools/perf/perf
  $ tools/perf/perf buildid-cache -p `realpath tools/perf/perf`
  $ tools/perf/perf test -v 'data symbol'
  113: Test data symbol                                                :
  --- start ---
  test child forked, pid 125646
  Recording workload...
  [ perf record: Woken up 3 times to write data ]
  [ perf record: Captured and wrote 0.577 MB /tmp/__perf_test.perf.data.Jhbdp (7794 samples) ]
  Cleaning up files...
  test child finished with -1
  ---- end ----
  Test data symbol: FAILED!

After:

  $ tools/perf/perf test -v 'data symbol'
  113: Test data symbol                                                :
  --- start ---
  test child forked, pid 125747
  perf does not have symbol 'buf1'
  perf is missing symbols - skipping test
  test child finished with -2
  ---- end ----
  Test data symbol: Skip
---
 tools/perf/tests/shell/test_data_symbol.sh | 5 +++++
 1 file changed, 5 insertions(+)
tools/perf/tests/shell/test_data_symbol.sh +5 -0
index e50e54e94f6f..3dfa91832aa8 100755
@@ -8,6 +8,9 @@ shelldir=$(dirname "$0")
 # shellcheck source=lib/waiting.sh
 . "${shelldir}"/lib/waiting.sh
 
+# shellcheck source=lib/perf_has_symbol.sh
+. "${shelldir}"/lib/perf_has_symbol.sh
+
 skip_if_no_mem_event() {
 	perf mem record -e list 2>&1 | grep -E -q 'available' && return 0
 	return 2
@@ -15,6 +18,8 @@ skip_if_no_mem_event() {
 
 skip_if_no_mem_event || exit 2
 
+skip_test_missing_symbol buf1
+
 TEST_PROGRAM="perf test -w datasym"
 PERF_DATA=$(mktemp /tmp/__perf_test.perf.data.XXXXX)
 ERR_FILE=$(mktemp /tmp/__perf_test.stderr.XXXXX)
Signature · 1 line
2.34.1
#10 lore raw
Quoted 10 lines · Arnaldo Carvalho de Melo

Em Thu, Nov 23, 2023 at 09:58:40AM +0200, Adrian Hunter escreveu:

Quoted 1 line

Here are some small changes to help reduce inexplicable perf test failures.

Looks ok, applying to perf-tools-next, had to remove a patch adding a new test, by Kan Liang, then reapply it on top of your series, git managed to sort it out:

Hi Arnaldo,

I didn’t find this in either of these places:
https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/log/?h=perf-tools-next
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/log/?h=perf-tools-next

Sorry if I missed checking correct location. Can you please point to git link ?

Thanks
Athira

Quoted 5 lines

https://lore.kernel.org/r/[email protected]

- Arnaldo

#11 lore raw

Em Mon, Nov 27, 2023 at 12:49:13PM +0530, Athira Rajeev escreveu:

Quoted 11 lines
Quoted 10 lines · Arnaldo Carvalho de Melo

Em Thu, Nov 23, 2023 at 09:58:40AM +0200, Adrian Hunter escreveu:

Quoted 1 line

Here are some small changes to help reduce inexplicable perf test failures.

Looks ok, applying to perf-tools-next, had to remove a patch adding a new test, by Kan Liang, then reapply it on top of your series, git managed to sort it out:

Quoted 5 lines

Hi Arnaldo,

I didn’t find this in either of these places:
https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/log/?h=perf-tools-next
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/log/?h=perf-tools-next

I was still testing it on the containers setup, it is at:

https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/log/?h=tmp.perf-tools-next
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/log/?h=tmp.perf-tools-next

And will move to the perf-tools-next branches soon, I just collected some more acked-by for some patches in the tmp branch and will push the updated branch.

- Arnaldo

Quoted 10 lines

Sorry if I missed checking correct location. Can you please point to git link ?

Thanks
Athira

Quoted 5 lines

https://lore.kernel.org/r/[email protected]

- Arnaldo

Signature · 1 line
- Arnaldo
#12 lore raw

Em Mon, Nov 27, 2023 at 10:23:45AM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 6 lines

Em Mon, Nov 27, 2023 at 12:49:13PM +0530, Athira Rajeev escreveu:

Quoted 4 lines
Quoted 1 line · Arnaldo Carvalho de Melo

I didn’t find this in either of these places:
https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/log/?h=perf-tools-next
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/log/?h=perf-tools-next

Quoted 8 lines

I was still testing it on the containers setup, it is at:

https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/log/?h=tmp.perf-tools-next
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/log/?h=tmp.perf-tools-next

And will move to the perf-tools-next branches soon, I just collected some more acked-by for some patches in the tmp branch and will push the updated branch.

Done.

- Arnaldo