Skip to contentSkip to conversation

perf arm-spe: extend Arm SPE test script with regression testing

PATCH In review German Gomez 2 participants 5 messages Nov 12, 2021 – Nov 23, 2021 perf events

lore
Share & tools
lore
#0 lore raw

cover perf arm-spe: extend Arm SPE test script with regression testing

This patch extends the test_arm_spe.sh test script in order to tests for regressions in the decoding flow of Arm SPE samples.

At the time of writing, perf-inject is not working for Arm SPE. In order to test/apply this patch, [1] needs to be resolved first.

[1] https://lore.kernel.org/all/[email protected]/T/#m06a8cc68f7b971b6eb10d11aa6e7385a7f6be29e

German Gomez (1):
perf arm-spe: extend Arm SPE test script with regression testing

 tools/perf/tests/shell/test_arm_spe.sh  |  32 ++++++++++++++++++++++++
 tools/perf/tests/shell/test_arm_spe.tgz | Bin 0 -> 253541 bytes
 2 files changed, 32 insertions(+)
 create mode 100644 tools/perf/tests/shell/test_arm_spe.tgz
Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/1 perf arm-spe: extend Arm SPE test script with regression testing

Extend the test_arm_spe.sh script to test for regressions in the decoding flow of Arm SPE samples. In order to support the tests, a set of perf.data files has been generated offline and is being hosted under tools/perf/tests/shell/test_arm_spe.tgz:

(See perf report --header-only to see the actual commands used)

- test_arm_spe.tgz
`- arm_spe_decode_contextidr.data
`- arm_spe_decode_contextidr.data.inject
`- arm_spe_decode_switch_events.data
`- arm_spe_decode_switch_events.data.inject

The purpose of each file is as follows:

- arm_spe_decode_contextidr.data is used to test decoding when SPE traces contain CONTEXT packets (for pid/tid tracking).
- arm_spe_decode_switch_events.data is used to test decoding when SPE traces _do not_ contain CONTEXT packets.
- arm_spe_*.data.inject were generated using perf-inject and the above files as inputs.

During testing, events are decoded from the arm_spe_*.data files, and the result are compared against the hosted arm_spe_*.data.inject files. If there are any differences, the test fails.

---
 tools/perf/tests/shell/test_arm_spe.sh  |  32 ++++++++++++++++++++++++
 tools/perf/tests/shell/test_arm_spe.tgz | Bin 0 -> 253541 bytes
 2 files changed, 32 insertions(+)
 create mode 100644 tools/perf/tests/shell/test_arm_spe.tgz
tools/perf/tests/shell/test_arm_spe.sh +32 -0
index e59044edc..181cd6946 100755
@@ -18,11 +18,13 @@ skip_if_no_arm_spe_event() {
 skip_if_no_arm_spe_event || exit 2
 
 perfdata=$(mktemp /tmp/__perf_test.perf.data.XXXXX)
+scratchdir=$(mktemp -d)
 glb_err=0
 
 cleanup_files()
 {
 	rm -f ${perfdata}
+	rm -rf ${scratchdir}
 	exit $glb_err
 }
 
@@ -85,5 +87,35 @@ arm_spe_snapshot_test() {
 	arm_spe_report "SPE snapshot testing" $err
 }
 
+arm_spe_decoder_regression_test() {
+	echo "Testing event decoding $scratchdir"
+
+	# unpack test data
+	tar xf ./tests/shell/test_arm_spe.tgz -C $scratchdir
+
+	# generate md5 file of ground truth files
+	for file in $scratchdir/arm_spe_*.data.inject; do
+		filename=$(basename $file)
+		perf script -f --fields event,ip,pid,tid -i $file > $scratchdir/$filename.script
+	done
+
+	md5sum $scratchdir/*.script > $scratchdir/hash.md5
+	rm -f $scratchdir/*.script
+
+	# run Arm SPE decoder
+	for file in $scratchdir/arm_spe_*.data; do
+		filename=$(basename $file)
+		perf inject -f --itrace --strip -i $file -o $scratchdir/$filename.inject
+		perf script -f --fields event,ip,pid,tid -i $scratchdir/$filename.inject > $scratchdir/$filename.inject.script
+	done
+
+	# compare decoder output
+	md5sum -c $scratchdir/hash.md5
+
+	err=$?
+	arm_spe_report "SPE event decoding testing" $err
+}
+
 arm_spe_snapshot_test
+arm_spe_decoder_regression_test
 exit $glb_err
tools/perf/tests/shell/test_arm_spe.tgz new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..7825dedb4eb326750d0d77416f602de589ca190d
GIT binary patch
Signature · 1 line
2.25.1
#2 lore raw
Quoted 6 lines · German Gomez

Extend the test_arm_spe.sh script to test for regressions in the decoding flow of Arm SPE samples. In order to support the tests, a set of perf.data files has been generated offline and is being hosted under tools/perf/tests/shell/test_arm_spe.tgz:

(See perf report --header-only to see the actual commands used)

Looking back at this patch, I don't think it can be applied as is. I notice that the docs for the --itrace argument aren't up to date (it says the default is --itrace=ibxwpe but lately it has changed ([1] & [2]). The value of --itrace should be locked in the test script.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/Documentation/itrace.txt?id=136057256686de39cc3a07c2e39ef6bc43003ff6#n26
[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/util/auxtrace.c?id=136057256686de39cc3a07c2e39ef6bc43003ff6#n1335

Quoted 8 lines

- test_arm_spe.tgz
`- arm_spe_decode_contextidr.data
`- arm_spe_decode_contextidr.data.inject
`- arm_spe_decode_switch_events.data
`- arm_spe_decode_switch_events.data.inject

[...]

#3 lore raw
In reply to German Gomez

Hi German,

Quoted 4 lines · German Gomez

Extend the test_arm_spe.sh script to test for regressions in the decoding flow of Arm SPE samples. In order to support the tests, a set of perf.data files has been generated offline and is being hosted under tools/perf/tests/shell/test_arm_spe.tgz:

Seems to me it's not a good idea to upstream perf data binaries into the mainline kernel. I understood you want to test the perf data with different context tracing modes (using contextidr vs switch events), since these two different modes we cannot capture them with the same kernel Image, I think this is the main reason you upstreamed the perf data binaries in this patch.

On the other hand, like CoreSight smoke testing, by default we can give priority for testing root PID namespace, so you could do the test with below commands, which is assumed that tracing PID in contextidr:

perf record -e arm_spe_0// -- test_program
perf report
perf script

Then, for testing non-root PID namespace, can we use the command "unshare" to create a namespace and run perf tool in a non-root PID namespace? In this way, you could record Arm SPE trace data and decode it on the fly. Finally we can avoid to upstream perf data binaries.

Thanks,
Leo

#4 lore raw

Hi Leo,

Thanks for the comments.

Quoted 7 lines · Leo Yan

Hi German,

Quoted 4 lines · German Gomez

Extend the test_arm_spe.sh script to test for regressions in the decoding flow of Arm SPE samples. In order to support the tests, a set of perf.data files has been generated offline and is being hosted under tools/perf/tests/shell/test_arm_spe.tgz:

Quoted 6 lines

Seems to me it's not a good idea to upstream perf data binaries into the mainline kernel. I understood you want to test the perf data with different context tracing modes (using contextidr vs switch events), since these two different modes we cannot capture them with the same kernel Image, I think this is the main reason you upstreamed the perf data binaries in this patch.

For this patch I wanted to test the decoding of SPE events, given fixed inputs, in order to lock the current implementation of the decoder.

Quoted 9 lines

On the other hand, like CoreSight smoke testing, by default we can give priority for testing root PID namespace, so you could do the test with below commands, which is assumed that tracing PID in contextidr:

perf record -e arm_spe_0// -- test_program
perf report
perf script

Quoted 5 lines

Then, for testing non-root PID namespace, can we use the command "unshare" to create a namespace and run perf tool in a non-root PID namespace? In this way, you could record Arm SPE trace data and decode it on the fly. Finally we can avoid to upstream perf data binaries.

I think adding additional smoke testing is a good idea. We could also check if CONTEXTIDR is enabled in the config (either /boot/config... or /proc/config.gz).

Thanks,
German

Quoted 3 lines

Thanks,
Leo