Skip to contentSkip to conversation

perf test shell trace_exit_race improvements

PATCH In review Arnaldo Carvalho de Melo 2 participants 6 messages Nov 15, 2024 – Nov 16, 2024 perf events

lore
Share & tools
lore

Arnaldo Carvalho de Melo

author maintainer
#0 lore raw

v2 cover perf test shell trace_exit_race improvements

Hi,

Here is v2, folding a slightly modified version from Howard
reviewing patch, with an extra patch to use --no-comm with 'perf trace' as discussed previously and in the patch log message,

Please see if everything is all right now,

- Arnaldo

Arnaldo Carvalho de Melo (2):
perf test shell trace_exit_race: Show what went wrong in verbose mode perf test shell trace_exit_race: Use --no-comm to avoid cases where
COMM isn't resolved

 tools/perf/tests/shell/trace_exit_race.sh | 26 ++++++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)
Signature · 1 line
2.47.0
author maintainer
#1 lore raw

PATCH v2 1/2 perf test shell trace_exit_race: Show what went wrong in verbose mode

From: Arnaldo Carvalho de Melo <[email protected]>

If it fails we need to check what was the reason, what were the lines that didn't match the expected format, so:

  root@number:~# perf test -v "trace exit race"
  --- start ---
  test child forked, pid 2028724
  Lines not matching the expected regexp: ' +[0-9]+\.[0-9]+ +true/[0-9]+ syscalls:sys_enter_exit_group\(\)$':
       0.000 :2028750/2028750 syscalls:sys_enter_exit_group()
  ---- end(-1) ----
  110: perf trace exit race                                            : FAILED!
  root@number:~#

In this case we're not resolving the process COMM for some reason and fallback to printing just the pid/tid, this will be fixed in a followup patch.

Howard Chu spotted a problem with single code surrounding a regexp, that made the test always fail, but since there were some failures when I tested (COMM not being resolved in some of the results) the end inverse grep would show some lines and thus didn't notice the single quote problem.

He also provided a patch to test if less than the number of expected matches took place but all of them with the expected output, in which case the inverse grep wouldn't show anything, confusing the tester.

Cc Adrian Hunter <[email protected]>
Cc Alexander Shishkin <[email protected]>
Cc Benjamin Peterson <[email protected]>
Cc Ian Rogers <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Kan Liang <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 tools/perf/tests/shell/trace_exit_race.sh | 24 +++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
tools/perf/tests/shell/trace_exit_race.sh +22 -2
index 3cf2d71a5c3b9278..8ea24f4256bc8f5e 100755
@@ -11,11 +11,19 @@
 
 skip_if_no_perf_trace || exit 2
 
+if [ "$1" = "-v" ]; then
+	verbose="1"
+fi
+
+iter=10
+regexp=" +[0-9]+\.[0-9]+ +true/[0-9]+ syscalls:sys_enter_exit_group\(\)$"
+
 trace_shutdown_race() {
-	for _ in $(seq 10); do
+	for _ in $(seq $iter); do
 		perf trace -e syscalls:sys_enter_exit_group true 2>>$file
 	done
-	[ "$(grep -c -E ' +[0-9]+\.[0-9]+ +true/[0-9]+ syscalls:sys_enter_exit_group\(\)$' $file)" = "10" ]
+	result="$(grep -c -E "$regexp" $file)"
+	[ $result = $iter ]
 }
 
 
@@ -27,5 +35,17 @@ export PERF_CONFIG=/dev/null
 
 trace_shutdown_race
 err=$?
+
+if [ $err != 0 ] && [ "${verbose}" = "1" ]; then
+	lines_not_matching=$(mktemp /tmp/temporary_file.XXXXX)
+	if grep -v -E "$regexp" $file > $lines_not_matching ; then
+		echo "Lines not matching the expected regexp: '$regexp':"
+		cat $lines_not_matching
+	else
+		echo "Missing output, expected $iter but only got $result"
+	fi
+	rm -f $lines_not_matching
+fi
+
 rm -f ${file}
 exit $err
Signature · 1 line
2.47.0
#2 lore raw

Re: [PATCH 1/2] perf test shell trace_exit_race: Show what went wrong in verbose mode

Hello Arnaldo,

Quoted 80 lines · Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

If it fails we need to check what was the reason, what were the lines that didn't match the expected format, so:

  root@number:~# perf test -v "trace exit race"
  --- start ---
  test child forked, pid 2028724
  Lines not matching the expected regexp: ' +[0-9]+\.[0-9]+ +true/[0-9]+ syscalls:sys_enter_exit_group\(\)$':
       0.000 :2028750/2028750 syscalls:sys_enter_exit_group()
  ---- end(-1) ----
  110: perf trace exit race                                            : FAILED!
  root@number:~#

In this case we're not resolving the process COMM for some reason and fallback to printing just the pid/tid, this will be fixed in a followup patch.

Howard Chu spotted a problem with single code surrounding a regexp, that made the test always fail, but since there were some failures when I tested (COMM not being resolved in some of the results) the end inverse grep would show some lines and thus didn't notice the single quote problem.

He also provided a patch to test if less than the number of expected matches took place but all of them with the expected output, in which case the inverse grep wouldn't show anything, confusing the tester.

Cc Adrian Hunter <[email protected]>
Cc Alexander Shishkin <[email protected]>
Cc Benjamin Peterson <[email protected]>
Cc Ian Rogers <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Kan Liang <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 tools/perf/tests/shell/trace_exit_race.sh | 24 +++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
tools/perf/tests/shell/trace_exit_race.sh +14 -2
index 3cf2d71a5c3b9278..8ea24f4256bc8f5e 100755
@@ -11,11 +11,19 @@
 
 skip_if_no_perf_trace || exit 2
 
+if [ "$1" = "-v" ]; then
+       verbose="1"
+fi
+
+iter=10
+regexp=" +[0-9]+\.[0-9]+ +true/[0-9]+ syscalls:sys_enter_exit_group\(\)$"
+
 trace_shutdown_race() {
-       for _ in $(seq 10); do
+       for _ in $(seq $iter); do
                perf trace -e syscalls:sys_enter_exit_group true 2>>$file
        done
-       [ "$(grep -c -E ' +[0-9]+\.[0-9]+ +true/[0-9]+ syscalls:sys_enter_exit_group\(\)$' $file)" = "10" ]
+       result="$(grep -c -E "$regexp" $file)"
+       [ $result = $iter ]
 }
 
 
@@ -27,5 +35,17 @@ export PERF_CONFIG=/dev/null
 
 trace_shutdown_race
 err=$?
+
+if [ $err != 0 ] && [ "${verbose}" = "1" ]; then
+       lines_not_matching=$(mktemp /tmp/temporary_file.XXXXX)
+       if grep -v -E "$regexp" $file > $lines_not_matching ; then

tools/perf/tests/shell/trace_exit_race.sh export PERF_CONFIG=/dev/null · line 51

Quoted 10 lines
tools/perf/tests/shell/trace_exit_race.sh +8 -0
+               echo "Lines not matching the expected regexp: '$regexp':"
+               cat $lines_not_matching
+       else
+               echo "Missing output, expected $iter but only got $result"
+       fi
+       rm -f $lines_not_matching
+fi
+
 rm -f ${file}
 exit $err

LGTM.

Thanks,
Howard

Quoted 3 lines

--
2.47.0

author maintainer
#3 lore raw

PATCH v2 2/2 perf test shell trace_exit_race: Use --no-comm to avoid cases where COMM isn't resolved

In reply to Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

The purpose of this test is to test for races in the exit of 'perf trace' missing the last events, it was failing when the COMM wasn't resolved either because we missed some PERF_RECORD_COMM or somehow raced on getting it from procfs.

Add --no-comm to the 'perf trace' command line so that we get a consistent, pid only output, which allows the test to achieve its goal.

This is the output from
'perf trace --no-comm -e syscalls:sys_enter_exit_group':

     0.000 21953 syscalls:sys_enter_exit_group()
     0.000 21955 syscalls:sys_enter_exit_group()
     0.000 21957 syscalls:sys_enter_exit_group()
     0.000 21959 syscalls:sys_enter_exit_group()
     0.000 21961 syscalls:sys_enter_exit_group()
     0.000 21963 syscalls:sys_enter_exit_group()
     0.000 21965 syscalls:sys_enter_exit_group()
     0.000 21967 syscalls:sys_enter_exit_group()
     0.000 21969 syscalls:sys_enter_exit_group()
     0.000 21971 syscalls:sys_enter_exit_group()

Now it passes:

  root@number:~# perf test "trace exit race"
  110: perf trace exit race                                            : Ok
  root@number:~#
  root@number:~# perf test -v "trace exit race"
  110: perf trace exit race                                            : Ok
  root@number:~#

If we artificially make it run just 9 times instead of the 10 it runs, i.e. by manually doing:

	trace_shutdown_race() {
		for _ in $(seq 9); do

that 9 is $iter, 10 in the patch, we get:

  root@number:~# vim ~acme/libexec/perf-core/tests/shell/trace_exit_race.sh
  root@number:~# perf test -v "trace exit race"
  --- start ---
  test child forked, pid 24629
  Missing output, expected 10 but only got 9
  ---- end(-1) ----
  110: perf trace exit race                                            : FAILED!
  root@number:~#

I.e. 9 'perf trace' calls produced the expected output, the inverse grep didn't show anything, so the patch provided by Howard for the previous patch kicks in and shows a more informative message.

Cc Adrian Hunter <[email protected]>
Cc Alexander Shishkin <[email protected]>
Cc Benjamin Peterson <[email protected]>
Cc Howard Chu <[email protected]>
Cc Ian Rogers <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Kan Liang <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 tools/perf/tests/shell/trace_exit_race.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/tests/shell/trace_exit_race.sh +2 -2
index 8ea24f4256bc8f5e..fbb0adc33a889516 100755
@@ -16,11 +16,11 @@ if [ "$1" = "-v" ]; then
 fi
 
 iter=10
-regexp=" +[0-9]+\.[0-9]+ +true/[0-9]+ syscalls:sys_enter_exit_group\(\)$"
+regexp=" +[0-9]+\.[0-9]+ [0-9]+ syscalls:sys_enter_exit_group\(\)$"
 
 trace_shutdown_race() {
 	for _ in $(seq $iter); do
-		perf trace -e syscalls:sys_enter_exit_group true 2>>$file
+		perf trace --no-comm -e syscalls:sys_enter_exit_group true 2>>$file
 	done
 	result="$(grep -c -E "$regexp" $file)"
 	[ $result = $iter ]
Signature · 1 line
2.47.0
Tested-by on v2 2/2
#4 lore raw

Re: [PATCH 2/2] perf test shell trace_exit_race: Use --no-comm to avoid cases where COMM isn't resolved

perf $ ./perf test "perf trace exit race" -v

110: perf trace exit race : Ok

Thanks,
Howard

Quoted 93 lines · Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

The purpose of this test is to test for races in the exit of 'perf trace' missing the last events, it was failing when the COMM wasn't resolved either because we missed some PERF_RECORD_COMM or somehow raced on getting it from procfs.

Add --no-comm to the 'perf trace' command line so that we get a consistent, pid only output, which allows the test to achieve its goal.

This is the output from
'perf trace --no-comm -e syscalls:sys_enter_exit_group':

     0.000 21953 syscalls:sys_enter_exit_group()
     0.000 21955 syscalls:sys_enter_exit_group()
     0.000 21957 syscalls:sys_enter_exit_group()
     0.000 21959 syscalls:sys_enter_exit_group()
     0.000 21961 syscalls:sys_enter_exit_group()
     0.000 21963 syscalls:sys_enter_exit_group()
     0.000 21965 syscalls:sys_enter_exit_group()
     0.000 21967 syscalls:sys_enter_exit_group()
     0.000 21969 syscalls:sys_enter_exit_group()
     0.000 21971 syscalls:sys_enter_exit_group()

Now it passes:

  root@number:~# perf test "trace exit race"
  110: perf trace exit race                                            : Ok
  root@number:~#
  root@number:~# perf test -v "trace exit race"
  110: perf trace exit race                                            : Ok
  root@number:~#

If we artificially make it run just 9 times instead of the 10 it runs, i.e. by manually doing:

        trace_shutdown_race() {
                for _ in $(seq 9); do

that 9 is $iter, 10 in the patch, we get:

  root@number:~# vim ~acme/libexec/perf-core/tests/shell/trace_exit_race.sh
  root@number:~# perf test -v "trace exit race"
  --- start ---
  test child forked, pid 24629
  Missing output, expected 10 but only got 9
  ---- end(-1) ----
  110: perf trace exit race                                            : FAILED!
  root@number:~#

I.e. 9 'perf trace' calls produced the expected output, the inverse grep didn't show anything, so the patch provided by Howard for the previous patch kicks in and shows a more informative message.

Cc Adrian Hunter <[email protected]>
Cc Alexander Shishkin <[email protected]>
Cc Benjamin Peterson <[email protected]>
Cc Howard Chu <[email protected]>
Cc Ian Rogers <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Kan Liang <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 tools/perf/tests/shell/trace_exit_race.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/tests/shell/trace_exit_race.sh +2 -3
index 8ea24f4256bc8f5e..fbb0adc33a889516 100755
@@ -16,11 +16,11 @@ if [ "$1" = "-v" ]; then
 fi
 
 iter=10
-regexp=" +[0-9]+\.[0-9]+ +true/[0-9]+ syscalls:sys_enter_exit_group\(\)$"
+regexp=" +[0-9]+\.[0-9]+ [0-9]+ syscalls:sys_enter_exit_group\(\)$"
 
 trace_shutdown_race() {
        for _ in $(seq $iter); do
-               perf trace -e syscalls:sys_enter_exit_group true 2>>$file
+               perf trace --no-comm -e syscalls:sys_enter_exit_group true 2>>$file
        done
        result="$(grep -c -E "$regexp" $file)"
        [ $result = $iter ]
--

2.47.0

author maintainer
#5 lore raw

Re: [PATCH 2/2] perf test shell trace_exit_race: Use --no-comm to avoid cases where COMM isn't resolved

Quoted 5 lines · Howard Chu

perf $ ./perf test "perf trace exit race" -v

110: perf trace exit race : Ok

Thanks, added to the cset.

- Arnaldo