Skip to contentSkip to conversation

perf test: Fix perf stat --bpf-counters on hybrid machines

PATCH Applied Namhyung Kim 4 participants 9 messages Mar 19 – Apr 3 perf events

lore
Share & tools
lore

Namhyung Kim

author maintainer
#0 lore raw

PATCH v3 1/1 perf test: Fix perf stat --bpf-counters on hybrid machines

The test constantly fails on my Intel hybrid machine. The issue was it has two events in the output even if I only gave it one event.

  $ perf stat -e instructions -- perf test -w sqrtloop

Performance counter stats for 'perf test -w sqrtloop':

         910,856,421      cpu_atom/instructions/                (28.05%)
      14,852,865,997      cpu_core/instructions/                (96.79%)
         1.014313341 seconds time elapsed

         1.004114000 seconds user
         0.008174000 seconds sys

Let's modify the awk script to add the values for each line and print the total. The variable 'i' has a number of input lines that have valid output and variable 'c' has the sum of actual counter values. That way it should work on any platforms.

---
v3) make sure to print numbers w/o thousand separators. v2) handle '<not counted>' and '<not supported>'.

 tools/perf/tests/shell/stat_bpf_counters.sh | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
tools/perf/tests/shell/stat_bpf_counters.sh +16 -4
index f43e28a136d3c9bc..35463358b273ce1c 100755
@@ -41,8 +41,14 @@ check_counts()
 test_bpf_counters()
 {
 	printf "Testing --bpf-counters "
-	base_instructions=$(perf stat --no-big-num -e instructions -- $workload 2>&1 | awk '/instructions/ {print $1}')
-	bpf_instructions=$(perf stat --no-big-num --bpf-counters -e instructions -- $workload  2>&1 | awk '/instructions/ {print $1}')
+	base_instructions=$(perf stat --no-big-num -e instructions -- $workload 2>&1 | \
+				awk -v i=0 -v c=0 '/instructions/ { \
+					if ($1 != "<not") { i++; c += $1 } \
+				} END { if (i > 0) printf "%.0f", c; else print "<not" }')
+	bpf_instructions=$(perf stat --no-big-num --bpf-counters -e instructions -- $workload  2>&1 | \
+				awk -v i=0 -v c=0 '/instructions/ { \
+					if ($1 != "<not") { i++; c += $1 } \
+				} END { if (i > 0) printf "%.0f", c; else print "<not" }')
 	check_counts $base_instructions $bpf_instructions
 	compare_number $base_instructions $bpf_instructions
 	echo "[Success]"
@@ -52,8 +58,14 @@ test_bpf_modifier()
 {
 	printf "Testing bpf event modifier "
 	stat_output=$(perf stat --no-big-num -e instructions/name=base_instructions/,instructions/name=bpf_instructions/b -- $workload 2>&1)
-	base_instructions=$(echo "$stat_output"| awk '/base_instructions/ {print $1}')
-	bpf_instructions=$(echo "$stat_output"| awk '/bpf_instructions/ {print $1}')
+	base_instructions=$(echo "$stat_output"| \
+				awk -v i=0 -v c=0 '/base_instructions/ { \
+					if ($1 != "<not") { i++; c += $1 } \
+				} END { if (i > 0) printf "%.0f", c; else print "<not" }')
+	bpf_instructions=$(echo "$stat_output"| \
+				awk -v i=0 -v c=0 '/bpf_instructions/ { \
+					if ($1 != "<not") { i++; c += $1 } \
+				} END { if (i > 0) printf "%.0f", c; else print "<not" }')
 	check_counts $base_instructions $bpf_instructions
 	compare_number $base_instructions $bpf_instructions
 	echo "[Success]"
Signature · 1 line
2.53.0
#1 lore raw
Quoted 15 lines · Namhyung Kim

The test constantly fails on my Intel hybrid machine. The issue was it has two events in the output even if I only gave it one event.

  $ perf stat -e instructions -- perf test -w sqrtloop
   Performance counter stats for 'perf test -w sqrtloop':

         910,856,421      cpu_atom/instructions/                (28.05%)
      14,852,865,997      cpu_core/instructions/                (96.79%)

         1.014313341 seconds time elapsed

         1.004114000 seconds user
         0.008174000 seconds sys
Quoted 4 lines

Let's modify the awk script to add the values for each line and print the total. The variable 'i' has a number of input lines that have valid output and variable 'c' has the sum of actual counter values. That way it should work on any platforms.

Still doesn't pass reliably for me, only once it did work:

⬢ [acme@toolbx perf-tools]$ git log --oneline -1
38415985abcdb64d (HEAD -> perf-tools, number/perf-tools, number/HEAD) perf test: Fix perf stat --bpf-counters on hybrid machines ⬢ [acme@toolbx perf-tools]$
logout
acme@x1:~/git/perf-tools$
acme@x1:~/git/perf-tools$

acme@x1:~/git/perf-tools$ sudo su -
root@x1:~# perf -v
perf version 7.0.rc4.g38415985abcd
root@x1:~# perf test 152
152: perf stat --bpf-counters test                                   : FAILED!
root@x1:~# set -o vi
root@x1:~# perf test -v 152
152: perf stat --bpf-counters test                                   : Ok
root@x1:~# perf test -v 152
--- start ---
test child forked, pid 1829364
Testing --bpf-counters The difference between 21947395146 and 16947118203 are greater than 20%.
---- end(-1) ----
152: perf stat --bpf-counters test                                   : FAILED!
root@x1:~# perf test -v 152
--- start ---
test child forked, pid 1829401
Testing --bpf-counters [Success]
Testing bpf event modifier The difference between 22461906845 and 17190344465 are greater than 20%.
---- end(-1) ----
152: perf stat --bpf-counters test                                   : FAILED!
root@x1:~# perf test -v 152
--- start ---
test child forked, pid 1829445
Testing --bpf-counters The difference between 24210793380 and 16852007232 are greater than 20%.
---- end(-1) ----
152: perf stat --bpf-counters test                                   : FAILED!
root@x1:~# perf test -v 152
--- start ---
test child forked, pid 1829481
Testing --bpf-counters The difference between 24293201069 and 17225814972 are greater than 20%.
---- end(-1) ----
152: perf stat --bpf-counters test                                   : FAILED!
root@x1:~# grep -m1 "model name" /proc/cpuinfo
model name	: 13th Gen Intel(R) Core(TM) i7-1365U
root@x1:~#
acme@x1:~/git/perf-tools$ uname -a
Linux x1 6.17.12-300.fc43.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Dec 13 05:06:24 UTC 2025 x86_64 GNU/Linux
acme@x1:~/git/perf-tools$ 
Quoted 49 lines

---
v3) make sure to print numbers w/o thousand separators. v2) handle '<not counted>' and '<not supported>'.

 tools/perf/tests/shell/stat_bpf_counters.sh | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
tools/perf/tests/shell/stat_bpf_counters.sh +16 -4
index f43e28a136d3c9bc..35463358b273ce1c 100755
@@ -41,8 +41,14 @@ check_counts()
 test_bpf_counters()
 {
 	printf "Testing --bpf-counters "
-	base_instructions=$(perf stat --no-big-num -e instructions -- $workload 2>&1 | awk '/instructions/ {print $1}')
-	bpf_instructions=$(perf stat --no-big-num --bpf-counters -e instructions -- $workload  2>&1 | awk '/instructions/ {print $1}')
+	base_instructions=$(perf stat --no-big-num -e instructions -- $workload 2>&1 | \
+				awk -v i=0 -v c=0 '/instructions/ { \
+					if ($1 != "<not") { i++; c += $1 } \
+				} END { if (i > 0) printf "%.0f", c; else print "<not" }')
+	bpf_instructions=$(perf stat --no-big-num --bpf-counters -e instructions -- $workload  2>&1 | \
+				awk -v i=0 -v c=0 '/instructions/ { \
+					if ($1 != "<not") { i++; c += $1 } \
+				} END { if (i > 0) printf "%.0f", c; else print "<not" }')
 	check_counts $base_instructions $bpf_instructions
 	compare_number $base_instructions $bpf_instructions
 	echo "[Success]"
@@ -52,8 +58,14 @@ test_bpf_modifier()
 {
 	printf "Testing bpf event modifier "
 	stat_output=$(perf stat --no-big-num -e instructions/name=base_instructions/,instructions/name=bpf_instructions/b -- $workload 2>&1)
-	base_instructions=$(echo "$stat_output"| awk '/base_instructions/ {print $1}')
-	bpf_instructions=$(echo "$stat_output"| awk '/bpf_instructions/ {print $1}')
+	base_instructions=$(echo "$stat_output"| \
+				awk -v i=0 -v c=0 '/base_instructions/ { \
+					if ($1 != "<not") { i++; c += $1 } \
+				} END { if (i > 0) printf "%.0f", c; else print "<not" }')
+	bpf_instructions=$(echo "$stat_output"| \
+				awk -v i=0 -v c=0 '/bpf_instructions/ { \
+					if ($1 != "<not") { i++; c += $1 } \
+				} END { if (i > 0) printf "%.0f", c; else print "<not" }')
 	check_counts $base_instructions $bpf_instructions
 	compare_number $base_instructions $bpf_instructions
 	echo "[Success]"
Signature · 1 line
2.53.0
reviewer
#2 lore raw
Quoted 67 lines · Arnaldo Carvalho de Melo
Quoted 15 lines · Namhyung Kim

The test constantly fails on my Intel hybrid machine. The issue was it has two events in the output even if I only gave it one event.

  $ perf stat -e instructions -- perf test -w sqrtloop
   Performance counter stats for 'perf test -w sqrtloop':

         910,856,421      cpu_atom/instructions/                (28.05%)
      14,852,865,997      cpu_core/instructions/                (96.79%)

         1.014313341 seconds time elapsed

         1.004114000 seconds user
         0.008174000 seconds sys
Quoted 4 lines

Let's modify the awk script to add the values for each line and print the total. The variable 'i' has a number of input lines that have valid output and variable 'c' has the sum of actual counter values. That way it should work on any platforms.

Still doesn't pass reliably for me, only once it did work:

⬢ [acme@toolbx perf-tools]$ git log --oneline -1
38415985abcdb64d (HEAD -> perf-tools, number/perf-tools, number/HEAD) perf test: Fix perf stat --bpf-counters on hybrid machines ⬢ [acme@toolbx perf-tools]$
logout
acme@x1:~/git/perf-tools$
acme@x1:~/git/perf-tools$

acme@x1:~/git/perf-tools$ sudo su -
root@x1:~# perf -v
perf version 7.0.rc4.g38415985abcd
root@x1:~# perf test 152
152: perf stat --bpf-counters test                                   : FAILED!
root@x1:~# set -o vi
root@x1:~# perf test -v 152
152: perf stat --bpf-counters test                                   : Ok
root@x1:~# perf test -v 152
--- start ---
test child forked, pid 1829364
Testing --bpf-counters The difference between 21947395146 and 16947118203 are greater than 20%.
---- end(-1) ----
152: perf stat --bpf-counters test                                   : FAILED!
root@x1:~# perf test -v 152
--- start ---
test child forked, pid 1829401
Testing --bpf-counters [Success]
Testing bpf event modifier The difference between 22461906845 and 17190344465 are greater than 20%.
---- end(-1) ----
152: perf stat --bpf-counters test                                   : FAILED!
root@x1:~# perf test -v 152
--- start ---
test child forked, pid 1829445
Testing --bpf-counters The difference between 24210793380 and 16852007232 are greater than 20%.
---- end(-1) ----
152: perf stat --bpf-counters test                                   : FAILED!
root@x1:~# perf test -v 152
--- start ---
test child forked, pid 1829481
Testing --bpf-counters The difference between 24293201069 and 17225814972 are greater than 20%.
---- end(-1) ----
152: perf stat --bpf-counters test                                   : FAILED!
root@x1:~# grep -m1 "model name" /proc/cpuinfo
model name      : 13th Gen Intel(R) Core(TM) i7-1365U
root@x1:~#
Quoted 3 lines
acme@x1:~/git/perf-tools$ uname -a
Linux x1 6.17.12-300.fc43.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Dec 13 05:06:24 UTC 2025 x86_64 GNU/Linux
acme@x1:~/git/perf-tools$

So I think the real issue here is the multiplexing value, which I've looked to fix with:
https://lore.kernel.org/lkml/[email protected]/

I have a patch to switch to task-clock which is a software event with a CPU mask of all online CPUs rather than the split seen on hybrid machines. I'll send it in reply here.

Thanks,
Ian

Quoted 49 lines
Quoted 49 lines

---
v3) make sure to print numbers w/o thousand separators. v2) handle '<not counted>' and '<not supported>'.

 tools/perf/tests/shell/stat_bpf_counters.sh | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
tools/perf/tests/shell/stat_bpf_counters.sh +16 -5
index f43e28a136d3c9bc..35463358b273ce1c 100755
@@ -41,8 +41,14 @@ check_counts()
 test_bpf_counters()
 {
      printf "Testing --bpf-counters "
-     base_instructions=$(perf stat --no-big-num -e instructions -- $workload 2>&1 | awk '/instructions/ {print $1}')
-     bpf_instructions=$(perf stat --no-big-num --bpf-counters -e instructions -- $workload  2>&1 | awk '/instructions/ {print $1}')
+     base_instructions=$(perf stat --no-big-num -e instructions -- $workload 2>&1 | \
+                             awk -v i=0 -v c=0 '/instructions/ { \
+                                     if ($1 != "<not") { i++; c += $1 } \
+                             } END { if (i > 0) printf "%.0f", c; else print "<not" }')
+     bpf_instructions=$(perf stat --no-big-num --bpf-counters -e instructions -- $workload  2>&1 | \
+                             awk -v i=0 -v c=0 '/instructions/ { \
+                                     if ($1 != "<not") { i++; c += $1 } \
+                             } END { if (i > 0) printf "%.0f", c; else print "<not" }')
      check_counts $base_instructions $bpf_instructions
      compare_number $base_instructions $bpf_instructions
      echo "[Success]"
@@ -52,8 +58,14 @@ test_bpf_modifier()
 {
      printf "Testing bpf event modifier "
      stat_output=$(perf stat --no-big-num -e instructions/name=base_instructions/,instructions/name=bpf_instructions/b -- $workload 2>&1)
-     base_instructions=$(echo "$stat_output"| awk '/base_instructions/ {print $1}')
-     bpf_instructions=$(echo "$stat_output"| awk '/bpf_instructions/ {print $1}')
+     base_instructions=$(echo "$stat_output"| \
+                             awk -v i=0 -v c=0 '/base_instructions/ { \
+                                     if ($1 != "<not") { i++; c += $1 } \
+                             } END { if (i > 0) printf "%.0f", c; else print "<not" }')
+     bpf_instructions=$(echo "$stat_output"| \
+                             awk -v i=0 -v c=0 '/bpf_instructions/ { \
+                                     if ($1 != "<not") { i++; c += $1 } \
+                             } END { if (i > 0) printf "%.0f", c; else print "<not" }')
      check_counts $base_instructions $bpf_instructions
      compare_number $base_instructions $bpf_instructions
      echo "[Success]"
--

2.53.0

reviewer
#3 lore raw

RFC PATCH 1/1 perf test: Fix perf stat --bpf-counters on hybrid machines

Events on hardware PMUs may open on >1 hardware PMU on hybrid machines. Unfortunately multiplexing introduces a scaling issue as reported in:
https://lore.kernel.org/lkml/[email protected]/

Switch to using task-clock which as a software event runs on all CPUs and has a single PMU.

---
 tools/perf/tests/shell/stat_bpf_counters.sh | 35 +++++++++++----------
 1 file changed, 18 insertions(+), 17 deletions(-)
tools/perf/tests/shell/stat_bpf_counters.sh +18 -17
index f43e28a136d3..58f7dc9692ac 100755
@@ -5,6 +5,7 @@
 set -e
 
 workload="perf test -w sqrtloop"
+event=task-clock
 
 # check whether $2 is within +/- 20% of $1
 compare_number()
@@ -25,15 +26,15 @@ compare_number()
 
 check_counts()
 {
-	base_instructions=$1
-	bpf_instructions=$2
+	base_count=$(echo "$1"|sed -e 's/[^0-9]*\([0-9]*\).*/\1/')
+	bpf_count=$(echo "$2"|sed -e 's/[^0-9]*\([0-9]*\).*/\1/')
 
-	if [ "$base_instructions" = "<not" ]; then
-		echo "Skipping: instructions event not counted"
+	if [ "$base_count" = "<not" ]; then
+		echo "Skipping: $event event not counted"
 		exit 2
 	fi
-	if [ "$bpf_instructions" = "<not" ]; then
-		echo "Failed: instructions not counted with --bpf-counters"
+	if [ "$bpf_count" = "<not" ]; then
+		echo "Failed: $event not counted with --bpf-counters"
 		exit 1
 	fi
 }
@@ -41,29 +42,29 @@ check_counts()
 test_bpf_counters()
 {
 	printf "Testing --bpf-counters "
-	base_instructions=$(perf stat --no-big-num -e instructions -- $workload 2>&1 | awk '/instructions/ {print $1}')
-	bpf_instructions=$(perf stat --no-big-num --bpf-counters -e instructions -- $workload  2>&1 | awk '/instructions/ {print $1}')
-	check_counts $base_instructions $bpf_instructions
-	compare_number $base_instructions $bpf_instructions
+	base_count=$(perf stat --no-big-num -e "$event" -- $workload 2>&1 | awk "/$event/ {print \$1}")
+	bpf_count=$(perf stat --no-big-num --bpf-counters -e "$event" -- $workload  2>&1 | awk "/$event/ {print \$1}")
+	check_counts "$base_count" "$bpf_count"
+	compare_number "$base_count" "$bpf_count"
 	echo "[Success]"
 }
 
 test_bpf_modifier()
 {
 	printf "Testing bpf event modifier "
-	stat_output=$(perf stat --no-big-num -e instructions/name=base_instructions/,instructions/name=bpf_instructions/b -- $workload 2>&1)
-	base_instructions=$(echo "$stat_output"| awk '/base_instructions/ {print $1}')
-	bpf_instructions=$(echo "$stat_output"| awk '/bpf_instructions/ {print $1}')
-	check_counts $base_instructions $bpf_instructions
-	compare_number $base_instructions $bpf_instructions
+	stat_output=$(perf stat --no-big-num -e "$event/name=base_$event/,$event/name=bpf_$event/b" -- $workload 2>&1)
+	base_count=$(echo "$stat_output"| awk "/base_$event/ {print \$1}")
+	bpf_count=$(echo "$stat_output"| awk "/bpf_$event/ {print \$1}")
+	check_counts "$base_count" "$bpf_count"
+	compare_number "$base_count" "$bpf_count"
 	echo "[Success]"
 }
 
 # skip if --bpf-counters is not supported
-if ! perf stat -e instructions --bpf-counters true > /dev/null 2>&1; then
+if ! perf stat -e $event --bpf-counters true > /dev/null 2>&1; then
 	if [ "$1" = "-v" ]; then
 		echo "Skipping: --bpf-counters not supported"
-		perf --no-pager stat -e instructions --bpf-counters true || true
+		perf --no-pager stat -e $event --bpf-counters true || true
 	fi
 	exit 2
 fi
Signature · 1 line
2.53.0.1018.g2bb0e51243-goog
reviewer
#4 lore raw
Quoted 10 lines · Ian Rogers

Events on hardware PMUs may open on >1 hardware PMU on hybrid machines. Unfortunately multiplexing introduces a scaling issue as reported in:
https://lore.kernel.org/lkml/[email protected]/

Switch to using task-clock which as a software event runs on all CPUs and has a single PMU.

With hindsight, maybe the better thing to do is Namhyung's patch plus the "--no-scale" option to perf stat to sidestep the scaling issue. It'd be great to make this test not "(exclusive)", it shouldn't be necessary. You can test for flakiness with the '-r' option to perf test which takes a repeat count.

Thanks,
Ian

Quoted 82 lines
---
 tools/perf/tests/shell/stat_bpf_counters.sh | 35 +++++++++++----------
 1 file changed, 18 insertions(+), 17 deletions(-)
tools/perf/tests/shell/stat_bpf_counters.sh +18 -18
index f43e28a136d3..58f7dc9692ac 100755
@@ -5,6 +5,7 @@
 set -e
 
 workload="perf test -w sqrtloop"
+event=task-clock
 
 # check whether $2 is within +/- 20% of $1
 compare_number()
@@ -25,15 +26,15 @@ compare_number()
 
 check_counts()
 {
-       base_instructions=$1
-       bpf_instructions=$2
+       base_count=$(echo "$1"|sed -e 's/[^0-9]*\([0-9]*\).*/\1/')
+       bpf_count=$(echo "$2"|sed -e 's/[^0-9]*\([0-9]*\).*/\1/')
 
-       if [ "$base_instructions" = "<not" ]; then
-               echo "Skipping: instructions event not counted"
+       if [ "$base_count" = "<not" ]; then
+               echo "Skipping: $event event not counted"
                exit 2
        fi
-       if [ "$bpf_instructions" = "<not" ]; then
-               echo "Failed: instructions not counted with --bpf-counters"
+       if [ "$bpf_count" = "<not" ]; then
+               echo "Failed: $event not counted with --bpf-counters"
                exit 1
        fi
 }
@@ -41,29 +42,29 @@ check_counts()
 test_bpf_counters()
 {
        printf "Testing --bpf-counters "
-       base_instructions=$(perf stat --no-big-num -e instructions -- $workload 2>&1 | awk '/instructions/ {print $1}')
-       bpf_instructions=$(perf stat --no-big-num --bpf-counters -e instructions -- $workload  2>&1 | awk '/instructions/ {print $1}')
-       check_counts $base_instructions $bpf_instructions
-       compare_number $base_instructions $bpf_instructions
+       base_count=$(perf stat --no-big-num -e "$event" -- $workload 2>&1 | awk "/$event/ {print \$1}")
+       bpf_count=$(perf stat --no-big-num --bpf-counters -e "$event" -- $workload  2>&1 | awk "/$event/ {print \$1}")
+       check_counts "$base_count" "$bpf_count"
+       compare_number "$base_count" "$bpf_count"
        echo "[Success]"
 }
 
 test_bpf_modifier()
 {
        printf "Testing bpf event modifier "
-       stat_output=$(perf stat --no-big-num -e instructions/name=base_instructions/,instructions/name=bpf_instructions/b -- $workload 2>&1)
-       base_instructions=$(echo "$stat_output"| awk '/base_instructions/ {print $1}')
-       bpf_instructions=$(echo "$stat_output"| awk '/bpf_instructions/ {print $1}')
-       check_counts $base_instructions $bpf_instructions
-       compare_number $base_instructions $bpf_instructions
+       stat_output=$(perf stat --no-big-num -e "$event/name=base_$event/,$event/name=bpf_$event/b" -- $workload 2>&1)
+       base_count=$(echo "$stat_output"| awk "/base_$event/ {print \$1}")
+       bpf_count=$(echo "$stat_output"| awk "/bpf_$event/ {print \$1}")
+       check_counts "$base_count" "$bpf_count"
+       compare_number "$base_count" "$bpf_count"
        echo "[Success]"
 }
 
 # skip if --bpf-counters is not supported
-if ! perf stat -e instructions --bpf-counters true > /dev/null 2>&1; then
+if ! perf stat -e $event --bpf-counters true > /dev/null 2>&1; then
        if [ "$1" = "-v" ]; then
                echo "Skipping: --bpf-counters not supported"
-               perf --no-pager stat -e instructions --bpf-counters true || true
+               perf --no-pager stat -e $event --bpf-counters true || true
        fi
        exit 2
 fi
--

2.53.0.1018.g2bb0e51243-goog

author maintainer Acked-by
#5 lore raw
Quoted 10 lines · Ian Rogers
Quoted 9 lines · Ian Rogers

Events on hardware PMUs may open on >1 hardware PMU on hybrid machines. Unfortunately multiplexing introduces a scaling issue as reported in:
https://lore.kernel.org/lkml/[email protected]/

Switch to using task-clock which as a software event runs on all CPUs and has a single PMU.

Quoted 7 lines
Quoted 1 line

With hindsight, maybe the better thing to do is Namhyung's patch plus the "--no-scale" option to perf stat to sidestep the scaling issue. It'd be great to make this test not "(exclusive)", it shouldn't be necessary. You can test for flakiness with the '-r' option to perf test which takes a repeat count.

I think your fix is simpler and better. :)

Thanks,
Namhyung

reviewer
#6 lore raw

PATCH v2 1/1 perf test: Fix perf stat --bpf-counters on hybrid machines

Events on hardware PMUs may open on >1 hardware PMU on hybrid machines. Unfortunately multiplexing introduces a scaling issue as reported in:
https://lore.kernel.org/lkml/[email protected]/

Switch to using task-clock which as a software event runs on all CPUs and has a single PMU.

For when hybrid event scaling is fixed, incorporate support for combining hybrid counts as posted by Namhyung Kim in:
https://lore.kernel.org/lkml/[email protected]/

---
v1: https://lore.kernel.org/lkml/[email protected]/

---
 tools/perf/tests/shell/stat_bpf_counters.sh | 55 ++++++++++++++-------
 1 file changed, 38 insertions(+), 17 deletions(-)
tools/perf/tests/shell/stat_bpf_counters.sh +38 -17
index f43e28a136d3..f8c24ccc7830 100755
@@ -5,6 +5,24 @@
 set -e
 
 workload="perf test -w sqrtloop"
+event=task-clock
+
+extract_count()
+{
+	local target_event="$1"
+
+	awk -v target="$target_event" -v i=0 -v c=0 '
+	$0 ~ target {
+		i++;
+		c += $1
+	}
+	END {
+		if (i > 0)
+			printf "%.0f\n", c;
+		else
+			print "<not counted>"
+	}'
+}
 
 # check whether $2 is within +/- 20% of $1
 compare_number()
@@ -25,15 +43,15 @@ compare_number()
 
 check_counts()
 {
-	base_instructions=$1
-	bpf_instructions=$2
+	base_count=$1
+	bpf_count=$2
 
-	if [ "$base_instructions" = "<not" ]; then
-		echo "Skipping: instructions event not counted"
+	if [ "$base_count" = "<not counted>" ]; then
+		echo "Skipping: $event event not counted"
 		exit 2
 	fi
-	if [ "$bpf_instructions" = "<not" ]; then
-		echo "Failed: instructions not counted with --bpf-counters"
+	if [ "$bpf_count" = "<not counted>" ]; then
+		echo "Failed: $event not counted with --bpf-counters"
 		exit 1
 	fi
 }
@@ -41,29 +59,32 @@ check_counts()
 test_bpf_counters()
 {
 	printf "Testing --bpf-counters "
-	base_instructions=$(perf stat --no-big-num -e instructions -- $workload 2>&1 | awk '/instructions/ {print $1}')
-	bpf_instructions=$(perf stat --no-big-num --bpf-counters -e instructions -- $workload  2>&1 | awk '/instructions/ {print $1}')
-	check_counts $base_instructions $bpf_instructions
-	compare_number $base_instructions $bpf_instructions
+	base_count=$(perf stat --no-big-num -e "$event" -- $workload 2>&1 \
+          | extract_count "$event")
+	bpf_count=$(perf stat --no-big-num --bpf-counters -e "$event" -- $workload  2>&1 \
+          | extract_count "$event")
+	check_counts "$base_count" "$bpf_count"
+	compare_number "$base_count" "$bpf_count"
 	echo "[Success]"
 }
 
 test_bpf_modifier()
 {
 	printf "Testing bpf event modifier "
-	stat_output=$(perf stat --no-big-num -e instructions/name=base_instructions/,instructions/name=bpf_instructions/b -- $workload 2>&1)
-	base_instructions=$(echo "$stat_output"| awk '/base_instructions/ {print $1}')
-	bpf_instructions=$(echo "$stat_output"| awk '/bpf_instructions/ {print $1}')
-	check_counts $base_instructions $bpf_instructions
-	compare_number $base_instructions $bpf_instructions
+	stat_output=$(perf stat --no-big-num \
+          -e "$event/name=base_$event/,$event/name=bpf_$event/b" -- $workload 2>&1)
+	base_count=$(echo "$stat_output" | extract_count "base_$event")
+	bpf_count=$(echo "$stat_output"|  extract_count "bpf_$event")
+	check_counts "$base_count" "$bpf_count"
+	compare_number "$base_count" "$bpf_count"
 	echo "[Success]"
 }
 
 # skip if --bpf-counters is not supported
-if ! perf stat -e instructions --bpf-counters true > /dev/null 2>&1; then
+if ! perf stat -e "$event" --bpf-counters true > /dev/null 2>&1; then
 	if [ "$1" = "-v" ]; then
 		echo "Skipping: --bpf-counters not supported"
-		perf --no-pager stat -e instructions --bpf-counters true || true
+		perf --no-pager stat -e "$event" --bpf-counters true || true
 	fi
 	exit 2
 fi
Signature · 1 line
2.53.0.1018.g2bb0e51243-goog
Tested-by on v2 1/1
#7 lore raw
Quoted 8 lines · Ian Rogers

Events on hardware PMUs may open on >1 hardware PMU on hybrid machines. Unfortunately multiplexing introduces a scaling issue as reported in:
https://lore.kernel.org/lkml/[email protected]/

Switch to using task-clock which as a software event runs on all CPUs and has a single PMU.

Quoted 6 lines

For when hybrid event scaling is fixed, incorporate support for combining hybrid counts as posted by Namhyung Kim in:
https://lore.kernel.org/lkml/[email protected]/

$ sudo ./perf test "perf stat --bpf-counters test" -vvv

Before:

152: perf stat --bpf-counters test:
--- start ---
test child forked, pid 444406
Testing --bpf-counters The difference between 400418163 and 11886805947 are greater than 20%.
---- end(-1) ----
152: perf stat --bpf-counters test :
FAILED!

After:

152: perf stat --bpf-counters test:
--- start ---
test child forked, pid 466896
Testing --bpf-counters [Success]
Testing bpf event modifier [Success]
---- end(0) ----
152: perf stat --bpf-counters test :
Ok

Thanks,
Tom

Quoted 121 lines

---
v1:
https://lore.kernel.org/lkml/[email protected]/
---
 tools/perf/tests/shell/stat_bpf_counters.sh | 55 ++++++++++++++----- --

 1 file changed, 38 insertions(+), 17 deletions(-)
tools/perf/tests/shell/stat_bpf_counters.sh +0 -0
index f43e28a136d3..f8c24ccc7830 100755
@@ -5,6 +5,24 @@

 set -e

 workload="perf test -w sqrtloop"

tools/perf/tests/shell/stat_bpf_counters.sh +18 -0
+event=task-clock
+
+extract_count()
+{
+	local target_event="$1"
+
+	awk -v target="$target_event" -v i=0 -v c=0 '
+	$0 ~ target {
+		i++;
+		c += $1
+	}
+	END {
+		if (i > 0)
+			printf "%.0f\n", c;
+		else
+			print "<not counted>"
+	}'
+}

 # check whether $2 is within +/- 20% of $1
 compare_number()
tools/perf/tests/shell/stat_bpf_counters.sh +0 -0
@@ -25,15 +43,15 @@ compare_number()


 check_counts()
 {

tools/perf/tests/shell/stat_bpf_counters.sh +2 -2
-	base_instructions=$1
-	bpf_instructions=$2
+	base_count=$1
+	bpf_count=$2

tools/perf/tests/shell/stat_bpf_counters.sh +2 -2
-	if [ "$base_instructions" = "<not" ]; then
-		echo "Skipping: instructions event not counted"
+	if [ "$base_count" = "<not counted>" ]; then
+		echo "Skipping: $event event not counted"

  exit 2
  fi

tools/perf/tests/shell/stat_bpf_counters.sh +0 -2
-	if [ "$bpf_instructions" = "<not" ]; then
-		echo "Failed: instructions not counted with --bpf-

counters"

tools/perf/tests/shell/stat_bpf_counters.sh +2 -0
+	if [ "$bpf_count" = "<not counted>" ]; then
+		echo "Failed: $event not counted with --bpf-

counters"
  exit 1
  fi
 }

tools/perf/tests/shell/stat_bpf_counters.sh +0 -0
@@ -41,29 +59,32 @@ check_counts()
  test_bpf_counters()
  {

  printf "Testing --bpf-counters "

tools/perf/tests/shell/stat_bpf_counters.sh +0 -3
-	base_instructions=$(perf stat --no-big-num -e instructions -
- $workload 2>&1 | awk '/instructions/ {print $1}')
-	bpf_instructions=$(perf stat --no-big-num --bpf-counters -e

instructions -- $workload  2>&1 | awk '/instructions/ {print $1}')

tools/perf/tests/shell/stat_bpf_counters.sh +1 -2
-	check_counts $base_instructions $bpf_instructions
-	compare_number $base_instructions $bpf_instructions
+	base_count=$(perf stat --no-big-num -e "$event" -- $workload

2>&1 \
+          | extract_count "$event")

tools/perf/tests/shell/stat_bpf_counters.sh +1 -0
+	bpf_count=$(perf stat --no-big-num --bpf-counters -e

"$event" -- $workload  2>&1 \
+          | extract_count "$event")

tools/perf/tests/shell/stat_bpf_counters.sh +2 -0
+	check_counts "$base_count" "$bpf_count"
+	compare_number "$base_count" "$bpf_count"

  echo "[Success]"
 }

 test_bpf_modifier()
 {
  printf "Testing bpf event modifier "

tools/perf/tests/shell/stat_bpf_counters.sh +0 -1
-	stat_output=$(perf stat --no-big-num -e

instructions/name=base_instructions/,instructions/name=bpf_instructio ns/b -- $workload 2>&1)

tools/perf/tests/shell/stat_bpf_counters.sh +0 -1
-	base_instructions=$(echo "$stat_output"| awk

'/base_instructions/ {print $1}')

tools/perf/tests/shell/stat_bpf_counters.sh +0 -1
-	bpf_instructions=$(echo "$stat_output"| awk

'/bpf_instructions/ {print $1}')

tools/perf/tests/shell/stat_bpf_counters.sh +2 -2
-	check_counts $base_instructions $bpf_instructions
-	compare_number $base_instructions $bpf_instructions
+	stat_output=$(perf stat --no-big-num \
+          -e "$event/name=base_$event/,$event/name=bpf_$event/b" --

$workload 2>&1)

tools/perf/tests/shell/stat_bpf_counters.sh +1 -0
+	base_count=$(echo "$stat_output" | extract_count

"base_$event")

tools/perf/tests/shell/stat_bpf_counters.sh +1 -0
+	bpf_count=$(echo "$stat_output"|  extract_count

"bpf_$event")

tools/perf/tests/shell/stat_bpf_counters.sh +2 -0
+	check_counts "$base_count" "$bpf_count"
+	compare_number "$base_count" "$bpf_count"

  echo "[Success]"
 }

 # skip if --bpf-counters is not supported
-if ! perf stat -e instructions --bpf-counters true > /dev/null 2>&1;
then
+if ! perf stat -e "$event" --bpf-counters true > /dev/null 2>&1;
then
 	if [ "$1" = "-v" ]; then
 		echo "Skipping: --bpf-counters not supported"
tools/perf/tests/shell/stat_bpf_counters.sh +0 -1
-		perf --no-pager stat -e instructions --bpf-counters

true || true

tools/perf/tests/shell/stat_bpf_counters.sh +1 -0
+		perf --no-pager stat -e "$event" --bpf-counters true

|| true
  fi
  exit 2
 fi

author maintainer Applied
#8 lore raw
In reply to Namhyung Kim
Quoted 8 lines · Namhyung Kim

The test constantly fails on my Intel hybrid machine. The issue was it has two events in the output even if I only gave it one event.

  $ perf stat -e instructions -- perf test -w sqrtloop

Performance counter stats for 'perf test -w sqrtloop':

[...]

Applied to perf-tools-next, thanks!

Best regards,
Namhyung