Skip to contentSkip to conversation

perf test: Avoid hard coded metrics in stat std output test

PATCH In review Ian Rogers 2 participants 6 messages Apr 17, 2024 – Apr 19, 2024 perf events

lore
Share & tools
lore

Ian Rogers

author reviewer
#0 lore raw

PATCH 1/1 perf test: Avoid hard coded metrics in stat std output test

Hard coded metric names fail on ARM testing.

---
 tools/perf/tests/shell/stat+std_output.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/tests/shell/stat+std_output.sh +1 -1
index cbf2894b2c84..845f83213855 100755
@@ -13,7 +13,7 @@ stat_output=$(mktemp /tmp/__perf_test.stat_output.std.XXXXX)
 
 event_name=(cpu-clock task-clock context-switches cpu-migrations page-faults stalled-cycles-frontend stalled-cycles-backend cycles instructions branches branch-misses)
 event_metric=("CPUs utilized" "CPUs utilized" "/sec" "/sec" "/sec" "frontend cycles idle" "backend cycles idle" "GHz" "insn per cycle" "/sec" "of all branches")
-skip_metric=("stalled cycles per insn" "tma_" "retiring" "frontend_bound" "bad_speculation" "backend_bound")
+skip_metric=($(perf list --raw Default 2> /dev/null))
 
 cleanup() {
   rm -f "${stat_output}"
Signature · 1 line
2.44.0.683.g7961c838ac-goog
#1 lore raw
Quoted 12 lines · Ian Rogers

Hard coded metric names fail on ARM testing.

---
 tools/perf/tests/shell/stat+std_output.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/tests/shell/stat+std_output.sh +0 -0
index cbf2894b2c84..845f83213855 100755

tools/perf/tests/shell/stat+std_output.sh stat_output=$(mktemp /tmp/__perf_test.stat_output.std.XXXXX) · line 16

Quoted 5 lines
tools/perf/tests/shell/stat+std_output.sh +1 -1
@@ -13,7 +13,7 @@ stat_output=$(mktemp /tmp/__perf_test.stat_output.std.XXXXX)
 
 event_name=(cpu-clock task-clock context-switches cpu-migrations page-faults stalled-cycles-frontend stalled-cycles-backend cycles instructions branches branch-misses)
 event_metric=("CPUs utilized" "CPUs utilized" "/sec" "/sec" "/sec" "frontend cycles idle" "backend cycles idle" "GHz" "insn per cycle" "/sec" "of all branches")
-skip_metric=("stalled cycles per insn" "tma_" "retiring" "frontend_bound" "bad_speculation" "backend_bound")
+skip_metric=($(perf list --raw Default 2> /dev/null))

The "perf list --raw Default" only gives the topdown metrics. The "stalled cycles per insn" is not covered.
The check should skip the line of "stalled cycles per insn" as well.

     3,856,436,920 stalled-cycles-frontend   #   74.09% frontend cycles idle
     1,600,790,871 stalled-cycles-backend    #   30.75% backend  cycles idle
     2,603,501,247 instructions              #    0.50  insns per cycle
                                             #    1.48  stalled cycles
per insn
       484,357,498 branches                  #  283.455 M/sec

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/builtin-stat.c#n24

The newer Intel CPU doesn't have the stalled-cycles-* events. But it seems power and older x86 CPU have the events.

Thanks,
Kan

Quoted 3 lines
tools/perf/tests/shell/stat+std_output.sh +0 -0
 cleanup() {
   rm -f "${stat_output}"
author reviewer
#2 lore raw
Quoted 33 lines · Kan
Quoted 12 lines · Ian Rogers

Hard coded metric names fail on ARM testing.

---
 tools/perf/tests/shell/stat+std_output.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/tests/shell/stat+std_output.sh +0 -0
index cbf2894b2c84..845f83213855 100755

tools/perf/tests/shell/stat+std_output.sh stat_output=$(mktemp /tmp/__perf_test.stat_output.std.XXXXX) · line 16

Quoted 5 lines
tools/perf/tests/shell/stat+std_output.sh +1 -1
@@ -13,7 +13,7 @@ stat_output=$(mktemp /tmp/__perf_test.stat_output.std.XXXXX)
 
 event_name=(cpu-clock task-clock context-switches cpu-migrations page-faults stalled-cycles-frontend stalled-cycles-backend cycles instructions branches branch-misses)
 event_metric=("CPUs utilized" "CPUs utilized" "/sec" "/sec" "/sec" "frontend cycles idle" "backend cycles idle" "GHz" "insn per cycle" "/sec" "of all branches")
-skip_metric=("stalled cycles per insn" "tma_" "retiring" "frontend_bound" "bad_speculation" "backend_bound")
+skip_metric=($(perf list --raw Default 2> /dev/null))

The "perf list --raw Default" only gives the topdown metrics. The "stalled cycles per insn" is not covered.
The check should skip the line of "stalled cycles per insn" as well.

     3,856,436,920 stalled-cycles-frontend   #   74.09% frontend cycles idle
     1,600,790,871 stalled-cycles-backend    #   30.75% backend  cycles idle
     2,603,501,247 instructions              #    0.50  insns per cycle
                                             #    1.48  stalled cycles
per insn
       484,357,498 branches                  #  283.455 M/sec
Quoted 4 lines

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/builtin-stat.c#n24

The newer Intel CPU doesn't have the stalled-cycles-* events. But it seems power and older x86 CPU have the events.

Oh, sigh. This test should really ignore lines like that. How much do we care about these metrics? The RISC-V event parsing change:
https://lore.kernel.org/lkml/[email protected]/
means that legacy hardware events will be uncommon and we need to adapt the hard coded metrics in stat-shadow.c to json ones. Once they are json metrics they will be in Default.

Thanks,
Ian

Quoted 6 lines

Thanks,
Kan

Quoted 3 lines
tools/perf/tests/shell/stat+std_output.sh +0 -0
 cleanup() {
   rm -f "${stat_output}"
#3 lore raw
Quoted 39 lines · Ian Rogers
Quoted 33 lines · Kan
Quoted 12 lines · Ian Rogers

Hard coded metric names fail on ARM testing.

---
 tools/perf/tests/shell/stat+std_output.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/tests/shell/stat+std_output.sh +0 -0
index cbf2894b2c84..845f83213855 100755

tools/perf/tests/shell/stat+std_output.sh stat_output=$(mktemp /tmp/__perf_test.stat_output.std.XXXXX) · line 16

Quoted 5 lines
tools/perf/tests/shell/stat+std_output.sh +1 -1
@@ -13,7 +13,7 @@ stat_output=$(mktemp /tmp/__perf_test.stat_output.std.XXXXX)
 
 event_name=(cpu-clock task-clock context-switches cpu-migrations page-faults stalled-cycles-frontend stalled-cycles-backend cycles instructions branches branch-misses)
 event_metric=("CPUs utilized" "CPUs utilized" "/sec" "/sec" "/sec" "frontend cycles idle" "backend cycles idle" "GHz" "insn per cycle" "/sec" "of all branches")
-skip_metric=("stalled cycles per insn" "tma_" "retiring" "frontend_bound" "bad_speculation" "backend_bound")
+skip_metric=($(perf list --raw Default 2> /dev/null))

The "perf list --raw Default" only gives the topdown metrics. The "stalled cycles per insn" is not covered.
The check should skip the line of "stalled cycles per insn" as well.

     3,856,436,920 stalled-cycles-frontend   #   74.09% frontend cycles idle
     1,600,790,871 stalled-cycles-backend    #   30.75% backend  cycles idle
     2,603,501,247 instructions              #    0.50  insns per cycle
                                             #    1.48  stalled cycles
per insn
       484,357,498 branches                  #  283.455 M/sec
Quoted 4 lines

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/builtin-stat.c#n24

The newer Intel CPU doesn't have the stalled-cycles-* events. But it seems power and older x86 CPU have the events.

Quoted 6 lines

Oh, sigh. This test should really ignore lines like that. How much do we care about these metrics? The RISC-V event parsing change:
https://lore.kernel.org/lkml/[email protected]/
means that legacy hardware events will be uncommon and we need to adapt the hard coded metrics in stat-shadow.c to json ones. Once they are json metrics they will be in Default.

It seems except the newer Intel CPU, all the other ARCHs support the two stalled-cycles-* events and the metric. For Intel, there are Topdown metrics. But it seems an important metrics for the other ARCHs.

RISC-V
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/perf/riscv_pmu_sbi.c#n134
Power
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/powerpc/perf/power9-pmu.c#n279
Arm
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/perf/arm_pmuv3.c#n53

So almost all json files have to be updated. I'm not sure if it's a practical way to fix the issue.

Thanks,
Kan

Quoted 11 lines

Thanks,
Ian

Quoted 6 lines

Thanks,
Kan

Quoted 3 lines
tools/perf/tests/shell/stat+std_output.sh +0 -0
 cleanup() {
   rm -f "${stat_output}"
author reviewer
#4 lore raw
Quoted 61 lines · Kan
Quoted 39 lines · Ian Rogers
Quoted 33 lines · Kan
Quoted 12 lines · Ian Rogers

Hard coded metric names fail on ARM testing.

---
 tools/perf/tests/shell/stat+std_output.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/tests/shell/stat+std_output.sh +0 -0
index cbf2894b2c84..845f83213855 100755

tools/perf/tests/shell/stat+std_output.sh stat_output=$(mktemp /tmp/__perf_test.stat_output.std.XXXXX) · line 16

Quoted 5 lines
tools/perf/tests/shell/stat+std_output.sh +1 -1
@@ -13,7 +13,7 @@ stat_output=$(mktemp /tmp/__perf_test.stat_output.std.XXXXX)
 
 event_name=(cpu-clock task-clock context-switches cpu-migrations page-faults stalled-cycles-frontend stalled-cycles-backend cycles instructions branches branch-misses)
 event_metric=("CPUs utilized" "CPUs utilized" "/sec" "/sec" "/sec" "frontend cycles idle" "backend cycles idle" "GHz" "insn per cycle" "/sec" "of all branches")
-skip_metric=("stalled cycles per insn" "tma_" "retiring" "frontend_bound" "bad_speculation" "backend_bound")
+skip_metric=($(perf list --raw Default 2> /dev/null))

The "perf list --raw Default" only gives the topdown metrics. The "stalled cycles per insn" is not covered.
The check should skip the line of "stalled cycles per insn" as well.

     3,856,436,920 stalled-cycles-frontend   #   74.09% frontend cycles idle
     1,600,790,871 stalled-cycles-backend    #   30.75% backend  cycles idle
     2,603,501,247 instructions              #    0.50  insns per cycle
                                             #    1.48  stalled cycles
per insn
       484,357,498 branches                  #  283.455 M/sec
Quoted 4 lines

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/builtin-stat.c#n24

The newer Intel CPU doesn't have the stalled-cycles-* events. But it seems power and older x86 CPU have the events.

Quoted 6 lines

Oh, sigh. This test should really ignore lines like that. How much do we care about these metrics? The RISC-V event parsing change:
https://lore.kernel.org/lkml/[email protected]/
means that legacy hardware events will be uncommon and we need to adapt the hard coded metrics in stat-shadow.c to json ones. Once they are json metrics they will be in Default.

It seems except the newer Intel CPU, all the other ARCHs support the two stalled-cycles-* events and the metric. For Intel, there are Topdown metrics. But it seems an important metrics for the other ARCHs.

RISC-V
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/perf/riscv_pmu_sbi.c#n134
Power
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/powerpc/perf/power9-pmu.c#n279
Arm
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/perf/arm_pmuv3.c#n53

Quoted 2 lines

So almost all json files have to be updated. I'm not sure if it's a practical way to fix the issue.

So I'd very much like to get rid of the hard coded metrics:
- they don't use or respect event groups,
- their ad hoc printing can introduce extra metric results
unexpectedly in output,
- they fall outside of optimizations like Weilin's metric event grouping work.
I'm hoping the python json generation of metrics makes their removal practical:
https://lore.kernel.org/lkml/[email protected]/

That's a lot to get landed for this fix:
- 40+ patches for python based json generation.
- 10+ patches for parse events changes.
So I think a version that hard codes ignoring the hard coded metrics is in order.

Thanks,
Ian

Quoted 13 lines

Thanks,
Kan

Quoted 11 lines

Thanks,
Ian

Quoted 6 lines

Thanks,
Kan

Quoted 3 lines
tools/perf/tests/shell/stat+std_output.sh +0 -0
 cleanup() {
   rm -f "${stat_output}"
#5 lore raw
Quoted 65 lines · Ian Rogers
Quoted 61 lines · Kan
Quoted 39 lines · Ian Rogers
Quoted 33 lines · Kan
Quoted 12 lines · Ian Rogers

Hard coded metric names fail on ARM testing.

---
 tools/perf/tests/shell/stat+std_output.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/tests/shell/stat+std_output.sh +0 -0
index cbf2894b2c84..845f83213855 100755

tools/perf/tests/shell/stat+std_output.sh stat_output=$(mktemp /tmp/__perf_test.stat_outputstd.XXXXX) · line 16

Quoted 5 lines
tools/perf/tests/shell/stat+std_output.sh +1 -1
@@ -13,7 +13,7 @@ stat_output=$(mktemp /tmp/__perf_test.stat_outputstd.XXXXX)
 
 event_name=(cpu-clock task-clock context-switches cpu-migrations page-faults stalled-cycles-frontend stalled-cycles-backend cycles instructions branches branch-misses)
 event_metric=("CPUs utilized" "CPUs utilized" "/sec" "/sec" "/sec" "frontend cycles idle" "backend cycles idle" "GHz" "insn per cycle" "/sec" "of all branches")
-skip_metric=("stalled cycles per insn" "tma_" "retiring" "frontend_bound" "bad_speculation" "backend_bound")
+skip_metric=($(perf list --raw Default 2> /dev/null))

The "perf list --raw Default" only gives the topdown metrics. The "stalled cycles per insn" is not covered.
The check should skip the line of "stalled cycles per insn" as well.

     3,856,436,920 stalled-cycles-frontend   #   74.09% frontend cycles idle
     1,600,790,871 stalled-cycles-backend    #   30.75% backend  cycles idle
     2,603,501,247 instructions              #    0.50  insns per cycle
                                             #    1.48  stalled cycles
per insn
       484,357,498 branches                  #  283.455 M/sec
Quoted 4 lines

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/builtin-stat.c#n24

The newer Intel CPU doesn't have the stalled-cycles-* events. But it seems power and older x86 CPU have the events.

Quoted 6 lines

Oh, sigh. This test should really ignore lines like that. How much do we care about these metrics? The RISC-V event parsing change:
https://lore.kernel.org/lkml/[email protected]/
means that legacy hardware events will be uncommon and we need to adapt the hard coded metrics in stat-shadow.c to json ones. Once they are json metrics they will be in Default.

It seems except the newer Intel CPU, all the other ARCHs support the two stalled-cycles-* events and the metric. For Intel, there are Topdown metrics. But it seems an important metrics for the other ARCHs.

RISC-V
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/perf/riscv_pmu_sbi.c#n134
Power
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/powerpc/perf/power9-pmu.c#n279
Arm
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/perf/arm_pmuv3.c#n53

Quoted 2 lines

So almost all json files have to be updated. I'm not sure if it's a practical way to fix the issue.

Quoted 8 lines

So I'd very much like to get rid of the hard coded metrics:
- they don't use or respect event groups,
- their ad hoc printing can introduce extra metric results
unexpectedly in output,
- they fall outside of optimizations like Weilin's metric event grouping work.
I'm hoping the python json generation of metrics makes their removal practical:
https://lore.kernel.org/lkml/[email protected]/

It seems I wasn't listed in the Cc list, so it's missed in my main inbox. But I was added to the same series for the Arm and AMD. :)

I will take a look at the patches.

Quoted 5 lines

That's a lot to get landed for this fix:
- 40+ patches for python based json generation.
- 10+ patches for parse events changes.
So I think a version that hard codes ignoring the hard coded metrics is in order.

Sounds good to me.

Thanks,
Kan