perf test: Avoid hard coded metrics in stat std output test
Ian Rogers
reviewerPATCH 1/1 perf test: Avoid hard coded metrics in stat std output test
Hard coded metric names fail on ARM testing.
--- tools/perf/tests/shell/stat+std_output.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/tests/shell/stat+std_output.sh +1 -1
@@ -13,7 +13,7 @@ stat_output=$(mktemp /tmp/__perf_test.stat_output.std.XXXXX) event_name=(cpu-clock task-clock context-switches cpu-migrations page-faults stalled-cycles-frontend stalled-cycles-backend cycles instructions branches branch-misses) event_metric=("CPUs utilized" "CPUs utilized" "/sec" "/sec" "/sec" "frontend cycles idle" "backend cycles idle" "GHz" "insn per cycle" "/sec" "of all branches") -skip_metric=("stalled cycles per insn" "tma_" "retiring" "frontend_bound" "bad_speculation" "backend_bound") +skip_metric=($(perf list --raw Default 2> /dev/null)) cleanup() { rm -f "${stat_output}"
Signature · 1 line
2.44.0.683.g7961c838ac-goog
Quoted 12 lines · Ian Rogers
Hard coded metric names fail on ARM testing.
Signed-off-by Ian Rogers <[email protected]>--- tools/perf/tests/shell/stat+std_output.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tools/perf/tests/shell/stat+std_output.sh +0
-0
tools/perf/tests/shell/stat+std_output.sh stat_output=$(mktemp /tmp/__perf_test.stat_output.std.XXXXX) · line 16
Quoted 5 lines
tools/perf/tests/shell/stat+std_output.sh +1
-1@@ -13,7 +13,7 @@ stat_output=$(mktemp /tmp/__perf_test.stat_output.std.XXXXX) event_name=(cpu-clock task-clock context-switches cpu-migrations page-faults stalled-cycles-frontend stalled-cycles-backend cycles instructions branches branch-misses) event_metric=("CPUs utilized" "CPUs utilized" "/sec" "/sec" "/sec" "frontend cycles idle" "backend cycles idle" "GHz" "insn per cycle" "/sec" "of all branches") -skip_metric=("stalled cycles per insn" "tma_" "retiring" "frontend_bound" "bad_speculation" "backend_bound") +skip_metric=($(perf list --raw Default 2> /dev/null))
The "perf list --raw Default" only gives the topdown metrics. The "stalled cycles per insn" is not covered.
The check should skip the line of "stalled cycles per insn" as well.
3,856,436,920 stalled-cycles-frontend # 74.09% frontend cycles idle
1,600,790,871 stalled-cycles-backend # 30.75% backend cycles idle
2,603,501,247 instructions # 0.50 insns per cycle
# 1.48 stalled cycles
per insn
484,357,498 branches # 283.455 M/secThe newer Intel CPU doesn't have the stalled-cycles-* events. But it seems power and older x86 CPU have the events.
Thanks,
Kan
Quoted 3 lines
tools/perf/tests/shell/stat+std_output.sh +0
-0cleanup() { rm -f "${stat_output}"
Ian Rogers
reviewerQuoted 33 lines · Kan
Quoted 12 lines · Ian Rogers
Hard coded metric names fail on ARM testing.
Signed-off-by Ian Rogers <[email protected]>--- tools/perf/tests/shell/stat+std_output.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tools/perf/tests/shell/stat+std_output.sh +0
-0tools/perf/tests/shell/stat+std_output.sh stat_output=$(mktemp /tmp/__perf_test.stat_output.std.XXXXX) · line 16
Quoted 5 lines
tools/perf/tests/shell/stat+std_output.sh +1
-1@@ -13,7 +13,7 @@ stat_output=$(mktemp /tmp/__perf_test.stat_output.std.XXXXX) event_name=(cpu-clock task-clock context-switches cpu-migrations page-faults stalled-cycles-frontend stalled-cycles-backend cycles instructions branches branch-misses) event_metric=("CPUs utilized" "CPUs utilized" "/sec" "/sec" "/sec" "frontend cycles idle" "backend cycles idle" "GHz" "insn per cycle" "/sec" "of all branches") -skip_metric=("stalled cycles per insn" "tma_" "retiring" "frontend_bound" "bad_speculation" "backend_bound") +skip_metric=($(perf list --raw Default 2> /dev/null))The "perf list --raw Default" only gives the topdown metrics. The "stalled cycles per insn" is not covered.
The check should skip the line of "stalled cycles per insn" as well.3,856,436,920 stalled-cycles-frontend # 74.09% frontend cycles idle 1,600,790,871 stalled-cycles-backend # 30.75% backend cycles idle 2,603,501,247 instructions # 0.50 insns per cycle # 1.48 stalled cycles per insn 484,357,498 branches # 283.455 M/sec
Quoted 4 lines
The newer Intel CPU doesn't have the stalled-cycles-* events. But it seems power and older x86 CPU have the events.
Oh, sigh. This test should really ignore lines like that. How much do we care about these metrics? The RISC-V event parsing change:
https://lore.kernel.org/lkml/[email protected]/
means that legacy hardware events will be uncommon and we need to adapt the hard coded metrics in stat-shadow.c to json ones. Once they are json metrics they will be in Default.
Thanks,
Ian
Quoted 6 lines
Thanks,
KanQuoted 3 lines
tools/perf/tests/shell/stat+std_output.sh +0
-0cleanup() { rm -f "${stat_output}"
Quoted 39 lines · Ian Rogers
Quoted 33 lines · Kan
Quoted 12 lines · Ian Rogers
Hard coded metric names fail on ARM testing.
Signed-off-by Ian Rogers <[email protected]>--- tools/perf/tests/shell/stat+std_output.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tools/perf/tests/shell/stat+std_output.sh +0
-0tools/perf/tests/shell/stat+std_output.sh stat_output=$(mktemp /tmp/__perf_test.stat_output.std.XXXXX) · line 16
Quoted 5 lines
tools/perf/tests/shell/stat+std_output.sh +1
-1@@ -13,7 +13,7 @@ stat_output=$(mktemp /tmp/__perf_test.stat_output.std.XXXXX) event_name=(cpu-clock task-clock context-switches cpu-migrations page-faults stalled-cycles-frontend stalled-cycles-backend cycles instructions branches branch-misses) event_metric=("CPUs utilized" "CPUs utilized" "/sec" "/sec" "/sec" "frontend cycles idle" "backend cycles idle" "GHz" "insn per cycle" "/sec" "of all branches") -skip_metric=("stalled cycles per insn" "tma_" "retiring" "frontend_bound" "bad_speculation" "backend_bound") +skip_metric=($(perf list --raw Default 2> /dev/null))The "perf list --raw Default" only gives the topdown metrics. The "stalled cycles per insn" is not covered.
The check should skip the line of "stalled cycles per insn" as well.3,856,436,920 stalled-cycles-frontend # 74.09% frontend cycles idle 1,600,790,871 stalled-cycles-backend # 30.75% backend cycles idle 2,603,501,247 instructions # 0.50 insns per cycle # 1.48 stalled cycles per insn 484,357,498 branches # 283.455 M/secQuoted 4 lines
The newer Intel CPU doesn't have the stalled-cycles-* events. But it seems power and older x86 CPU have the events.
Quoted 6 lines
Oh, sigh. This test should really ignore lines like that. How much do we care about these metrics? The RISC-V event parsing change:
https://lore.kernel.org/lkml/[email protected]/
means that legacy hardware events will be uncommon and we need to adapt the hard coded metrics instat-shadow.cto json ones. Once they are json metrics they will be in Default.
It seems except the newer Intel CPU, all the other ARCHs support the two stalled-cycles-* events and the metric. For Intel, there are Topdown metrics. But it seems an important metrics for the other ARCHs.
RISC-V
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/perf/riscv_pmu_sbi.c#n134
Power
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/powerpc/perf/power9-pmu.c#n279
Arm
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/perf/arm_pmuv3.c#n53
So almost all json files have to be updated. I'm not sure if it's a practical way to fix the issue.
Thanks,
Kan
Quoted 11 lines
Thanks,
IanQuoted 6 lines
Thanks,
KanQuoted 3 lines
tools/perf/tests/shell/stat+std_output.sh +0
-0cleanup() { rm -f "${stat_output}"
Ian Rogers
reviewerQuoted 61 lines · Kan
Quoted 39 lines · Ian Rogers
Quoted 33 lines · Kan
Quoted 12 lines · Ian Rogers
Hard coded metric names fail on ARM testing.
Signed-off-by Ian Rogers <[email protected]>--- tools/perf/tests/shell/stat+std_output.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tools/perf/tests/shell/stat+std_output.sh +0
-0tools/perf/tests/shell/stat+std_output.sh stat_output=$(mktemp /tmp/__perf_test.stat_output.std.XXXXX) · line 16
Quoted 5 lines
tools/perf/tests/shell/stat+std_output.sh +1
-1@@ -13,7 +13,7 @@ stat_output=$(mktemp /tmp/__perf_test.stat_output.std.XXXXX) event_name=(cpu-clock task-clock context-switches cpu-migrations page-faults stalled-cycles-frontend stalled-cycles-backend cycles instructions branches branch-misses) event_metric=("CPUs utilized" "CPUs utilized" "/sec" "/sec" "/sec" "frontend cycles idle" "backend cycles idle" "GHz" "insn per cycle" "/sec" "of all branches") -skip_metric=("stalled cycles per insn" "tma_" "retiring" "frontend_bound" "bad_speculation" "backend_bound") +skip_metric=($(perf list --raw Default 2> /dev/null))The "perf list --raw Default" only gives the topdown metrics. The "stalled cycles per insn" is not covered.
The check should skip the line of "stalled cycles per insn" as well.3,856,436,920 stalled-cycles-frontend # 74.09% frontend cycles idle 1,600,790,871 stalled-cycles-backend # 30.75% backend cycles idle 2,603,501,247 instructions # 0.50 insns per cycle # 1.48 stalled cycles per insn 484,357,498 branches # 283.455 M/secQuoted 4 lines
The newer Intel CPU doesn't have the stalled-cycles-* events. But it seems power and older x86 CPU have the events.
Quoted 6 lines
Oh, sigh. This test should really ignore lines like that. How much do we care about these metrics? The RISC-V event parsing change:
https://lore.kernel.org/lkml/[email protected]/
means that legacy hardware events will be uncommon and we need to adapt the hard coded metrics instat-shadow.cto json ones. Once they are json metrics they will be in Default.It seems except the newer Intel CPU, all the other ARCHs support the two stalled-cycles-* events and the metric. For Intel, there are Topdown metrics. But it seems an important metrics for the other ARCHs.
RISC-V
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/perf/riscv_pmu_sbi.c#n134
Power
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/powerpc/perf/power9-pmu.c#n279
Arm
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/perf/arm_pmuv3.c#n53
Quoted 2 lines
So almost all json files have to be updated. I'm not sure if it's a practical way to fix the issue.
So I'd very much like to get rid of the hard coded metrics:
- they don't use or respect event groups,
- their ad hoc printing can introduce extra metric results
unexpectedly in output,
- they fall outside of optimizations like Weilin's metric event grouping work.
I'm hoping the python json generation of metrics makes their removal practical:
https://lore.kernel.org/lkml/[email protected]/
That's a lot to get landed for this fix:
- 40+ patches for python based json generation.
- 10+ patches for parse events changes.
So I think a version that hard codes ignoring the hard coded metrics is in order.
Thanks,
Ian
Quoted 13 lines
Thanks,
KanQuoted 11 lines
Thanks,
IanQuoted 6 lines
Thanks,
KanQuoted 3 lines
tools/perf/tests/shell/stat+std_output.sh +0
-0cleanup() { rm -f "${stat_output}"
Quoted 65 lines · Ian Rogers
Quoted 61 lines · Kan
Quoted 39 lines · Ian Rogers
Quoted 33 lines · Kan
Quoted 12 lines · Ian Rogers
Hard coded metric names fail on ARM testing.
Signed-off-by Ian Rogers <[email protected]>--- tools/perf/tests/shell/stat+std_output.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tools/perf/tests/shell/stat+std_output.sh +0
-0tools/perf/tests/shell/stat+std_output.sh stat_output=$(mktemp /tmp/__perf_test.stat_outputstd.XXXXX) · line 16
Quoted 5 lines
tools/perf/tests/shell/stat+std_output.sh +1
-1@@ -13,7 +13,7 @@ stat_output=$(mktemp /tmp/__perf_test.stat_outputstd.XXXXX) event_name=(cpu-clock task-clock context-switches cpu-migrations page-faults stalled-cycles-frontend stalled-cycles-backend cycles instructions branches branch-misses) event_metric=("CPUs utilized" "CPUs utilized" "/sec" "/sec" "/sec" "frontend cycles idle" "backend cycles idle" "GHz" "insn per cycle" "/sec" "of all branches") -skip_metric=("stalled cycles per insn" "tma_" "retiring" "frontend_bound" "bad_speculation" "backend_bound") +skip_metric=($(perf list --raw Default 2> /dev/null))The "perf list --raw Default" only gives the topdown metrics. The "stalled cycles per insn" is not covered.
The check should skip the line of "stalled cycles per insn" as well.3,856,436,920 stalled-cycles-frontend # 74.09% frontend cycles idle 1,600,790,871 stalled-cycles-backend # 30.75% backend cycles idle 2,603,501,247 instructions # 0.50 insns per cycle # 1.48 stalled cycles per insn 484,357,498 branches # 283.455 M/secQuoted 4 lines
The newer Intel CPU doesn't have the stalled-cycles-* events. But it seems power and older x86 CPU have the events.
Quoted 6 lines
Oh, sigh. This test should really ignore lines like that. How much do we care about these metrics? The RISC-V event parsing change:
https://lore.kernel.org/lkml/[email protected]/
means that legacy hardware events will be uncommon and we need to adapt the hard coded metrics instat-shadow.cto json ones. Once they are json metrics they will be in Default.It seems except the newer Intel CPU, all the other ARCHs support the two stalled-cycles-* events and the metric. For Intel, there are Topdown metrics. But it seems an important metrics for the other ARCHs.
RISC-V
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/perf/riscv_pmu_sbi.c#n134
Power
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/powerpc/perf/power9-pmu.c#n279
Arm
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/perf/arm_pmuv3.c#n53Quoted 2 lines
So almost all json files have to be updated. I'm not sure if it's a practical way to fix the issue.
Quoted 8 lines
So I'd very much like to get rid of the hard coded metrics:
- they don't use or respect event groups,
- their ad hoc printing can introduce extra metric results
unexpectedly in output,
- they fall outside of optimizations like Weilin's metric event grouping work.
I'm hoping the python json generation of metrics makes their removal practical:
https://lore.kernel.org/lkml/[email protected]/
It seems I wasn't listed in the Cc list, so it's missed in my main inbox. But I was added to the same series for the Arm and AMD. :)
I will take a look at the patches.
Quoted 5 lines
That's a lot to get landed for this fix:
- 40+ patches for python based json generation.
- 10+ patches for parse events changes.
So I think a version that hard codes ignoring the hard coded metrics is in order.
Sounds good to me.
Thanks,
Kan