'perf top' broken on intel hybrid systems
Hi,
So I finally got a recent Intel hybrid system:
root@fedora:~# grep -m1 "model name" /proc/cpuinfo model name : Intel(R) Core(TM) i7-14700K root@fedora:~#
Most things work, but:
root@fedora:~# perf top
Error:
The cycles:P event is not supported.
root@fedora:~#
root@fedora:~# perf top -e cycles:p Error: The cycles:p event is not supported. root@fedora:~# perf top -e cycles:pp Error: The cycles:pp event is not supported. ^[[Aroot@fedora:~# perf top -e cycles:ppp Error: The cycles:ppp event is not supported. root@fedora:~# root@fedora:~# perf top -e cycles Error: The cycles event is not supported. root@fedora:~#
root@fedora:~# perf top -e instructions Error: The instructions event is not supported. root@fedora:~#root@fedora:~# perf top -e cache-misses Error: The cache-misses event is not supported. root@fedora:~#
------------------------------------------------------------perf_event_attr:
type 0 (PERF_TYPE_HARDWARE)
size 136
config 0xa00000000
{ sample_period, sample_freq } 4000
sample_type IP|TID|TIME|ID|CPU|PERIOD
read_format ID|LOST
disabled 1
inherit 1
mmap 1
comm 1
freq 1
task 1
sample_id_all 1
exclude_guest 1
mmap2 1
comm_exec 1
ksymbol 1
bpf_event 1------------------------------------------------------------sys_perf_event_open: pid -1 cpu 0 group_fd -1 flags 0x8 sys_perf_event_open failed, error -2
Error:
The cycles event is not supported.
root@fedora:~#
But...
root@fedora:~# perf record ^C[ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 2.631 MB perf.data (24079 samples) ]
root@fedora:~# perf evlist -v
cpu_atom/cycles:P/: type: 0 (PERF_TYPE_HARDWARE), size: 136, config: 0xa00000000, { sample_period, sample_freq }: 4000, sample_type: IP|TID|TIME|CPU|PERIOD|IDENTIFIER, read_format: ID|LOST, disabled: 1, inherit: 1, freq: 1, precise_ip: 3, sample_id_all: 1
cpu_core/cycles:P/: type: 0 (PERF_TYPE_HARDWARE), size: 136, config: 0x400000000, { sample_period, sample_freq }: 4000, sample_type: IP|TID|TIME|CPU|PERIOD|IDENTIFIER, read_format: ID|LOST, disabled: 1, inherit: 1, freq: 1, precise_ip: 3, sample_id_all: 1
dummy:u: type: 1 (PERF_TYPE_SOFTWARE), size: 136, config: 0x9 (PERF_COUNT_SW_DUMMY), { sample_period, sample_freq }: 1, sample_type: IP|TID|TIME|CPU|IDENTIFIER, read_format: ID|LOST, inherit: 1, exclude_kernel: 1, exclude_hv: 1, mmap: 1, comm: 1, task: 1, sample_id_all: 1, exclude_guest: 1, mmap2: 1, comm_exec: 1, ksymbol: 1, bpf_event: 1
root@fedora:~#root@fedora:~# perf top -e cpu_core/cpu-cycles:P/
event syntax error: 'cpu_core/cpu-cycles:P/'
\___ Bad event or PMUUnable to find PMU or event on a PMU of 'cpu_core'
Initial error:
event syntax error: 'cpu_core/cpu-cycles:P/'
\___ unknown term 'cpu-cycles:P' for pmu 'cpu_core'
valid terms: event,pc,edge,offcore_rsp,ldlat,inv,umask,frontend,cmask,config,config1,config2,config3,name,period,freq,branch_type,time,call-graph,stack-size,no-inherit,inherit,max-stack,nr,no-overwrite,overwrite,driver-config,percore,aux-output,aux-sample-size,metric-id,raw,legacy-cache,hardware Run 'perf list' for a list of valid events
Usage: perf top [<options>]
-e, --event <event> event selector. use 'perf list' to list available events
root@fedora:~#
So it seems its just 'perf top' breakage:
root@fedora:~# perf top -e cpu_core/cpu-cycles/ Error: The cpu_core/cpu-cycles/ event is not supported. root@fedora:~# perf top -e cpu_atom/cpu-cycles/ Error: The cpu_atom/cpu-cycles/ event is not supported. root@fedora:~#
I bisected this to:
⬢[acme@toolbox perf-tools-next]$ git bisect good 251aa040244a3b17068e4e6ec61f138d7e50681a is the first bad commit commit 251aa040244a3b17068e4e6ec61f138d7e50681a
Author: Ian Rogers <[email protected]> Date: Thu Jun 1 01:29:53 2023 -0700
perf parse-events: Wildcard most "numeric" events
Numeric events are either raw events or those with ABI defined numbers matched by the lexer. PERF_TYPE_HARDWARE and PERF_TYPE_HW_CACHE events should wildcard match on hybrid systems. So "cycles" should match each PMU type with an extended type, not just PERF_TYPE_HARDWARE.
Change wildcard matching to add the event even if wildcard PMU scanning fails, there will be no extended type but this best matches previous behavior.
Only set the extended type when the event type supports it and when perf_pmus__supports_extended_type is true. This new function returns true if >1 core PMU and avoids potential errors on older kernels.
Modify evsel__compute_group_pmu_name using a helper perf_pmu__is_software to determine when grouping should occur. Try to use PMUs, and evsel__find_pmu, as being more dependable than evsel->pmu_name.
Set a parse events error if a hardware term's PMU lookup fails, to provide extra diagnostics.
tools/perf/util/parse-events.c | 104 +++++++++++++++++++++++++++++------------ tools/perf/util/parse-events.y | 10 ++-- tools/perf/util/pmu.c | 16 +++++++ tools/perf/util/pmu.h | 5 ++ tools/perf/util/pmus.c | 5 ++ tools/perf/util/pmus.h | 1 + 6 files changed, 106 insertions(+), 35 deletions(-)
⬢[acme@toolbox perf-tools-next]$
- Arnaldo
Quoted 59 lines · Arnaldo Carvalho de Melo
Hi,
So I finally got a recent Intel hybrid system:
root@fedora:~# grep -m1 "model name" /proc/cpuinfo model name : Intel(R) Core(TM) i7-14700K root@fedora:~#Most things work, but:
root@fedora:~# perf topError:
The cycles:P event is not supported.
root@fedora:~#root@fedora:~# perf top -e cycles:p Error: The cycles:p event is not supported. root@fedora:~# perf top -e cycles:pp Error: The cycles:pp event is not supported. ^[[Aroot@fedora:~# perf top -e cycles:ppp Error: The cycles:ppp event is not supported. root@fedora:~# root@fedora:~# perf top -e cycles Error: The cycles event is not supported. root@fedora:~#root@fedora:~# perf top -e instructions Error: The instructions event is not supported. root@fedora:~#root@fedora:~# perf top -e cache-misses Error: The cache-misses event is not supported. root@fedora:~#------------------------------------------------------------
perf_event_attr:type 0 (PERF_TYPE_HARDWARE) size 136 config 0xa00000000 { sample_period, sample_freq } 4000 sample_type IP|TID|TIME|ID|CPU|PERIOD read_format ID|LOST disabled 1 inherit 1 mmap 1 comm 1 freq 1 task 1 sample_id_all 1 exclude_guest 1 mmap2 1 comm_exec 1 ksymbol 1 bpf_event 1
Quoted 3 lines
------------------------------------------------------------
sys_perf_event_open: pid -1 cpu 0group_fd-1 flags0x8sys_perf_event_openfailed, error -2
The error is because the perf top always tries to open an event on the user_requested_cpus, which are all CPUs by default.
Something as below should fix it. For hybrid, open a PMU event on an unsupported CPU should be error out.
tools/perf/builtin-top.c +0 -0
@@ -1027,8 +1027,8 @@ static int perf_top__start_counters(struct
perf_top *top)
evlist__for_each_entry(evlist, counter) { try_again:
- if (evsel__open(counter,top->evlist->core.user_requested_cpus,
- top->evlist->core.threads) < 0) {
+ if (evsel__open(counter, counter->core.cpus,
+ counter->core.threads) < 0) {
/*
* Specially handle overwrite fall back.
Quoted 21 lines
Error:
The cycles event is not supported.
root@fedora:~#But...
root@fedora:~# perf record ^C[ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 2.631 MB perf.data (24079 samples) ]root@fedora:~# perf evlist -v cpu_atom/cycles:P/: type: 0 (PERF_TYPE_HARDWARE), size: 136, config: 0xa00000000, { sample_period, sample_freq }: 4000, sample_type: IP|TID|TIME|CPU|PERIOD|IDENTIFIER, read_format: ID|LOST, disabled: 1, inherit: 1, freq: 1, precise_ip: 3, sample_id_all: 1 cpu_core/cycles:P/: type: 0 (PERF_TYPE_HARDWARE), size: 136, config: 0x400000000, { sample_period, sample_freq }: 4000, sample_type: IP|TID|TIME|CPU|PERIOD|IDENTIFIER, read_format: ID|LOST, disabled: 1, inherit: 1, freq: 1, precise_ip: 3, sample_id_all: 1 dummy:u: type: 1 (PERF_TYPE_SOFTWARE), size: 136, config: 0x9 (PERF_COUNT_SW_DUMMY), { sample_period, sample_freq }: 1, sample_type: IP|TID|TIME|CPU|IDENTIFIER, read_format: ID|LOST, inherit: 1, exclude_kernel: 1, exclude_hv: 1, mmap: 1, comm: 1, task: 1, sample_id_all: 1, exclude_guest: 1, mmap2: 1, comm_exec: 1, ksymbol: 1, bpf_event: 1 root@fedora:~#root@fedora:~# perf top -e cpu_core/cpu-cycles:P/ event syntax error: 'cpu_core/cpu-cycles:P/' \___ Bad event or PMU
Quoted 5 lines
Unable to find PMU or event on a PMU of '
cpu_core'Initial error:
event syntax error: 'cpu_core/cpu-cycles:P/'
\___ unknown term 'cpu-cycles:P' for pmu 'cpu_core'
The P or other event modifier should be after the last '/'. Something as below:
perf top -e cpu_core/cpu-cycles/P
Even for the non-hybrid or perf record, the /cpu-cycles:P/ format is incorrect.
It looks like by design, not a bug.
$ perf record -e cpu/cycles:P/
event syntax error: 'cpu/cycles:P/'
\___ unknown term 'cycles:P' for pmu 'cpu'valid terms:
event,pc,edge,any,offcore_rsp,ldlat,inv,umask,frontend,cmask,config,config1,config2,config3,name,period,freq,branch_type,time,call-graph,stack-size,no-inherit,inherit,max-stack,nr,no-overwrite,overwrite,driver-config,percore,aux-output,aux-sample-size,metric-id,raw,legacy-cache,hardware
Initial error:
event syntax error: 'cpu/cycles:P/'
\___ unknown term 'cycles:P' for pmu 'cpu'
$ perf record -e cpu/cycles/P ^C[ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.695 MB perf.data (254 samples) ]
Thanks,
Kan
Quoted 78 lines
valid terms: event,pc,edge,offcore_rsp,ldlat,inv,umask,frontend,cmask,config,config1,config2,config3,name,period,freq,branch_type,time,call-graph,stack-size,no-inherit,inherit,max-stack,nr,no-overwrite,overwrite,driver-config,percore,aux-output,aux-sample-size,metric-id,raw,legacy-cache,hardware Run 'perf list' for a list of valid events
Usage: perf top [<options>]
-e, --event <event> event selector. use 'perf list' to list available events
root@fedora:~#So it seems its just 'perf top' breakage:
root@fedora:~# perf top -e cpu_core/cpu-cycles/ Error: The cpu_core/cpu-cycles/ event is not supported. root@fedora:~# perf top -e cpu_atom/cpu-cycles/ Error: The cpu_atom/cpu-cycles/ event is not supported. root@fedora:~#I bisected this to:
⬢[acme@toolbox perf-tools-next]$ git bisect good
251aa040244a3b17068e4e6ec61f138d7e50681ais the first bad commit commit251aa040244a3b17068e4e6ec61f138d7e50681aAuthor: Ian Rogers <[email protected]> Date: Thu Jun 1 01:29:53 2023 -0700perf parse-events: Wildcard most "numeric" events
Numeric events are either raw events or those with ABI defined numbers matched by the lexer.
PERF_TYPE_HARDWAREandPERF_TYPE_HW_CACHEevents should wildcard match on hybrid systems. So "cycles" should match each PMU type with an extended type, not justPERF_TYPE_HARDWARE.Change wildcard matching to add the event even if wildcard PMU scanning fails, there will be no extended type but this best matches previous behavior.
Only set the extended type when the event type supports it and when
perf_pmus__supports_extended_typeis true. This new function returns true if >1 core PMU and avoids potential errors on older kernels.Modify
evsel__compute_group_pmu_nameusing a helperperf_pmu__is_softwareto determine when grouping should occur. Try to use PMUs, andevsel__find_pmu, as being more dependable thanevsel->pmu_name.Set a parse events error if a hardware term's PMU lookup fails, to provide extra diagnostics.
Fixes 8bc75f699c141420 ("perf parse-events: Support wildcards on raw events")Reported-by Kan Liang <[email protected]>Signed-off-by Ian Rogers <[email protected]>Tested-by Kan Liang <[email protected]>Cc Adrian Hunter <[email protected]>Cc Alexander Shishkin <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jiri Olsa <[email protected]>Cc Mark Rutland <[email protected]>Cc Namhyung Kim <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Ravi Bangoria <[email protected]>Cc Rob Herring <[email protected]>Cc Thomas Richter <[email protected]>Cc Xing Zhengjun <[email protected]>Signed-off-by Arnaldo Carvalho de Melo <[email protected]>tools/perf/util/parse-events.c | 104 +++++++++++++++++++++++++++++------------ tools/perf/util/parse-events.y | 10 ++-- tools/perf/util/pmu.c | 16 +++++++ tools/perf/util/pmu.h | 5 ++ tools/perf/util/pmus.c | 5 ++ tools/perf/util/pmus.h | 1 + 6 files changed, 106 insertions(+), 35 deletions(-)⬢[acme@toolbox perf-tools-next]$
- Arnaldo
tools/perf/builtin-top.c
Quoted 8 lines
The error is because the perf top always tries to open an event on the
user_requested_cpus, which are all CPUs by default.Something as below should fix it. For hybrid, open a PMU event on an unsupported CPU should be error out.
tools/perf/builtin-top.c +0
-0
This fixes "perf top" on my i7-1355U. It would be great if you could get this patch into upstream and stable branches.
Thanks,
Signature · 1 line
Leah Neukirchen <[email protected]> https://leahneukirchen.org/
Ian Rogers
reviewerQuoted 10 lines · Ian Rogers
Quoted 8 lines · Leah Neukirchen
Quoted 8 lines
The error is because the perf top always tries to open an event on the
user_requested_cpus, which are all CPUs by default.Something as below should fix it. For hybrid, open a PMU event on an unsupported CPU should be error out.
Quoted 8 lines
Quoted 4 lines
tools/perf/builtin-top.c
Quoted 1 line
tools/perf/builtin-top.c +0
-0This fixes "perf top" on my i7-1355U. It would be great if you could get this patch into upstream and stable branches.
I'll try to take a look ASAP. As we don't need this for record there is some existing logic that perf top is clearly missing. We should also have a test on perf top.
So for stat and record there is a call to evlist__create_maps:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/builtin-stat.c?h=perf-tools-next#n2730
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/builtin-record.c?h=perf-tools-next#n4210
In evlist__create_maps we do perf_evlist__propagate_maps and that will:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/lib/perf/evlist.c?h=perf-tools-next#n43
```
} else if (evlist->has_user_cpus && evsel->is_pmu_core) {
/*
* User requested CPUs on a core PMU, ensure the requested CPUs
* are valid by intersecting with those of the PMU.
*/
perf_cpu_map__put(evsel->cpus);
evsel->cpus = perf_cpu_map__intersect(evlist->user_requested_cpus,
evsel->own_cpus);
```I think the fix should be looking to do the same map set up in top as stat and record, presumably there was a refactor and top was overlooked. I'll try to address this and add a basic sanity test of doing a few iterations of perf top on stdio, which would have been sufficient to catch this.
Thanks,
Ian
Arnaldo Carvalho de Melo
maintainerEm Fri, Dec 08, 2023 at 02:39:37PM -0500, Liang, Kan escreveu:
Quoted 28 lines · Arnaldo Carvalho de Melo
Quoted 12 lines
So I finally got a recent Intel hybrid system:
root@fedora:~# grep -m1 "model name" /proc/cpuinfo model name : Intel(R) Core(TM) i7-14700K root@fedora:~# Most things work, but:root@fedora:~# perf topError:
The cycles:P event is not supported.
root@fedora:~#Quoted 14 lines
root@fedora:~# perf top -e cycles:p Error: The cycles:p event is not supported. root@fedora:~# perf top -e cycles:pp Error: The cycles:pp event is not supported. ^[[Aroot@fedora:~# perf top -e cycles:ppp Error: The cycles:ppp event is not supported. root@fedora:~# root@fedora:~# perf top -e cycles Error: The cycles event is not supported. root@fedora:~#
Quoted 2 lines
The error is because the perf top always tries to open an event on the
user_requested_cpus, which are all CPUs by default.
But what is wrong with that for the default event, CPU cycles?
It should work for all CPUs, its the most basic event, right?
We should have a rough idea where CPU (no matter which CPUs) cycles are being used.
- Arnaldo
Ian Rogers
reviewerQuoted 33 lines · Arnaldo Carvalho de Melo
Em Fri, Dec 08, 2023 at 02:39:37PM -0500, Liang, Kan escreveu:
Quoted 28 lines · Arnaldo Carvalho de Melo
Quoted 12 lines
So I finally got a recent Intel hybrid system:
root@fedora:~# grep -m1 "model name" /proc/cpuinfo model name : Intel(R) Core(TM) i7-14700K root@fedora:~# Most things work, but:root@fedora:~# perf topError:
The cycles:P event is not supported.
root@fedora:~#Quoted 14 lines
root@fedora:~# perf top -e cycles:p Error: The cycles:p event is not supported. root@fedora:~# perf top -e cycles:pp Error: The cycles:pp event is not supported. ^[[Aroot@fedora:~# perf top -e cycles:ppp Error: The cycles:ppp event is not supported. root@fedora:~# root@fedora:~# perf top -e cycles Error: The cycles event is not supported. root@fedora:~#Quoted 2 lines
The error is because the perf top always tries to open an event on the
user_requested_cpus, which are all CPUs by default.
Quoted 6 lines
But what is wrong with that for the default event, CPU cycles?
It should work for all CPUs, its the most basic event, right?
We should have a rough idea where CPU (no matter which CPUs) cycles are being used.
Generally we wouldn't want to aggregate cycles events on two differing PMUs as the CPUs for them likely behave quite differently. Most of the cycle event opening is now done by parsing "cycles:P" which will give two evsels, one for each core PMU. You're right that the correct PMU can be determined for the legacy events without an extended type if the CPU isn't the -1 any CPU value. We'd need special parsing to make this work in the context of perf top, which doesn't seem desirable. I think we should support wild card PMUs and fix the CPU maps like we do in perf stat and perf record.
Thanks,
Ian