Skip to contentSkip to conversation

'perf top' broken on intel hybrid systems

DISCUSSION Arnaldo Carvalho de Melo 4 participants 6 messages Dec 8, 2023 – Dec 11, 2023 1 message not in the archive perf events

lore
Share & tools
lore

Arnaldo Carvalho de Melo

author maintainer Tested-by Kan Liang
#0 lore raw

Hi,

So I finally got a recent Intel hybrid system:

root@fedora:~# grep -m1 "model name" /proc/cpuinfo
model name	: Intel(R) Core(TM) i7-14700K
root@fedora:~#

Most things work, but:

root@fedora:~# perf top

Error:
The cycles:P event is not supported.
root@fedora:~#

root@fedora:~# perf top -e cycles:p
Error:
The cycles:p event is not supported.
root@fedora:~# perf top -e cycles:pp
Error:
The cycles:pp event is not supported.
^[[Aroot@fedora:~# perf top -e cycles:ppp
Error:
The cycles:ppp event is not supported.
root@fedora:~#
root@fedora:~# perf top -e cycles
Error:
The cycles event is not supported.
root@fedora:~#
root@fedora:~# perf top -e instructions
Error:
The instructions event is not supported.
root@fedora:~#root@fedora:~# perf top -e cache-misses
Error:
The cache-misses event is not supported.
root@fedora:~#

------------------------------------------------------------
perf_event_attr:

  type                             0 (PERF_TYPE_HARDWARE)
  size                             136
  config                           0xa00000000
  { sample_period, sample_freq }   4000
  sample_type                      IP|TID|TIME|ID|CPU|PERIOD
  read_format                      ID|LOST
  disabled                         1
  inherit                          1
  mmap                             1
  comm                             1
  freq                             1
  task                             1
  sample_id_all                    1
  exclude_guest                    1
  mmap2                            1
  comm_exec                        1
  ksymbol                          1
  bpf_event                        1

------------------------------------------------------------
sys_perf_event_open: pid -1 cpu 0 group_fd -1 flags 0x8 sys_perf_event_open failed, error -2
Error:
The cycles event is not supported.
root@fedora:~#

But...

root@fedora:~# perf record
^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 2.631 MB perf.data (24079 samples) ]
root@fedora:~# perf evlist -v
cpu_atom/cycles:P/: type: 0 (PERF_TYPE_HARDWARE), size: 136, config: 0xa00000000, { sample_period, sample_freq }: 4000, sample_type: IP|TID|TIME|CPU|PERIOD|IDENTIFIER, read_format: ID|LOST, disabled: 1, inherit: 1, freq: 1, precise_ip: 3, sample_id_all: 1
cpu_core/cycles:P/: type: 0 (PERF_TYPE_HARDWARE), size: 136, config: 0x400000000, { sample_period, sample_freq }: 4000, sample_type: IP|TID|TIME|CPU|PERIOD|IDENTIFIER, read_format: ID|LOST, disabled: 1, inherit: 1, freq: 1, precise_ip: 3, sample_id_all: 1
dummy:u: type: 1 (PERF_TYPE_SOFTWARE), size: 136, config: 0x9 (PERF_COUNT_SW_DUMMY), { sample_period, sample_freq }: 1, sample_type: IP|TID|TIME|CPU|IDENTIFIER, read_format: ID|LOST, inherit: 1, exclude_kernel: 1, exclude_hv: 1, mmap: 1, comm: 1, task: 1, sample_id_all: 1, exclude_guest: 1, mmap2: 1, comm_exec: 1, ksymbol: 1, bpf_event: 1
root@fedora:~#
root@fedora:~# perf top -e cpu_core/cpu-cycles:P/
event syntax error: 'cpu_core/cpu-cycles:P/'
                              \___ Bad event or PMU

Unable to find PMU or event on a PMU of 'cpu_core'

Initial error:
event syntax error: 'cpu_core/cpu-cycles:P/'
\___ unknown term 'cpu-cycles:P' for pmu 'cpu_core'

valid terms: event,pc,edge,offcore_rsp,ldlat,inv,umask,frontend,cmask,config,config1,config2,config3,name,period,freq,branch_type,time,call-graph,stack-size,no-inherit,inherit,max-stack,nr,no-overwrite,overwrite,driver-config,percore,aux-output,aux-sample-size,metric-id,raw,legacy-cache,hardware Run 'perf list' for a list of valid events

Usage: perf top [<options>]

-e, --event <event> event selector. use 'perf list' to list available events
root@fedora:~#

So it seems its just 'perf top' breakage:

root@fedora:~# perf top -e cpu_core/cpu-cycles/
Error:
The cpu_core/cpu-cycles/ event is not supported.
root@fedora:~# perf top -e cpu_atom/cpu-cycles/
Error:
The cpu_atom/cpu-cycles/ event is not supported.
root@fedora:~#

I bisected this to:

⬢[acme@toolbox perf-tools-next]$ git bisect good 251aa040244a3b17068e4e6ec61f138d7e50681a is the first bad commit commit 251aa040244a3b17068e4e6ec61f138d7e50681a

Author: Ian Rogers <[email protected]>
Date:   Thu Jun 1 01:29:53 2023 -0700

perf parse-events: Wildcard most "numeric" events

Numeric events are either raw events or those with ABI defined numbers matched by the lexer. PERF_TYPE_HARDWARE and PERF_TYPE_HW_CACHE events should wildcard match on hybrid systems. So "cycles" should match each PMU type with an extended type, not just PERF_TYPE_HARDWARE.

Change wildcard matching to add the event even if wildcard PMU scanning fails, there will be no extended type but this best matches previous behavior.

Only set the extended type when the event type supports it and when perf_pmus__supports_extended_type is true. This new function returns true if >1 core PMU and avoids potential errors on older kernels.

Modify evsel__compute_group_pmu_name using a helper perf_pmu__is_software to determine when grouping should occur. Try to use PMUs, and evsel__find_pmu, as being more dependable than evsel->pmu_name.

Set a parse events error if a hardware term's PMU lookup fails, to provide extra diagnostics.

Fixes 8bc75f699c141420 ("perf parse-events: Support wildcards on raw events")
Cc Adrian Hunter <[email protected]>
Cc Alexander Shishkin <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Ravi Bangoria <[email protected]>
Cc Rob Herring <[email protected]>
Cc Thomas Richter <[email protected]>
Cc Xing Zhengjun <[email protected]>
 tools/perf/util/parse-events.c | 104 +++++++++++++++++++++++++++++------------
 tools/perf/util/parse-events.y |  10 ++--
 tools/perf/util/pmu.c          |  16 +++++++
 tools/perf/util/pmu.h          |   5 ++
 tools/perf/util/pmus.c         |   5 ++
 tools/perf/util/pmus.h         |   1 +
 6 files changed, 106 insertions(+), 35 deletions(-)

⬢[acme@toolbox perf-tools-next]$

- Arnaldo

#1 lore raw
Quoted 59 lines · Arnaldo Carvalho de Melo

Hi,

So I finally got a recent Intel hybrid system:

root@fedora:~# grep -m1 "model name" /proc/cpuinfo
model name	: Intel(R) Core(TM) i7-14700K
root@fedora:~#

Most things work, but:

root@fedora:~# perf top

Error:
The cycles:P event is not supported.
root@fedora:~#

root@fedora:~# perf top -e cycles:p
Error:
The cycles:p event is not supported.
root@fedora:~# perf top -e cycles:pp
Error:
The cycles:pp event is not supported.
^[[Aroot@fedora:~# perf top -e cycles:ppp
Error:
The cycles:ppp event is not supported.
root@fedora:~#
root@fedora:~# perf top -e cycles
Error:
The cycles event is not supported.
root@fedora:~#
root@fedora:~# perf top -e instructions
Error:
The instructions event is not supported.
root@fedora:~#root@fedora:~# perf top -e cache-misses
Error:
The cache-misses event is not supported.
root@fedora:~#

------------------------------------------------------------
perf_event_attr:

  type                             0 (PERF_TYPE_HARDWARE)
  size                             136
  config                           0xa00000000
  { sample_period, sample_freq }   4000
  sample_type                      IP|TID|TIME|ID|CPU|PERIOD
  read_format                      ID|LOST
  disabled                         1
  inherit                          1
  mmap                             1
  comm                             1
  freq                             1
  task                             1
  sample_id_all                    1
  exclude_guest                    1
  mmap2                            1
  comm_exec                        1
  ksymbol                          1
  bpf_event                        1
Quoted 3 lines

------------------------------------------------------------
sys_perf_event_open: pid -1 cpu 0 group_fd -1 flags 0x8 sys_perf_event_open failed, error -2

The error is because the perf top always tries to open an event on the user_requested_cpus, which are all CPUs by default.

Something as below should fix it. For hybrid, open a PMU event on an unsupported CPU should be error out.

tools/perf/builtin-top.c +0 -0
index 1e42bd1c7d5a..7aa29e6264e4 100644
@@ -1027,8 +1027,8 @@ static int perf_top__start_counters(struct

perf_top *top)

        evlist__for_each_entry(evlist, counter) {

try_again:
- if (evsel__open(counter,
top->evlist->core.user_requested_cpus,
- top->evlist->core.threads) < 0) {
+ if (evsel__open(counter, counter->core.cpus,
+ counter->core.threads) < 0) {

/*
* Specially handle overwrite fall back.

Quoted 21 lines

Error:
The cycles event is not supported.
root@fedora:~#

But...

root@fedora:~# perf record
^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 2.631 MB perf.data (24079 samples) ]
root@fedora:~# perf evlist -v
cpu_atom/cycles:P/: type: 0 (PERF_TYPE_HARDWARE), size: 136, config: 0xa00000000, { sample_period, sample_freq }: 4000, sample_type: IP|TID|TIME|CPU|PERIOD|IDENTIFIER, read_format: ID|LOST, disabled: 1, inherit: 1, freq: 1, precise_ip: 3, sample_id_all: 1
cpu_core/cycles:P/: type: 0 (PERF_TYPE_HARDWARE), size: 136, config: 0x400000000, { sample_period, sample_freq }: 4000, sample_type: IP|TID|TIME|CPU|PERIOD|IDENTIFIER, read_format: ID|LOST, disabled: 1, inherit: 1, freq: 1, precise_ip: 3, sample_id_all: 1
dummy:u: type: 1 (PERF_TYPE_SOFTWARE), size: 136, config: 0x9 (PERF_COUNT_SW_DUMMY), { sample_period, sample_freq }: 1, sample_type: IP|TID|TIME|CPU|IDENTIFIER, read_format: ID|LOST, inherit: 1, exclude_kernel: 1, exclude_hv: 1, mmap: 1, comm: 1, task: 1, sample_id_all: 1, exclude_guest: 1, mmap2: 1, comm_exec: 1, ksymbol: 1, bpf_event: 1
root@fedora:~#
root@fedora:~# perf top -e cpu_core/cpu-cycles:P/
event syntax error: 'cpu_core/cpu-cycles:P/'
                              \___ Bad event or PMU
Quoted 5 lines

Unable to find PMU or event on a PMU of 'cpu_core'

Initial error:
event syntax error: 'cpu_core/cpu-cycles:P/'
\___ unknown term 'cpu-cycles:P' for pmu 'cpu_core'

The P or other event modifier should be after the last '/'. Something as below:
perf top -e cpu_core/cpu-cycles/P

Even for the non-hybrid or perf record, the /cpu-cycles:P/ format is incorrect.

It looks like by design, not a bug.

$ perf record -e cpu/cycles:P/
event syntax error: 'cpu/cycles:P/'
                         \___ unknown term 'cycles:P' for pmu 'cpu'

valid terms:
event,pc,edge,any,offcore_rsp,ldlat,inv,umask,frontend,cmask,config,config1,config2,config3,name,period,freq,branch_type,time,call-graph,stack-size,no-inherit,inherit,max-stack,nr,no-overwrite,overwrite,driver-config,percore,aux-output,aux-sample-size,metric-id,raw,legacy-cache,hardware

Initial error:
event syntax error: 'cpu/cycles:P/'
\___ unknown term 'cycles:P' for pmu 'cpu'

$ perf record -e cpu/cycles/P
^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.695 MB perf.data (254 samples) ]

Thanks,
Kan

Quoted 78 lines

valid terms: event,pc,edge,offcore_rsp,ldlat,inv,umask,frontend,cmask,config,config1,config2,config3,name,period,freq,branch_type,time,call-graph,stack-size,no-inherit,inherit,max-stack,nr,no-overwrite,overwrite,driver-config,percore,aux-output,aux-sample-size,metric-id,raw,legacy-cache,hardware Run 'perf list' for a list of valid events

Usage: perf top [<options>]

-e, --event <event> event selector. use 'perf list' to list available events
root@fedora:~#

So it seems its just 'perf top' breakage:

root@fedora:~# perf top -e cpu_core/cpu-cycles/
Error:
The cpu_core/cpu-cycles/ event is not supported.
root@fedora:~# perf top -e cpu_atom/cpu-cycles/
Error:
The cpu_atom/cpu-cycles/ event is not supported.
root@fedora:~#

I bisected this to:

⬢[acme@toolbox perf-tools-next]$ git bisect good 251aa040244a3b17068e4e6ec61f138d7e50681a is the first bad commit commit 251aa040244a3b17068e4e6ec61f138d7e50681a

Author: Ian Rogers <[email protected]>
Date:   Thu Jun 1 01:29:53 2023 -0700

perf parse-events: Wildcard most "numeric" events

Numeric events are either raw events or those with ABI defined numbers matched by the lexer. PERF_TYPE_HARDWARE and PERF_TYPE_HW_CACHE events should wildcard match on hybrid systems. So "cycles" should match each PMU type with an extended type, not just PERF_TYPE_HARDWARE.

Change wildcard matching to add the event even if wildcard PMU scanning fails, there will be no extended type but this best matches previous behavior.

Only set the extended type when the event type supports it and when perf_pmus__supports_extended_type is true. This new function returns true if >1 core PMU and avoids potential errors on older kernels.

Modify evsel__compute_group_pmu_name using a helper perf_pmu__is_software to determine when grouping should occur. Try to use PMUs, and evsel__find_pmu, as being more dependable than evsel->pmu_name.

Set a parse events error if a hardware term's PMU lookup fails, to provide extra diagnostics.

Fixes 8bc75f699c141420 ("perf parse-events: Support wildcards on raw events")
Cc Adrian Hunter <[email protected]>
Cc Alexander Shishkin <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Ravi Bangoria <[email protected]>
Cc Rob Herring <[email protected]>
Cc Thomas Richter <[email protected]>
Cc Xing Zhengjun <[email protected]>
 tools/perf/util/parse-events.c | 104 +++++++++++++++++++++++++++++------------
 tools/perf/util/parse-events.y |  10 ++--
 tools/perf/util/pmu.c          |  16 +++++++
 tools/perf/util/pmu.h          |   5 ++
 tools/perf/util/pmus.c         |   5 ++
 tools/perf/util/pmus.h         |   1 +
 6 files changed, 106 insertions(+), 35 deletions(-)

⬢[acme@toolbox perf-tools-next]$

- Arnaldo

#2 lore raw

tools/perf/builtin-top.c

Quoted 8 lines

The error is because the perf top always tries to open an event on the user_requested_cpus, which are all CPUs by default.

Something as below should fix it. For hybrid, open a PMU event on an unsupported CPU should be error out.

tools/perf/builtin-top.c +0 -0

This fixes "perf top" on my i7-1355U. It would be great if you could get this patch into upstream and stable branches.

Thanks,

Signature · 1 line
Leah Neukirchen  <[email protected]>  https://leahneukirchen.org/
reviewer
#4 lore raw
In reply to a message not in the archive
Quoted 10 lines · Ian Rogers
Quoted 8 lines · Leah Neukirchen
Quoted 8 lines

The error is because the perf top always tries to open an event on the user_requested_cpus, which are all CPUs by default.

Something as below should fix it. For hybrid, open a PMU event on an unsupported CPU should be error out.

Quoted 8 lines
Quoted 4 lines

tools/perf/builtin-top.c

Quoted 1 line
tools/perf/builtin-top.c +0 -0

This fixes "perf top" on my i7-1355U. It would be great if you could get this patch into upstream and stable branches.

I'll try to take a look ASAP. As we don't need this for record there is some existing logic that perf top is clearly missing. We should also have a test on perf top.

So for stat and record there is a call to evlist__create_maps:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/builtin-stat.c?h=perf-tools-next#n2730
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/builtin-record.c?h=perf-tools-next#n4210

In evlist__create_maps we do perf_evlist__propagate_maps and that will:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/lib/perf/evlist.c?h=perf-tools-next#n43

```
} else if (evlist->has_user_cpus && evsel->is_pmu_core) {
/*
* User requested CPUs on a core PMU, ensure the requested CPUs
* are valid by intersecting with those of the PMU.
*/
perf_cpu_map__put(evsel->cpus);
evsel->cpus = perf_cpu_map__intersect(evlist->user_requested_cpus,
evsel->own_cpus);
```

I think the fix should be looking to do the same map set up in top as stat and record, presumably there was a refactor and top was overlooked. I'll try to address this and add a basic sanity test of doing a few iterations of perf top on stdio, which would have been sufficient to catch this.

Thanks,
Ian

author maintainer
#5 lore raw
In reply to Liang, Kan

Em Fri, Dec 08, 2023 at 02:39:37PM -0500, Liang, Kan escreveu:

Quoted 28 lines · Arnaldo Carvalho de Melo
Quoted 12 lines

So I finally got a recent Intel hybrid system:

root@fedora:~# grep -m1 "model name" /proc/cpuinfo
model name	: Intel(R) Core(TM) i7-14700K
root@fedora:~#
	Most things work, but:
root@fedora:~# perf top

Error:
The cycles:P event is not supported.
root@fedora:~#

Quoted 14 lines
root@fedora:~# perf top -e cycles:p
Error:
The cycles:p event is not supported.
root@fedora:~# perf top -e cycles:pp
Error:
The cycles:pp event is not supported.
^[[Aroot@fedora:~# perf top -e cycles:ppp
Error:
The cycles:ppp event is not supported.
root@fedora:~#
root@fedora:~# perf top -e cycles
Error:
The cycles event is not supported.
root@fedora:~#
Quoted 2 lines

The error is because the perf top always tries to open an event on the user_requested_cpus, which are all CPUs by default.

But what is wrong with that for the default event, CPU cycles?

It should work for all CPUs, its the most basic event, right?

We should have a rough idea where CPU (no matter which CPUs) cycles are being used.

- Arnaldo

reviewer
#6 lore raw
Quoted 33 lines · Arnaldo Carvalho de Melo

Em Fri, Dec 08, 2023 at 02:39:37PM -0500, Liang, Kan escreveu:

Quoted 28 lines · Arnaldo Carvalho de Melo
Quoted 12 lines

So I finally got a recent Intel hybrid system:

root@fedora:~# grep -m1 "model name" /proc/cpuinfo
model name  : Intel(R) Core(TM) i7-14700K
root@fedora:~#
    Most things work, but:
root@fedora:~# perf top

Error:
The cycles:P event is not supported.
root@fedora:~#

Quoted 14 lines
root@fedora:~# perf top -e cycles:p
Error:
The cycles:p event is not supported.
root@fedora:~# perf top -e cycles:pp
Error:
The cycles:pp event is not supported.
^[[Aroot@fedora:~# perf top -e cycles:ppp
Error:
The cycles:ppp event is not supported.
root@fedora:~#
root@fedora:~# perf top -e cycles
Error:
The cycles event is not supported.
root@fedora:~#
Quoted 2 lines

The error is because the perf top always tries to open an event on the user_requested_cpus, which are all CPUs by default.

Quoted 6 lines

But what is wrong with that for the default event, CPU cycles?

It should work for all CPUs, its the most basic event, right?

We should have a rough idea where CPU (no matter which CPUs) cycles are being used.

Generally we wouldn't want to aggregate cycles events on two differing PMUs as the CPUs for them likely behave quite differently. Most of the cycle event opening is now done by parsing "cycles:P" which will give two evsels, one for each core PMU. You're right that the correct PMU can be determined for the legacy events without an extended type if the CPU isn't the -1 any CPU value. We'd need special parsing to make this work in the context of perf top, which doesn't seem desirable. I think we should support wild card PMUs and fix the CPU maps like we do in perf stat and perf record.

Thanks,
Ian