Skip to contentSkip to conversation

perf stat: Fix crash on arm64

PATCH Superseded Breno Leitao 5 participants 14 messages Feb 5 – Mar 24 perf events

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 perf stat: Fix crash on arm64

Perf stat is crashing on arm64 hosts with the following issue:

	# make -C tools/perf DEBUG=1
	# perf stat sleep 1
	perf: util/evsel.c:2034: get_group_fd: Assertion `!(!leader->core.fd)' failed.
	[1]    1220794 IOT instruction (core dumped)  ./perf stat

The sorting function introduced by commit a745c0831c15c ("perf stat:
Sort default events/metrics") compares events based on their individual properties. This can cause events from different groups to be interleaved, resulting in group members appearing before their leaders in the sorted evlist.

When the iterator opens events in list order, a group member may be processed before its leader has been opened.

For example, CPU_CYCLES (idx=32) with leader STALL_SLOT_BACKEND (idx=37) could be sorted before its leader, causing the crash when CPU_CYCLES tries to get its group fd from the not-yet-opened leader.

Fix this by comparing events based on their leader's attributes instead of their own attributes when the events are in different groups. This ensures all members of a group share the same sort key as their leader, keeping groups together and guaranteeing leaders are opened before their members.

Fixes a745c0831c15c ("perf stat: Sort default events/metrics")

---
Cc; [email protected]

---
 tools/perf/builtin-stat.c | 26 +++++++++++++++++---------
 1 file changed, 17 insertions(+), 9 deletions(-)
tools/perf/builtin-stat.c +17 -9
index ab40d85fb1259..3a423ca31d8d3 100644
@@ -1938,25 +1938,33 @@ static int default_evlist_evsel_cmp(void *priv __maybe_unused,
 	const struct evsel *lhs = container_of(lhs_core, struct evsel, core);
 	const struct perf_evsel *rhs_core = container_of(r, struct perf_evsel, node);
 	const struct evsel *rhs = container_of(rhs_core, struct evsel, core);
+	const struct evsel *lhs_leader = evsel__leader(lhs);
+	const struct evsel *rhs_leader = evsel__leader(rhs);
 
-	if (evsel__leader(lhs) == evsel__leader(rhs)) {
+	if (lhs_leader == rhs_leader) {
 		/* Within the same group, respect the original order. */
 		return lhs_core->idx - rhs_core->idx;
 	}
 
+	/*
+	 * Compare using leader's attributes so that all members of a group
+	 * stay together. This ensures leaders are opened before their members.
+	 */
+
 	/* Sort default metrics evsels first, and default show events before those. */
-	if (lhs->default_metricgroup != rhs->default_metricgroup)
-		return lhs->default_metricgroup ? -1 : 1;
+	if (lhs_leader->default_metricgroup != rhs_leader->default_metricgroup)
+		return lhs_leader->default_metricgroup ? -1 : 1;
 
-	if (lhs->default_show_events != rhs->default_show_events)
-		return lhs->default_show_events ? -1 : 1;
+	if (lhs_leader->default_show_events != rhs_leader->default_show_events)
+		return lhs_leader->default_show_events ? -1 : 1;
 
 	/* Sort by PMU type (prefers legacy types first). */
-	if (lhs->pmu != rhs->pmu)
-		return lhs->pmu->type - rhs->pmu->type;
+	if (lhs_leader->pmu != rhs_leader->pmu)
+		return lhs_leader->pmu->type - rhs_leader->pmu->type;
 
-	/* Sort by name. */
-	return strcmp(evsel__name((struct evsel *)lhs), evsel__name((struct evsel *)rhs));
+	/* Sort by leader's name. */
+	return strcmp(evsel__name((struct evsel *)lhs_leader),
+		      evsel__name((struct evsel *)rhs_leader));
 }
 
 /*

---
base-commit: 5fd0a1df5d05ad066e5618ccdd3d0fa6cb686c27
change-id: 20260205-perf_stat-a0a2a37e21c5

Best regards,

Signature · 1 line
Breno Leitao <[email protected]>
Tested-by on v1 1/1
#1 lore raw
Quoted 87 lines · Breno Leitao

Perf stat is crashing on arm64 hosts with the following issue:

	# make -C tools/perf DEBUG=1
	# perf stat sleep 1
	perf: util/evsel.c:2034: get_group_fd: Assertion `!(!leader->core.fd)' failed.
	[1]    1220794 IOT instruction (core dumped)  ./perf stat

The sorting function introduced by commit a745c0831c15c ("perf stat:
Sort default events/metrics") compares events based on their individual properties. This can cause events from different groups to be interleaved, resulting in group members appearing before their leaders in the sorted evlist.

When the iterator opens events in list order, a group member may be processed before its leader has been opened.

For example, CPU_CYCLES (idx=32) with leader STALL_SLOT_BACKEND (idx=37) could be sorted before its leader, causing the crash when CPU_CYCLES tries to get its group fd from the not-yet-opened leader.

Fix this by comparing events based on their leader's attributes instead of their own attributes when the events are in different groups. This ensures all members of a group share the same sort key as their leader, keeping groups together and guaranteeing leaders are opened before their members.

Fixes a745c0831c15c ("perf stat: Sort default events/metrics")

---
Cc; [email protected]

---
 tools/perf/builtin-stat.c | 26 +++++++++++++++++---------
 1 file changed, 17 insertions(+), 9 deletions(-)
tools/perf/builtin-stat.c +17 -10
index ab40d85fb1259..3a423ca31d8d3 100644
@@ -1938,25 +1938,33 @@ static int default_evlist_evsel_cmp(void *priv __maybe_unused,
 	const struct evsel *lhs = container_of(lhs_core, struct evsel, core);
 	const struct perf_evsel *rhs_core = container_of(r, struct perf_evsel, node);
 	const struct evsel *rhs = container_of(rhs_core, struct evsel, core);
+	const struct evsel *lhs_leader = evsel__leader(lhs);
+	const struct evsel *rhs_leader = evsel__leader(rhs);
 
-	if (evsel__leader(lhs) == evsel__leader(rhs)) {
+	if (lhs_leader == rhs_leader) {
 		/* Within the same group, respect the original order. */
 		return lhs_core->idx - rhs_core->idx;
 	}
 
+	/*
+	 * Compare using leader's attributes so that all members of a group
+	 * stay together. This ensures leaders are opened before their members.
+	 */
+
 	/* Sort default metrics evsels first, and default show events before those. */
-	if (lhs->default_metricgroup != rhs->default_metricgroup)
-		return lhs->default_metricgroup ? -1 : 1;
+	if (lhs_leader->default_metricgroup != rhs_leader->default_metricgroup)
+		return lhs_leader->default_metricgroup ? -1 : 1;
 
-	if (lhs->default_show_events != rhs->default_show_events)
-		return lhs->default_show_events ? -1 : 1;
+	if (lhs_leader->default_show_events != rhs_leader->default_show_events)
+		return lhs_leader->default_show_events ? -1 : 1;
 
 	/* Sort by PMU type (prefers legacy types first). */
-	if (lhs->pmu != rhs->pmu)
-		return lhs->pmu->type - rhs->pmu->type;
+	if (lhs_leader->pmu != rhs_leader->pmu)
+		return lhs_leader->pmu->type - rhs_leader->pmu->type;
 
-	/* Sort by name. */
-	return strcmp(evsel__name((struct evsel *)lhs), evsel__name((struct evsel *)rhs));
+	/* Sort by leader's name. */
+	return strcmp(evsel__name((struct evsel *)lhs_leader),
+		      evsel__name((struct evsel *)rhs_leader));
 }
 
 /*
 
---

base-commit: 5fd0a1df5d05ad066e5618ccdd3d0fa6cb686c27
change-id: 20260205-perf_stat-a0a2a37e21c5

Quoted 4 lines

Best regards,

Signature · 1 line
Breno Leitao <[email protected]>
reviewer
#2 lore raw
In reply to Breno Leitao
Quoted 8 lines · Breno Leitao

Perf stat is crashing on arm64 hosts with the following issue:

        # make -C tools/perf DEBUG=1
        # perf stat sleep 1
        perf: util/evsel.c:2034: get_group_fd: Assertion `!(!leader->core.fd)' failed.
        [1]    1220794 IOT instruction (core dumped)  ./perf stat
Quoted 5 lines

The sorting function introduced by commit a745c0831c15c ("perf stat:
Sort default events/metrics") compares events based on their individual properties. This can cause events from different groups to be interleaved, resulting in group members appearing before their leaders in the sorted evlist.

Hi, sorry for the issue. I can see what you're saying but why is this an arm64 issue? The legacy Default metrics are common to all architectures:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/pmu-events/arch/common/common/metrics.json?h=perf-tools-next

Quoted 6 lines

When the iterator opens events in list order, a group member may be processed before its leader has been opened.

For example, CPU_CYCLES (idx=32) with leader STALL_SLOT_BACKEND (idx=37) could be sorted before its leader, causing the crash when CPU_CYCLES tries to get its group fd from the not-yet-opened leader.

Which metric is this?

Quoted 5 lines

Fix this by comparing events based on their leader's attributes instead of their own attributes when the events are in different groups. This ensures all members of a group share the same sort key as their leader, keeping groups together and guaranteeing leaders are opened before their members.

This makes sense but I'm not understanding why this problem wasn't seen previously. I'm guessing that in a metric like backend_cycles_idle:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/pmu-events/arch/common/common/metrics.json?h=perf-tools-next#n63

```
        "BriefDescription": "Backend stalls per cycle",
        "MetricExpr": "stalled\\-cycles\\-backend / cpu\\-cycles",
        "MetricGroup": "Default",
        "MetricName": "backend_cycles_idle",
        "MetricThreshold": "backend_cycles_idle > 0.2",
        "DefaultShowEvents": "1"
```

The PMUs for cpu-cycles and stalled-cycles differ? This may mean we also need to be smarting in determining PMUs for legacy events.

It'd be interesting to see what events are coming from the kernel, e.g.:

```
$ ls /sys/bus/event_source/devices/*/events
/sys/bus/event_source/devices/cpu_atom/events:
branch-instructions  cache-misses      instructions  ref-cycles
topdown-fe-bound
branch-misses        cache-references  mem-loads     topdown-bad-spec
topdown-retiring
bus-cycles           cpu-cycles        mem-stores    topdown-be-bound
...
```
and the cpuid to match it up with the json.
```
$ perf stat -v sleep 1 2>&1 |head -1
Using CPUID GenuineIntel-6-B7-1
$ ./tools/perf/pmu-events/models.py x86 GenuineIntel-6-B7-1
tools/perf/pmu-events/arch/
alderlake
```
this information is in the verbose output too:
```
$ perf stat -vv sleep 1
...
------------------------------------------------------------
perf_event_attr:
 type                             1 (PERF_TYPE_SOFTWARE)
 size                             136
 config                           0x2 (PERF_COUNT_SW_PAGE_FAULTS)
 sample_type                      IDENTIFIER
 read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
 disabled                         1
 inherit                          1
 enable_on_exec                   1
------------------------------------------------------------
sys_perf_event_open: pid 608809  cpu -1  group_fd -1  flags 0x8 = 7
------------------------------------------------------------
perf_event_attr:
 type                             0 (PERF_TYPE_HARDWARE)
 size                             136
 config                           0xa00000001
(cpu_atom/PERF_COUNT_HW_INSTRUCTIONS/)
 sample_type                      IDENTIFIER
 read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
 disabled                         1
 inherit                          1
 enable_on_exec                   1
------------------------------------------------------------
sys_perf_event_open: pid 608809  cpu -1  group_fd -1  flags 0x8 = 8
...
```

Thanks,
Ian

Quoted 65 lines
Fixes a745c0831c15c ("perf stat: Sort default events/metrics")

---
Cc; [email protected]

---
 tools/perf/builtin-stat.c | 26 +++++++++++++++++---------
 1 file changed, 17 insertions(+), 9 deletions(-)
tools/perf/builtin-stat.c +17 -10
index ab40d85fb1259..3a423ca31d8d3 100644
@@ -1938,25 +1938,33 @@ static int default_evlist_evsel_cmp(void *priv __maybe_unused,
        const struct evsel *lhs = container_of(lhs_core, struct evsel, core);
        const struct perf_evsel *rhs_core = container_of(r, struct perf_evsel, node);
        const struct evsel *rhs = container_of(rhs_core, struct evsel, core);
+       const struct evsel *lhs_leader = evsel__leader(lhs);
+       const struct evsel *rhs_leader = evsel__leader(rhs);
 
-       if (evsel__leader(lhs) == evsel__leader(rhs)) {
+       if (lhs_leader == rhs_leader) {
                /* Within the same group, respect the original order. */
                return lhs_core->idx - rhs_core->idx;
        }
 
+       /*
+        * Compare using leader's attributes so that all members of a group
+        * stay together. This ensures leaders are opened before their members.
+        */
+
        /* Sort default metrics evsels first, and default show events before those. */
-       if (lhs->default_metricgroup != rhs->default_metricgroup)
-               return lhs->default_metricgroup ? -1 : 1;
+       if (lhs_leader->default_metricgroup != rhs_leader->default_metricgroup)
+               return lhs_leader->default_metricgroup ? -1 : 1;
 
-       if (lhs->default_show_events != rhs->default_show_events)
-               return lhs->default_show_events ? -1 : 1;
+       if (lhs_leader->default_show_events != rhs_leader->default_show_events)
+               return lhs_leader->default_show_events ? -1 : 1;
 
        /* Sort by PMU type (prefers legacy types first). */
-       if (lhs->pmu != rhs->pmu)
-               return lhs->pmu->type - rhs->pmu->type;
+       if (lhs_leader->pmu != rhs_leader->pmu)
+               return lhs_leader->pmu->type - rhs_leader->pmu->type;
 
-       /* Sort by name. */
-       return strcmp(evsel__name((struct evsel *)lhs), evsel__name((struct evsel *)rhs));
+       /* Sort by leader's name. */
+       return strcmp(evsel__name((struct evsel *)lhs_leader),
+                     evsel__name((struct evsel *)rhs_leader));
 }
 
 /*
 
---

base-commit: 5fd0a1df5d05ad066e5618ccdd3d0fa6cb686c27
change-id: 20260205-perf_stat-a0a2a37e21c5

Best regards,
--
Breno Leitao <[email protected]>

#3 lore raw

Hi Ian,

Quoted 15 lines · Ian Rogers
Quoted 8 lines · Breno Leitao

Perf stat is crashing on arm64 hosts with the following issue:

        # make -C tools/perf DEBUG=1
        # perf stat sleep 1
        perf: util/evsel.c:2034: get_group_fd: Assertion `!(!leader->core.fd)' failed.
        [1]    1220794 IOT instruction (core dumped)  ./perf stat
Quoted 5 lines

The sorting function introduced by commit a745c0831c15c ("perf stat:
Sort default events/metrics") compares events based on their individual properties. This can cause events from different groups to be interleaved, resulting in group members appearing before their leaders in the sorted evlist.

Quoted 4 lines

Hi, sorry for the issue. I can see what you're saying but why is this an arm64 issue? The legacy Default metrics are common to all architectures:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/pmu-events/arch/common/common/metrics.json?h=perf-tools-next

Since you are mentioning common metrics, I found the common metrics does not work on Arm64 platform (I built with NO_JEVENTS=1 or enabled jevnts but both don't work).

The latest perf will have no any output if the CPU type is missed in json and rallback to common metrics. The failure path is:

add_default_events()
metricgroup__parse_groups()
pmu_metrics_table__find() => return NULL

In my case, pmu_metrics_table__find() always return NULL, as a result, perf stat sleep 1 directly bail out without any output.

I expect Breno's env might have the corresponding CPU json files, this is possible different from my test machine.

Thanks,
Leo

#4 lore raw
Quoted 22 lines · Leo Yan
Quoted 10 lines
Quoted 5 lines

The sorting function introduced by commit a745c0831c15c ("perf stat:
Sort default events/metrics") compares events based on their individual properties. This can cause events from different groups to be interleaved, resulting in group members appearing before their leaders in the sorted evlist.

Hi, sorry for the issue. I can see what you're saying but why is this an arm64 issue? The legacy Default metrics are common to all architectures:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/pmu-events/arch/common/common/metrics.json?h=perf-tools-next

Since you are mentioning common metrics, I found the common metrics does not work on Arm64 platform (I built with NO_JEVENTS=1 or enabled jevnts but both don't work).

The latest perf will have no any output if the CPU type is missed in json and rallback to common metrics. The failure path is:

add_default_events()
metricgroup__parse_groups()
pmu_metrics_table__find() => return NULL

Quoted 5 lines

In my case, pmu_metrics_table__find() always return NULL, as a result, perf stat sleep 1 directly bail out without any output.

I expect Breno's env might have the corresponding CPU json files, this is possible different from my test machine.

On my local env, I need a fix:

tools/perf/pmu-events/empty-pmu-events.c +11 -8
index e4d00f6b2b5d..f74acc206856 100644
@@ -3237,14 +3237,6 @@ const struct pmu_events_table *perf_pmu__default_core_events_table(void)
         return NULL;
 }
 
-const struct pmu_metrics_table *pmu_metrics_table__find(void)
-{
-        struct perf_cpu cpu = {-1};
-        const struct pmu_events_map *map = map_for_cpu(cpu);
-
-        return map ? &map->metric_table : NULL;
-}
-
 const struct pmu_metrics_table *pmu_metrics_table__default(void)
 {
         int i = 0;
@@ -3261,6 +3253,17 @@ const struct pmu_metrics_table *pmu_metrics_table__default(void)
         return NULL;
 }
 
+const struct pmu_metrics_table *pmu_metrics_table__find(void)
+{
+        struct perf_cpu cpu = {-1};
+        const struct pmu_events_map *map = map_for_cpu(cpu);
+
+       if (map)
+               return &map->metric_table;
+
+       return pmu_metrics_table__default();
+}
+
 

I have no deep understanding for jevents, seems to me, Breno's issue is a different one from me. Please kindly confirm.

Thanks,
Leo

reviewer
#5 lore raw
Quoted 18 lines · Leo Yan
Quoted 15 lines · Leo Yan
Quoted 10 lines
Quoted 5 lines

The sorting function introduced by commit a745c0831c15c ("perf stat:
Sort default events/metrics") compares events based on their individual properties. This can cause events from different groups to be interleaved, resulting in group members appearing before their leaders in the sorted evlist.

Hi, sorry for the issue. I can see what you're saying but why is this an arm64 issue? The legacy Default metrics are common to all architectures:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/pmu-events/arch/common/common/metrics.json?h=perf-tools-next

Since you are mentioning common metrics, I found the common metrics does not work on Arm64 platform (I built with NO_JEVENTS=1 or enabled jevnts but both don't work).

Quoted 7 lines
Quoted 7 lines

The latest perf will have no any output if the CPU type is missed in json and rallback to common metrics. The failure path is:

add_default_events()
metricgroup__parse_groups()
pmu_metrics_table__find() => return NULL

The return is correct but the early return is wrong, the metric code was updated to always consider the default table and skip a NULL table:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/metricgroup.c?h=perf-tools-next#n430
I'll send a patch for the early return.

Quoted 43 lines
Quoted 5 lines

In my case, pmu_metrics_table__find() always return NULL, as a result, perf stat sleep 1 directly bail out without any output.

I expect Breno's env might have the corresponding CPU json files, this is possible different from my test machine.

On my local env, I need a fix:

tools/perf/pmu-events/empty-pmu-events.c +11 -8
index e4d00f6b2b5d..f74acc206856 100644
@@ -3237,14 +3237,6 @@ const struct pmu_events_table *perf_pmu__default_core_events_table(void)
         return NULL;
 }
 
-const struct pmu_metrics_table *pmu_metrics_table__find(void)
-{
-        struct perf_cpu cpu = {-1};
-        const struct pmu_events_map *map = map_for_cpu(cpu);
-
-        return map ? &map->metric_table : NULL;
-}
-
 const struct pmu_metrics_table *pmu_metrics_table__default(void)
 {
         int i = 0;
@@ -3261,6 +3253,17 @@ const struct pmu_metrics_table *pmu_metrics_table__default(void)
         return NULL;
 }
 
+const struct pmu_metrics_table *pmu_metrics_table__find(void)
+{
+        struct perf_cpu cpu = {-1};
+        const struct pmu_events_map *map = map_for_cpu(cpu);
+
+       if (map)
+               return &map->metric_table;
+
+       return pmu_metrics_table__default();
+}
+
 
Quoted 2 lines

I have no deep understanding for jevents, seems to me, Breno's issue is a different one from me. Please kindly confirm.

I think it is a different issue, they have metrics while you don't. Your report does highlight we're missing a NO_JEVENTS=1 build-test, but the build is working for me. I'll send out two patches for these issues.

Thanks,
Ian

Quoted 2 lines

Thanks,
Leo

#6 lore raw
Quoted 4 lines · Ian Rogers

I think it is a different issue, they have metrics while you don't. Your report does highlight we're missing a NO_JEVENTS=1 build-test, but the build is working for me. I'll send out two patches for these issues.

Hi Ian, Leo, Acme

I wanted to follow up on this. Are there any next steps I should take?

Thanks,
--breno

#7 lore raw

Hi Breno,

Quoted 9 lines · Breno Leitao
Quoted 4 lines · Ian Rogers

I think it is a different issue, they have metrics while you don't. Your report does highlight we're missing a NO_JEVENTS=1 build-test, but the build is working for me. I'll send out two patches for these issues.

Hi Ian, Leo, Acme

I wanted to follow up on this. Are there any next steps I should take?

Sorry for not tracking this issue.

I can reproduce the issue on my Orion6 with setting a _dummy_ CPUID:

 $ export PERF_CPUID=0x00000000410fd490
 $ perf stat -C 5 -vvv
 ...
 Aborted
 perf: util/evsel.c:2156: get_group_fd: Assertion `!(!leader->core.fd)' failed

Because we are working on different hardwares, I am a bit suspect I reproduced the issue with difference sequence as yours. Anyway, I do see that an event can be opened prior to its leader event, see the log below.

Thus, your patch seems make sense to me as we need to ensure the leader event to be opened first. Ian, how about you think?

I have a side question (maybe this is for James :): why other arch (or platforms) don't have reproduce the issue? I am wandering if any error or special stuffs in Arm CPU's json files.

Thanks,
Leo

---8<---

Opening: BR_MIS_PRED
------------------------------------------------------------
perf_event_attr:

  type                             11 (armv8_pmuv3_1)
  size                             144
  config                           0x10 (br_mis_pred)
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
  disabled                         1
  inherit                          1

------------------------------------------------------------
evsel=BR_MIS_PRED (supported=1) leader=BR_MIS_PRED
sys_perf_event_open: pid -1 cpu 5 group_fd -1 flags 0x8 sys_perf_event_open failed, error -2
Warning:
BR_MIS_PRED event is not supported by the kernel. The BR_MIS_PRED event is not supported.
Opening: OP_SPEC
------------------------------------------------------------
perf_event_attr:

  type                             11 (armv8_pmuv3_1)
  size                             144
  config                           0x3b (op_spec)
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
  inherit                          1

------------------------------------------------------------
evsel=OP_SPEC (supported=1) leader=BR_MIS_PRED leader->core.fd=0xaaab19edd950
broken group leader for OP_SPEC
Warning:
skipping event OP_SPEC that kernel failed to open.
The sys_perf_event_open() syscall failed for event (OP_SPEC): Success "dmesg | grep -i perf" may provide additional information.

Opening: STALL_SLOT
------------------------------------------------------------
perf_event_attr:

  type                             11 (armv8_pmuv3_1)
  size                             144
  config                           0x3f (stall_slot)
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
  inherit                          1

------------------------------------------------------------
evsel=STALL_SLOT (supported=1) leader=BR_MIS_PRED leader->core.fd=0xaaab19edd950
broken group leader for STALL_SLOT
Warning:
skipping event STALL_SLOT that kernel failed to open.
The sys_perf_event_open() syscall failed for event (STALL_SLOT): Success "dmesg | grep -i perf" may provide additional information.

Opening: CPU_CYCLES
------------------------------------------------------------
perf_event_attr:

  type                             11 (armv8_pmuv3_1)
  size                             144
  config                           0x11 (cpu_cycles)
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
  inherit                          1

------------------------------------------------------------
evsel=CPU_CYCLES (supported=1) leader=BR_MIS_PRED leader->core.fd=0xaaab19edd950
broken group leader for CPU_CYCLES
Warning:
skipping event CPU_CYCLES that kernel failed to open.
The sys_perf_event_open() syscall failed for event (CPU_CYCLES): Success "dmesg | grep -i perf" may provide additional information.

Opening: OP_RETIRED
------------------------------------------------------------
perf_event_attr:

  type                             11 (armv8_pmuv3_1)
  size                             144
  config                           0x3a (op_retired)
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
  inherit                          1

------------------------------------------------------------
evsel=OP_RETIRED (supported=1) leader=BR_MIS_PRED leader->core.fd=0xaaab19edd950
broken group leader for OP_RETIRED
Warning:
skipping event OP_RETIRED that kernel failed to open.
The sys_perf_event_open() syscall failed for event (OP_RETIRED): Success "dmesg | grep -i perf" may provide additional information.

Opening: BR_MIS_PRED
------------------------------------------------------------
perf_event_attr:

  type                             11 (armv8_pmuv3_1)
  size                             144
  config                           0x10 (br_mis_pred)
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
  inherit                          1

------------------------------------------------------------
evsel=BR_MIS_PRED (supported=1) leader=STALL_SLOT_BACKEND leader->core.fd=(nil)
perf: util/evsel.c:2156: get_group_fd: Assertion `!(!leader->core.fd)' failed.

#8 lore raw
Quoted 22 lines · Leo Yan
Quoted 7 lines · Breno Leitao
Quoted 4 lines · Ian Rogers

I think it is a different issue, they have metrics while you don't. Your report does highlight we're missing a NO_JEVENTS=1 build-test, but the build is working for me. I'll send out two patches for these issues.

I wanted to follow up on this. Are there any next steps I should take?

I can reproduce the issue on my Orion6 with setting a _dummy_ CPUID:

 $ export PERF_CPUID=0x00000000410fd490
 $ perf stat -C 5 -vvv
 ...
 Aborted
 perf: util/evsel.c:2156: get_group_fd: Assertion `!(!leader->core.fd)' failed

Because we are working on different hardwares, I am a bit suspect I reproduced the issue with difference sequence as yours. Anyway, I do see that an event can be opened prior to its leader event, see the log below.

Quoted 6 lines

Thus, your patch seems make sense to me as we need to ensure the leader event to be opened first. Ian, how about you think?

I have a side question (maybe this is for James :): why other arch (or platforms) don't have reproduce the issue? I am wandering if any error or special stuffs in Arm CPU's json files.

I managed to reproduce using Leo's method:

(gdb) run stat -C 1
Starting program: /root/bin/perf stat -C 1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
Failure to read '#slots'Failure to read '#slots'Failure to read '#slots'Failure to read '#slots'
Program received signal SIGSEGV, Segmentation fault.
metricgroup__copy_metric_events (evlist=0x555555c85790, cgrp=0x0, new_metric_events=0x555555c86250, old_metric_events=0x555555c86d60) at util/metricgroup.c:1662
1662 evsel = evlist__find_evsel(evlist, old_me->evsel->core.idx);
(gdb) bt

#0  metricgroup__copy_metric_events (evlist=0x555555c85790, cgrp=0x0, new_metric_events=0x555555c86250, old_metric_events=0x555555c86d60) at util/metricgroup.c:1662
#1  0x00005555555db404 in add_default_events () at builtin-stat.c:2102
#2  0x00005555555de000 in cmd_stat (argc=0, argv=0x7ffffffffc30) at builtin-stat.c:2830
#3  0x00005555556778e0 in run_builtin (p=0x555555bd5558 <commands+360>, argc=3, argv=0x7ffffffffc30) at perf.c:348
#4  0x0000555555677b5c in handle_internal_command (argc=3, argv=0x7ffffffffc30) at perf.c:398
#5  0x0000555555677d20 in run_argv (argcp=0x7ffffffffa4c, argv=0x7ffffffffa40) at perf.c:442
#6  0x0000555555678064 in main (argc=3, argv=0x7ffffffffc30) at perf.c:549

(gdb)

acme@raspberrypi:~/git/perf-tools $ git log --no-merges --oneline -15 tools/
493ad070cbcb (HEAD -> perf-tools, tag: perf-tools-fixes-for-v7.0-2-2026-03-23, x1/perf-tools, number/perf-tools) tools headers: Synchronize linux/build_bug.h with the kernel sources 0a8b2a0857ed tools headers UAPI: Sync x86's asm/kvm.h with the kernel sources
3c71ae8ec9ad tools headers UAPI: Sync linux/kvm.h with the kernel sources
4ddd7588fae6 tools arch x86: Sync the msr-index.h copy with the kernel sources
563d39928db6 perf kvm stat: Fix relative paths for including headers
72a8b9c060d3 perf parse-events: Fix big-endian 'overwrite' by writing correct union member
8dd1d9a33532 perf metricgroup: Fix metricgroup__has_metric_or_groups()
5d4c6c132ea9 selftests/hid: fix compilation when bpf_wq and hid_device are not exported
81f86728a980 tools headers: Skip arm64 cputype.h check
e2715ea5fb35 bootconfig: Add bootconfig tests about braces
d2ea4ff1ce50 KVM: selftests: Verify SEV+ guests can read and write EFER, CR0, CR4, and CR8
146c9ab38b48 powerpc/selftests/copyloops: extend selftest to exercise __copy_tofrom_user_power7_vmx
68e76fc12df0 selftests: rtnetlink: add neighbour update test
35b16a7a2c4f (tag: perf-tools-fixes-for-v7.0-1-2026-03-13, five/perf-tools) perf synthetic-events: Fix stale build ID in module MMAP2 records 3e745694b032 selftests: kvm: add a test that VMX validates controls on RSM
acme@raspberrypi:~/git/perf-tools $

Below goes the full -vvv log FWIW.

- Arnaldo

root@raspberrypi:~# export PERF_CPUID=0x00000000410fd490
root@raspberrypi:~# perf stat -C 1 -vvv
Using CPUID 0x00000000410fd490
metric expr 100 * (STALL_SLOT_BACKEND / (CPU_CYCLES * #slots) - BR_MIS_PRED * 3 / CPU_CYCLES) for backend_bound
parsing metric: 100 * (STALL_SLOT_BACKEND / (CPU_CYCLES * #slots) - BR_MIS_PRED * 3 / CPU_CYCLES)
Failure to read '#slots'literal: #slots = nan
syntax error
metric expr 100 * ((1 - OP_RETIRED / OP_SPEC) * (1 - (STALL_SLOT if strcmp_cpuid_str(0x410fd493) | strcmp_cpuid_str(0x410fd490) ^ 1 else STALL_SLOT - CPU_CYCLES) / (CPU_CYCLES * #slots)) + BR_MIS_PRED * 4 / CPU_CYCLES) for bad_speculation
parsing metric: 100 * ((1 - OP_RETIRED / OP_SPEC) * (1 - (STALL_SLOT if strcmp_cpuid_str(0x410fd493) | strcmp_cpuid_str(0x410fd490) ^ 1 else STALL_SLOT - CPU_CYCLES) / (CPU_CYCLES * #slots)) + BR_MIS_PRED * 4 / CPU_CYCLES)
Failure to read '#slots'literal: #slots = nan
syntax error
metric expr 100 * ((STALL_SLOT_FRONTEND if strcmp_cpuid_str(0x410fd493) | strcmp_cpuid_str(0x410fd490) ^ 1 else STALL_SLOT_FRONTEND - CPU_CYCLES) / (CPU_CYCLES * #slots) - BR_MIS_PRED / CPU_CYCLES) for frontend_bound
parsing metric: 100 * ((STALL_SLOT_FRONTEND if strcmp_cpuid_str(0x410fd493) | strcmp_cpuid_str(0x410fd490) ^ 1 else STALL_SLOT_FRONTEND - CPU_CYCLES) / (CPU_CYCLES * #slots) - BR_MIS_PRED / CPU_CYCLES)
Failure to read '#slots'literal: #slots = nan
syntax error
metric expr 100 * (OP_RETIRED / OP_SPEC * (1 - (STALL_SLOT if strcmp_cpuid_str(0x410fd493) | strcmp_cpuid_str(0x410fd490) ^ 1 else STALL_SLOT - CPU_CYCLES) / (CPU_CYCLES * #slots))) for retiring
parsing metric: 100 * (OP_RETIRED / OP_SPEC * (1 - (STALL_SLOT if strcmp_cpuid_str(0x410fd493) | strcmp_cpuid_str(0x410fd490) ^ 1 else STALL_SLOT - CPU_CYCLES) / (CPU_CYCLES * #slots)))
Failure to read '#slots'literal: #slots = nan
syntax error
metric expr (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) / (duration_time * 1e9) for CPUs_utilized
parsing metric: (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) / (duration_time * 1e9)
literal: #target_cpu = 1.000000
metric expr stalled\-cycles\-backend / cpu\-cycles for backend_cycles_idle
parsing metric: backend_cycles_idle
metric expr stalled\-cycles\-backend / cpu\-cycles for backend_cycles_idle
parsing metric: stalled\-cycles\-backend / cpu\-cycles
metric expr branches / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for branch_frequency
parsing metric: branches / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@)
literal: #target_cpu = 1.000000
metric expr branch\-misses / branches for branch_miss_rate
parsing metric: branch_miss_rate
metric expr branch\-misses / branches for branch_miss_rate
parsing metric: branch\-misses / branches
metric expr software@context\-switches\,name\=context\-switches@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for cs_per_second
parsing metric: software@context\-switches\,name\=context\-switches@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@)
literal: #target_cpu = 1.000000
metric expr cpu\-cycles / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for cycles_frequency
parsing metric: cpu\-cycles / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@)
literal: #target_cpu = 1.000000
metric expr stalled\-cycles\-frontend / cpu\-cycles for frontend_cycles_idle
parsing metric: frontend_cycles_idle
metric expr stalled\-cycles\-frontend / cpu\-cycles for frontend_cycles_idle
parsing metric: stalled\-cycles\-frontend / cpu\-cycles
metric expr instructions / cpu\-cycles for insn_per_cycle
parsing metric: insn_per_cycle
metric expr instructions / cpu\-cycles for insn_per_cycle
parsing metric: instructions / cpu\-cycles
metric expr software@cpu\-migrations\,name\=cpu\-migrations@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for migrations_per_second
parsing metric: software@cpu\-migrations\,name\=cpu\-migrations@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@)
literal: #target_cpu = 1.000000
metric expr software@page\-faults\,name\=page\-faults@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for page_faults_per_second
parsing metric: software@page\-faults\,name\=page\-faults@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@)
literal: #target_cpu = 1.000000
metric expr max(stalled\-cycles\-frontend, stalled\-cycles\-backend) / instructions for stalled_cycles_per_instruction
parsing metric: max(stalled\-cycles\-frontend, stalled\-cycles\-backend) / instructions
found event software@context-switches,name=context-switches@
found event duration_time
found event software@page-faults,name=page-faults@
found event cpu-cycles
found event branches
found event software@cpu-migrations,name=cpu-migrations@
found event software@cpu-clock,name=cpu-clock@
Parsing metric events 'software/context-switches,name=context-switches,metric-id=software!3context!1switches!0name!2context!1switches!3/,software/page-faults,name=page-faults,metric-id=software!3page!1faults!0name!2page!1faults!3/,cpu-cycles/metric-id=cpu!1cycles/,branches/metric-id=branches/,software/cpu-migrations,name=cpu-migrations,metric-id=software!3cpu!1migrations!0name!2cpu!1migrations!3/,software/cpu-clock,name=cpu-clock,metric-id=software!3cpu!1clock!0name!2cpu!1clock!3/,duration_time'
Attempt to add: software/context-switches,name=context-switches,metric-id=software!3context!1switches!0name!2context!1switches!3/
..after resolving event: software/config=0x3,name=context-switches,metric-id=software!3context!1switches!0name!2context!1switches!3/
Attempt to add: software/page-faults,name=page-faults,metric-id=software!3page!1faults!0name!2page!1faults!3/
..after resolving event: software/config=0x2,name=page-faults,metric-id=software!3page!1faults!0name!2page!1faults!3/
hwmon_pmu: not a hwmon file 'cpu-cycles'
Attempt to add: armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
..after resolving event: armv8_cortex_a76/metric-id=cpu!1cycles,legacy-hardware-config=0/
cpu-cycles -> armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'branches'
Attempt to add: armv8_cortex_a76/metric-id=branches,branches/
..after resolving event: armv8_cortex_a76/metric-id=branches,legacy-hardware-config=0x4/
branches -> armv8_cortex_a76/metric-id=branches,branches/
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon file 'branches'
Attempt to add: software/cpu-migrations,name=cpu-migrations,metric-id=software!3cpu!1migrations!0name!2cpu!1migrations!3/
..after resolving event: software/config=0x4,name=cpu-migrations,metric-id=software!3cpu!1migrations!0name!2cpu!1migrations!3/
Attempt to add: software/cpu-clock,name=cpu-clock,metric-id=software!3cpu!1clock!0name!2cpu!1clock!3/
..after resolving event: software/config=0,name=cpu-clock,metric-id=software!3cpu!1clock!0name!2cpu!1clock!3/
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
Attempt to add: tool/duration_time/
..after resolving event: tool/config=0x1/
duration_time -> tool/duration_time/
found event duration_time
Parsing metric events 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
Attempt to add: tool/duration_time/
..after resolving event: tool/config=0x1/
duration_time -> tool/duration_time/
Matched metric-id duration_time to duration_time
Events in 'frontend_bound' fully contained within 'retiring'
Events in 'bad_speculation' fully contained within 'retiring'
Events in 'backend_bound' fully contained within 'retiring'
found event duration_time
found event stalled-cycles-backend
found event instructions
found event stalled-cycles-frontend
Parsing metric events '{stalled-cycles-backend/metric-id=stalled!1cycles!1backend/,instructions/metric-id=instructions/,stalled-cycles-frontend/metric-id=stalled!1cycles!1frontend/}:W,duration_time'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
Attempt to add: armv8_cortex_a76/metric-id=stalled!1cycles!1backend,stalled-cycles-backend/
..after resolving event: armv8_cortex_a76/metric-id=stalled!1cycles!1backend,legacy-hardware-config=0x8/
stalled-cycles-backend -> armv8_cortex_a76/metric-id=stalled!1cycles!1backend,stalled-cycles-backend/
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'instructions'
Attempt to add: armv8_cortex_a76/metric-id=instructions,instructions/
..after resolving event: armv8_cortex_a76/metric-id=instructions,legacy-hardware-config=0x1/
instructions -> armv8_cortex_a76/metric-id=instructions,instructions/
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
Attempt to add: armv8_cortex_a76/metric-id=stalled!1cycles!1frontend,stalled-cycles-frontend/
..after resolving event: armv8_cortex_a76/metric-id=stalled!1cycles!1frontend,legacy-hardware-config=0x7/
stalled-cycles-frontend -> armv8_cortex_a76/metric-id=stalled!1cycles!1frontend,stalled-cycles-frontend/
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
Attempt to add: tool/duration_time/
..after resolving event: tool/config=0x1/
duration_time -> tool/duration_time/
Matched metric-id stalled-cycles-backend to stalled-cycles-backend
Matched metric-id instructions to instructions
Matched metric-id stalled-cycles-frontend to stalled-cycles-frontend
Matched metric-id duration_time to duration_time
Matched metric-id software@page-faults,name=page-faults@ to page-faults
Matched metric-id software@cpu-clock,name=cpu-clock@ to cpu-clock
Matched metric-id software@cpu-migrations,name=cpu-migrations@ to cpu-migrations
Matched metric-id software@cpu-clock,name=cpu-clock@ to cpu-clock
found event duration_time
found event cpu-cycles
found event instructions
Parsing metric events '{cpu-cycles/metric-id=cpu!1cycles/,instructions/metric-id=instructions/}:W,duration_time'
hwmon_pmu: not a hwmon file 'cpu-cycles'
Attempt to add: armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
..after resolving event: armv8_cortex_a76/metric-id=cpu!1cycles,legacy-hardware-config=0/
cpu-cycles -> armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'instructions'
Attempt to add: armv8_cortex_a76/metric-id=instructions,instructions/
..after resolving event: armv8_cortex_a76/metric-id=instructions,legacy-hardware-config=0x1/
instructions -> armv8_cortex_a76/metric-id=instructions,instructions/
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
Attempt to add: tool/duration_time/
..after resolving event: tool/config=0x1/
duration_time -> tool/duration_time/
Matched metric-id cpu-cycles to cpu-cycles
Matched metric-id instructions to instructions
Matched metric-id duration_time to duration_time
found event duration_time
found event cpu-cycles
found event stalled-cycles-frontend
Parsing metric events '{cpu-cycles/metric-id=cpu!1cycles/,stalled-cycles-frontend/metric-id=stalled!1cycles!1frontend/}:W,duration_time'
hwmon_pmu: not a hwmon file 'cpu-cycles'
Attempt to add: armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
..after resolving event: armv8_cortex_a76/metric-id=cpu!1cycles,legacy-hardware-config=0/
cpu-cycles -> armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
Attempt to add: armv8_cortex_a76/metric-id=stalled!1cycles!1frontend,stalled-cycles-frontend/
..after resolving event: armv8_cortex_a76/metric-id=stalled!1cycles!1frontend,legacy-hardware-config=0x7/
stalled-cycles-frontend -> armv8_cortex_a76/metric-id=stalled!1cycles!1frontend,stalled-cycles-frontend/
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
Attempt to add: tool/duration_time/
..after resolving event: tool/config=0x1/
duration_time -> tool/duration_time/
Matched metric-id cpu-cycles to cpu-cycles
Matched metric-id stalled-cycles-frontend to stalled-cycles-frontend
Matched metric-id duration_time to duration_time
Matched metric-id cpu-cycles to cpu-cycles
Matched metric-id software@cpu-clock,name=cpu-clock@ to cpu-clock
Matched metric-id software@context-switches,name=context-switches@ to context-switches
Matched metric-id software@cpu-clock,name=cpu-clock@ to cpu-clock
found event duration_time
found event branch-misses
found event branches
Parsing metric events '{branch-misses/metric-id=branch!1misses/,branches/metric-id=branches/}:W,duration_time'
hwmon_pmu: not a hwmon file 'branch-misses'
Attempt to add: armv8_cortex_a76/metric-id=branch!1misses,branch-misses/
..after resolving event: armv8_cortex_a76/metric-id=branch!1misses,legacy-hardware-config=0x5/
branch-misses -> armv8_cortex_a76/metric-id=branch!1misses,branch-misses/
hwmon_pmu: not a hwmon file 'branch-misses'
hwmon_pmu: not a hwmon file 'branch-misses'
hwmon_pmu: not a hwmon file 'branch-misses'
hwmon_pmu: not a hwmon file 'branch-misses'
hwmon_pmu: not a hwmon file 'branch-misses'
hwmon_pmu: not a hwmon file 'branches'
Attempt to add: armv8_cortex_a76/metric-id=branches,branches/
..after resolving event: armv8_cortex_a76/metric-id=branches,legacy-hardware-config=0x4/
branches -> armv8_cortex_a76/metric-id=branches,branches/
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
Attempt to add: tool/duration_time/
..after resolving event: tool/config=0x1/
duration_time -> tool/duration_time/
Matched metric-id branch-misses to branch-misses
Matched metric-id branches to branches
Matched metric-id duration_time to duration_time
Matched metric-id branches to branches
Matched metric-id software@cpu-clock,name=cpu-clock@ to cpu-clock
found event duration_time
found event cpu-cycles
found event stalled-cycles-backend
Parsing metric events '{cpu-cycles/metric-id=cpu!1cycles/,stalled-cycles-backend/metric-id=stalled!1cycles!1backend/}:W,duration_time'
hwmon_pmu: not a hwmon file 'cpu-cycles'
Attempt to add: armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
..after resolving event: armv8_cortex_a76/metric-id=cpu!1cycles,legacy-hardware-config=0/
cpu-cycles -> armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
Attempt to add: armv8_cortex_a76/metric-id=stalled!1cycles!1backend,stalled-cycles-backend/
..after resolving event: armv8_cortex_a76/metric-id=stalled!1cycles!1backend,legacy-hardware-config=0x8/
stalled-cycles-backend -> armv8_cortex_a76/metric-id=stalled!1cycles!1backend,stalled-cycles-backend/
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
Attempt to add: tool/duration_time/
..after resolving event: tool/config=0x1/
duration_time -> tool/duration_time/
Matched metric-id cpu-cycles to cpu-cycles
Matched metric-id stalled-cycles-backend to stalled-cycles-backend
Matched metric-id duration_time to duration_time
Matched metric-id software@cpu-clock,name=cpu-clock@ to cpu-clock
Matched metric-id duration_time to duration_time
Segmentation fault
root@raspberrypi:~#
#9 lore raw
Quoted 23 lines · Arnaldo Carvalho de Melo
Quoted 21 lines · Leo Yan
Quoted 7 lines · Breno Leitao
Quoted 4 lines · Ian Rogers

I think it is a different issue, they have metrics while you don't. Your report does highlight we're missing a NO_JEVENTS=1 build-test, but the build is working for me. I'll send out two patches for these issues.

I wanted to follow up on this. Are there any next steps I should take?

I can reproduce the issue on my Orion6 with setting a _dummy_ CPUID:

 $ export PERF_CPUID=0x00000000410fd490
 $ perf stat -C 5 -vvv
 ...
 Aborted
 perf: util/evsel.c:2156: get_group_fd: Assertion `!(!leader->core.fd)' failed

Because we are working on different hardwares, I am a bit suspect I reproduced the issue with difference sequence as yours. Anyway, I do see that an event can be opened prior to its leader event, see the log below.

Quoted 8 lines
Quoted 6 lines

Thus, your patch seems make sense to me as we need to ensure the leader event to be opened first. Ian, how about you think?

I have a side question (maybe this is for James :): why other arch (or platforms) don't have reproduce the issue? I am wandering if any error or special stuffs in Arm CPU's json files.

I managed to reproduce using Leo's method:

But then I applied Breno's patch and got the same SEGV as below.

- Arnaldo

Quoted 349 lines

(gdb) run stat -C 1
Starting program: /root/bin/perf stat -C 1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
Failure to read '#slots'Failure to read '#slots'Failure to read '#slots'Failure to read '#slots'
Program received signal SIGSEGV, Segmentation fault.
metricgroup__copy_metric_events (evlist=0x555555c85790, cgrp=0x0, new_metric_events=0x555555c86250, old_metric_events=0x555555c86d60) at util/metricgroup.c:1662
1662 evsel = evlist__find_evsel(evlist, old_me->evsel->core.idx);
(gdb) bt

#0  metricgroup__copy_metric_events (evlist=0x555555c85790, cgrp=0x0, new_metric_events=0x555555c86250, old_metric_events=0x555555c86d60) at util/metricgroup.c:1662
#1  0x00005555555db404 in add_default_events () at builtin-stat.c:2102
#2  0x00005555555de000 in cmd_stat (argc=0, argv=0x7ffffffffc30) at builtin-stat.c:2830
#3  0x00005555556778e0 in run_builtin (p=0x555555bd5558 <commands+360>, argc=3, argv=0x7ffffffffc30) at perf.c:348
#4  0x0000555555677b5c in handle_internal_command (argc=3, argv=0x7ffffffffc30) at perf.c:398
#5  0x0000555555677d20 in run_argv (argcp=0x7ffffffffa4c, argv=0x7ffffffffa40) at perf.c:442
#6  0x0000555555678064 in main (argc=3, argv=0x7ffffffffc30) at perf.c:549

(gdb)

acme@raspberrypi:~/git/perf-tools $ git log --no-merges --oneline -15 tools/
493ad070cbcb (HEAD -> perf-tools, tag: perf-tools-fixes-for-v7.0-2-2026-03-23, x1/perf-tools, number/perf-tools) tools headers: Synchronize linux/build_bug.h with the kernel sources 0a8b2a0857ed tools headers UAPI: Sync x86's asm/kvm.h with the kernel sources
3c71ae8ec9ad tools headers UAPI: Sync linux/kvm.h with the kernel sources
4ddd7588fae6 tools arch x86: Sync the msr-index.h copy with the kernel sources
563d39928db6 perf kvm stat: Fix relative paths for including headers
72a8b9c060d3 perf parse-events: Fix big-endian 'overwrite' by writing correct union member
8dd1d9a33532 perf metricgroup: Fix metricgroup__has_metric_or_groups()
5d4c6c132ea9 selftests/hid: fix compilation when bpf_wq and hid_device are not exported
81f86728a980 tools headers: Skip arm64 cputype.h check
e2715ea5fb35 bootconfig: Add bootconfig tests about braces
d2ea4ff1ce50 KVM: selftests: Verify SEV+ guests can read and write EFER, CR0, CR4, and CR8
146c9ab38b48 powerpc/selftests/copyloops: extend selftest to exercise __copy_tofrom_user_power7_vmx
68e76fc12df0 selftests: rtnetlink: add neighbour update test
35b16a7a2c4f (tag: perf-tools-fixes-for-v7.0-1-2026-03-13, five/perf-tools) perf synthetic-events: Fix stale build ID in module MMAP2 records 3e745694b032 selftests: kvm: add a test that VMX validates controls on RSM
acme@raspberrypi:~/git/perf-tools $

Below goes the full -vvv log FWIW.

- Arnaldo

root@raspberrypi:~# export PERF_CPUID=0x00000000410fd490
root@raspberrypi:~# perf stat -C 1 -vvv
Using CPUID 0x00000000410fd490
metric expr 100 * (STALL_SLOT_BACKEND / (CPU_CYCLES * #slots) - BR_MIS_PRED * 3 / CPU_CYCLES) for backend_bound
parsing metric: 100 * (STALL_SLOT_BACKEND / (CPU_CYCLES * #slots) - BR_MIS_PRED * 3 / CPU_CYCLES)
Failure to read '#slots'literal: #slots = nan
syntax error
metric expr 100 * ((1 - OP_RETIRED / OP_SPEC) * (1 - (STALL_SLOT if strcmp_cpuid_str(0x410fd493) | strcmp_cpuid_str(0x410fd490) ^ 1 else STALL_SLOT - CPU_CYCLES) / (CPU_CYCLES * #slots)) + BR_MIS_PRED * 4 / CPU_CYCLES) for bad_speculation
parsing metric: 100 * ((1 - OP_RETIRED / OP_SPEC) * (1 - (STALL_SLOT if strcmp_cpuid_str(0x410fd493) | strcmp_cpuid_str(0x410fd490) ^ 1 else STALL_SLOT - CPU_CYCLES) / (CPU_CYCLES * #slots)) + BR_MIS_PRED * 4 / CPU_CYCLES)
Failure to read '#slots'literal: #slots = nan
syntax error
metric expr 100 * ((STALL_SLOT_FRONTEND if strcmp_cpuid_str(0x410fd493) | strcmp_cpuid_str(0x410fd490) ^ 1 else STALL_SLOT_FRONTEND - CPU_CYCLES) / (CPU_CYCLES * #slots) - BR_MIS_PRED / CPU_CYCLES) for frontend_bound
parsing metric: 100 * ((STALL_SLOT_FRONTEND if strcmp_cpuid_str(0x410fd493) | strcmp_cpuid_str(0x410fd490) ^ 1 else STALL_SLOT_FRONTEND - CPU_CYCLES) / (CPU_CYCLES * #slots) - BR_MIS_PRED / CPU_CYCLES)
Failure to read '#slots'literal: #slots = nan
syntax error
metric expr 100 * (OP_RETIRED / OP_SPEC * (1 - (STALL_SLOT if strcmp_cpuid_str(0x410fd493) | strcmp_cpuid_str(0x410fd490) ^ 1 else STALL_SLOT - CPU_CYCLES) / (CPU_CYCLES * #slots))) for retiring
parsing metric: 100 * (OP_RETIRED / OP_SPEC * (1 - (STALL_SLOT if strcmp_cpuid_str(0x410fd493) | strcmp_cpuid_str(0x410fd490) ^ 1 else STALL_SLOT - CPU_CYCLES) / (CPU_CYCLES * #slots)))
Failure to read '#slots'literal: #slots = nan
syntax error
metric expr (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) / (duration_time * 1e9) for CPUs_utilized
parsing metric: (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) / (duration_time * 1e9)
literal: #target_cpu = 1.000000
metric expr stalled\-cycles\-backend / cpu\-cycles for backend_cycles_idle
parsing metric: backend_cycles_idle
metric expr stalled\-cycles\-backend / cpu\-cycles for backend_cycles_idle
parsing metric: stalled\-cycles\-backend / cpu\-cycles
metric expr branches / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for branch_frequency
parsing metric: branches / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@)
literal: #target_cpu = 1.000000
metric expr branch\-misses / branches for branch_miss_rate
parsing metric: branch_miss_rate
metric expr branch\-misses / branches for branch_miss_rate
parsing metric: branch\-misses / branches
metric expr software@context\-switches\,name\=context\-switches@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for cs_per_second
parsing metric: software@context\-switches\,name\=context\-switches@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@)
literal: #target_cpu = 1.000000
metric expr cpu\-cycles / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for cycles_frequency
parsing metric: cpu\-cycles / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@)
literal: #target_cpu = 1.000000
metric expr stalled\-cycles\-frontend / cpu\-cycles for frontend_cycles_idle
parsing metric: frontend_cycles_idle
metric expr stalled\-cycles\-frontend / cpu\-cycles for frontend_cycles_idle
parsing metric: stalled\-cycles\-frontend / cpu\-cycles
metric expr instructions / cpu\-cycles for insn_per_cycle
parsing metric: insn_per_cycle
metric expr instructions / cpu\-cycles for insn_per_cycle
parsing metric: instructions / cpu\-cycles
metric expr software@cpu\-migrations\,name\=cpu\-migrations@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for migrations_per_second
parsing metric: software@cpu\-migrations\,name\=cpu\-migrations@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@)
literal: #target_cpu = 1.000000
metric expr software@page\-faults\,name\=page\-faults@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for page_faults_per_second
parsing metric: software@page\-faults\,name\=page\-faults@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@)
literal: #target_cpu = 1.000000
metric expr max(stalled\-cycles\-frontend, stalled\-cycles\-backend) / instructions for stalled_cycles_per_instruction
parsing metric: max(stalled\-cycles\-frontend, stalled\-cycles\-backend) / instructions
found event software@context-switches,name=context-switches@
found event duration_time
found event software@page-faults,name=page-faults@
found event cpu-cycles
found event branches
found event software@cpu-migrations,name=cpu-migrations@
found event software@cpu-clock,name=cpu-clock@
Parsing metric events 'software/context-switches,name=context-switches,metric-id=software!3context!1switches!0name!2context!1switches!3/,software/page-faults,name=page-faults,metric-id=software!3page!1faults!0name!2page!1faults!3/,cpu-cycles/metric-id=cpu!1cycles/,branches/metric-id=branches/,software/cpu-migrations,name=cpu-migrations,metric-id=software!3cpu!1migrations!0name!2cpu!1migrations!3/,software/cpu-clock,name=cpu-clock,metric-id=software!3cpu!1clock!0name!2cpu!1clock!3/,duration_time'
Attempt to add: software/context-switches,name=context-switches,metric-id=software!3context!1switches!0name!2context!1switches!3/
..after resolving event: software/config=0x3,name=context-switches,metric-id=software!3context!1switches!0name!2context!1switches!3/
Attempt to add: software/page-faults,name=page-faults,metric-id=software!3page!1faults!0name!2page!1faults!3/
..after resolving event: software/config=0x2,name=page-faults,metric-id=software!3page!1faults!0name!2page!1faults!3/
hwmon_pmu: not a hwmon file 'cpu-cycles'
Attempt to add: armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
..after resolving event: armv8_cortex_a76/metric-id=cpu!1cycles,legacy-hardware-config=0/
cpu-cycles -> armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'branches'
Attempt to add: armv8_cortex_a76/metric-id=branches,branches/
..after resolving event: armv8_cortex_a76/metric-id=branches,legacy-hardware-config=0x4/
branches -> armv8_cortex_a76/metric-id=branches,branches/
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon file 'branches'
Attempt to add: software/cpu-migrations,name=cpu-migrations,metric-id=software!3cpu!1migrations!0name!2cpu!1migrations!3/
..after resolving event: software/config=0x4,name=cpu-migrations,metric-id=software!3cpu!1migrations!0name!2cpu!1migrations!3/
Attempt to add: software/cpu-clock,name=cpu-clock,metric-id=software!3cpu!1clock!0name!2cpu!1clock!3/
..after resolving event: software/config=0,name=cpu-clock,metric-id=software!3cpu!1clock!0name!2cpu!1clock!3/
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
Attempt to add: tool/duration_time/
..after resolving event: tool/config=0x1/
duration_time -> tool/duration_time/
found event duration_time
Parsing metric events 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
Attempt to add: tool/duration_time/
..after resolving event: tool/config=0x1/
duration_time -> tool/duration_time/
Matched metric-id duration_time to duration_time
Events in 'frontend_bound' fully contained within 'retiring'
Events in 'bad_speculation' fully contained within 'retiring'
Events in 'backend_bound' fully contained within 'retiring'
found event duration_time
found event stalled-cycles-backend
found event instructions
found event stalled-cycles-frontend
Parsing metric events '{stalled-cycles-backend/metric-id=stalled!1cycles!1backend/,instructions/metric-id=instructions/,stalled-cycles-frontend/metric-id=stalled!1cycles!1frontend/}:W,duration_time'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
Attempt to add: armv8_cortex_a76/metric-id=stalled!1cycles!1backend,stalled-cycles-backend/
..after resolving event: armv8_cortex_a76/metric-id=stalled!1cycles!1backend,legacy-hardware-config=0x8/
stalled-cycles-backend -> armv8_cortex_a76/metric-id=stalled!1cycles!1backend,stalled-cycles-backend/
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'instructions'
Attempt to add: armv8_cortex_a76/metric-id=instructions,instructions/
..after resolving event: armv8_cortex_a76/metric-id=instructions,legacy-hardware-config=0x1/
instructions -> armv8_cortex_a76/metric-id=instructions,instructions/
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
Attempt to add: armv8_cortex_a76/metric-id=stalled!1cycles!1frontend,stalled-cycles-frontend/
..after resolving event: armv8_cortex_a76/metric-id=stalled!1cycles!1frontend,legacy-hardware-config=0x7/
stalled-cycles-frontend -> armv8_cortex_a76/metric-id=stalled!1cycles!1frontend,stalled-cycles-frontend/
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
Attempt to add: tool/duration_time/
..after resolving event: tool/config=0x1/
duration_time -> tool/duration_time/
Matched metric-id stalled-cycles-backend to stalled-cycles-backend
Matched metric-id instructions to instructions
Matched metric-id stalled-cycles-frontend to stalled-cycles-frontend
Matched metric-id duration_time to duration_time
Matched metric-id software@page-faults,name=page-faults@ to page-faults
Matched metric-id software@cpu-clock,name=cpu-clock@ to cpu-clock
Matched metric-id software@cpu-migrations,name=cpu-migrations@ to cpu-migrations
Matched metric-id software@cpu-clock,name=cpu-clock@ to cpu-clock
found event duration_time
found event cpu-cycles
found event instructions
Parsing metric events '{cpu-cycles/metric-id=cpu!1cycles/,instructions/metric-id=instructions/}:W,duration_time'
hwmon_pmu: not a hwmon file 'cpu-cycles'
Attempt to add: armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
..after resolving event: armv8_cortex_a76/metric-id=cpu!1cycles,legacy-hardware-config=0/
cpu-cycles -> armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'instructions'
Attempt to add: armv8_cortex_a76/metric-id=instructions,instructions/
..after resolving event: armv8_cortex_a76/metric-id=instructions,legacy-hardware-config=0x1/
instructions -> armv8_cortex_a76/metric-id=instructions,instructions/
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon file 'instructions'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
Attempt to add: tool/duration_time/
..after resolving event: tool/config=0x1/
duration_time -> tool/duration_time/
Matched metric-id cpu-cycles to cpu-cycles
Matched metric-id instructions to instructions
Matched metric-id duration_time to duration_time
found event duration_time
found event cpu-cycles
found event stalled-cycles-frontend
Parsing metric events '{cpu-cycles/metric-id=cpu!1cycles/,stalled-cycles-frontend/metric-id=stalled!1cycles!1frontend/}:W,duration_time'
hwmon_pmu: not a hwmon file 'cpu-cycles'
Attempt to add: armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
..after resolving event: armv8_cortex_a76/metric-id=cpu!1cycles,legacy-hardware-config=0/
cpu-cycles -> armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
Attempt to add: armv8_cortex_a76/metric-id=stalled!1cycles!1frontend,stalled-cycles-frontend/
..after resolving event: armv8_cortex_a76/metric-id=stalled!1cycles!1frontend,legacy-hardware-config=0x7/
stalled-cycles-frontend -> armv8_cortex_a76/metric-id=stalled!1cycles!1frontend,stalled-cycles-frontend/
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon file 'stalled-cycles-frontend'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
Attempt to add: tool/duration_time/
..after resolving event: tool/config=0x1/
duration_time -> tool/duration_time/
Matched metric-id cpu-cycles to cpu-cycles
Matched metric-id stalled-cycles-frontend to stalled-cycles-frontend
Matched metric-id duration_time to duration_time
Matched metric-id cpu-cycles to cpu-cycles
Matched metric-id software@cpu-clock,name=cpu-clock@ to cpu-clock
Matched metric-id software@context-switches,name=context-switches@ to context-switches
Matched metric-id software@cpu-clock,name=cpu-clock@ to cpu-clock
found event duration_time
found event branch-misses
found event branches
Parsing metric events '{branch-misses/metric-id=branch!1misses/,branches/metric-id=branches/}:W,duration_time'
hwmon_pmu: not a hwmon file 'branch-misses'
Attempt to add: armv8_cortex_a76/metric-id=branch!1misses,branch-misses/
..after resolving event: armv8_cortex_a76/metric-id=branch!1misses,legacy-hardware-config=0x5/
branch-misses -> armv8_cortex_a76/metric-id=branch!1misses,branch-misses/
hwmon_pmu: not a hwmon file 'branch-misses'
hwmon_pmu: not a hwmon file 'branch-misses'
hwmon_pmu: not a hwmon file 'branch-misses'
hwmon_pmu: not a hwmon file 'branch-misses'
hwmon_pmu: not a hwmon file 'branch-misses'
hwmon_pmu: not a hwmon file 'branches'
Attempt to add: armv8_cortex_a76/metric-id=branches,branches/
..after resolving event: armv8_cortex_a76/metric-id=branches,legacy-hardware-config=0x4/
branches -> armv8_cortex_a76/metric-id=branches,branches/
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon file 'branches'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
Attempt to add: tool/duration_time/
..after resolving event: tool/config=0x1/
duration_time -> tool/duration_time/
Matched metric-id branch-misses to branch-misses
Matched metric-id branches to branches
Matched metric-id duration_time to duration_time
Matched metric-id branches to branches
Matched metric-id software@cpu-clock,name=cpu-clock@ to cpu-clock
found event duration_time
found event cpu-cycles
found event stalled-cycles-backend
Parsing metric events '{cpu-cycles/metric-id=cpu!1cycles/,stalled-cycles-backend/metric-id=stalled!1cycles!1backend/}:W,duration_time'
hwmon_pmu: not a hwmon file 'cpu-cycles'
Attempt to add: armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
..after resolving event: armv8_cortex_a76/metric-id=cpu!1cycles,legacy-hardware-config=0/
cpu-cycles -> armv8_cortex_a76/metric-id=cpu!1cycles,cpu-cycles/
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'cpu-cycles'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
Attempt to add: armv8_cortex_a76/metric-id=stalled!1cycles!1backend,stalled-cycles-backend/
..after resolving event: armv8_cortex_a76/metric-id=stalled!1cycles!1backend,legacy-hardware-config=0x8/
stalled-cycles-backend -> armv8_cortex_a76/metric-id=stalled!1cycles!1backend,stalled-cycles-backend/
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon file 'stalled-cycles-backend'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
hwmon_pmu: not a hwmon type 'duration' in file name 'duration_time'
Attempt to add: tool/duration_time/
..after resolving event: tool/config=0x1/
duration_time -> tool/duration_time/
Matched metric-id cpu-cycles to cpu-cycles
Matched metric-id stalled-cycles-backend to stalled-cycles-backend
Matched metric-id duration_time to duration_time
Matched metric-id software@cpu-clock,name=cpu-clock@ to cpu-clock
Matched metric-id duration_time to duration_time
Segmentation fault
root@raspberrypi:~#
#10 lore raw

Hi Arnaldo,

On Mon, Mar 23, 2026 at 11:36:22AM -0300, Arnaldo Carvalho de Melo wrote:

[...]

Quoted 3 lines
Quoted 1 line

I managed to reproduce using Leo's method:

But then I applied Breno's patch and got the same SEGV as below.

Are you sure ?

When my replied previously, I tried Breno and confirmed the Segmentation fault can be dismissed. Do you want me to test again?

Thanks,
Leo

reviewer
#11 lore raw
In reply to Leo Yan
Quoted 24 lines · Leo Yan

Hi Breno,

Quoted 9 lines · Breno Leitao
Quoted 4 lines · Ian Rogers

I think it is a different issue, they have metrics while you don't. Your report does highlight we're missing a NO_JEVENTS=1 build-test, but the build is working for me. I'll send out two patches for these issues.

Hi Ian, Leo, Acme

I wanted to follow up on this. Are there any next steps I should take?

Sorry for not tracking this issue.

I can reproduce the issue on my Orion6 with setting a _dummy_ CPUID:

 $ export PERF_CPUID=0x00000000410fd490
 $ perf stat -C 5 -vvv
 ...
 Aborted
 perf: util/evsel.c:2156: get_group_fd: Assertion `!(!leader->core.fd)' failed
Quoted 7 lines

Because we are working on different hardwares, I am a bit suspect I reproduced the issue with difference sequence as yours. Anyway, I do see that an event can be opened prior to its leader event, see the log below.

Thus, your patch seems make sense to me as we need to ensure the leader event to be opened first. Ian, how about you think?

Because so many things depend on the event ordering, the patch makes me very nervous, particularly how it will change architectural requirement handling. Ordering the Intel slots and metric events is a challenge. There is also how handling uncore events, which are deliberately parsed out-of-order, will change. I've got a feeling the test coverage for this isn't adequate, and finding the bugs requires running on particular machines. It will also require examining the default perf stat output to ensure this isn't broken; hopefully the code is somewhat robust.

Ideally, the impact of the change on all these issues would be in the commit message, but it's more realistic to cover each issue with testing. I'll try to ask an AI buddy to help with this. Could you rebase and send a v2? I'm curious to see what sashiko will throw up.

Thanks,
Ian

Quoted 112 lines

I have a side question (maybe this is for James :): why other arch (or platforms) don't have reproduce the issue? I am wandering if any error or special stuffs in Arm CPU's json files.

Thanks,
Leo

---8<---

Opening: BR_MIS_PRED
------------------------------------------------------------
perf_event_attr:

  type                             11 (armv8_pmuv3_1)
  size                             144
  config                           0x10 (br_mis_pred)
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
  disabled                         1
  inherit                          1

------------------------------------------------------------
evsel=BR_MIS_PRED (supported=1) leader=BR_MIS_PRED
sys_perf_event_open: pid -1 cpu 5 group_fd -1 flags 0x8 sys_perf_event_open failed, error -2
Warning:
BR_MIS_PRED event is not supported by the kernel. The BR_MIS_PRED event is not supported.
Opening: OP_SPEC
------------------------------------------------------------
perf_event_attr:

  type                             11 (armv8_pmuv3_1)
  size                             144
  config                           0x3b (op_spec)
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
  inherit                          1

------------------------------------------------------------
evsel=OP_SPEC (supported=1) leader=BR_MIS_PRED leader->core.fd=0xaaab19edd950
broken group leader for OP_SPEC
Warning:
skipping event OP_SPEC that kernel failed to open.
The sys_perf_event_open() syscall failed for event (OP_SPEC): Success "dmesg | grep -i perf" may provide additional information.

Opening: STALL_SLOT
------------------------------------------------------------
perf_event_attr:

  type                             11 (armv8_pmuv3_1)
  size                             144
  config                           0x3f (stall_slot)
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
  inherit                          1

------------------------------------------------------------
evsel=STALL_SLOT (supported=1) leader=BR_MIS_PRED leader->core.fd=0xaaab19edd950
broken group leader for STALL_SLOT
Warning:
skipping event STALL_SLOT that kernel failed to open.
The sys_perf_event_open() syscall failed for event (STALL_SLOT): Success "dmesg | grep -i perf" may provide additional information.

Opening: CPU_CYCLES
------------------------------------------------------------
perf_event_attr:

  type                             11 (armv8_pmuv3_1)
  size                             144
  config                           0x11 (cpu_cycles)
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
  inherit                          1

------------------------------------------------------------
evsel=CPU_CYCLES (supported=1) leader=BR_MIS_PRED leader->core.fd=0xaaab19edd950
broken group leader for CPU_CYCLES
Warning:
skipping event CPU_CYCLES that kernel failed to open.
The sys_perf_event_open() syscall failed for event (CPU_CYCLES): Success "dmesg | grep -i perf" may provide additional information.

Opening: OP_RETIRED
------------------------------------------------------------
perf_event_attr:

  type                             11 (armv8_pmuv3_1)
  size                             144
  config                           0x3a (op_retired)
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
  inherit                          1

------------------------------------------------------------
evsel=OP_RETIRED (supported=1) leader=BR_MIS_PRED leader->core.fd=0xaaab19edd950
broken group leader for OP_RETIRED
Warning:
skipping event OP_RETIRED that kernel failed to open.
The sys_perf_event_open() syscall failed for event (OP_RETIRED): Success "dmesg | grep -i perf" may provide additional information.

Opening: BR_MIS_PRED
------------------------------------------------------------
perf_event_attr:

  type                             11 (armv8_pmuv3_1)
  size                             144
  config                           0x10 (br_mis_pred)
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
  inherit                          1

------------------------------------------------------------
evsel=BR_MIS_PRED (supported=1) leader=STALL_SLOT_BACKEND leader->core.fd=(nil)
perf: util/evsel.c:2156: get_group_fd: Assertion `!(!leader->core.fd)' failed.

#12 lore raw
Quoted 33 lines · Ian Rogers
Quoted 24 lines · Leo Yan

Hi Breno,

Quoted 9 lines · Breno Leitao
Quoted 4 lines · Ian Rogers

I think it is a different issue, they have metrics while you don't. Your report does highlight we're missing a NO_JEVENTS=1 build-test, but the build is working for me. I'll send out two patches for these issues.

Hi Ian, Leo, Acme

I wanted to follow up on this. Are there any next steps I should take?

Sorry for not tracking this issue.

I can reproduce the issue on my Orion6 with setting a _dummy_ CPUID:

 $ export PERF_CPUID=0x00000000410fd490
 $ perf stat -C 5 -vvv
 ...
 Aborted
 perf: util/evsel.c:2156: get_group_fd: Assertion `!(!leader->core.fd)' failed
Quoted 7 lines

Because we are working on different hardwares, I am a bit suspect I reproduced the issue with difference sequence as yours. Anyway, I do see that an event can be opened prior to its leader event, see the log below.

Thus, your patch seems make sense to me as we need to ensure the leader event to be opened first. Ian, how about you think?

Quoted 9 lines

Because so many things depend on the event ordering, the patch makes me very nervous, particularly how it will change architectural requirement handling. Ordering the Intel slots and metric events is a challenge. There is also how handling uncore events, which are deliberately parsed out-of-order, will change. I've got a feeling the test coverage for this isn't adequate, and finding the bugs requires running on particular machines. It will also require examining the default perf stat output to ensure this isn't broken; hopefully the code is somewhat robust.

Sure thing. please let me know if there is any action on my side, and I am happy to help.

Quoted 4 lines

Ideally, the impact of the change on all these issues would be in the commit message, but it's more realistic to cover each issue with testing. I'll try to ask an AI buddy to help with this. Could you rebase and send a v2? I'm curious to see what sashiko will throw up.

Ack. I will resping it, so, we can check what sashiko thinks about it.

Thanks,
--breno

#13 lore raw
In reply to Ian Rogers

Hello Ian, thanks for the quick reply!

Quoted 9 lines · Ian Rogers
Quoted 8 lines · Breno Leitao

Perf stat is crashing on arm64 hosts with the following issue:

        # make -C tools/perf DEBUG=1
        # perf stat sleep 1
        perf: util/evsel.c:2034: get_group_fd: Assertion `!(!leader->core.fd)' failed.
        [1]    1220794 IOT instruction (core dumped)  ./perf stat
Quoted 8 lines
Quoted 5 lines

The sorting function introduced by commit a745c0831c15c ("perf stat:
Sort default events/metrics") compares events based on their individual properties. This can cause events from different groups to be interleaved, resulting in group members appearing before their leaders in the sorted evlist.

Hi, sorry for the issue. I can see what you're saying but why is this an arm64 issue?

Sorry, It's not ARM64-specific - the bug is in the generic sort code. It just happens to manifest on ARM64.

Quoted 4 lines

The legacy Default metrics are common to all
architectures:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/pmu-events/arch/common/common/metrics.json?h=perf-tools-next

Quoted 8 lines
Quoted 6 lines

When the iterator opens events in list order, a group member may be processed before its leader has been opened.

For example, CPU_CYCLES (idx=32) with leader STALL_SLOT_BACKEND (idx=37) could be sorted before its leader, causing the crash when CPU_CYCLES tries to get its group fd from the not-yet-opened leader.

Which metric is this?

These are ARM neoverse metrics, they can be found in tools/perf/pmu-events/arch/arm64/arm/neoverse-n*

Quoted 10 lines
Quoted 5 lines

Fix this by comparing events based on their leader's attributes instead of their own attributes when the events are in different groups. This ensures all members of a group share the same sort key as their leader, keeping groups together and guaranteeing leaders are opened before their members.

This makes sense but I'm not understanding why this problem wasn't seen previously. I'm guessing that in a metric like backend_cycles_idle:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/pmu-events/arch/common/common/metrics.json?h=perf-tools-next#n63

Quoted 8 lines
```
        "BriefDescription": "Backend stalls per cycle",
        "MetricExpr": "stalled\\-cycles\\-backend / cpu\\-cycles",
        "MetricGroup": "Default",
        "MetricName": "backend_cycles_idle",
        "MetricThreshold": "backend_cycles_idle > 0.2",
        "DefaultShowEvents": "1"
```

I was able to limit this to the following json:

  [
      {
          "ArchStdEvent": "backend_bound",
          "MetricExpr": "(100 * ((STALL_SLOT_BACKEND / (CPU_CYCLES * #slots)) - ((BR_MIS_PRED * 3) / CPU_CYCLES)))"
      },
      {
          "ArchStdEvent": "frontend_bound",
          "MetricExpr": "(100 * (((STALL_SLOT_FRONTEND) / (CPU_CYCLES * #slots)) - (BR_MIS_PRED / CPU_CYCLES)))"
      }
  ]

and then

	# ./tools/perf/perf stat -v sleep 0.01
	Using CPUID 0x00000000410fd4f0
	metric expr 100 * (STALL_SLOT_BACKEND / (CPU_CYCLES * #slots) - BR_MIS_PRED * 3 / CPU_CYCLES) for backend_bound
	metric expr 100 * (STALL_SLOT_FRONTEND / (CPU_CYCLES * #slots) - BR_MIS_PRED / CPU_CYCLES) for frontend_bound
	metric expr (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) / (duration_time * 1e9) for CPUs_utilized
	metric expr stalled\-cycles\-backend / cpu\-cycles for backend_cycles_idle
	metric expr stalled\-cycles\-backend / cpu\-cycles for backend_cycles_idle
	metric expr branches / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for branch_frequency
	metric expr branch\-misses / branches for branch_miss_rate
	metric expr branch\-misses / branches for branch_miss_rate
	metric expr software@context\-switches\,name\=context\-switches@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for cs_per_second
	metric expr cpu\-cycles / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for cycles_frequency
	metric expr stalled\-cycles\-frontend / cpu\-cycles for frontend_cycles_idle
	metric expr stalled\-cycles\-frontend / cpu\-cycles for frontend_cycles_idle
	metric expr instructions / cpu\-cycles for insn_per_cycle
	metric expr instructions / cpu\-cycles for insn_per_cycle
	metric expr software@cpu\-migrations\,name\=cpu\-migrations@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for migrations_per_second
	metric expr software@page\-faults\,name\=page\-faults@ * 1e9 / (software@cpu\-clock\,name\=cpu\-clock@ if #target_cpu else software@task\-clock\,name\=task\-clock@) for page_faults_per_second
	metric expr max(stalled\-cycles\-frontend, stalled\-cycles\-backend) / instructions for stalled_cycles_per_instruction
	hwmon_pmu: failure to open '/sys/class/hwmon/hwmon4/name'
	hwmon_pmu: failure to open '/sys/class/hwmon/hwmon5/name'
	hwmon_pmu: failure to open '/sys/class/hwmon/hwmon3/name'
	found event software@context-switches,name=context-switches@
	found event duration_time
	found event software@page-faults,name=page-faults@
	found event software@task-clock,name=task-clock@
	found event cpu-cycles
	found event branches
	found event software@cpu-migrations,name=cpu-migrations@
	Parsing metric events 'software/context-switches,name=context-switches,metric-id=software!3context!1switches!0name!2context!1switches!3/,software/page-faults,name=page-faults,metric-id=software!3page!1faults!0name!2page!1faults!3/,software/task-clock,name=task-clock,metric-id=software!3task!1clock!0name!2task!1clock!3/,cpu-cycles/metric-id=cpu!1cycles/,branches/metric-id=branches/,software/cpu-migrations,name=cpu-migrations,metric-id=software!3cpu!1migrations!0name!2cpu!1migrations!3/,duration_time'
	cpu-cycles -> armv8_pmuv3_0/metric-id=cpu!1cycles,cpu-cycles/
	branches -> armv8_pmuv3_0/metric-id=branches,branches/
	duration_time -> tool/duration_time/
	found event STALL_SLOT_FRONTEND
	found event duration_time
	found event BR_MIS_PRED
	found event CPU_CYCLES
	Parsing metric events '{STALL_SLOT_FRONTEND/metric-id=STALL_SLOT_FRONTEND/,BR_MIS_PRED/metric-id=BR_MIS_PRED/,CPU_CYCLES/metric-id=CPU_CYCLES/}:W,duration_time'
	STALL_SLOT_FRONTEND -> armv8_pmuv3_0/metric-id=STALL_SLOT_FRONTEND,STALL_SLOT_FRONTEND/
	BR_MIS_PRED -> armv8_pmuv3_0/metric-id=BR_MIS_PRED,BR_MIS_PRED/
	CPU_CYCLES -> armv8_pmuv3_0/metric-id=CPU_CYCLES,CPU_CYCLES/
	duration_time -> tool/duration_time/
	Matched metric-id STALL_SLOT_FRONTEND to STALL_SLOT_FRONTEND
	Matched metric-id BR_MIS_PRED to BR_MIS_PRED
	Matched metric-id CPU_CYCLES to CPU_CYCLES
	Matched metric-id duration_time to duration_time
	found event STALL_SLOT_BACKEND
	found event duration_time
	found event BR_MIS_PRED
	found event CPU_CYCLES
	Parsing metric events '{STALL_SLOT_BACKEND/metric-id=STALL_SLOT_BACKEND/,BR_MIS_PRED/metric-id=BR_MIS_PRED/,CPU_CYCLES/metric-id=CPU_CYCLES/}:W,duration_time'
	STALL_SLOT_BACKEND -> armv8_pmuv3_0/metric-id=STALL_SLOT_BACKEND,STALL_SLOT_BACKEND/
	BR_MIS_PRED -> armv8_pmuv3_0/metric-id=BR_MIS_PRED,BR_MIS_PRED/
	CPU_CYCLES -> armv8_pmuv3_0/metric-id=CPU_CYCLES,CPU_CYCLES/
	duration_time -> tool/duration_time/
	Matched metric-id STALL_SLOT_BACKEND to STALL_SLOT_BACKEND
	Matched metric-id BR_MIS_PRED to BR_MIS_PRED
	Matched metric-id CPU_CYCLES to CPU_CYCLES
	Matched metric-id duration_time to duration_time
	found event duration_time
	found event stalled-cycles-backend
	found event instructions
	found event stalled-cycles-frontend
	Parsing metric events '{stalled-cycles-backend/metric-id=stalled!1cycles!1backend/,instructions/metric-id=instructions/,stalled-cycles-frontend/metric-id=stalled!1cycles!1frontend/}:W,duration_time'
	stalled-cycles-backend -> armv8_pmuv3_0/metric-id=stalled!1cycles!1backend,stalled-cycles-backend/
	instructions -> armv8_pmuv3_0/metric-id=instructions,instructions/
	stalled-cycles-frontend -> armv8_pmuv3_0/metric-id=stalled!1cycles!1frontend,stalled-cycles-frontend/
	duration_time -> tool/duration_time/
	Matched metric-id stalled-cycles-backend to stalled-cycles-backend
	Matched metric-id instructions to instructions
	Matched metric-id stalled-cycles-frontend to stalled-cycles-frontend
	Matched metric-id duration_time to duration_time
	Matched metric-id software@page-faults,name=page-faults@ to page-faults
	Matched metric-id software@task-clock,name=task-clock@ to task-clock
	Matched metric-id software@task-clock,name=task-clock@ to task-clock
	Matched metric-id software@cpu-migrations,name=cpu-migrations@ to cpu-migrations
	found event duration_time
	found event cpu-cycles
	found event instructions
	Parsing metric events '{cpu-cycles/metric-id=cpu!1cycles/,instructions/metric-id=instructions/}:W,duration_time'
	cpu-cycles -> armv8_pmuv3_0/metric-id=cpu!1cycles,cpu-cycles/
	instructions -> armv8_pmuv3_0/metric-id=instructions,instructions/
	duration_time -> tool/duration_time/
	Matched metric-id cpu-cycles to cpu-cycles
	Matched metric-id instructions to instructions
	Matched metric-id duration_time to duration_time
	found event duration_time
	found event cpu-cycles
	found event stalled-cycles-frontend
	Parsing metric events '{cpu-cycles/metric-id=cpu!1cycles/,stalled-cycles-frontend/metric-id=stalled!1cycles!1frontend/}:W,duration_time'
	cpu-cycles -> armv8_pmuv3_0/metric-id=cpu!1cycles,cpu-cycles/
	stalled-cycles-frontend -> armv8_pmuv3_0/metric-id=stalled!1cycles!1frontend,stalled-cycles-frontend/
	duration_time -> tool/duration_time/
	Matched metric-id cpu-cycles to cpu-cycles
	Matched metric-id stalled-cycles-frontend to stalled-cycles-frontend
	Matched metric-id duration_time to duration_time
	Matched metric-id software@task-clock,name=task-clock@ to task-clock
	Matched metric-id cpu-cycles to cpu-cycles
	Matched metric-id software@context-switches,name=context-switches@ to context-switches
	Matched metric-id software@task-clock,name=task-clock@ to task-clock
	found event duration_time
	found event branch-misses
	found event branches
	Parsing metric events '{branch-misses/metric-id=branch!1misses/,branches/metric-id=branches/}:W,duration_time'
	branch-misses -> armv8_pmuv3_0/metric-id=branch!1misses,branch-misses/
	branches -> armv8_pmuv3_0/metric-id=branches,branches/
	duration_time -> tool/duration_time/
	Matched metric-id branch-misses to branch-misses
	Matched metric-id branches to branches
	Matched metric-id duration_time to duration_time
	Matched metric-id software@task-clock,name=task-clock@ to task-clock
	Matched metric-id branches to branches
	found event duration_time
	found event cpu-cycles
	found event stalled-cycles-backend
	Parsing metric events '{cpu-cycles/metric-id=cpu!1cycles/,stalled-cycles-backend/metric-id=stalled!1cycles!1backend/}:W,duration_time'
	cpu-cycles -> armv8_pmuv3_0/metric-id=cpu!1cycles,cpu-cycles/
	stalled-cycles-backend -> armv8_pmuv3_0/metric-id=stalled!1cycles!1backend,stalled-cycles-backend/
	duration_time -> tool/duration_time/
	Matched metric-id cpu-cycles to cpu-cycles
	Matched metric-id stalled-cycles-backend to stalled-cycles-backend
	Matched metric-id duration_time to duration_time
	Matched metric-id software@task-clock,name=task-clock@ to task-clock
	Matched metric-id duration_time to duration_time
	copying metric event for cgroup 'root': context-switches (idx=0)
	copying metric event for cgroup 'root': page-faults (idx=1)
	copying metric event for cgroup 'root': task-clock (idx=2)
	copying metric event for cgroup 'root': cpu-cycles (idx=3)
	copying metric event for cgroup 'root': branches (idx=4)
	copying metric event for cgroup 'root': cpu-migrations (idx=5)
	copying metric event for cgroup 'root': STALL_SLOT_FRONTEND (idx=7)
	copying metric event for cgroup 'root': stalled-cycles-backend (idx=29)
	copying metric event for cgroup 'root': STALL_SLOT_BACKEND (idx=11)
	copying metric event for cgroup 'root': stalled-cycles-backend (idx=15)
	copying metric event for cgroup 'root': instructions (idx=20)
	copying metric event for cgroup 'root': stalled-cycles-frontend (idx=23)
	copying metric event for cgroup 'root': branch-misses (idx=25)
	copying metric event for cgroup 'root': context-switches (idx=6)
	copying metric event for cgroup 'root': page-faults (idx=8)
	copying metric event for cgroup 'root': task-clock (idx=9)
	copying metric event for cgroup 'root': cpu-cycles (idx=13)
	copying metric event for cgroup 'root': branches (idx=12)
	copying metric event for cgroup 'root': cpu-migrations (idx=7)
	copying metric event for cgroup 'root': STALL_SLOT_FRONTEND (idx=25)
	copying metric event for cgroup 'root': stalled-cycles-backend (idx=19)
	copying metric event for cgroup 'root': STALL_SLOT_BACKEND (idx=29)
	copying metric event for cgroup 'root': stalled-cycles-backend (idx=20)
	copying metric event for cgroup 'root': instructions (idx=15)
	copying metric event for cgroup 'root': stalled-cycles-frontend (idx=17)
	copying metric event for cgroup 'root': branch-misses (idx=10)
	Control descriptor is not initialized
	perf: util/evsel.c:2034: get_group_fd: Assertion `!(!leader->core.fd)' failed.
	[1]    832866 IOT instruction (core dumped)  ./tools/perf/perf stat -v sleep 0.01
Quoted 6 lines

The PMUs for cpu-cycles and stalled-cycles differ? This may mean we also need to be smarting in determining PMUs for legacy events.

It'd be interesting to see what events are coming from the kernel, e.g.:
```

$ ls /sys/bus/event_source/devices/*/events
# ls /sys/bus/event_source/devices/*/events
	/sys/bus/event_source/devices/armv8_pmuv3_0/events:
	br_mis_pred          cti_trigout7        l1d_tlb_refill      l2d_tlb_refill         mem_access_checked_wr  stall_backend_mem
	br_mis_pred_retired  dtlb_walk           l1i_cache           l3d_cache              memory_error           stall_frontend
	br_pred              exc_return          l1i_cache_lmiss     l3d_cache_allocate     op_retired             stall_slot
	br_retired           exc_taken           l1i_cache_refill    l3d_cache_lmiss_rd     op_spec                stall_slot_backend
	bus_access           inst_retired        l1i_tlb             l3d_cache_refill       remote_access          stall_slot_frontend
	bus_cycles           inst_spec           l1i_tlb_refill      ld_align_lat           sample_collision       trb_wrap
	cid_write_retired    itlb_walk           l2d_cache           ldst_align_lat         sample_feed            trcextout0
	cnt_cycles           l1d_cache           l2d_cache_allocate  ll_cache_miss_rd       sample_filtrate        trcextout1
	cpu_cycles           l1d_cache_lmiss_rd  l2d_cache_lmiss_rd  ll_cache_rd            sample_pop             trcextout2
	cti_trigout4         l1d_cache_refill    l2d_cache_refill    mem_access             st_align_lat           trcextout3
	cti_trigout5         l1d_cache_wb        l2d_cache_wb        mem_access_checked     stall                  ttbr_write_retired
	cti_trigout6         l1d_tlb             l2d_tlb             mem_access_checked_rd  stall_backend

/sys/bus/event_source/devices/cs_etm/events:
autofdo

/sys/bus/event_source/devices/nvidia_cnvlink_pmu_0/events:
cycles rd_bytes_rem rd_cum_outs_rem rd_req_rem total_bytes_rem total_req_rem wr_bytes_rem wr_req_rem
rd_bytes_loc rd_cum_outs_loc rd_req_loc total_bytes_loc total_req_loc wr_bytes_loc wr_req_loc

/sys/bus/event_source/devices/nvidia_nvlink_c2c0_pmu_0/events:
cycles rd_bytes_rem rd_cum_outs_rem rd_req_rem total_bytes_rem total_req_rem wr_bytes_rem wr_req_rem
rd_bytes_loc rd_cum_outs_loc rd_req_loc total_bytes_loc total_req_loc wr_bytes_loc wr_req_loc

/sys/bus/event_source/devices/nvidia_nvlink_c2c1_pmu_0/events:
cycles rd_bytes_rem rd_cum_outs_rem rd_req_rem total_bytes_rem total_req_rem wr_bytes_rem wr_req_rem
rd_bytes_loc rd_cum_outs_loc rd_req_loc total_bytes_loc total_req_loc wr_bytes_loc wr_req_loc

/sys/bus/event_source/devices/nvidia_pcie_pmu_0/events:
cycles rd_bytes_rem rd_cum_outs_rem rd_req_rem total_bytes_rem total_req_rem wr_bytes_rem wr_req_rem
rd_bytes_loc rd_cum_outs_loc rd_req_loc total_bytes_loc total_req_loc wr_bytes_loc wr_req_loc

/sys/bus/event_source/devices/nvidia_scf_pmu_0/events:
bus_cycles gmem_rd_data scf_cache socket_1_rd_access socket_2_wb_data
cmem_rd_access gmem_rd_outstanding scf_cache_allocate socket_1_rd_data socket_2_wr_access
cmem_rd_data gmem_wb_access scf_cache_refill socket_1_rd_outstanding socket_2_wr_data
cmem_rd_outstanding gmem_wb_data scf_cache_wb socket_1_wb_access socket_3_rd_access
cmem_wb_access gmem_wr_access socket_0_rd_access socket_1_wb_data socket_3_rd_data
cmem_wb_data gmem_wr_data socket_0_rd_data socket_1_wr_access socket_3_rd_outstanding
cmem_wr_access gmem_wr_total_bytes socket_0_rd_outstanding socket_1_wr_data socket_3_wb_access
cmem_wr_data remote_socket_rd_access socket_0_wb_access socket_2_rd_access socket_3_wb_data
cmem_wr_total_bytes remote_socket_rd_data socket_0_wb_data socket_2_rd_data socket_3_wr_access
cycles remote_socket_rd_outstanding socket_0_wr_access socket_2_rd_outstanding socket_3_wr_data
gmem_rd_access remote_socket_wr_total_bytes socket_0_wr_data socket_2_wb_access

/sys/bus/event_source/devices/smmuv3_pmcg_11002/events:
config_cache_miss config_struct_access cycles pcie_ats_trans_rq tlb_miss transaction trans_table_walk_access

/sys/bus/event_source/devices/smmuv3_pmcg_11042/events:
cycles pcie_ats_trans_passed tlb_miss transaction

/sys/bus/event_source/devices/smmuv3_pmcg_11062/events:
cycles pcie_ats_trans_passed tlb_miss transaction

/sys/bus/event_source/devices/smmuv3_pmcg_11082/events:
cycles pcie_ats_trans_passed tlb_miss transaction

/sys/bus/event_source/devices/smmuv3_pmcg_110a2/events:
cycles pcie_ats_trans_passed tlb_miss transaction

/sys/bus/event_source/devices/smmuv3_pmcg_12002/events:
config_cache_miss config_struct_access cycles pcie_ats_trans_rq tlb_miss transaction trans_table_walk_access

/sys/bus/event_source/devices/smmuv3_pmcg_12042/events:
cycles pcie_ats_trans_passed tlb_miss transaction

/sys/bus/event_source/devices/smmuv3_pmcg_12062/events:
cycles pcie_ats_trans_passed tlb_miss transaction

/sys/bus/event_source/devices/smmuv3_pmcg_12082/events:
cycles pcie_ats_trans_passed tlb_miss transaction

/sys/bus/event_source/devices/smmuv3_pmcg_120a2/events:
cycles pcie_ats_trans_passed tlb_miss transaction

/sys/bus/event_source/devices/smmuv3_pmcg_15002/events:
config_cache_miss config_struct_access cycles pcie_ats_trans_rq tlb_miss transaction trans_table_walk_access

/sys/bus/event_source/devices/smmuv3_pmcg_15042/events:
cycles pcie_ats_trans_passed tlb_miss transaction

/sys/bus/event_source/devices/smmuv3_pmcg_15062/events:
cycles pcie_ats_trans_passed tlb_miss transaction

/sys/bus/event_source/devices/smmuv3_pmcg_15082/events:
cycles pcie_ats_trans_passed tlb_miss transaction

/sys/bus/event_source/devices/smmuv3_pmcg_150a2/events:
cycles pcie_ats_trans_passed tlb_miss transaction

/sys/bus/event_source/devices/smmuv3_pmcg_16002/events:
config_cache_miss config_struct_access cycles pcie_ats_trans_rq tlb_miss transaction trans_table_walk_access

/sys/bus/event_source/devices/smmuv3_pmcg_16042/events:
cycles pcie_ats_trans_passed tlb_miss transaction

/sys/bus/event_source/devices/smmuv3_pmcg_16062/events:
cycles pcie_ats_trans_passed tlb_miss transaction

/sys/bus/event_source/devices/smmuv3_pmcg_5002/events:
config_cache_miss config_struct_access cycles pcie_ats_trans_rq tlb_miss transaction trans_table_walk_access

/sys/bus/event_source/devices/smmuv3_pmcg_5042/events:
cycles pcie_ats_trans_passed tlb_miss transaction

/sys/bus/event_source/devices/smmuv3_pmcg_5062/events:
cycles pcie_ats_trans_passed tlb_miss transaction

Quoted 4 lines
```
and the cpuid to match it up with the json.
```
$ perf stat -v sleep 1 2>&1 |head -1
	# perf stat -v sleep 1 2>&1 |head -1
	Using CPUID 0x00000000410fd4f0
Quoted 3 lines

this information is in the verbose output too:
```

$ perf stat -vv sleep 1
	------------------------------------------------------------
	perf_event_attr:
	type                             4294967294 (tool)
	size                             144
	config                           0x1 (duration_time)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
	disabled                         1
	inherit                          1
	enable_on_exec                   1
	------------------------------------------------------------
	------------------------------------------------------------
	perf_event_attr:
	type                             4294967294 (tool)
	size                             144
	config                           0x1 (duration_time)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
	disabled                         1
	inherit                          1
	enable_on_exec                   1
	------------------------------------------------------------
	------------------------------------------------------------
	perf_event_attr:
	type                             4294967294 (tool)
	size                             144
	config                           0x1 (duration_time)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
	disabled                         1
	inherit                          1
	enable_on_exec                   1
	------------------------------------------------------------
	------------------------------------------------------------
	perf_event_attr:
	type                             4294967294 (tool)
	size                             144
	config                           0x1 (duration_time)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
	disabled                         1
	inherit                          1
	enable_on_exec                   1
	------------------------------------------------------------
	------------------------------------------------------------
	perf_event_attr:
	type                             4294967294 (tool)
	size                             144
	config                           0x1 (duration_time)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
	disabled                         1
	inherit                          1
	enable_on_exec                   1
	------------------------------------------------------------
	------------------------------------------------------------
	perf_event_attr:
	type                             4294967294 (tool)
	size                             144
	config                           0x1 (duration_time)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
	disabled                         1
	inherit                          1
	enable_on_exec                   1
	------------------------------------------------------------
	------------------------------------------------------------
	perf_event_attr:
	type                             1 (PERF_TYPE_SOFTWARE)
	size                             144
	config                           0x3 (PERF_COUNT_SW_CONTEXT_SWITCHES)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
	disabled                         1
	inherit                          1
	enable_on_exec                   1
	------------------------------------------------------------
	sys_perf_event_open: pid 865887  cpu -1  group_fd -1  flags 0x8 = 3
	------------------------------------------------------------
	perf_event_attr:
	type                             1 (PERF_TYPE_SOFTWARE)
	size                             144
	config                           0x4 (PERF_COUNT_SW_CPU_MIGRATIONS)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
	disabled                         1
	inherit                          1
	enable_on_exec                   1
	------------------------------------------------------------
	sys_perf_event_open: pid 865887  cpu -1  group_fd -1  flags 0x8 = 4
	------------------------------------------------------------
	perf_event_attr:
	type                             1 (PERF_TYPE_SOFTWARE)
	size                             144
	config                           0x2 (PERF_COUNT_SW_PAGE_FAULTS)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
	disabled                         1
	inherit                          1
	enable_on_exec                   1
	------------------------------------------------------------
	sys_perf_event_open: pid 865887  cpu -1  group_fd -1  flags 0x8 = 5
	------------------------------------------------------------
	perf_event_attr:
	type                             1 (PERF_TYPE_SOFTWARE)
	size                             144
	config                           0x1 (PERF_COUNT_SW_TASK_CLOCK)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
	disabled                         1
	inherit                          1
	enable_on_exec                   1
	------------------------------------------------------------
	sys_perf_event_open: pid 865887  cpu -1  group_fd -1  flags 0x8 = 7
	------------------------------------------------------------
	perf_event_attr:
	type                             0 (PERF_TYPE_HARDWARE)
	size                             144
	config                           0x5 (PERF_COUNT_HW_BRANCH_MISSES)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
	disabled                         1
	inherit                          1
	enable_on_exec                   1
	------------------------------------------------------------
	sys_perf_event_open: pid 865887  cpu -1  group_fd -1  flags 0x8 = 8
	------------------------------------------------------------
	perf_event_attr:
	type                             0 (PERF_TYPE_HARDWARE)
	size                             144
	config                           0x4 (PERF_COUNT_HW_BRANCH_INSTRUCTIONS)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
	inherit                          1
	------------------------------------------------------------
	sys_perf_event_open: pid 865887  cpu -1  group_fd 8  flags 0x8 = 9
	------------------------------------------------------------
	perf_event_attr:
	type                             0 (PERF_TYPE_HARDWARE)
	size                             144
	config                           0x4 (PERF_COUNT_HW_BRANCH_INSTRUCTIONS)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
	disabled                         1
	inherit                          1
	enable_on_exec                   1
	------------------------------------------------------------
	sys_perf_event_open: pid 865887  cpu -1  group_fd -1  flags 0x8 = 10
	------------------------------------------------------------
	perf_event_attr:
	type                             0 (PERF_TYPE_HARDWARE)
	size                             144
	config                           0 (PERF_COUNT_HW_CPU_CYCLES)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
	disabled                         1
	inherit                          1
	enable_on_exec                   1
	------------------------------------------------------------
	sys_perf_event_open: pid 865887  cpu -1  group_fd -1  flags 0x8 = 11
	------------------------------------------------------------
	perf_event_attr:
	type                             0 (PERF_TYPE_HARDWARE)
	size                             144
	config                           0 (PERF_COUNT_HW_CPU_CYCLES)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
	disabled                         1
	inherit                          1
	enable_on_exec                   1
	------------------------------------------------------------
	sys_perf_event_open: pid 865887  cpu -1  group_fd -1  flags 0x8 = 12
	------------------------------------------------------------
	perf_event_attr:
	type                             0 (PERF_TYPE_HARDWARE)
	size                             144
	config                           0x1 (PERF_COUNT_HW_INSTRUCTIONS)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
	inherit                          1
	------------------------------------------------------------
	sys_perf_event_open: pid 865887  cpu -1  group_fd 12  flags 0x8 = 13
	------------------------------------------------------------
	perf_event_attr:
	type                             0 (PERF_TYPE_HARDWARE)
	size                             144
	config                           0 (PERF_COUNT_HW_CPU_CYCLES)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
	disabled                         1
	inherit                          1
	enable_on_exec                   1
	------------------------------------------------------------
	sys_perf_event_open: pid 865887  cpu -1  group_fd -1  flags 0x8 = 14
	------------------------------------------------------------
	perf_event_attr:
	type                             0 (PERF_TYPE_HARDWARE)
	size                             144
	config                           0x7 (PERF_COUNT_HW_STALLED_CYCLES_FRONTEND)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
	inherit                          1
	------------------------------------------------------------
	sys_perf_event_open: pid 865887  cpu -1  group_fd 14  flags 0x8 = 15
	------------------------------------------------------------
	perf_event_attr:
	type                             0 (PERF_TYPE_HARDWARE)
	size                             144
	config                           0 (PERF_COUNT_HW_CPU_CYCLES)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
	disabled                         1
	inherit                          1
	enable_on_exec                   1
	------------------------------------------------------------
	sys_perf_event_open: pid 865887  cpu -1  group_fd -1  flags 0x8 = 16
	------------------------------------------------------------
	perf_event_attr:
	type                             0 (PERF_TYPE_HARDWARE)
	size                             144
	config                           0x8 (PERF_COUNT_HW_STALLED_CYCLES_BACKEND)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
	inherit                          1
	------------------------------------------------------------
	sys_perf_event_open: pid 865887  cpu -1  group_fd 16  flags 0x8 = 17
	------------------------------------------------------------
	perf_event_attr:
	type                             0 (PERF_TYPE_HARDWARE)
	size                             144
	config                           0x8 (PERF_COUNT_HW_STALLED_CYCLES_BACKEND)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
	disabled                         1
	inherit                          1
	enable_on_exec                   1
	------------------------------------------------------------
	sys_perf_event_open: pid 865887  cpu -1  group_fd -1  flags 0x8 = 18
	------------------------------------------------------------
	perf_event_attr:
	type                             0 (PERF_TYPE_HARDWARE)
	size                             144
	config                           0x1 (PERF_COUNT_HW_INSTRUCTIONS)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
	inherit                          1
	------------------------------------------------------------
	sys_perf_event_open: pid 865887  cpu -1  group_fd 18  flags 0x8 = 19
	------------------------------------------------------------
	perf_event_attr:
	type                             0 (PERF_TYPE_HARDWARE)
	size                             144
	config                           0x7 (PERF_COUNT_HW_STALLED_CYCLES_FRONTEND)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
	inherit                          1
	------------------------------------------------------------
	sys_perf_event_open: pid 865887  cpu -1  group_fd 18  flags 0x8 = 20
	------------------------------------------------------------
	perf_event_attr:
	type                             4294967294 (tool)
	size                             144
	config                           0x1 (duration_time)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
	disabled                         1
	inherit                          1
	enable_on_exec                   1
	------------------------------------------------------------
	------------------------------------------------------------
	perf_event_attr:
	type                             4294967294 (tool)
	size                             144
	config                           0x1 (duration_time)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
	disabled                         1
	inherit                          1
	enable_on_exec                   1
	------------------------------------------------------------
	------------------------------------------------------------
	perf_event_attr:
	type                             4294967294 (tool)
	size                             144
	config                           0x1 (duration_time)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
	disabled                         1
	inherit                          1
	enable_on_exec                   1
	------------------------------------------------------------
	------------------------------------------------------------
	perf_event_attr:
	type                             4294967294 (tool)
	size                             144
	config                           0x1 (duration_time)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
	disabled                         1
	inherit                          1
	enable_on_exec                   1
	------------------------------------------------------------
	------------------------------------------------------------
	perf_event_attr:
	type                             10 (armv8_pmuv3_0)
	size                             144
	config                           0x10 (br_mis_pred)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
	disabled                         1
	inherit                          1
	enable_on_exec                   1
	------------------------------------------------------------
	sys_perf_event_open: pid 865887  cpu -1  group_fd -1  flags 0x8 = 21
	------------------------------------------------------------
	perf_event_attr:
	type                             10 (armv8_pmuv3_0)
	size                             144
	config                           0x3b (op_spec)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
	inherit                          1
	------------------------------------------------------------
	sys_perf_event_open: pid 865887  cpu -1  group_fd 21  flags 0x8 = 22
	------------------------------------------------------------
	perf_event_attr:
	type                             10 (armv8_pmuv3_0)
	size                             144
	config                           0x3f (stall_slot)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
	inherit                          1
	------------------------------------------------------------
	sys_perf_event_open: pid 865887  cpu -1  group_fd 21  flags 0x8 = 23
	------------------------------------------------------------
	perf_event_attr:
	type                             10 (armv8_pmuv3_0)
	size                             144
	config                           0x11 (cpu_cycles)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
	inherit                          1
	------------------------------------------------------------
	sys_perf_event_open: pid 865887  cpu -1  group_fd 21  flags 0x8 = 24
	------------------------------------------------------------
	perf_event_attr:
	type                             10 (armv8_pmuv3_0)
	size                             144
	config                           0x3a (op_retired)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
	inherit                          1
	------------------------------------------------------------
	sys_perf_event_open: pid 865887  cpu -1  group_fd 21  flags 0x8 = 25
	------------------------------------------------------------
	perf_event_attr:
	type                             10 (armv8_pmuv3_0)
	size                             144
	config                           0x11 (cpu_cycles)
	sample_type                      IDENTIFIER
	read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID|GROUP
	inherit                          1
	------------------------------------------------------------

Thanks for your help,
--breno