Skip to contentSkip to conversation

perf stat: Include PMU name and split uncore events per PMU in metric-only JSON output

PATCH Superseded Chun-Tse Shao 3 participants 4 messages Jul 29 – Aug 4 perf events

v3 of 8 Superseded Latest is v8 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 perf stat: Include PMU name and split uncore events per PMU in metric-only JSON output

When running perf stat with -A (--no-aggr) and --metric-only in JSON output mode (-j), perf stat evaluates metric expressions across all matching PMUs (including uncore PMUs like uncore_iio_0, uncore_iio_1, etc.).

However, perf stat previously formatted JSON output by printing only "cpu" : "<id>" and grouping all metric values on a single line per CPU without identifying which PMU instance evaluated each metric. As a result, when an uncore event spans multiple PMU boxes, perf stat printed repeated, ambiguous metric keys without PMU names.

Fix this by:
1. Including "pmu" : "<pmu_name>" in print_aggr_id_json when evsel->pmu is a non-core or hybrid PMU in AGGR_NONE mode (-A).
2. Starting a new JSON metric line in AGGR_NONE mode (-A) whenever the underlying PMU instance changes across PMU events.
3. Updating perf_json_output_lint.py to recognize the new "pmu" key in the JSON test suite.

Before the fix:

  $ perf stat -M iio_bandwidth_read -a -A --metric-only -j -I 1000
  {"interval" : 1.001017947, "cpu" : "0", "MB/s  iio_bandwidth_read" : "0.0", "MB/s  iio_bandwidth_read" : "0.0", "MB/s  iio_bandwidth_read" : "22.5", "MB/s  iio_bandwidth_read" : "0.0", "MB/s  iio_bandwidth_read" : "0.2", "MB/s  iio_bandwidth_read" : "0.0", "MB/s  iio_bandwidth_read" : "0.0", "MB/s  iio_bandwidth_read" : "0.0", "MB/s  iio_bandwidth_read" : "0.0", "MB/s  iio_bandwidth_read" : "0.0"}

There is no way to determine which uncore device generated the metrics.

After:

  $ perf stat -M iio_bandwidth_read -a -A --metric-only -j -I 1000
  {"interval" : 1.000314908, "cpu" : "0", "pmu" : "uncore_iio_0", "MB/s  iio_bandwidth_read" : "0.0"}
  {"interval" : 1.000314908, "cpu" : "0", "pmu" : "uncore_iio_1", "MB/s  iio_bandwidth_read" : "0.1"}
  {"interval" : 1.000314908, "cpu" : "0", "pmu" : "uncore_iio_11", "MB/s  iio_bandwidth_read" : "0.0"}
  ...
  {"interval" : 1.000314908, "cpu" : "56", "pmu" : "uncore_iio_0", "MB/s  iio_bandwidth_read" : "0.0"}
  {"interval" : 1.000314908, "cpu" : "56", "pmu" : "uncore_iio_1", "MB/s  iio_bandwidth_read" : "0.0"}
  {"interval" : 1.000314908, "cpu" : "56", "pmu" : "uncore_iio_11", "MB/s  iio_bandwidth_read" : "0.0"}
Assisted-by Gemini:gemini-3.1-pro-preview

---

v3:
- Add before the fix output.

v2: lore.kernel.org/[email protected]
- Fix print_metric_begin() initialization in print_no_aggr_metric() to ensure tool events always open lines cleanly.
- Simplify and fix PMU line-split transition tracking when switching between different PMU instances or transitioning from tool events.
- Add bounds check on aggr_map index in print_no_aggr_metric() to prevent out-of-bounds array reads if sysfs CPU topology is incomplete.

v1: lore.kernel.org/[email protected]

 .../tests/shell/lib/perf_json_output_lint.py  |  1 +
 tools/perf/util/stat-display.c                | 48 ++++++++++++++-----
 2 files changed, 37 insertions(+), 12 deletions(-)
tools/perf/tests/shell/lib/perf_json_output_lint.py +1 -0
index dafbde56cc76..36767905fcdb 100644
@@ -64,6 +64,7 @@ def check_json_output(expected_items):
       'metric-threshold': lambda x: x in ['unknown', 'good', 'less good', 'nearly bad', 'bad'],
       'metricgroup': lambda x: True,
       'node': lambda x: True,
+      'pmu': lambda x: True,
       'pcnt-running': lambda x: isfloat(x),
       'socket': lambda x: True,
       'thread': lambda x: True,
tools/perf/util/stat-display.c +36 -12

115 lines not on this page · raw message

--
2.55.0.508.g3f0d502094-goog

#1 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Flawed line-split transition logic causes spurious empty JSON objects and incorrect PMU…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Flawed line-split transition logic causes spurious empty JSON objects and incorrect PMU attribution for tool events. --

--- Patch [1]: [PATCH v3] perf stat: Include PMU name and split uncore events per PMU in metric-only JSON output --- commit 20e3cd08fd5e415b458343262ee76bc1f5f6ce85
Author: Chun-Tse Shao <[email protected]>

perf stat: Include PMU name and split uncore events per PMU in metric-only JSON output

This commit modifies perf stat JSON output in --metric-only and -A mode to group metrics by PMU. It achieves this by adding a "pmu" key to the JSON output and starting a new JSON line when the underlying PMU instance changes.

Quoted 4 lines
tools/perf/util/stat-display.c +0 -0
index b337cc23f4131..6785e2c442bd7 100644

[ ... ]

Quoted 9 lines
tools/perf/util/stat-display.c +2 -0
@@ -1146,13 +1149,34 @@ static void print_no_aggr_metric(struct perf_stat_config *config,
 				if (config->aggr_map->map[aggr_idx].cpu.cpu == cpu.cpu)
 					break;
 			}
+			if (aggr_idx >= config->aggr_map->nr)
+				continue;
 
 			os->evsel = counter;
 			os->id = aggr_cpu_id__cpu(cpu, /*data=*/NULL);

tools/perf/util/stat-display.c static void print_no_aggr_metric(struct perf_stat_config *config, · line 1163

Quoted 10 lines
tools/perf/util/stat-display.c +7 -3
-			if (first) {
-				print_metric_begin(config, evlist, os, aggr_idx);
-				first = false;
+
+			if (config->metric_only) {
+				struct perf_pmu *pmu = counter->pmu;
+
+				if (!evsel__is_tool(counter)) {
+					if (config->json_output && line_open &&
+					    (!last_pmu || pmu != last_pmu)) {

[Severity: Medium]
Does this line-split transition logic handle tool events correctly depending on their order in the event list?

If a tool event is processed first, line_open becomes true, but last_pmu remains NULL. When the next event (a PMU event) is processed, the condition (!last_pmu || pmu != last_pmu) evaluates to true. This prematurely closes the line and creates a spurious empty JSON object for the tool event.

Conversely, if a tool event comes after a PMU event, the else-if block below is taken because !evsel__is_tool(counter) evaluates to false. This bypasses the line-splitting check entirely, causing the tool event to silently merge into the preceding PMU's JSON object and incorrectly inherit its PMU attribution.

Quoted 15 lines
tools/perf/util/stat-display.c +14 -0
+						print_metric_end(config, os);
+						line_open = false;
+					}
+					if (!line_open) {
+						print_metric_begin(config, evlist, os,
+								   aggr_idx, counter);
+						line_open = true;
+					}
+					last_pmu = pmu;
+				} else if (!line_open) {
+					print_metric_begin(config, evlist, os,
+							   aggr_idx, counter);
+					line_open = true;
+				}
 			}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
maintainer
#2 lore raw
Quoted 45 lines · On Wed, Jul 29, 2026 at 05:17:41PM +0000, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Flawed line-split transition logic causes spurious empty JSON objects and incorrect PMU attribution for tool events. --

--- Patch [1]: [PATCH v3] perf stat: Include PMU name and split uncore events per PMU in metric-only JSON output --- commit 20e3cd08fd5e415b458343262ee76bc1f5f6ce85
Author: Chun-Tse Shao <[email protected]>

perf stat: Include PMU name and split uncore events per PMU in metric-only JSON output

This commit modifies perf stat JSON output in --metric-only and -A mode to group metrics by PMU. It achieves this by adding a "pmu" key to the JSON output and starting a new JSON line when the underlying PMU instance changes.

Quoted 4 lines
tools/perf/util/stat-display.c +0 -0
index b337cc23f4131..6785e2c442bd7 100644

[ ... ]

Quoted 9 lines
tools/perf/util/stat-display.c +2 -0
@@ -1146,13 +1149,34 @@ static void print_no_aggr_metric(struct perf_stat_config *config,
 				if (config->aggr_map->map[aggr_idx].cpu.cpu == cpu.cpu)
 					break;
 			}
+			if (aggr_idx >= config->aggr_map->nr)
+				continue;
 
 			os->evsel = counter;
 			os->id = aggr_cpu_id__cpu(cpu, /*data=*/NULL);

tools/perf/util/stat-display.c static void print_no_aggr_metric(struct perf_stat_config *config, · line 1163

Quoted 10 lines
tools/perf/util/stat-display.c +7 -3
-			if (first) {
-				print_metric_begin(config, evlist, os, aggr_idx);
-				first = false;
+
+			if (config->metric_only) {
+				struct perf_pmu *pmu = counter->pmu;
+
+				if (!evsel__is_tool(counter)) {
+					if (config->json_output && line_open &&
+					    (!last_pmu || pmu != last_pmu)) {

[Severity: Medium]
Does this line-split transition logic handle tool events correctly depending on their order in the event list?

Quoted 4 lines

If a tool event is processed first, line_open becomes true, but last_pmu remains NULL. When the next event (a PMU event) is processed, the condition (!last_pmu || pmu != last_pmu) evaluates to true. This prematurely closes the line and creates a spurious empty JSON object for the tool event.

Something like this?

  $ sudo ./perf stat -aA -e user_time -M insn_per_cycle -j --metric-only true
  {"cpu" : "0", "pmu" : "tool"}
  {"cpu" : "0", "pmu" : "cpu_core", "instructions  insn_per_cycle" : "0.8"}
  {"cpu" : "1", "pmu" : "tool"}
  {"cpu" : "1", "pmu" : "cpu_core", "instructions  insn_per_cycle" : "0.1"}
  {"cpu" : "2", "pmu" : "tool"}
  {"cpu" : "2", "pmu" : "cpu_core", "instructions  insn_per_cycle" : "0.5"}
  {"cpu" : "3", "pmu" : "tool"}
  {"cpu" : "3", "pmu" : "cpu_core", "instructions  insn_per_cycle" : "0.3"}
  ...

Thanks,
Namhyung

Quoted 25 lines

Conversely, if a tool event comes after a PMU event, the else-if block below is taken because !evsel__is_tool(counter) evaluates to false. This bypasses the line-splitting check entirely, causing the tool event to silently merge into the preceding PMU's JSON object and incorrectly inherit its PMU attribution.

Quoted 15 lines
tools/perf/util/stat-display.c +14 -0
+						print_metric_end(config, os);
+						line_open = false;
+					}
+					if (!line_open) {
+						print_metric_begin(config, evlist, os,
+								   aggr_idx, counter);
+						line_open = true;
+					}
+					last_pmu = pmu;
+				} else if (!line_open) {
+					print_metric_begin(config, evlist, os,
+							   aggr_idx, counter);
+					line_open = true;
+				}
 			}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#3 lore raw
Quoted 65 lines · Namhyung Kim
Quoted 45 lines · On Wed, Jul 29, 2026 at 05:17:41PM +0000, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Flawed line-split transition logic causes spurious empty JSON objects and incorrect PMU attribution for tool events. --

--- Patch [1]: [PATCH v3] perf stat: Include PMU name and split uncore events per PMU in metric-only JSON output --- commit 20e3cd08fd5e415b458343262ee76bc1f5f6ce85
Author: Chun-Tse Shao <[email protected]>

perf stat: Include PMU name and split uncore events per PMU in metric-only JSON output

This commit modifies perf stat JSON output in --metric-only and -A mode to group metrics by PMU. It achieves this by adding a "pmu" key to the JSON output and starting a new JSON line when the underlying PMU instance changes.

Quoted 4 lines
tools/perf/util/stat-display.c +0 -0
index b337cc23f4131..6785e2c442bd7 100644

[ ... ]

Quoted 9 lines
tools/perf/util/stat-display.c +2 -0
@@ -1146,13 +1149,34 @@ static void print_no_aggr_metric(struct perf_stat_config *config,
                            if (config->aggr_map->map[aggr_idx].cpu.cpu == cpu.cpu)
                                    break;
                    }
+                   if (aggr_idx >= config->aggr_map->nr)
+                           continue;
 
                    os->evsel = counter;
                    os->id = aggr_cpu_id__cpu(cpu, /*data=*/NULL);

tools/perf/util/stat-display.c static void print_no_aggr_metric(struct perf_stat_config *config, · line 1163

Quoted 10 lines
tools/perf/util/stat-display.c +7 -3
-                   if (first) {
-                           print_metric_begin(config, evlist, os, aggr_idx);
-                           first = false;
+
+                   if (config->metric_only) {
+                           struct perf_pmu *pmu = counter->pmu;
+
+                           if (!evsel__is_tool(counter)) {
+                                   if (config->json_output && line_open &&
+                                       (!last_pmu || pmu != last_pmu)) {

[Severity: Medium]
Does this line-split transition logic handle tool events correctly depending on their order in the event list?

Quoted 4 lines

If a tool event is processed first, line_open becomes true, but last_pmu remains NULL. When the next event (a PMU event) is processed, the condition (!last_pmu || pmu != last_pmu) evaluates to true. This prematurely closes the line and creates a spurious empty JSON object for the tool event.

Something like this?

  $ sudo ./perf stat -aA -e user_time -M insn_per_cycle -j --metric-only true
  {"cpu" : "0", "pmu" : "tool"}
  {"cpu" : "0", "pmu" : "cpu_core", "instructions  insn_per_cycle" : "0.8"}
  {"cpu" : "1", "pmu" : "tool"}
  {"cpu" : "1", "pmu" : "cpu_core", "instructions  insn_per_cycle" : "0.1"}
  {"cpu" : "2", "pmu" : "tool"}
  {"cpu" : "2", "pmu" : "cpu_core", "instructions  insn_per_cycle" : "0.5"}
  {"cpu" : "3", "pmu" : "tool"}
  {"cpu" : "3", "pmu" : "cpu_core", "instructions  insn_per_cycle" : "0.3"}
  ...
Quoted 2 lines

Thanks,
Namhyung

Thanks for the test, I sent the v4 patch for fixing this problem:
lore.kernel.org/[email protected]

Test:

  $ perf stat -aA -e user_time -M insn_per_cycle -j --metric-only true
  {"cpu" : "0", "instructions  insn_per_cycle" : "0.9"}
  {"cpu" : "1", "instructions  insn_per_cycle" : "0.6"}
  {"cpu" : "2", "instructions  insn_per_cycle" : "0.2"}
  {"cpu" : "3", "instructions  insn_per_cycle" : "0.1"}
  {"cpu" : "4", "instructions  insn_per_cycle" : "0.2"}
  {"cpu" : "5", "instructions  insn_per_cycle" : "0.1"}
  {"cpu" : "6", "instructions  insn_per_cycle" : "0.1"}
  {"cpu" : "7", "instructions  insn_per_cycle" : "0.1"}
  {"cpu" : "8", "instructions  insn_per_cycle" : "0.1"}
  {"cpu" : "9", "instructions  insn_per_cycle" : "0.1"}
  ...

-CT

Quoted 26 lines
Quoted 26 lines

Conversely, if a tool event comes after a PMU event, the else-if block below is taken because !evsel__is_tool(counter) evaluates to false. This bypasses the line-splitting check entirely, causing the tool event to silently merge into the preceding PMU's JSON object and incorrectly inherit its PMU attribution.

Quoted 5 lines
tools/perf/util/stat-display.c +5 -0
+                                           print_metric_end(config, os);
+                                           line_open = false;
+                                   }
+                                   if (!line_open) {
+                                           print_metric_begin(config, evlist, os,

tools/perf/util/stat-display.c static void print_no_aggr_metric(struct perf_stat_config *config, · line 1178

Quoted 10 lines
tools/perf/util/stat-display.c +9 -0
+                                                              aggr_idx, counter);
+                                           line_open = true;
+                                   }
+                                   last_pmu = pmu;
+                           } else if (!line_open) {
+                                   print_metric_begin(config, evlist, os,
+                                                      aggr_idx, counter);
+                                   line_open = true;
+                           }
                    }

--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1