Skip to contentSkip to conversation

perf jevents: Add NVIDIA Tegra410 uncore DDR and PCIe metrics

PATCH In review Chun-Tse Shao 5 participants 14 messages Sep 28 – Oct 19 perf events

lore
Share & tools
lore
#0 lore raw

cover perf jevents: Add NVIDIA Tegra410 uncore DDR and PCIe metrics

Add DDR bandwidth/latency and PCIe bandwidth metrics for the NVIDIA Tegra410 SoC to arm64_metrics.py. The metrics use the sysfs events of the Tegra410 UCF, CMEM latency and PCIE PMUs, and follow the formulas in Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst.

Patch 1 adds the DDR metrics and makes pmu-events/Build also run arm64_metrics.py for the nvidia vendor models. Patch 2 adds the PCIe metrics, in total and per PCIe Root Complex (RC).

Tested on a 2-socket Tegra410 system:

- memcpy with multiload (from multichase) on socket 0 CPUs and memory:
lpm_ddr_bw is 740-744 GB/s at steady state (perf stat -I 2000), multiload reports 708063 MiB/s (742 GB/s).
- Read only (stream-sum) and write only (memset) loads on socket 1:
lpm_ddr_rd_bw is 608 GB/s with 2.4 GB/s of writes, lpm_ddr_wr_bw is 628 GB/s with 1.1 GB/s of reads.
- memcpy on socket 0 CPUs with memory on socket 1: socket 0's lpm_ddr_rem_rd_bw and socket 1's lpm_ddr_rd_bw count nearly the same bytes (258.03 vs 258.73 GB).
- lpm_ddr_lat idle is 150 ns in the default aggregation, and 132 ns and 194 ns for socket 0 and 1 with --per-socket. Under the memcpy load it is 555 ns by default and 563 ns for socket 0 with --per-socket, and stays at 551-558 ns per interval with -I 1000.
- 8 GiB O_DIRECT dd read from an NVMe drive behind RC 0 of socket 0:
lpm_pcie_wr_bw_0 counts the 8 GiB at 8.7 GB/s (dd: 8.7 GB/s). With the buffer on node 0 it shows up in lpm_pcie_loc_wr_bw_0, with the buffer on node 1 in lpm_pcie_rem_wr_bw_0. With --per-socket, socket 0 shows the same and no metric of either socket is nan.
- With the nvidia_t410 metrics forced on a machine without these PMUs (PERF_CPUID=0x000000004e0f0100 on an x86 JEVENTS_ARCH=all build), the PCIe metrics read 0 rather than failing to parse, also with --per-socket.
- "perf test 10" (PMU JSON event tests) passes on the Tegra410 system and with the x86 JEVENTS_ARCH=all build.

Chun-Tse Shao (2):
perf jevents: Add NVIDIA Tegra410 uncore DDR metrics perf jevents: Add NVIDIA Tegra410 uncore PCIe metrics

 tools/perf/pmu-events/Build            |   2 +-
 tools/perf/pmu-events/arm64_metrics.py | 126 ++++++++++++++++++++++++-
 tools/perf/pmu-events/metric.py        |   2 +-
 3 files changed, 126 insertions(+), 4 deletions(-)

base-commit: 0ae6fc78c5ce0dfd18d8712a50f0fd4602eff103 --
2.56.0.rc1.315.gc6ed9934b7-goog

#1 lore raw

PATCH 1/2 perf jevents: Add NVIDIA Tegra410 uncore DDR metrics

Add DDR bandwidth and latency metrics for the NVIDIA Tegra410 SoC. The formulas follow Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst:

AVG_MEM_READ_BANDWIDTH_IN_GBPS = MEM_BYTES_RD / ELAPSED_TIME_IN_NS AVG_MEM_WRITE_BANDWIDTH_IN_GBPS = MEM_BYTES_WR / ELAPSED_TIME_IN_NS

FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS AVG_LATENCY_IN_CYCLES = RD_CUM_OUTS / RD_REQ AVERAGE_LATENCY_IN_NS = AVG_LATENCY_IN_CYCLES / FREQ_IN_GHZ

The bandwidth metrics use the mem_bytes_rd/mem_bytes_wr events of the Unified Coherence Fabric (UCF) PMU with source/destination filters:

lpm_ddr_bw, lpm_ddr_rd_bw, lpm_ddr_wr_bw: all sources to local CMEM. lpm_ddr_loc_rd_bw, lpm_ddr_loc_wr_bw: local CPU and non-CPU sources to
local CMEM.
lpm_ddr_rem_rd_bw, lpm_ddr_rem_wr_bw: local CPU and non-CPU sources to
remote memory.

The latency metrics use the CPU Memory (CMEM) latency PMU:

lpm_ddr_lat, lpm_ddr_rd_lat: average DDR read latency in ns.
lpm_ddr_wr_lat: the PMU only measures reads, so this is always 0.

There is one CMEM latency PMU per socket and perf sums the counts of the PMUs in an aggregation, e.g. of both sockets in the default aggregation mode. Use aggr_nr() to divide the summed cycles by the number of PMUs, otherwise the frequency is too high and the latency too low by that factor.

The events are sysfs events of the nvidia_ucf_pmu_<socket-id> and nvidia_cmem_latency_pmu_<socket-id> PMUs rather than json events, so add the "nvidia" PMU prefix to the prefixes that skip the named json event check in metric.py.

pmu-events/Build only runs arm64_metrics.py for the "arm" vendor models, so also run it for the "nvidia" vendor models to generate the Tegra410 metrics.

Assisted-by Gemini:gemini-3.1-pro-preview
---
 tools/perf/pmu-events/Build            |  2 +-
 tools/perf/pmu-events/arm64_metrics.py | 66 +++++++++++++++++++++++++-
 tools/perf/pmu-events/metric.py        |  2 +-
 3 files changed, 66 insertions(+), 4 deletions(-)
tools/perf/pmu-events/Build +1 -1
index 1ac067be9d34..8dc8cfcb8b04 100644
@@ -78,7 +78,7 @@ endif
 
 ifeq ($(JEVENTS_ARCH),$(filter $(JEVENTS_ARCH),arm64 all))
 # Generate ARM Json
-ARMS := $(shell ls -d pmu-events/arch/arm64/arm/*|grep -v cmn)
+ARMS := $(shell ls -d pmu-events/arch/arm64/arm/* pmu-events/arch/arm64/nvidia/*|grep -v cmn)
 ARM_METRICS = $(foreach x,$(ARMS),$(OUTPUT)$(x)/extra-metrics.json)
 ARM_METRICGROUPS = $(foreach x,$(ARMS),$(OUTPUT)$(x)/extra-metricgroups.json)
 GEN_JSON += $(ARM_METRICS) $(ARM_METRICGROUPS)
tools/perf/pmu-events/arm64_metrics.py +64 -2

84 lines not on this page · raw message

tools/perf/pmu-events/metric.py +1 -1
index f5d81eccd5c3..6fb69024d1fd 100644
@@ -91,7 +91,7 @@ def CheckEveryEvent(*names: str) -> None:
       name = name[:name.find(':')]
     elif '/' in name:
       name = name[:name.find('/')]
-      if any([name.startswith(x) for x in ['amd', 'arm', 'cpu', 'msr', 'power', 'cha', 'uncore']]):
+      if any([name.startswith(x) for x in ['amd', 'arm', 'cpu', 'msr', 'power', 'cha', 'uncore', 'nvidia']]):
         continue
     if name not in all_events_all_models:
       raise ValueError(f"Is {name} a named json event?")
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#2 lore raw

Re: [PATCH 1/2] perf jevents: Add NVIDIA Tegra410 uncore DDR metrics

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 13 lines

Add DDR bandwidth and latency metrics for the NVIDIA Tegra410 SoC. The formulas follow Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst:

AVG_MEM_READ_BANDWIDTH_IN_GBPS = MEM_BYTES_RD / ELAPSED_TIME_IN_NS AVG_MEM_WRITE_BANDWIDTH_IN_GBPS = MEM_BYTES_WR / ELAPSED_TIME_IN_NS

FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS AVG_LATENCY_IN_CYCLES = RD_CUM_OUTS / RD_REQ AVERAGE_LATENCY_IN_NS = AVG_LATENCY_IN_CYCLES / FREQ_IN_GHZ

The bandwidth metrics use the mem_bytes_rd/mem_bytes_wr events of the Unified Coherence Fabric (UCF) PMU with source/destination filters:

Quoted 8 lines

lpm_ddr_bw, lpm_ddr_rd_bw, lpm_ddr_wr_bw: all sources to local CMEM. lpm_ddr_loc_rd_bw, lpm_ddr_loc_wr_bw: local CPU and non-CPU sources to
[ ... ]
pmu-events/Build only runs arm64_metrics.py for the "arm" vendor models, so also run it for the "nvidia" vendor models to generate the Tegra410 metrics.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Tested-by on v1 1/2
#3 lore raw

RE: [PATCH 1/2] perf jevents: Add NVIDIA Tegra410 uncore DDR metrics

In reply to Chun-Tse Shao

Hi Chun-Tse,

I tested this patch on 2-socket Tegra410. Both DDR bandwidth and read latency metrics give correct values in the default, --per-socket, and -A modes.

Some small comments inline below.

Quoted 40 lines

-----Original Message-----
From: Chun-Tse Shao <[email protected]>
Sent: Monday, September 28, 2026 5:51 PM
To: [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Besar Wicaksono <[email protected]>; [email protected]; linux- [email protected]; Chun-Tse Shao <[email protected]>
Subject: [PATCH 1/2] perf jevents: Add NVIDIA Tegra410 uncore DDR metrics

External email: Use caution opening links or attachments

Add DDR bandwidth and latency metrics for the NVIDIA Tegra410 SoC. The formulas follow Documentation/admin-guide/perf/nvidia-tegra410- pmu.rst:

AVG_MEM_READ_BANDWIDTH_IN_GBPS = MEM_BYTES_RD /
ELAPSED_TIME_IN_NS
AVG_MEM_WRITE_BANDWIDTH_IN_GBPS = MEM_BYTES_WR /
ELAPSED_TIME_IN_NS

FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS AVG_LATENCY_IN_CYCLES = RD_CUM_OUTS / RD_REQ AVERAGE_LATENCY_IN_NS = AVG_LATENCY_IN_CYCLES / FREQ_IN_GHZ

The bandwidth metrics use the mem_bytes_rd/mem_bytes_wr events of the
Unified Coherence Fabric (UCF) PMU with source/destination filters:

lpm_ddr_bw, lpm_ddr_rd_bw, lpm_ddr_wr_bw: all sources to local CMEM. lpm_ddr_loc_rd_bw, lpm_ddr_loc_wr_bw: local CPU and non-CPU sources
to
local CMEM.
lpm_ddr_rem_rd_bw, lpm_ddr_rem_wr_bw: local CPU and non-CPU
sources to
remote memory.

Quoted 5 lines

The latency metrics use the CPU Memory (CMEM) latency PMU:

lpm_ddr_lat, lpm_ddr_rd_lat: average DDR read latency in ns.
lpm_ddr_wr_lat: the PMU only measures reads, so this is always 0.

A question on naming: what does the "lpm_" prefix stand for? Is it the convention for all metrics generated by the Python scripts (I see it on the Intel/AMD and common metrics too)?

Quoted 122 lines

There is one CMEM latency PMU per socket and perf sums the counts of the PMUs in an aggregation, e.g. of both sockets in the default aggregation mode. Use aggr_nr() to divide the summed cycles by the number of PMUs, otherwise the frequency is too high and the latency too low by that factor.

The events are sysfs events of the nvidia_ucf_pmu_<socket-id> and nvidia_cmem_latency_pmu_<socket-id> PMUs rather than json events, so add
the "nvidia" PMU prefix to the prefixes that skip the named json event check in metric.py.

pmu-events/Build only runs arm64_metrics.py for the "arm" vendor models,
so also run it for the "nvidia" vendor models to generate the Tegra410 metrics.

Assisted-by Gemini:gemini-3.1-pro-preview
---
 tools/perf/pmu-events/Build            |  2 +-
 tools/perf/pmu-events/arm64_metrics.py | 66
+++++++++++++++++++++++++-
 tools/perf/pmu-events/metric.py        |  2 +-
 3 files changed, 66 insertions(+), 4 deletions(-)
tools/perf/pmu-events/Build +1 -1
index 1ac067be9d34..8dc8cfcb8b04 100644
@@ -78,7 +78,7 @@ endif
 
 ifeq ($(JEVENTS_ARCH),$(filter $(JEVENTS_ARCH),arm64 all))
 # Generate ARM Json
-ARMS := $(shell ls -d pmu-events/arch/arm64/arm/*|grep -v cmn)
+ARMS := $(shell ls -d pmu-events/arch/arm64/arm/* pmu-
 events/arch/arm64/nvidia/*|grep -v cmn)
 ARM_METRICS = $(foreach x,$(ARMS),$(OUTPUT)$(x)/extra-metrics.json)
 ARM_METRICGROUPS = $(foreach x,$(ARMS),$(OUTPUT)$(x)/extra-

metricgroups.json)
GEN_JSON += $(ARM_METRICS) $(ARM_METRICGROUPS)

tools/perf/pmu- renamed from tools/perf/pmu-events/arm64_metrics.py +0 -0

events/arm64_metrics.py
index 4ecda96d11fa..b0d0651fec12 100755

tools/perf/pmu-events/arm64_metrics.py +7 -2
@@ -2,14 +2,75 @@
 # SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
 import argparse
 import os
-from metric import (JsonEncodeMetric,
 JsonEncodeMetricGroupDescriptions, LoadEvents,
-                    MetricGroup)
+from typing import Optional
+from metric import (aggr_nr, d_ratio, Event, JsonEncodeMetric,
+                    JsonEncodeMetricGroupDescriptions, LoadEvents, Metric,
 MetricGroup)
 from common_metrics import Cycles
 
 # Global command line arguments.
 _args = None
 
 
+def NvidiaT410Uncore() -> Optional[MetricGroup]:
+    """Uncore metrics for the NVIDIA Tegra410 SoC.
+
+    The formulas follow Documentation/admin-guide/perf/nvidia-tegra410-

pmu.rst.

tools/perf/pmu-events/arm64_metrics.py +11 -0
+    """
+    assert _args is not None
+    if _args.vendor != "nvidia" or _args.model != "t410":
+        return None
+
+    interval_sec = Event("duration_time")
+    mb_per_sec = f"{1/1e6}MB/s"
+
+    def DdrBw() -> MetricGroup:
+        # Unified Coherence Fabric (UCF) PMU:
+        # AVG_MEM_{READ,WRITE}_BANDWIDTH = MEM_BYTES_{RD,WR} /

ELAPSED_TIME

tools/perf/pmu-events/arm64_metrics.py +4 -0
+        loc_src = "src_loc_cpu=0x1,src_loc_noncpu=0x1"
+        rd = Event("nvidia_ucf_pmu/mem_bytes_rd,dst_loc_cmem=0x1/")
+        wr = Event("nvidia_ucf_pmu/mem_bytes_wr,dst_loc_cmem=0x1/")
+        loc_rd =

Event(f"nvidia_ucf_pmu/mem_bytes_rd,{loc_src},dst_loc_cmem=0x1/")
+ loc_wr =
Event(f"nvidia_ucf_pmu/mem_bytes_wr,{loc_src},dst_loc_cmem=0x1/")
+ rem_rd =
Event(f"nvidia_ucf_pmu/mem_bytes_rd,{loc_src},dst_rem=0x1/")
+ rem_wr =
Event(f"nvidia_ucf_pmu/mem_bytes_wr,{loc_src},dst_rem=0x1/")

tools/perf/pmu-events/arm64_metrics.py +20 -0
+        return MetricGroup("lpm_ddr_bw", [
+            Metric("lpm_ddr_bw", "Total DDR bandwidth",
+                   d_ratio(rd + wr, interval_sec), mb_per_sec),
+            Metric("lpm_ddr_rd_bw", "DDR read bandwidth",
+                   d_ratio(rd, interval_sec), mb_per_sec),
+            Metric("lpm_ddr_wr_bw", "DDR write bandwidth",
+                   d_ratio(wr, interval_sec), mb_per_sec),
+            Metric("lpm_ddr_loc_rd_bw", "Local DDR read bandwidth",
+                   d_ratio(loc_rd, interval_sec), mb_per_sec),
+            Metric("lpm_ddr_loc_wr_bw", "Local DDR write bandwidth",
+                   d_ratio(loc_wr, interval_sec), mb_per_sec),
+            Metric("lpm_ddr_rem_rd_bw", "Remote DDR read bandwidth",
+                   d_ratio(rem_rd, interval_sec), mb_per_sec),
+            Metric("lpm_ddr_rem_wr_bw", "Remote DDR write bandwidth",
+                   d_ratio(rem_wr, interval_sec), mb_per_sec),
+        ], description="NVIDIA Tegra410 DDR bandwidth")
+
+    def DdrLat() -> MetricGroup:
+        # CPU Memory (CMEM) latency PMU, one per socket:
+        # AVERAGE_LATENCY_IN_NS = (RD_CUM_OUTS / RD_REQ) / (CYCLES /

ELAPSED_TIME_IN_NS)

tools/perf/pmu-events/arm64_metrics.py +4 -0
+        rd_cum_outs = Event("nvidia_cmem_latency_pmu/rd_cum_outs/")
+        rd_req = Event("nvidia_cmem_latency_pmu/rd_req/")
+        cycles = Event("nvidia_cmem_latency_pmu/cycles/")
+        # perf sums the counts of the PMUs in an aggregation, e.g. both

sockets

tools/perf/pmu-events/arm64_metrics.py +2 -0
+        # by default, so divide cycles by their number to get the frequency.
+        rd_lat = d_ratio(1e9 * interval_sec * rd_cum_outs * aggr_nr(cycles),

rd_req * cycles)

tools/perf/pmu-events/arm64_metrics.py +2 -0
+        return MetricGroup("lpm_ddr_lat", [
+            Metric("lpm_ddr_lat", "Average DDR read latency in ns", rd_lat,
Quoted 2 lines

"1ns"),
+ Metric("lpm_ddr_rd_lat", "DDR read latency in ns", rd_lat, "1ns"),

Could we keep just lpm_ddr_rd_lat ?

tools/perf/pmu-events/arm64_metrics.py line 68

Quoted 5 lines

+ # The CMEM latency PMU only measures reads, write latency is
always 0.

tools/perf/pmu-events/arm64_metrics.py +3 -0
+            Metric("lpm_ddr_wr_lat", "DDR write latency in ns",
+                   rd_cum_outs - rd_cum_outs, "1ns"),
+        ], description="NVIDIA Tegra410 DDR latency")

This always prints 0.0 ns, which reads like a real measurement of zero-latency writes. Since the hardware can't measure write latency, I'd drop this metric rather than report a placeholder.

Thanks,
Besar

Quoted 34 lines
tools/perf/pmu-events/arm64_metrics.py +6 -0
+
+    return MetricGroup("lpm_uncore", [DdrBw(), DdrLat()],
+                       description="NVIDIA Tegra410 uncore metrics")
+
+
 def main() -> None:
     global _args
@@ -37,6 +98,7 @@ def main() -> None:
 
     all_metrics = MetricGroup("", [
         Cycles(),
+        NvidiaT410Uncore(),
     ])
 
     if _args.metricgroups:
tools/perf/pmu- renamed from tools/perf/pmu-events/metric.py +0 -0

events/metric.py
index f5d81eccd5c3..6fb69024d1fd 100644

tools/perf/pmu-events/metric.py +0 -1
@@ -91,7 +91,7 @@ def CheckEveryEvent(*names: str) -> None:
       name = name[:name.find(':')]
     elif '/' in name:
       name = name[:name.find('/')]
-      if any([name.startswith(x) for x in ['amd', 'arm', 'cpu', 'msr', 'power',

'cha', 'uncore']]):
+ if any([name.startswith(x) for x in ['amd', 'arm', 'cpu', 'msr', 'power',
'cha', 'uncore', 'nvidia']]):
continue
if name not in all_events_all_models:
raise ValueError(f"Is {name} a named json event?")
--
2.56.0.rc1.315.gc6ed9934b7-goog

reviewer
#4 lore raw

Re: [PATCH 1/2] perf jevents: Add NVIDIA Tegra410 uncore DDR metrics

Quoted 51 lines · Besar Wicaksono

Hi Chun-Tse,

I tested this patch on 2-socket Tegra410. Both DDR bandwidth and read latency metrics give correct values in the default, --per-socket, and -A modes.

Some small comments inline below.

Quoted 42 lines

-----Original Message-----
From: Chun-Tse Shao <[email protected]>
Sent: Monday, September 28, 2026 5:51 PM
To: [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Besar Wicaksono <[email protected]>; [email protected]; linux- [email protected]; Chun-Tse Shao <[email protected]>
Subject: [PATCH 1/2] perf jevents: Add NVIDIA Tegra410 uncore DDR metrics

External email: Use caution opening links or attachments

Add DDR bandwidth and latency metrics for the NVIDIA Tegra410 SoC. The formulas follow Documentation/admin-guide/perf/nvidia-tegra410- pmu.rst:

AVG_MEM_READ_BANDWIDTH_IN_GBPS = MEM_BYTES_RD /
ELAPSED_TIME_IN_NS
AVG_MEM_WRITE_BANDWIDTH_IN_GBPS = MEM_BYTES_WR /
ELAPSED_TIME_IN_NS

FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS AVG_LATENCY_IN_CYCLES = RD_CUM_OUTS / RD_REQ AVERAGE_LATENCY_IN_NS = AVG_LATENCY_IN_CYCLES / FREQ_IN_GHZ

The bandwidth metrics use the mem_bytes_rd/mem_bytes_wr events of the
Unified Coherence Fabric (UCF) PMU with source/destination filters:

lpm_ddr_bw, lpm_ddr_rd_bw, lpm_ddr_wr_bw: all sources to local CMEM. lpm_ddr_loc_rd_bw, lpm_ddr_loc_wr_bw: local CPU and non-CPU sources
to
local CMEM.
lpm_ddr_rem_rd_bw, lpm_ddr_rem_wr_bw: local CPU and non-CPU
sources to
remote memory.

The latency metrics use the CPU Memory (CMEM) latency PMU:

Quoted 7 lines
Quoted 3 lines

lpm_ddr_lat, lpm_ddr_rd_lat: average DDR read latency in ns.
lpm_ddr_wr_lat: the PMU only measures reads, so this is always 0.

A question on naming: what does the "lpm_" prefix stand for? Is it the convention for all metrics generated by the Python scripts (I see it on the Intel/AMD and common metrics too)?

Hi Besar, naming is one of the hardest problems in Computer Science :-) When we first added the metrics there was a request to differentiate between metrics coming from the python scripts and metrics coming from vendor code, like Intel top-down. We agreed on having a prefix, and lpm_ was suggested as meaning Linux perf metric. All of those words seem redundant given the metric is being measured on Linux using perf, but it was adopted as the best suggestion. If you have a better proposal, I think we can change it.

Thanks,
Ian

Quoted 177 lines
Quoted 122 lines

There is one CMEM latency PMU per socket and perf sums the counts of the PMUs in an aggregation, e.g. of both sockets in the default aggregation mode. Use aggr_nr() to divide the summed cycles by the number of PMUs, otherwise the frequency is too high and the latency too low by that factor.

The events are sysfs events of the nvidia_ucf_pmu_<socket-id> and nvidia_cmem_latency_pmu_<socket-id> PMUs rather than json events, so add
the "nvidia" PMU prefix to the prefixes that skip the named json event check in metric.py.

pmu-events/Build only runs arm64_metrics.py for the "arm" vendor models,
so also run it for the "nvidia" vendor models to generate the Tegra410 metrics.

Assisted-by Gemini:gemini-3.1-pro-preview
---
 tools/perf/pmu-events/Build            |  2 +-
 tools/perf/pmu-events/arm64_metrics.py | 66
+++++++++++++++++++++++++-
 tools/perf/pmu-events/metric.py        |  2 +-
 3 files changed, 66 insertions(+), 4 deletions(-)
tools/perf/pmu-events/Build +1 -1
index 1ac067be9d34..8dc8cfcb8b04 100644
@@ -78,7 +78,7 @@ endif
 
 ifeq ($(JEVENTS_ARCH),$(filter $(JEVENTS_ARCH),arm64 all))
 # Generate ARM Json
-ARMS := $(shell ls -d pmu-events/arch/arm64/arm/*|grep -v cmn)
+ARMS := $(shell ls -d pmu-events/arch/arm64/arm/* pmu-
 events/arch/arm64/nvidia/*|grep -v cmn)
 ARM_METRICS = $(foreach x,$(ARMS),$(OUTPUT)$(x)/extra-metrics.json)
 ARM_METRICGROUPS = $(foreach x,$(ARMS),$(OUTPUT)$(x)/extra-

metricgroups.json)
GEN_JSON += $(ARM_METRICS) $(ARM_METRICGROUPS)

tools/perf/pmu- renamed from tools/perf/pmu-events/arm64_metrics.py +0 -0

events/arm64_metrics.py
index 4ecda96d11fa..b0d0651fec12 100755

tools/perf/pmu-events/arm64_metrics.py +7 -2
@@ -2,14 +2,75 @@
 # SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
 import argparse
 import os
-from metric import (JsonEncodeMetric,
 JsonEncodeMetricGroupDescriptions, LoadEvents,
-                    MetricGroup)
+from typing import Optional
+from metric import (aggr_nr, d_ratio, Event, JsonEncodeMetric,
+                    JsonEncodeMetricGroupDescriptions, LoadEvents, Metric,
 MetricGroup)
 from common_metrics import Cycles
 
 # Global command line arguments.
 _args = None
 
 
+def NvidiaT410Uncore() -> Optional[MetricGroup]:
+    """Uncore metrics for the NVIDIA Tegra410 SoC.
+
+    The formulas follow Documentation/admin-guide/perf/nvidia-tegra410-

pmu.rst.

tools/perf/pmu-events/arm64_metrics.py +11 -0
+    """
+    assert _args is not None
+    if _args.vendor != "nvidia" or _args.model != "t410":
+        return None
+
+    interval_sec = Event("duration_time")
+    mb_per_sec = f"{1/1e6}MB/s"
+
+    def DdrBw() -> MetricGroup:
+        # Unified Coherence Fabric (UCF) PMU:
+        # AVG_MEM_{READ,WRITE}_BANDWIDTH = MEM_BYTES_{RD,WR} /

ELAPSED_TIME

tools/perf/pmu-events/arm64_metrics.py +4 -0
+        loc_src = "src_loc_cpu=0x1,src_loc_noncpu=0x1"
+        rd = Event("nvidia_ucf_pmu/mem_bytes_rd,dst_loc_cmem=0x1/")
+        wr = Event("nvidia_ucf_pmu/mem_bytes_wr,dst_loc_cmem=0x1/")
+        loc_rd =

Event(f"nvidia_ucf_pmu/mem_bytes_rd,{loc_src},dst_loc_cmem=0x1/")
+ loc_wr =
Event(f"nvidia_ucf_pmu/mem_bytes_wr,{loc_src},dst_loc_cmem=0x1/")
+ rem_rd =
Event(f"nvidia_ucf_pmu/mem_bytes_rd,{loc_src},dst_rem=0x1/")
+ rem_wr =
Event(f"nvidia_ucf_pmu/mem_bytes_wr,{loc_src},dst_rem=0x1/")

tools/perf/pmu-events/arm64_metrics.py +20 -0
+        return MetricGroup("lpm_ddr_bw", [
+            Metric("lpm_ddr_bw", "Total DDR bandwidth",
+                   d_ratio(rd + wr, interval_sec), mb_per_sec),
+            Metric("lpm_ddr_rd_bw", "DDR read bandwidth",
+                   d_ratio(rd, interval_sec), mb_per_sec),
+            Metric("lpm_ddr_wr_bw", "DDR write bandwidth",
+                   d_ratio(wr, interval_sec), mb_per_sec),
+            Metric("lpm_ddr_loc_rd_bw", "Local DDR read bandwidth",
+                   d_ratio(loc_rd, interval_sec), mb_per_sec),
+            Metric("lpm_ddr_loc_wr_bw", "Local DDR write bandwidth",
+                   d_ratio(loc_wr, interval_sec), mb_per_sec),
+            Metric("lpm_ddr_rem_rd_bw", "Remote DDR read bandwidth",
+                   d_ratio(rem_rd, interval_sec), mb_per_sec),
+            Metric("lpm_ddr_rem_wr_bw", "Remote DDR write bandwidth",
+                   d_ratio(rem_wr, interval_sec), mb_per_sec),
+        ], description="NVIDIA Tegra410 DDR bandwidth")
+
+    def DdrLat() -> MetricGroup:
+        # CPU Memory (CMEM) latency PMU, one per socket:
+        # AVERAGE_LATENCY_IN_NS = (RD_CUM_OUTS / RD_REQ) / (CYCLES /

ELAPSED_TIME_IN_NS)

tools/perf/pmu-events/arm64_metrics.py +4 -0
+        rd_cum_outs = Event("nvidia_cmem_latency_pmu/rd_cum_outs/")
+        rd_req = Event("nvidia_cmem_latency_pmu/rd_req/")
+        cycles = Event("nvidia_cmem_latency_pmu/cycles/")
+        # perf sums the counts of the PMUs in an aggregation, e.g. both

sockets

tools/perf/pmu-events/arm64_metrics.py +2 -0
+        # by default, so divide cycles by their number to get the frequency.
+        rd_lat = d_ratio(1e9 * interval_sec * rd_cum_outs * aggr_nr(cycles),

rd_req * cycles)

tools/perf/pmu-events/arm64_metrics.py +2 -0
+        return MetricGroup("lpm_ddr_lat", [
+            Metric("lpm_ddr_lat", "Average DDR read latency in ns", rd_lat,
Quoted 2 lines

"1ns"),
+ Metric("lpm_ddr_rd_lat", "DDR read latency in ns", rd_lat, "1ns"),

Could we keep just lpm_ddr_rd_lat ?

tools/perf/pmu-events/arm64_metrics.py line 68

Quoted 5 lines

+ # The CMEM latency PMU only measures reads, write latency is
always 0.

tools/perf/pmu-events/arm64_metrics.py +3 -0
+            Metric("lpm_ddr_wr_lat", "DDR write latency in ns",
+                   rd_cum_outs - rd_cum_outs, "1ns"),
+        ], description="NVIDIA Tegra410 DDR latency")

This always prints 0.0 ns, which reads like a real measurement of zero-latency writes. Since the hardware can't measure write latency, I'd drop this metric rather than report a placeholder.

Thanks,
Besar

Quoted 34 lines
tools/perf/pmu-events/arm64_metrics.py +6 -0
+
+    return MetricGroup("lpm_uncore", [DdrBw(), DdrLat()],
+                       description="NVIDIA Tegra410 uncore metrics")
+
+
 def main() -> None:
     global _args
@@ -37,6 +98,7 @@ def main() -> None:
 
     all_metrics = MetricGroup("", [
         Cycles(),
+        NvidiaT410Uncore(),
     ])
 
     if _args.metricgroups:
tools/perf/pmu- renamed from tools/perf/pmu-events/metric.py +0 -0

events/metric.py
index f5d81eccd5c3..6fb69024d1fd 100644

tools/perf/pmu-events/metric.py +0 -1
@@ -91,7 +91,7 @@ def CheckEveryEvent(*names: str) -> None:
       name = name[:name.find(':')]
     elif '/' in name:
       name = name[:name.find('/')]
-      if any([name.startswith(x) for x in ['amd', 'arm', 'cpu', 'msr', 'power',

'cha', 'uncore']]):
+ if any([name.startswith(x) for x in ['amd', 'arm', 'cpu', 'msr', 'power',
'cha', 'uncore', 'nvidia']]):
continue
if name not in all_events_all_models:
raise ValueError(f"Is {name} a named json event?")
--
2.56.0.rc1.315.gc6ed9934b7-goog

#5 lore raw

RE: [PATCH 1/2] perf jevents: Add NVIDIA Tegra410 uncore DDR metrics

Quoted 86 lines

-----Original Message-----
From: Ian Rogers <[email protected]>
Sent: Tuesday, October 13, 2026 9:30 PM
To: Besar Wicaksono <[email protected]>
Cc: Chun-Tse Shao <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; [email protected]; Matt Ochs <[email protected]>; Sean Kelley <[email protected]>; Rich Wiley <[email protected]>; Vikram Sethi <[email protected]>; Raj Joshi <[email protected]>
Subject: Re: [PATCH 1/2] perf jevents: Add NVIDIA Tegra410 uncore DDR metrics

External email: Use caution opening links or attachments

Quoted 14 lines · Besar Wicaksono

Hi Chun-Tse,

I tested this patch on 2-socket Tegra410. Both DDR bandwidth and read latency metrics give correct values in the default, --per-socket, and -A modes.

Some small comments inline below.

Quoted 5 lines

-----Original Message-----
From: Chun-Tse Shao <[email protected]>
Sent: Monday, September 28, 2026 5:51 PM
To: [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected];

[email protected];

Quoted 11 lines
Quoted 11 lines

[email protected]; [email protected]; [email protected]; [email protected]; Besar Wicaksono <[email protected]>; [email protected]; linux- [email protected]; Chun-Tse Shao <[email protected]>
Subject: [PATCH 1/2] perf jevents: Add NVIDIA Tegra410 uncore DDR metrics

External email: Use caution opening links or attachments

Add DDR bandwidth and latency metrics for the NVIDIA Tegra410 SoC.

The

Quoted 8 lines
Quoted 8 lines

formulas follow Documentation/admin-guide/perf/nvidia-tegra410- pmu.rst:

AVG_MEM_READ_BANDWIDTH_IN_GBPS = MEM_BYTES_RD /
ELAPSED_TIME_IN_NS
AVG_MEM_WRITE_BANDWIDTH_IN_GBPS = MEM_BYTES_WR /
ELAPSED_TIME_IN_NS

Quoted 5 lines
Quoted 5 lines

FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS AVG_LATENCY_IN_CYCLES = RD_CUM_OUTS / RD_REQ AVERAGE_LATENCY_IN_NS = AVG_LATENCY_IN_CYCLES / FREQ_IN_GHZ

The bandwidth metrics use the mem_bytes_rd/mem_bytes_wr events

of

Quoted 4 lines
Quoted 4 lines

the
Unified Coherence Fabric (UCF) PMU with source/destination filters:

lpm_ddr_bw, lpm_ddr_rd_bw, lpm_ddr_wr_bw: all sources to local

CMEM.

Quoted 1 line
Quoted 1 line

lpm_ddr_loc_rd_bw, lpm_ddr_loc_wr_bw: local CPU and non-CPU

sources

Quoted 8 lines
Quoted 8 lines

to
local CMEM.
lpm_ddr_rem_rd_bw, lpm_ddr_rem_wr_bw: local CPU and non-CPU
sources to
remote memory.

The latency metrics use the CPU Memory (CMEM) latency PMU:

Quoted 7 lines
Quoted 3 lines

lpm_ddr_lat, lpm_ddr_rd_lat: average DDR read latency in ns.
lpm_ddr_wr_lat: the PMU only measures reads, so this is always 0.

A question on naming: what does the "lpm_" prefix stand for? Is it the convention for all metrics generated by the Python scripts (I see it on the Intel/AMD and common metrics too)?

Quoted 9 lines

Hi Besar, naming is one of the hardest problems in Computer Science :-) When we first added the metrics there was a request to differentiate between metrics coming from the python scripts and metrics coming from vendor code, like Intel top-down. We agreed on having a prefix, and lpm_ was suggested as meaning Linux perf metric. All of those words seem redundant given the metric is being measured on Linux using perf, but it was adopted as the best suggestion. If you have a better proposal, I think we can change it.

Hi Ian,

Thanks for the explanation, this makes sense, and it keeps them distinguishable from the vendor JSON metrics.

Regards,
Besar

Quoted 189 lines

Thanks,
Ian

Quoted 1 line
Quoted 1 line

There is one CMEM latency PMU per socket and perf sums the counts of

the

Quoted 2 lines
Quoted 2 lines

PMUs in an aggregation, e.g. of both sockets in the default aggregation mode. Use aggr_nr() to divide the summed cycles by the number of

PMUs,

Quoted 5 lines
Quoted 5 lines

otherwise the frequency is too high and the latency too low by that factor.

The events are sysfs events of the nvidia_ucf_pmu_<socket-id> and nvidia_cmem_latency_pmu_<socket-id> PMUs rather than json events,

so

Quoted 24 lines
Quoted 24 lines

add
the "nvidia" PMU prefix to the prefixes that skip the named json event check in metric.py.

pmu-events/Build only runs arm64_metrics.py for the "arm" vendor models,
so also run it for the "nvidia" vendor models to generate the Tegra410 metrics.

Assisted-by Gemini:gemini-3.1-pro-preview
---
 tools/perf/pmu-events/Build            |  2 +-
 tools/perf/pmu-events/arm64_metrics.py | 66
+++++++++++++++++++++++++-
 tools/perf/pmu-events/metric.py        |  2 +-
 3 files changed, 66 insertions(+), 4 deletions(-)
tools/perf/pmu-events/Build +0 -0
index 1ac067be9d34..8dc8cfcb8b04 100644
@@ -78,7 +78,7 @@ endif
 
Quoted 6 lines

tools/perf/pmu-events/Build endif

Quoted 6 lines
tools/perf/pmu-events/Build +1 -1
 ifeq ($(JEVENTS_ARCH),$(filter $(JEVENTS_ARCH),arm64 all))
 # Generate ARM Json
-ARMS := $(shell ls -d pmu-events/arch/arm64/arm/*|grep -v cmn)
+ARMS := $(shell ls -d pmu-events/arch/arm64/arm/* pmu-
 events/arch/arm64/nvidia/*|grep -v cmn)
 ARM_METRICS = $(foreach x,$(ARMS),$(OUTPUT)$(x)/extra-

metrics.json)

Quoted 21 lines
Quoted 21 lines
tools/perf/pmu-events/Build +0 -0
 ARM_METRICGROUPS = $(foreach x,$(ARMS),$(OUTPUT)$(x)/extra-

metricgroups.json)
GEN_JSON += $(ARM_METRICS) $(ARM_METRICGROUPS)

tools/perf/pmu- renamed from tools/perf/pmu-events/arm64_metrics.py +0 -0

events/arm64_metrics.py
index 4ecda96d11fa..b0d0651fec12 100755

tools/perf/pmu-events/arm64_metrics.py +3 -2
@@ -2,14 +2,75 @@
 # SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
 import argparse
 import os
-from metric import (JsonEncodeMetric,
 JsonEncodeMetricGroupDescriptions, LoadEvents,
-                    MetricGroup)
+from typing import Optional
+from metric import (aggr_nr, d_ratio, Event, JsonEncodeMetric,
+                    JsonEncodeMetricGroupDescriptions, LoadEvents, Metric,
 MetricGroup)
 from common_metrics import Cycles
 
Quoted 8 lines

tools/perf/pmu-events/arm64_metrics.py

Quoted 8 lines
tools/perf/pmu-events/arm64_metrics.py +4 -0
 # Global command line arguments.
 _args = None
 
 
+def NvidiaT410Uncore() -> Optional[MetricGroup]:
+    """Uncore metrics for the NVIDIA Tegra410 SoC.
+
+    The formulas follow Documentation/admin-guide/perf/nvidia-

tegra410-

Quoted 4 lines
Quoted 4 lines

pmu.rst.

tools/perf/pmu-events/arm64_metrics.py +3 -0
+    """
+    assert _args is not None
+    if _args.vendor != "nvidia" or _args.model != "t410":
Quoted 8 lines

tools/perf/pmu-events/arm64_metrics.py

Quoted 8 lines
tools/perf/pmu-events/arm64_metrics.py +8 -0
+        return None
+
+    interval_sec = Event("duration_time")
+    mb_per_sec = f"{1/1e6}MB/s"
+
+    def DdrBw() -> MetricGroup:
+        # Unified Coherence Fabric (UCF) PMU:
+        # AVG_MEM_{READ,WRITE}_BANDWIDTH = MEM_BYTES_{RD,WR}

/

Quoted 4 lines

tools/perf/pmu-events/arm64_metrics.py

Quoted 4 lines

ELAPSED_TIME

tools/perf/pmu-events/arm64_metrics.py +3 -0
+        loc_src = "src_loc_cpu=0x1,src_loc_noncpu=0x1"
+        rd = Event("nvidia_ucf_pmu/mem_bytes_rd,dst_loc_cmem=0x1/")
+        wr =

Event("nvidia_ucf_pmu/mem_bytes_wr,dst_loc_cmem=0x1/")

Quoted 4 lines
Quoted 4 lines

+ loc_rd =
Event(f"nvidia_ucf_pmu/mem_bytes_rd,{loc_src},dst_loc_cmem=0x1/")
+ loc_wr =

Event(f"nvidia_ucf_pmu/mem_bytes_wr,{loc_src},dst_loc_cmem=0x1/")

Quoted 16 lines
Quoted 16 lines

+ rem_rd =
Event(f"nvidia_ucf_pmu/mem_bytes_rd,{loc_src},dst_rem=0x1/")
+ rem_wr =
Event(f"nvidia_ucf_pmu/mem_bytes_wr,{loc_src},dst_rem=0x1/")

tools/perf/pmu-events/arm64_metrics.py +12 -0
+        return MetricGroup("lpm_ddr_bw", [
+            Metric("lpm_ddr_bw", "Total DDR bandwidth",
+                   d_ratio(rd + wr, interval_sec), mb_per_sec),
+            Metric("lpm_ddr_rd_bw", "DDR read bandwidth",
+                   d_ratio(rd, interval_sec), mb_per_sec),
+            Metric("lpm_ddr_wr_bw", "DDR write bandwidth",
+                   d_ratio(wr, interval_sec), mb_per_sec),
+            Metric("lpm_ddr_loc_rd_bw", "Local DDR read bandwidth",
+                   d_ratio(loc_rd, interval_sec), mb_per_sec),
+            Metric("lpm_ddr_loc_wr_bw", "Local DDR write bandwidth",
+                   d_ratio(loc_wr, interval_sec), mb_per_sec),
+            Metric("lpm_ddr_rem_rd_bw", "Remote DDR read bandwidth",
Quoted 8 lines

tools/perf/pmu-events/arm64_metrics.py

Quoted 8 lines
tools/perf/pmu-events/arm64_metrics.py +8 -0
+                   d_ratio(rem_rd, interval_sec), mb_per_sec),
+            Metric("lpm_ddr_rem_wr_bw", "Remote DDR write bandwidth",
+                   d_ratio(rem_wr, interval_sec), mb_per_sec),
+        ], description="NVIDIA Tegra410 DDR bandwidth")
+
+    def DdrLat() -> MetricGroup:
+        # CPU Memory (CMEM) latency PMU, one per socket:
+        # AVERAGE_LATENCY_IN_NS = (RD_CUM_OUTS / RD_REQ) /

(CYCLES /

Quoted 66 lines
Quoted 11 lines

ELAPSED_TIME_IN_NS)

tools/perf/pmu-events/arm64_metrics.py +4 -0
+        rd_cum_outs = Event("nvidia_cmem_latency_pmu/rd_cum_outs/")
+        rd_req = Event("nvidia_cmem_latency_pmu/rd_req/")
+        cycles = Event("nvidia_cmem_latency_pmu/cycles/")
+        # perf sums the counts of the PMUs in an aggregation, e.g. both

sockets

tools/perf/pmu-events/arm64_metrics.py +2 -0
+        # by default, so divide cycles by their number to get the frequency.
+        rd_lat = d_ratio(1e9 * interval_sec * rd_cum_outs * aggr_nr(cycles),

rd_req * cycles)

tools/perf/pmu-events/arm64_metrics.py +2 -0
+        return MetricGroup("lpm_ddr_lat", [
+            Metric("lpm_ddr_lat", "Average DDR read latency in ns", rd_lat,
Quoted 2 lines

"1ns"),
+ Metric("lpm_ddr_rd_lat", "DDR read latency in ns", rd_lat, "1ns"),

Could we keep just lpm_ddr_rd_lat ?

tools/perf/pmu-events/arm64_metrics.py line 68

Quoted 5 lines

+ # The CMEM latency PMU only measures reads, write latency is
always 0.

tools/perf/pmu-events/arm64_metrics.py +3 -0
+            Metric("lpm_ddr_wr_lat", "DDR write latency in ns",
+                   rd_cum_outs - rd_cum_outs, "1ns"),
+        ], description="NVIDIA Tegra410 DDR latency")

This always prints 0.0 ns, which reads like a real measurement of zero-latency writes. Since the hardware can't measure write latency, I'd drop this metric rather than report a placeholder.

Thanks,
Besar

Quoted 34 lines
tools/perf/pmu-events/arm64_metrics.py +6 -0
+
+    return MetricGroup("lpm_uncore", [DdrBw(), DdrLat()],
+                       description="NVIDIA Tegra410 uncore metrics")
+
+
 def main() -> None:
     global _args
@@ -37,6 +98,7 @@ def main() -> None:
 
     all_metrics = MetricGroup("", [
         Cycles(),
+        NvidiaT410Uncore(),
     ])
 
     if _args.metricgroups:
tools/perf/pmu- renamed from tools/perf/pmu-events/metric.py +0 -0

events/metric.py
index f5d81eccd5c3..6fb69024d1fd 100644

tools/perf/pmu-events/metric.py +0 -1
@@ -91,7 +91,7 @@ def CheckEveryEvent(*names: str) -> None:
       name = name[:name.find(':')]
     elif '/' in name:
       name = name[:name.find('/')]
-      if any([name.startswith(x) for x in ['amd', 'arm', 'cpu', 'msr', 'power',

'cha', 'uncore']]):
+ if any([name.startswith(x) for x in ['amd', 'arm', 'cpu', 'msr', 'power',
'cha', 'uncore', 'nvidia']]):
continue
if name not in all_events_all_models:
raise ValueError(f"Is {name} a named json event?")
--
2.56.0.rc1.315.gc6ed9934b7-goog

#6 lore raw

PATCH 2/2 perf jevents: Add NVIDIA Tegra410 uncore PCIe metrics

In reply to Chun-Tse Shao

Add PCIe bandwidth metrics for the NVIDIA Tegra410 SoC. The formulas follow Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst:

AVG_RD_BANDWIDTH_IN_GBPS = RD_BYTES / ELAPSED_TIME_IN_NS AVG_WR_BANDWIDTH_IN_GBPS = WR_BYTES / ELAPSED_TIME_IN_NS

There is one PCIE PMU per socket and PCIe Root Complex (RC), named nvidia_pcie_pmu_<socket-id>_rc_<pcie-rc-id>, with RCs 0 to 5 in each socket. The metrics sum the rd_bytes/wr_bytes events of both sockets:

lpm_pcie_bw, lpm_pcie_rd_bw, lpm_pcie_wr_bw: all RCs. lpm_pcie_bw_<rc>, lpm_pcie_rd_bw_<rc>, lpm_pcie_wr_bw_<rc>: one RC. lpm_pcie_loc_rd_bw_<rc>, lpm_pcie_loc_wr_bw_<rc>: one RC to local
CMEM, GMEM, PCIe peer and CXL memory.
lpm_pcie_rem_rd_bw_<rc>, lpm_pcie_rem_wr_bw_<rc>: one RC to remote
memory.

The per RC metrics are in the lpm_pcie_rc<rc>_bw metric groups.

A socket's event counts as 0 if its PMU isn't present (has_event() is 0), e.g. on a single socket system, or if it isn't counted in the aggregation (source_count() is 0), e.g. the other socket's event with --per-socket. Otherwise the metric fails to parse or is nan.

Assisted-by Gemini:gemini-3.1-pro-preview
---
 tools/perf/pmu-events/arm64_metrics.py | 68 ++++++++++++++++++++++++--
 1 file changed, 64 insertions(+), 4 deletions(-)
tools/perf/pmu-events/arm64_metrics.py +64 -4
index b0d0651fec12..1ca3d8ddb4b7 100755
@@ -2,9 +2,10 @@
 # SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
 import argparse
 import os
-from typing import Optional
-from metric import (aggr_nr, d_ratio, Event, JsonEncodeMetric,
-                    JsonEncodeMetricGroupDescriptions, LoadEvents, Metric, MetricGroup)
+from typing import List, Optional, Union
+from metric import (aggr_nr, d_ratio, has_event, source_count, Event, Expression,
+                    JsonEncodeMetric, JsonEncodeMetricGroupDescriptions, LoadEvents, Metric,
+                    MetricGroup, Select)
 from common_metrics import Cycles
 
 # Global command line arguments.
@@ -67,7 +68,66 @@ def NvidiaT410Uncore() -> Optional[MetricGroup]:
                    rd_cum_outs - rd_cum_outs, "1ns"),
         ], description="NVIDIA Tegra410 DDR latency")
 
-    return MetricGroup("lpm_uncore", [DdrBw(), DdrLat()],
+    def PcieBw() -> MetricGroup:
+        # PCIE PMU, one per socket and PCIe Root Complex (RC):
+        # AVG_{RD,WR}_BANDWIDTH = {RD,WR}_BYTES / ELAPSED_TIME
+        loc_dst = "dst_loc_cmem=0x1,dst_loc_gmem=0x1,dst_loc_pcie_p2p=0x1,dst_loc_pcie_cxl=0x1"
+        rem_dst = "dst_rem=0x1"
+
+        def SocketSum(rc: str, event: str) -> Expression:
+            # Sum the event over both sockets. A PMU counts as 0 if it isn't
+            # present (has_event() is 0) or isn't counted in the aggregation,
+            # e.g. the other socket with --per-socket (source_count() is 0).
+            def Count(e: Event) -> Expression:
+                return Select(Select(e, source_count(e), 0), has_event(e), 0)
+
+            return (Count(Event(f"nvidia_pcie_pmu_0_{rc}/{event}/")) +
+                    Count(Event(f"nvidia_pcie_pmu_1_{rc}/{event}/")))
+
+        # nvidia_pcie_pmu_<socket-id>_rc matches the PMUs of all RCs in the socket.
+        rd = SocketSum("rc", "rd_bytes")
+        wr = SocketSum("rc", "wr_bytes")
+        metrics: List[Union[Metric, MetricGroup]] = [
+            Metric("lpm_pcie_bw", "Total PCIe bandwidth across all Root Complexes",
+                   d_ratio(rd + wr, interval_sec), mb_per_sec),
+            Metric("lpm_pcie_rd_bw", "Total PCIe read bandwidth across all Root Complexes",
+                   d_ratio(rd, interval_sec), mb_per_sec),
+            Metric("lpm_pcie_wr_bw", "Total PCIe write bandwidth across all Root Complexes",
+                   d_ratio(wr, interval_sec), mb_per_sec),
+        ]
+        for i in range(6):
+            rc = f"rc_{i}"
+            rc_rd = SocketSum(rc, "rd_bytes")
+            rc_wr = SocketSum(rc, "wr_bytes")
+            loc_rd = SocketSum(rc, f"rd_bytes,{loc_dst}")
+            loc_wr = SocketSum(rc, f"wr_bytes,{loc_dst}")
+            rem_rd = SocketSum(rc, f"rd_bytes,{rem_dst}")
+            rem_wr = SocketSum(rc, f"wr_bytes,{rem_dst}")
+            metrics.append(MetricGroup(f"lpm_pcie_rc{i}_bw", [
+                Metric(f"lpm_pcie_bw_{i}", f"PCIe total bandwidth on Root Complex {i}",
+                       d_ratio(rc_rd + rc_wr, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_rd_bw_{i}", f"PCIe read bandwidth on Root Complex {i}",
+                       d_ratio(rc_rd, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_wr_bw_{i}", f"PCIe write bandwidth on Root Complex {i}",
+                       d_ratio(rc_wr, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_loc_rd_bw_{i}",
+                       f"PCIe local read bandwidth on Root Complex {i}",
+                       d_ratio(loc_rd, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_loc_wr_bw_{i}",
+                       f"PCIe local write bandwidth on Root Complex {i}",
+                       d_ratio(loc_wr, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_rem_rd_bw_{i}",
+                       f"PCIe remote read bandwidth on Root Complex {i}",
+                       d_ratio(rem_rd, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_rem_wr_bw_{i}",
+                       f"PCIe remote write bandwidth on Root Complex {i}",
+                       d_ratio(rem_wr, interval_sec), mb_per_sec),
+            ], description=f"PCIe bandwidth on Root Complex {i}"))
+
+        return MetricGroup("lpm_pcie_bw", metrics,
+                           description="NVIDIA Tegra410 PCIe Root Complex bandwidth")
+
+    return MetricGroup("lpm_uncore", [DdrBw(), DdrLat(), PcieBw()],
                        description="NVIDIA Tegra410 uncore metrics")
 
 
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#7 lore raw

Re: [PATCH 2/2] perf jevents: Add NVIDIA Tegra410 uncore PCIe metrics

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 17 lines

Add PCIe bandwidth metrics for the NVIDIA Tegra410 SoC. The formulas follow Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst:

AVG_RD_BANDWIDTH_IN_GBPS = RD_BYTES / ELAPSED_TIME_IN_NS AVG_WR_BANDWIDTH_IN_GBPS = WR_BYTES / ELAPSED_TIME_IN_NS

There is one PCIE PMU per socket and PCIe Root Complex (RC), named nvidia_pcie_pmu_<socket-id>_rc_<pcie-rc-id>, with RCs 0 to 5 in each socket. The metrics sum the rd_bytes/wr_bytes events of both sockets:

lpm_pcie_bw, lpm_pcie_rd_bw, lpm_pcie_wr_bw: all RCs. lpm_pcie_bw_<rc>, lpm_pcie_rd_bw_<rc>, lpm_pcie_wr_bw_<rc>: one RC. lpm_pcie_loc_rd_bw_<rc>, lpm_pcie_loc_wr_bw_<rc>: one RC to local
CMEM, GMEM, PCIe peer and CXL memory.
lpm_pcie_rem_rd_bw_<rc>, lpm_pcie_rem_wr_bw_<rc>: one RC to remote
memory.

Quoted 8 lines

The per RC metrics are in the lpm_pcie_rc<rc>_bw metric groups.

A socket's event counts as 0 if its PMU isn't present (has_event() is 0), e.g. on a single socket system, or if it isn't counted in the aggregation (source_count() is 0), e.g. the other socket's event with --per-socket. Otherwise the metric fails to parse or is nan.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#8 lore raw

RE: [PATCH 2/2] perf jevents: Add NVIDIA Tegra410 uncore PCIe metrics

In reply to Chun-Tse Shao

Hi Chun-Tse,

I tested this on 2-sockets Tegra410 and found some issues.
1. With --per-socket or -A, every lpm_pcie_* metric is only reported for socket 0, so socket 1's PCIe traffic never shows up. Example output:

perf stat -a -A -M lpm_pcie_rc1_bw -- sleep 1

Performance counter stats for 'system wide':

CPU0 15,204 nvidia_pcie_pmu_0_rc_1/wr_bytes/ # 0.0 MB/s lpm_pcie_wr_bw_1
CPU0 # 33.8 MB/s lpm_pcie_bw_1

CPU0           33,910,952      nvidia_pcie_pmu_0_rc_1/rd_bytes/ #     33.8 MB/s  lpm_pcie_rd_bw_1    
CPU88                   0      nvidia_pcie_pmu_1_rc_1/wr_bytes/                                      
CPU88                   0      nvidia_pcie_pmu_1_rc_1/rd_bytes/                                      
CPU0        1,002,286,615      tool/duration_time/                                                   
CPU0                    0      nvidia_pcie_pmu_0_rc_1/wr_bytes,dst_rem=0x1/ #      0.0 MB/s  lpm_pcie_rem_wr_bw_1
CPU88                   0      nvidia_pcie_pmu_1_rc_1/wr_bytes,dst_rem=0x1/                                      
CPU0        1,002,275,059      tool/duration_time/                                                   
CPU0                    0      nvidia_pcie_pmu_0_rc_1/rd_bytes,dst_rem=0x1/ #      0.0 MB/s  lpm_pcie_rem_rd_bw_1
CPU88                   0      nvidia_pcie_pmu_1_rc_1/rd_bytes,dst_rem=0x1/ 

<snip>

Note CPU88 (socket 1_ has no lpm_pcie_* values.

2. In the default mode, lpm_pcie_*_<i> adds RC i of socket 0 and RC i of socket 1. These are separate root complexes with different devices, so the "Root Complex i" value can combine traffic from two unrelated links.
Could the metrics follow the hardware hierarchy instead, starting at the socket and drilling down to the RC, with the same destination split at each level?

    - per socket, all RCs (nvidia_pcie_pmu_<s>_rc):
        lpm_pcie_s<s>_{bw,rd_bw,wr_bw}
        lpm_pcie_s<s>_{loc,rem}_{rd,wr}_bw
    - per socket and RC (nvidia_pcie_pmu_<s>_rc_<i>):
        lpm_pcie_s<s>_{bw,rd_bw,wr_bw}_<i>
        lpm_pcie_s<s>_{loc,rem}_{rd,wr}_bw_<i>
      grouped as lpm_pcie_s<s>_rc<i>_bw.

Thanks,
Besar

Quoted 148 lines

-----Original Message-----
From: Chun-Tse Shao <[email protected]>
Sent: Monday, September 28, 2026 5:51 PM
To: [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Besar Wicaksono <[email protected]>; [email protected]; linux- [email protected]; Chun-Tse Shao <[email protected]>
Subject: [PATCH 2/2] perf jevents: Add NVIDIA Tegra410 uncore PCIe metrics

External email: Use caution opening links or attachments

Add PCIe bandwidth metrics for the NVIDIA Tegra410 SoC. The formulas follow Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst:

AVG_RD_BANDWIDTH_IN_GBPS = RD_BYTES / ELAPSED_TIME_IN_NS AVG_WR_BANDWIDTH_IN_GBPS = WR_BYTES / ELAPSED_TIME_IN_NS

There is one PCIE PMU per socket and PCIe Root Complex (RC), named nvidia_pcie_pmu_<socket-id>_rc_<pcie-rc-id>, with RCs 0 to 5 in each socket. The metrics sum the rd_bytes/wr_bytes events of both sockets:

lpm_pcie_bw, lpm_pcie_rd_bw, lpm_pcie_wr_bw: all RCs. lpm_pcie_bw_<rc>, lpm_pcie_rd_bw_<rc>, lpm_pcie_wr_bw_<rc>: one RC. lpm_pcie_loc_rd_bw_<rc>, lpm_pcie_loc_wr_bw_<rc>: one RC to local
CMEM, GMEM, PCIe peer and CXL memory.
lpm_pcie_rem_rd_bw_<rc>, lpm_pcie_rem_wr_bw_<rc>: one RC to remote
memory.

The per RC metrics are in the lpm_pcie_rc<rc>_bw metric groups.

A socket's event counts as 0 if its PMU isn't present (has_event() is 0), e.g. on a single socket system, or if it isn't counted in the aggregation (source_count() is 0), e.g. the other socket's event with --per-socket. Otherwise the metric fails to parse or is nan.

Assisted-by Gemini:gemini-3.1-pro-preview
---
 tools/perf/pmu-events/arm64_metrics.py | 68
++++++++++++++++++++++++--
 1 file changed, 64 insertions(+), 4 deletions(-)
tools/perf/pmu- renamed from tools/perf/pmu-events/arm64_metrics.py +0 -0

events/arm64_metrics.py
index b0d0651fec12..1ca3d8ddb4b7 100755

tools/perf/pmu-events/arm64_metrics.py +4 -3
@@ -2,9 +2,10 @@
 # SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
 import argparse
 import os
-from typing import Optional
-from metric import (aggr_nr, d_ratio, Event, JsonEncodeMetric,
-                    JsonEncodeMetricGroupDescriptions, LoadEvents, Metric,
 MetricGroup)
+from typing import List, Optional, Union
+from metric import (aggr_nr, d_ratio, has_event, source_count, Event,
 Expression,
+                    JsonEncodeMetric, JsonEncodeMetricGroupDescriptions,
 LoadEvents, Metric,
+                    MetricGroup, Select)

from common_metrics import Cycles

# Global command line arguments.

tools/perf/pmu-events/arm64_metrics.py +4 -1
@@ -67,7 +68,66 @@ def NvidiaT410Uncore() -> Optional[MetricGroup]:
                    rd_cum_outs - rd_cum_outs, "1ns"),
         ], description="NVIDIA Tegra410 DDR latency")
 
-    return MetricGroup("lpm_uncore", [DdrBw(), DdrLat()],
+    def PcieBw() -> MetricGroup:
+        # PCIE PMU, one per socket and PCIe Root Complex (RC):
+        # AVG_{RD,WR}_BANDWIDTH = {RD,WR}_BYTES / ELAPSED_TIME
+        loc_dst =

"dst_loc_cmem=0x1,dst_loc_gmem=0x1,dst_loc_pcie_p2p=0x1,dst_loc_pci e_cxl=0x1"

tools/perf/pmu-events/arm64_metrics.py +13 -0
+        rem_dst = "dst_rem=0x1"
+
+        def SocketSum(rc: str, event: str) -> Expression:
+            # Sum the event over both sockets. A PMU counts as 0 if it isn't
+            # present (has_event() is 0) or isn't counted in the aggregation,
+            # e.g. the other socket with --per-socket (source_count() is 0).
+            def Count(e: Event) -> Expression:
+                return Select(Select(e, source_count(e), 0), has_event(e), 0)
+
+            return (Count(Event(f"nvidia_pcie_pmu_0_{rc}/{event}/")) +
+                    Count(Event(f"nvidia_pcie_pmu_1_{rc}/{event}/")))
+
+        # nvidia_pcie_pmu_<socket-id>_rc matches the PMUs of all RCs in the

socket.

tools/perf/pmu-events/arm64_metrics.py +4 -0
+        rd = SocketSum("rc", "rd_bytes")
+        wr = SocketSum("rc", "wr_bytes")
+        metrics: List[Union[Metric, MetricGroup]] = [
+            Metric("lpm_pcie_bw", "Total PCIe bandwidth across all Root

Complexes",

tools/perf/pmu-events/arm64_metrics.py +2 -0
+                   d_ratio(rd + wr, interval_sec), mb_per_sec),
+            Metric("lpm_pcie_rd_bw", "Total PCIe read bandwidth across all

Root Complexes",

tools/perf/pmu-events/arm64_metrics.py +2 -0
+                   d_ratio(rd, interval_sec), mb_per_sec),
+            Metric("lpm_pcie_wr_bw", "Total PCIe write bandwidth across all

Root Complexes",

tools/perf/pmu-events/arm64_metrics.py +12 -0
+                   d_ratio(wr, interval_sec), mb_per_sec),
+        ]
+        for i in range(6):
+            rc = f"rc_{i}"
+            rc_rd = SocketSum(rc, "rd_bytes")
+            rc_wr = SocketSum(rc, "wr_bytes")
+            loc_rd = SocketSum(rc, f"rd_bytes,{loc_dst}")
+            loc_wr = SocketSum(rc, f"wr_bytes,{loc_dst}")
+            rem_rd = SocketSum(rc, f"rd_bytes,{rem_dst}")
+            rem_wr = SocketSum(rc, f"wr_bytes,{rem_dst}")
+            metrics.append(MetricGroup(f"lpm_pcie_rc{i}_bw", [
+                Metric(f"lpm_pcie_bw_{i}", f"PCIe total bandwidth on Root

Complex {i}",

tools/perf/pmu-events/arm64_metrics.py +2 -0
+                       d_ratio(rc_rd + rc_wr, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_rd_bw_{i}", f"PCIe read bandwidth on Root

Complex {i}",

tools/perf/pmu-events/arm64_metrics.py +2 -0
+                       d_ratio(rc_rd, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_wr_bw_{i}", f"PCIe write bandwidth on Root

Complex {i}",

tools/perf/pmu-events/arm64_metrics.py +17 -0
+                       d_ratio(rc_wr, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_loc_rd_bw_{i}",
+                       f"PCIe local read bandwidth on Root Complex {i}",
+                       d_ratio(loc_rd, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_loc_wr_bw_{i}",
+                       f"PCIe local write bandwidth on Root Complex {i}",
+                       d_ratio(loc_wr, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_rem_rd_bw_{i}",
+                       f"PCIe remote read bandwidth on Root Complex {i}",
+                       d_ratio(rem_rd, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_rem_wr_bw_{i}",
+                       f"PCIe remote write bandwidth on Root Complex {i}",
+                       d_ratio(rem_wr, interval_sec), mb_per_sec),
+            ], description=f"PCIe bandwidth on Root Complex {i}"))
+
+        return MetricGroup("lpm_pcie_bw", metrics,
+                           description="NVIDIA Tegra410 PCIe Root Complex

bandwidth")

tools/perf/pmu-events/arm64_metrics.py +2 -1
+
+    return MetricGroup("lpm_uncore", [DdrBw(), DdrLat(), PcieBw()],
                        description="NVIDIA Tegra410 uncore metrics")
 
 
--

2.56.0.rc1.315.gc6ed9934b7-goog

#9 lore raw

Re: [PATCH 2/2] perf jevents: Add NVIDIA Tegra410 uncore PCIe metrics

Quoted 33 lines · Besar Wicaksono

Hi Chun-Tse,

I tested this on 2-sockets Tegra410 and found some issues.
1. With --per-socket or -A, every lpm_pcie_* metric is only reported for socket 0, so socket 1's PCIe traffic never shows up. Example output:

perf stat -a -A -M lpm_pcie_rc1_bw -- sleep 1

Performance counter stats for 'system wide':

CPU0 15,204 nvidia_pcie_pmu_0_rc_1/wr_bytes/ # 0.0 MB/s lpm_pcie_wr_bw_1
CPU0 # 33.8 MB/s lpm_pcie_bw_1

CPU0           33,910,952      nvidia_pcie_pmu_0_rc_1/rd_bytes/ #     33.8 MB/s  lpm_pcie_rd_bw_1
CPU88                   0      nvidia_pcie_pmu_1_rc_1/wr_bytes/
CPU88                   0      nvidia_pcie_pmu_1_rc_1/rd_bytes/
CPU0        1,002,286,615      tool/duration_time/
CPU0                    0      nvidia_pcie_pmu_0_rc_1/wr_bytes,dst_rem=0x1/ #      0.0 MB/s  lpm_pcie_rem_wr_bw_1
CPU88                   0      nvidia_pcie_pmu_1_rc_1/wr_bytes,dst_rem=0x1/
CPU0        1,002,275,059      tool/duration_time/
CPU0                    0      nvidia_pcie_pmu_0_rc_1/rd_bytes,dst_rem=0x1/ #      0.0 MB/s  lpm_pcie_rem_rd_bw_1
CPU88                   0      nvidia_pcie_pmu_1_rc_1/rd_bytes,dst_rem=0x1/

<snip>

Note CPU88 (socket 1_ has no lpm_pcie_* values.

2. In the default mode, lpm_pcie_*_<i> adds RC i of socket 0 and RC i of socket 1. These are separate root complexes with different devices, so the "Root Complex i" value can combine traffic from two unrelated links.
Could the metrics follow the hardware hierarchy instead, starting at the socket and drilling down to the RC, with the same destination split at each level?

Quoted 7 lines
    - per socket, all RCs (nvidia_pcie_pmu_<s>_rc):
        lpm_pcie_s<s>_{bw,rd_bw,wr_bw}
        lpm_pcie_s<s>_{loc,rem}_{rd,wr}_bw
    - per socket and RC (nvidia_pcie_pmu_<s>_rc_<i>):
        lpm_pcie_s<s>_{bw,rd_bw,wr_bw}_<i>
        lpm_pcie_s<s>_{loc,rem}_{rd,wr}_bw_<i>
      grouped as lpm_pcie_s<s>_rc<i>_bw.

Thanks for testing and for the suggestion. I think both problems come from how perf expects a PMU to expose its cpumask. Ideally perf wants a single PMU for both sockets, with one CPU per socket in its cpumask, e.g.:

```
/sys/bus/event_source/devices/nvidia_pcie_rc_0/cpumask: 0,88
```

We can work around the one-PMU-per-socket layout in the metrics, but would a single PMU per RC be a viable plan for you to implement?

Thanks,
CT

Quoted 153 lines

Thanks,
Besar

Quoted 148 lines

-----Original Message-----
From: Chun-Tse Shao <[email protected]>
Sent: Monday, September 28, 2026 5:51 PM
To: [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Besar Wicaksono <[email protected]>; [email protected]; linux- [email protected]; Chun-Tse Shao <[email protected]>
Subject: [PATCH 2/2] perf jevents: Add NVIDIA Tegra410 uncore PCIe metrics

External email: Use caution opening links or attachments

Add PCIe bandwidth metrics for the NVIDIA Tegra410 SoC. The formulas follow Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst:

AVG_RD_BANDWIDTH_IN_GBPS = RD_BYTES / ELAPSED_TIME_IN_NS AVG_WR_BANDWIDTH_IN_GBPS = WR_BYTES / ELAPSED_TIME_IN_NS

There is one PCIE PMU per socket and PCIe Root Complex (RC), named nvidia_pcie_pmu_<socket-id>_rc_<pcie-rc-id>, with RCs 0 to 5 in each socket. The metrics sum the rd_bytes/wr_bytes events of both sockets:

lpm_pcie_bw, lpm_pcie_rd_bw, lpm_pcie_wr_bw: all RCs. lpm_pcie_bw_<rc>, lpm_pcie_rd_bw_<rc>, lpm_pcie_wr_bw_<rc>: one RC. lpm_pcie_loc_rd_bw_<rc>, lpm_pcie_loc_wr_bw_<rc>: one RC to local
CMEM, GMEM, PCIe peer and CXL memory.
lpm_pcie_rem_rd_bw_<rc>, lpm_pcie_rem_wr_bw_<rc>: one RC to remote
memory.

The per RC metrics are in the lpm_pcie_rc<rc>_bw metric groups.

A socket's event counts as 0 if its PMU isn't present (has_event() is 0), e.g. on a single socket system, or if it isn't counted in the aggregation (source_count() is 0), e.g. the other socket's event with --per-socket. Otherwise the metric fails to parse or is nan.

Assisted-by Gemini:gemini-3.1-pro-preview
---
 tools/perf/pmu-events/arm64_metrics.py | 68
++++++++++++++++++++++++--
 1 file changed, 64 insertions(+), 4 deletions(-)
tools/perf/pmu- renamed from tools/perf/pmu-events/arm64_metrics.py +0 -0

events/arm64_metrics.py
index b0d0651fec12..1ca3d8ddb4b7 100755

tools/perf/pmu-events/arm64_metrics.py +4 -3
@@ -2,9 +2,10 @@
 # SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
 import argparse
 import os
-from typing import Optional
-from metric import (aggr_nr, d_ratio, Event, JsonEncodeMetric,
-                    JsonEncodeMetricGroupDescriptions, LoadEvents, Metric,
 MetricGroup)
+from typing import List, Optional, Union
+from metric import (aggr_nr, d_ratio, has_event, source_count, Event,
 Expression,
+                    JsonEncodeMetric, JsonEncodeMetricGroupDescriptions,
 LoadEvents, Metric,
+                    MetricGroup, Select)

from common_metrics import Cycles

# Global command line arguments.

tools/perf/pmu-events/arm64_metrics.py +4 -1
@@ -67,7 +68,66 @@ def NvidiaT410Uncore() -> Optional[MetricGroup]:
                    rd_cum_outs - rd_cum_outs, "1ns"),
         ], description="NVIDIA Tegra410 DDR latency")
 
-    return MetricGroup("lpm_uncore", [DdrBw(), DdrLat()],
+    def PcieBw() -> MetricGroup:
+        # PCIE PMU, one per socket and PCIe Root Complex (RC):
+        # AVG_{RD,WR}_BANDWIDTH = {RD,WR}_BYTES / ELAPSED_TIME
+        loc_dst =

"dst_loc_cmem=0x1,dst_loc_gmem=0x1,dst_loc_pcie_p2p=0x1,dst_loc_pci e_cxl=0x1"

tools/perf/pmu-events/arm64_metrics.py +13 -0
+        rem_dst = "dst_rem=0x1"
+
+        def SocketSum(rc: str, event: str) -> Expression:
+            # Sum the event over both sockets. A PMU counts as 0 if it isn't
+            # present (has_event() is 0) or isn't counted in the aggregation,
+            # e.g. the other socket with --per-socket (source_count() is 0).
+            def Count(e: Event) -> Expression:
+                return Select(Select(e, source_count(e), 0), has_event(e), 0)
+
+            return (Count(Event(f"nvidia_pcie_pmu_0_{rc}/{event}/")) +
+                    Count(Event(f"nvidia_pcie_pmu_1_{rc}/{event}/")))
+
+        # nvidia_pcie_pmu_<socket-id>_rc matches the PMUs of all RCs in the

socket.

tools/perf/pmu-events/arm64_metrics.py +4 -0
+        rd = SocketSum("rc", "rd_bytes")
+        wr = SocketSum("rc", "wr_bytes")
+        metrics: List[Union[Metric, MetricGroup]] = [
+            Metric("lpm_pcie_bw", "Total PCIe bandwidth across all Root

Complexes",

tools/perf/pmu-events/arm64_metrics.py +2 -0
+                   d_ratio(rd + wr, interval_sec), mb_per_sec),
+            Metric("lpm_pcie_rd_bw", "Total PCIe read bandwidth across all

Root Complexes",

tools/perf/pmu-events/arm64_metrics.py +2 -0
+                   d_ratio(rd, interval_sec), mb_per_sec),
+            Metric("lpm_pcie_wr_bw", "Total PCIe write bandwidth across all

Root Complexes",

tools/perf/pmu-events/arm64_metrics.py +12 -0
+                   d_ratio(wr, interval_sec), mb_per_sec),
+        ]
+        for i in range(6):
+            rc = f"rc_{i}"
+            rc_rd = SocketSum(rc, "rd_bytes")
+            rc_wr = SocketSum(rc, "wr_bytes")
+            loc_rd = SocketSum(rc, f"rd_bytes,{loc_dst}")
+            loc_wr = SocketSum(rc, f"wr_bytes,{loc_dst}")
+            rem_rd = SocketSum(rc, f"rd_bytes,{rem_dst}")
+            rem_wr = SocketSum(rc, f"wr_bytes,{rem_dst}")
+            metrics.append(MetricGroup(f"lpm_pcie_rc{i}_bw", [
+                Metric(f"lpm_pcie_bw_{i}", f"PCIe total bandwidth on Root

Complex {i}",

tools/perf/pmu-events/arm64_metrics.py +2 -0
+                       d_ratio(rc_rd + rc_wr, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_rd_bw_{i}", f"PCIe read bandwidth on Root

Complex {i}",

tools/perf/pmu-events/arm64_metrics.py +2 -0
+                       d_ratio(rc_rd, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_wr_bw_{i}", f"PCIe write bandwidth on Root

Complex {i}",

tools/perf/pmu-events/arm64_metrics.py +17 -0
+                       d_ratio(rc_wr, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_loc_rd_bw_{i}",
+                       f"PCIe local read bandwidth on Root Complex {i}",
+                       d_ratio(loc_rd, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_loc_wr_bw_{i}",
+                       f"PCIe local write bandwidth on Root Complex {i}",
+                       d_ratio(loc_wr, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_rem_rd_bw_{i}",
+                       f"PCIe remote read bandwidth on Root Complex {i}",
+                       d_ratio(rem_rd, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_rem_wr_bw_{i}",
+                       f"PCIe remote write bandwidth on Root Complex {i}",
+                       d_ratio(rem_wr, interval_sec), mb_per_sec),
+            ], description=f"PCIe bandwidth on Root Complex {i}"))
+
+        return MetricGroup("lpm_pcie_bw", metrics,
+                           description="NVIDIA Tegra410 PCIe Root Complex

bandwidth")

tools/perf/pmu-events/arm64_metrics.py +2 -1
+
+    return MetricGroup("lpm_uncore", [DdrBw(), DdrLat(), PcieBw()],
                        description="NVIDIA Tegra410 uncore metrics")
 
 
--

2.56.0.rc1.315.gc6ed9934b7-goog

#10 lore raw

RE: [PATCH 2/2] perf jevents: Add NVIDIA Tegra410 uncore PCIe metrics

Quoted 67 lines

-----Original Message-----
From: Chun-Tse Shao <[email protected]>
Sent: Friday, October 16, 2026 6:40 PM
To: Besar Wicaksono <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux-perf- [email protected]; [email protected]; Matt Ochs <[email protected]>; Sean Kelley <[email protected]>; Rich Wiley <[email protected]>; Vikram Sethi <[email protected]>; Raj Joshi <[email protected]>
Subject: Re: [PATCH 2/2] perf jevents: Add NVIDIA Tegra410 uncore PCIe metrics

External email: Use caution opening links or attachments

Quoted 7 lines · Besar Wicaksono

Hi Chun-Tse,

I tested this on 2-sockets Tegra410 and found some issues.
1. With --per-socket or -A, every lpm_pcie_* metric is only reported for socket 0, so socket 1's PCIe traffic never shows up. Example output:

Quoted 5 lines

perf stat -a -A -M lpm_pcie_rc1_bw -- sleep 1

Performance counter stats for 'system wide':

CPU0 15,204 nvidia_pcie_pmu_0_rc_1/wr_bytes/ # 0.0 MB/s

lpm_pcie_wr_bw_1

Quoted 2 lines

CPU0 # 33.8 MB/s lpm_pcie_bw_1
CPU0 33,910,952 nvidia_pcie_pmu_0_rc_1/rd_bytes/ # 33.8

MB/s lpm_pcie_rd_bw_1

Quoted 4 lines
CPU88                   0      nvidia_pcie_pmu_1_rc_1/wr_bytes/
CPU88                   0      nvidia_pcie_pmu_1_rc_1/rd_bytes/
CPU0        1,002,286,615      tool/duration_time/
CPU0                    0      nvidia_pcie_pmu_0_rc_1/wr_bytes,dst_rem=0x1/ #

0.0 MB/s lpm_pcie_rem_wr_bw_1

Quoted 3 lines
CPU88                   0      nvidia_pcie_pmu_1_rc_1/wr_bytes,dst_rem=0x1/
CPU0        1,002,275,059      tool/duration_time/
CPU0                    0      nvidia_pcie_pmu_0_rc_1/rd_bytes,dst_rem=0x1/ #

0.0 MB/s lpm_pcie_rem_rd_bw_1

Quoted 12 lines

CPU88 0 nvidia_pcie_pmu_1_rc_1/rd_bytes,dst_rem=0x1/
<snip>

Note CPU88 (socket 1_ has no lpm_pcie_* values.

2. In the default mode, lpm_pcie_*_<i> adds RC i of socket 0 and RC i of socket 1. These are separate root complexes with different devices, so the "Root Complex i" value can combine traffic from two unrelated links.
Could the metrics follow the hardware hierarchy instead, starting at the socket and drilling down to the RC, with the same destination split at each level?

Quoted 7 lines
    - per socket, all RCs (nvidia_pcie_pmu_<s>_rc):
        lpm_pcie_s<s>_{bw,rd_bw,wr_bw}
        lpm_pcie_s<s>_{loc,rem}_{rd,wr}_bw
    - per socket and RC (nvidia_pcie_pmu_<s>_rc_<i>):
        lpm_pcie_s<s>_{bw,rd_bw,wr_bw}_<i>
        lpm_pcie_s<s>_{loc,rem}_{rd,wr}_bw_<i>
      grouped as lpm_pcie_s<s>_rc<i>_bw.
Quoted 9 lines

Thanks for testing and for the suggestion. I think both problems come from how perf expects a PMU to expose its cpumask. Ideally perf wants a single PMU for both sockets, with one CPU per socket in its cpumask, e.g.:

```
/sys/bus/event_source/devices/nvidia_pcie_rc_0/cpumask: 0,88
```

I see this as a driver design choice: expose each physical PMU separately, or group several physical instances under one logical PMU. Either way, the PMU name and cpumask tell userspace which CPU(s) and root complex are associated with the PMU. With the first approach, each perf PMU has a cpumask of a single CPU. Tegra410 follows the first approach and exposes each physical PMU as a separate perf PMU.

Quoted 3 lines

We can work around the one-PMU-per-socket layout in the metrics, but would a single PMU per RC be a viable plan for you to implement?

Grouping instances across sockets is an interesting alternative, but it would require changes to the arm_cspmu driver design. We would also need to evaluate how such grouping could be supported generically in arm_cspmu, and whether the ACPI APMT would need changes.

For this patch, I think userspace gives us more flexibility in how we present and aggregate counts across sockets and root complexes, so I'd prefer to address it there and keep the current driver model.

FYI, I've posted the Tegra410 uncore JSON series:
https://lore.kernel.org/all/[email protected]/
It adds events and metrics for the same counters (e.g. ucf_avg_*, cmem_avg_rd_latency, pcie_avg_*), so it would be good to align the two and avoid duplicate metrics.

Thanks,
Besar

Quoted 163 lines

Thanks,
CT

Quoted 9 lines

Thanks,
Besar

Quoted 5 lines

-----Original Message-----
From: Chun-Tse Shao <[email protected]>
Sent: Monday, September 28, 2026 5:51 PM
To: [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected];

[email protected];

Quoted 16 lines
Quoted 16 lines

[email protected]; [email protected]; [email protected]; [email protected]; Besar Wicaksono <[email protected]>; [email protected]; linux- [email protected]; Chun-Tse Shao <[email protected]>
Subject: [PATCH 2/2] perf jevents: Add NVIDIA Tegra410 uncore PCIe metrics

External email: Use caution opening links or attachments

Add PCIe bandwidth metrics for the NVIDIA Tegra410 SoC. The formulas follow Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst:

AVG_RD_BANDWIDTH_IN_GBPS = RD_BYTES / ELAPSED_TIME_IN_NS AVG_WR_BANDWIDTH_IN_GBPS = WR_BYTES / ELAPSED_TIME_IN_NS

Quoted 6 lines
Quoted 6 lines

There is one PCIE PMU per socket and PCIe Root Complex (RC), named nvidia_pcie_pmu_<socket-id>_rc_<pcie-rc-id>, with RCs 0 to 5 in each socket. The metrics sum the rd_bytes/wr_bytes events of both sockets:

lpm_pcie_bw, lpm_pcie_rd_bw, lpm_pcie_wr_bw: all RCs. lpm_pcie_bw_<rc>, lpm_pcie_rd_bw_<rc>, lpm_pcie_wr_bw_<rc>: one

RC.

Quoted 3 lines
Quoted 3 lines

lpm_pcie_loc_rd_bw_<rc>, lpm_pcie_loc_wr_bw_<rc>: one RC to local
CMEM, GMEM, PCIe peer and CXL memory.
lpm_pcie_rem_rd_bw_<rc>, lpm_pcie_rem_wr_bw_<rc>: one RC to

remote

Quoted 37 lines
Quoted 37 lines

memory.

The per RC metrics are in the lpm_pcie_rc<rc>_bw metric groups.

A socket's event counts as 0 if its PMU isn't present (has_event() is 0), e.g. on a single socket system, or if it isn't counted in the aggregation (source_count() is 0), e.g. the other socket's event with --per-socket. Otherwise the metric fails to parse or is nan.

Assisted-by Gemini:gemini-3.1-pro-preview
---
 tools/perf/pmu-events/arm64_metrics.py | 68
++++++++++++++++++++++++--
 1 file changed, 64 insertions(+), 4 deletions(-)
tools/perf/pmu- renamed from tools/perf/pmu-events/arm64_metrics.py +0 -0

events/arm64_metrics.py
index b0d0651fec12..1ca3d8ddb4b7 100755

tools/perf/pmu-events/arm64_metrics.py +4 -3
@@ -2,9 +2,10 @@
 # SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
 import argparse
 import os
-from typing import Optional
-from metric import (aggr_nr, d_ratio, Event, JsonEncodeMetric,
-                    JsonEncodeMetricGroupDescriptions, LoadEvents, Metric,
 MetricGroup)
+from typing import List, Optional, Union
+from metric import (aggr_nr, d_ratio, has_event, source_count, Event,
 Expression,
+                    JsonEncodeMetric, JsonEncodeMetricGroupDescriptions,
 LoadEvents, Metric,
+                    MetricGroup, Select)

from common_metrics import Cycles

Quoted 2 lines

tools/perf/pmu-events/arm64_metrics.py def NvidiaT410Uncore() ->

Quoted 2 lines

# Global command line arguments.

tools/perf/pmu-events/arm64_metrics.py +0 -0
@@ -67,7 +68,66 @@ def NvidiaT410Uncore() ->

Optional[MetricGroup]:

Quoted 9 lines

tools/perf/pmu-events/arm64_metrics.py def NvidiaT410Uncore() -> Optional[MetricGroup]: · line 132

Quoted 9 lines
tools/perf/pmu-events/arm64_metrics.py +4 -1
                    rd_cum_outs - rd_cum_outs, "1ns"),
         ], description="NVIDIA Tegra410 DDR latency")
 
-    return MetricGroup("lpm_uncore", [DdrBw(), DdrLat()],
+    def PcieBw() -> MetricGroup:
+        # PCIE PMU, one per socket and PCIe Root Complex (RC):
+        # AVG_{RD,WR}_BANDWIDTH = {RD,WR}_BYTES / ELAPSED_TIME
+        loc_dst =
 

"dst_loc_cmem=0x1,dst_loc_gmem=0x1,dst_loc_pcie_p2p=0x1,dst_loc_pci

Quoted 6 lines
Quoted 6 lines

e_cxl=0x1"

tools/perf/pmu-events/arm64_metrics.py +5 -0
+        rem_dst = "dst_rem=0x1"
+
+        def SocketSum(rc: str, event: str) -> Expression:
+            # Sum the event over both sockets. A PMU counts as 0 if it isn't
+            # present (has_event() is 0) or isn't counted in the aggregation,
Quoted 8 lines

tools/perf/pmu-events/arm64_metrics.py def NvidiaT410Uncore() -> Optional[MetricGroup]:

Quoted 8 lines
tools/perf/pmu-events/arm64_metrics.py +8 -0
+            # e.g. the other socket with --per-socket (source_count() is 0).
+            def Count(e: Event) -> Expression:
+                return Select(Select(e, source_count(e), 0), has_event(e), 0)
+
+            return (Count(Event(f"nvidia_pcie_pmu_0_{rc}/{event}/")) +
+                    Count(Event(f"nvidia_pcie_pmu_1_{rc}/{event}/")))
+
+        # nvidia_pcie_pmu_<socket-id>_rc matches the PMUs of all RCs in

the

Quoted 22 lines
Quoted 22 lines

socket.

tools/perf/pmu-events/arm64_metrics.py +4 -0
+        rd = SocketSum("rc", "rd_bytes")
+        wr = SocketSum("rc", "wr_bytes")
+        metrics: List[Union[Metric, MetricGroup]] = [
+            Metric("lpm_pcie_bw", "Total PCIe bandwidth across all Root

Complexes",

tools/perf/pmu-events/arm64_metrics.py +2 -0
+                   d_ratio(rd + wr, interval_sec), mb_per_sec),
+            Metric("lpm_pcie_rd_bw", "Total PCIe read bandwidth across all

Root Complexes",

tools/perf/pmu-events/arm64_metrics.py +2 -0
+                   d_ratio(rd, interval_sec), mb_per_sec),
+            Metric("lpm_pcie_wr_bw", "Total PCIe write bandwidth across all

Root Complexes",

tools/perf/pmu-events/arm64_metrics.py +10 -0
+                   d_ratio(wr, interval_sec), mb_per_sec),
+        ]
+        for i in range(6):
+            rc = f"rc_{i}"
+            rc_rd = SocketSum(rc, "rd_bytes")
+            rc_wr = SocketSum(rc, "wr_bytes")
+            loc_rd = SocketSum(rc, f"rd_bytes,{loc_dst}")
+            loc_wr = SocketSum(rc, f"wr_bytes,{loc_dst}")
+            rem_rd = SocketSum(rc, f"rd_bytes,{rem_dst}")
+            rem_wr = SocketSum(rc, f"wr_bytes,{rem_dst}")
Quoted 8 lines

tools/perf/pmu-events/arm64_metrics.py

Quoted 8 lines
tools/perf/pmu-events/arm64_metrics.py +2 -0
+            metrics.append(MetricGroup(f"lpm_pcie_rc{i}_bw", [
+                Metric(f"lpm_pcie_bw_{i}", f"PCIe total bandwidth on Root

Complex {i}",

tools/perf/pmu-events/arm64_metrics.py +2 -0
+                       d_ratio(rc_rd + rc_wr, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_rd_bw_{i}", f"PCIe read bandwidth on Root

Complex {i}",

tools/perf/pmu-events/arm64_metrics.py +2 -0
+                       d_ratio(rc_rd, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_wr_bw_{i}", f"PCIe write bandwidth on

Root

Quoted 27 lines
Quoted 26 lines

Complex {i}",

tools/perf/pmu-events/arm64_metrics.py +17 -0
+                       d_ratio(rc_wr, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_loc_rd_bw_{i}",
+                       f"PCIe local read bandwidth on Root Complex {i}",
+                       d_ratio(loc_rd, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_loc_wr_bw_{i}",
+                       f"PCIe local write bandwidth on Root Complex {i}",
+                       d_ratio(loc_wr, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_rem_rd_bw_{i}",
+                       f"PCIe remote read bandwidth on Root Complex {i}",
+                       d_ratio(rem_rd, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_rem_wr_bw_{i}",
+                       f"PCIe remote write bandwidth on Root Complex {i}",
+                       d_ratio(rem_wr, interval_sec), mb_per_sec),
+            ], description=f"PCIe bandwidth on Root Complex {i}"))
+
+        return MetricGroup("lpm_pcie_bw", metrics,
+                           description="NVIDIA Tegra410 PCIe Root Complex

bandwidth")

tools/perf/pmu-events/arm64_metrics.py +2 -1
+
+    return MetricGroup("lpm_uncore", [DdrBw(), DdrLat(), PcieBw()],
                        description="NVIDIA Tegra410 uncore metrics")
 
 
--

2.56.0.rc1.315.gc6ed9934b7-goog

#11 lore raw

Re: [PATCH 2/2] perf jevents: Add NVIDIA Tegra410 uncore PCIe metrics

Hi Besar,

Thanks, that makes sense. I'll drop my series in favor of yours.

Two things from mine that your JSON doesn't have yet:
1. DDR bandwidth from all sources to local CMEM (mem_bytes_{rd,wr} with only dst_loc_cmem=1). Today it needs the sum of the local_cpu, local_noncpu and remote _to_local_cmem metrics.
2. Read + write total bandwidth, for UCF memory and PCIe.

Agreed on handling the PCIe socket grouping in userspace; the --metric-only issue I reported on your 3/6 is part of that.

Thanks,
CT

Quoted 270 lines · Besar Wicaksono
Quoted 68 lines

-----Original Message-----
From: Chun-Tse Shao <[email protected]>
Sent: Friday, October 16, 2026 6:40 PM
To: Besar Wicaksono <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux-perf- [email protected]; [email protected]; Matt Ochs <[email protected]>; Sean Kelley <[email protected]>; Rich Wiley <[email protected]>; Vikram Sethi <[email protected]>; Raj Joshi <[email protected]>
Subject: Re: [PATCH 2/2] perf jevents: Add NVIDIA Tegra410 uncore PCIe metrics

External email: Use caution opening links or attachments

Quoted 7 lines · Besar Wicaksono

Hi Chun-Tse,

I tested this on 2-sockets Tegra410 and found some issues.
1. With --per-socket or -A, every lpm_pcie_* metric is only reported for socket 0, so socket 1's PCIe traffic never shows up. Example output:

Quoted 5 lines

perf stat -a -A -M lpm_pcie_rc1_bw -- sleep 1

Performance counter stats for 'system wide':

CPU0 15,204 nvidia_pcie_pmu_0_rc_1/wr_bytes/ # 0.0 MB/s

lpm_pcie_wr_bw_1

Quoted 2 lines

CPU0 # 33.8 MB/s lpm_pcie_bw_1
CPU0 33,910,952 nvidia_pcie_pmu_0_rc_1/rd_bytes/ # 33.8

MB/s lpm_pcie_rd_bw_1

Quoted 4 lines
CPU88                   0      nvidia_pcie_pmu_1_rc_1/wr_bytes/
CPU88                   0      nvidia_pcie_pmu_1_rc_1/rd_bytes/
CPU0        1,002,286,615      tool/duration_time/
CPU0                    0      nvidia_pcie_pmu_0_rc_1/wr_bytes,dst_rem=0x1/ #

0.0 MB/s lpm_pcie_rem_wr_bw_1

Quoted 3 lines
CPU88                   0      nvidia_pcie_pmu_1_rc_1/wr_bytes,dst_rem=0x1/
CPU0        1,002,275,059      tool/duration_time/
CPU0                    0      nvidia_pcie_pmu_0_rc_1/rd_bytes,dst_rem=0x1/ #

0.0 MB/s lpm_pcie_rem_rd_bw_1

Quoted 12 lines

CPU88 0 nvidia_pcie_pmu_1_rc_1/rd_bytes,dst_rem=0x1/
<snip>

Note CPU88 (socket 1_ has no lpm_pcie_* values.

2. In the default mode, lpm_pcie_*_<i> adds RC i of socket 0 and RC i of socket 1. These are separate root complexes with different devices, so the "Root Complex i" value can combine traffic from two unrelated links.
Could the metrics follow the hardware hierarchy instead, starting at the socket and drilling down to the RC, with the same destination split at each level?

Quoted 7 lines
    - per socket, all RCs (nvidia_pcie_pmu_<s>_rc):
        lpm_pcie_s<s>_{bw,rd_bw,wr_bw}
        lpm_pcie_s<s>_{loc,rem}_{rd,wr}_bw
    - per socket and RC (nvidia_pcie_pmu_<s>_rc_<i>):
        lpm_pcie_s<s>_{bw,rd_bw,wr_bw}_<i>
        lpm_pcie_s<s>_{loc,rem}_{rd,wr}_bw_<i>
      grouped as lpm_pcie_s<s>_rc<i>_bw.
Quoted 9 lines

Thanks for testing and for the suggestion. I think both problems come from how perf expects a PMU to expose its cpumask. Ideally perf wants a single PMU for both sockets, with one CPU per socket in its cpumask, e.g.:

```
/sys/bus/event_source/devices/nvidia_pcie_rc_0/cpumask: 0,88
```

I see this as a driver design choice: expose each physical PMU separately, or group several physical instances under one logical PMU. Either way, the PMU name and cpumask tell userspace which CPU(s) and root complex are associated with the PMU. With the first approach, each perf PMU has a cpumask of a single CPU. Tegra410 follows the first approach and exposes each physical PMU as a separate perf PMU.

Quoted 3 lines

We can work around the one-PMU-per-socket layout in the metrics, but would a single PMU per RC be a viable plan for you to implement?

Grouping instances across sockets is an interesting alternative, but it would require changes to the arm_cspmu driver design. We would also need to evaluate how such grouping could be supported generically in arm_cspmu, and whether the ACPI APMT would need changes.

For this patch, I think userspace gives us more flexibility in how we present and aggregate counts across sockets and root complexes, so I'd prefer to address it there and keep the current driver model.

FYI, I've posted the Tegra410 uncore JSON series:
https://lore.kernel.org/all/[email protected]/
It adds events and metrics for the same counters (e.g. ucf_avg_*, cmem_avg_rd_latency, pcie_avg_*), so it would be good to align the two and avoid duplicate metrics.

Thanks,
Besar

Quoted 163 lines

Thanks,
CT

Quoted 9 lines

Thanks,
Besar

Quoted 5 lines

-----Original Message-----
From: Chun-Tse Shao <[email protected]>
Sent: Monday, September 28, 2026 5:51 PM
To: [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected];

[email protected];

Quoted 16 lines
Quoted 16 lines

[email protected]; [email protected]; [email protected]; [email protected]; Besar Wicaksono <[email protected]>; [email protected]; linux- [email protected]; Chun-Tse Shao <[email protected]>
Subject: [PATCH 2/2] perf jevents: Add NVIDIA Tegra410 uncore PCIe metrics

External email: Use caution opening links or attachments

Add PCIe bandwidth metrics for the NVIDIA Tegra410 SoC. The formulas follow Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst:

AVG_RD_BANDWIDTH_IN_GBPS = RD_BYTES / ELAPSED_TIME_IN_NS AVG_WR_BANDWIDTH_IN_GBPS = WR_BYTES / ELAPSED_TIME_IN_NS

Quoted 6 lines
Quoted 6 lines

There is one PCIE PMU per socket and PCIe Root Complex (RC), named nvidia_pcie_pmu_<socket-id>_rc_<pcie-rc-id>, with RCs 0 to 5 in each socket. The metrics sum the rd_bytes/wr_bytes events of both sockets:

lpm_pcie_bw, lpm_pcie_rd_bw, lpm_pcie_wr_bw: all RCs. lpm_pcie_bw_<rc>, lpm_pcie_rd_bw_<rc>, lpm_pcie_wr_bw_<rc>: one

RC.

Quoted 3 lines
Quoted 3 lines

lpm_pcie_loc_rd_bw_<rc>, lpm_pcie_loc_wr_bw_<rc>: one RC to local
CMEM, GMEM, PCIe peer and CXL memory.
lpm_pcie_rem_rd_bw_<rc>, lpm_pcie_rem_wr_bw_<rc>: one RC to

remote

Quoted 37 lines
Quoted 37 lines

memory.

The per RC metrics are in the lpm_pcie_rc<rc>_bw metric groups.

A socket's event counts as 0 if its PMU isn't present (has_event() is 0), e.g. on a single socket system, or if it isn't counted in the aggregation (source_count() is 0), e.g. the other socket's event with --per-socket. Otherwise the metric fails to parse or is nan.

Assisted-by Gemini:gemini-3.1-pro-preview
---
 tools/perf/pmu-events/arm64_metrics.py | 68
++++++++++++++++++++++++--
 1 file changed, 64 insertions(+), 4 deletions(-)
tools/perf/pmu- renamed from tools/perf/pmu-events/arm64_metrics.py +0 -0

events/arm64_metrics.py
index b0d0651fec12..1ca3d8ddb4b7 100755

tools/perf/pmu-events/arm64_metrics.py +4 -3
@@ -2,9 +2,10 @@
 # SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
 import argparse
 import os
-from typing import Optional
-from metric import (aggr_nr, d_ratio, Event, JsonEncodeMetric,
-                    JsonEncodeMetricGroupDescriptions, LoadEvents, Metric,
 MetricGroup)
+from typing import List, Optional, Union
+from metric import (aggr_nr, d_ratio, has_event, source_count, Event,
 Expression,
+                    JsonEncodeMetric, JsonEncodeMetricGroupDescriptions,
 LoadEvents, Metric,
+                    MetricGroup, Select)

from common_metrics import Cycles

Quoted 2 lines

tools/perf/pmu-events/arm64_metrics.py def NvidiaT410Uncore() ->

Quoted 2 lines

# Global command line arguments.

tools/perf/pmu-events/arm64_metrics.py +0 -0
@@ -67,7 +68,66 @@ def NvidiaT410Uncore() ->

Optional[MetricGroup]:

Quoted 9 lines

tools/perf/pmu-events/arm64_metrics.py def NvidiaT410Uncore() -> Optional[MetricGroup]: · line 132

Quoted 9 lines
tools/perf/pmu-events/arm64_metrics.py +4 -1
                    rd_cum_outs - rd_cum_outs, "1ns"),
         ], description="NVIDIA Tegra410 DDR latency")
 
-    return MetricGroup("lpm_uncore", [DdrBw(), DdrLat()],
+    def PcieBw() -> MetricGroup:
+        # PCIE PMU, one per socket and PCIe Root Complex (RC):
+        # AVG_{RD,WR}_BANDWIDTH = {RD,WR}_BYTES / ELAPSED_TIME
+        loc_dst =
 

"dst_loc_cmem=0x1,dst_loc_gmem=0x1,dst_loc_pcie_p2p=0x1,dst_loc_pci

Quoted 6 lines
Quoted 6 lines

e_cxl=0x1"

tools/perf/pmu-events/arm64_metrics.py +5 -0
+        rem_dst = "dst_rem=0x1"
+
+        def SocketSum(rc: str, event: str) -> Expression:
+            # Sum the event over both sockets. A PMU counts as 0 if it isn't
+            # present (has_event() is 0) or isn't counted in the aggregation,
Quoted 8 lines

tools/perf/pmu-events/arm64_metrics.py def NvidiaT410Uncore() -> Optional[MetricGroup]:

Quoted 8 lines
tools/perf/pmu-events/arm64_metrics.py +8 -0
+            # e.g. the other socket with --per-socket (source_count() is 0).
+            def Count(e: Event) -> Expression:
+                return Select(Select(e, source_count(e), 0), has_event(e), 0)
+
+            return (Count(Event(f"nvidia_pcie_pmu_0_{rc}/{event}/")) +
+                    Count(Event(f"nvidia_pcie_pmu_1_{rc}/{event}/")))
+
+        # nvidia_pcie_pmu_<socket-id>_rc matches the PMUs of all RCs in

the

Quoted 22 lines
Quoted 22 lines

socket.

tools/perf/pmu-events/arm64_metrics.py +4 -0
+        rd = SocketSum("rc", "rd_bytes")
+        wr = SocketSum("rc", "wr_bytes")
+        metrics: List[Union[Metric, MetricGroup]] = [
+            Metric("lpm_pcie_bw", "Total PCIe bandwidth across all Root

Complexes",

tools/perf/pmu-events/arm64_metrics.py +2 -0
+                   d_ratio(rd + wr, interval_sec), mb_per_sec),
+            Metric("lpm_pcie_rd_bw", "Total PCIe read bandwidth across all

Root Complexes",

tools/perf/pmu-events/arm64_metrics.py +2 -0
+                   d_ratio(rd, interval_sec), mb_per_sec),
+            Metric("lpm_pcie_wr_bw", "Total PCIe write bandwidth across all

Root Complexes",

tools/perf/pmu-events/arm64_metrics.py +10 -0
+                   d_ratio(wr, interval_sec), mb_per_sec),
+        ]
+        for i in range(6):
+            rc = f"rc_{i}"
+            rc_rd = SocketSum(rc, "rd_bytes")
+            rc_wr = SocketSum(rc, "wr_bytes")
+            loc_rd = SocketSum(rc, f"rd_bytes,{loc_dst}")
+            loc_wr = SocketSum(rc, f"wr_bytes,{loc_dst}")
+            rem_rd = SocketSum(rc, f"rd_bytes,{rem_dst}")
+            rem_wr = SocketSum(rc, f"wr_bytes,{rem_dst}")
Quoted 8 lines

tools/perf/pmu-events/arm64_metrics.py

Quoted 8 lines
tools/perf/pmu-events/arm64_metrics.py +2 -0
+            metrics.append(MetricGroup(f"lpm_pcie_rc{i}_bw", [
+                Metric(f"lpm_pcie_bw_{i}", f"PCIe total bandwidth on Root

Complex {i}",

tools/perf/pmu-events/arm64_metrics.py +2 -0
+                       d_ratio(rc_rd + rc_wr, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_rd_bw_{i}", f"PCIe read bandwidth on Root

Complex {i}",

tools/perf/pmu-events/arm64_metrics.py +2 -0
+                       d_ratio(rc_rd, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_wr_bw_{i}", f"PCIe write bandwidth on

Root

Quoted 27 lines
Quoted 26 lines

Complex {i}",

tools/perf/pmu-events/arm64_metrics.py +17 -0
+                       d_ratio(rc_wr, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_loc_rd_bw_{i}",
+                       f"PCIe local read bandwidth on Root Complex {i}",
+                       d_ratio(loc_rd, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_loc_wr_bw_{i}",
+                       f"PCIe local write bandwidth on Root Complex {i}",
+                       d_ratio(loc_wr, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_rem_rd_bw_{i}",
+                       f"PCIe remote read bandwidth on Root Complex {i}",
+                       d_ratio(rem_rd, interval_sec), mb_per_sec),
+                Metric(f"lpm_pcie_rem_wr_bw_{i}",
+                       f"PCIe remote write bandwidth on Root Complex {i}",
+                       d_ratio(rem_wr, interval_sec), mb_per_sec),
+            ], description=f"PCIe bandwidth on Root Complex {i}"))
+
+        return MetricGroup("lpm_pcie_bw", metrics,
+                           description="NVIDIA Tegra410 PCIe Root Complex

bandwidth")

tools/perf/pmu-events/arm64_metrics.py +2 -1
+
+    return MetricGroup("lpm_uncore", [DdrBw(), DdrLat(), PcieBw()],
                        description="NVIDIA Tegra410 uncore metrics")
 
 
--

2.56.0.rc1.315.gc6ed9934b7-goog

#12 lore raw
In reply to Chun-Tse Shao
Quoted 4 lines · Chun-Tse Shao

Add DDR bandwidth/latency and PCIe bandwidth metrics for the NVIDIA Tegra410 SoC to arm64_metrics.py. The metrics use the sysfs events of the Tegra410 UCF, CMEM latency and PCIE PMUs, and follow the formulas in Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst.

Ian,

Can you take a look and please provide your Reviewed-by?

Thanks,

- Arnaldo

Quoted 44 lines

Patch 1 adds the DDR metrics and makes pmu-events/Build also run arm64_metrics.py for the nvidia vendor models. Patch 2 adds the PCIe metrics, in total and per PCIe Root Complex (RC).

Tested on a 2-socket Tegra410 system:

- memcpy with multiload (from multichase) on socket 0 CPUs and memory:

   lpm_ddr_bw is 740-744 GB/s at steady state (perf stat -I 2000),
   multiload reports 708063 MiB/s (742 GB/s).

- Read only (stream-sum) and write only (memset) loads on socket 1:
lpm_ddr_rd_bw is 608 GB/s with 2.4 GB/s of writes, lpm_ddr_wr_bw is 628 GB/s with 1.1 GB/s of reads.
- memcpy on socket 0 CPUs with memory on socket 1: socket 0's lpm_ddr_rem_rd_bw and socket 1's lpm_ddr_rd_bw count nearly the same bytes (258.03 vs 258.73 GB).
- lpm_ddr_lat idle is 150 ns in the default aggregation, and 132 ns and 194 ns for socket 0 and 1 with --per-socket. Under the memcpy load it is 555 ns by default and 563 ns for socket 0 with --per-socket, and stays at 551-558 ns per interval with -I 1000.
- 8 GiB O_DIRECT dd read from an NVMe drive behind RC 0 of socket 0:
lpm_pcie_wr_bw_0 counts the 8 GiB at 8.7 GB/s (dd: 8.7 GB/s). With the buffer on node 0 it shows up in lpm_pcie_loc_wr_bw_0, with the buffer on node 1 in lpm_pcie_rem_wr_bw_0. With --per-socket, socket 0 shows the same and no metric of either socket is nan.
- With the nvidia_t410 metrics forced on a machine without these PMUs (PERF_CPUID=0x000000004e0f0100 on an x86 JEVENTS_ARCH=all build), the PCIe metrics read 0 rather than failing to parse, also with --per-socket.
- "perf test 10" (PMU JSON event tests) passes on the Tegra410 system and with the x86 JEVENTS_ARCH=all build.

Chun-Tse Shao (2):
perf jevents: Add NVIDIA Tegra410 uncore DDR metrics perf jevents: Add NVIDIA Tegra410 uncore PCIe metrics

 tools/perf/pmu-events/Build            |   2 +-
 tools/perf/pmu-events/arm64_metrics.py | 126 ++++++++++++++++++++++++-
 tools/perf/pmu-events/metric.py        |   2 +-
 3 files changed, 126 insertions(+), 4 deletions(-)

base-commit: 0ae6fc78c5ce0dfd18d8712a50f0fd4602eff103 --
2.56.0.rc1.315.gc6ed9934b7-goog

#13 lore raw
In reply to Chun-Tse Shao

Hi Chun-Tse,

Thanks for the series.
FYI, I am working on the JSON patch series for Tegra410 uncore PMUs, which adds the events and metrics for the same counters in UCF, CMEM latency, PCIE, and others.
I will cc you when I post it.

Thanks,
Besar

Quoted 68 lines

-----Original Message-----
From: Chun-Tse Shao <[email protected]>
Sent: Monday, September 28, 2026 5:51 PM
To: [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Besar Wicaksono <[email protected]>; [email protected]; linux- [email protected]; Chun-Tse Shao <[email protected]>
Subject: [PATCH 0/2] perf jevents: Add NVIDIA Tegra410 uncore DDR and PCIe metrics

External email: Use caution opening links or attachments

Add DDR bandwidth/latency and PCIe bandwidth metrics for the NVIDIA Tegra410 SoC to arm64_metrics.py. The metrics use the sysfs events of the Tegra410 UCF, CMEM latency and PCIE PMUs, and follow the formulas in
Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst.

Patch 1 adds the DDR metrics and makes pmu-events/Build also run arm64_metrics.py for the nvidia vendor models. Patch 2 adds the PCIe metrics, in total and per PCIe Root Complex (RC).

Tested on a 2-socket Tegra410 system:

- memcpy with multiload (from multichase) on socket 0 CPUs and memory:

   lpm_ddr_bw is 740-744 GB/s at steady state (perf stat -I 2000),
   multiload reports 708063 MiB/s (742 GB/s).

- Read only (stream-sum) and write only (memset) loads on socket 1:
lpm_ddr_rd_bw is 608 GB/s with 2.4 GB/s of writes, lpm_ddr_wr_bw is 628 GB/s with 1.1 GB/s of reads.
- memcpy on socket 0 CPUs with memory on socket 1: socket 0's lpm_ddr_rem_rd_bw and socket 1's lpm_ddr_rd_bw count nearly the
same
bytes (258.03 vs 258.73 GB).
- lpm_ddr_lat idle is 150 ns in the default aggregation, and 132 ns and 194 ns for socket 0 and 1 with --per-socket. Under the memcpy load it is 555 ns by default and 563 ns for socket 0 with --per-socket, and stays at 551-558 ns per interval with -I 1000.
- 8 GiB O_DIRECT dd read from an NVMe drive behind RC 0 of socket 0:
lpm_pcie_wr_bw_0 counts the 8 GiB at 8.7 GB/s (dd: 8.7 GB/s). With the buffer on node 0 it shows up in lpm_pcie_loc_wr_bw_0, with the buffer on node 1 in lpm_pcie_rem_wr_bw_0. With --per-socket, socket 0 shows the same and no metric of either socket is nan.
- With the nvidia_t410 metrics forced on a machine without these PMUs (PERF_CPUID=0x000000004e0f0100 on an x86 JEVENTS_ARCH=all build),
the
PCIe metrics read 0 rather than failing to parse, also with --per-socket.
- "perf test 10" (PMU JSON event tests) passes on the Tegra410 system and with the x86 JEVENTS_ARCH=all build.

Chun-Tse Shao (2):
perf jevents: Add NVIDIA Tegra410 uncore DDR metrics perf jevents: Add NVIDIA Tegra410 uncore PCIe metrics

 tools/perf/pmu-events/Build            |   2 +-
 tools/perf/pmu-events/arm64_metrics.py | 126
++++++++++++++++++++++++-
 tools/perf/pmu-events/metric.py        |   2 +-
 3 files changed, 126 insertions(+), 4 deletions(-)

base-commit: 0ae6fc78c5ce0dfd18d8712a50f0fd4602eff103 --
2.56.0.rc1.315.gc6ed9934b7-goog