Skip to contentSkip to conversation

Add IOMMU TLB and interrupt metrics

PATCH Applied Chun-Tse Shao 3 participants 5 messages May 28 – May 30 perf events

lore
Share & tools
lore
#0 lore raw

v2 cover Add IOMMU TLB and interrupt metrics

This patch series adds IOMMU Translation Lookaside Buffer (TLB) and interrupt cache metrics to perf jevents for both AMD and Intel platforms. This enhances I/O performance observability, allowing fleet-wide monitoring of IOMMU overhead.

The changes are split into two patches:
1. perf jevents: Add IOMMU metrics for AMD
- Adds IOMMU TLB and interrupt metrics for Zen 2+ processors using standard AMD IOMMU PMU events.
- Note that pde events on AMD cover both 2M and 1G pages, so 1G pages are implicitly included.
- Added code comments to clarify this hardware detail and fixed indentation to match the file's style.

2. perf jevents: Add IOMMU metrics for Intel
- Adds IOMMU TLB and interrupt metrics using uncore IIO IOMMU events.
- Supports Emerald Rapids (TLB-only) and Granite Rapids (TLB + Interrupt) by dynamically detecting event availability and making interrupt metrics optional.
- Clamped calculated interrupt cache miss metric to zero to prevent negative values due to counter multiplexing or sampling skid.

v2:
Split the changes into separate AMD and Intel commits as requested.

v1: lore.kernel.org/[email protected]

Chun-Tse Shao (2):
perf jevents: Add IOMMU metrics for AMD
perf jevents: Add IOMMU metrics for Intel

 tools/perf/pmu-events/amd_metrics.py   | 57 +++++++++++++++++++++++
 tools/perf/pmu-events/intel_metrics.py | 62 ++++++++++++++++++++++++++
 2 files changed, 119 insertions(+)

--
2.54.0.823.g6e5bcc1fc9-goog

#1 lore raw

PATCH v2 1/2 perf jevents: Add IOMMU metrics for AMD

Add IOMMU Translation Lookaside Buffer (TLB) and interrupt cache metrics to perf jevents for AMD platforms. This enhances I/O performance observability, allowing fleet-wide monitoring of IOMMU overhead.

These metrics are supported on Zen 2 and newer processors (Rome, Milan, Genoa, Turin) and are implemented using the standard amd_iommu PMU events. The implementation uses the existing _zen_model helper to ensure these are only generated for Zen 2+. Note that the pde events on AMD cover both 2M and 1G pages, so 1G pages are implicitly included in the total hits/misses metrics (sum of pte and pde events).

The following metrics are added:
- iotlb_total_hit: Total IOTLB hits (4K, 2M, 1G pages).
- iotlb_total_miss: Total IOTLB misses.
- iotlb_miss_rate: IOTLB miss rate.
- iotlb_interrupt_cache_hit: Interrupt cache hits.
- iotlb_interrupt_cache_miss: Interrupt cache misses.
- iotlb_interrupt_cache_lookup: Interrupt cache lookups.
- iotlb_interrupt_cache_miss_rate: Interrupt cache miss rate.

Tested:

  # perf stat -M \
    iotlb_total_hit,iotlb_total_miss,iotlb_miss_rate \
    --per-socket --metric-only -a -j -- sleep 10
  {"socket" : "S0", "counters" : 10,
   "hits  iotlb_total_hit" : "3579249.0",
   "%  iotlb_miss_rate" : "0.0",
   "misses  iotlb_total_miss" : "3.0"}
  {"socket" : "S1", "counters" : 10,
   "hits  iotlb_total_hit" : "0.0",
   "%  iotlb_miss_rate" : "0.0",
   "misses  iotlb_total_miss" : "0.0"}
Assisted-by Gemini:gemini-3.1-pro-preview
---
 tools/perf/pmu-events/amd_metrics.py | 57 ++++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)
tools/perf/pmu-events/amd_metrics.py +57 -0
index 971f6e7af1f8..dccfcacaf148 100755
@@ -265,6 +265,62 @@ def AmdDtlb() -> Optional[MetricGroup]:
     ], description="Data TLB metrics")
 
 
+def AmdIotlb() -> Optional[MetricGroup]:
+    global _zen_model
+    if _zen_model < 2:
+        return None
+
+    # On AMD, the pde events cover both 2M and 1G pages.
+    total_hit = Event("amd_iommu/mem_iommu_tlb_pte_hit/") + Event(
+        "amd_iommu/mem_iommu_tlb_pde_hit/"
+    )
+    total_miss = Event("amd_iommu/mem_iommu_tlb_pte_mis/") + Event(
+        "amd_iommu/mem_iommu_tlb_pde_mis/"
+    )
+    miss_rate = d_ratio(total_miss, total_miss + total_hit)
+
+    interrupt_cache_hit = Event("amd_iommu/int_dte_hit/")
+    interrupt_cache_miss = Event("amd_iommu/int_dte_mis/")
+    interrupt_cache_lookup = interrupt_cache_hit + interrupt_cache_miss
+    interrupt_cache_miss_rate = d_ratio(
+        interrupt_cache_miss, interrupt_cache_miss + interrupt_cache_hit
+    )
+
+    return MetricGroup(
+        "iotlb",
+        [
+            Metric("iotlb_total_hit", "IOTLB total hit", total_hit, "hits"),
+            Metric("iotlb_total_miss", "IOTLB total miss", total_miss, "misses"),
+            Metric("iotlb_miss_rate", "IOTLB miss rate", miss_rate, "100%"),
+            Metric(
+                "iotlb_interrupt_cache_hit",
+                "IOTLB interrupt cache hit",
+                interrupt_cache_hit,
+                "hits",
+            ),
+            Metric(
+                "iotlb_interrupt_cache_miss",
+                "IOTLB interrupt cache miss",
+                interrupt_cache_miss,
+                "misses",
+            ),
+            Metric(
+                "iotlb_interrupt_cache_lookup",
+                "IOTLB interrupt cache lookup",
+                interrupt_cache_lookup,
+                "lookups",
+            ),
+            Metric(
+                "iotlb_interrupt_cache_miss_rate",
+                "IOTLB interrupt cache miss rate",
+                interrupt_cache_miss_rate,
+                "100%",
+            ),
+        ],
+        description="IOMMU TLB metrics",
+    )
+
+
 def AmdItlb():
     global _zen_model
     l2h = Event("bp_l1_tlb_miss_l2_tlb_hit", "bp_l1_tlb_miss_l2_hit")
@@ -473,6 +529,7 @@ def main() -> None:
         AmdBr(),
         AmdCtxSw(),
         AmdDtlb(),
+        AmdIotlb(),
         AmdItlb(),
         AmdLdSt(),
         AmdUpc(),
Signature · 1 line
2.54.0.823.g6e5bcc1fc9-goog
Reviewed-by on v2 1/2
#2 lore raw

Re: [PATCH v2 1/2] perf jevents: Add IOMMU metrics for AMD

Quoted 33 lines · Chun-Tse Shao

Add IOMMU Translation Lookaside Buffer (TLB) and interrupt cache metrics to perf jevents for AMD platforms. This enhances I/O performance observability, allowing fleet-wide monitoring of IOMMU overhead.

These metrics are supported on Zen 2 and newer processors (Rome, Milan, Genoa, Turin) and are implemented using the standard amd_iommu PMU events. The implementation uses the existing _zen_model helper to ensure these are only generated for Zen 2+. Note that the pde events on AMD cover both 2M and 1G pages, so 1G pages are implicitly included in the total hits/misses metrics (sum of pte and pde events).

The following metrics are added:
- iotlb_total_hit: Total IOTLB hits (4K, 2M, 1G pages).
- iotlb_total_miss: Total IOTLB misses.
- iotlb_miss_rate: IOTLB miss rate.
- iotlb_interrupt_cache_hit: Interrupt cache hits.
- iotlb_interrupt_cache_miss: Interrupt cache misses.
- iotlb_interrupt_cache_lookup: Interrupt cache lookups.
- iotlb_interrupt_cache_miss_rate: Interrupt cache miss rate.

Tested:

  # perf stat -M \
    iotlb_total_hit,iotlb_total_miss,iotlb_miss_rate \
    --per-socket --metric-only -a -j -- sleep 10
  {"socket" : "S0", "counters" : 10,
   "hits  iotlb_total_hit" : "3579249.0",
   "%  iotlb_miss_rate" : "0.0",
   "misses  iotlb_total_miss" : "3.0"}
  {"socket" : "S1", "counters" : 10,
   "hits  iotlb_total_hit" : "0.0",
   "%  iotlb_miss_rate" : "0.0",
   "misses  iotlb_total_miss" : "0.0"}
Quoted 3 lines
Assisted-by Gemini:gemini-3.1-pro-preview

---

Quoted 78 lines
 tools/perf/pmu-events/amd_metrics.py | 57 ++++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)
tools/perf/pmu-events/amd_metrics.py +57 -0
index 971f6e7af1f8..dccfcacaf148 100755
@@ -265,6 +265,62 @@ def AmdDtlb() -> Optional[MetricGroup]:
     ], description="Data TLB metrics")
 
 
+def AmdIotlb() -> Optional[MetricGroup]:
+    global _zen_model
+    if _zen_model < 2:
+        return None
+
+    # On AMD, the pde events cover both 2M and 1G pages.
+    total_hit = Event("amd_iommu/mem_iommu_tlb_pte_hit/") + Event(
+        "amd_iommu/mem_iommu_tlb_pde_hit/"
+    )
+    total_miss = Event("amd_iommu/mem_iommu_tlb_pte_mis/") + Event(
+        "amd_iommu/mem_iommu_tlb_pde_mis/"
+    )
+    miss_rate = d_ratio(total_miss, total_miss + total_hit)
+
+    interrupt_cache_hit = Event("amd_iommu/int_dte_hit/")
+    interrupt_cache_miss = Event("amd_iommu/int_dte_mis/")
+    interrupt_cache_lookup = interrupt_cache_hit + interrupt_cache_miss
+    interrupt_cache_miss_rate = d_ratio(
+        interrupt_cache_miss, interrupt_cache_miss + interrupt_cache_hit
+    )
+
+    return MetricGroup(
+        "iotlb",
+        [
+            Metric("iotlb_total_hit", "IOTLB total hit", total_hit, "hits"),
+            Metric("iotlb_total_miss", "IOTLB total miss", total_miss, "misses"),
+            Metric("iotlb_miss_rate", "IOTLB miss rate", miss_rate, "100%"),
+            Metric(
+                "iotlb_interrupt_cache_hit",
+                "IOTLB interrupt cache hit",
+                interrupt_cache_hit,
+                "hits",
+            ),
+            Metric(
+                "iotlb_interrupt_cache_miss",
+                "IOTLB interrupt cache miss",
+                interrupt_cache_miss,
+                "misses",
+            ),
+            Metric(
+                "iotlb_interrupt_cache_lookup",
+                "IOTLB interrupt cache lookup",
+                interrupt_cache_lookup,
+                "lookups",
+            ),
+            Metric(
+                "iotlb_interrupt_cache_miss_rate",
+                "IOTLB interrupt cache miss rate",
+                interrupt_cache_miss_rate,
+                "100%",
+            ),
+        ],
+        description="IOMMU TLB metrics",
+    )
+
+
 def AmdItlb():
     global _zen_model
     l2h = Event("bp_l1_tlb_miss_l2_tlb_hit", "bp_l1_tlb_miss_l2_hit")
@@ -473,6 +529,7 @@ def main() -> None:
         AmdBr(),
         AmdCtxSw(),
         AmdDtlb(),
+        AmdIotlb(),
         AmdItlb(),
         AmdLdSt(),
         AmdUpc(),
maintainer Applied
#3 lore raw

Re: [PATCH v2 1/2] perf jevents: Add IOMMU metrics for AMD

Quoted 34 lines · Sandipan Das
Quoted 33 lines · Chun-Tse Shao

Add IOMMU Translation Lookaside Buffer (TLB) and interrupt cache metrics to perf jevents for AMD platforms. This enhances I/O performance observability, allowing fleet-wide monitoring of IOMMU overhead.

These metrics are supported on Zen 2 and newer processors (Rome, Milan, Genoa, Turin) and are implemented using the standard amd_iommu PMU events. The implementation uses the existing _zen_model helper to ensure these are only generated for Zen 2+. Note that the pde events on AMD cover both 2M and 1G pages, so 1G pages are implicitly included in the total hits/misses metrics (sum of pte and pde events).

The following metrics are added:
- iotlb_total_hit: Total IOTLB hits (4K, 2M, 1G pages).
- iotlb_total_miss: Total IOTLB misses.
- iotlb_miss_rate: IOTLB miss rate.
- iotlb_interrupt_cache_hit: Interrupt cache hits.
- iotlb_interrupt_cache_miss: Interrupt cache misses.
- iotlb_interrupt_cache_lookup: Interrupt cache lookups.
- iotlb_interrupt_cache_miss_rate: Interrupt cache miss rate.

Tested:

  # perf stat -M \
    iotlb_total_hit,iotlb_total_miss,iotlb_miss_rate \
    --per-socket --metric-only -a -j -- sleep 10
  {"socket" : "S0", "counters" : 10,
   "hits  iotlb_total_hit" : "3579249.0",
   "%  iotlb_miss_rate" : "0.0",
   "misses  iotlb_total_miss" : "3.0"}
  {"socket" : "S1", "counters" : 10,
   "hits  iotlb_total_hit" : "0.0",
   "%  iotlb_miss_rate" : "0.0",
   "misses  iotlb_total_miss" : "0.0"}
Quoted 5 lines
Quoted 3 lines
Assisted-by Gemini:gemini-3.1-pro-preview

---

Thanks, applied both patches and collected Ian's reviewed-by, since he provided it for the combined patch and requested that it was split, which this v2 does.

- Arnaldo

#4 lore raw

PATCH v2 2/2 perf jevents: Add IOMMU metrics for Intel

In reply to Chun-Tse Shao

Add IOMMU Translation Lookaside Buffer (TLB) and interrupt cache metrics to perf jevents for Intel platforms. This enhances I/O performance observability, allowing fleet-wide monitoring of IOMMU overhead.

These metrics are supported on platforms that expose the required uncore IIO IOMMU events (such as Emerald Rapids and Granite Rapids). The Intel implementation dynamically detects event availability at generation time. It requires at least the TLB events to expose the metric group, while the interrupt cache events are optional. This allows platforms like Emerald Rapids, which lack IOMMU interrupt cache events, to still expose the IOMMU TLB metrics.

The following metrics are added:
- iotlb_total_hit: Total IOTLB hits (4K, 2M, 1G pages).
- iotlb_total_miss: Total IOTLB misses.
- iotlb_miss_rate: IOTLB miss rate.
- iotlb_interrupt_cache_hit: Interrupt cache hits.
- iotlb_interrupt_cache_miss: Interrupt cache misses (calculated as lookup - hit, clamped to zero).
- iotlb_interrupt_cache_lookup: Interrupt cache lookups.
- iotlb_interrupt_cache_miss_rate: Interrupt cache miss rate.

Tested:

  # perf stat -M \
    iotlb_total_hit,iotlb_total_miss,iotlb_miss_rate \
    --per-socket --metric-only -a -j -- sleep 10
  {"socket" : "S0", "counters" : 10,
   "hits  iotlb_total_hit" : "3579249.0",
   "%  iotlb_miss_rate" : "0.0",
   "misses  iotlb_total_miss" : "3.0"}
  {"socket" : "S1", "counters" : 10,
   "hits  iotlb_total_hit" : "0.0",
   "%  iotlb_miss_rate" : "0.0",
   "misses  iotlb_total_miss" : "0.0"}
Assisted-by Gemini:gemini-3.1-pro-preview
---
 tools/perf/pmu-events/intel_metrics.py | 62 ++++++++++++++++++++++++++
 1 file changed, 62 insertions(+)
tools/perf/pmu-events/intel_metrics.py +62 -0
index 52035433b505..c3a5c2965f74 100755
@@ -457,6 +457,67 @@ def IntelIlp() -> MetricGroup:
     ])
 
 
+def IntelIotlb() -> Optional[MetricGroup]:
+  try:
+    total_hit = (
+        Event("UNC_IIO_IOMMU0.4K_HITS")
+        + Event("UNC_IIO_IOMMU0.2M_HITS")
+        + Event("UNC_IIO_IOMMU0.1G_HITS")
+    )
+    total_miss = Event("UNC_IIO_IOMMU0.MISSES")
+  except:
+    return None
+
+  miss_rate = d_ratio(total_miss, total_miss + total_hit)
+  metrics = [
+      Metric("iotlb_total_hit", "IOTLB total hit", total_hit, "hits"),
+      Metric("iotlb_total_miss", "IOTLB total miss", total_miss, "misses"),
+      Metric("iotlb_miss_rate", "IOTLB miss rate", miss_rate, "100%"),
+  ]
+
+  try:
+    interrupt_cache_hit = Event("UNC_IIO_IOMMU3.INT_CACHE_HITS")
+    interrupt_cache_lookup = Event("UNC_IIO_IOMMU3.INT_CACHE_LOOKUPS")
+    interrupt_cache_miss = max(interrupt_cache_lookup - interrupt_cache_hit, 0)
+    interrupt_cache_miss_rate = d_ratio(
+        interrupt_cache_miss, interrupt_cache_miss + interrupt_cache_hit
+    )
+    metrics += [
+        Metric(
+            "iotlb_interrupt_cache_hit",
+            "IOTLB interrupt cache hit",
+            interrupt_cache_hit,
+            "hits",
+        ),
+        Metric(
+            "iotlb_interrupt_cache_miss",
+            "IOTLB interrupt cache miss",
+            interrupt_cache_miss,
+            "misses",
+        ),
+        Metric(
+            "iotlb_interrupt_cache_lookup",
+            "IOTLB interrupt cache lookup",
+            interrupt_cache_lookup,
+            "lookups",
+        ),
+        Metric(
+            "iotlb_interrupt_cache_miss_rate",
+            "IOTLB interrupt cache miss rate",
+            interrupt_cache_miss_rate,
+            "100%",
+        ),
+    ]
+  except:
+    pass
+
+  return MetricGroup(
+      "iotlb",
+      metrics,
+      description="IOMMU TLB metrics",
+  )
+
+
 def IntelL2() -> Optional[MetricGroup]:
     try:
         DC_HIT = Event("L2_RQSTS.DEMAND_DATA_RD_HIT")
@@ -1105,6 +1166,7 @@ def main() -> None:
         IntelCtxSw(),
         IntelFpu(),
         IntelIlp(),
+        IntelIotlb(),
         IntelL2(),
         IntelLdSt(),
         IntelMissLat(),
Signature · 1 line
2.54.0.823.g6e5bcc1fc9-goog