Add IOMMU TLB and interrupt metrics
v2 cover Add IOMMU TLB and interrupt metrics
This patch series adds IOMMU Translation Lookaside Buffer (TLB) and interrupt cache metrics to perf jevents for both AMD and Intel platforms. This enhances I/O performance observability, allowing fleet-wide monitoring of IOMMU overhead.
The changes are split into two patches:
1. perf jevents: Add IOMMU metrics for AMD
- Adds IOMMU TLB and interrupt metrics for Zen 2+ processors using standard AMD IOMMU PMU events.
- Note that pde events on AMD cover both 2M and 1G pages, so 1G pages are implicitly included.
- Added code comments to clarify this hardware detail and fixed indentation to match the file's style.
2. perf jevents: Add IOMMU metrics for Intel
- Adds IOMMU TLB and interrupt metrics using uncore IIO IOMMU events.
- Supports Emerald Rapids (TLB-only) and Granite Rapids (TLB + Interrupt) by dynamically detecting event availability and making interrupt metrics optional.
- Clamped calculated interrupt cache miss metric to zero to prevent negative values due to counter multiplexing or sampling skid.
v2:
Split the changes into separate AMD and Intel commits as requested.
v1: lore.kernel.org/[email protected]
Chun-Tse Shao (2):
perf jevents: Add IOMMU metrics for AMD
perf jevents: Add IOMMU metrics for Intel
tools/perf/pmu-events/amd_metrics.py | 57 +++++++++++++++++++++++ tools/perf/pmu-events/intel_metrics.py | 62 ++++++++++++++++++++++++++ 2 files changed, 119 insertions(+)
--
2.54.0.823.g6e5bcc1fc9-goog
PATCH v2 1/2 perf jevents: Add IOMMU metrics for AMD
Add IOMMU Translation Lookaside Buffer (TLB) and interrupt cache metrics to perf jevents for AMD platforms. This enhances I/O performance observability, allowing fleet-wide monitoring of IOMMU overhead.
These metrics are supported on Zen 2 and newer processors (Rome, Milan, Genoa, Turin) and are implemented using the standard amd_iommu PMU events. The implementation uses the existing _zen_model helper to ensure these are only generated for Zen 2+. Note that the pde events on AMD cover both 2M and 1G pages, so 1G pages are implicitly included in the total hits/misses metrics (sum of pte and pde events).
The following metrics are added:
- iotlb_total_hit: Total IOTLB hits (4K, 2M, 1G pages).
- iotlb_total_miss: Total IOTLB misses.
- iotlb_miss_rate: IOTLB miss rate.
- iotlb_interrupt_cache_hit: Interrupt cache hits.
- iotlb_interrupt_cache_miss: Interrupt cache misses.
- iotlb_interrupt_cache_lookup: Interrupt cache lookups.
- iotlb_interrupt_cache_miss_rate: Interrupt cache miss rate.
Tested:
# perf stat -M \
iotlb_total_hit,iotlb_total_miss,iotlb_miss_rate \
--per-socket --metric-only -a -j -- sleep 10
{"socket" : "S0", "counters" : 10,
"hits iotlb_total_hit" : "3579249.0",
"% iotlb_miss_rate" : "0.0",
"misses iotlb_total_miss" : "3.0"}
{"socket" : "S1", "counters" : 10,
"hits iotlb_total_hit" : "0.0",
"% iotlb_miss_rate" : "0.0",
"misses iotlb_total_miss" : "0.0"}--- tools/perf/pmu-events/amd_metrics.py | 57 ++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+)
tools/perf/pmu-events/amd_metrics.py +57 -0
@@ -265,6 +265,62 @@ def AmdDtlb() -> Optional[MetricGroup]: ], description="Data TLB metrics") +def AmdIotlb() -> Optional[MetricGroup]: + global _zen_model + if _zen_model < 2: + return None + + # On AMD, the pde events cover both 2M and 1G pages. + total_hit = Event("amd_iommu/mem_iommu_tlb_pte_hit/") + Event( + "amd_iommu/mem_iommu_tlb_pde_hit/" + ) + total_miss = Event("amd_iommu/mem_iommu_tlb_pte_mis/") + Event( + "amd_iommu/mem_iommu_tlb_pde_mis/" + ) + miss_rate = d_ratio(total_miss, total_miss + total_hit) + + interrupt_cache_hit = Event("amd_iommu/int_dte_hit/") + interrupt_cache_miss = Event("amd_iommu/int_dte_mis/") + interrupt_cache_lookup = interrupt_cache_hit + interrupt_cache_miss + interrupt_cache_miss_rate = d_ratio( + interrupt_cache_miss, interrupt_cache_miss + interrupt_cache_hit + ) + + return MetricGroup( + "iotlb", + [ + Metric("iotlb_total_hit", "IOTLB total hit", total_hit, "hits"), + Metric("iotlb_total_miss", "IOTLB total miss", total_miss, "misses"), + Metric("iotlb_miss_rate", "IOTLB miss rate", miss_rate, "100%"), + Metric( + "iotlb_interrupt_cache_hit", + "IOTLB interrupt cache hit", + interrupt_cache_hit, + "hits", + ), + Metric( + "iotlb_interrupt_cache_miss", + "IOTLB interrupt cache miss", + interrupt_cache_miss, + "misses", + ), + Metric( + "iotlb_interrupt_cache_lookup", + "IOTLB interrupt cache lookup", + interrupt_cache_lookup, + "lookups", + ), + Metric( + "iotlb_interrupt_cache_miss_rate", + "IOTLB interrupt cache miss rate", + interrupt_cache_miss_rate, + "100%", + ), + ], + description="IOMMU TLB metrics", + ) + + def AmdItlb(): global _zen_model l2h = Event("bp_l1_tlb_miss_l2_tlb_hit", "bp_l1_tlb_miss_l2_hit")
@@ -473,6 +529,7 @@ def main() -> None: AmdBr(), AmdCtxSw(), AmdDtlb(), + AmdIotlb(), AmdItlb(), AmdLdSt(), AmdUpc(),
Signature · 1 line
2.54.0.823.g6e5bcc1fc9-goog
Re: [PATCH v2 1/2] perf jevents: Add IOMMU metrics for AMD
Quoted 33 lines · Chun-Tse Shao
Add IOMMU Translation Lookaside Buffer (TLB) and interrupt cache metrics to perf jevents for AMD platforms. This enhances I/O performance observability, allowing fleet-wide monitoring of IOMMU overhead.
These metrics are supported on Zen 2 and newer processors (Rome, Milan, Genoa, Turin) and are implemented using the standard
amd_iommuPMU events. The implementation uses the existing_zen_modelhelper to ensure these are only generated for Zen 2+. Note that the pde events on AMD cover both 2M and 1G pages, so 1G pages are implicitly included in the total hits/misses metrics (sum of pte and pde events).The following metrics are added:
-iotlb_total_hit: Total IOTLB hits (4K, 2M, 1G pages).
-iotlb_total_miss: Total IOTLB misses.
-iotlb_miss_rate: IOTLB miss rate.
-iotlb_interrupt_cache_hit: Interrupt cache hits.
-iotlb_interrupt_cache_miss: Interrupt cache misses.
-iotlb_interrupt_cache_lookup: Interrupt cache lookups.
-iotlb_interrupt_cache_miss_rate: Interrupt cache miss rate.Tested:
# perf stat -M \ iotlb_total_hit,iotlb_total_miss,iotlb_miss_rate \ --per-socket --metric-only -a -j -- sleep 10 {"socket" : "S0", "counters" : 10, "hits iotlb_total_hit" : "3579249.0", "% iotlb_miss_rate" : "0.0", "misses iotlb_total_miss" : "3.0"} {"socket" : "S1", "counters" : 10, "hits iotlb_total_hit" : "0.0", "% iotlb_miss_rate" : "0.0", "misses iotlb_total_miss" : "0.0"}
Quoted 3 lines
Signed-off-by Chun-Tse Shao <[email protected]>Assisted-by Gemini:gemini-3.1-pro-preview---
Quoted 78 lines
tools/perf/pmu-events/amd_metrics.py | 57 ++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+)tools/perf/pmu-events/amd_metrics.py +57
-0@@ -265,6 +265,62 @@ def AmdDtlb() -> Optional[MetricGroup]: ], description="Data TLB metrics") +def AmdIotlb() -> Optional[MetricGroup]: + global _zen_model + if _zen_model < 2: + return None + + # On AMD, the pde events cover both 2M and 1G pages. + total_hit = Event("amd_iommu/mem_iommu_tlb_pte_hit/") + Event( + "amd_iommu/mem_iommu_tlb_pde_hit/" + ) + total_miss = Event("amd_iommu/mem_iommu_tlb_pte_mis/") + Event( + "amd_iommu/mem_iommu_tlb_pde_mis/" + ) + miss_rate = d_ratio(total_miss, total_miss + total_hit) + + interrupt_cache_hit = Event("amd_iommu/int_dte_hit/") + interrupt_cache_miss = Event("amd_iommu/int_dte_mis/") + interrupt_cache_lookup = interrupt_cache_hit + interrupt_cache_miss + interrupt_cache_miss_rate = d_ratio( + interrupt_cache_miss, interrupt_cache_miss + interrupt_cache_hit + ) + + return MetricGroup( + "iotlb", + [ + Metric("iotlb_total_hit", "IOTLB total hit", total_hit, "hits"), + Metric("iotlb_total_miss", "IOTLB total miss", total_miss, "misses"), + Metric("iotlb_miss_rate", "IOTLB miss rate", miss_rate, "100%"), + Metric( + "iotlb_interrupt_cache_hit", + "IOTLB interrupt cache hit", + interrupt_cache_hit, + "hits", + ), + Metric( + "iotlb_interrupt_cache_miss", + "IOTLB interrupt cache miss", + interrupt_cache_miss, + "misses", + ), + Metric( + "iotlb_interrupt_cache_lookup", + "IOTLB interrupt cache lookup", + interrupt_cache_lookup, + "lookups", + ), + Metric( + "iotlb_interrupt_cache_miss_rate", + "IOTLB interrupt cache miss rate", + interrupt_cache_miss_rate, + "100%", + ), + ], + description="IOMMU TLB metrics", + ) + + def AmdItlb(): global _zen_model l2h = Event("bp_l1_tlb_miss_l2_tlb_hit", "bp_l1_tlb_miss_l2_hit")@@ -473,6 +529,7 @@ def main() -> None: AmdBr(), AmdCtxSw(), AmdDtlb(), + AmdIotlb(), AmdItlb(), AmdLdSt(), AmdUpc(),
Re: [PATCH v2 1/2] perf jevents: Add IOMMU metrics for AMD
Quoted 34 lines · Sandipan Das
Quoted 33 lines · Chun-Tse Shao
Add IOMMU Translation Lookaside Buffer (TLB) and interrupt cache metrics to perf jevents for AMD platforms. This enhances I/O performance observability, allowing fleet-wide monitoring of IOMMU overhead.
These metrics are supported on Zen 2 and newer processors (Rome, Milan, Genoa, Turin) and are implemented using the standard
amd_iommuPMU events. The implementation uses the existing_zen_modelhelper to ensure these are only generated for Zen 2+. Note that the pde events on AMD cover both 2M and 1G pages, so 1G pages are implicitly included in the total hits/misses metrics (sum of pte and pde events).The following metrics are added:
-iotlb_total_hit: Total IOTLB hits (4K, 2M, 1G pages).
-iotlb_total_miss: Total IOTLB misses.
-iotlb_miss_rate: IOTLB miss rate.
-iotlb_interrupt_cache_hit: Interrupt cache hits.
-iotlb_interrupt_cache_miss: Interrupt cache misses.
-iotlb_interrupt_cache_lookup: Interrupt cache lookups.
-iotlb_interrupt_cache_miss_rate: Interrupt cache miss rate.Tested:
# perf stat -M \ iotlb_total_hit,iotlb_total_miss,iotlb_miss_rate \ --per-socket --metric-only -a -j -- sleep 10 {"socket" : "S0", "counters" : 10, "hits iotlb_total_hit" : "3579249.0", "% iotlb_miss_rate" : "0.0", "misses iotlb_total_miss" : "3.0"} {"socket" : "S1", "counters" : 10, "hits iotlb_total_hit" : "0.0", "% iotlb_miss_rate" : "0.0", "misses iotlb_total_miss" : "0.0"}
Quoted 5 lines
Quoted 3 lines
Signed-off-by Chun-Tse Shao <[email protected]>Assisted-by Gemini:gemini-3.1-pro-preview---
Reviewed-by Sandipan Das <[email protected]>
Thanks, applied both patches and collected Ian's reviewed-by, since he provided it for the combined patch and requested that it was split, which this v2 does.
- Arnaldo
PATCH v2 2/2 perf jevents: Add IOMMU metrics for Intel
In reply to Chun-Tse ShaoAdd IOMMU Translation Lookaside Buffer (TLB) and interrupt cache metrics to perf jevents for Intel platforms. This enhances I/O performance observability, allowing fleet-wide monitoring of IOMMU overhead.
These metrics are supported on platforms that expose the required uncore IIO IOMMU events (such as Emerald Rapids and Granite Rapids). The Intel implementation dynamically detects event availability at generation time. It requires at least the TLB events to expose the metric group, while the interrupt cache events are optional. This allows platforms like Emerald Rapids, which lack IOMMU interrupt cache events, to still expose the IOMMU TLB metrics.
The following metrics are added:
- iotlb_total_hit: Total IOTLB hits (4K, 2M, 1G pages).
- iotlb_total_miss: Total IOTLB misses.
- iotlb_miss_rate: IOTLB miss rate.
- iotlb_interrupt_cache_hit: Interrupt cache hits.
- iotlb_interrupt_cache_miss: Interrupt cache misses (calculated as lookup - hit, clamped to zero).
- iotlb_interrupt_cache_lookup: Interrupt cache lookups.
- iotlb_interrupt_cache_miss_rate: Interrupt cache miss rate.
Tested:
# perf stat -M \
iotlb_total_hit,iotlb_total_miss,iotlb_miss_rate \
--per-socket --metric-only -a -j -- sleep 10
{"socket" : "S0", "counters" : 10,
"hits iotlb_total_hit" : "3579249.0",
"% iotlb_miss_rate" : "0.0",
"misses iotlb_total_miss" : "3.0"}
{"socket" : "S1", "counters" : 10,
"hits iotlb_total_hit" : "0.0",
"% iotlb_miss_rate" : "0.0",
"misses iotlb_total_miss" : "0.0"}--- tools/perf/pmu-events/intel_metrics.py | 62 ++++++++++++++++++++++++++ 1 file changed, 62 insertions(+)
tools/perf/pmu-events/intel_metrics.py +62 -0
@@ -457,6 +457,67 @@ def IntelIlp() -> MetricGroup: ]) +def IntelIotlb() -> Optional[MetricGroup]: + try: + total_hit = ( + Event("UNC_IIO_IOMMU0.4K_HITS") + + Event("UNC_IIO_IOMMU0.2M_HITS") + + Event("UNC_IIO_IOMMU0.1G_HITS") + ) + total_miss = Event("UNC_IIO_IOMMU0.MISSES") + except: + return None + + miss_rate = d_ratio(total_miss, total_miss + total_hit) + metrics = [ + Metric("iotlb_total_hit", "IOTLB total hit", total_hit, "hits"), + Metric("iotlb_total_miss", "IOTLB total miss", total_miss, "misses"), + Metric("iotlb_miss_rate", "IOTLB miss rate", miss_rate, "100%"), + ] + + try: + interrupt_cache_hit = Event("UNC_IIO_IOMMU3.INT_CACHE_HITS") + interrupt_cache_lookup = Event("UNC_IIO_IOMMU3.INT_CACHE_LOOKUPS") + interrupt_cache_miss = max(interrupt_cache_lookup - interrupt_cache_hit, 0) + interrupt_cache_miss_rate = d_ratio( + interrupt_cache_miss, interrupt_cache_miss + interrupt_cache_hit + ) + metrics += [ + Metric( + "iotlb_interrupt_cache_hit", + "IOTLB interrupt cache hit", + interrupt_cache_hit, + "hits", + ), + Metric( + "iotlb_interrupt_cache_miss", + "IOTLB interrupt cache miss", + interrupt_cache_miss, + "misses", + ), + Metric( + "iotlb_interrupt_cache_lookup", + "IOTLB interrupt cache lookup", + interrupt_cache_lookup, + "lookups", + ), + Metric( + "iotlb_interrupt_cache_miss_rate", + "IOTLB interrupt cache miss rate", + interrupt_cache_miss_rate, + "100%", + ), + ] + except: + pass + + return MetricGroup( + "iotlb", + metrics, + description="IOMMU TLB metrics", + ) + + def IntelL2() -> Optional[MetricGroup]: try: DC_HIT = Event("L2_RQSTS.DEMAND_DATA_RD_HIT")
@@ -1105,6 +1166,7 @@ def main() -> None: IntelCtxSw(), IntelFpu(), IntelIlp(), + IntelIotlb(), IntelL2(), IntelLdSt(), IntelMissLat(),
Signature · 1 line
2.54.0.823.g6e5bcc1fc9-goog