iommu/vt-d: Support performance monitoring for IOMMU
Subsystems (5)
v4 cover iommu/vt-d: Support performance monitoring for IOMMU
From: Kan Liang <[email protected]>
Changes since V3:
- Fix issues found by "./scripts/checkpatch.pl --strict" scripts Doesn't fix the "macros should not use a trailing semicolon" warning, since the macros has to include several struct and functions to further reduce the code duplication.
- To fix the 0 value filter issue, introduce new "*_filter_en" attrs for each filter. The end user must set both filter_en and filter to utilize a filter.
- Re-organize the layout of config1 and config2.
Changes since V2:
- Move ecmd_submit_sync() to iommu.c to avoid #ifdef CONFIG_INTEL_IOMMU
Changes since V1:
- The cap and ecap registers are always in the first page. It's not necessary to use the reg size in dmar_validate_one_drhd(). (Patch 1)
- Move reg_size up and pair it with reg_base_addr in struct dmar_drhd_unit (Patch 1)
- Update the year of Copyright (Patch 2)
- Return 0 if PMS is not supported (Patch 2)
- Refine the comments and add a pr_warn for per-counter capabilities check (Patch 2)
- Remove unnecessary iommu_pmu->num_cntr = i (Patch 2)
- Remove has_ob of ecmd_submit_sync() (Patch 3)
- Remove the helpers from non-INTEL_IOMMU. (Patch 3)
- Still keep #ifdef CONFIG_INTEL_IOMMU for ecmd_submit_sync() to avoid compile warning with non-INTEL_IOMMU config.
- Use pr_warn_once() to replace WARN_ONCE() (Patch 4 & 6)
- Free iommu PMU if it fails to be registered. (Patch 4)
- Remove unnecessary 'handled' variable. (Patch 6)
A performance monitoring infrastructure, perfmon, is introduced with the VT-d Spec 4.0. The purpose of perfmon is to support collection of information about key events occurring during operation of the remapping hardware, to aid performance tuning and debug. The patch series is to support the perfmon for IOMMU.
To facilitate the perfmon support, the patch series also supports two new generic features of VT-d Spec 4.0.
- Support the 'size' field to retrieve the accurate size of the register set for each dmar device from DRHD. (Patch 1)
- Support the new Enhanced Command Interface. (Patch 3)
With the patch series, users can collect the performance data of IOMMU via Linux perf tool. For example,
$ perf stat -e dmar1/iommu_requests,filter_ats_en=0x1,filter_ats=0x1/ -a sleep 1
Performance counter stats for 'system wide':
368,947 dmar1/iommu_requests,filter_ats_en=0x1,filter_ats=0x1/
1.002592074 seconds time elapsed
Kan Liang (7):
iommu/vt-d: Support size of the register set in DRHD iommu/vt-d: Retrieve IOMMU perfmon capability information iommu/vt-d: Support Enhanced Command Interface iommu/vt-d: Add IOMMU perfmon support iommu/vt-d: Support cpumask for IOMMU perfmon iommu/vt-d: Add IOMMU perfmon overflow handler support iommu/vt-d: Enable IOMMU perfmon support
.../sysfs-bus-event_source-devices-iommu | 37 + drivers/iommu/intel/Kconfig | 11 + drivers/iommu/intel/Makefile | 1 + drivers/iommu/intel/dmar.c | 33 +- drivers/iommu/intel/iommu.c | 59 ++ drivers/iommu/intel/iommu.h | 102 +- drivers/iommu/intel/perfmon.c | 877 ++++++++++++++++++ drivers/iommu/intel/perfmon.h | 64 ++ drivers/iommu/intel/svm.c | 2 +- include/acpi/actbl1.h | 2 +- include/linux/cpuhotplug.h | 1 + include/linux/dmar.h | 1 + 12 files changed, 1182 insertions(+), 8 deletions(-) create mode 100644 Documentation/ABI/testing/sysfs-bus-event_source-devices-iommu create mode 100644 drivers/iommu/intel/perfmon.c create mode 100644 drivers/iommu/intel/perfmon.h
Signature · 1 line
2.35.1
PATCH v4 1/7 iommu/vt-d: Support size of the register set in DRHD
From: Kan Liang <[email protected]>
A new field, which indicates the size of the remapping hardware register set for this remapping unit, is introduced in the DMA-remapping hardware unit definition (DRHD) structure with the VT-d Spec 4.0. With this information, SW doesn't need to 'guess' the size of the register set anymore.
Update the struct acpi_dmar_hardware_unit to reflect the field. Store the size of the register set in struct dmar_drhd_unit for each dmar device.
The 'size' information is ResvZ for the old BIOS and platforms. Fall back to the old guessing method. There is nothing changed.
--- drivers/iommu/intel/dmar.c | 11 +++++++---- include/acpi/actbl1.h | 2 +- include/linux/dmar.h | 1 + 3 files changed, 9 insertions(+), 5 deletions(-)
drivers/iommu/intel/dmar.c +7 -4
@@ -427,6 +427,8 @@ static int dmar_parse_one_drhd(struct acpi_dmar_header *header, void *arg)memcpy (dmaru->hdr, header, header->length); dmaru->reg_base_addr = drhd->address; dmaru->segment = drhd->segment; +/* The size of the register set is 2 ^ N 4 KB pages. */ + dmaru->reg_size =1UL << (drhd->size +12 ); dmaru->include_all = drhd->flags &0x1 ;/* BIT0: INCLUDE_ALL */ dmaru->devices = dmar_alloc_dev_scope((void *)(drhd + 1), ((void *)drhd) + drhd->header.length,
@@ -956,17 +958,18 @@ static void unmap_iommu(struct intel_iommu *iommu)/** * map_iommu: map the iommu's registers * @iommu: the iommu to map -* @phys_addr: the physical address of the base resgister +* @drhd: DMA remapping hardware definition structure * * Memory map the iommu's registers. Start w/ a single page, and * possibly expand if that turns out to be insufficent. */ -static int map_iommu (struct intel_iommu *iommu,u64 phys_addr) +static int map_iommu (struct intel_iommu *iommu,struct dmar_drhd_unit *drhd) { +u64 phys_addr = drhd->reg_base_addr;int map_size, err=0 ; iommu->reg_phys = phys_addr; - iommu->reg_size =VTD_PAGE_SIZE ; + iommu->reg_size = drhd->reg_size;if (!request_mem_region (iommu->reg_phys, iommu->reg_size, iommu->name)) {pr_err ("Can't reserve memory\n" );
@@ -1050,7 +1053,7 @@ static int alloc_iommu(struct dmar_drhd_unit *drhd) }sprintf (iommu->name,"dmar%d" , iommu->seq_id); - err =map_iommu (iommu, drhd->reg_base_addr); + err =map_iommu (iommu, drhd);if (err) {pr_err ("Failed to map %s\n" , iommu->name); goto error_free_seq_id;
include/acpi/actbl1.h +1 -1
@@ -802,7 +802,7 @@ struct acpi_dmar_pci_path {struct acpi_dmar_hardware_unit {struct acpi_dmar_header header;u8 flags; -u8 reserved; +u8 size;/* Size of the register set */ u16 segment;u64 address;/* Register Base Address */ };
include/linux/dmar.h +1 -0
@@ -39,6 +39,7 @@ struct dmar_drhd_unit {struct list_head list;/* list of drhd units */ struct acpi_dmar_header *hdr;/* ACPI header */ u64 reg_base_addr;/* register base address*/ +unsigned long reg_size;/* size of register set */ struct dmar_dev_scope *devices;/* target device array */ int devices_cnt;/* target device count */ u16 segment;/* PCI domain */
Signature · 1 line
2.35.1
PATCH v4 2/7 iommu/vt-d: Retrieve IOMMU perfmon capability information
In reply to [email protected]From: Kan Liang <[email protected]>
The performance monitoring infrastructure, perfmon, is to support collection of information about key events occurring during operation of the remapping hardware, to aid performance tuning and debug. Each remapping hardware unit has capability registers that indicate support for performance monitoring features and enumerate the capabilities.
Add alloc_iommu_pmu() to retrieve IOMMU perfmon capability information for each iommu unit. The information is stored in the iommu->pmu data structure. Capability registers are read-only, so it's safe to prefetch and store them in the pmu structure. This could avoid unnecessary VMEXIT when this code is running in the virtualization environment.
Add free_iommu_pmu() to free the saved capability information when freeing the iommu unit.
Add a kernel config option for the IOMMU perfmon feature. Unless a user explicitly uses the perf tool to monitor the IOMMU perfmon event, there isn't any impact for the existing IOMMU. Enable it by default.
--- drivers/iommu/intel/Kconfig | 11 +++ drivers/iommu/intel/Makefile | 1 + drivers/iommu/intel/dmar.c | 7 ++ drivers/iommu/intel/iommu.h | 43 ++++++++- drivers/iommu/intel/perfmon.c | 172 ++++++++++++++++++++++++++++++++++ drivers/iommu/intel/perfmon.h | 40 ++++++++ 6 files changed, 273 insertions(+), 1 deletion(-) create mode 100644 drivers/iommu/intel/perfmon.c create mode 100644 drivers/iommu/intel/perfmon.h
drivers/iommu/intel/Kconfig +11 -0
@@ -96,4 +96,15 @@ config INTEL_IOMMU_SCALABLE_MODE_DEFAULT_ON passing intel_iommu=sm_on to the kernel. If not sure, please use the default value. +config INTEL_IOMMU_PERF_EVENTS + def_bool y + bool "Intel IOMMU performance events" + depends on INTEL_IOMMU && PERF_EVENTS + help + Selecting this option will enable the performance monitoring + infrastructure in the Intel IOMMU. It collects information about + key events occurring during operation of the remapping hardware, + to aid performance tuning and debug. These are available on modern + processors which support Intel VT-d 4.0 and later. + endif # INTEL_IOMMU
drivers/iommu/intel/Makefile +1 -0
@@ -6,3 +6,4 @@ obj-$(CONFIG_DMAR_PERF) += perf.o obj-$(CONFIG_INTEL_IOMMU_DEBUGFS) += debugfs.o obj-$(CONFIG_INTEL_IOMMU_SVM) += svm.o obj-$(CONFIG_IRQ_REMAP) += irq_remapping.o +obj-$(CONFIG_INTEL_IOMMU_PERF_EVENTS) += perfmon.o
drivers/iommu/intel/dmar.c +7 -0
@@ -34,6 +34,7 @@#include "../irq_remapping.h" #include "perf.h" #include "trace.h" +#include "perfmon.h" typedef int (*dmar_res_handler_t )(struct acpi_dmar_header *,void *);struct dmar_res_callback {
@@ -1106,6 +1107,9 @@ static int alloc_iommu(struct dmar_drhd_unit *drhd)if (sts &DMA_GSTS_QIES ) iommu->gcmd |=DMA_GCMD_QIE ; +if (alloc_iommu_pmu (iommu)) +pr_debug ("Cannot alloc PMU for iommu (seq_id = %d)\n" , iommu->seq_id); +raw_spin_lock_init (&iommu->register_lock);/*
@@ -1140,6 +1144,7 @@ static int alloc_iommu(struct dmar_drhd_unit *drhd) err_sysfs:iommu_device_sysfs_remove (&iommu->iommu); err_unmap: +free_iommu_pmu (iommu);unmap_iommu (iommu); error_free_seq_id:ida_free (&dmar_seq_ids, iommu->seq_id);
@@ -1155,6 +1160,8 @@ static void free_iommu(struct intel_iommu *iommu)iommu_device_sysfs_remove (&iommu->iommu); } +free_iommu_pmu (iommu); +if (iommu->irq) {if (iommu->pr_irq) {free_irq (iommu->pr_irq, iommu);
drivers/iommu/intel/iommu.h +42 -1
79 lines not on this page · raw message
drivers/iommu/intel/perfmon.c new file +172 -0
172 lines not on this page · raw message
drivers/iommu/intel/perfmon.h new file +40 -0
@@ -0,0 +1,40 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* +* PERFCFGOFF_REG, PERFFRZOFF_REG +* PERFOVFOFF_REG, PERFCNTROFF_REG +*/ +#define IOMMU_PMU_NUM_OFF_REGS 4 +#define IOMMU_PMU_OFF_REGS_STEP 4 + +#define IOMMU_PMU_CFG_OFFSET 0x100 +#define IOMMU_PMU_CFG_CNTRCAP_OFFSET 0x80 +#define IOMMU_PMU_CFG_CNTREVCAP_OFFSET 0x84 +#define IOMMU_PMU_CFG_SIZE 0x8 +#define IOMMU_PMU_CFG_FILTERS_OFFSET 0x4 + +#define IOMMU_PMU_CAP_REGS_STEP 8 + +#define iommu_cntrcap_pcc (p) ((p) & 0x1) +#define iommu_cntrcap_cw (p) (((p) >> 8) & 0xff) +#define iommu_cntrcap_ios (p) (((p) >> 16) & 0x1) +#define iommu_cntrcap_egcnt (p) (((p) >> 28) & 0xf) + +#define iommu_event_select (p) ((p) & 0xfffffff) +#define iommu_event_group (p) (((p) >> 28) & 0xf) + +#ifdef CONFIG_INTEL_IOMMU_PERF_EVENTS +int alloc_iommu_pmu (struct intel_iommu *iommu); +void free_iommu_pmu (struct intel_iommu *iommu); +#else +static inline int +alloc_iommu_pmu (struct intel_iommu *iommu) +{ +return 0 ; +} + +static inline void +free_iommu_pmu (struct intel_iommu *iommu) +{ +} +#endif /* CONFIG_INTEL_IOMMU_PERF_EVENTS */
Signature · 1 line
2.35.1
Re: [PATCH V4 2/7] iommu/vt-d: Retrieve IOMMU perfmon capability information
Sorry for the late reply, I've just stumbled upon this code and a few things look confusing to me:
On Sat, Jan 28, 2023 at 12:04:23PM -0800, [email protected] wrote:
<...>
Quoted 12 lines
drivers/iommu/intel/perfmon.c new file +12
-0+/* +* Check per-counter capabilities. All counters should have the +* same capabilities on Interrupt on Overflow Support and Counter +* Width. +*/ +for (i =0 ; i < iommu_pmu->num_cntr; i++) { + cap =dmar_readl (iommu_pmu->cfg_reg + + i *IOMMU_PMU_CFG_OFFSET + +IOMMU_PMU_CFG_CNTRCAP_OFFSET ); +if (!iommu_cntrcap_pcc (cap)) +continue ; +
drivers/iommu/intel/perfmon.c line 117
Quoted 10 lines
drivers/iommu/intel/perfmon.c new file +10
-0+/* +* It's possible that some counters have a different +* capability because of e.g., HW bug. Check the corner +* case here and simply drop those counters. +*/ +if ((iommu_cntrcap_cw (cap) != iommu_pmu->cntr_width) || + !iommu_cntrcap_ios (cap)) { + iommu_pmu->num_cntr = i; +pr_warn ("PMU counter capability inconsistent, counter number reduced to %d\n" , + iommu_pmu->num_cntr);
1. AFAICS according to the VT-d spec section 11.4.13.2 (about the global Counter Width in the PERFCAP register): "If per-counter capabilities are supported, the counter width indicated in the PERFCNTRCAP registers overrides this value."
which, I'd assume, means that a per-counter Counter Width is allowed to be different from the global one (i.e. it is not necessarily a HW bug)?
2. By truncating the number of counters here, we disregard all subsequent counters, including possibly valid ones (which would pass this check)?
3. If we disregard all subsequent counters anyway, why don't we break from the loop here?
Quoted 19 lines
drivers/iommu/intel/perfmon.c new file +19
-0+ } + +/* Clear the pre-defined events group */ +for (j =0 ; j < iommu_pmu->num_eg; j++) + iommu_pmu->cntr_evcap[i][j] =0 ; + +/* Override with per-counter event capabilities */ +for (j =0 ; j <iommu_cntrcap_egcnt (cap); j++) { + cap =dmar_readl (iommu_pmu->cfg_reg + i *IOMMU_PMU_CFG_OFFSET + +IOMMU_PMU_CFG_CNTREVCAP_OFFSET + + (j *IOMMU_PMU_OFF_REGS_STEP )); + iommu_pmu->cntr_evcap[i][iommu_event_group (cap)] =iommu_event_select (cap); +/* +* Some events may only be supported by a specific counter. +* Track them in the evcap as well. +*/ + iommu_pmu->evcap[iommu_event_group (cap)] |=iommu_event_select (cap); + } + }
<...>
PATCH v4 3/7 iommu/vt-d: Support Enhanced Command Interface
In reply to [email protected]From: Kan Liang <[email protected]>
The Enhanced Command Register is to submit command and operand of enhanced commands to DMA Remapping hardware. It can supports up to 256 enhanced commands.
There is a HW register to indicate the availability of all 256 enhanced commands. Each bit stands for each command. But there isn't an existing interface to read/write all 256 bits. Introduce the u64 ecmdcap[4] to store the existence of each enhanced command. Read 4 times to get all of them in map_iommu().
Add a helper to facilitate an enhanced command launch. Make sure hardware complete the command. Also add a helper to facilitate the check of PMU essentials. These helpers will be used later.
--- drivers/iommu/intel/dmar.c | 10 +++++++ drivers/iommu/intel/iommu.c | 56 +++++++++++++++++++++++++++++++++++++ drivers/iommu/intel/iommu.h | 33 ++++++++++++++++++++++ 3 files changed, 99 insertions(+)
drivers/iommu/intel/dmar.c +10 -0
@@ -1017,6 +1017,16 @@ static int map_iommu(struct intel_iommu *iommu, struct dmar_drhd_unit *drhd) goto release; } } + +if (cap_ecmds (iommu->cap)) { +int i; + +for (i =0 ; i <DMA_MAX_NUM_ECMDCAP ; i++) { + iommu->ecmdcap[i] =dmar_readq (iommu->reg +DMAR_ECCAP_REG + + i *DMA_ECMD_REG_STEP ); + } + } + err =0 ; goto out;
drivers/iommu/intel/iommu.c +56 -0
59 lines not on this page · raw message
drivers/iommu/intel/iommu.h +33 -0
@@ -130,6 +130,10 @@#define DMAR_PERFOVFOFF_REG 0x318#define DMAR_PERFCNTROFF_REG 0x31c#define DMAR_PERFEVNTCAP_REG 0x380 +#define DMAR_ECMD_REG 0x400 +#define DMAR_ECEO_REG 0x408 +#define DMAR_ECRSP_REG 0x410 +#define DMAR_ECCAP_REG 0x430#define DMAR_VCCAP_REG 0xe30/* Virtual command capability register */ #define DMAR_VCMD_REG 0xe00/* Virtual command register */ #define DMAR_VCRSP_REG 0xe10/* Virtual command response register */
@@ -304,6 +308,26 @@#define DMA_CCMD_SID (s) (((u64)((s) & 0xffff)) << 16)#define DMA_CCMD_DID (d) ((u64)((d) & 0xffff)) +/* ECMD_REG */ +#define DMA_MAX_NUM_ECMD 256 +#define DMA_MAX_NUM_ECMDCAP (DMA_MAX_NUM_ECMD / 64) +#define DMA_ECMD_REG_STEP 8 +#define DMA_ECMD_ENABLE 0xf0 +#define DMA_ECMD_DISABLE 0xf1 +#define DMA_ECMD_FREEZE 0xf4 +#define DMA_ECMD_UNFREEZE 0xf5 +#define DMA_ECMD_OA_SHIFT 16 +#define DMA_ECMD_ECRSP_IP 0x1 +#define DMA_ECMD_ECCAP3 3 +#define DMA_ECMD_ECCAP3_ECNTS BIT_ULL(48) +#define DMA_ECMD_ECCAP3_DCNTS BIT_ULL(49) +#define DMA_ECMD_ECCAP3_FCNTS BIT_ULL(52) +#define DMA_ECMD_ECCAP3_UFCNTS BIT_ULL(53) +#define DMA_ECMD_ECCAP3_ESSENTIAL (DMA_ECMD_ECCAP3_ECNTS | \ + DMA_ECMD_ECCAP3_DCNTS | \ + DMA_ECMD_ECCAP3_FCNTS | \ + DMA_ECMD_ECCAP3_UFCNTS) +/* FECTL_REG */ #define DMA_FECTL_IM (((u32)1) << 31)
@@ -605,6 +629,7 @@ struct intel_iommu {u64 cap;u64 ecap;u64 vccap; +u64 ecmdcap[DMA_MAX_NUM_ECMDCAP ];u32 gcmd;/* Holds TE, EAFL. Don't need SRTP, SFL, WBF */ raw_spinlock_t register_lock;/* protect register handling */ int seq_id;/* sequence id of the iommu */
@@ -840,6 +865,14 @@ extern const struct iommu_ops intel_iommu_ops;extern int intel_iommu_sm;extern int iommu_calculate_agaw (struct intel_iommu *iommu);extern int iommu_calculate_max_sagaw (struct intel_iommu *iommu); +int ecmd_submit_sync (struct intel_iommu *iommu,u8 ecmd,u64 oa,u64 ob); + +static inline bool ecmd_has_pmu_essential (struct intel_iommu *iommu) +{ +return (iommu->ecmdcap[DMA_ECMD_ECCAP3 ] &DMA_ECMD_ECCAP3_ESSENTIAL ) == +DMA_ECMD_ECCAP3_ESSENTIAL ; +} +extern int dmar_disabled;extern int intel_iommu_enabled;#else
Signature · 1 line
2.35.1
PATCH v4 4/7 iommu/vt-d: Add IOMMU perfmon support
In reply to [email protected]From: Kan Liang <[email protected]>
Implement the IOMMU performance monitor capability, which supports the collection of information about key events occurring during operation of the remapping hardware, to aid performance tuning and debug.
The IOMMU perfmon support is implemented as part of the IOMMU driver and interfaces with the Linux perf subsystem.
The IOMMU PMU has the following unique features compared with the other PMUs.
- Support counting. Not support sampling.
- Does not support per-thread counting. The scope is system-wide.
- Support per-counter capability register. The event constraints can be enumerated.
- The available event and event group can also be enumerated.
- Extra Enhanced Commands are introduced to control the counters.
Add a new variable, struct iommu_pmu *pmu, to in the struct intel_iommu to track the PMU related information.
Add iommu_pmu_register() and iommu_pmu_unregister() to register and unregister a IOMMU PMU. The register function setup the IOMMU PMU ops and invoke the standard perf_pmu_register() interface to register a PMU in the perf subsystem. This patch only exposes the functions. The following patch will enable them in the IOMMU driver.
The IOMMU PMUs can be found under /sys/bus/event_source/devices/dmar*
The available filters and event format can be found at the format folder
$ ls /sys/bus/event_source/devices/dmar1/format/ event event_group filter_ats filter_ats_en filter_page_table filter_page_table_en
The supported events can be found at the events folder
$ ls /sys/bus/event_source/devices/dmar1/events/ ats_blocked fs_nonleaf_hit int_cache_hit_posted iommu_mem_blocked iotlb_hit pasid_cache_lookup ss_nonleaf_hit ctxt_cache_hit fs_nonleaf_lookup int_cache_lookup iommu_mrds iotlb_lookup pg_req_posted ss_nonleaf_lookup ctxt_cache_lookup int_cache_hit_nonposted iommu_clocks iommu_requests pasid_cache_hit pw_occupancy
The command below illustrates filter usage with a simple example.
$ perf stat -e dmar1/iommu_requests,filter_ats_en=0x1,filter_ats=0x1/ -a sleep 1
Performance counter stats for 'system wide':
368,947 dmar1/iommu_requests,filter_ats_en=0x1,filter_ats=0x1/
1.002592074 seconds time elapsed
--- .../sysfs-bus-event_source-devices-iommu | 29 + drivers/iommu/intel/iommu.h | 15 + drivers/iommu/intel/perfmon.c | 526 ++++++++++++++++++ drivers/iommu/intel/perfmon.h | 24 + 4 files changed, 594 insertions(+) create mode 100644 Documentation/ABI/testing/sysfs-bus-event_source-devices-iommu
Documentation/ABI/testing/sysfs-bus-event_source-devices-iommu new file +29 -0
@@ -0,0 +1,29 @@ +What: /sys/bus/event_source/devices/dmar*/format +Date: Jan 2023 +KernelVersion: 6.3 +Contact: Kan Liang <[email protected]> +Description: Read-only. Attribute group to describe the magic bits + that go into perf_event_attr.config, + perf_event_attr.config1 or perf_event_attr.config2 for + the IOMMU pmu. (See also + ABI/testing/sysfs-bus-event_source-devices-format). + + Each attribute in this group defines a bit range in + perf_event_attr.config, perf_event_attr.config1, + or perf_event_attr.config2. All supported attributes + are listed below (See the VT-d Spec 4.0 for possible + attribute values):: + + event = "config:0-27" - event ID + event_group = "config:28-31" - event group ID + + filter_requester_en = "config1:0" - Enable Requester ID filter + filter_domain_en = "config1:1" - Enable Domain ID filter + filter_pasid_en = "config1:2" - Enable PASID filter + filter_ats_en = "config1:3" - Enable Address Type filter + filter_page_table_en= "config1:4" - Enable Page Table Level filter + filter_requester_id = "config1:16-31" - Requester ID filter + filter_domain = "config1:32-47" - Domain ID filter + filter_pasid = "config2:0-21" - PASID filter + filter_ats = "config2:24-28" - Address Type filter + filter_page_table = "config2:32-36" - Page Table Level filter
drivers/iommu/intel/iommu.h +15 -0
@@ -22,6 +22,7 @@#include <linux/ioasid.h> #include <linux/bitfield.h> #include <linux/xarray.h> +#include <linux/perf_event.h> #include <asm/cacheflush.h> #include <asm/iommu.h>
@@ -606,6 +607,16 @@ struct dmar_domain {iommu core */ }; +/* +* In theory, the VT-d 4.0 spec can support up to 2 ^ 16 counters. +* But in practice, there are only 14 counters for the existing +* platform. Setting the max number of counters to 64 should be good +* enough for a long time. Also, supporting more than 64 counters +* requires more extras, e.g., extra freeze and overflow registers, +* which is not necessary for now. +*/ +#define IOMMU_PMU_IDX_MAX 64 +struct iommu_pmu {struct intel_iommu *iommu;u32 num_cntr;/* Number of counters */
@@ -620,6 +631,10 @@ struct iommu_pmu {u64 *evcap;/* Indicates all supported events */ u32 **cntr_evcap;/* Supported events of each counter. */ + +struct pmu pmu; +DECLARE_BITMAP (used_mask,IOMMU_PMU_IDX_MAX ); +struct perf_event *event_list[IOMMU_PMU_IDX_MAX ]; };struct intel_iommu {
drivers/iommu/intel/perfmon.c +526 -0
546 lines not on this page · raw message
drivers/iommu/intel/perfmon.h +24 -0
@@ -7,6 +7,14 @@#define IOMMU_PMU_NUM_OFF_REGS 4#define IOMMU_PMU_OFF_REGS_STEP 4 +#define IOMMU_PMU_FILTER_REQUESTER_ID 0x01 +#define IOMMU_PMU_FILTER_DOMAIN 0x02 +#define IOMMU_PMU_FILTER_PASID 0x04 +#define IOMMU_PMU_FILTER_ATS 0x08 +#define IOMMU_PMU_FILTER_PAGE_TABLE 0x10 + +#define IOMMU_PMU_FILTER_EN BIT(31) +#define IOMMU_PMU_CFG_OFFSET 0x100#define IOMMU_PMU_CFG_CNTRCAP_OFFSET 0x80#define IOMMU_PMU_CFG_CNTREVCAP_OFFSET 0x84
@@ -20,12 +28,18 @@#define iommu_cntrcap_ios (p) (((p) >> 16) & 0x1)#define iommu_cntrcap_egcnt (p) (((p) >> 28) & 0xf) +#define IOMMU_EVENT_CFG_EGI_SHIFT 8 +#define IOMMU_EVENT_CFG_ES_SHIFT 32 +#define IOMMU_EVENT_CFG_INT BIT_ULL(1) +#define iommu_event_select (p) ((p) & 0xfffffff)#define iommu_event_group (p) (((p) >> 28) & 0xf)#ifdef CONFIG_INTEL_IOMMU_PERF_EVENTS int alloc_iommu_pmu (struct intel_iommu *iommu);void free_iommu_pmu (struct intel_iommu *iommu); +void iommu_pmu_register (struct intel_iommu *iommu); +void iommu_pmu_unregister (struct intel_iommu *iommu);#else static inline int alloc_iommu_pmu (struct intel_iommu *iommu)
@@ -37,4 +51,14 @@ static inline void free_iommu_pmu(struct intel_iommu *iommu) { } + +static inline void +iommu_pmu_register (struct intel_iommu *iommu) +{ +} + +static inline void +iommu_pmu_unregister (struct intel_iommu *iommu) +{ +}#endif /* CONFIG_INTEL_IOMMU_PERF_EVENTS */
Signature · 1 line
2.35.1
PATCH v4 5/7 iommu/vt-d: Support cpumask for IOMMU perfmon
In reply to [email protected]From: Kan Liang <[email protected]>
The perf subsystem assumes that all counters are by default per-CPU. So the user space tool reads a counter from each CPU. However, the IOMMU counters are system-wide and can be read from any CPU. Here we use a CPU mask to restrict counting to one CPU to handle the issue. (with CPU hotplug notifier to choose a different CPU if the chosen one is taken off-line).
The CPU is exposed to /sys/bus/event_source/devices/dmar*/cpumask for the user space perf tool.
--- .../sysfs-bus-event_source-devices-iommu | 8 ++ drivers/iommu/intel/perfmon.c | 113 ++++++++++++++++-- include/linux/cpuhotplug.h | 1 + 3 files changed, 114 insertions(+), 8 deletions(-)
Documentation/ABI/testing/sysfs-bus-event_source-devices-iommu +8 -0
@@ -27,3 +27,11 @@ Description: Read-only. Attribute group to describe the magic bits filter_pasid = "config2:0-21" - PASID filter filter_ats = "config2:24-28" - Address Type filter filter_page_table = "config2:32-36" - Page Table Level filter + +What: /sys/bus/event_source/devices/dmar*/cpumask +Date: Jan 2023 +KernelVersion: 6.3 +Contact: Kan Liang <[email protected]> +Description: Read-only. This file always returns the CPU to which the + IOMMU pmu is bound for access to all IOMMU pmu performance + monitoring events.
drivers/iommu/intel/perfmon.c +105 -8
134 lines not on this page · raw message
include/linux/cpuhotplug.h +1 -0
@@ -221,6 +221,7 @@ enum cpuhp_state {CPUHP_AP_PERF_X86_CQM_ONLINE ,CPUHP_AP_PERF_X86_CSTATE_ONLINE ,CPUHP_AP_PERF_X86_IDXD_ONLINE , +CPUHP_AP_PERF_X86_IOMMU_PERF_ONLINE ,CPUHP_AP_PERF_S390_CF_ONLINE ,CPUHP_AP_PERF_S390_SF_ONLINE , CPUHP_AP_PERF_ARM_CCI_ONLINE,
Signature · 1 line
2.35.1
PATCH v4 6/7 iommu/vt-d: Add IOMMU perfmon overflow handler support
In reply to [email protected]From: Kan Liang <[email protected]>
While enabled to count events and an event occurrence causes the counter value to increment and roll over to or past zero, this is termed a counter overflow. The overflow can trigger an interrupt. The IOMMU perfmon needs to handle the case properly.
New HW IRQs are allocated for each IOMMU device for perfmon. The IRQ IDs are after the SVM range.
In the overflow handler, the counter is not frozen. It's very unlikely that the same counter overflows again during the period. But it's possible that other counters overflow at the same time. Read the overflow register at the end of the handler and check whether there are more.
--- drivers/iommu/intel/dmar.c | 2 + drivers/iommu/intel/iommu.h | 11 ++++- drivers/iommu/intel/perfmon.c | 82 +++++++++++++++++++++++++++++++++++ drivers/iommu/intel/svm.c | 2 +- 4 files changed, 95 insertions(+), 2 deletions(-)
drivers/iommu/intel/dmar.c +2 -0
@@ -1879,6 +1879,8 @@ static inline int dmar_msi_reg(struct intel_iommu *iommu, int irq)return DMAR_FECTL_REG ; else if (iommu->pr_irq == irq) return DMAR_PECTL_REG; + else if (iommu->perf_irq == irq) + return DMAR_PERFINTRCTL_REG; elseBUG (); }
drivers/iommu/intel/iommu.h +10 -1
@@ -130,6 +130,8 @@#define DMAR_PERFCFGOFF_REG 0x310#define DMAR_PERFOVFOFF_REG 0x318#define DMAR_PERFCNTROFF_REG 0x31c +#define DMAR_PERFINTRSTS_REG 0x324 +#define DMAR_PERFINTRCTL_REG 0x328#define DMAR_PERFEVNTCAP_REG 0x380#define DMAR_ECMD_REG 0x400#define DMAR_ECEO_REG 0x408
@@ -357,6 +359,9 @@#define DMA_VCS_PAS ((u64)1) +/* PERFINTRSTS_REG */ +#define DMA_PERFINTRSTS_PIS ((u32)1) +#define IOMMU_WAIT_OP (iommu, offset, op, cond, sts) \ do { \ cycles_t start_time = get_cycles(); \
@@ -635,8 +640,12 @@ struct iommu_pmu {struct pmu pmu;DECLARE_BITMAP (used_mask,IOMMU_PMU_IDX_MAX );struct perf_event *event_list[IOMMU_PMU_IDX_MAX ]; +unsigned char irq_name[16 ]; }; +#define IOMMU_IRQ_ID_OFFSET_PRQ (DMAR_UNITS_SUPPORTED) +#define IOMMU_IRQ_ID_OFFSET_PERF (2 * DMAR_UNITS_SUPPORTED) +struct intel_iommu {void __iomem *reg;/* Pointer to hardware regs, virtual addr */ u64 reg_phys;/* physical address of hw register set */
@@ -650,7 +659,7 @@ struct intel_iommu {int seq_id;/* sequence id of the iommu */ int agaw;/* agaw of this iommu */ int msagaw;/* max sagaw of this iommu */ -unsigned int irq, pr_irq; +unsigned int irq, pr_irq, perf_irq;u16 segment;/* PCI segment# */ unsigned char name[13 ];/* Device Name */
drivers/iommu/intel/perfmon.c +82 -0
108 lines not on this page · raw message
drivers/iommu/intel/svm.c +1 -1
@@ -78,7 +78,7 @@ int intel_svm_enable_prq(struct intel_iommu *iommu) } iommu->prq =page_address (pages); - irq =dmar_alloc_hwirq (DMAR_UNITS_SUPPORTED + iommu->seq_id, iommu->node, iommu); + irq =dmar_alloc_hwirq (IOMMU_IRQ_ID_OFFSET_PRQ + iommu->seq_id, iommu->node, iommu);if (irq <=0 ) {pr_err ("IOMMU: %s: Failed to create IRQ vector for page request queue\n" , iommu->name);
Signature · 1 line
2.35.1
PATCH v4 7/7 iommu/vt-d: Enable IOMMU perfmon support
In reply to [email protected]From: Kan Liang <[email protected]>
Register and enable an IOMMU perfmon for each active IOMMU device.
The failure of IOMMU perfmon registration doesn't impact other functionalities of an IOMMU device.
--- drivers/iommu/intel/dmar.c | 3 +++ drivers/iommu/intel/iommu.c | 3 +++ 2 files changed, 6 insertions(+)
drivers/iommu/intel/dmar.c +3 -0
@@ -1144,6 +1144,8 @@ static int alloc_iommu(struct dmar_drhd_unit *drhd) err =iommu_device_register (&iommu->iommu, &intel_iommu_ops,NULL );if (err) goto err_sysfs; + +iommu_pmu_register (iommu); } drhd->iommu = iommu;
@@ -1166,6 +1168,7 @@ static int alloc_iommu(struct dmar_drhd_unit *drhd)static void free_iommu (struct intel_iommu *iommu) {if (intel_iommu_enabled && !iommu->drhd->ignored) { +iommu_pmu_unregister (iommu);iommu_device_unregister (&iommu->iommu);iommu_device_sysfs_remove (&iommu->iommu); }
drivers/iommu/intel/iommu.c +3 -0
@@ -29,6 +29,7 @@#include "../iommu-sva.h" #include "pasid.h" #include "cap_audit.h" +#include "perfmon.h" #define ROOT_SIZE VTD_PAGE_SIZE#define CONTEXT_SIZE VTD_PAGE_SIZE
@@ -4013,6 +4014,8 @@ int __init intel_iommu_init(void) intel_iommu_groups, "%s", iommu->name);iommu_device_register (&iommu->iommu, &intel_iommu_ops,NULL ); + +iommu_pmu_register (iommu); }up_read (&dmar_global_lock);
Signature · 1 line
2.35.1
Baolu Lu
maintainerQuoted 57 lines · On 2023/1/29 4:04, [email protected] wrote:
From: Kan Liang<[email protected]>
Changes since V3:
- Fix issues found by "./scripts/checkpatch.pl--strict" scripts Doesn't fix the "macros should not use a trailing semicolon" warning, since the macros has to include several struct and functions to further reduce the code duplication.
- To fix the 0 value filter issue, introduce new "*_filter_en" attrs for each filter. The end user must set bothfilter_enand filter to utilize a filter.
- Re-organize the layout of config1 and config2.Changes since V2:
- Moveecmd_submit_sync()toiommu.cto avoid #ifdefCONFIG_INTEL_IOMMUChanges since V1:
- The cap and ecap registers are always in the first page. It's not necessary to use the reg size indmar_validate_one_drhd(). (Patch 1)
- Movereg_sizeup and pair it withreg_base_addrin structdmar_drhd_unit(Patch 1)
- Update the year of Copyright (Patch 2)
- Return 0 if PMS is not supported (Patch 2)
- Refine the comments and add apr_warnfor per-counter capabilities check (Patch 2)
- Remove unnecessaryiommu_pmu->num_cntr= i (Patch 2)
- Removehas_obofecmd_submit_sync()(Patch 3)
- Remove the helpers from non-INTEL_IOMMU. (Patch 3)
- Still keep #ifdefCONFIG_INTEL_IOMMUforecmd_submit_sync()to avoid compile warning with non-INTEL_IOMMU config.
- Usepr_warn_once()to replaceWARN_ONCE()(Patch 4 & 6)
- Free iommu PMU if it fails to be registered. (Patch 4)
- Remove unnecessary 'handled' variable. (Patch 6)A performance monitoring infrastructure, perfmon, is introduced with the VT-d Spec 4.0. The purpose of perfmon is to support collection of information about key events occurring during operation of the remapping hardware, to aid performance tuning and debug. The patch series is to support the perfmon for IOMMU.
To facilitate the perfmon support, the patch series also supports two new generic features of VT-d Spec 4.0.
- Support the 'size' field to retrieve the accurate size of the register set for each dmar device from DRHD. (Patch 1)
- Support the new Enhanced Command Interface. (Patch 3)With the patch series, users can collect the performance data of IOMMU via Linux perf tool. For example,
$ perf stat -e dmar1/iommu_requests,filter_ats_en=0x1,filter_ats=0x1/ -a sleep 1Performance counter stats for 'system wide':
368,947 dmar1/iommu_requests,filter_ats_en=0x1,filter_ats=0x1/1.002592074 seconds time elapsed
Quoted 8 lines
Kan Liang (7):
iommu/vt-d: Support size of the register set in DRHD iommu/vt-d: Retrieve IOMMU perfmon capability information iommu/vt-d: Support Enhanced Command Interface iommu/vt-d: Add IOMMU perfmon support iommu/vt-d: Support cpumask for IOMMU perfmon iommu/vt-d: Add IOMMU perfmon overflow handler support iommu/vt-d: Enable IOMMU perfmon support
Patches queued for v6.3. Thank you!
https://lore.kernel.org/linux-iommu/[email protected]/
Best regards,
baolu