Skip to contentSkip to conversation

AMD IOMMUv2 Performance Counter patches

PATCH In review Steven L. Kinney 4 participants 12 messages Jan 21, 2013 – Jan 28, 2013

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover AMD IOMMUv2 Performance Counter patches

From: "Steven L. Kinney" <[email protected]>

These patches implement the AMD IOMMUv2.5 Performance Counter functionality via custom perf PMU and implement static counting for various IOMMU translations. The patches address three areas of functionality:

1) Add a PCI quirk for the enablement of IOMMUv2 EFR PC within a specific
AMD family/model in which BIOS has not enabled. Based on code implemented by Andreas Herrmann at AMD Dresden.
2) Extend the AMD IOMMU initialization to include IOMMUv2 PC enablement
and access to IOMMUv2 counter registers. For all AMD family/models that implement IOMMUv2.5 functionality.
3) Code the perf IOMMUv2 PMU to manage IOMMUv2 perf events, which call
the function(s) extending the AMD IOMMU core driver. For all AMD family/models that implement IOMMUv2.5 functionality.

The command-line, to invoke the iommuv2 PMU, is:

perf stat -e iommuv2/config=[data],config1=[data]/{u,r} [command]

For information regarding AMD IOMMU v2.5 PC configuration, see the public specification.

Steven L. Kinney (3):
AMD x86 quirks: Quirk for enabling IOMMUv2 PC feature AMD IOMMUv2 PC resource management hooks
AMD IOMMUv2 PC perf PMU implementation

 arch/x86/kernel/cpu/Makefile                 |    1 +
 arch/x86/kernel/cpu/perf_event_amd_iommuv2.c |  429 ++++++++++++++++++++++++++
 arch/x86/kernel/cpu/perf_event_amd_iommuv2.h |   42 +++
 arch/x86/kernel/quirks.c                     |   17 +
 drivers/iommu/Kconfig                        |   10 +
 drivers/iommu/amd_iommu_init.c               |   99 ++++++
 drivers/iommu/amd_iommu_types.h              |   12 +
 include/linux/pci_ids.h                      |    2 +
 8 files changed, 612 insertions(+)
 create mode 100644 arch/x86/kernel/cpu/perf_event_amd_iommuv2.c
 create mode 100644 arch/x86/kernel/cpu/perf_event_amd_iommuv2.h
Signature · 1 line
1.7.9.5
#1 lore raw

PATCH 1/3 AMD x86 quirks: Quirk for enabling IOMMUv2 PC feature

From: "Steven L. Kinney" <[email protected]>

Original code base by Andreas Herrmann at AMD with minor changes.

---
 arch/x86/kernel/quirks.c |   17 +++++++++++++++++
 include/linux/pci_ids.h  |    2 ++
 2 files changed, 19 insertions(+)
arch/x86/kernel/quirks.c +17 -0
index 26ee48a..ab27b88 100644
@@ -567,3 +567,20 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, PCI_DEVICE_ID_AMD_15H_NB_F5,
 			quirk_amd_nb_node);
 
 #endif
+
+#if defined(CONFIG_PCI) && defined(CONFIG_AMD_IOMMU_V2_PC)
+
+static void amd_force_iommu_v2_pc(struct pci_dev *dev)
+{
+	u32 val;
+
+	pci_read_config_dword(dev, 0x70, &val);
+	val |= (1<<9);
+	pci_write_config_dword(dev, 0x70, val);
+	dev_printk(KERN_DEBUG, &dev->dev, "Enabled IOMMUv2 PC\n");
+}
+
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, PCI_DEVICE_ID_AMD_15H_M10H_IOMMU,
+			amd_force_iommu_v2_pc);
+
+#endif
include/linux/pci_ids.h +2 -0
index 0eb6579..c6ce47b 100644
@@ -396,6 +396,8 @@
 /* AMD RD890 Chipset */
 #define PCI_DEVICE_ID_RD890_IOMMU	0x5a23
 
+#define PCI_DEVICE_ID_AMD_15H_M10H_IOMMU       0x1419
+
 #define PCI_VENDOR_ID_ADL		0x1005
 #define PCI_DEVICE_ID_ADL_2301		0x2301
 
Signature · 1 line
1.7.9.5
maintainer
#2 lore raw

Re: [PATCH 1/3] AMD x86 quirks: Quirk for enabling IOMMUv2 PC feature

Quoted 3 lines · Steven L. Kinney

From: "Steven L. Kinney" <[email protected]>

Original code base by Andreas Herrmann at AMD with minor changes.

Why is it safe to enable this feature when it was disabled by BIOS? Was it disabled for no reason?

Joerg

#3 lore raw

RE: [PATCH 1/3] AMD x86 quirks: Quirk for enabling IOMMUv2 PC feature

Hi Joerg,

It was disabled for a reason that does not pertain to IOMMUv2 PC static counts (reading from the ICounter associated with the selected bank/counter). Testing with perf shows expected results.

Thanks,

Steve

-----Original Message-----
From: Joerg Roedel [mailto:[email protected]]
Sent: Monday, January 28, 2013 8:29 AM
To: Kinney, Steven
Cc: Thomas Gleixner; Ingo Molnar; H. Peter Anvin; [email protected]; Bjorn Helgaas; Greg Kroah-Hartman; Sebastian Andrzej Siewior; Myron Stowe; Hiroshi DOYU; Stephen Warren; Jiri Kosina; Kukjin Kim; [email protected]; [email protected]; Peter Zijlstra; Paul Mackerras; Arnaldo Carvalho de Melo; Thomas Renninger; Andi Kleen; Cyrill Gorcunov
Subject: Re: [PATCH 1/3] AMD x86 quirks: Quirk for enabling IOMMUv2 PC feature

Quoted 3 lines · Steven L. Kinney

From: "Steven L. Kinney" <[email protected]>

Original code base by Andreas Herrmann at AMD with minor changes.

Why is it safe to enable this feature when it was disabled by BIOS? Was it disabled for no reason?

Joerg

maintainer
#4 lore raw

Re: [PATCH 1/3] AMD x86 quirks: Quirk for enabling IOMMUv2 PC feature

Quoted 1 line · Steven

Testing with perf shows expected results.

Can you give me an impression on how the results look like when perf is used? Since the hardware is widely available yet I can't try this myself.

Joerg

#5 lore raw

PATCH 2/3 AMD IOMMUv2 PC resource management hooks

In reply to Steven L. Kinney

From: "Steven L. Kinney" <[email protected]>

Add functionality to check the availability of the AMD IOMMUv2 Performance Counters and export this functionality to other core drivers, such as in this case, a perf IOMMUv2 PMU. This feature is not bound to any specific AMD family/model other than the presence of the IOMMUv2 with PC enabled.

The IOMMUv2 PC support static counting only at this time.

---
 drivers/iommu/Kconfig           |   10 ++++
 drivers/iommu/amd_iommu_init.c  |   99 +++++++++++++++++++++++++++++++++++++++
 drivers/iommu/amd_iommu_types.h |   12 +++++
 3 files changed, 121 insertions(+)
drivers/iommu/Kconfig +10 -0
index e39f9db..4a941dd 100644
@@ -73,6 +73,16 @@ config AMD_IOMMU_V2
 	  hardware. Select this option if you want to use devices that support
 	  the PCI PRI and PASID interface.
 
+# AMD IOMMUv2 Performance Counter support
+config AMD_IOMMU_V2_PC
+	bool "AMD IOMMUv2 Performance Counter (EXPERIMENTAL)"
+	depends on AMD_IOMMU_V2
+	---help---
+	  This option enables support for AMD IOMMUv2 Performance Counters.
+	  Select this option if you want to enable IOMMUv2 Performance
+	  Counter support.
+	  If unsure, say N.
+
 # Intel IOMMU support
 config DMAR_TABLE
 	bool
drivers/iommu/amd_iommu_init.c +99 -0

114 lines not on this page · raw message

drivers/iommu/amd_iommu_types.h +12 -0
index e38ab43..fdf236b 100644
@@ -38,7 +38,12 @@
 #define RLOOKUP_TABLE_ENTRY_SIZE	(sizeof(void *))
 
 /* Length of the MMIO region for the AMD IOMMU */
+#ifdef CONFIG_AMD_IOMMU_V2_PC
+#define MMIO_REGION_LENGTH       0x80000
+#define IOMMU_V2_PC_REG_OFFSET	 0x40000
+#else
 #define MMIO_REGION_LENGTH       0x4000
+#endif
 
 /* Capability offsets used by the driver */
 #define MMIO_CAP_HDR_OFFSET	0x00
@@ -77,6 +82,7 @@
 #define MMIO_STATUS_OFFSET	0x2020
 #define MMIO_PPR_HEAD_OFFSET	0x2030
 #define MMIO_PPR_TAIL_OFFSET	0x2038
+#define MMIO_CNTR_CONF_OFFSET	0x4000
 
 
 /* Extended Feature Bits */
@@ -585,6 +591,12 @@ struct amd_iommu {
 
 	/* The l2 indirect registers */
 	u32 stored_l2[0x83];
+
+#ifdef CONFIG_AMD_IOMMU_V2_PC
+	/* The maximum PC banks and counters/bank (PCSup=1) */
+	u8 max_banks;
+	u8 max_counters;
+#endif
 };
 
 struct devid_map {
Signature · 1 line
1.7.9.5
#6 lore raw

Re: [PATCH 2/3] AMD IOMMUv2 PC resource management hooks

drivers/iommu/amd_iommu_init.c bool amd_iommu_v2_supported(void) · line 2116

Quoted 13 lines · Steven L. Kinney
drivers/iommu/amd_iommu_init.c +13 -0
+u8 amd_iommu_v2_get_max_pc_banks(u16 devid)
+{
+	struct amd_iommu *iommu;
+
+	/* locate the iommu governing the devid */
+	iommu = amd_iommu_rlookup_table[devid];
+
+	if (iommu)
+		return iommu->max_banks;
+
+	return -ENODEV;
+}
+EXPORT_SYMBOL(amd_iommu_v2_get_max_pc_banks);

Hi Steven, may not it be better to return int here, instead of u8?

#7 lore raw

RE: [PATCH 2/3] AMD IOMMUv2 PC resource management hooks

Hi Cyrill,

Agree, will clean this up.

Thanks

-----Original Message-----
From: Cyrill Gorcunov [mailto:[email protected]] On Behalf Of Cyrill Gorcunov
Sent: Monday, January 21, 2013 3:32 PM
To: Kinney, Steven
Cc: Thomas Gleixner; Ingo Molnar; H. Peter Anvin; [email protected]; Joerg Roedel; Bjorn Helgaas; Greg Kroah-Hartman; Sebastian Andrzej Siewior; Myron Stowe; Hiroshi DOYU; Stephen Warren; Jiri Kosina; Kukjin Kim; [email protected]; iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; Peter Zijlstra; Paul Mackerras; Arnaldo Carvalho de Melo; Thomas Renninger; Andi Kleen
Subject: Re: [PATCH 2/3] AMD IOMMUv2 PC resource management hooks

drivers/iommu/amd_iommu_init.c bool amd_iommu_v2_supported(void) · line 2116

Quoted 12 lines · Steven L. Kinney
drivers/iommu/amd_iommu_init.c +12 -0
+u8 amd_iommu_v2_get_max_pc_banks(u16 devid) {
+	struct amd_iommu *iommu;
+
+	/* locate the iommu governing the devid */
+	iommu = amd_iommu_rlookup_table[devid];
+
+	if (iommu)
+		return iommu->max_banks;
+
+	return -ENODEV;
+}
+EXPORT_SYMBOL(amd_iommu_v2_get_max_pc_banks);

Hi Steven, may not it be better to return int here, instead of u8?

#8 lore raw

Re: [PATCH 2/3] AMD IOMMUv2 PC resource management hooks

Quoted 2 lines · Steven

Hi Cyrill,
Agree, will clean this up.

Yeah, but no rush. Lets wait for other opinions as well.

#9 lore raw

PATCH 3/3 AMD IOMMUv2 PC perf PMU implementation

In reply to Steven L. Kinney

From: "Steven L. Kinney" <[email protected]>

Implement a perf PMU to handle IOMMUv2 PC perf events. This PMU will handle static counter perf events relative to the AMD IOMMUv2 Performance Counters.

To invoke the AMD IOMMUv2 PMU issue a perf tool command such as:

./perf stat -e iommuv2/config=<config-data>,config1=<config1-data>/u <command>

For example:

./perf stat -e iommuv2/config=08000000000000005,config1=0/u <command>

The resulting count will be how many IOMMUv2 totsal peripheral memory operations were performed during the command execution window.

---
 arch/x86/kernel/cpu/Makefile                 |    1 +
 arch/x86/kernel/cpu/perf_event_amd_iommuv2.c |  429 ++++++++++++++++++++++++++
 arch/x86/kernel/cpu/perf_event_amd_iommuv2.h |   42 +++
 3 files changed, 472 insertions(+)
 create mode 100644 arch/x86/kernel/cpu/perf_event_amd_iommuv2.c
 create mode 100644 arch/x86/kernel/cpu/perf_event_amd_iommuv2.h
arch/x86/kernel/cpu/Makefile +1 -0
index a0e067d..4872b99 100644
@@ -32,6 +32,7 @@ obj-$(CONFIG_PERF_EVENTS)		+= perf_event.o
 
 ifdef CONFIG_PERF_EVENTS
 obj-$(CONFIG_CPU_SUP_AMD)		+= perf_event_amd.o
+obj-$(CONFIG_AMD_IOMMU_V2_PC)           += perf_event_amd_iommuv2.o
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_p6.o perf_event_knc.o perf_event_p4.o
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_lbr.o perf_event_intel_ds.o perf_event_intel.o
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_uncore.o
arch/x86/kernel/cpu/perf_event_amd_iommuv2.c new file +429 -0

429 lines not on this page · raw message

arch/x86/kernel/cpu/perf_event_amd_iommuv2.h new file +42 -0

42 lines not on this page · raw message

Signature · 1 line
1.7.9.5
#10 lore raw
In reply to Steven L. Kinney
Quoted 16 lines · Steven L. Kinney

From: "Steven L. Kinney" <[email protected]>

These patches implement the AMD IOMMUv2.5 Performance Counter functionality via custom perf PMU and implement static counting for various IOMMU translations. The patches address three areas of functionality:

1) Add a PCI quirk for the enablement of IOMMUv2 EFR PC within a specific
AMD family/model in which BIOS has not enabled. Based on code implemented by Andreas Herrmann at AMD Dresden.
2) Extend the AMD IOMMU initialization to include IOMMUv2 PC enablement
and access to IOMMUv2 counter registers. For all AMD family/models that implement IOMMUv2.5 functionality.
3) Code the perf IOMMUv2 PMU to manage IOMMUv2 perf events, which call
the function(s) extending the AMD IOMMU core driver. For all AMD family/models that implement IOMMUv2.5 functionality.

Quoted 6 lines

The command-line, to invoke the iommuv2 PMU, is:

perf stat -e iommuv2/config=[data],config1=[data]/{u,r} [command]

For information regarding AMD IOMMU v2.5 PC configuration, see the public specification.

Is there a name for it? I see
http://support.amd.com/us/Processor_TechDocs/48882.pdf
but that looks to be for v2.0?

Quoted 26 lines

Steven L. Kinney (3):
AMD x86 quirks: Quirk for enabling IOMMUv2 PC feature AMD IOMMUv2 PC resource management hooks
AMD IOMMUv2 PC perf PMU implementation

 arch/x86/kernel/cpu/Makefile                 |    1 +
 arch/x86/kernel/cpu/perf_event_amd_iommuv2.c |  429 ++++++++++++++++++++++++++
 arch/x86/kernel/cpu/perf_event_amd_iommuv2.h |   42 +++
 arch/x86/kernel/quirks.c                     |   17 +
 drivers/iommu/Kconfig                        |   10 +
 drivers/iommu/amd_iommu_init.c               |   99 ++++++
 drivers/iommu/amd_iommu_types.h              |   12 +
 include/linux/pci_ids.h                      |    2 +
 8 files changed, 612 insertions(+)
 create mode 100644 arch/x86/kernel/cpu/perf_event_amd_iommuv2.c
 create mode 100644 arch/x86/kernel/cpu/perf_event_amd_iommuv2.h
Signature · 7 lines
1.7.9.5


_______________________________________________
iommu mailing list
iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#11 lore raw

Hi Konrad,

Thanks for your response. When writing the driver, at AMD, I am provided an internal IOMMUv2.x specification that may not be out in the public as of yet; I am following up on this and will comment on the availability concerning this version. Also, I checked the v2.0 of the IOMMU Specification, specifically the following:

MMIO Offset [40-7F][0-F]08h IOMMU Counter Source Register - Table 64: Architectural Counter Input Group : This is the CSource and the format can be found in /sys/bus/event_sources/devices/iommuv2/forrmat/ MMIO Offset [40-7F][0-F]20h IOMMU DeviceID Match Register - For the following three registers, the config value defines the Match and config1 value defines the Mask; see the specification v2.0 for details; also see sys/...../format for details. MMIO Offset [40-7F][0-F]10h IOMMU PASID Match Register -
MMIO Offset [40-7F][0-F]18h IOMMU Domain Match Register -

The one caveat between the v2.0 and v2.5 concerns a slight expansion of the CSource register Input Group, but any event defined within v2.0 will execute properly regarding the driver patch(es) sent to the kernel community.

Please let me know, in any way, how I can better this driver code and I will make it so. Also, Will have an answer for you regarding AMD IOMMUv2.5 ASAP.

Thanks again for your comments, appreciated.

-----Original Message-----
From: Konrad Rzeszutek Wilk [mailto:[email protected]]
Sent: Tuesday, January 22, 2013 11:48 AM
To: Kinney, Steven
Cc: Thomas Gleixner; Ingo Molnar; H. Peter Anvin; [email protected]; Joerg Roedel; Kukjin Kim; Peter Zijlstra; Stephen Warren; Greg Kroah-Hartman; [email protected]; iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; Cyrill Gorcunov; Sebastian Andrzej Siewior; Andi Kleen; Paul Mackerras; Arnaldo Carvalho de Melo; Jiri Kosina; Bjorn Helgaas; Myron Stowe; Thomas Renninger
Subject: Re: [PATCH 0/3] AMD IOMMUv2 Performance Counter patches

Quoted 16 lines · Steven L. Kinney

From: "Steven L. Kinney" <[email protected]>

These patches implement the AMD IOMMUv2.5 Performance Counter functionality via custom perf PMU and implement static counting for various IOMMU translations. The patches address three areas of functionality:

1) Add a PCI quirk for the enablement of IOMMUv2 EFR PC within a specific
AMD family/model in which BIOS has not enabled. Based on code implemented by Andreas Herrmann at AMD Dresden.
2) Extend the AMD IOMMU initialization to include IOMMUv2 PC enablement
and access to IOMMUv2 counter registers. For all AMD family/models that implement IOMMUv2.5 functionality.
3) Code the perf IOMMUv2 PMU to manage IOMMUv2 perf events, which call
the function(s) extending the AMD IOMMU core driver. For all AMD family/models that implement IOMMUv2.5 functionality.

Quoted 6 lines

The command-line, to invoke the iommuv2 PMU, is:

perf stat -e iommuv2/config=[data],config1=[data]/{u,r} [command]

For information regarding AMD IOMMU v2.5 PC configuration, see the public specification.

Is there a name for it? I see
http://support.amd.com/us/Processor_TechDocs/48882.pdf
but that looks to be for v2.0?

Quoted 26 lines

Steven L. Kinney (3):
AMD x86 quirks: Quirk for enabling IOMMUv2 PC feature AMD IOMMUv2 PC resource management hooks
AMD IOMMUv2 PC perf PMU implementation

 arch/x86/kernel/cpu/Makefile                 |    1 +
 arch/x86/kernel/cpu/perf_event_amd_iommuv2.c |  429 ++++++++++++++++++++++++++
 arch/x86/kernel/cpu/perf_event_amd_iommuv2.h |   42 +++
 arch/x86/kernel/quirks.c                     |   17 +
 drivers/iommu/Kconfig                        |   10 +
 drivers/iommu/amd_iommu_init.c               |   99 ++++++
 drivers/iommu/amd_iommu_types.h              |   12 +
 include/linux/pci_ids.h                      |    2 +
 8 files changed, 612 insertions(+)
 create mode 100644 arch/x86/kernel/cpu/perf_event_amd_iommuv2.c
 create mode 100644 arch/x86/kernel/cpu/perf_event_amd_iommuv2.h

--
1.7.9.5

_______________________________________________
iommu mailing list iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu