Skip to contentSkip to conversation

iommu/amd: SVA Support (Part 4) - SVA and IOPF

PATCH Applied Vasant Hegde 3 participants 24 messages Apr 18, 2024 – May 6, 2024

Subsystems (2)
lore
Share & tools
lore

Vasant Hegde

author maintainer
#0 lore raw

v8 cover iommu/amd: SVA Support (Part 4) - SVA and IOPF

This is part 4 of the 4-part series to introduce Share Virtual Address (SVA) support for devices, which can support PCI ATS, PASID and PRI capabilities. These devices are referred to as SVA-capable devices in this series.

It contains the following enhancements:

* Patch 1 - 3:
Rename, add support to enable/disable features, update DTE etc.

* Patch 4 - 12:
Add IOPF support

* Patch 13 - 15:
Introduce SVA support

This patch series is based on top of v6.9-rc4.

This is also available at github :
https://github.com/AMDESE/linux-iommu/tree/iommu_sva_part4_v8_v6.9_rc4

Thanks everyone who reviewed previous version and provided valuable feedbacks.

Changes from v7 -> v8:
- Rebased on top of v6.9-rc4
- Added "iommu.h" to ppr.c to fix build error

v7: https://lore.kernel.org/linux-iommu/[email protected]/T/#t

Changes from v6 -> v7:
- Rebased on top of iommu/next
- Trivial white space fix
- Added Reviewed-by tag

v6: https://lore.kernel.org/linux-iommu/[email protected]/T/#t

Changes from v5 -> v6:
- Rebased on top of SVA Part3 v7
- Reorganized, added few patches
- Patch to track per device max PASIDs
- Patch to setup GCR3 table for SVA capable domain
- Patch to enable PASID/PRI only when domain is capable of PRI support
- Dropped below patches as they are not needed anymore
- iommu/amd: Add GCR3 [un]initialization function
- iommu/amd: Add support for add/remove device for IOPF
- Rework sev[/remove_dev]_dev_pasid() functions
- Dropped late gcr3 setup patch

v5: https://lore.kernel.org/linux-iommu/[email protected]/T/#t

Changes from v4 -> v5:
- Rebased on top of v6.7-rc8 + SVA Part3 patches
- Few minor improvements like renaming structure name for better, introducing macros, etc

v4: https://lore.kernel.org/linux-iommu/[email protected]/

Changes from v3 -> v4:
- Moved amd_iommu_dev_update_dte() after set/clear_dte() so that we can avoid forward declaration
- Dropped "iommu/amd: Do not override PASID entry in GCR3 table"
- Added patch to fix PPR interrupt processing logic
- Renamed enable_iommus_v2() -> enable_iommus_ppr()
- Added ops->domain_alloc_sva()
- Added domain_alloc_sva() support and reorganize SVA patches
- In error path iommu_call_iopf_notifier() calls amd_iommu_complete_ppr() instead of amd_iommu_page_response()

v3: https://lore.kernel.org/linux-iommu/[email protected]/T/#t

Changes from v2 -> v3:
- Rename sva.c -> pasid.c
- Changed amd_iommu_sva_supported() -> amd_iommu_pasid_supported()
- Added patch to update/flush DTE
- Rework part of SVA support
- Move IOPF enablement to PASID bind time

v2: https://lore.kernel.org/linux-iommu/[email protected]/T/#u

Changes from v1 -> v2:
- Added new patch to fix PASID override issue in GCR3 table
- Complete rework of SVA code on top of Tina's SVA series.
- Rework SVA enable code
- Reworked IOPF handler code on top of Baolu's IOPF improvement series.

v1: https://lore.kernel.org/linux-iommu/[email protected]/T/#t

Jason Gunthorpe (1):
iommu: Add ops->domain_alloc_sva()

Suravee Suthikulpanit (3):
iommu/amd: Move PPR-related functions into ppr.c iommu/amd: Define per-IOMMU iopf_queue iommu/amd: Add support for page response

Vasant Hegde (9):
iommu/amd: Rename amd_iommu_v2_supported() as amd_iommu_pasid_supported() iommu/amd: Introduce per device DTE update function iommu/amd: Fix PPR interrupt processing logic
iommu/amd: Introduce iommu_dev_data.max_pasids
iommu/amd: Setup GCR3 table in advance if domain is SVA capable iommu/amd: Enable PCI features based on attached domain capability iommu/amd: Add support for enable/disable IOPF
iommu/amd: Initial SVA support for AMD IOMMU
iommu/amd: Add SVA domain support

Wei Huang (2):
iommu/amd: Add support for enabling/disabling IOMMU features iommu/amd: Add IO page fault notifier handler

 drivers/iommu/amd/Kconfig           |   2 +
 drivers/iommu/amd/Makefile          |   2 +-
 drivers/iommu/amd/amd_iommu.h       |  49 ++++-
 drivers/iommu/amd/amd_iommu_types.h |  33 ++++
 drivers/iommu/amd/init.c            |  85 +++------
 drivers/iommu/amd/iommu.c           | 283 +++++++++++++++++----------
 drivers/iommu/amd/pasid.c           | 202 ++++++++++++++++++++
 drivers/iommu/amd/ppr.c             | 286 ++++++++++++++++++++++++++++
 drivers/iommu/iommu-sva.c           |  16 +-
 include/linux/iommu.h               |   3 +
 10 files changed, 788 insertions(+), 173 deletions(-)
 create mode 100644 drivers/iommu/amd/pasid.c
 create mode 100644 drivers/iommu/amd/ppr.c
Signature · 1 line
2.31.1
author maintainer
#1 lore raw

PATCH v8 1/15 iommu/amd: Rename amd_iommu_v2_supported() as amd_iommu_pasid_supported()

To reflect its usage. No functional changes intended.

---
 drivers/iommu/amd/amd_iommu.h | 2 +-
 drivers/iommu/amd/init.c      | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
drivers/iommu/amd/amd_iommu.h +1 -1
index f482aab420f7..d77660551f2a 100644
@@ -38,7 +38,7 @@ extern int amd_iommu_guest_ir;
 extern enum io_pgtable_fmt amd_iommu_pgtable;
 extern int amd_iommu_gpt_level;
 
-bool amd_iommu_v2_supported(void);
+bool amd_iommu_pasid_supported(void);
 
 /* Device capabilities */
 int amd_iommu_pdev_enable_cap_pri(struct pci_dev *pdev);
drivers/iommu/amd/init.c +1 -1
index ac6754a85f35..5687c19825d7 100644
@@ -3690,7 +3690,7 @@ __setup("ivrs_ioapic",		parse_ivrs_ioapic);
 __setup("ivrs_hpet",		parse_ivrs_hpet);
 __setup("ivrs_acpihid",		parse_ivrs_acpihid);
 
-bool amd_iommu_v2_supported(void)
+bool amd_iommu_pasid_supported(void)
 {
 	/* CPU page table size should match IOMMU guest page table size */
 	if (cpu_feature_enabled(X86_FEATURE_LA57) &&
Signature · 1 line
2.31.1
author maintainer
#2 lore raw

PATCH v8 2/15 iommu/amd: Introduce per device DTE update function

In reply to Vasant Hegde

Consolidate per device update and flush logic into separate function. Also make it as global function as it will be used in subsequent series to update the DTE.

---
 drivers/iommu/amd/amd_iommu.h |  1 +
 drivers/iommu/amd/iommu.c     | 26 ++++++++++++++++++--------
 2 files changed, 19 insertions(+), 8 deletions(-)
drivers/iommu/amd/amd_iommu.h +1 -0
index d77660551f2a..98aa3ce8473f 100644
@@ -56,6 +56,7 @@ int amd_iommu_clear_gcr3(struct iommu_dev_data *dev_data, ioasid_t pasid);
 void amd_iommu_flush_all_caches(struct amd_iommu *iommu);
 void amd_iommu_update_and_flush_device_table(struct protection_domain *domain);
 void amd_iommu_domain_update(struct protection_domain *domain);
+void amd_iommu_dev_update_dte(struct iommu_dev_data *dev_data, bool set);
 void amd_iommu_domain_flush_complete(struct protection_domain *domain);
 void amd_iommu_domain_flush_pages(struct protection_domain *domain,
 				  u64 address, size_t size);
drivers/iommu/amd/iommu.c +18 -8
index e692217fcb28..394623abbaa4 100644
@@ -2002,6 +2002,21 @@ static void clear_dte_entry(struct amd_iommu *iommu, u16 devid)
 	amd_iommu_apply_erratum_63(iommu, devid);
 }
 
+/* Update and flush DTE for the given device */
+void amd_iommu_dev_update_dte(struct iommu_dev_data *dev_data, bool set)
+{
+	struct amd_iommu *iommu = get_amd_iommu_from_dev(dev_data->dev);
+
+	if (set)
+		set_dte_entry(iommu, dev_data);
+	else
+		clear_dte_entry(iommu, dev_data->devid);
+
+	clone_aliases(iommu, dev_data->dev);
+	device_flush_dte(dev_data);
+	iommu_completion_wait(iommu);
+}
+
 static int do_attach(struct iommu_dev_data *dev_data,
 		     struct protection_domain *domain)
 {
@@ -2036,10 +2051,7 @@ static int do_attach(struct iommu_dev_data *dev_data,
 	}
 
 	/* Update device table */
-	set_dte_entry(iommu, dev_data);
-	clone_aliases(iommu, dev_data->dev);
-
-	device_flush_dte(dev_data);
+	amd_iommu_dev_update_dte(dev_data, true);
 
 	return ret;
 }
@@ -2058,11 +2070,9 @@ static void do_detach(struct iommu_dev_data *dev_data)
 	/* Update data structures */
 	dev_data->domain = NULL;
 	list_del(&dev_data->list);
-	clear_dte_entry(iommu, dev_data->devid);
-	clone_aliases(iommu, dev_data->dev);
 
-	/* Flush the DTE entry */
-	device_flush_dte(dev_data);
+	/* Clear DTE and flush the entry */
+	amd_iommu_dev_update_dte(dev_data, false);
 
 	/* Flush IOTLB and wait for the flushes to finish */
 	amd_iommu_domain_flush_all(domain);
Signature · 1 line
2.31.1
author maintainer
#3 lore raw

PATCH v8 3/15 iommu/amd: Add support for enabling/disabling IOMMU features

In reply to Vasant Hegde

From: Wei Huang <[email protected]>

Add support for struct iommu_ops.dev_{enable/disable}_feat. Please note that the empty feature switches will be populated by subsequent patches.

---
 drivers/iommu/amd/iommu.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
drivers/iommu/amd/iommu.c +28 -0
index 394623abbaa4..a22ecd8479db 100644
@@ -2791,6 +2791,32 @@ static const struct iommu_dirty_ops amd_dirty_ops = {
 	.read_and_clear_dirty = amd_iommu_read_and_clear_dirty,
 };
 
+static int amd_iommu_dev_enable_feature(struct device *dev,
+					enum iommu_dev_features feat)
+{
+	int ret;
+
+	switch (feat) {
+	default:
+		ret = -EINVAL;
+		break;
+	}
+	return ret;
+}
+
+static int amd_iommu_dev_disable_feature(struct device *dev,
+					 enum iommu_dev_features feat)
+{
+	int ret;
+
+	switch (feat) {
+	default:
+		ret = -EINVAL;
+		break;
+	}
+	return ret;
+}
+
 const struct iommu_ops amd_iommu_ops = {
 	.capable = amd_iommu_capable,
 	.domain_alloc = amd_iommu_domain_alloc,
@@ -2803,6 +2829,8 @@ const struct iommu_ops amd_iommu_ops = {
 	.is_attach_deferred = amd_iommu_is_attach_deferred,
 	.pgsize_bitmap	= AMD_IOMMU_PGSIZES,
 	.def_domain_type = amd_iommu_def_domain_type,
+	.dev_enable_feat = amd_iommu_dev_enable_feature,
+	.dev_disable_feat = amd_iommu_dev_disable_feature,
 	.default_domain_ops = &(const struct iommu_domain_ops) {
 		.attach_dev	= amd_iommu_attach_device,
 		.map_pages	= amd_iommu_map_pages,
Signature · 1 line
2.31.1
author maintainer
#4 lore raw

PATCH v8 4/15 iommu/amd: Move PPR-related functions into ppr.c

In reply to Vasant Hegde

From: Suravee Suthikulpanit <[email protected]>

In preparation to subsequent PPR-related patches, and also remove static declaration for certain helper functions so that it can be reused in other files.

Also rename below functions:
alloc_ppr_log -> amd_iommu_alloc_ppr_log
iommu_enable_ppr_log -> amd_iommu_enable_ppr_log
free_ppr_log -> amd_iommu_free_ppr_log
iommu_poll_ppr_log -> amd_iommu_poll_ppr_log

---
 drivers/iommu/amd/Makefile    |   2 +-
 drivers/iommu/amd/amd_iommu.h |  17 ++++-
 drivers/iommu/amd/init.c      |  65 +++----------------
 drivers/iommu/amd/iommu.c     |  55 +---------------
 drivers/iommu/amd/ppr.c       | 114 ++++++++++++++++++++++++++++++++++
 5 files changed, 139 insertions(+), 114 deletions(-)
 create mode 100644 drivers/iommu/amd/ppr.c
drivers/iommu/amd/Makefile +1 -1
index f454fbb1569e..93b11b6d764f 100644
@@ -1,3 +1,3 @@
 # SPDX-License-Identifier: GPL-2.0-only
-obj-$(CONFIG_AMD_IOMMU) += iommu.o init.o quirks.o io_pgtable.o io_pgtable_v2.o
+obj-$(CONFIG_AMD_IOMMU) += iommu.o init.o quirks.o io_pgtable.o io_pgtable_v2.o ppr.o
 obj-$(CONFIG_AMD_IOMMU_DEBUGFS) += debugfs.o
drivers/iommu/amd/amd_iommu.h +14 -3
index 98aa3ce8473f..159e9a43aa61 100644
@@ -17,10 +17,16 @@ irqreturn_t amd_iommu_int_thread_pprlog(int irq, void *data);
 irqreturn_t amd_iommu_int_thread_galog(int irq, void *data);
 irqreturn_t amd_iommu_int_handler(int irq, void *data);
 void amd_iommu_apply_erratum_63(struct amd_iommu *iommu, u16 devid);
+void amd_iommu_restart_log(struct amd_iommu *iommu, const char *evt_type,
+			   u8 cntrl_intr, u8 cntrl_log,
+			   u32 status_run_mask, u32 status_overflow_mask);
 void amd_iommu_restart_event_logging(struct amd_iommu *iommu);
 void amd_iommu_restart_ga_log(struct amd_iommu *iommu);
 void amd_iommu_restart_ppr_log(struct amd_iommu *iommu);
 void amd_iommu_set_rlookup_table(struct amd_iommu *iommu, u16 devid);
+void iommu_feature_enable(struct amd_iommu *iommu, u8 bit);
+void *__init iommu_alloc_4k_pages(struct amd_iommu *iommu,
+				  gfp_t gfp, size_t size);
 
 #ifdef CONFIG_AMD_IOMMU_DEBUGFS
 void amd_iommu_debugfs_setup(struct amd_iommu *iommu);
@@ -49,6 +55,14 @@ int amd_iommu_set_gcr3(struct iommu_dev_data *dev_data,
 		       ioasid_t pasid, unsigned long gcr3);
 int amd_iommu_clear_gcr3(struct iommu_dev_data *dev_data, ioasid_t pasid);
 
+/* PPR */
+int __init amd_iommu_alloc_ppr_log(struct amd_iommu *iommu);
+void __init amd_iommu_free_ppr_log(struct amd_iommu *iommu);
+void amd_iommu_enable_ppr_log(struct amd_iommu *iommu);
+void amd_iommu_poll_ppr_log(struct amd_iommu *iommu);
+int amd_iommu_complete_ppr(struct pci_dev *pdev, u32 pasid,
+			   int status, int tag);
+
 /*
  * This function flushes all internal caches of
  * the IOMMU used by this driver.
@@ -74,9 +88,6 @@ static inline int amd_iommu_create_irq_domain(struct amd_iommu *iommu)
 }
 #endif
 
-int amd_iommu_complete_ppr(struct pci_dev *pdev, u32 pasid,
-			   int status, int tag);
-
 static inline bool is_rd890_iommu(struct pci_dev *pdev)
 {
 	return (pdev->vendor == PCI_VENDOR_ID_ATI) &&
drivers/iommu/amd/init.c +9 -56

113 lines not on this page · raw message

drivers/iommu/amd/iommu.c +1 -54

67 lines not on this page · raw message

drivers/iommu/amd/ppr.c new file +114 -0

114 lines not on this page · raw message

Signature · 1 line
2.31.1
author maintainer
#5 lore raw

PATCH v8 5/15 iommu/amd: Fix PPR interrupt processing logic

In reply to Vasant Hegde

* Do not re-read ppr head pointer as its just updated by the driver.

* Do not read PPR buffer tail pointer inside while loop. If IOMMU generates PPR events continuously then completing interrupt processing takes long time. In worst case it may cause infinite loop.

---
 drivers/iommu/amd/ppr.c | 4 ----
 1 file changed, 4 deletions(-)
drivers/iommu/amd/ppr.c +0 -4
index 1f76ed549ec1..65db1745f40c 100644
@@ -106,9 +106,5 @@ void amd_iommu_poll_ppr_log(struct amd_iommu *iommu)
 		writel(head, iommu->mmio_base + MMIO_PPR_HEAD_OFFSET);
 
 		/* TODO: PPR Handler will be added when we add IOPF support */
-
-		/* Refresh ring-buffer information */
-		head = readl(iommu->mmio_base + MMIO_PPR_HEAD_OFFSET);
-		tail = readl(iommu->mmio_base + MMIO_PPR_TAIL_OFFSET);
 	}
 }
Signature · 1 line
2.31.1
author maintainer
#6 lore raw

PATCH v8 6/15 iommu/amd: Introduce iommu_dev_data.max_pasids

In reply to Vasant Hegde

This variable will track the number of PASIDs supported by the device. If IOMMU or device doesn't support PASID then it will be zero.

This will be used while allocating GCR3 table to decide required number of PASID table levels. Also in PASID bind path it will use this variable to check whether device supports PASID or not.

---
 drivers/iommu/amd/amd_iommu_types.h |  1 +
 drivers/iommu/amd/iommu.c           | 12 ++++++++++++
 2 files changed, 13 insertions(+)
drivers/iommu/amd/amd_iommu_types.h +1 -0
index d1fed5fc219b..03442e86ae44 100644
@@ -813,6 +813,7 @@ struct iommu_dev_data {
 	struct device *dev;
 	u16 devid;			  /* PCI Device ID */
 
+	u32 max_pasids;			  /* Max supported PASIDs */
 	u32 flags;			  /* Holds AMD_IOMMU_DEVICE_FLAG_<*> */
 	int ats_qdep;
 	u8 ats_enabled  :1;		  /* ATS state */
drivers/iommu/amd/iommu.c +12 -0
index 0decab958106..7daf6d75d964 100644
@@ -2104,6 +2104,7 @@ static struct iommu_device *amd_iommu_probe_device(struct device *dev)
 {
 	struct iommu_device *iommu_dev;
 	struct amd_iommu *iommu;
+	struct iommu_dev_data *dev_data;
 	int ret;
 
 	if (!check_device(dev))
@@ -2130,6 +2131,17 @@ static struct iommu_device *amd_iommu_probe_device(struct device *dev)
 		iommu_dev = &iommu->iommu;
 	}
 
+	/*
+	 * If IOMMU and device supports PASID then it will contain max
+	 * supported PASIDs, else it will be zero.
+	 */
+	dev_data = dev_iommu_priv_get(dev);
+	if (amd_iommu_pasid_supported() && dev_is_pci(dev) &&
+	    pdev_pasid_supported(dev_data)) {
+		dev_data->max_pasids = min_t(u32, iommu->iommu.max_pasids,
+					     pci_max_pasids(to_pci_dev(dev)));
+	}
+
 	iommu_completion_wait(iommu);
 
 	return iommu_dev;
Signature · 1 line
2.31.1
author maintainer
#7 lore raw

PATCH v8 7/15 iommu/amd: Setup GCR3 table in advance if domain is SVA capable

In reply to Vasant Hegde

SVA can be supported if domain is in passthrough mode or paging domain with v2 page table. Current code sets up GCR3 table for domain with v2 page table only. Setup GCR3 table for all SVA capable domains.

- Move GCR3 init/destroy to separate function.

- Change default GCR3 table to use MAX supported PASIDs. Ideally it should use 1 level PASID table as its using PASID zero only. But we don't have support to extend PASID table yet. We will fix this later.

- When domain is configured with passthrough mode, allocate default GCR3 table only if device is SVA capable.

Note that in attach_device() path it will not know whether device will use SVA or not. If device is attached to passthrough domain and if it doesn't use SVA then GCR3 table will never be used. We will endup wasting memory allocated for GCR3 table. This is done to avoid DTE update when attaching PASID to device.

---
 drivers/iommu/amd/iommu.c | 86 ++++++++++++++++++++++++++++++++-------
 1 file changed, 71 insertions(+), 15 deletions(-)
drivers/iommu/amd/iommu.c +71 -15
index 7daf6d75d964..288cf7485306 100644
@@ -89,6 +89,21 @@ static inline bool pdom_is_v2_pgtbl_mode(struct protection_domain *pdom)
 	return (pdom && (pdom->pd_mode == PD_MODE_V2));
 }
 
+static inline bool pdom_is_in_pt_mode(struct protection_domain *pdom)
+{
+	return (pdom->domain.type == IOMMU_DOMAIN_IDENTITY);
+}
+
+/*
+ * We cannot support PASID w/ existing v1 page table in the same domain
+ * since it will be nested. However, existing domain w/ v2 page table
+ * or passthrough mode can be used for PASID.
+ */
+static inline bool pdom_is_sva_capable(struct protection_domain *pdom)
+{
+	return pdom_is_v2_pgtbl_mode(pdom) || pdom_is_in_pt_mode(pdom);
+}
+
 static inline int get_acpihid_device_id(struct device *dev,
 					struct acpihid_map_entry **entry)
 {
@@ -1964,6 +1979,58 @@ void amd_iommu_dev_update_dte(struct iommu_dev_data *dev_data, bool set)
 	iommu_completion_wait(iommu);
 }
 
+/*
+ * If domain is SVA capable then initialize GCR3 table. Also if domain is
+ * in v2 page table mode then update GCR3[0].
+ */
+static int init_gcr3_table(struct iommu_dev_data *dev_data,
+			   struct protection_domain *pdom)
+{
+	struct amd_iommu *iommu = get_amd_iommu_from_dev_data(dev_data);
+	int max_pasids = dev_data->max_pasids;
+	int ret = 0;
+
+	 /*
+	  * If domain is in pt mode then setup GCR3 table only if device
+	  * is PASID capable
+	  */
+	if (pdom_is_in_pt_mode(pdom) && !pdev_pasid_supported(dev_data))
+		return ret;
+
+	/*
+	 * By default, setup GCR3 table to support MAX PASIDs
+	 * supported by the device/IOMMU.
+	 */
+	ret = setup_gcr3_table(&dev_data->gcr3_info, iommu,
+			       max_pasids > 0 ?  max_pasids : 1);
+	if (ret)
+		return ret;
+
+	/* Setup GCR3[0] only if domain is setup with v2 page table mode */
+	if (!pdom_is_v2_pgtbl_mode(pdom))
+		return ret;
+
+	ret = update_gcr3(dev_data, 0, iommu_virt_to_phys(pdom->iop.pgd), true);
+	if (ret)
+		free_gcr3_table(&dev_data->gcr3_info);
+
+	return ret;
+}
+
+static void destroy_gcr3_table(struct iommu_dev_data *dev_data,
+			       struct protection_domain *pdom)
+{
+	struct gcr3_tbl_info *gcr3_info = &dev_data->gcr3_info;
+
+	if (pdom_is_v2_pgtbl_mode(pdom))
+		update_gcr3(dev_data, 0, 0, false);
+
+	if (gcr3_info->gcr3_tbl == NULL)
+		return;
+
+	free_gcr3_table(gcr3_info);
+}
+
 static int do_attach(struct iommu_dev_data *dev_data,
 		     struct protection_domain *domain)
 {
@@ -1982,19 +2049,10 @@ static int do_attach(struct iommu_dev_data *dev_data,
 	domain->dev_iommu[iommu->index] += 1;
 	domain->dev_cnt                 += 1;
 
-	/* Init GCR3 table and update device table */
-	if (domain->pd_mode == PD_MODE_V2) {
-		/* By default, setup GCR3 table to support single PASID */
-		ret = setup_gcr3_table(&dev_data->gcr3_info, iommu, 1);
+	if (pdom_is_sva_capable(domain)) {
+		ret = init_gcr3_table(dev_data, domain);
 		if (ret)
 			return ret;
-
-		ret = update_gcr3(dev_data, 0,
-				  iommu_virt_to_phys(domain->iop.pgd), true);
-		if (ret) {
-			free_gcr3_table(&dev_data->gcr3_info);
-			return ret;
-		}
 	}
 
 	/* Update device table */
@@ -2009,10 +2067,8 @@ static void do_detach(struct iommu_dev_data *dev_data)
 	struct amd_iommu *iommu = get_amd_iommu_from_dev_data(dev_data);
 
 	/* Clear GCR3 table */
-	if (domain->pd_mode == PD_MODE_V2) {
-		update_gcr3(dev_data, 0, 0, false);
-		free_gcr3_table(&dev_data->gcr3_info);
-	}
+	if (pdom_is_sva_capable(domain))
+		destroy_gcr3_table(dev_data, domain);
 
 	/* Update data structures */
 	dev_data->domain = NULL;
Signature · 1 line
2.31.1
#8 lore raw

Re: [PATCH v8 07/15] iommu/amd: Setup GCR3 table in advance if domain is SVA capable

Hi,

Quoted 135 lines · Vasant Hegde

SVA can be supported if domain is in passthrough mode or paging domain with v2 page table. Current code sets up GCR3 table for domain with v2 page table only. Setup GCR3 table for all SVA capable domains.

- Move GCR3 init/destroy to separate function.

- Change default GCR3 table to use MAX supported PASIDs. Ideally it should use 1 level PASID table as its using PASID zero only. But we don't have support to extend PASID table yet. We will fix this later.

- When domain is configured with passthrough mode, allocate default GCR3 table only if device is SVA capable.

Note that in attach_device() path it will not know whether device will use SVA or not. If device is attached to passthrough domain and if it doesn't use SVA then GCR3 table will never be used. We will endup wasting memory allocated for GCR3 table. This is done to avoid DTE update when attaching PASID to device.

---
  drivers/iommu/amd/iommu.c | 86 ++++++++++++++++++++++++++++++++-------
  1 file changed, 71 insertions(+), 15 deletions(-)
drivers/iommu/amd/iommu.c +69 -11
index 7daf6d75d964..288cf7485306 100644
@@ -89,6 +89,21 @@ static inline bool pdom_is_v2_pgtbl_mode(struct protection_domain *pdom)
  	return (pdom && (pdom->pd_mode == PD_MODE_V2));
  }
 
+static inline bool pdom_is_in_pt_mode(struct protection_domain *pdom)
+{
+	return (pdom->domain.type == IOMMU_DOMAIN_IDENTITY);
+}
+
+/*
+ * We cannot support PASID w/ existing v1 page table in the same domain
+ * since it will be nested. However, existing domain w/ v2 page table
+ * or passthrough mode can be used for PASID.
+ */
+static inline bool pdom_is_sva_capable(struct protection_domain *pdom)
+{
+	return pdom_is_v2_pgtbl_mode(pdom) || pdom_is_in_pt_mode(pdom);
+}
+
  static inline int get_acpihid_device_id(struct device *dev,
  					struct acpihid_map_entry **entry)
  {
@@ -1964,6 +1979,58 @@ void amd_iommu_dev_update_dte(struct iommu_dev_data *dev_data, bool set)
  	iommu_completion_wait(iommu);
  }
 
+/*
+ * If domain is SVA capable then initialize GCR3 table. Also if domain is
+ * in v2 page table mode then update GCR3[0].
+ */
+static int init_gcr3_table(struct iommu_dev_data *dev_data,
+			   struct protection_domain *pdom)
+{
+	struct amd_iommu *iommu = get_amd_iommu_from_dev_data(dev_data);
+	int max_pasids = dev_data->max_pasids;
+	int ret = 0;
+
+	 /*
+	  * If domain is in pt mode then setup GCR3 table only if device
+	  * is PASID capable
+	  */
+	if (pdom_is_in_pt_mode(pdom) && !pdev_pasid_supported(dev_data))
+		return ret;
+
+	/*
+	 * By default, setup GCR3 table to support MAX PASIDs
+	 * supported by the device/IOMMU.
+	 */
+	ret = setup_gcr3_table(&dev_data->gcr3_info, iommu,
+			       max_pasids > 0 ?  max_pasids : 1);
+	if (ret)
+		return ret;
+
+	/* Setup GCR3[0] only if domain is setup with v2 page table mode */
+	if (!pdom_is_v2_pgtbl_mode(pdom))
+		return ret;
+
+	ret = update_gcr3(dev_data, 0, iommu_virt_to_phys(pdom->iop.pgd), true);
+	if (ret)
+		free_gcr3_table(&dev_data->gcr3_info);
+
+	return ret;
+}
+
+static void destroy_gcr3_table(struct iommu_dev_data *dev_data,
+			       struct protection_domain *pdom)
+{
+	struct gcr3_tbl_info *gcr3_info = &dev_data->gcr3_info;
+
+	if (pdom_is_v2_pgtbl_mode(pdom))
+		update_gcr3(dev_data, 0, 0, false);
+
+	if (gcr3_info->gcr3_tbl == NULL)
+		return;
+
+	free_gcr3_table(gcr3_info);
+}
+
  static int do_attach(struct iommu_dev_data *dev_data,
  		     struct protection_domain *domain)
  {
@@ -1982,19 +2049,10 @@ static int do_attach(struct iommu_dev_data *dev_data,
  	domain->dev_iommu[iommu->index] += 1;
  	domain->dev_cnt                 += 1;
 
-	/* Init GCR3 table and update device table */
-	if (domain->pd_mode == PD_MODE_V2) {
-		/* By default, setup GCR3 table to support single PASID */
-		ret = setup_gcr3_table(&dev_data->gcr3_info, iommu, 1);
+	if (pdom_is_sva_capable(domain)) {
+		ret = init_gcr3_table(dev_data, domain);
  		if (ret)
  			return ret;
-
-		ret = update_gcr3(dev_data, 0,
-				  iommu_virt_to_phys(domain->iop.pgd), true);
-		if (ret) {
-			free_gcr3_table(&dev_data->gcr3_info);
-			return ret;
-		}
  	}
 
  	/* Update device table */
@@ -2009,10 +2067,8 @@ static void do_detach(struct iommu_dev_data *dev_data)
  	struct amd_iommu *iommu = get_amd_iommu_from_dev_data(dev_data);
 

drivers/iommu/amd/iommu.c static void do_detach(struct iommu_dev_data *dev_data) · line 2074

Quoted 10 lines
drivers/iommu/amd/iommu.c +2 -4
  	/* Clear GCR3 table */
-	if (domain->pd_mode == PD_MODE_V2) {
-		update_gcr3(dev_data, 0, 0, false);
-		free_gcr3_table(&dev_data->gcr3_info);
-	}
+	if (pdom_is_sva_capable(domain))
+		destroy_gcr3_table(dev_data, domain);
 
  	/* Update data structures */
  	dev_data->domain = NULL;

This change seems to interact poorly with the amdgpu driver. It fails to initialize properly (monitors go to sleep).

The full log is attached, this snippet happens immediately before the first warning:
amdgpu 0000:2d:00.0: amdgpu: STB initialized to 2048 entries
[drm] Loading DMUB firmware via PSP: version=0x02020020
[drm] use_doorbell being set to: [true]
[drm] use_doorbell being set to: [true]
[drm] Found VCN firmware Version ENC: 1.30 DEC: 3 VEP: 0 Revision: 4
amdgpu 0000:2d:00.0: amdgpu: Will use PSP to load VCN firmware
iommu ivhd0: AMD-Vi: Event logged [ILLEGAL_DEV_TABLE_ENTRY device=0000:2d:00.0 pasid=0x00000 address=0x11c300000 flags=0x0080]
AMD-Vi: DTE[0]: 7d80000000000003
AMD-Vi: DTE[1]: 0000100011700014
AMD-Vi: DTE[2]: 0000000000000000
AMD-Vi: DTE[3]: 0000000000000000
r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control rx/tx
amdgpu 0000:2d:00.0: amdgpu: PSP load kdb failed!
[drm:psp_v11_0_ring_destroy [amdgpu]] *ERROR* Fail to stop psp ring
[drm:amdgpu_fill_buffer [amdgpu]] *ERROR* Trying to clear memory with ring turned off.

Please let me know if this is the wrong place to report this or there's anything else you need.

Kind regards,
Klara Modin

# bad: [d04466706db5e241ee026f17b5f920e50dee26b5] Add linux-next specific files for 20240430
git bisect start 'next/master'
# status: waiting for good commit(s), bad commit known
# good: [98369dccd2f8e16bf4c6621053af7aa4821dcf8e] Merge tag 'wq-for-6.9-rc6-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq
git bisect good 98369dccd2f8e16bf4c6621053af7aa4821dcf8e
# good: [caacb2666b21a77d201acd8e0ee8c79a8446c9d9] Merge branch 'main' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git
git bisect good caacb2666b21a77d201acd8e0ee8c79a8446c9d9
# good: [3a9b7696d1ffbf779546f88c538167c9406e1bec] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/libata/linux
git bisect good 3a9b7696d1ffbf779546f88c538167c9406e1bec
# bad: [521ff90e51ca05aaa37ed4314a6276be2026009e] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/westeri/thunderbolt.git
git bisect bad 521ff90e51ca05aaa37ed4314a6276be2026009e
# bad: [6b06cd258b09108594411d81191b5fb6908ec466] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git
git bisect bad 6b06cd258b09108594411d81191b5fb6908ec466
# bad: [3ccf6929e33ddeed4a98eb3f8ceee6629bc27871] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/spi.git
git bisect bad 3ccf6929e33ddeed4a98eb3f8ceee6629bc27871
# good: [9dfa204dca85149b12be3bfd6e34017fb5b0b61b] Merge branch 'next-integrity' of git://git.kernel.org/pub/scm/linux/kernel/git/zohar/linux-integrity
git bisect good 9dfa204dca85149b12be3bfd6e34017fb5b0b61b
# bad: [720f4287ef1bfae57a1d01f35e02fe778a52de9f] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu.git
git bisect bad 720f4287ef1bfae57a1d01f35e02fe778a52de9f
# bad: [5dc72c8a146ddcfc568265c8088e90d8e996d340] Merge branch 'memory-observability' into x86/amd
git bisect bad 5dc72c8a146ddcfc568265c8088e90d8e996d340
# bad: [80af5a45202422db957549a241e00bf4d4e0ce89] iommu: Add ops->domain_alloc_sva()
git bisect bad 80af5a45202422db957549a241e00bf4d4e0ce89
# bad: [c9e8701132e6cc162d082e7dad8a2e9110f5f8fd] iommu/amd: Setup GCR3 table in advance if domain is SVA capable
git bisect bad c9e8701132e6cc162d082e7dad8a2e9110f5f8fd
# good: [db44bd517ff2de4224fdb0e9c0db426572f9fa11] iommu/amd: Add support for enabling/disabling IOMMU features
git bisect good db44bd517ff2de4224fdb0e9c0db426572f9fa11
# good: [7c5b7176f0c3996adbe853adb1f857bd4f82b1e2] iommu/amd: Fix PPR interrupt processing logic
git bisect good 7c5b7176f0c3996adbe853adb1f857bd4f82b1e2
# good: [a0c47f233e683e6a81fced5c9c9cef6fa0da9446] iommu/amd: Introduce iommu_dev_data.max_pasids
git bisect good a0c47f233e683e6a81fced5c9c9cef6fa0da9446

# first bad commit: [c9e8701132e6cc162d082e7dad8a2e9110f5f8fd] iommu/amd: Setup GCR3 table in advance if domain is SVA capable
maintainer
#9 lore raw

Re: [PATCH v8 07/15] iommu/amd: Setup GCR3 table in advance if domain is SVA capable

Vasant,

Quoted 2 lines · Klara Modin

This change seems to interact poorly with the amdgpu driver. It fails to initialize properly (monitors go to sleep).

Can you have a look please? We need to resolve this otherwise I have to exclude the x86/amd branch from next, as I can not send code with known regressions upstream.

Regards,

Joerg

author maintainer
#10 lore raw

Re: [PATCH v8 07/15] iommu/amd: Setup GCR3 table in advance if domain is SVA capable

Hi Joerg,

Quoted 9 lines · Joerg Roedel

Vasant,

Quoted 2 lines · Klara Modin

This change seems to interact poorly with the amdgpu driver. It fails to initialize properly (monitors go to sleep).

Can you have a look please? We need to resolve this otherwise I have to exclude the x86/amd branch from next, as I can not send code with known regressions upstream.

Sorry for the delay. I was on vacation last week.

Looking into the report nothing strikes me immediately. DTE setting looks to be fine. I will see if I can find a system internally to reproduce. Will update soon.

-Vasant

author maintainer
#11 lore raw

Re: [PATCH v8 07/15] iommu/amd: Setup GCR3 table in advance if domain is SVA capable

In reply to Klara Modin

Hi Klara,

Quoted 168 lines · Klara Modin

Hi,

Quoted 131 lines · Vasant Hegde

SVA can be supported if domain is in passthrough mode or paging domain with v2 page table. Current code sets up GCR3 table for domain with v2 page table only. Setup GCR3 table for all SVA capable domains.

   - Move GCR3 init/destroy to separate function.

   - Change default GCR3 table to use MAX supported PASIDs. Ideally it should use 1 level PASID table as its using PASID zero only. But we don't have support to extend PASID table yet. We will fix this later.

   - When domain is configured with passthrough mode, allocate default GCR3 table only if device is SVA capable.

Note that in attach_device() path it will not know whether device will use SVA or not. If device is attached to passthrough domain and if it doesn't use SVA then GCR3 table will never be used. We will endup wasting memory allocated for GCR3 table. This is done to avoid DTE update when attaching PASID to device.

---
  drivers/iommu/amd/iommu.c | 86 ++++++++++++++++++++++++++++++++-------

  1 file changed, 71 insertions(+), 15 deletions(-)
drivers/iommu/amd/iommu.c +14 -0
index 7daf6d75d964..288cf7485306 100644
@@ -89,6 +89,21 @@ static inline bool pdom_is_v2_pgtbl_mode(struct
 protection_domain *pdom)
       return (pdom && (pdom->pd_mode == PD_MODE_V2));
   }
   +static inline bool pdom_is_in_pt_mode(struct protection_domain *pdom)
+{
+    return (pdom->domain.type == IOMMU_DOMAIN_IDENTITY);
+}
+
+/*
+ * We cannot support PASID w/ existing v1 page table in the same domain
+ * since it will be nested. However, existing domain w/ v2 page table
+ * or passthrough mode can be used for PASID.
+ */
+static inline bool pdom_is_sva_capable(struct protection_domain *pdom)
+{
+    return pdom_is_v2_pgtbl_mode(pdom) || pdom_is_in_pt_mode(pdom);
+}
+
   static inline int get_acpihid_device_id(struct device *dev,
                       struct acpihid_map_entry **entry)

  {

drivers/iommu/amd/iommu.c +0 -0
@@ -1964,6 +1979,58 @@ void amd_iommu_dev_update_dte(struct iommu_dev_data
 *dev_data, bool set)
       iommu_completion_wait(iommu);
   }

  +/*
+ * If domain is SVA capable then initialize GCR3 table. Also if domain is
+ * in v2 page table mode then update GCR3[0].
+ */

drivers/iommu/amd/iommu.c +48 -0
+static int init_gcr3_table(struct iommu_dev_data *dev_data,
+               struct protection_domain *pdom)
+{
+    struct amd_iommu *iommu = get_amd_iommu_from_dev_data(dev_data);
+    int max_pasids = dev_data->max_pasids;
+    int ret = 0;
+
+     /*
+      * If domain is in pt mode then setup GCR3 table only if device
+      * is PASID capable
+      */
+    if (pdom_is_in_pt_mode(pdom) && !pdev_pasid_supported(dev_data))
+        return ret;
+
+    /*
+     * By default, setup GCR3 table to support MAX PASIDs
+     * supported by the device/IOMMU.
+     */
+    ret = setup_gcr3_table(&dev_data->gcr3_info, iommu,
+                   max_pasids > 0 ?  max_pasids : 1);
+    if (ret)
+        return ret;
+
+    /* Setup GCR3[0] only if domain is setup with v2 page table mode */
+    if (!pdom_is_v2_pgtbl_mode(pdom))
+        return ret;
+
+    ret = update_gcr3(dev_data, 0, iommu_virt_to_phys(pdom->iop.pgd), true);
+    if (ret)
+        free_gcr3_table(&dev_data->gcr3_info);
+
+    return ret;
+}
+
+static void destroy_gcr3_table(struct iommu_dev_data *dev_data,
+                   struct protection_domain *pdom)
+{
+    struct gcr3_tbl_info *gcr3_info = &dev_data->gcr3_info;
+
+    if (pdom_is_v2_pgtbl_mode(pdom))
+        update_gcr3(dev_data, 0, 0, false);
+
+    if (gcr3_info->gcr3_tbl == NULL)
+        return;
+
+    free_gcr3_table(gcr3_info);
+}
+

  static int do_attach(struct iommu_dev_data *dev_data, struct protection_domain *domain) {

drivers/iommu/amd/iommu.c +0 -0
@@ -1982,19 +2049,10 @@ static int do_attach(struct iommu_dev_data *dev_data,
       domain->dev_iommu[iommu->index] += 1;
       domain->dev_cnt                 += 1;

  -    /* Init GCR3 table and update device table */
-    if (domain->pd_mode == PD_MODE_V2) {
-        /* By default, setup GCR3 table to support single PASID */ -        ret = setup_gcr3_table(&dev_data->gcr3_info, iommu, 1); +    if (pdom_is_sva_capable(domain)) {
+        ret = init_gcr3_table(dev_data, domain);
          if (ret)
              return ret;
-
-        ret = update_gcr3(dev_data, 0,
-                  iommu_virt_to_phys(domain->iop.pgd), true); -        if (ret) {
-            free_gcr3_table(&dev_data->gcr3_info);
-            return ret;
-        }
      }
        /* Update device table */

drivers/iommu/amd/iommu.c static void do_detach(struct iommu_dev_data *dev_data) · line 2072

Quoted 10 lines
drivers/iommu/amd/iommu.c +2 -4
@@ -2009,10 +2067,8 @@ static void do_detach(struct iommu_dev_data *dev_data)
       struct amd_iommu *iommu = get_amd_iommu_from_dev_data(dev_data);
         /* Clear GCR3 table */
-    if (domain->pd_mode == PD_MODE_V2) {
-        update_gcr3(dev_data, 0, 0, false);
-        free_gcr3_table(&dev_data->gcr3_info);
-    }
+    if (pdom_is_sva_capable(domain))
+        destroy_gcr3_table(dev_data, domain);
         /* Update data structures */
       dev_data->domain = NULL;

This change seems to interact poorly with the amdgpu driver. It fails to initialize properly (monitors go to sleep).

The full log is attached, this snippet happens immediately before the first warning:
amdgpu 0000:2d:00.0: amdgpu: STB initialized to 2048 entries [drm] Loading DMUB firmware via PSP: version=0x02020020
[drm] use_doorbell being set to: [true]
[drm] use_doorbell being set to: [true]
[drm] Found VCN firmware Version ENC: 1.30 DEC: 3 VEP: 0 Revision: 4 amdgpu 0000:2d:00.0: amdgpu: Will use PSP to load VCN firmware iommu ivhd0: AMD-Vi: Event logged [ILLEGAL_DEV_TABLE_ENTRY device=0000:2d:00.0 pasid=0x00000 address=0x11c300000 flags=0x0080]
AMD-Vi: DTE[0]: 7d80000000000003
AMD-Vi: DTE[1]: 0000100011700014
AMD-Vi: DTE[2]: 0000000000000000
AMD-Vi: DTE[3]: 0000000000000000
r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control rx/tx amdgpu 0000:2d:00.0: amdgpu: PSP load kdb failed! [drm:psp_v11_0_ring_destroy [amdgpu]] *ERROR* Fail to stop psp ring [drm:amdgpu_fill_buffer [amdgpu]] *ERROR* Trying to clear memory with ring turned off.

Quoted 2 lines

Please let me know if this is the wrong place to report this or there's anything else you need.

Thanks for the report.

Looking into the attached dmesg :

2024-04-29T10:35:05+02:00 soda.int.kasm.eu kernel: AMD-Vi: Using global IVHD

EFR:0x0, EFR2:0x0

--> This means BIOS didn't setup the EFR flags in ACPI IVRS table (its BIOS issue).

Commit 8e0179733172 (iommu/amd: Enable Guest Translation before registering devices) changed GT feature enablement in control register.

Now it tries to enable CONTROL[GT] flag before reading IOMMU feature register (late_iommu_features_init()).

In attach device path, it assumed GT is supported (based on IOMMU EFR register value) and enabled DTE[GV]. Hence we hit ILLEGAL_DEV_TABLE_ENTRY error.

Can you please try below fix?

-Vasant

---

drivers/iommu/amd/init.c +2 -2
index fd3e76e43699..b292181995b7 100644
@@ -2046,6 +2046,8 @@ static int __init iommu_init_pci(struct amd_iommu *iommu)
 			amd_iommu_max_glx_val = glxval;
 		else
 			amd_iommu_max_glx_val = min(amd_iommu_max_glx_val, glxval);
+
+		iommu_enable_gt(iommu);
 	}
 
 	if (check_feature(FEATURE_PPR) && amd_iommu_alloc_ppr_log(iommu))
@@ -2732,7 +2734,6 @@ static void early_enable_iommu(struct amd_iommu *iommu)
 	iommu_enable_command_buffer(iommu);
 	iommu_enable_event_buffer(iommu);
 	iommu_set_exclusion_range(iommu);
-	iommu_enable_gt(iommu);
 	iommu_enable_ga(iommu);
 	iommu_enable_xt(iommu);
 	iommu_enable_irtcachedis(iommu);
@@ -2789,7 +2790,6 @@ static void early_enable_iommus(void)
 			iommu_disable_irtcachedis(iommu);
 			iommu_enable_command_buffer(iommu);
 			iommu_enable_event_buffer(iommu);
-			iommu_enable_gt(iommu);
 			iommu_enable_ga(iommu);
 			iommu_enable_xt(iommu);
 			iommu_enable_irtcachedis(iommu);
Tested-by
#12 lore raw

Re: [PATCH v8 07/15] iommu/amd: Setup GCR3 table in advance if domain is SVA capable

Quoted 52 lines · Vasant Hegde

Hi Klara,

Thanks for the report.

Looking into the attached dmesg :

2024-04-29T10:35:05+02:00 soda.int.kasm.eu kernel: AMD-Vi: Using global IVHD

EFR:0x0, EFR2:0x0

--> This means BIOS didn't setup the EFR flags in ACPI IVRS table (its BIOS issue).

Commit 8e0179733172 (iommu/amd: Enable Guest Translation before registering devices) changed GT feature enablement in control register.

Now it tries to enable CONTROL[GT] flag before reading IOMMU feature register (late_iommu_features_init()).

In attach device path, it assumed GT is supported (based on IOMMU EFR register value) and enabled DTE[GV]. Hence we hit ILLEGAL_DEV_TABLE_ENTRY error.

Can you please try below fix?

(quoted diff) +0 -2
-Vasant
 
---
drivers/iommu/amd/init.c +2 -1
index fd3e76e43699..b292181995b7 100644
@@ -2046,6 +2046,8 @@ static int __init iommu_init_pci(struct amd_iommu *iommu)
  			amd_iommu_max_glx_val = glxval;
  		else
  			amd_iommu_max_glx_val = min(amd_iommu_max_glx_val, glxval);
+
+		iommu_enable_gt(iommu);
  	}
 
  	if (check_feature(FEATURE_PPR) && amd_iommu_alloc_ppr_log(iommu))
@@ -2732,7 +2734,6 @@ static void early_enable_iommu(struct amd_iommu *iommu)
  	iommu_enable_command_buffer(iommu);
  	iommu_enable_event_buffer(iommu);
  	iommu_set_exclusion_range(iommu);
-	iommu_enable_gt(iommu);
  	iommu_enable_ga(iommu);
  	iommu_enable_xt(iommu);
  	iommu_enable_irtcachedis(iommu);

drivers/iommu/amd/init.c static void early_enable_iommus(void) · line 2795

Quoted 7 lines
drivers/iommu/amd/init.c +0 -1
@@ -2789,7 +2790,6 @@ static void early_enable_iommus(void)
  			iommu_disable_irtcachedis(iommu);
  			iommu_enable_command_buffer(iommu);
  			iommu_enable_event_buffer(iommu);
-			iommu_enable_gt(iommu);
  			iommu_enable_ga(iommu);
  			iommu_enable_xt(iommu);
  			iommu_enable_irtcachedis(iommu);

This fixes the issue for me.

Thanks,

maintainer
#13 lore raw

Re: [PATCH v8 07/15] iommu/amd: Setup GCR3 table in advance if domain is SVA capable

Quoted 4 lines · Klara Modin

This fixes the issue for me.

Thanks,

Thanks Klara.

Vasant, can you please send this as a proper patch later today, including a Fixed tag? Thanks!

Regards,

Joerg

author maintainer
#14 lore raw

Re: [PATCH v8 07/15] iommu/amd: Setup GCR3 table in advance if domain is SVA capable

Hi,

Quoted 5 lines · Joerg Roedel
Quoted 4 lines · Klara Modin

This fixes the issue for me.

Thanks,

Thank for confirming Klara.

Quoted 5 lines

Thanks Klara.

Vasant, can you please send this as a proper patch later today, including a Fixed tag? Thanks!

Done. I have posted proper patch.

-Vasant

author maintainer
#15 lore raw

PATCH v8 8/15 iommu/amd: Enable PCI features based on attached domain capability

In reply to Vasant Hegde

Commit eda8c2860ab6 ("iommu/amd: Enable device ATS/PASID/PRI capabilities independently") changed the way it enables device capability while attaching devices. I missed to account the attached domain capability. Meaning if domain is not capable of handling PASID/PRI (ex: paging domain with v1 page table) then enabling device feature is not required.

This patch enables PASID/PRI only if domain is capable of handling SVA. Also move pci feature enablement to do_attach() function so that we make SVA capability in one place. Finally make PRI enable/disable functions as static functions.

---
 drivers/iommu/amd/amd_iommu.h |  4 ----
 drivers/iommu/amd/iommu.c     | 22 ++++++++++++++--------
 2 files changed, 14 insertions(+), 12 deletions(-)
drivers/iommu/amd/amd_iommu.h +0 -4
index 159e9a43aa61..f8919c54d9ae 100644
@@ -46,10 +46,6 @@ extern int amd_iommu_gpt_level;
 
 bool amd_iommu_pasid_supported(void);
 
-/* Device capabilities */
-int amd_iommu_pdev_enable_cap_pri(struct pci_dev *pdev);
-void amd_iommu_pdev_disable_cap_pri(struct pci_dev *pdev);
-
 /* GCR3 setup */
 int amd_iommu_set_gcr3(struct iommu_dev_data *dev_data,
 		       ioasid_t pasid, unsigned long gcr3);
drivers/iommu/amd/iommu.c +14 -8
index 288cf7485306..1319f93bb473 100644
@@ -399,7 +399,7 @@ static inline void pdev_disable_cap_ats(struct pci_dev *pdev)
 	}
 }
 
-int amd_iommu_pdev_enable_cap_pri(struct pci_dev *pdev)
+static inline int pdev_enable_cap_pri(struct pci_dev *pdev)
 {
 	struct iommu_dev_data *dev_data = dev_iommu_priv_get(&pdev->dev);
 	int ret = -EINVAL;
@@ -407,6 +407,9 @@ int amd_iommu_pdev_enable_cap_pri(struct pci_dev *pdev)
 	if (dev_data->pri_enabled)
 		return 0;
 
+	if (!dev_data->ats_enabled)
+		return 0;
+
 	if (dev_data->flags & AMD_IOMMU_DEVICE_FLAG_PRI_SUP) {
 		/*
 		 * First reset the PRI state of the device.
@@ -423,7 +426,7 @@ int amd_iommu_pdev_enable_cap_pri(struct pci_dev *pdev)
 	return ret;
 }
 
-void amd_iommu_pdev_disable_cap_pri(struct pci_dev *pdev)
+static inline void pdev_disable_cap_pri(struct pci_dev *pdev)
 {
 	struct iommu_dev_data *dev_data = dev_iommu_priv_get(&pdev->dev);
 
@@ -465,15 +468,14 @@ static void pdev_enable_caps(struct pci_dev *pdev)
 {
 	pdev_enable_cap_ats(pdev);
 	pdev_enable_cap_pasid(pdev);
-	amd_iommu_pdev_enable_cap_pri(pdev);
-
+	pdev_enable_cap_pri(pdev);
 }
 
 static void pdev_disable_caps(struct pci_dev *pdev)
 {
 	pdev_disable_cap_ats(pdev);
 	pdev_disable_cap_pasid(pdev);
-	amd_iommu_pdev_disable_cap_pri(pdev);
+	pdev_disable_cap_pri(pdev);
 }
 
 /*
@@ -2035,6 +2037,7 @@ static int do_attach(struct iommu_dev_data *dev_data,
 		     struct protection_domain *domain)
 {
 	struct amd_iommu *iommu = get_amd_iommu_from_dev_data(dev_data);
+	struct pci_dev *pdev;
 	int ret = 0;
 
 	/* Update data structures */
@@ -2049,10 +2052,16 @@ static int do_attach(struct iommu_dev_data *dev_data,
 	domain->dev_iommu[iommu->index] += 1;
 	domain->dev_cnt                 += 1;
 
+	pdev = dev_is_pci(dev_data->dev) ? to_pci_dev(dev_data->dev) : NULL;
 	if (pdom_is_sva_capable(domain)) {
 		ret = init_gcr3_table(dev_data, domain);
 		if (ret)
 			return ret;
+
+		if (pdev)
+			pdev_enable_caps(pdev);
+	} else if (pdev) {
+		pdev_enable_cap_ats(pdev);
 	}
 
 	/* Update device table */
@@ -2107,9 +2116,6 @@ static int attach_device(struct device *dev,
 		goto out;
 	}
 
-	if (dev_is_pci(dev))
-		pdev_enable_caps(to_pci_dev(dev));
-
 	ret = do_attach(dev_data, domain);
 
 out:
Signature · 1 line
2.31.1
author maintainer
#16 lore raw

PATCH v8 9/15 iommu/amd: Define per-IOMMU iopf_queue

In reply to Vasant Hegde

From: Suravee Suthikulpanit <[email protected]>

AMD IOMMU hardware supports PCI Peripheral Paging Request (PPR) using a PPR log, which is a circular buffer containing requests from downstream end-point devices.

There is one PPR log per IOMMU instance. Therefore, allocate an iopf_queue per IOMMU instance during driver initialization, and free the queue during driver deinitialization.

Also rename enable_iommus_v2() -> enable_iommus_ppr() to reflect its usage. And add amd_iommu_gt_ppr_supported() check before enabling PPR log.

---
 drivers/iommu/amd/amd_iommu.h       |  4 ++++
 drivers/iommu/amd/amd_iommu_types.h |  4 ++++
 drivers/iommu/amd/init.c            | 18 +++++++++++++++--
 drivers/iommu/amd/ppr.c             | 31 +++++++++++++++++++++++++++++
 4 files changed, 55 insertions(+), 2 deletions(-)
drivers/iommu/amd/amd_iommu.h +4 -0
index f8919c54d9ae..66bf20d4a91a 100644
@@ -46,6 +46,10 @@ extern int amd_iommu_gpt_level;
 
 bool amd_iommu_pasid_supported(void);
 
+/* IOPF */
+int amd_iommu_iopf_init(struct amd_iommu *iommu);
+void amd_iommu_iopf_uninit(struct amd_iommu *iommu);
+
 /* GCR3 setup */
 int amd_iommu_set_gcr3(struct iommu_dev_data *dev_data,
 		       ioasid_t pasid, unsigned long gcr3);
drivers/iommu/amd/amd_iommu_types.h +4 -0
index 03442e86ae44..1a014dca7c2e 100644
@@ -762,6 +762,10 @@ struct amd_iommu {
 	/* DebugFS Info */
 	struct dentry *debugfs;
 #endif
+
+	/* IOPF support */
+	struct iopf_queue *iopf_queue;
+	unsigned char iopfq_name[32];
 };
 
 static inline struct amd_iommu *dev_to_amd_iommu(struct device *dev)
drivers/iommu/amd/init.c +16 -2
index 269446717818..7f9b5c710058 100644
@@ -1639,6 +1639,7 @@ static void __init free_iommu_one(struct amd_iommu *iommu)
 	amd_iommu_free_ppr_log(iommu);
 	free_ga_log(iommu);
 	iommu_unmap_mmio_space(iommu);
+	amd_iommu_iopf_uninit(iommu);
 }
 
 static void __init free_iommu_all(void)
@@ -2108,6 +2109,16 @@ static int __init iommu_init_pci(struct amd_iommu *iommu)
 	if (ret)
 		return ret;
 
+	/*
+	 * Allocate per IOMMU IOPF queue here so that in attach device path,
+	 * PRI capable device can be added to IOPF queue
+	 */
+	if (amd_iommu_gt_ppr_supported()) {
+		ret = amd_iommu_iopf_init(iommu);
+		if (ret)
+			return ret;
+	}
+
 	iommu_device_register(&iommu->iommu, &amd_iommu_ops, NULL);
 
 	return pci_enable_device(iommu->dev);
@@ -2793,10 +2804,13 @@ static void early_enable_iommus(void)
 	}
 }
 
-static void enable_iommus_v2(void)
+static void enable_iommus_ppr(void)
 {
 	struct amd_iommu *iommu;
 
+	if (!amd_iommu_gt_ppr_supported())
+		return;
+
 	for_each_iommu(iommu)
 		amd_iommu_enable_ppr_log(iommu);
 }
@@ -3134,7 +3148,7 @@ static int amd_iommu_enable_interrupts(void)
 	 * PPR and GA log interrupt for all IOMMUs.
 	 */
 	enable_iommus_vapic();
-	enable_iommus_v2();
+	enable_iommus_ppr();
 
 out:
 	return ret;
drivers/iommu/amd/ppr.c +31 -0
index 65db1745f40c..d4e08800beac 100644
@@ -108,3 +108,34 @@ void amd_iommu_poll_ppr_log(struct amd_iommu *iommu)
 		/* TODO: PPR Handler will be added when we add IOPF support */
 	}
 }
+
+/**************************************************************
+ *
+ * IOPF handling stuff
+ */
+
+/* Setup per-IOMMU IOPF queue if not exist. */
+int amd_iommu_iopf_init(struct amd_iommu *iommu)
+{
+	int ret = 0;
+
+	if (iommu->iopf_queue)
+		return ret;
+
+	snprintf(iommu->iopfq_name, sizeof(iommu->iopfq_name),
+		 "amdiommu-%#x-iopfq",
+		 PCI_SEG_DEVID_TO_SBDF(iommu->pci_seg->id, iommu->devid));
+
+	iommu->iopf_queue = iopf_queue_alloc(iommu->iopfq_name);
+	if (!iommu->iopf_queue)
+		ret = -ENOMEM;
+
+	return ret;
+}
+
+/* Destroy per-IOMMU IOPF queue if no longer needed. */
+void amd_iommu_iopf_uninit(struct amd_iommu *iommu)
+{
+	iopf_queue_free(iommu->iopf_queue);
+	iommu->iopf_queue = NULL;
+}
Signature · 1 line
2.31.1
author maintainer
#17 lore raw

PATCH v8 10/15 iommu/amd: Add support for page response

In reply to Vasant Hegde

From: Suravee Suthikulpanit <[email protected]>

This generates AMD IOMMU COMPLETE_PPR_REQUEST for the specified device with the specified PRI Response Code.

Also update amd_iommu_complete_ppr() to accept 'struct device' instead of pdev as it just need device reference.

---
 drivers/iommu/amd/amd_iommu.h | 5 +++--
 drivers/iommu/amd/iommu.c     | 8 ++++----
 drivers/iommu/amd/ppr.c       | 6 ++++++
 3 files changed, 13 insertions(+), 6 deletions(-)
drivers/iommu/amd/amd_iommu.h +3 -2
index 66bf20d4a91a..bb9a4c2e40da 100644
@@ -49,6 +49,8 @@ bool amd_iommu_pasid_supported(void);
 /* IOPF */
 int amd_iommu_iopf_init(struct amd_iommu *iommu);
 void amd_iommu_iopf_uninit(struct amd_iommu *iommu);
+void amd_iommu_page_response(struct device *dev, struct iopf_fault *evt,
+			     struct iommu_page_response *resp);
 
 /* GCR3 setup */
 int amd_iommu_set_gcr3(struct iommu_dev_data *dev_data,
@@ -60,8 +62,7 @@ int __init amd_iommu_alloc_ppr_log(struct amd_iommu *iommu);
 void __init amd_iommu_free_ppr_log(struct amd_iommu *iommu);
 void amd_iommu_enable_ppr_log(struct amd_iommu *iommu);
 void amd_iommu_poll_ppr_log(struct amd_iommu *iommu);
-int amd_iommu_complete_ppr(struct pci_dev *pdev, u32 pasid,
-			   int status, int tag);
+int amd_iommu_complete_ppr(struct device *dev, u32 pasid, int status, int tag);
 
 /*
  * This function flushes all internal caches of
drivers/iommu/amd/iommu.c +4 -4
index 1319f93bb473..11b50a6c744e 100644
@@ -1628,15 +1628,14 @@ void amd_iommu_domain_update(struct protection_domain *domain)
 	amd_iommu_domain_flush_all(domain);
 }
 
-int amd_iommu_complete_ppr(struct pci_dev *pdev, u32 pasid,
-			   int status, int tag)
+int amd_iommu_complete_ppr(struct device *dev, u32 pasid, int status, int tag)
 {
 	struct iommu_dev_data *dev_data;
 	struct amd_iommu *iommu;
 	struct iommu_cmd cmd;
 
-	dev_data = dev_iommu_priv_get(&pdev->dev);
-	iommu    = get_amd_iommu_from_dev(&pdev->dev);
+	dev_data = dev_iommu_priv_get(dev);
+	iommu    = get_amd_iommu_from_dev(dev);
 
 	build_complete_ppr(&cmd, dev_data->devid, pasid, status,
 			   tag, dev_data->pri_tlp);
@@ -2852,6 +2851,7 @@ const struct iommu_ops amd_iommu_ops = {
 	.def_domain_type = amd_iommu_def_domain_type,
 	.dev_enable_feat = amd_iommu_dev_enable_feature,
 	.dev_disable_feat = amd_iommu_dev_disable_feature,
+	.page_response = amd_iommu_page_response,
 	.default_domain_ops = &(const struct iommu_domain_ops) {
 		.attach_dev	= amd_iommu_attach_device,
 		.map_pages	= amd_iommu_map_pages,
drivers/iommu/amd/ppr.c +6 -0
index d4e08800beac..a33ce537b76e 100644
@@ -139,3 +139,9 @@ void amd_iommu_iopf_uninit(struct amd_iommu *iommu)
 	iopf_queue_free(iommu->iopf_queue);
 	iommu->iopf_queue = NULL;
 }
+
+void amd_iommu_page_response(struct device *dev, struct iopf_fault *evt,
+			     struct iommu_page_response *resp)
+{
+	amd_iommu_complete_ppr(dev, resp->pasid, resp->code, resp->grpid);
+}
Signature · 1 line
2.31.1
author maintainer
#18 lore raw

PATCH v8 11/15 iommu/amd: Add IO page fault notifier handler

In reply to Vasant Hegde

From: Wei Huang <[email protected]>

Whenever there is a page fault IOMMU logs entry to ppr log and sends interrupt to host. We have to handle the page fault and respond to IOMMU.

Add support to validate page fault request and hook it to core iommu page fault handler.

---
 drivers/iommu/amd/amd_iommu_types.h |   8 +++
 drivers/iommu/amd/ppr.c             | 100 +++++++++++++++++++++++++++-
 2 files changed, 107 insertions(+), 1 deletion(-)
drivers/iommu/amd/amd_iommu_types.h +8 -0
index 1a014dca7c2e..61ca32b7bb07 100644
@@ -251,6 +251,14 @@
 #define PPR_ENTRY_SIZE		16
 #define PPR_LOG_SIZE		(PPR_ENTRY_SIZE * PPR_LOG_ENTRIES)
 
+/* PAGE_SERVICE_REQUEST PPR Log Buffer Entry flags */
+#define PPR_FLAG_EXEC		0x002	/* Execute permission requested */
+#define PPR_FLAG_READ		0x004	/* Read permission requested */
+#define PPR_FLAG_WRITE		0x020	/* Write permission requested */
+#define PPR_FLAG_US		0x040	/* 1: User, 0: Supervisor */
+#define PPR_FLAG_RVSD		0x080	/* Reserved bit not zero */
+#define PPR_FLAG_GN		0x100	/* GVA and PASID is valid */
+
 #define PPR_REQ_TYPE(x)		(((x) >> 60) & 0xfULL)
 #define PPR_FLAGS(x)		(((x) >> 48) & 0xfffULL)
 #define PPR_DEVID(x)		((x) & 0xffffULL)
drivers/iommu/amd/ppr.c +99 -1

112 lines not on this page · raw message

Signature · 1 line
2.31.1
author maintainer
#19 lore raw

PATCH v8 12/15 iommu/amd: Add support for enable/disable IOPF

In reply to Vasant Hegde

Return success from enable_feature(IOPF) path as this interface is going away. Instead we will enable/disable IOPF support in attach/detach device path.

In attach device path, if device is capable of PRI, then we will add it to per IOMMU IOPF queue and enable PPR support in IOMMU. Also it will attach device to domain even if it fails to enable PRI or add device to IOPF queue as device can continue to work without PRI support.

In detach device patch it follows following sequence:
- Flush the queue for the given device
- Disable PPR support in DTE[devid]
- Remove device from IOPF queue
- Disable device PRI

Also add IOMMU_IOPF as dependency to AMD_IOMMU driver.

---
 drivers/iommu/amd/Kconfig     |  1 +
 drivers/iommu/amd/amd_iommu.h |  4 ++++
 drivers/iommu/amd/iommu.c     | 39 ++++++++++++++++++++++++++-------
 drivers/iommu/amd/ppr.c       | 41 +++++++++++++++++++++++++++++++++++
 4 files changed, 77 insertions(+), 8 deletions(-)
drivers/iommu/amd/Kconfig +1 -0
index 443b2c13c37b..d563f6d496ca 100644
@@ -10,6 +10,7 @@ config AMD_IOMMU
 	select IOMMU_API
 	select IOMMU_IOVA
 	select IOMMU_IO_PGTABLE
+	select IOMMU_IOPF
 	select IOMMUFD_DRIVER if IOMMUFD
 	depends on X86_64 && PCI && ACPI && HAVE_CMPXCHG_DOUBLE
 	help
drivers/iommu/amd/amd_iommu.h +4 -0
index bb9a4c2e40da..dabbb85a71e9 100644
@@ -51,6 +51,10 @@ int amd_iommu_iopf_init(struct amd_iommu *iommu);
 void amd_iommu_iopf_uninit(struct amd_iommu *iommu);
 void amd_iommu_page_response(struct device *dev, struct iopf_fault *evt,
 			     struct iommu_page_response *resp);
+int amd_iommu_iopf_add_device(struct amd_iommu *iommu,
+			      struct iommu_dev_data *dev_data);
+void amd_iommu_iopf_remove_device(struct amd_iommu *iommu,
+				  struct iommu_dev_data *dev_data);
 
 /* GCR3 setup */
 int amd_iommu_set_gcr3(struct iommu_dev_data *dev_data,
drivers/iommu/amd/iommu.c +31 -8
index 11b50a6c744e..9d5bffa16cb6 100644
@@ -2057,8 +2057,17 @@ static int do_attach(struct iommu_dev_data *dev_data,
 		if (ret)
 			return ret;
 
-		if (pdev)
+		if (pdev) {
 			pdev_enable_caps(pdev);
+
+			/*
+			 * Device can continue to function even if IOPF
+			 * enablement failed. Hence in error path just
+			 * disable device PRI support.
+			 */
+			if (amd_iommu_iopf_add_device(iommu, dev_data))
+				pdev_disable_cap_pri(pdev);
+		}
 	} else if (pdev) {
 		pdev_enable_cap_ats(pdev);
 	}
@@ -2130,12 +2139,11 @@ static int attach_device(struct device *dev,
  */
 static void detach_device(struct device *dev)
 {
-	struct protection_domain *domain;
-	struct iommu_dev_data *dev_data;
+	struct iommu_dev_data *dev_data = dev_iommu_priv_get(dev);
+	struct protection_domain *domain = dev_data->domain;
+	struct amd_iommu *iommu = get_amd_iommu_from_dev_data(dev_data);
 	unsigned long flags;
-
-	dev_data = dev_iommu_priv_get(dev);
-	domain   = dev_data->domain;
+	bool ppr = dev_data->ppr;
 
 	spin_lock_irqsave(&domain->lock, flags);
 
@@ -2150,8 +2158,19 @@ static void detach_device(struct device *dev)
 	if (WARN_ON(!dev_data->domain))
 		goto out;
 
+	if (ppr) {
+		iopf_queue_flush_dev(dev);
+
+		/* Updated here so that it gets reflected in DTE */
+		dev_data->ppr = false;
+	}
+
 	do_detach(dev_data);
 
+	/* Remove IOPF handler */
+	if (ppr)
+		amd_iommu_iopf_remove_device(iommu, dev_data);
+
 	if (dev_is_pci(dev))
 		pdev_disable_caps(to_pci_dev(dev));
 
@@ -2814,9 +2833,11 @@ static const struct iommu_dirty_ops amd_dirty_ops = {
 static int amd_iommu_dev_enable_feature(struct device *dev,
 					enum iommu_dev_features feat)
 {
-	int ret;
+	int ret = 0;
 
 	switch (feat) {
+	case IOMMU_DEV_FEAT_IOPF:
+		break;
 	default:
 		ret = -EINVAL;
 		break;
@@ -2827,9 +2848,11 @@ static int amd_iommu_dev_enable_feature(struct device *dev,
 static int amd_iommu_dev_disable_feature(struct device *dev,
 					 enum iommu_dev_features feat)
 {
-	int ret;
+	int ret = 0;
 
 	switch (feat) {
+	case IOMMU_DEV_FEAT_IOPF:
+		break;
 	default:
 		ret = -EINVAL;
 		break;
drivers/iommu/amd/ppr.c +41 -0

44 lines not on this page · raw message

Signature · 1 line
2.31.1
author maintainer
#20 lore raw

PATCH v8 13/15 iommu/amd: Initial SVA support for AMD IOMMU

In reply to Vasant Hegde

This includes :
- Add data structure to track per protection domain dev/pasid binding details

    protection_domain->dev_data_list will track attached list of
    dev_data/PASIDs.

- Move 'to_pdomain()' to header file

- Add iommu_sva_set_dev_pasid(). It will check whether PASID is supported or not. Also adds PASID to SVA protection domain list as well as to device GCR3 table.

- Add iommu_ops.remove_dev_pasid support. It will unbind PASID from device. Also remove pasid data from protection domain device list.

- Add IOMMU_SVA as dependency to AMD_IOMMU driver

For a given PASID, iommu_set_dev_pasid() will bind all devices to same SVA protection domain (1 PASID : 1 SVA protection domain : N devices). This protection domain is different from device protection domain (one that's mapped in attach_device() path). IOMMU uses domain ID for caching, invalidation, etc. In SVA mode it will use per-device-domain-ID. Hence in invalidation path we retrieve domain ID from gcr3_info_table structure and use that for invalidation.

---
 drivers/iommu/amd/Kconfig           |   1 +
 drivers/iommu/amd/Makefile          |   2 +-
 drivers/iommu/amd/amd_iommu.h       |  11 +++
 drivers/iommu/amd/amd_iommu_types.h |  19 +++++
 drivers/iommu/amd/iommu.c           |   7 +-
 drivers/iommu/amd/pasid.c           | 122 ++++++++++++++++++++++++++++
 6 files changed, 156 insertions(+), 6 deletions(-)
 create mode 100644 drivers/iommu/amd/pasid.c
drivers/iommu/amd/Kconfig +1 -0
index d563f6d496ca..68d8fc107cb9 100644
@@ -10,6 +10,7 @@ config AMD_IOMMU
 	select IOMMU_API
 	select IOMMU_IOVA
 	select IOMMU_IO_PGTABLE
+	select IOMMU_SVA
 	select IOMMU_IOPF
 	select IOMMUFD_DRIVER if IOMMUFD
 	depends on X86_64 && PCI && ACPI && HAVE_CMPXCHG_DOUBLE
drivers/iommu/amd/Makefile +1 -1
index 93b11b6d764f..9de33b2d42f5 100644
@@ -1,3 +1,3 @@
 # SPDX-License-Identifier: GPL-2.0-only
-obj-$(CONFIG_AMD_IOMMU) += iommu.o init.o quirks.o io_pgtable.o io_pgtable_v2.o ppr.o
+obj-$(CONFIG_AMD_IOMMU) += iommu.o init.o quirks.o io_pgtable.o io_pgtable_v2.o ppr.o pasid.o
 obj-$(CONFIG_AMD_IOMMU_DEBUGFS) += debugfs.o
drivers/iommu/amd/amd_iommu.h +11 -0
index dabbb85a71e9..49462d073132 100644
@@ -44,6 +44,12 @@ extern int amd_iommu_guest_ir;
 extern enum io_pgtable_fmt amd_iommu_pgtable;
 extern int amd_iommu_gpt_level;
 
+/* Protection domain ops */
+int iommu_sva_set_dev_pasid(struct iommu_domain *domain,
+			    struct device *dev, ioasid_t pasid);
+void amd_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid);
+
+/* SVA/PASID */
 bool amd_iommu_pasid_supported(void);
 
 /* IOPF */
@@ -174,6 +180,11 @@ static inline struct amd_iommu *get_amd_iommu_from_dev_data(struct iommu_dev_dat
 	return iommu_get_iommu_dev(dev_data->dev, struct amd_iommu, iommu);
 }
 
+static inline struct protection_domain *to_pdomain(struct iommu_domain *dom)
+{
+	return container_of(dom, struct protection_domain, domain);
+}
+
 bool translation_pre_enabled(struct amd_iommu *iommu);
 bool amd_iommu_is_attach_deferred(struct device *dev);
 int __init add_special_device(u8 type, u8 id, u32 *devid, bool cmd_line);
drivers/iommu/amd/amd_iommu_types.h +19 -0
index 61ca32b7bb07..ed9971cd3f0e 100644
@@ -8,7 +8,9 @@
 #ifndef _ASM_X86_AMD_IOMMU_TYPES_H
 #define _ASM_X86_AMD_IOMMU_TYPES_H
 
+#include <linux/iommu.h>
 #include <linux/types.h>
+#include <linux/mmu_notifier.h>
 #include <linux/mutex.h>
 #include <linux/msi.h>
 #include <linux/list.h>
@@ -511,6 +513,11 @@ extern struct kmem_cache *amd_iommu_irq_cache;
 	list_for_each_entry((iommu), &amd_iommu_list, list)
 #define for_each_iommu_safe(iommu, next) \
 	list_for_each_entry_safe((iommu), (next), &amd_iommu_list, list)
+/* Making iterating over protection_domain->dev_data_list easier */
+#define for_each_pdom_dev_data(pdom_dev_data, pdom) \
+	list_for_each_entry(pdom_dev_data, &pdom->dev_data_list, list)
+#define for_each_pdom_dev_data_safe(pdom_dev_data, next, pdom) \
+	list_for_each_entry_safe((pdom_dev_data), (next), &pdom->dev_data_list, list)
 
 struct amd_iommu;
 struct iommu_domain;
@@ -552,6 +559,16 @@ enum protection_domain_mode {
 	PD_MODE_V2,
 };
 
+/* Track dev_data/PASID list for the protection domain */
+struct pdom_dev_data {
+	/* Points to attached device data */
+	struct iommu_dev_data *dev_data;
+	/* PASID attached to the protection domain */
+	ioasid_t pasid;
+	/* For protection_domain->dev_data_list */
+	struct list_head list;
+};
+
 /*
  * This structure contains generic data for  IOMMU protection domains
  * independent of their use.
@@ -568,6 +585,8 @@ struct protection_domain {
 	bool dirty_tracking;	/* dirty tracking is enabled in the domain */
 	unsigned dev_cnt;	/* devices assigned to this domain */
 	unsigned dev_iommu[MAX_IOMMUS]; /* per-IOMMU reference count */
+
+	struct list_head dev_data_list; /* List of pdom_dev_data */
 };
 
 /*
drivers/iommu/amd/iommu.c +2 -5
index 9d5bffa16cb6..e0f770c9c9a8 100644
@@ -194,11 +194,6 @@ static struct amd_iommu *rlookup_amd_iommu(struct device *dev)
 	return __rlookup_amd_iommu(seg, PCI_SBDF_TO_DEVID(devid));
 }
 
-static struct protection_domain *to_pdomain(struct iommu_domain *dom)
-{
-	return container_of(dom, struct protection_domain, domain);
-}
-
 static struct iommu_dev_data *alloc_dev_data(struct amd_iommu *iommu, u16 devid)
 {
 	struct iommu_dev_data *dev_data;
@@ -2345,6 +2340,7 @@ static struct protection_domain *protection_domain_alloc(unsigned int type)
 
 	spin_lock_init(&domain->lock);
 	INIT_LIST_HEAD(&domain->dev_list);
+	INIT_LIST_HEAD(&domain->dev_data_list);
 	domain->nid = NUMA_NO_NODE;
 
 	switch (type) {
@@ -2874,6 +2870,7 @@ const struct iommu_ops amd_iommu_ops = {
 	.def_domain_type = amd_iommu_def_domain_type,
 	.dev_enable_feat = amd_iommu_dev_enable_feature,
 	.dev_disable_feat = amd_iommu_dev_disable_feature,
+	.remove_dev_pasid = amd_iommu_remove_dev_pasid,
 	.page_response = amd_iommu_page_response,
 	.default_domain_ops = &(const struct iommu_domain_ops) {
 		.attach_dev	= amd_iommu_attach_device,
drivers/iommu/amd/pasid.c new file +122 -0

122 lines not on this page · raw message

Signature · 1 line
2.31.1
author maintainer
#21 lore raw

PATCH v8 14/15 iommu: Add ops->domain_alloc_sva()

In reply to Vasant Hegde

From: Jason Gunthorpe <[email protected]>

Make a new op that receives the device and the mm_struct that the SVA domain should be created for. Unlike domain_alloc_paging() the dev argument is never NULL here.

This allows drivers to fully initialize the SVA domain and allocate the mmu_notifier during allocation. It allows the notifier lifetime to follow the lifetime of the iommu_domain.

Since we have only one call site, upgrade the new op to return ERR_PTR instead of NULL.

[Removed smmu3 related changes - Vasant]
---
 drivers/iommu/iommu-sva.c | 16 +++++++++++-----
 include/linux/iommu.h     |  3 +++
 2 files changed, 14 insertions(+), 5 deletions(-)
drivers/iommu/iommu-sva.c +11 -5
index 640acc804e8c..18a35e798b72 100644
@@ -108,8 +108,8 @@ struct iommu_sva *iommu_sva_bind_device(struct device *dev, struct mm_struct *mm
 
 	/* Allocate a new domain and set it on device pasid. */
 	domain = iommu_sva_domain_alloc(dev, mm);
-	if (!domain) {
-		ret = -ENOMEM;
+	if (IS_ERR(domain)) {
+		ret = PTR_ERR(domain);
 		goto out_free_handle;
 	}
 
@@ -283,9 +283,15 @@ struct iommu_domain *iommu_sva_domain_alloc(struct device *dev,
 	const struct iommu_ops *ops = dev_iommu_ops(dev);
 	struct iommu_domain *domain;
 
-	domain = ops->domain_alloc(IOMMU_DOMAIN_SVA);
-	if (!domain)
-		return NULL;
+	if (ops->domain_alloc_sva) {
+		domain = ops->domain_alloc_sva(dev, mm);
+		if (IS_ERR(domain))
+			return domain;
+	} else {
+		domain = ops->domain_alloc(IOMMU_DOMAIN_SVA);
+		if (!domain)
+			return ERR_PTR(-ENOMEM);
+	}
 
 	domain->type = IOMMU_DOMAIN_SVA;
 	mmgrab(mm);
include/linux/iommu.h +3 -0
index 2e925b5eba53..8aabe83af8f2 100644
@@ -518,6 +518,7 @@ static inline int __iommu_copy_struct_from_user_array(
  *                     Upon failure, ERR_PTR must be returned.
  * @domain_alloc_paging: Allocate an iommu_domain that can be used for
  *                       UNMANAGED, DMA, and DMA_FQ domain types.
+ * @domain_alloc_sva: Allocate an iommu_domain for Shared Virtual Addressing.
  * @probe_device: Add device to iommu driver handling
  * @release_device: Remove device from iommu driver handling
  * @probe_finalize: Do final setup work after the device is added to an IOMMU
@@ -558,6 +559,8 @@ struct iommu_ops {
 		struct device *dev, u32 flags, struct iommu_domain *parent,
 		const struct iommu_user_data *user_data);
 	struct iommu_domain *(*domain_alloc_paging)(struct device *dev);
+	struct iommu_domain *(*domain_alloc_sva)(struct device *dev,
+						 struct mm_struct *mm);
 
 	struct iommu_device *(*probe_device)(struct device *dev);
 	void (*release_device)(struct device *dev);
Signature · 1 line
2.31.1
author maintainer
#22 lore raw

PATCH v8 15/15 iommu/amd: Add SVA domain support

In reply to Vasant Hegde

- Allocate SVA domain and setup mmu notifier. In free path unregister mmu notifier and free protection domain.

- Add mmu notifier callback function. It will retrieve SVA protection domain and invalidates IO/TLB.

---
 drivers/iommu/amd/amd_iommu.h       |  5 ++
 drivers/iommu/amd/amd_iommu_types.h |  1 +
 drivers/iommu/amd/iommu.c           | 10 ++--
 drivers/iommu/amd/pasid.c           | 80 +++++++++++++++++++++++++++++
 4 files changed, 93 insertions(+), 3 deletions(-)
drivers/iommu/amd/amd_iommu.h +5 -0
index 49462d073132..0005da7af739 100644
@@ -45,6 +45,11 @@ extern enum io_pgtable_fmt amd_iommu_pgtable;
 extern int amd_iommu_gpt_level;
 
 /* Protection domain ops */
+struct protection_domain *protection_domain_alloc(unsigned int type);
+void protection_domain_free(struct protection_domain *domain);
+struct iommu_domain *amd_iommu_domain_alloc_sva(struct device *dev,
+						struct mm_struct *mm);
+void amd_iommu_domain_free(struct iommu_domain *dom);
 int iommu_sva_set_dev_pasid(struct iommu_domain *domain,
 			    struct device *dev, ioasid_t pasid);
 void amd_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid);
drivers/iommu/amd/amd_iommu_types.h +1 -0
index ed9971cd3f0e..2b76b5dedc1d 100644
@@ -586,6 +586,7 @@ struct protection_domain {
 	unsigned dev_cnt;	/* devices assigned to this domain */
 	unsigned dev_iommu[MAX_IOMMUS]; /* per-IOMMU reference count */
 
+	struct mmu_notifier mn;	/* mmu notifier for the SVA domain */
 	struct list_head dev_data_list; /* List of pdom_dev_data */
 };
 
drivers/iommu/amd/iommu.c +7 -3
index e0f770c9c9a8..f7ba6552ccad 100644
@@ -2280,7 +2280,7 @@ static void cleanup_domain(struct protection_domain *domain)
 	WARN_ON(domain->dev_cnt != 0);
 }
 
-static void protection_domain_free(struct protection_domain *domain)
+void protection_domain_free(struct protection_domain *domain)
 {
 	if (!domain)
 		return;
@@ -2323,7 +2323,7 @@ static int protection_domain_init_v2(struct protection_domain *pdom)
 	return 0;
 }
 
-static struct protection_domain *protection_domain_alloc(unsigned int type)
+struct protection_domain *protection_domain_alloc(unsigned int type)
 {
 	struct io_pgtable_ops *pgtbl_ops;
 	struct protection_domain *domain;
@@ -2346,6 +2346,7 @@ static struct protection_domain *protection_domain_alloc(unsigned int type)
 	switch (type) {
 	/* No need to allocate io pgtable ops in passthrough mode */
 	case IOMMU_DOMAIN_IDENTITY:
+	case IOMMU_DOMAIN_SVA:
 		return domain;
 	case IOMMU_DOMAIN_DMA:
 		pgtable = amd_iommu_pgtable;
@@ -2465,7 +2466,7 @@ amd_iommu_domain_alloc_user(struct device *dev, u32 flags,
 	return do_iommu_domain_alloc(type, dev, flags);
 }
 
-static void amd_iommu_domain_free(struct iommu_domain *dom)
+void amd_iommu_domain_free(struct iommu_domain *dom)
 {
 	struct protection_domain *domain;
 	unsigned long flags;
@@ -2833,6 +2834,7 @@ static int amd_iommu_dev_enable_feature(struct device *dev,
 
 	switch (feat) {
 	case IOMMU_DEV_FEAT_IOPF:
+	case IOMMU_DEV_FEAT_SVA:
 		break;
 	default:
 		ret = -EINVAL;
@@ -2848,6 +2850,7 @@ static int amd_iommu_dev_disable_feature(struct device *dev,
 
 	switch (feat) {
 	case IOMMU_DEV_FEAT_IOPF:
+	case IOMMU_DEV_FEAT_SVA:
 		break;
 	default:
 		ret = -EINVAL;
@@ -2860,6 +2863,7 @@ const struct iommu_ops amd_iommu_ops = {
 	.capable = amd_iommu_capable,
 	.domain_alloc = amd_iommu_domain_alloc,
 	.domain_alloc_user = amd_iommu_domain_alloc_user,
+	.domain_alloc_sva = amd_iommu_domain_alloc_sva,
 	.probe_device = amd_iommu_probe_device,
 	.release_device = amd_iommu_release_device,
 	.probe_finalize = amd_iommu_probe_finalize,
drivers/iommu/amd/pasid.c +80 -0

89 lines not on this page · raw message

Signature · 1 line
2.31.1
maintainer Applied
#23 lore raw
In reply to Vasant Hegde
Quoted 19 lines · Vasant Hegde

Jason Gunthorpe (1):
iommu: Add ops->domain_alloc_sva()

Suravee Suthikulpanit (3):
iommu/amd: Move PPR-related functions into ppr.c iommu/amd: Define per-IOMMU iopf_queue iommu/amd: Add support for page response

Vasant Hegde (9):
iommu/amd: Rename amd_iommu_v2_supported() as amd_iommu_pasid_supported() iommu/amd: Introduce per device DTE update function iommu/amd: Fix PPR interrupt processing logic
iommu/amd: Introduce iommu_dev_data.max_pasids
iommu/amd: Setup GCR3 table in advance if domain is SVA capable iommu/amd: Enable PCI features based on attached domain capability iommu/amd: Add support for enable/disable IOPF
iommu/amd: Initial SVA support for AMD IOMMU
iommu/amd: Add SVA domain support

Quoted 3 lines

Wei Huang (2):
iommu/amd: Add support for enabling/disabling IOMMU features iommu/amd: Add IO page fault notifier handler

Applied, thanks.