Skip to contentSkip to conversation

Make set_dev_pasid op supporting domain replacement

PATCH Superseded Yi Liu 6 participants 34 messages Sep 12, 2024 – Oct 15, 2024

Subsystems (6)

v2 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Yi Liu

author
#0 lore raw

v2 cover Make set_dev_pasid op supporting domain replacement

This splits the preparation works of the iommu and the Intel iommu driver out from the iommufd pasid attach/replace series. [1]

To support domain replacement, the definition of the set_dev_pasid op needs to be enhanced. Meanwhile, the existing set_dev_pasid callbacks should be extended as well to suit the new definition.

This series first prepares the Intel iommu set_dev_pasid op for the new definition, adds the missing set_dev_pasid support for nested domain, makes ARM SMMUv3 set_dev_pasid op to suit the new definition, and in the end enhances the definition of set_dev_pasid op. The AMD set_dev_pasid callback is extended to fail if the caller tries to do domain replacement to meet the new definition of set_dev_pasid op. AMD iommu driver would support it later per Vasant [2].

[1] https://lore.kernel.org/linux-iommu/[email protected]/
[2] https://lore.kernel.org/linux-iommu/[email protected]/

v2:
- Make ARM SMMUv3 set_dev_pasid op support domain replacement (Jason)
- Drop patch 03 of v1 (Kevin)
- Multiple tweaks in VT-d driver (Kevin)

v1: https://lore.kernel.org/linux-iommu/[email protected]/

Regards,
Yi Liu

Jason Gunthorpe (1):
iommu/arm-smmu-v3: Make smmuv3 set_dev_pasid() op support replace

Lu Baolu (1):
iommu/vt-d: Add set_dev_pasid callback for nested domain

Yi Liu (4):
iommu: Pass old domain to set_dev_pasid op iommu/vt-d: Move intel_drain_pasid_prq() into
intel_pasid_tear_down_entry()
iommu/vt-d: Make intel_iommu_set_dev_pasid() to handle domain
replacement
iommu: Make set_dev_pasid op support domain replacement

 drivers/iommu/amd/amd_iommu.h                 |   3 +-
 drivers/iommu/amd/pasid.c                     |   6 +-
 .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c   |   5 +-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   |   8 +-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   |   2 +-
 drivers/iommu/intel/iommu.c                   | 122 ++++++++++++------
 drivers/iommu/intel/iommu.h                   |   3 +
 drivers/iommu/intel/nested.c                  |   1 +
 drivers/iommu/intel/pasid.c                   |  13 +-
 drivers/iommu/intel/pasid.h                   |   8 +-
 drivers/iommu/intel/svm.c                     |   6 +-
 drivers/iommu/iommu.c                         |   3 +-
 include/linux/iommu.h                         |   5 +-
 13 files changed, 129 insertions(+), 56 deletions(-)
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH v2 1/6 iommu: Pass old domain to set_dev_pasid op

To support domain replacement for pasid, the underlying iommu driver needs to know the old domain hence be able to clean up the existing attachment. It would be much convenient for iommu layer to pass down the old domain. Otherwise, iommu drivers would need to track domain for pasids by themselves, this would duplicate code among the iommu drivers. Or iommu drivers would rely group->pasid_array to get domain, which may not always the correct one.

---
 drivers/iommu/amd/amd_iommu.h                   | 3 ++-
 drivers/iommu/amd/pasid.c                       | 3 ++-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 ++-
 drivers/iommu/intel/iommu.c                     | 6 ++++--
 drivers/iommu/intel/svm.c                       | 3 ++-
 drivers/iommu/iommu.c                           | 3 ++-
 include/linux/iommu.h                           | 2 +-
 7 files changed, 15 insertions(+), 8 deletions(-)
drivers/iommu/amd/amd_iommu.h +2 -1
index 2d5945c982bd..271406a45e33 100644
@@ -51,7 +51,8 @@ struct iommu_domain *amd_iommu_domain_alloc_sva(struct device *dev,
 						struct mm_struct *mm);
 void amd_iommu_domain_free(struct iommu_domain *dom);
 int iommu_sva_set_dev_pasid(struct iommu_domain *domain,
-			    struct device *dev, ioasid_t pasid);
+			    struct device *dev, ioasid_t pasid,
+			    struct iommu_domain *old);
 void amd_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid,
 				struct iommu_domain *domain);
 
drivers/iommu/amd/pasid.c +2 -1
index a68215f2b3e1..77bf5f5f947a 100644
@@ -100,7 +100,8 @@ static const struct mmu_notifier_ops sva_mn = {
 };
 
 int iommu_sva_set_dev_pasid(struct iommu_domain *domain,
-			    struct device *dev, ioasid_t pasid)
+			    struct device *dev, ioasid_t pasid,
+			    struct iommu_domain *old)
 {
 	struct pdom_dev_data *pdom_dev_data;
 	struct protection_domain *sva_pdom = to_pdomain(domain);
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +2 -1
index a7c36654dee5..645da7b69bed 100644
@@ -332,7 +332,8 @@ void arm_smmu_sva_notifier_synchronize(void)
 }
 
 static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
-				      struct device *dev, ioasid_t id)
+				      struct device *dev, ioasid_t id,
+				      struct iommu_domain *old)
 {
 	struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
 	struct arm_smmu_master *master = dev_iommu_priv_get(dev);
drivers/iommu/intel/iommu.c +4 -2
index 34006b6e89eb..1a2a5cf4ef60 100644
@@ -4283,7 +4283,8 @@ static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid,
 }
 
 static int intel_iommu_set_dev_pasid(struct iommu_domain *domain,
-				     struct device *dev, ioasid_t pasid)
+				     struct device *dev, ioasid_t pasid,
+				     struct iommu_domain *old)
 {
 	struct device_domain_info *info = dev_iommu_priv_get(dev);
 	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
@@ -4569,7 +4570,8 @@ static int identity_domain_attach_dev(struct iommu_domain *domain, struct device
 }
 
 static int identity_domain_set_dev_pasid(struct iommu_domain *domain,
-					 struct device *dev, ioasid_t pasid)
+					 struct device *dev, ioasid_t pasid,
+					 struct iommu_domain *old)
 {
 	struct device_domain_info *info = dev_iommu_priv_get(dev);
 	struct intel_iommu *iommu = info->iommu;
drivers/iommu/intel/svm.c +2 -1
index 0e3a9b38bef2..5ae1df7598b7 100644
@@ -194,7 +194,8 @@ static const struct mmu_notifier_ops intel_mmuops = {
 };
 
 static int intel_svm_set_dev_pasid(struct iommu_domain *domain,
-				   struct device *dev, ioasid_t pasid)
+				   struct device *dev, ioasid_t pasid,
+				   struct iommu_domain *old)
 {
 	struct device_domain_info *info = dev_iommu_priv_get(dev);
 	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
drivers/iommu/iommu.c +2 -1
index 83c8e617a2c5..f3f81c04b8fb 100644
@@ -3331,7 +3331,8 @@ static int __iommu_set_group_pasid(struct iommu_domain *domain,
 	int ret;
 
 	for_each_group_device(group, device) {
-		ret = domain->ops->set_dev_pasid(domain, device->dev, pasid);
+		ret = domain->ops->set_dev_pasid(domain, device->dev,
+						 pasid, NULL);
 		if (ret)
 			goto err_revert;
 	}
include/linux/iommu.h +1 -1
index bd722f473635..32dce80aa7fd 100644
@@ -642,7 +642,7 @@ struct iommu_ops {
 struct iommu_domain_ops {
 	int (*attach_dev)(struct iommu_domain *domain, struct device *dev);
 	int (*set_dev_pasid)(struct iommu_domain *domain, struct device *dev,
-			     ioasid_t pasid);
+			     ioasid_t pasid, struct iommu_domain *old);
 
 	int (*map_pages)(struct iommu_domain *domain, unsigned long iova,
 			 phys_addr_t paddr, size_t pgsize, size_t pgcount,
Signature · 1 line
2.34.1
Reviewed-by on v2 1/6
#2 lore raw

Re: [PATCH v2 1/6] iommu: Pass old domain to set_dev_pasid op

Quoted 10 lines · Yi Liu

To support domain replacement for pasid, the underlying iommu driver needs to know the old domain hence be able to clean up the existing attachment. It would be much convenient for iommu layer to pass down the old domain. Otherwise, iommu drivers would need to track domain for pasids by themselves, this would duplicate code among the iommu drivers. Or iommu drivers would rely group->pasid_array to get domain, which may not always the correct one.

Quoted 9 lines
---
 drivers/iommu/amd/amd_iommu.h                   | 3 ++-
 drivers/iommu/amd/pasid.c                       | 3 ++-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 ++-
 drivers/iommu/intel/iommu.c                     | 6 ++++--
 drivers/iommu/intel/svm.c                       | 3 ++-
 drivers/iommu/iommu.c                           | 3 ++-
 include/linux/iommu.h                           | 2 +-
 7 files changed, 15 insertions(+), 8 deletions(-)

Jason

Reviewed-by on v2 1/6
#3 lore raw

RE: [PATCH v2 1/6] iommu: Pass old domain to set_dev_pasid op

In reply to Yi Liu
Quoted 14 lines

From: Liu, Yi L <[email protected]>
Sent: Thursday, September 12, 2024 9:04 PM

To support domain replacement for pasid, the underlying iommu driver needs
to know the old domain hence be able to clean up the existing attachment. It would be much convenient for iommu layer to pass down the old domain. Otherwise, iommu drivers would need to track domain for pasids by themselves,
this would duplicate code among the iommu drivers. Or iommu drivers would
rely group->pasid_array to get domain, which may not always the correct one.

Quoted 2 lines
author
#4 lore raw

PATCH v2 2/6 iommu/vt-d: Move intel_drain_pasid_prq() into intel_pasid_tear_down_entry()

In reply to Yi Liu

Draining PRQ is mostly conjuncted with pasid teardown, and with more callers coming, move it into it in the intel_pasid_tear_down_entry(). But there is scenario that only teardown pasid entry but no PRQ drain, so passing a flag to mark it.

---
 drivers/iommu/intel/iommu.c |  8 ++++----
 drivers/iommu/intel/pasid.c | 13 +++++++++++--
 drivers/iommu/intel/pasid.h |  8 +++++---
 drivers/iommu/intel/svm.c   |  3 ++-
 4 files changed, 22 insertions(+), 10 deletions(-)
drivers/iommu/intel/iommu.c +4 -4
index 1a2a5cf4ef60..80b587de226d 100644
@@ -3376,7 +3376,7 @@ void device_block_translation(struct device *dev)
 	if (!dev_is_real_dma_subdevice(dev)) {
 		if (sm_supported(iommu))
 			intel_pasid_tear_down_entry(iommu, dev,
-						    IOMMU_NO_PASID, false);
+						    IOMMU_NO_PASID, 0);
 		else
 			domain_context_clear(info);
 	}
@@ -4258,7 +4258,7 @@ static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid,
 	unsigned long flags;
 
 	if (domain->type == IOMMU_DOMAIN_IDENTITY) {
-		intel_pasid_tear_down_entry(iommu, dev, pasid, false);
+		intel_pasid_tear_down_entry(iommu, dev, pasid, 0);
 		return;
 	}
 
@@ -4278,8 +4278,8 @@ static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid,
 	domain_detach_iommu(dmar_domain, iommu);
 	intel_iommu_debugfs_remove_dev_pasid(dev_pasid);
 	kfree(dev_pasid);
-	intel_pasid_tear_down_entry(iommu, dev, pasid, false);
-	intel_drain_pasid_prq(dev, pasid);
+	intel_pasid_tear_down_entry(iommu, dev, pasid,
+				    INTEL_PASID_TEARDOWN_DRAIN_PRQ);
 }
 
 static int intel_iommu_set_dev_pasid(struct iommu_domain *domain,
drivers/iommu/intel/pasid.c +11 -2
index b51fc268dc84..ceb9c5274a39 100644
@@ -236,8 +236,13 @@ devtlb_invalidation_with_pasid(struct intel_iommu *iommu,
 		qi_flush_dev_iotlb_pasid(iommu, sid, pfsid, pasid, qdep, 0, 64 - VTD_PAGE_SHIFT);
 }
 
+/*
+ * Not all PASID entry destroy requires PRQ drain as it can be handled in
+ * the remove_dev_pasid path. Caller should be clear about it and set the
+ * @flags properly.
+ */
 void intel_pasid_tear_down_entry(struct intel_iommu *iommu, struct device *dev,
-				 u32 pasid, bool fault_ignore)
+				 u32 pasid, u32 flags)
 {
 	struct pasid_entry *pte;
 	u16 did, pgtt;
@@ -251,7 +256,8 @@ void intel_pasid_tear_down_entry(struct intel_iommu *iommu, struct device *dev,
 
 	did = pasid_get_domain_id(pte);
 	pgtt = pasid_pte_get_pgtt(pte);
-	intel_pasid_clear_entry(dev, pasid, fault_ignore);
+	intel_pasid_clear_entry(dev, pasid,
+				flags & INTEL_PASID_TEARDOWN_IGNORE_FAULT);
 	spin_unlock(&iommu->lock);
 
 	if (!ecap_coherent(iommu->ecap))
@@ -267,6 +273,9 @@ void intel_pasid_tear_down_entry(struct intel_iommu *iommu, struct device *dev,
 	/* Device IOTLB doesn't need to be flushed in caching mode. */
 	if (!cap_caching_mode(iommu->cap))
 		devtlb_invalidation_with_pasid(iommu, dev, pasid);
+
+	if (flags & INTEL_PASID_TEARDOWN_DRAIN_PRQ)
+		intel_drain_pasid_prq(dev, pasid);
 }
 
 /*
drivers/iommu/intel/pasid.h +5 -3
index dde6d3ba5ae0..6eb849ec5fb8 100644
@@ -303,9 +303,11 @@ int intel_pasid_setup_pass_through(struct intel_iommu *iommu,
 				   struct device *dev, u32 pasid);
 int intel_pasid_setup_nested(struct intel_iommu *iommu, struct device *dev,
 			     u32 pasid, struct dmar_domain *domain);
-void intel_pasid_tear_down_entry(struct intel_iommu *iommu,
-				 struct device *dev, u32 pasid,
-				 bool fault_ignore);
+
+#define INTEL_PASID_TEARDOWN_IGNORE_FAULT	(1U << 0)
+#define INTEL_PASID_TEARDOWN_DRAIN_PRQ		(1U << 1)
+void intel_pasid_tear_down_entry(struct intel_iommu *iommu, struct device *dev,
+				 u32 pasid, u32 flags);
 void intel_pasid_setup_page_snoop_control(struct intel_iommu *iommu,
 					  struct device *dev, u32 pasid);
 int intel_pasid_setup_sm_context(struct device *dev);
drivers/iommu/intel/svm.c +2 -1
index 5ae1df7598b7..3c1e105b9da6 100644
@@ -176,7 +176,8 @@ static void intel_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
 	list_for_each_entry(dev_pasid, &domain->dev_pasids, link_domain) {
 		info = dev_iommu_priv_get(dev_pasid->dev);
 		intel_pasid_tear_down_entry(info->iommu, dev_pasid->dev,
-					    dev_pasid->pasid, true);
+					    dev_pasid->pasid,
+					    INTEL_PASID_TEARDOWN_IGNORE_FAULT);
 	}
 	spin_unlock_irqrestore(&domain->lock, flags);
 
Signature · 1 line
2.34.1
maintainer
#5 lore raw

Re: [PATCH v2 2/6] iommu/vt-d: Move intel_drain_pasid_prq() into intel_pasid_tear_down_entry()

Quoted 3 lines · Yi Liu

Draining PRQ is mostly conjuncted with pasid teardown, and with more callers coming, move it into it in the intel_pasid_tear_down_entry(). But there is scenario that only teardown pasid entry but no PRQ drain, so passing a flag to mark it.

Is it a reasonable case where PRI needs to be drained but the pasid entry won't be torn down? For example, could this happen when a PRI is disabled?

Quoted 8 lines
---
  drivers/iommu/intel/iommu.c |  8 ++++----
  drivers/iommu/intel/pasid.c | 13 +++++++++++--
  drivers/iommu/intel/pasid.h |  8 +++++---
  drivers/iommu/intel/svm.c   |  3 ++-
  4 files changed, 22 insertions(+), 10 deletions(-)

Thanks,
baolu

author
#6 lore raw

Re: [PATCH v2 2/6] iommu/vt-d: Move intel_drain_pasid_prq() into intel_pasid_tear_down_entry()

Quoted 8 lines · Baolu Lu
Quoted 3 lines · Yi Liu

Draining PRQ is mostly conjuncted with pasid teardown, and with more callers coming, move it into it in the intel_pasid_tear_down_entry(). But there is scenario that only teardown pasid entry but no PRQ drain, so passing a flag to mark it.

Is it a reasonable case where PRI needs to be drained but the pasid entry won't be torn down? For example, could this happen when a PRI is disabled?

in concept, yes. But it seems no more than a debugging method in my opinion. I cannot map it to a usage so far.

Quoted 11 lines
Quoted 8 lines

---

  drivers/iommu/intel/iommu.c |  8 ++++----
  drivers/iommu/intel/pasid.c | 13 +++++++++++--
  drivers/iommu/intel/pasid.h |  8 +++++---
  drivers/iommu/intel/svm.c   |  3 ++-
  4 files changed, 22 insertions(+), 10 deletions(-)

Thanks,
baolu

Signature · 2 lines
Regards,
Yi Liu
maintainer
#7 lore raw

Re: [PATCH v2 2/6] iommu/vt-d: Move intel_drain_pasid_prq() into intel_pasid_tear_down_entry()

In reply to Yi Liu
Quoted 6 lines · Yi Liu
drivers/iommu/intel/pasid.c +0 -0
index b51fc268dc84..ceb9c5274a39 100644
@@ -236,8 +236,13 @@ devtlb_invalidation_with_pasid(struct intel_iommu *iommu,
  		qi_flush_dev_iotlb_pasid(iommu, sid, pfsid, pasid, qdep, 0, 64 - VTD_PAGE_SHIFT);

drivers/iommu/intel/pasid.c devtlb_invalidation_with_pasid(struct intel_iommu *iommu, · line 245

Quoted 10 lines
drivers/iommu/intel/pasid.c +6 -1
  }
 
+/*
+ * Not all PASID entry destroy requires PRQ drain as it can be handled in
+ * the remove_dev_pasid path. Caller should be clear about it and set the
+ * @flags properly.
+ */
  void intel_pasid_tear_down_entry(struct intel_iommu *iommu, struct device *dev,
-				 u32 pasid, bool fault_ignore)
+				 u32 pasid, u32 flags)

As a generic opt-in bit flags, why not using 'unsigned int'?

Thanks,
baolu

author
#8 lore raw

Re: [PATCH v2 2/6] iommu/vt-d: Move intel_drain_pasid_prq() into intel_pasid_tear_down_entry()

Quoted 12 lines · Baolu Lu
Quoted 11 lines · Yi Liu
drivers/iommu/intel/pasid.c +0 -0
index b51fc268dc84..ceb9c5274a39 100644
@@ -236,8 +236,13 @@ devtlb_invalidation_with_pasid(struct intel_iommu 
 *iommu,

          qi_flush_dev_iotlb_pasid(iommu, sid, pfsid, pasid, qdep, 0, 64
- VTD_PAGE_SHIFT);
  }
+/*
+ * Not all PASID entry destroy requires PRQ drain as it can be handled in

Quoted 8 lines
Quoted 6 lines

+ * the remove_dev_pasid path. Caller should be clear about it and set the
+ * @flags properly.
+ */
  void intel_pasid_tear_down_entry(struct intel_iommu *iommu, struct device *dev, -                 u32 pasid, bool fault_ignore)
+                 u32 pasid, u32 flags)

As a generic opt-in bit flags, why not using 'unsigned int'?

aha, I can use 'unsigned int'.

Signature · 2 lines
Regards,
Yi Liu
Reviewed-by on v2 2/6
#9 lore raw

RE: [PATCH v2 2/6] iommu/vt-d: Move intel_drain_pasid_prq() into intel_pasid_tear_down_entry()

In reply to Yi Liu
Quoted 8 lines

From: Liu, Yi L <[email protected]>
Sent: Thursday, September 12, 2024 9:04 PM

+/*
+ * Not all PASID entry destroy requires PRQ drain as it can be handled in
+ * the remove_dev_pasid path. Caller should be clear about it and set the
+ * @flags properly.
+ */

/*
* Caller can request to drain PRQ in this helper if it hasn't done so,
* e.g. in a path which doesn't follow remove_dev_pasid(). */

author
#10 lore raw

PATCH v2 3/6 iommu/vt-d: Make intel_iommu_set_dev_pasid() to handle domain replacement

In reply to Yi Liu

set_dev_pasid op is going to support domain replacement and keep the old hardware config if it fails. Make the Intel iommu driver be prepared for it.

---
 drivers/iommu/intel/iommu.c | 98 ++++++++++++++++++++++++-------------
 1 file changed, 65 insertions(+), 33 deletions(-)
drivers/iommu/intel/iommu.c +65 -33
index 80b587de226d..6f5a8e549f3f 100644
@@ -4248,8 +4248,8 @@ static int intel_iommu_iotlb_sync_map(struct iommu_domain *domain,
 	return 0;
 }
 
-static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid,
-					 struct iommu_domain *domain)
+static void domain_remove_dev_pasid(struct iommu_domain *domain,
+				    struct device *dev, ioasid_t pasid)
 {
 	struct device_domain_info *info = dev_iommu_priv_get(dev);
 	struct dev_pasid_info *curr, *dev_pasid = NULL;
@@ -4257,11 +4257,6 @@ static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid,
 	struct dmar_domain *dmar_domain;
 	unsigned long flags;
 
-	if (domain->type == IOMMU_DOMAIN_IDENTITY) {
-		intel_pasid_tear_down_entry(iommu, dev, pasid, 0);
-		return;
-	}
-
 	dmar_domain = to_dmar_domain(domain);
 	spin_lock_irqsave(&dmar_domain->lock, flags);
 	list_for_each_entry(curr, &dmar_domain->dev_pasids, link_domain) {
@@ -4278,13 +4273,24 @@ static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid,
 	domain_detach_iommu(dmar_domain, iommu);
 	intel_iommu_debugfs_remove_dev_pasid(dev_pasid);
 	kfree(dev_pasid);
+}
+
+static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid,
+					 struct iommu_domain *domain)
+{
+	struct device_domain_info *info = dev_iommu_priv_get(dev);
+	struct intel_iommu *iommu = info->iommu;
+
 	intel_pasid_tear_down_entry(iommu, dev, pasid,
 				    INTEL_PASID_TEARDOWN_DRAIN_PRQ);
+	if (domain->type == IOMMU_DOMAIN_IDENTITY)
+		return;
+	domain_remove_dev_pasid(domain, dev, pasid);
 }
 
-static int intel_iommu_set_dev_pasid(struct iommu_domain *domain,
-				     struct device *dev, ioasid_t pasid,
-				     struct iommu_domain *old)
+static struct dev_pasid_info *
+domain_prepare_dev_pasid(struct iommu_domain *domain,
+			 struct device *dev, ioasid_t pasid)
 {
 	struct device_domain_info *info = dev_iommu_priv_get(dev);
 	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
@@ -4293,22 +4299,13 @@ static int intel_iommu_set_dev_pasid(struct iommu_domain *domain,
 	unsigned long flags;
 	int ret;
 
-	if (!pasid_supported(iommu) || dev_is_real_dma_subdevice(dev))
-		return -EOPNOTSUPP;
-
-	if (domain->dirty_ops)
-		return -EINVAL;
-
-	if (context_copied(iommu, info->bus, info->devfn))
-		return -EBUSY;
-
 	ret = prepare_domain_attach_device(domain, dev);
 	if (ret)
-		return ret;
+		return ERR_PTR(ret);
 
 	dev_pasid = kzalloc(sizeof(*dev_pasid), GFP_KERNEL);
 	if (!dev_pasid)
-		return -ENOMEM;
+		return ERR_PTR(-ENOMEM);
 
 	ret = domain_attach_iommu(dmar_domain, iommu);
 	if (ret)
@@ -4318,6 +4315,47 @@ static int intel_iommu_set_dev_pasid(struct iommu_domain *domain,
 	if (ret)
 		goto out_detach_iommu;
 
+	dev_pasid->dev = dev;
+	dev_pasid->pasid = pasid;
+	spin_lock_irqsave(&dmar_domain->lock, flags);
+	list_add(&dev_pasid->link_domain, &dmar_domain->dev_pasids);
+	spin_unlock_irqrestore(&dmar_domain->lock, flags);
+
+	return dev_pasid;
+out_detach_iommu:
+	domain_detach_iommu(dmar_domain, iommu);
+out_free:
+	kfree(dev_pasid);
+	return ERR_PTR(ret);
+}
+
+static int intel_iommu_set_dev_pasid(struct iommu_domain *domain,
+				     struct device *dev, ioasid_t pasid,
+				     struct iommu_domain *old)
+{
+	struct device_domain_info *info = dev_iommu_priv_get(dev);
+	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
+	struct intel_iommu *iommu = info->iommu;
+	struct dev_pasid_info *dev_pasid;
+	int ret;
+
+	if (!pasid_supported(iommu) || dev_is_real_dma_subdevice(dev))
+		return -EOPNOTSUPP;
+
+	if (domain->dirty_ops)
+		return -EINVAL;
+
+	if (context_copied(iommu, info->bus, info->devfn))
+		return -EBUSY;
+
+	dev_pasid = domain_prepare_dev_pasid(domain, dev, pasid);
+	if (IS_ERR(dev_pasid))
+		return PTR_ERR(dev_pasid);
+
+	/* Clear the old configuration if it already exists */
+	intel_pasid_tear_down_entry(iommu, dev, pasid,
+				    INTEL_PASID_TEARDOWN_DRAIN_PRQ);
+
 	if (dmar_domain->use_first_level)
 		ret = domain_setup_first_level(iommu, dmar_domain,
 					       dev, pasid);
@@ -4325,24 +4363,18 @@ static int intel_iommu_set_dev_pasid(struct iommu_domain *domain,
 		ret = intel_pasid_setup_second_level(iommu, dmar_domain,
 						     dev, pasid);
 	if (ret)
-		goto out_unassign_tag;
+		goto out_undo_dev_pasid;
 
-	dev_pasid->dev = dev;
-	dev_pasid->pasid = pasid;
-	spin_lock_irqsave(&dmar_domain->lock, flags);
-	list_add(&dev_pasid->link_domain, &dmar_domain->dev_pasids);
-	spin_unlock_irqrestore(&dmar_domain->lock, flags);
+	if (old)
+		domain_remove_dev_pasid(old, dev, pasid);
 
 	if (domain->type & __IOMMU_DOMAIN_PAGING)
 		intel_iommu_debugfs_create_dev_pasid(dev_pasid);
 
 	return 0;
-out_unassign_tag:
-	cache_tag_unassign_domain(dmar_domain, dev, pasid);
-out_detach_iommu:
-	domain_detach_iommu(dmar_domain, iommu);
-out_free:
-	kfree(dev_pasid);
+
+out_undo_dev_pasid:
+	domain_remove_dev_pasid(domain, dev, pasid);
 	return ret;
 }
 
Signature · 1 line
2.34.1
maintainer
#11 lore raw

Re: [PATCH v2 3/6] iommu/vt-d: Make intel_iommu_set_dev_pasid() to handle domain replacement

Quoted 42 lines · Yi Liu

set_dev_pasid op is going to support domain replacement and keep the old hardware config if it fails. Make the Intel iommu driver be prepared for it.

---
  drivers/iommu/intel/iommu.c | 98 ++++++++++++++++++++++++-------------
  1 file changed, 65 insertions(+), 33 deletions(-)
drivers/iommu/intel/iommu.c +4 -7
index 80b587de226d..6f5a8e549f3f 100644
@@ -4248,8 +4248,8 @@ static int intel_iommu_iotlb_sync_map(struct iommu_domain *domain,
  	return 0;
  }
 
-static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid,
-					 struct iommu_domain *domain)
+static void domain_remove_dev_pasid(struct iommu_domain *domain,
+				    struct device *dev, ioasid_t pasid)
  {
  	struct device_domain_info *info = dev_iommu_priv_get(dev);
  	struct dev_pasid_info *curr, *dev_pasid = NULL;
@@ -4257,11 +4257,6 @@ static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid,
  	struct dmar_domain *dmar_domain;
  	unsigned long flags;
 
-	if (domain->type == IOMMU_DOMAIN_IDENTITY) {
-		intel_pasid_tear_down_entry(iommu, dev, pasid, 0);
-		return;
-	}
-
  	dmar_domain = to_dmar_domain(domain);
  	spin_lock_irqsave(&dmar_domain->lock, flags);
  	list_for_each_entry(curr, &dmar_domain->dev_pasids, link_domain) {
@@ -4278,13 +4273,24 @@ static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid,
  	domain_detach_iommu(dmar_domain, iommu);
  	intel_iommu_debugfs_remove_dev_pasid(dev_pasid);
  	kfree(dev_pasid);
+}
+

drivers/iommu/intel/iommu.c static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid, · line 4287

Quoted 10 lines
drivers/iommu/intel/iommu.c +8 -0
+static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid,
+					 struct iommu_domain *domain)
+{
+	struct device_domain_info *info = dev_iommu_priv_get(dev);
+	struct intel_iommu *iommu = info->iommu;
+
  	intel_pasid_tear_down_entry(iommu, dev, pasid,
  				    INTEL_PASID_TEARDOWN_DRAIN_PRQ);
+	if (domain->type == IOMMU_DOMAIN_IDENTITY)
+		return;

The static identity domain is not capable of handling page requests. Therefore there is no need to drain PRQ for an identity domain removal.

So it probably should be something like this:

	if (domain->type == IOMMU_DOMAIN_IDENTITY) {
		intel_pasid_tear_down_entry(iommu, dev, pasid, 0);
		return;
	}

	intel_pasid_tear_down_entry(iommu, dev, pasid,
                                    INTEL_PASID_TEARDOWN_DRAIN_PRQ);

drivers/iommu/intel/iommu.c static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid, · line 4293

Quoted 9 lines
drivers/iommu/intel/iommu.c +4 -3
+	domain_remove_dev_pasid(domain, dev, pasid);
  }
 
-static int intel_iommu_set_dev_pasid(struct iommu_domain *domain,
-				     struct device *dev, ioasid_t pasid,
-				     struct iommu_domain *old)
+static struct dev_pasid_info *
+domain_prepare_dev_pasid(struct iommu_domain *domain,
+			 struct device *dev, ioasid_t pasid)

Why do you want to return a struct pointer instead of an integer? The returned pointer is not used after it is returned.

Also, how about renaming this helper to domain_add_dev_pasid() to pair it with domain_remove_dev_pasid()?

Quoted 18 lines
drivers/iommu/intel/iommu.c +0 -9
  {
  	struct device_domain_info *info = dev_iommu_priv_get(dev);
  	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
@@ -4293,22 +4299,13 @@ static int intel_iommu_set_dev_pasid(struct iommu_domain *domain,
  	unsigned long flags;
  	int ret;
 
-	if (!pasid_supported(iommu) || dev_is_real_dma_subdevice(dev))
-		return -EOPNOTSUPP;
-
-	if (domain->dirty_ops)
-		return -EINVAL;
-
-	if (context_copied(iommu, info->bus, info->devfn))
-		return -EBUSY;
-
  	ret = prepare_domain_attach_device(domain, dev);
  	if (ret)

drivers/iommu/intel/iommu.c static int intel_iommu_set_dev_pasid(struct iommu_domain *domain, · line 4311

Quoted 10 lines
drivers/iommu/intel/iommu.c +2 -2
-		return ret;
+		return ERR_PTR(ret);
 
  	dev_pasid = kzalloc(sizeof(*dev_pasid), GFP_KERNEL);
  	if (!dev_pasid)
-		return -ENOMEM;
+		return ERR_PTR(-ENOMEM);
 
  	ret = domain_attach_iommu(dmar_domain, iommu);
  	if (ret)

Thanks,
baolu

maintainer
#12 lore raw

Re: [PATCH v2 3/6] iommu/vt-d: Make intel_iommu_set_dev_pasid() to handle domain replacement

Quoted 62 lines · Baolu Lu
Quoted 45 lines · Yi Liu

set_dev_pasid op is going to support domain replacement and keep the old hardware config if it fails. Make the Intel iommu driver be prepared for it.

---
  drivers/iommu/intel/iommu.c | 98 ++++++++++++++++++++++++-------------

  1 file changed, 65 insertions(+), 33 deletions(-)
drivers/iommu/intel/iommu.c +2 -2
index 80b587de226d..6f5a8e549f3f 100644
@@ -4248,8 +4248,8 @@ static int intel_iommu_iotlb_sync_map(struct 
 iommu_domain *domain,
       return 0;
   }
-static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t 
 pasid,
-                     struct iommu_domain *domain)
+static void domain_remove_dev_pasid(struct iommu_domain *domain,
+                    struct device *dev, ioasid_t pasid)
   {
       struct device_domain_info *info = dev_iommu_priv_get(dev);

      struct dev_pasid_info *curr, *dev_pasid = NULL;

drivers/iommu/intel/iommu.c +3 -5
@@ -4257,11 +4257,6 @@ static void intel_iommu_remove_dev_pasid(struct 
 device *dev, ioasid_t pasid,
       struct dmar_domain *dmar_domain;
       unsigned long flags;
-    if (domain->type == IOMMU_DOMAIN_IDENTITY) {
-        intel_pasid_tear_down_entry(iommu, dev, pasid, 0);
-        return;
-    }
-
       dmar_domain = to_dmar_domain(domain);
       spin_lock_irqsave(&dmar_domain->lock, flags);
       list_for_each_entry(curr, &dmar_domain->dev_pasids, link_domain) {
@@ -4278,13 +4273,24 @@ static void 
 intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid,
       domain_detach_iommu(dmar_domain, iommu);
       intel_iommu_debugfs_remove_dev_pasid(dev_pasid);
       kfree(dev_pasid);
+}
+
+static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t 

drivers/iommu/intel/iommu.c static void · line 4289

Quoted 10 lines
drivers/iommu/intel/iommu.c +7 -0
 pasid,
+                     struct iommu_domain *domain)
+{
+    struct device_domain_info *info = dev_iommu_priv_get(dev);
+    struct intel_iommu *iommu = info->iommu;
+
       intel_pasid_tear_down_entry(iommu, dev, pasid,
                       INTEL_PASID_TEARDOWN_DRAIN_PRQ);
+    if (domain->type == IOMMU_DOMAIN_IDENTITY)
+        return;

The static identity domain is not capable of handling page requests. Therefore there is no need to drain PRQ for an identity domain removal.

So it probably should be something like this:

Quoted 7 lines

    if (domain->type == IOMMU_DOMAIN_IDENTITY) { intel_pasid_tear_down_entry(iommu, dev, pasid, 0); return;
    }

    intel_pasid_tear_down_entry(iommu, dev, pasid, INTEL_PASID_TEARDOWN_DRAIN_PRQ);

Just revisited this. It seems that we just need to drain PRQ if the attached domain is iopf-capable. Therefore, how about making it like this?

	unsigned int flags = 0;

	if (domain->iopf_handler)
		flags |= INTEL_PASID_TEARDOWN_DRAIN_PRQ;

	intel_pasid_tear_down_entry(iommu, dev, pasid, flags);

	/* Identity domain has no meta data for pasid. */
	if (domain->type == IOMMU_DOMAIN_IDENTITY)
		return;

Thanks,
baolu

author
#13 lore raw

Re: [PATCH v2 3/6] iommu/vt-d: Make intel_iommu_set_dev_pasid() to handle domain replacement

Quoted 77 lines · Baolu Lu
Quoted 62 lines · Baolu Lu
Quoted 45 lines · Yi Liu

set_dev_pasid op is going to support domain replacement and keep the old hardware config if it fails. Make the Intel iommu driver be prepared for it.

---
  drivers/iommu/intel/iommu.c | 98 ++++++++++++++++++++++++-------------

  1 file changed, 65 insertions(+), 33 deletions(-)
drivers/iommu/intel/iommu.c +2 -2
index 80b587de226d..6f5a8e549f3f 100644
@@ -4248,8 +4248,8 @@ static int intel_iommu_iotlb_sync_map(struct 
 iommu_domain *domain,
       return 0;
   }
-static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t 
 pasid,
-                     struct iommu_domain *domain)
+static void domain_remove_dev_pasid(struct iommu_domain *domain,
+                    struct device *dev, ioasid_t pasid)
   {
       struct device_domain_info *info = dev_iommu_priv_get(dev);

      struct dev_pasid_info *curr, *dev_pasid = NULL;

drivers/iommu/intel/iommu.c +3 -5
@@ -4257,11 +4257,6 @@ static void intel_iommu_remove_dev_pasid(struct 
 device *dev, ioasid_t pasid,
       struct dmar_domain *dmar_domain;
       unsigned long flags;
-    if (domain->type == IOMMU_DOMAIN_IDENTITY) {
-        intel_pasid_tear_down_entry(iommu, dev, pasid, 0);
-        return;
-    }
-
       dmar_domain = to_dmar_domain(domain);
       spin_lock_irqsave(&dmar_domain->lock, flags);
       list_for_each_entry(curr, &dmar_domain->dev_pasids, link_domain) {
@@ -4278,13 +4273,24 @@ static void intel_iommu_remove_dev_pasid(struct 
 device *dev, ioasid_t pasid,
       domain_detach_iommu(dmar_domain, iommu);
       intel_iommu_debugfs_remove_dev_pasid(dev_pasid);
       kfree(dev_pasid);
+}
+
+static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t 

drivers/iommu/intel/iommu.c static void intel_iommu_remove_dev_pasid(struct · line 4289

Quoted 10 lines
drivers/iommu/intel/iommu.c +7 -0
 pasid,
+                     struct iommu_domain *domain)
+{
+    struct device_domain_info *info = dev_iommu_priv_get(dev);
+    struct intel_iommu *iommu = info->iommu;
+
       intel_pasid_tear_down_entry(iommu, dev, pasid,
                       INTEL_PASID_TEARDOWN_DRAIN_PRQ);
+    if (domain->type == IOMMU_DOMAIN_IDENTITY)
+        return;

The static identity domain is not capable of handling page requests. Therefore there is no need to drain PRQ for an identity domain removal.

So it probably should be something like this:

Quoted 7 lines

     if (domain->type == IOMMU_DOMAIN_IDENTITY) { intel_pasid_tear_down_entry(iommu, dev, pasid, 0); return;
     }

     intel_pasid_tear_down_entry(iommu, dev, pasid, INTEL_PASID_TEARDOWN_DRAIN_PRQ);

Just revisited this. It seems that we just need to drain PRQ if the attached domain is iopf-capable. Therefore, how about making it like this?

    unsigned int flags = 0;

Quoted 9 lines

    if (domain->iopf_handler)
        flags |= INTEL_PASID_TEARDOWN_DRAIN_PRQ;

    intel_pasid_tear_down_entry(iommu, dev, pasid, flags);

    /* Identity domain has no meta data for pasid. */ if (domain->type == IOMMU_DOMAIN_IDENTITY) return;

got it.

Signature · 2 lines
Regards,
Yi Liu
#14 lore raw

RE: [PATCH v2 3/6] iommu/vt-d: Make intel_iommu_set_dev_pasid() to handle domain replacement

In reply to Baolu Lu
Quoted 34 lines

From: Baolu Lu <[email protected]>
Sent: Friday, September 13, 2024 10:17 AM

Quoted 18 lines · Baolu Lu
Quoted 11 lines · Yi Liu

+static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid,
+                     struct iommu_domain *domain)

drivers/iommu/intel/iommu.c +4 -0
+{
+    struct device_domain_info *info = dev_iommu_priv_get(dev);
+    struct intel_iommu *iommu = info->iommu;
+

      intel_pasid_tear_down_entry(iommu, dev, pasid, INTEL_PASID_TEARDOWN_DRAIN_PRQ); +    if (domain->type == IOMMU_DOMAIN_IDENTITY) +        return;

The static identity domain is not capable of handling page requests. Therefore there is no need to drain PRQ for an identity domain removal.

So it probably should be something like this:

Quoted 7 lines

    if (domain->type == IOMMU_DOMAIN_IDENTITY) { intel_pasid_tear_down_entry(iommu, dev, pasid, 0); return;
    }

    intel_pasid_tear_down_entry(iommu, dev, pasid, INTEL_PASID_TEARDOWN_DRAIN_PRQ);

Just revisited this. It seems that we just need to drain PRQ if the attached domain is iopf-capable. Therefore, how about making it like this?

Quoted 11 lines
	unsigned int flags = 0;

	if (domain->iopf_handler)
		flags |= INTEL_PASID_TEARDOWN_DRAIN_PRQ;

	intel_pasid_tear_down_entry(iommu, dev, pasid, flags);

	/* Identity domain has no meta data for pasid. */
	if (domain->type == IOMMU_DOMAIN_IDENTITY)
		return;

this is the right thing to do, but also suggesting a bug in existing code. intel_pasid_tear_down_entry() is not just for PRQ drain. It's also about iotlb/devtlb invalidation. From device p.o.v it has no idea about the translation mode in the IOMMU side and always caches the valid mappings in devtlb when ATS is enabled.

Existing code skips all those housekeeping for identify domain by early return before intel_pasid_tear_down_entry(). We need a separate fix for it before this series?

maintainer
#15 lore raw

Re: [PATCH v2 3/6] iommu/vt-d: Make intel_iommu_set_dev_pasid() to handle domain replacement

Quoted 46 lines · Kevin
Quoted 34 lines

From: Baolu Lu <[email protected]>
Sent: Friday, September 13, 2024 10:17 AM

Quoted 18 lines · Baolu Lu

drivers/iommu/intel/iommu.c static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid, · line 4287

Quoted 11 lines · Yi Liu

+static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid,
+                     struct iommu_domain *domain)

drivers/iommu/intel/iommu.c +6 -0
+{
+    struct device_domain_info *info = dev_iommu_priv_get(dev);
+    struct intel_iommu *iommu = info->iommu;
+
       intel_pasid_tear_down_entry(iommu, dev, pasid,
                       INTEL_PASID_TEARDOWN_DRAIN_PRQ);
+    if (domain->type == IOMMU_DOMAIN_IDENTITY)
+        return;

The static identity domain is not capable of handling page requests. Therefore there is no need to drain PRQ for an identity domain removal.

So it probably should be something like this:

Quoted 7 lines
      if (domain->type == IOMMU_DOMAIN_IDENTITY) {
          intel_pasid_tear_down_entry(iommu, dev, pasid, 0);
          return;
      }

    intel_pasid_tear_down_entry(iommu, dev, pasid, INTEL_PASID_TEARDOWN_DRAIN_PRQ);

Just revisited this. It seems that we just need to drain PRQ if the attached domain is iopf-capable. Therefore, how about making it like this?

Quoted 11 lines
	unsigned int flags = 0;

	if (domain->iopf_handler)
		flags |= INTEL_PASID_TEARDOWN_DRAIN_PRQ;

	intel_pasid_tear_down_entry(iommu, dev, pasid, flags);

	/* Identity domain has no meta data for pasid. */
	if (domain->type == IOMMU_DOMAIN_IDENTITY)
		return;
Quoted 5 lines

this is the right thing to do, but also suggesting a bug in existing code. intel_pasid_tear_down_entry() is not just for PRQ drain. It's also about iotlb/devtlb invalidation. From device p.o.v it has no idea about the translation mode in the IOMMU side and always caches the valid mappings in devtlb when ATS is enabled.

Yes. You are right.

intel_pasid_tear_down_entry() takes care of iotlb/devtlb invalidation. So it's fine as long as intel_pasid_tear_down_entry() is called for the IDENTITY domain path, right?

Quoted 3 lines

Existing code skips all those housekeeping for identify domain by early return before intel_pasid_tear_down_entry(). We need a separate fix for it before this series?

Existing code doesn't skip intel_pasid_tear_down_entry().

        if (domain->type == IOMMU_DOMAIN_IDENTITY) {
                intel_pasid_tear_down_entry(iommu, dev, pasid, false);
                return;
        }

Or anything I overlooked?

Thanks,
baolu

#16 lore raw

RE: [PATCH v2 3/6] iommu/vt-d: Make intel_iommu_set_dev_pasid() to handle domain replacement

Quoted 67 lines

From: Baolu Lu <[email protected]>
Sent: Wednesday, October 9, 2024 9:09 AM

Quoted 6 lines · Kevin
Quoted 6 lines

From: Baolu Lu <[email protected]>
Sent: Friday, September 13, 2024 10:17 AM

Quoted 2 lines · Baolu Lu
Quoted 1 line · Yi Liu

+static void intel_iommu_remove_dev_pasid(struct device *dev,

ioasid_t

Quoted 40 lines
Quoted 28 lines
Quoted 16 lines

drivers/iommu/intel/iommu.c static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid, · line 4287

Quoted 10 lines

pasid,
+                     struct iommu_domain *domain)

drivers/iommu/intel/iommu.c +6 -0
+{
+    struct device_domain_info *info = dev_iommu_priv_get(dev);
+    struct intel_iommu *iommu = info->iommu;
+
       intel_pasid_tear_down_entry(iommu, dev, pasid,
                       INTEL_PASID_TEARDOWN_DRAIN_PRQ);
+    if (domain->type == IOMMU_DOMAIN_IDENTITY)
+        return;

The static identity domain is not capable of handling page requests. Therefore there is no need to drain PRQ for an identity domain removal.

So it probably should be something like this:

Quoted 7 lines
      if (domain->type == IOMMU_DOMAIN_IDENTITY) {
          intel_pasid_tear_down_entry(iommu, dev, pasid, 0);
          return;
      }

    intel_pasid_tear_down_entry(iommu, dev, pasid, INTEL_PASID_TEARDOWN_DRAIN_PRQ);

Just revisited this. It seems that we just need to drain PRQ if the attached domain is iopf-capable. Therefore, how about making it like this?

Quoted 11 lines
	unsigned int flags = 0;

	if (domain->iopf_handler)
		flags |= INTEL_PASID_TEARDOWN_DRAIN_PRQ;

	intel_pasid_tear_down_entry(iommu, dev, pasid, flags);

	/* Identity domain has no meta data for pasid. */
	if (domain->type == IOMMU_DOMAIN_IDENTITY)
		return;
Quoted 5 lines

this is the right thing to do, but also suggesting a bug in existing code. intel_pasid_tear_down_entry() is not just for PRQ drain. It's also about iotlb/devtlb invalidation. From device p.o.v it has no idea about the translation mode in the IOMMU side and always caches the valid mappings in devtlb when ATS is enabled.

Yes. You are right.

intel_pasid_tear_down_entry() takes care of iotlb/devtlb invalidation. So it's fine as long as intel_pasid_tear_down_entry() is called for the IDENTITY domain path, right?

Quoted 3 lines

Existing code skips all those housekeeping for identify domain by early return before intel_pasid_tear_down_entry(). We need a separate fix for it before this series?

Quoted 8 lines

Existing code doesn't skip intel_pasid_tear_down_entry().

         if (domain->type == IOMMU_DOMAIN_IDENTITY) {
                 intel_pasid_tear_down_entry(iommu, dev, pasid, false);
                 return;
         }

Or anything I overlooked?

No. Looks I was confused by this patch and misunderstood the existing code.

author
#17 lore raw

Re: [PATCH v2 3/6] iommu/vt-d: Make intel_iommu_set_dev_pasid() to handle domain replacement

In reply to Baolu Lu
Quoted 51 lines · Baolu Lu
Quoted 50 lines · Yi Liu

set_dev_pasid op is going to support domain replacement and keep the old hardware config if it fails. Make the Intel iommu driver be prepared for it.

---
  drivers/iommu/intel/iommu.c | 98 ++++++++++++++++++++++++-------------

  1 file changed, 65 insertions(+), 33 deletions(-)
drivers/iommu/intel/iommu.c +2 -2
index 80b587de226d..6f5a8e549f3f 100644
@@ -4248,8 +4248,8 @@ static int intel_iommu_iotlb_sync_map(struct 
 iommu_domain *domain,
       return 0;
   }
-static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t 
 pasid,
-                     struct iommu_domain *domain)
+static void domain_remove_dev_pasid(struct iommu_domain *domain,
+                    struct device *dev, ioasid_t pasid)
   {
       struct device_domain_info *info = dev_iommu_priv_get(dev);

      struct dev_pasid_info *curr, *dev_pasid = NULL;

drivers/iommu/intel/iommu.c +7 -5
@@ -4257,11 +4257,6 @@ static void intel_iommu_remove_dev_pasid(struct 
 device *dev, ioasid_t pasid,
       struct dmar_domain *dmar_domain;
       unsigned long flags;
-    if (domain->type == IOMMU_DOMAIN_IDENTITY) {
-        intel_pasid_tear_down_entry(iommu, dev, pasid, 0);
-        return;
-    }
-
       dmar_domain = to_dmar_domain(domain);
       spin_lock_irqsave(&dmar_domain->lock, flags);
       list_for_each_entry(curr, &dmar_domain->dev_pasids, link_domain) {
@@ -4278,13 +4273,24 @@ static void intel_iommu_remove_dev_pasid(struct 
 device *dev, ioasid_t pasid,
       domain_detach_iommu(dmar_domain, iommu);
       intel_iommu_debugfs_remove_dev_pasid(dev_pasid);
       kfree(dev_pasid);
+}
+
+static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t 
 pasid,
+                     struct iommu_domain *domain)
+{
+    struct device_domain_info *info = dev_iommu_priv_get(dev);
+    struct intel_iommu *iommu = info->iommu;
Quoted 8 lines

drivers/iommu/intel/iommu.c static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid, · line 4287

Quoted 5 lines
drivers/iommu/intel/iommu.c +3 -0
+
       intel_pasid_tear_down_entry(iommu, dev, pasid,
                       INTEL_PASID_TEARDOWN_DRAIN_PRQ);
+    if (domain->type == IOMMU_DOMAIN_IDENTITY)
+        return;

The static identity domain is not capable of handling page requests. Therefore there is no need to drain PRQ for an identity domain removal.

oh, yes. so maybe for SI domain, there is no PRQ at all.

Quoted 10 lines

So it probably should be something like this:

    if (domain->type == IOMMU_DOMAIN_IDENTITY) { intel_pasid_tear_down_entry(iommu, dev, pasid, 0); return;
    }

    intel_pasid_tear_down_entry(iommu, dev, pasid, INTEL_PASID_TEARDOWN_DRAIN_PRQ);

Quoted 11 lines

drivers/iommu/intel/iommu.c static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid, · line 4293

Quoted 8 lines
drivers/iommu/intel/iommu.c +4 -3
+    domain_remove_dev_pasid(domain, dev, pasid);
   }
-static int intel_iommu_set_dev_pasid(struct iommu_domain *domain,
-                     struct device *dev, ioasid_t pasid,
-                     struct iommu_domain *old)
+static struct dev_pasid_info *
+domain_prepare_dev_pasid(struct iommu_domain *domain,
+             struct device *dev, ioasid_t pasid)

Why do you want to return a struct pointer instead of an integer? The returned pointer is not used after it is returned.

it's for the intel_iommu_debugfs_create_dev_pasid(). it needs the dev_pasid.

Quoted 2 lines

Also, how about renaming this helper to domain_add_dev_pasid() to pair it with domain_remove_dev_pasid()?

sure.

Quoted 29 lines
Quoted 16 lines
drivers/iommu/intel/iommu.c +0 -0
   {
       struct device_domain_info *info = dev_iommu_priv_get(dev);

      struct dmar_domain *dmar_domain = to_dmar_domain(domain);

drivers/iommu/intel/iommu.c +0 -9
@@ -4293,22 +4299,13 @@ static int intel_iommu_set_dev_pasid(struct 
 iommu_domain *domain,
       unsigned long flags;
       int ret;
-    if (!pasid_supported(iommu) || dev_is_real_dma_subdevice(dev))
-        return -EOPNOTSUPP;
-
-    if (domain->dirty_ops)
-        return -EINVAL;
-
-    if (context_copied(iommu, info->bus, info->devfn))
-        return -EBUSY;
-

drivers/iommu/intel/iommu.c static int intel_iommu_set_dev_pasid(struct · line 4309

Quoted 10 lines
drivers/iommu/intel/iommu.c +2 -2
       ret = prepare_domain_attach_device(domain, dev);
       if (ret)
-        return ret;
+        return ERR_PTR(ret);
       dev_pasid = kzalloc(sizeof(*dev_pasid), GFP_KERNEL);
       if (!dev_pasid)
-        return -ENOMEM;
+        return ERR_PTR(-ENOMEM);
       ret = domain_attach_iommu(dmar_domain, iommu);
       if (ret)

Thanks,
baolu

Signature · 2 lines
Regards,
Yi Liu
maintainer
#18 lore raw

Re: [PATCH v2 3/6] iommu/vt-d: Make intel_iommu_set_dev_pasid() to handle domain replacement

In reply to Yi Liu
Quoted 20 lines · Yi Liu
drivers/iommu/intel/iommu.c +3 -7
@@ -4325,24 +4363,18 @@ static int intel_iommu_set_dev_pasid(struct iommu_domain *domain,
  		ret = intel_pasid_setup_second_level(iommu, dmar_domain,
  						     dev, pasid);
  	if (ret)
-		goto out_unassign_tag;
+		goto out_undo_dev_pasid;
 
-	dev_pasid->dev = dev;
-	dev_pasid->pasid = pasid;
-	spin_lock_irqsave(&dmar_domain->lock, flags);
-	list_add(&dev_pasid->link_domain, &dmar_domain->dev_pasids);
-	spin_unlock_irqrestore(&dmar_domain->lock, flags);
+	if (old)
+		domain_remove_dev_pasid(old, dev, pasid);
 
  	if (domain->type & __IOMMU_DOMAIN_PAGING)
  		intel_iommu_debugfs_create_dev_pasid(dev_pasid);
 
  	return 0;
-out_unassign_tag:

drivers/iommu/intel/iommu.c static int intel_iommu_set_dev_pasid(struct iommu_domain *domain, · line 4379

Quoted 10 lines
drivers/iommu/intel/iommu.c +3 -5
-	cache_tag_unassign_domain(dmar_domain, dev, pasid);
-out_detach_iommu:
-	domain_detach_iommu(dmar_domain, iommu);
-out_free:
-	kfree(dev_pasid);
+
+out_undo_dev_pasid:
+	domain_remove_dev_pasid(domain, dev, pasid);
  	return ret;
  }

Do you need to re-install the old domain to the pasid entry in the failure path?

Thanks,
baolu

author
#19 lore raw

Re: [PATCH v2 3/6] iommu/vt-d: Make intel_iommu_set_dev_pasid() to handle domain replacement

Quoted 24 lines · Baolu Lu
Quoted 23 lines · Yi Liu
drivers/iommu/intel/iommu.c +3 -12
@@ -4325,24 +4363,18 @@ static int intel_iommu_set_dev_pasid(struct 
 iommu_domain *domain,
           ret = intel_pasid_setup_second_level(iommu, dmar_domain,
                                dev, pasid);
       if (ret)
-        goto out_unassign_tag;
+        goto out_undo_dev_pasid;
-    dev_pasid->dev = dev;
-    dev_pasid->pasid = pasid;
-    spin_lock_irqsave(&dmar_domain->lock, flags);
-    list_add(&dev_pasid->link_domain, &dmar_domain->dev_pasids);
-    spin_unlock_irqrestore(&dmar_domain->lock, flags);
+    if (old)
+        domain_remove_dev_pasid(old, dev, pasid);
       if (domain->type & __IOMMU_DOMAIN_PAGING)
           intel_iommu_debugfs_create_dev_pasid(dev_pasid);
       return 0;
-out_unassign_tag:
-    cache_tag_unassign_domain(dmar_domain, dev, pasid);
-out_detach_iommu:
-    domain_detach_iommu(dmar_domain, iommu);
-out_free:
-    kfree(dev_pasid);
Quoted 8 lines

drivers/iommu/intel/iommu.c static int intel_iommu_set_dev_pasid(struct iommu_domain *domain, · line 4379

Quoted 5 lines
drivers/iommu/intel/iommu.c +3 -0
+
+out_undo_dev_pasid:
+    domain_remove_dev_pasid(domain, dev, pasid);
       return ret;
   }

Do you need to re-install the old domain to the pasid entry in the failure path?

yes, but no. The old domain is still installed in the pasid entry when the failure happened. :)

Signature · 2 lines
Regards,
Yi Liu
maintainer
#20 lore raw

Re: [PATCH v2 3/6] iommu/vt-d: Make intel_iommu_set_dev_pasid() to handle domain replacement

Quoted 34 lines · Yi Liu
Quoted 24 lines · Baolu Lu
Quoted 23 lines · Yi Liu
drivers/iommu/intel/iommu.c +3 -12
@@ -4325,24 +4363,18 @@ static int intel_iommu_set_dev_pasid(struct 
 iommu_domain *domain,
           ret = intel_pasid_setup_second_level(iommu, dmar_domain,
                                dev, pasid);
       if (ret)
-        goto out_unassign_tag;
+        goto out_undo_dev_pasid;
-    dev_pasid->dev = dev;
-    dev_pasid->pasid = pasid;
-    spin_lock_irqsave(&dmar_domain->lock, flags);
-    list_add(&dev_pasid->link_domain, &dmar_domain->dev_pasids);
-    spin_unlock_irqrestore(&dmar_domain->lock, flags);
+    if (old)
+        domain_remove_dev_pasid(old, dev, pasid);
       if (domain->type & __IOMMU_DOMAIN_PAGING)
           intel_iommu_debugfs_create_dev_pasid(dev_pasid);
       return 0;
-out_unassign_tag:
-    cache_tag_unassign_domain(dmar_domain, dev, pasid);
-out_detach_iommu:
-    domain_detach_iommu(dmar_domain, iommu);
-out_free:
-    kfree(dev_pasid);
Quoted 8 lines

drivers/iommu/intel/iommu.c static int intel_iommu_set_dev_pasid(struct iommu_domain *domain, · line 4379

Quoted 5 lines
drivers/iommu/intel/iommu.c +3 -0
+
+out_undo_dev_pasid:
+    domain_remove_dev_pasid(domain, dev, pasid);
       return ret;
   }

Do you need to re-install the old domain to the pasid entry in the failure path?

Quoted 2 lines

yes, but no. The old domain is still installed in the pasid entry when the failure happened. :)

I am afraid not. The old domain has already been cleaned up by intel_pasid_tear_down_entry(). Or not?

Thanks,
baolu

author
#21 lore raw

Re: [PATCH v2 3/6] iommu/vt-d: Make intel_iommu_set_dev_pasid() to handle domain replacement

Quoted 36 lines · Baolu Lu
Quoted 32 lines · Yi Liu
Quoted 32 lines · Baolu Lu
Quoted 17 lines · Yi Liu
drivers/iommu/intel/iommu.c +3 -7
@@ -4325,24 +4363,18 @@ static int intel_iommu_set_dev_pasid(struct iommu_domain *domain,
          ret = intel_pasid_setup_second_level(iommu, dmar_domain,
                               dev, pasid);
      if (ret)
-        goto out_unassign_tag;
+        goto out_undo_dev_pasid;
-    dev_pasid->dev = dev;
-    dev_pasid->pasid = pasid;
-    spin_lock_irqsave(&dmar_domain->lock, flags);
-    list_add(&dev_pasid->link_domain, &dmar_domain->dev_pasids);
-    spin_unlock_irqrestore(&dmar_domain->lock, flags);
+    if (old)
+        domain_remove_dev_pasid(old, dev, pasid);
      if (domain->type & __IOMMU_DOMAIN_PAGING)
          intel_iommu_debugfs_create_dev_pasid(dev_pasid);
      return 0;
-out_unassign_tag:

drivers/iommu/intel/iommu.c static int intel_iommu_set_dev_pasid(struct iommu_domain *domain, · line 4376

Quoted 10 lines
drivers/iommu/intel/iommu.c +3 -5
-    cache_tag_unassign_domain(dmar_domain, dev, pasid);
-out_detach_iommu:
-    domain_detach_iommu(dmar_domain, iommu);
-out_free:
-    kfree(dev_pasid);
+
+out_undo_dev_pasid:
+    domain_remove_dev_pasid(domain, dev, pasid);
      return ret;
  }

Do you need to re-install the old domain to the pasid entry in the failure path?

Quoted 5 lines
Quoted 2 lines

yes, but no. The old domain is still installed in the pasid entry when the failure happened. :)

I am afraid not. The old domain has already been cleaned up by intel_pasid_tear_down_entry(). Or not?

oops, yes. The tear down was lifted to this function now instead of in the setup helpers. I may move them back to the setup helpers just like v1 does.

Good catch!

Regards,
Yi Liu

author
#22 lore raw

PATCH v2 4/6 iommu/vt-d: Add set_dev_pasid callback for nested domain

In reply to Yi Liu

From: Lu Baolu <[email protected]>

Extend intel_iommu_set_dev_pasid() to set a nested type domain to a PASID of a device.

---
 drivers/iommu/intel/iommu.c  | 22 +++++++++++++++++-----
 drivers/iommu/intel/iommu.h  |  3 +++
 drivers/iommu/intel/nested.c |  1 +
 3 files changed, 21 insertions(+), 5 deletions(-)
drivers/iommu/intel/iommu.c +17 -5
index 6f5a8e549f3f..749ee7741ec4 100644
@@ -284,6 +284,11 @@ static int __init intel_iommu_setup(char *str)
 }
 __setup("intel_iommu=", intel_iommu_setup);
 
+static int domain_type_is_nested(struct dmar_domain *domain)
+{
+	return domain->domain.type == IOMMU_DOMAIN_NESTED;
+}
+
 static int domain_pfn_supported(struct dmar_domain *domain, unsigned long pfn)
 {
 	int addr_width = agaw_to_width(domain->agaw) - VTD_PAGE_SHIFT;
@@ -4299,7 +4304,12 @@ domain_prepare_dev_pasid(struct iommu_domain *domain,
 	unsigned long flags;
 	int ret;
 
-	ret = prepare_domain_attach_device(domain, dev);
+	/* Nested type domain should prepare its parent domain */
+	if (domain_type_is_nested(dmar_domain))
+		ret = prepare_domain_attach_device(
+				&dmar_domain->s2_domain->domain, dev);
+	else
+		ret = prepare_domain_attach_device(domain, dev);
 	if (ret)
 		return ERR_PTR(ret);
 
@@ -4329,9 +4339,9 @@ domain_prepare_dev_pasid(struct iommu_domain *domain,
 	return ERR_PTR(ret);
 }
 
-static int intel_iommu_set_dev_pasid(struct iommu_domain *domain,
-				     struct device *dev, ioasid_t pasid,
-				     struct iommu_domain *old)
+int intel_iommu_set_dev_pasid(struct iommu_domain *domain,
+			      struct device *dev, ioasid_t pasid,
+			      struct iommu_domain *old)
 {
 	struct device_domain_info *info = dev_iommu_priv_get(dev);
 	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
@@ -4356,7 +4366,9 @@ static int intel_iommu_set_dev_pasid(struct iommu_domain *domain,
 	intel_pasid_tear_down_entry(iommu, dev, pasid,
 				    INTEL_PASID_TEARDOWN_DRAIN_PRQ);
 
-	if (dmar_domain->use_first_level)
+	if (domain_type_is_nested(dmar_domain))
+		ret = intel_pasid_setup_nested(iommu, dev, pasid, dmar_domain);
+	else if (dmar_domain->use_first_level)
 		ret = domain_setup_first_level(iommu, dmar_domain,
 					       dev, pasid);
 	else
drivers/iommu/intel/iommu.h +3 -0
index b3b295e60626..36cf59bfe4ea 100644
@@ -1109,6 +1109,9 @@ void device_block_translation(struct device *dev);
 int prepare_domain_attach_device(struct iommu_domain *domain,
 				 struct device *dev);
 void domain_update_iommu_cap(struct dmar_domain *domain);
+int intel_iommu_set_dev_pasid(struct iommu_domain *domain,
+			      struct device *dev, ioasid_t pasid,
+			      struct iommu_domain *old);
 
 int dmar_ir_support(void);
 
drivers/iommu/intel/nested.c +1 -0
index 36a91b1b52be..c5a94c00b396 100644
@@ -131,6 +131,7 @@ static int intel_nested_cache_invalidate_user(struct iommu_domain *domain,
 
 static const struct iommu_domain_ops intel_nested_domain_ops = {
 	.attach_dev		= intel_nested_attach_dev,
+	.set_dev_pasid		= intel_iommu_set_dev_pasid,
 	.free			= intel_nested_domain_free,
 	.cache_invalidate_user	= intel_nested_cache_invalidate_user,
 };
Signature · 1 line
2.34.1
maintainer
#23 lore raw

Re: [PATCH v2 4/6] iommu/vt-d: Add set_dev_pasid callback for nested domain

Quoted 4 lines · Yi Liu
drivers/iommu/intel/iommu.c +0 -0
@@ -4299,7 +4304,12 @@ domain_prepare_dev_pasid(struct iommu_domain *domain,
  	unsigned long flags;
  	int ret;
 

drivers/iommu/intel/iommu.c domain_prepare_dev_pasid(struct iommu_domain *domain, · line 4315

Quoted 10 lines
drivers/iommu/intel/iommu.c +6 -1
-	ret = prepare_domain_attach_device(domain, dev);
+	/* Nested type domain should prepare its parent domain */
+	if (domain_type_is_nested(dmar_domain))
+		ret = prepare_domain_attach_device(
+				&dmar_domain->s2_domain->domain, dev);
+	else
+		ret = prepare_domain_attach_device(domain, dev);
  	if (ret)
  		return ERR_PTR(ret);
 

'prepare' is a bit confusing in this context. It actually means checking whether a domain is compatible with the hardware capabilities of RID or PASID. Or not?

I know this confusion comes from the naming of prepare_domain_attach_device(). Hence, do you mind renaming this helper in a small patch?

Thanks,
baolu

author
#24 lore raw

Re: [PATCH v2 4/6] iommu/vt-d: Add set_dev_pasid callback for nested domain

Quoted 18 lines · Baolu Lu

drivers/iommu/intel/iommu.c domain_prepare_dev_pasid(struct iommu_domain *domain, · line 4313

Quoted 12 lines · Yi Liu
drivers/iommu/intel/iommu.c +6 -1
@@ -4299,7 +4304,12 @@ domain_prepare_dev_pasid(struct iommu_domain *domain,
       unsigned long flags;
       int ret;
-    ret = prepare_domain_attach_device(domain, dev);
+    /* Nested type domain should prepare its parent domain */
+    if (domain_type_is_nested(dmar_domain))
+        ret = prepare_domain_attach_device(
+                &dmar_domain->s2_domain->domain, dev);
+    else
+        ret = prepare_domain_attach_device(domain, dev);
       if (ret)
           return ERR_PTR(ret);

'prepare' is a bit confusing in this context. It actually means checking whether a domain is compatible with the hardware capabilities of RID or PASID. Or not?

Quoted 3 lines

I know this confusion comes from the naming of prepare_domain_attach_device(). Hence, do you mind renaming this helper in a small patch?

good suggestion.

Signature · 2 lines
Regards,
Yi Liu
Reviewed-by on v2 4/6
#25 lore raw

RE: [PATCH v2 4/6] iommu/vt-d: Add set_dev_pasid callback for nested domain

In reply to Yi Liu
Quoted 11 lines

From: Liu, Yi L <[email protected]>
Sent: Thursday, September 12, 2024 9:04 PM

From: Lu Baolu <[email protected]>

Extend intel_iommu_set_dev_pasid() to set a nested type domain to a PASID of a device.

author
#26 lore raw

PATCH v2 5/6 iommu/arm-smmu-v3: Make smmuv3 set_dev_pasid() op support replace

In reply to Yi Liu

From: Jason Gunthorpe <[email protected]>

set_dev_pasid() op is going to be enhanced to support domain replacement of a pasid. This prepares for this op definition.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c     | 8 +++++---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h     | 2 +-
 3 files changed, 7 insertions(+), 5 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +1 -1
index 645da7b69bed..1d3e71569775 100644
@@ -349,7 +349,7 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
 	 * get reassigned
 	 */
 	arm_smmu_make_sva_cd(&target, master, domain->mm, smmu_domain->cd.asid);
-	ret = arm_smmu_set_pasid(master, smmu_domain, id, &target);
+	ret = arm_smmu_set_pasid(master, smmu_domain, id, &target, old);
 
 	mmput(domain->mm);
 	return ret;
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +5 -3
index ed2b106e02dd..f7a73b854151 100644
@@ -2824,7 +2824,8 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
 }
 
 static int arm_smmu_s1_set_dev_pasid(struct iommu_domain *domain,
-				      struct device *dev, ioasid_t id)
+				     struct device *dev, ioasid_t id,
+				     struct iommu_domain *old)
 {
 	struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
 	struct arm_smmu_master *master = dev_iommu_priv_get(dev);
@@ -2850,7 +2851,7 @@ static int arm_smmu_s1_set_dev_pasid(struct iommu_domain *domain,
 	 */
 	arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
 	return arm_smmu_set_pasid(master, to_smmu_domain(domain), id,
-				  &target_cd);
+				  &target_cd, old);
 }
 
 static void arm_smmu_update_ste(struct arm_smmu_master *master,
@@ -2880,7 +2881,7 @@ static void arm_smmu_update_ste(struct arm_smmu_master *master,
 
 int arm_smmu_set_pasid(struct arm_smmu_master *master,
 		       struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
-		       struct arm_smmu_cd *cd)
+		       struct arm_smmu_cd *cd, struct iommu_domain *old)
 {
 	struct iommu_domain *sid_domain = iommu_get_domain_for_dev(master->dev);
 	struct arm_smmu_attach_state state = {
@@ -2890,6 +2891,7 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
 		 * already attached, no need to set old_domain.
 		 */
 		.ssid = pasid,
+		.old_domain = old,
 	};
 	struct arm_smmu_cd *cdptr;
 	int ret;
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +1 -1
index 14bca41a981b..a942f5d40051 100644
@@ -806,7 +806,7 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
 
 int arm_smmu_set_pasid(struct arm_smmu_master *master,
 		       struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
-		       struct arm_smmu_cd *cd);
+		       struct arm_smmu_cd *cd, struct iommu_domain *old);
 
 void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid);
 void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
Signature · 1 line
2.34.1
Reviewed-by on v2 5/6
#27 lore raw

RE: [PATCH v2 5/6] iommu/arm-smmu-v3: Make smmuv3 set_dev_pasid() op support replace

Quoted 10 lines

From: Liu, Yi L <[email protected]>
Sent: Thursday, September 12, 2024 9:04 PM

From: Jason Gunthorpe <[email protected]>

set_dev_pasid() op is going to be enhanced to support domain replacement of a pasid. This prepares for this op definition.

maintainer
#28 lore raw

Re: [PATCH v2 5/6] iommu/arm-smmu-v3: Make smmuv3 set_dev_pasid() op support replace

In reply to Yi Liu
Quoted 59 lines · Yi Liu

From: Jason Gunthorpe <[email protected]>

set_dev_pasid() op is going to be enhanced to support domain replacement of a pasid. This prepares for this op definition.

---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c     | 8 +++++---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h     | 2 +-
 3 files changed, 7 insertions(+), 5 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +1 -1
index 645da7b69bed..1d3e71569775 100644
@@ -349,7 +349,7 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
 	 * get reassigned
 	 */
 	arm_smmu_make_sva_cd(&target, master, domain->mm, smmu_domain->cd.asid);
-	ret = arm_smmu_set_pasid(master, smmu_domain, id, &target);
+	ret = arm_smmu_set_pasid(master, smmu_domain, id, &target, old);
 
 	mmput(domain->mm);
 	return ret;
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +4 -3
index ed2b106e02dd..f7a73b854151 100644
@@ -2824,7 +2824,8 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
 }
 
 static int arm_smmu_s1_set_dev_pasid(struct iommu_domain *domain,
-				      struct device *dev, ioasid_t id)
+				     struct device *dev, ioasid_t id,
+				     struct iommu_domain *old)
 {
 	struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
 	struct arm_smmu_master *master = dev_iommu_priv_get(dev);
@@ -2850,7 +2851,7 @@ static int arm_smmu_s1_set_dev_pasid(struct iommu_domain *domain,
 	 */
 	arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
 	return arm_smmu_set_pasid(master, to_smmu_domain(domain), id,
-				  &target_cd);
+				  &target_cd, old);
 }
 
 static void arm_smmu_update_ste(struct arm_smmu_master *master,
@@ -2880,7 +2881,7 @@ static void arm_smmu_update_ste(struct arm_smmu_master *master,
 
 int arm_smmu_set_pasid(struct arm_smmu_master *master,
 		       struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
-		       struct arm_smmu_cd *cd)
+		       struct arm_smmu_cd *cd, struct iommu_domain *old)
 {
 	struct iommu_domain *sid_domain = iommu_get_domain_for_dev(master->dev);
 	struct arm_smmu_attach_state state = {

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c int arm_smmu_set_pasid(struct arm_smmu_master *master, · line 2894

Quoted 4 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +1 -0
@@ -2890,6 +2891,7 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
 		 * already attached, no need to set old_domain.
 		 */
 		.ssid = pasid,
+		.old_domain = old,

nit: The existing comment says "no need to set old_domain" and now you're doing exactly that! Please can you update the commentary (probably just remove the comment entirely)?

Cheers,

Will

author
#29 lore raw

Re: [PATCH v2 5/6] iommu/arm-smmu-v3: Make smmuv3 set_dev_pasid() op support replace

Quoted 52 lines · Will Deacon
Quoted 51 lines · Yi Liu

From: Jason Gunthorpe <[email protected]>

set_dev_pasid() op is going to be enhanced to support domain replacement of a pasid. This prepares for this op definition.

---
  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +-
  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c     | 8 +++++---
  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h     | 2 +-
  3 files changed, 7 insertions(+), 5 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +1 -1
index 645da7b69bed..1d3e71569775 100644
@@ -349,7 +349,7 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
  	 * get reassigned
  	 */
  	arm_smmu_make_sva_cd(&target, master, domain->mm, smmu_domain->cd.asid);
-	ret = arm_smmu_set_pasid(master, smmu_domain, id, &target);
+	ret = arm_smmu_set_pasid(master, smmu_domain, id, &target, old);
 
  	mmput(domain->mm);
  	return ret;
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +3 -2
index ed2b106e02dd..f7a73b854151 100644
@@ -2824,7 +2824,8 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
  }
 
  static int arm_smmu_s1_set_dev_pasid(struct iommu_domain *domain,
-				      struct device *dev, ioasid_t id)
+				     struct device *dev, ioasid_t id,
+				     struct iommu_domain *old)
  {
  	struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
  	struct arm_smmu_master *master = dev_iommu_priv_get(dev);
@@ -2850,7 +2851,7 @@ static int arm_smmu_s1_set_dev_pasid(struct iommu_domain *domain,
  	 */
  	arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
  	return arm_smmu_set_pasid(master, to_smmu_domain(domain), id,
-				  &target_cd);
+				  &target_cd, old);
  }
 
  static void arm_smmu_update_ste(struct arm_smmu_master *master,
@@ -2880,7 +2881,7 @@ static void arm_smmu_update_ste(struct arm_smmu_master *master,
 
Quoted 16 lines

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c int arm_smmu_set_pasid(struct arm_smmu_master *master, · line 2894

Quoted 12 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +2 -1
  int arm_smmu_set_pasid(struct arm_smmu_master *master,
  		       struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
-		       struct arm_smmu_cd *cd)
+		       struct arm_smmu_cd *cd, struct iommu_domain *old)
  {
  	struct iommu_domain *sid_domain = iommu_get_domain_for_dev(master->dev);
  	struct arm_smmu_attach_state state = {
@@ -2890,6 +2891,7 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
  		 * already attached, no need to set old_domain.
  		 */
  		.ssid = pasid,
+		.old_domain = old,

nit: The existing comment says "no need to set old_domain" and now you're doing exactly that! Please can you update the commentary (probably just remove the comment entirely)?

oops, indeed, should remove this comment entirely. thanks for spotting it.

Signature · 2 lines
Regards,
Yi Liu
#30 lore raw

Re: [PATCH v2 5/6] iommu/arm-smmu-v3: Make smmuv3 set_dev_pasid() op support replace

In reply to Will Deacon
Quoted 9 lines · Will Deacon

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c int arm_smmu_set_pasid(struct arm_smmu_master *master, · line 2894

Quoted 5 lines
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +1 -0
@@ -2890,6 +2891,7 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
 		 * already attached, no need to set old_domain.
 		 */
 		.ssid = pasid,
+		.old_domain = old,

nit: The existing comment says "no need to set old_domain" and now you're doing exactly that! Please can you update the commentary (probably just remove the comment entirely)?

+1

Jason

author
#31 lore raw

PATCH v2 6/6 iommu: Make set_dev_pasid op support domain replacement

In reply to Yi Liu

The iommu core is going to support domain replacement for pasid, it needs to make the set_dev_pasid op support replacing domain and keep the old domain config in the failure case.

AMD iommu driver does not support domain replacement for pasid yet, so it would fail the set_dev_pasid op to keep the old config if the input @old is non-NULL.

---
 drivers/iommu/amd/pasid.c | 3 +++
 include/linux/iommu.h     | 3 ++-
 2 files changed, 5 insertions(+), 1 deletion(-)
drivers/iommu/amd/pasid.c +3 -0
index 77bf5f5f947a..30e27bda3fac 100644
@@ -109,6 +109,9 @@ int iommu_sva_set_dev_pasid(struct iommu_domain *domain,
 	unsigned long flags;
 	int ret = -EINVAL;
 
+	if (old)
+		return -EOPNOTSUPP;
+
 	/* PASID zero is used for requests from the I/O device without PASID */
 	if (!is_pasid_valid(dev_data, pasid))
 		return ret;
include/linux/iommu.h +2 -1
index 32dce80aa7fd..27f923450a7c 100644
@@ -616,7 +616,8 @@ struct iommu_ops {
  * * EBUSY	- device is attached to a domain and cannot be changed
  * * ENODEV	- device specific errors, not able to be attached
  * * <others>	- treated as ENODEV by the caller. Use is discouraged
- * @set_dev_pasid: set an iommu domain to a pasid of device
+ * @set_dev_pasid: set or replace an iommu domain to a pasid of device. The pasid of
+ *                 the device should be left in the old config in error case.
  * @map_pages: map a physically contiguous set of pages of the same size to
  *             an iommu domain.
  * @unmap_pages: unmap a number of pages of the same size from an iommu domain
Signature · 1 line
2.34.1
Reviewed-by on v2 6/6
#32 lore raw

Re: [PATCH v2 6/6] iommu: Make set_dev_pasid op support domain replacement

Quoted 8 lines · Yi Liu

The iommu core is going to support domain replacement for pasid, it needs to make the set_dev_pasid op support replacing domain and keep the old domain config in the failure case.

AMD iommu driver does not support domain replacement for pasid yet, so it would fail the set_dev_pasid op to keep the old config if the input @old is non-NULL.

Quoted 6 lines
---
 drivers/iommu/amd/pasid.c | 3 +++
 include/linux/iommu.h     | 3 ++-
 2 files changed, 5 insertions(+), 1 deletion(-)

Jason

Reviewed-by on v2 6/6
#33 lore raw

RE: [PATCH v2 6/6] iommu: Make set_dev_pasid op support domain replacement

In reply to Yi Liu
Quoted 7 lines

From: Liu, Yi L <[email protected]>
Sent: Thursday, September 12, 2024 9:04 PM

The iommu core is going to support domain replacement for pasid, it needs to make the set_dev_pasid op support replacing domain and keep the old domain config in the failure case.

Quoted 6 lines

AMD iommu driver does not support domain replacement for pasid yet, so it would fail the set_dev_pasid op to keep the old config if the input @old is non-NULL.