Skip to contentSkip to conversation

iommu: Make pasid array per device

PATCH Superseded Lu Baolu 4 participants 17 messages Aug 1, 2023 – Aug 4, 2023

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Lu Baolu

author
#0 lore raw

cover iommu: Make pasid array per device

The PCI PASID enabling interface guarantees that the address space used by each PASID is unique. This is achieved by checking that the PCI ACS path is enabled for the device. If the path is not enabled, then the PASID feature cannot be used.

if (!pci_acs_path_enabled(pdev, NULL, PCI_ACS_RR | PCI_ACS_UF))
return -EINVAL;

The PASID array is not an attribute of the IOMMU group. It is more natural to store the PASID array in the per-device IOMMU data. This makes the code clearer and easier to understand. No functional changes are intended.

Please help review and suggest.

Lu Baolu (2):
iommu: Consolidate pasid dma ownership check
iommu: Move pasid array from group to device

 include/linux/iommu.h |   2 +
 drivers/iommu/iommu.c | 105 +++++++++++++++++-------------------------
 2 files changed, 43 insertions(+), 64 deletions(-)
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH 1/2 iommu: Consolidate pasid dma ownership check

When switching device DMA ownership, it is required that all the device's pasid DMA be disabled. This is done by checking if the pasid array of the group is empty. Consolidate all the open code into a single helper. No intentional functionality change.

---
 drivers/iommu/iommu.c | 27 ++++++++++++++++++++-------
 1 file changed, 20 insertions(+), 7 deletions(-)
drivers/iommu/iommu.c +20 -7
index 4352a149a935..1a8fb30341e6 100644
@@ -3034,6 +3034,17 @@ static bool iommu_is_default_domain(struct iommu_group *group)
 	return false;
 }
 
+/*
+ * Assert no PASID DMA when claiming or releasing group's DMA ownership.
+ * The iommu_xxx_device_pasid() interfaces are only for device drivers
+ * that have claimed the DMA ownership. Otherwise, it's a driver bug.
+ */
+static void assert_pasid_dma_ownership(struct iommu_group *group)
+{
+	lockdep_assert_held(&group->mutex);
+	WARN_ON(!xa_empty(&group->pasid_array));
+}
+
 /**
  * iommu_device_use_default_domain() - Device driver wants to handle device
  *                                     DMA through the kernel DMA API.
@@ -3052,14 +3063,14 @@ int iommu_device_use_default_domain(struct device *dev)
 
 	mutex_lock(&group->mutex);
 	if (group->owner_cnt) {
-		if (group->owner || !iommu_is_default_domain(group) ||
-		    !xa_empty(&group->pasid_array)) {
+		if (group->owner || !iommu_is_default_domain(group)) {
 			ret = -EBUSY;
 			goto unlock_out;
 		}
 	}
 
 	group->owner_cnt++;
+	assert_pasid_dma_ownership(group);
 
 unlock_out:
 	mutex_unlock(&group->mutex);
@@ -3084,7 +3095,8 @@ void iommu_device_unuse_default_domain(struct device *dev)
 		return;
 
 	mutex_lock(&group->mutex);
-	if (!WARN_ON(!group->owner_cnt || !xa_empty(&group->pasid_array)))
+	assert_pasid_dma_ownership(group);
+	if (!WARN_ON(!group->owner_cnt))
 		group->owner_cnt--;
 
 	mutex_unlock(&group->mutex);
@@ -3118,8 +3130,7 @@ static int __iommu_take_dma_ownership(struct iommu_group *group, void *owner)
 {
 	int ret;
 
-	if ((group->domain && group->domain != group->default_domain) ||
-	    !xa_empty(&group->pasid_array))
+	if (group->domain && group->domain != group->default_domain)
 		return -EBUSY;
 
 	ret = __iommu_group_alloc_blocking_domain(group);
@@ -3129,8 +3140,10 @@ static int __iommu_take_dma_ownership(struct iommu_group *group, void *owner)
 	if (ret)
 		return ret;
 
+	assert_pasid_dma_ownership(group);
 	group->owner = owner;
 	group->owner_cnt++;
+
 	return 0;
 }
 
@@ -3206,10 +3219,10 @@ EXPORT_SYMBOL_GPL(iommu_device_claim_dma_owner);
 
 static void __iommu_release_dma_ownership(struct iommu_group *group)
 {
-	if (WARN_ON(!group->owner_cnt || !group->owner ||
-		    !xa_empty(&group->pasid_array)))
+	if (WARN_ON(!group->owner_cnt || !group->owner))
 		return;
 
+	assert_pasid_dma_ownership(group);
 	group->owner_cnt = 0;
 	group->owner = NULL;
 	__iommu_group_set_domain_nofail(group, group->default_domain);
Signature · 1 line
2.34.1
author
#7 lore raw

PATCH 2/2 iommu: Move pasid array from group to device

In reply to Lu Baolu

The PASID (Process Address Space ID) feature is a device feature that allows a device driver to manage the PASID value and attach or detach its domain to the pasid. The pasid array, which is used to store the domain of each pasid, is currently stored in iommu group struct, but it would be more natural to move it to the device so that the device drivers don't need to understand the internal iommu group concept.

---
 include/linux/iommu.h |  2 ++
 drivers/iommu/iommu.c | 80 ++++++++++++-------------------------------
 2 files changed, 24 insertions(+), 58 deletions(-)
include/linux/iommu.h +2 -0
index b1dcb1b9b170..90be3efd0e91 100644
@@ -406,6 +406,7 @@ struct iommu_fault_param {
  * @iopf_param:	 I/O Page Fault queue and data
  * @fwspec:	 IOMMU fwspec data
  * @iommu_dev:	 IOMMU device this device is linked to
+ * @pasid_array: pasid-indexed array of domains attached to pasid
  * @priv:	 IOMMU Driver private data
  * @max_pasids:  number of PASIDs this device can consume
  * @attach_deferred: the dma domain attachment is deferred
@@ -420,6 +421,7 @@ struct dev_iommu {
 	struct iopf_device_param	*iopf_param;
 	struct iommu_fwspec		*fwspec;
 	struct iommu_device		*iommu_dev;
+	struct xarray			pasid_array;
 	void				*priv;
 	u32				max_pasids;
 	u32				attach_deferred:1;
drivers/iommu/iommu.c +22 -58

147 lines not on this page · raw message

Signature · 1 line
2.34.1
#2 lore raw

RE: [PATCH 1/2] iommu: Consolidate pasid dma ownership check

In reply to Lu Baolu
Quoted 7 lines

From: Lu Baolu <[email protected]>
Sent: Tuesday, August 1, 2023 2:31 PM

When switching device DMA ownership, it is required that all the device's pasid DMA be disabled. This is done by checking if the pasid array of the group is empty. Consolidate all the open code into a single helper. No intentional functionality change.

...

Quoted 9 lines

/**
* iommu_device_use_default_domain() - Device driver wants to handle
device
* DMA through the kernel DMA API.

drivers/iommu/iommu.c +0 -0
@@ -3052,14 +3063,14 @@ int iommu_device_use_default_domain(struct
 device *dev)
 
 	mutex_lock(&group->mutex);
 	if (group->owner_cnt) {

drivers/iommu/iommu.c int iommu_device_use_default_domain(struct · line 3074

Quoted 10 lines
drivers/iommu/iommu.c +2 -2
-		if (group->owner || !iommu_is_default_domain(group) ||
-		    !xa_empty(&group->pasid_array)) {
+		if (group->owner || !iommu_is_default_domain(group)) {
 			ret = -EBUSY;
 			goto unlock_out;
 		}
 	}
 
 	group->owner_cnt++;
+	assert_pasid_dma_ownership(group);

Old code returns error if pasid_xrrary is not empty.

New code continues to take ownership with a warning.

this is a functional change. Is it intended or not?

author
#3 lore raw

Re: [PATCH 1/2] iommu: Consolidate pasid dma ownership check

Quoted 17 lines · Kevin
Quoted 17 lines

/**
* iommu_device_use_default_domain() - Device driver wants to handle
device
* DMA through the kernel DMA API.

drivers/iommu/iommu.c +1 -2
@@ -3052,14 +3063,14 @@ int iommu_device_use_default_domain(struct
 device *dev)
 
  	mutex_lock(&group->mutex);
  	if (group->owner_cnt) {
-		if (group->owner || !iommu_is_default_domain(group) ||
-		    !xa_empty(&group->pasid_array)) {
+		if (group->owner || !iommu_is_default_domain(group)) {
  			ret = -EBUSY;
  			goto unlock_out;
  		}
  	}
 
Quoted 7 lines

drivers/iommu/iommu.c int iommu_device_use_default_domain(struct device *dev) · line 3073

Quoted 2 lines
drivers/iommu/iommu.c +1 -0
  	group->owner_cnt++;
+	assert_pasid_dma_ownership(group);

Old code returns error if pasid_xrrary is not empty.

New code continues to take ownership with a warning.

this is a functional change. Is it intended or not?

If iommu_device_use_default_domain() is called with pasid_array not empty, there must be a bug somewhere in the device driver. We should WARN it instead of returning an error. Probably this is a functional change? If so, I can add this in the commit message.

Best regards,
baolu

#8 lore raw

Re: [PATCH 2/2] iommu: Move pasid array from group to device

In reply to Lu Baolu

Hi Baolu,

Although this patch moves the domain reference pointer from a per-group structure to a per-device structure, the domain life-cycle is still expected to be managed per-group (i.e., iommu_domain_free() is called in iommu_group_release()). Is this what we expect?

Regards,
-Tina

Quoted 192 lines · Lu Baolu

The PASID (Process Address Space ID) feature is a device feature that allows a device driver to manage the PASID value and attach or detach its domain to the pasid. The pasid array, which is used to store the domain of each pasid, is currently stored in iommu group struct, but it would be more natural to move it to the device so that the device drivers don't need to understand the internal iommu group concept.

---
  include/linux/iommu.h |  2 ++
  drivers/iommu/iommu.c | 80 ++++++++++++-------------------------------
  2 files changed, 24 insertions(+), 58 deletions(-)
include/linux/iommu.h +2 -0
index b1dcb1b9b170..90be3efd0e91 100644
@@ -406,6 +406,7 @@ struct iommu_fault_param {
   * @iopf_param:	 I/O Page Fault queue and data
   * @fwspec:	 IOMMU fwspec data
   * @iommu_dev:	 IOMMU device this device is linked to
+ * @pasid_array: pasid-indexed array of domains attached to pasid
   * @priv:	 IOMMU Driver private data
   * @max_pasids:  number of PASIDs this device can consume
   * @attach_deferred: the dma domain attachment is deferred
@@ -420,6 +421,7 @@ struct dev_iommu {
  	struct iopf_device_param	*iopf_param;
  	struct iommu_fwspec		*fwspec;
  	struct iommu_device		*iommu_dev;
+	struct xarray			pasid_array;
  	void				*priv;
  	u32				max_pasids;
  	u32				attach_deferred:1;
drivers/iommu/iommu.c +22 -58
index 1a8fb30341e6..7fd53b9e9754 100644
@@ -48,7 +48,6 @@ struct iommu_group {
  	struct kobject kobj;
  	struct kobject *devices_kobj;
  	struct list_head devices;
-	struct xarray pasid_array;
  	struct mutex mutex;
  	void *iommu_data;
  	void (*iommu_data_release)(void *iommu_data);
@@ -302,6 +301,7 @@ static struct dev_iommu *dev_iommu_get(struct device *dev)
  		return NULL;
 
  	mutex_init(&param->lock);
+	xa_init(&param->pasid_array);
  	dev->iommu = param;
  	return param;
  }
@@ -891,7 +891,6 @@ struct iommu_group *iommu_group_alloc(void)
  	mutex_init(&group->mutex);
  	INIT_LIST_HEAD(&group->devices);
  	INIT_LIST_HEAD(&group->entry);
-	xa_init(&group->pasid_array);
 
  	ret = ida_alloc(&iommu_group_ida, GFP_KERNEL);
  	if (ret < 0) {
@@ -3041,8 +3040,11 @@ static bool iommu_is_default_domain(struct iommu_group *group)
   */
  static void assert_pasid_dma_ownership(struct iommu_group *group)
  {
+	struct group_device *device;
+
  	lockdep_assert_held(&group->mutex);
-	WARN_ON(!xa_empty(&group->pasid_array));
+	for_each_group_device(group, device)
+		WARN_ON(!xa_empty(&device->dev->iommu->pasid_array));
  }
 
  /**
@@ -3281,33 +3283,6 @@ bool iommu_group_dma_owner_claimed(struct iommu_group *group)
  }
  EXPORT_SYMBOL_GPL(iommu_group_dma_owner_claimed);
 
-static int __iommu_set_group_pasid(struct iommu_domain *domain,
-				   struct iommu_group *group, ioasid_t pasid)
-{
-	struct group_device *device;
-	int ret = 0;
-
-	for_each_group_device(group, device) {
-		ret = domain->ops->set_dev_pasid(domain, device->dev, pasid);
-		if (ret)
-			break;
-	}
-
-	return ret;
-}
-
-static void __iommu_remove_group_pasid(struct iommu_group *group,
-				       ioasid_t pasid)
-{
-	struct group_device *device;
-	const struct iommu_ops *ops;
-
-	for_each_group_device(group, device) {
-		ops = dev_iommu_ops(device->dev);
-		ops->remove_dev_pasid(device->dev, pasid);
-	}
-}
-
  /*
   * iommu_attach_device_pasid() - Attach a domain to pasid of device
   * @domain: the iommu domain.
@@ -3319,32 +3294,28 @@ static void __iommu_remove_group_pasid(struct iommu_group *group,
  int iommu_attach_device_pasid(struct iommu_domain *domain,
  			      struct device *dev, ioasid_t pasid)
  {
-	struct iommu_group *group;
+	struct dev_iommu *param = dev->iommu;
  	void *curr;
  	int ret;
 
  	if (!domain->ops->set_dev_pasid)
  		return -EOPNOTSUPP;
 
-	group = iommu_group_get(dev);
-	if (!group)
+	if (!param)
  		return -ENODEV;
 
-	mutex_lock(&group->mutex);
-	curr = xa_cmpxchg(&group->pasid_array, pasid, NULL, domain, GFP_KERNEL);
+	mutex_lock(&param->lock);
+	curr = xa_cmpxchg(&param->pasid_array, pasid, NULL, domain, GFP_KERNEL);
  	if (curr) {
  		ret = xa_err(curr) ? : -EBUSY;
  		goto out_unlock;
  	}
 
-	ret = __iommu_set_group_pasid(domain, group, pasid);
-	if (ret) {
-		__iommu_remove_group_pasid(group, pasid);
-		xa_erase(&group->pasid_array, pasid);
-	}
+	ret = domain->ops->set_dev_pasid(domain, dev, pasid);
+	if (ret)
+		xa_erase(&param->pasid_array, pasid);
  out_unlock:
-	mutex_unlock(&group->mutex);
-	iommu_group_put(group);
+	mutex_unlock(&param->lock);
 
  	return ret;
  }
@@ -3362,14 +3333,13 @@ EXPORT_SYMBOL_GPL(iommu_attach_device_pasid);
  void iommu_detach_device_pasid(struct iommu_domain *domain, struct device *dev,
  			       ioasid_t pasid)
  {
-	struct iommu_group *group = iommu_group_get(dev);
+	const struct iommu_ops *ops = dev_iommu_ops(dev);
+	struct dev_iommu *param = dev->iommu;
 
-	mutex_lock(&group->mutex);
-	__iommu_remove_group_pasid(group, pasid);
-	WARN_ON(xa_erase(&group->pasid_array, pasid) != domain);
-	mutex_unlock(&group->mutex);
-
-	iommu_group_put(group);
+	mutex_lock(&param->lock);
+	ops->remove_dev_pasid(dev, pasid);
+	WARN_ON(xa_erase(&param->pasid_array, pasid) != domain);
+	mutex_unlock(&param->lock);
  }
  EXPORT_SYMBOL_GPL(iommu_detach_device_pasid);
 
@@ -3392,18 +3362,12 @@ struct iommu_domain *iommu_get_domain_for_dev_pasid(struct device *dev,
  						    unsigned int type)
  {
  	struct iommu_domain *domain;
-	struct iommu_group *group;
 
-	group = iommu_group_get(dev);
-	if (!group)
-		return NULL;
-
-	xa_lock(&group->pasid_array);
-	domain = xa_load(&group->pasid_array, pasid);
+	xa_lock(&dev->iommu->pasid_array);
+	domain = xa_load(&dev->iommu->pasid_array, pasid);
  	if (type && domain && domain->type != type)
  		domain = ERR_PTR(-EBUSY);
-	xa_unlock(&group->pasid_array);
-	iommu_group_put(group);
+	xa_unlock(&dev->iommu->pasid_array);
 
  	return domain;
  }
#4 lore raw

Re: [PATCH 1/2] iommu: Consolidate pasid dma ownership check

In reply to Baolu Lu

Hi,

Quoted 22 lines · Baolu Lu
Quoted 21 lines · Kevin
Quoted 9 lines

  /**
   * iommu_device_use_default_domain() - Device driver wants to handle device
   *                                     DMA through the kernel DMA API.

drivers/iommu/iommu.c +0 -0
@@ -3052,14 +3063,14 @@ int iommu_device_use_default_domain(struct
 device *dev)
 
       mutex_lock(&group->mutex);
       if (group->owner_cnt) {

drivers/iommu/iommu.c int iommu_device_use_default_domain(struct · line 3074

Quoted 10 lines
drivers/iommu/iommu.c +2 -2
-        if (group->owner || !iommu_is_default_domain(group) ||
-            !xa_empty(&group->pasid_array)) {
+        if (group->owner || !iommu_is_default_domain(group)) {
               ret = -EBUSY;
               goto unlock_out;
           }
       }
 
       group->owner_cnt++;
+    assert_pasid_dma_ownership(group);

Old code returns error if pasid_xrrary is not empty.

Quoted 8 lines
Quoted 3 lines

New code continues to take ownership with a warning.

this is a functional change. Is it intended or not?

If iommu_device_use_default_domain() is called with pasid_array not empty, there must be a bug somewhere in the device driver. We should WARN it instead of returning an error. Probably this is a functional change? If so, I can add this in the commit message.

I think in case of a bug, returning error is still expected by callers. Otherwise, the callers might think everything is just fine, as the return value tells success.

Regards,
-Tina

Quoted 4 lines

Best regards,
baolu

author
#9 lore raw

Re: [PATCH 2/2] iommu: Move pasid array from group to device

In reply to tina.zhang
Quoted 1 line · tina.zhang

Hi Baolu,

Hi Tina,

Quoted 4 lines

Although this patch moves the domain reference pointer from a per-group structure to a per-device structure, the domain life-cycle is still expected to be managed per-group (i.e., iommu_domain_free() is called in iommu_group_release()). Is this what we expect?

The lifecycle of an iommu domain is independent of the lifecycle of the iommu group that it is attached to. The system domains, such as the default domain and the blocking domain, are allocated and managed by the iommu core. These domains are freed when the group is freed.

However, any device driver can allocate its own iommu domains. The device driver can set/remove the domain to/from the RID or PASID of the device.

Best regards,
baolu

#5 lore raw

RE: [PATCH 1/2] iommu: Consolidate pasid dma ownership check

In reply to Baolu Lu
Quoted 26 lines

From: Baolu Lu <[email protected]>
Sent: Tuesday, August 1, 2023 3:44 PM

Quoted 5 lines · Kevin

Review int

Quoted 5 lines

/**
* iommu_device_use_default_domain() - Device driver wants to handle
device
* DMA through the kernel DMA API.

(quoted diff) +0 -0
@@ -3052,14 +3063,14 @@ int

iommu_device_use_default_domain(struct

Quoted 16 lines
Quoted 4 lines
drivers/iommu/iommu.c +0 -0
 device *dev)
 
  	mutex_lock(&group->mutex);
  	if (group->owner_cnt) {

drivers/iommu/iommu.c int iommu_device_use_default_domain(struct device *dev) · line 3073

Quoted 10 lines
drivers/iommu/iommu.c +2 -2
-		if (group->owner || !iommu_is_default_domain(group) ||
-		    !xa_empty(&group->pasid_array)) {
+		if (group->owner || !iommu_is_default_domain(group)) {
  			ret = -EBUSY;
  			goto unlock_out;
  		}
  	}
 
  	group->owner_cnt++;
+	assert_pasid_dma_ownership(group);

Old code returns error if pasid_xrrary is not empty.

Quoted 9 lines
Quoted 3 lines

New code continues to take ownership with a warning.

this is a functional change. Is it intended or not?

If iommu_device_use_default_domain() is called with pasid_array not empty, there must be a bug somewhere in the device driver. We should WARN it instead of returning an error. Probably this is a functional change? If so, I can add this in the commit message.

IMHO we should WARN *and* return an error.

author
#6 lore raw

Re: [PATCH 1/2] iommu: Consolidate pasid dma ownership check

Quoted 28 lines · Kevin
Quoted 28 lines

From: Baolu Lu<[email protected]>
Sent: Tuesday, August 1, 2023 3:44 PM

Quoted 5 lines · Kevin

Review int

Quoted 5 lines

/**
* iommu_device_use_default_domain() - Device driver wants to handle
device
* DMA through the kernel DMA API.

(quoted diff) +0 -0
@@ -3052,14 +3063,14 @@ int

iommu_device_use_default_domain(struct

Quoted 18 lines
Quoted 4 lines
drivers/iommu/iommu.c +0 -0
 device *dev)
 
   	mutex_lock(&group->mutex);
   	if (group->owner_cnt) {

drivers/iommu/iommu.c int iommu_device_use_default_domain(struct device *dev) · line 3073

Quoted 10 lines
drivers/iommu/iommu.c +2 -2
-		if (group->owner || !iommu_is_default_domain(group) ||
-		    !xa_empty(&group->pasid_array)) {
+		if (group->owner || !iommu_is_default_domain(group)) {
   			ret = -EBUSY;
   			goto unlock_out;
   		}
   	}
 
   	group->owner_cnt++;
+	assert_pasid_dma_ownership(group);

Old code returns error if pasid_xrrary is not empty.

New code continues to take ownership with a warning.

Quoted 7 lines
Quoted 6 lines
Quoted 1 line

this is a functional change. Is it intended or not?

If iommu_device_use_default_domain() is called with pasid_array not empty, there must be a bug somewhere in the device driver. We should WARN it instead of returning an error. Probably this is a functional change? If so, I can add this in the commit message.

IMHO we should WARN*and* return an error.

Okay, fine to me. Will make this in the next version.

Best regards,
baolu

#10 lore raw
In reply to Lu Baolu
Quoted 5 lines · Lu Baolu

The PCI PASID enabling interface guarantees that the address space used by each PASID is unique. This is achieved by checking that the PCI ACS path is enabled for the device. If the path is not enabled, then the PASID feature cannot be used.

Quoted 7 lines

if (!pci_acs_path_enabled(pdev, NULL, PCI_ACS_RR | PCI_ACS_UF))
return -EINVAL;

The PASID array is not an attribute of the IOMMU group. It is more natural to store the PASID array in the per-device IOMMU data. This makes the code clearer and easier to understand. No functional changes are intended.

Is there a reason to do this?

*PCI* requires the ACS/etc because PCI kind of messed up how switches handled PASID so PASID doesn't work otherwise.

But there is nothing that says other bus type can't have working (non-PCI) PASID and still have device isolation issues.

So unless there is a really strong reason to do this we should keep the PASID list in the group just like the domain.

Jason

#11 lore raw
Quoted 19 lines

From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, August 2, 2023 10:16 PM

Quoted 5 lines · Lu Baolu

The PCI PASID enabling interface guarantees that the address space used by each PASID is unique. This is achieved by checking that the PCI ACS path is enabled for the device. If the path is not enabled, then the PASID feature cannot be used.

Quoted 7 lines

if (!pci_acs_path_enabled(pdev, NULL, PCI_ACS_RR | PCI_ACS_UF))
return -EINVAL;

The PASID array is not an attribute of the IOMMU group. It is more natural to store the PASID array in the per-device IOMMU data. This makes the code clearer and easier to understand. No functional changes are intended.

Is there a reason to do this?

Quoted 9 lines

*PCI* requires the ACS/etc because PCI kind of messed up how switches handled PASID so PASID doesn't work otherwise.

But there is nothing that says other bus type can't have working (non-PCI) PASID and still have device isolation issues.

So unless there is a really strong reason to do this we should keep the PASID list in the group just like the domain.

this comes from the consensus in [1].

[1] https://lore.kernel.org/linux-iommu/[email protected]/

#12 lore raw
Quoted 25 lines · Kevin
Quoted 25 lines

From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, August 2, 2023 10:16 PM

Quoted 5 lines · Lu Baolu

The PCI PASID enabling interface guarantees that the address space used by each PASID is unique. This is achieved by checking that the PCI ACS path is enabled for the device. If the path is not enabled, then the PASID feature cannot be used.

Quoted 7 lines

if (!pci_acs_path_enabled(pdev, NULL, PCI_ACS_RR | PCI_ACS_UF))
return -EINVAL;

The PASID array is not an attribute of the IOMMU group. It is more natural to store the PASID array in the per-device IOMMU data. This makes the code clearer and easier to understand. No functional changes are intended.

Is there a reason to do this?

*PCI* requires the ACS/etc because PCI kind of messed up how switches handled PASID so PASID doesn't work otherwise.

But there is nothing that says other bus type can't have working (non-PCI) PASID and still have device isolation issues.

Quoted 7 lines
Quoted 3 lines

So unless there is a really strong reason to do this we should keep the PASID list in the group just like the domain.

this comes from the consensus in [1].

[1] https://lore.kernel.org/linux-iommu/[email protected]/

That consensus was that we don't have PASID support if there is multi-device groups, at least in iommufd.. That makes sense. If we want to change the core code to enforce this that also makes sense

But this series is just moving the array?

Jason

#13 lore raw
Quoted 37 lines

From: Jason Gunthorpe <[email protected]>
Sent: Thursday, August 3, 2023 11:19 PM

Quoted 5 lines · Kevin
Quoted 5 lines

From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, August 2, 2023 10:16 PM

Quoted 1 line · Lu Baolu

The PCI PASID enabling interface guarantees that the address space

used

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

by each PASID is unique. This is achieved by checking that the PCI ACS path is enabled for the device. If the path is not enabled, then the PASID feature cannot be used.

if (!pci_acs_path_enabled(pdev, NULL, PCI_ACS_RR | PCI_ACS_UF))
return -EINVAL;

The PASID array is not an attribute of the IOMMU group. It is more natural to store the PASID array in the per-device IOMMU data. This makes the code clearer and easier to understand. No functional

changes

Quoted 16 lines
Quoted 4 lines
Quoted 1 line

are intended.

Is there a reason to do this?

Quoted 9 lines

*PCI* requires the ACS/etc because PCI kind of messed up how switches handled PASID so PASID doesn't work otherwise.

But there is nothing that says other bus type can't have working (non-PCI) PASID and still have device isolation issues.

So unless there is a really strong reason to do this we should keep the PASID list in the group just like the domain.

this comes from the consensus in [1].

Quoted 8 lines
Quoted 1 line

[1] https://lore.kernel.org/linux-iommu/[email protected]/

That consensus was that we don't have PASID support if there is multi-device groups, at least in iommufd.. That makes sense. If we want to change the core code to enforce this that also makes sense

But this series is just moving the array?

This is a preparation series for supporting PASID in iommufd (will be sent out probably after next version of the nesting series).

It only moves the array by taking only PCI into consideration. We could add an explicit enforcement in iommu core for all types of devices.

author
#14 lore raw
In reply to Jason Gunthorpe
Quoted 28 lines · Jason Gunthorpe
Quoted 27 lines · Kevin
Quoted 27 lines

From: Jason Gunthorpe<[email protected]>
Sent: Wednesday, August 2, 2023 10:16 PM

Quoted 5 lines · Lu Baolu

The PCI PASID enabling interface guarantees that the address space used by each PASID is unique. This is achieved by checking that the PCI ACS path is enabled for the device. If the path is not enabled, then the PASID feature cannot be used.

Quoted 7 lines

if (!pci_acs_path_enabled(pdev, NULL, PCI_ACS_RR | PCI_ACS_UF))
return -EINVAL;

The PASID array is not an attribute of the IOMMU group. It is more natural to store the PASID array in the per-device IOMMU data. This makes the code clearer and easier to understand. No functional changes are intended.

Is there a reason to do this?

*PCI* requires the ACS/etc because PCI kind of messed up how switches handled PASID so PASID doesn't work otherwise.

But there is nothing that says other bus type can't have working (non-PCI) PASID and still have device isolation issues.

So unless there is a really strong reason to do this we should keep the PASID list in the group just like the domain.

Quoted 6 lines
Quoted 3 lines

this comes from the consensus in [1].

[1]https://lore.kernel.org/linux-iommu/[email protected]/

That consensus was that we don't have PASID support if there is multi-device groups, at least in iommufd.. That makes sense. If we want to change the core code to enforce this that also makes sense

In my initial plan, I had a third patch that would have enforced single- device groups for PASID interfaces in the core. But I ultimately dropped it because it is the fact for PCI devices, but I am not sure about other buses although perhaps there is none.

Quoted 1 line

But this series is just moving the array?

So I took the first step by moving the pasid_array from iommu group to the device. :-)

Best regards,
baolu

author
#15 lore raw
Quoted 41 lines · Baolu Lu
Quoted 28 lines · Jason Gunthorpe
Quoted 27 lines · Kevin
Quoted 27 lines

From: Jason Gunthorpe<[email protected]>
Sent: Wednesday, August 2, 2023 10:16 PM

Quoted 5 lines · Lu Baolu

The PCI PASID enabling interface guarantees that the address space used by each PASID is unique. This is achieved by checking that the PCI ACS path is enabled for the device. If the path is not enabled, then the PASID feature cannot be used.

Quoted 7 lines

     if (!pci_acs_path_enabled(pdev, NULL, PCI_ACS_RR | PCI_ACS_UF)) return -EINVAL;

The PASID array is not an attribute of the IOMMU group. It is more natural to store the PASID array in the per-device IOMMU data. This makes the code clearer and easier to understand. No functional changes are intended.

Is there a reason to do this?

*PCI*  requires the ACS/etc because PCI kind of messed up how switches handled PASID so PASID doesn't work otherwise.

But there is nothing that says other bus type can't have working (non-PCI) PASID and still have device isolation issues.

So unless there is a really strong reason to do this we should keep the PASID list in the group just like the domain.

Quoted 6 lines
Quoted 3 lines

this comes from the consensus in [1].

[1]https://lore.kernel.org/linux-iommu/[email protected]/

That consensus was that we don't have PASID support if there is multi-device groups, at least in iommufd.. That makes sense. If we want to change the core code to enforce this that also makes sense

In my initial plan, I had a third patch that would have enforced single- device groups for PASID interfaces in the core. But I ultimately dropped it because it is the fact for PCI devices, but I am not sure about other buses although perhaps there is none.

Quoted 4 lines
Quoted 1 line

But this series is just moving the array?

So I took the first step by moving the pasid_array from iommu group to the device. 😄

In my mind, iommu_group was introduced to solve the PCI alias and P2P transactions which bypass IOMMU translation. When we enter the PASID world, the architecture should disallow these anymore. Hence, it's safe to move pasid_array to device.

This was the motivation of this series.

Best regards,
baolu

#16 lore raw
Quoted 48 lines · Baolu Lu
Quoted 42 lines · Baolu Lu
Quoted 29 lines · Jason Gunthorpe
Quoted 28 lines · Kevin
Quoted 28 lines

From: Jason Gunthorpe<[email protected]>
Sent: Wednesday, August 2, 2023 10:16 PM

Quoted 6 lines · Lu Baolu

The PCI PASID enabling interface guarantees that the address space used
by each PASID is unique. This is achieved by checking that the PCI ACS path is enabled for the device. If the path is not enabled, then the PASID feature cannot be used.

Quoted 7 lines

     if (!pci_acs_path_enabled(pdev, NULL, PCI_ACS_RR | PCI_ACS_UF)) return -EINVAL;

The PASID array is not an attribute of the IOMMU group. It is more natural to store the PASID array in the per-device IOMMU data. This makes the code clearer and easier to understand. No functional changes are intended.

Is there a reason to do this?

*PCI*  requires the ACS/etc because PCI kind of messed up how switches handled PASID so PASID doesn't work otherwise.

But there is nothing that says other bus type can't have working (non-PCI) PASID and still have device isolation issues.

So unless there is a really strong reason to do this we should keep the PASID list in the group just like the domain.

Quoted 6 lines
Quoted 3 lines

this comes from the consensus in [1].

[1]https://lore.kernel.org/linux-iommu/[email protected]/

That consensus was that we don't have PASID support if there is multi-device groups, at least in iommufd.. That makes sense. If we want to change the core code to enforce this that also makes sense

In my initial plan, I had a third patch that would have enforced single- device groups for PASID interfaces in the core. But I ultimately dropped it because it is the fact for PCI devices, but I am not sure about other buses although perhaps there is none.

Quoted 4 lines
Quoted 1 line

But this series is just moving the array?

So I took the first step by moving the pasid_array from iommu group to the device. 😄

Quoted 6 lines

In my mind, iommu_group was introduced to solve the PCI alias and P2P transactions which bypass IOMMU translation. When we enter the PASID world, the architecture should disallow these anymore. Hence, it's safe to move pasid_array to device.

This was the motivation of this series.

I think you should add a protection as well, directly prevent multi-device groups being used with pasid.

Jason