Skip to contentSkip to conversation

iommu/vt-d: Move PCI PASID enablement to probe path

PATCH Superseded Lu Baolu 4 participants 19 messages Aug 16, 2024 – Aug 26, 2024

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Lu Baolu

author maintainer
#0 lore raw

PATCH 1/1 iommu/vt-d: Move PCI PASID enablement to probe path

Currently, PCI PASID is enabled alongside PCI ATS when an iommu domain is attached to the device and disabled when the device transitions to block translation mode. This approach is inappropriate as PCI PASID is a device feature independent of the type of the attached domain.

Enable PCI PASID during the IOMMU device probe and disables it during the release path.

---
 drivers/iommu/intel/iommu.c | 27 +++++++++++++--------------
 1 file changed, 13 insertions(+), 14 deletions(-)
drivers/iommu/intel/iommu.c +13 -14
index 9ff8b83c19a3..5a8080c71b04 100644
@@ -1322,15 +1322,6 @@ static void iommu_enable_pci_caps(struct device_domain_info *info)
 		return;
 
 	pdev = to_pci_dev(info->dev);
-
-	/* The PCIe spec, in its wisdom, declares that the behaviour of
-	   the device if you enable PASID support after ATS support is
-	   undefined. So always enable PASID support on devices which
-	   have it, even if we can't yet know if we're ever going to
-	   use it. */
-	if (info->pasid_supported && !pci_enable_pasid(pdev, info->pasid_supported & ~1))
-		info->pasid_enabled = 1;
-
 	if (info->ats_supported && pci_ats_page_aligned(pdev) &&
 	    !pci_enable_ats(pdev, VTD_PAGE_SHIFT)) {
 		info->ats_enabled = 1;
@@ -1352,11 +1343,6 @@ static void iommu_disable_pci_caps(struct device_domain_info *info)
 		info->ats_enabled = 0;
 		domain_update_iotlb(info->domain);
 	}
-
-	if (info->pasid_enabled) {
-		pci_disable_pasid(pdev);
-		info->pasid_enabled = 0;
-	}
 }
 
 static void intel_flush_iotlb_all(struct iommu_domain *domain)
@@ -4110,6 +4096,16 @@ static struct iommu_device *intel_iommu_probe_device(struct device *dev)
 		}
 	}
 
+	/*
+	 * The PCIe spec, in its wisdom, declares that the behaviour of the
+	 * device is undefined if you enable PASID support after ATS support.
+	 * So always enable PASID support on devices which have it, even if
+	 * we can't yet know if we're ever going to use it.
+	 */
+	if (info->pasid_supported &&
+	    !pci_enable_pasid(pdev, info->pasid_supported & ~1))
+		info->pasid_enabled = 1;
+
 	intel_iommu_debugfs_create_dev(info);
 
 	return &iommu->iommu;
@@ -4128,6 +4124,9 @@ static void intel_iommu_release_device(struct device *dev)
 	struct device_domain_info *info = dev_iommu_priv_get(dev);
 	struct intel_iommu *iommu = info->iommu;
 
+	if (info->pasid_enabled)
+		pci_disable_pasid(to_pci_dev(dev));
+
 	mutex_lock(&iommu->iopf_lock);
 	if (dev_is_pci(dev) && pci_ats_supported(to_pci_dev(dev)))
 		device_rbtree_remove(info);
Signature · 1 line
2.34.1
#1 lore raw

Hi Lu,

Quoted 4 lines · Lu Baolu

Currently, PCI PASID is enabled alongside PCI ATS when an iommu domain is attached to the device and disabled when the device transitions to block translation mode. This approach is inappropriate as PCI PASID is a device feature independent of the type of the attached domain.

Reading through other thread, I thought we want to enable both PASID and PRI in device probe path. Did I miss something?

-Vasant

author maintainer
#2 lore raw
Quoted 7 lines · Vasant Hegde
Quoted 4 lines · Lu Baolu

Currently, PCI PASID is enabled alongside PCI ATS when an iommu domain is attached to the device and disabled when the device transitions to block translation mode. This approach is inappropriate as PCI PASID is a device feature independent of the type of the attached domain.

Reading through other thread, I thought we want to enable both PASID and PRI in device probe path. Did I miss something?

PRI is different. PRI should be enabled when the first iopf-capable domain is attached to device or its PASID, and disabled when the last such domain is detached.

Thanks,
baolu

#3 lore raw
Quoted 6 lines · Baolu Lu
Quoted 5 lines · Vasant Hegde
Quoted 4 lines · Lu Baolu

Currently, PCI PASID is enabled alongside PCI ATS when an iommu domain is attached to the device and disabled when the device transitions to block translation mode. This approach is inappropriate as PCI PASID is a device feature independent of the type of the attached domain.

Quoted 6 lines
Quoted 2 lines

Reading through other thread, I thought we want to enable both PASID and PRI in device probe path. Did I miss something?

PRI is different. PRI should be enabled when the first iopf-capable domain is attached to device or its PASID, and disabled when the last such domain is detached.

+1

Jason

#4 lore raw
In reply to Baolu Lu

Hi,

Quoted 6 lines · Baolu Lu
Quoted 5 lines · Vasant Hegde
Quoted 4 lines · Lu Baolu

Currently, PCI PASID is enabled alongside PCI ATS when an iommu domain is attached to the device and disabled when the device transitions to block translation mode. This approach is inappropriate as PCI PASID is a device feature independent of the type of the attached domain.

Quoted 6 lines
Quoted 2 lines

Reading through other thread, I thought we want to enable both PASID and PRI in device probe path. Did I miss something?

PRI is different. PRI should be enabled when the first iopf-capable domain is attached to device or its PASID, and disabled when the last such domain is detached.

Right. That's what AMD driver also does (We enable it when we attach IOPF capable domain). But looking into pci_enable_pri() :

202 /*

203          * VFs must not implement the PRI Capability.  If their PF
204          * implements PRI, it is shared by the VFs, so if the PF PRI is
205          * enabled, it is also enabled for the VF.
206          */

207 if (pdev->is_virtfn) {
208 if (pci_physfn(pdev)->pri_enabled)
209 return 0;
210 return -EINVAL;
211 }
212

If we try to enable PRI for VF without first enabling it in PF it will fail right?

Now if PF is attached to non-IOPF capable domain (like in AMD case attaching to domain with V1 page table) and we try to attach VF to IOPF capable domain (say AMD v2 page table -OR- nested domain) it will fail right?

-Vasant

author maintainer
#5 lore raw
Quoted 29 lines · Vasant Hegde
Quoted 11 lines · Baolu Lu
Quoted 7 lines · Vasant Hegde
Quoted 4 lines · Lu Baolu

Currently, PCI PASID is enabled alongside PCI ATS when an iommu domain is attached to the device and disabled when the device transitions to block translation mode. This approach is inappropriate as PCI PASID is a device feature independent of the type of the attached domain.

Reading through other thread, I thought we want to enable both PASID and PRI in device probe path. Did I miss something?

PRI is different. PRI should be enabled when the first iopf-capable domain is attached to device or its PASID, and disabled when the last such domain is detached.

Right. That's what AMD driver also does (We enable it when we attach IOPF capable domain). But looking into pci_enable_pri() :

202 /*

203          * VFs must not implement the PRI Capability.  If their PF
204          * implements PRI, it is shared by the VFs, so if the PF PRI is
205          * enabled, it is also enabled for the VF.
206          */

207 if (pdev->is_virtfn) {
208 if (pci_physfn(pdev)->pri_enabled)
209 return 0;
210 return -EINVAL;
211 }
212

Quoted 5 lines

If we try to enable PRI for VF without first enabling it in PF it will fail right?

Now if PF is attached to non-IOPF capable domain (like in AMD case attaching to domain with V1 page table) and we try to attach VF to IOPF capable domain (say AMD v2 page table -OR- nested domain) it will fail right?

Yeah! So, the iommu driver should basically control the PRI switch on the PF whenever someone wants to use it on a VF.

We could simplify things by turning on PRI for the whole PF when the first iopf-capable domain is attached to a VF. Then, we'd only turn it off once all VFs have such domains detached.

Thanks,
baolu
Yeah, so the iommu driver should flip the PRI switch on the PF whenever someone wants to turn it on for its VF.

We could change things up so that when the first iopf-capable domain is attached to any VF or PF, we turn on PRI for the whole device. Then, we turn it off when every VF and PF have such domains detached.

author maintainer
#6 lore raw
Quoted 5 lines · Baolu Lu

Yeah, so the iommu driver should flip the PRI switch on the PF whenever someone wants to turn it on for its VF.

We could change things up so that when the first iopf-capable domain is attached to any VF or PF, we turn on PRI for the whole device. Then, we turn it off when every VF and PF have such domains detached.

Please ignore this. I forgot to remove it when writing the words. Sorry about it.

Thanks,
baolu

#7 lore raw
In reply to Baolu Lu
Quoted 30 lines · Baolu Lu
Quoted 29 lines · Vasant Hegde
Quoted 11 lines · Baolu Lu
Quoted 7 lines · Vasant Hegde
Quoted 4 lines · Lu Baolu

Currently, PCI PASID is enabled alongside PCI ATS when an iommu domain is attached to the device and disabled when the device transitions to block translation mode. This approach is inappropriate as PCI PASID is a device feature independent of the type of the attached domain.

Reading through other thread, I thought we want to enable both PASID and PRI in device probe path. Did I miss something?

PRI is different. PRI should be enabled when the first iopf-capable domain is attached to device or its PASID, and disabled when the last such domain is detached.

Right. That's what AMD driver also does (We enable it when we attach IOPF capable domain). But looking into pci_enable_pri() :

202 /*

203          * VFs must not implement the PRI Capability.  If their PF
204          * implements PRI, it is shared by the VFs, so if the PF PRI is
205          * enabled, it is also enabled for the VF.
206          */

207 if (pdev->is_virtfn) {
208 if (pci_physfn(pdev)->pri_enabled)
209 return 0;
210 return -EINVAL;
211 }
212

Quoted 8 lines
Quoted 5 lines

If we try to enable PRI for VF without first enabling it in PF it will fail right?

Now if PF is attached to non-IOPF capable domain (like in AMD case attaching to domain with V1 page table) and we try to attach VF to IOPF capable domain (say AMD v2 page table -OR- nested domain) it will fail right?

Yeah! So, the iommu driver should basically control the PRI switch on the PF whenever someone wants to use it on a VF.

PRI enable sounds like PASID enable to me.

The ATS control is per VF/PF, and PRI does nothing unless ATS returns a non-present indication.

Like PASID, it seems the purpose of PRI caps is to negotiate if the CPU can process PRI TLPs globally.

So, I'd guess that just like PASID we should turn it on at PF probe time if the IOMMU can globall handle PRI.

Enabling ATS will cause PRI TLPs to be sent.

Probably more of this code should be lifted out of the iommu drivers..

Jason

author maintainer
#8 lore raw
Quoted 46 lines · Jason Gunthorpe
Quoted 30 lines · Baolu Lu
Quoted 29 lines · Vasant Hegde
Quoted 11 lines · Baolu Lu
Quoted 7 lines · Vasant Hegde
Quoted 4 lines · Lu Baolu

Currently, PCI PASID is enabled alongside PCI ATS when an iommu domain is attached to the device and disabled when the device transitions to block translation mode. This approach is inappropriate as PCI PASID is a device feature independent of the type of the attached domain.

Reading through other thread, I thought we want to enable both PASID and PRI in device probe path. Did I miss something?

PRI is different. PRI should be enabled when the first iopf-capable domain is attached to device or its PASID, and disabled when the last such domain is detached.

Right. That's what AMD driver also does (We enable it when we attach IOPF capable domain). But looking into pci_enable_pri() :

202 /*

203          * VFs must not implement the PRI Capability.  If their PF
204          * implements PRI, it is shared by the VFs, so if the PF PRI is
205          * enabled, it is also enabled for the VF.
206          */

207 if (pdev->is_virtfn) {
208 if (pci_physfn(pdev)->pri_enabled)
209 return 0;
210 return -EINVAL;
211 }
212

Quoted 7 lines
Quoted 5 lines

If we try to enable PRI for VF without first enabling it in PF it will fail right?

Now if PF is attached to non-IOPF capable domain (like in AMD case attaching to domain with V1 page table) and we try to attach VF to IOPF capable domain (say AMD v2 page table -OR- nested domain) it will fail right?

Yeah! So, the iommu driver should basically control the PRI switch on the PF whenever someone wants to use it on a VF.

PRI enable sounds like PASID enable to me.

The ATS control is per VF/PF, and PRI does nothing unless ATS returns a non-present indication.

Like PASID, it seems the purpose of PRI caps is to negotiate if the CPU can process PRI TLPs globally.

Quoted 6 lines

So, I'd guess that just like PASID we should turn it on at PF probe time if the IOMMU can globall handle PRI.

Enabling ATS will cause PRI TLPs to be sent.

Probably more of this code should be lifted out of the iommu drivers..

Some architectures, including VT-d non-scalable mode, doesn't support ATS translation and translated requests when it is working in the IDENTITY domain mode. In that case, probably PCI ATS still need to be disabled when such domain is attached and re-enabled when the domain is detached.

Thanks,
baolu

#9 lore raw

Hi All,

Quoted 54 lines · Baolu Lu
Quoted 53 lines · Jason Gunthorpe
Quoted 31 lines · Baolu Lu
Quoted 30 lines · Vasant Hegde
Quoted 6 lines · Baolu Lu
Quoted 5 lines · Vasant Hegde
Quoted 4 lines · Lu Baolu

Currently, PCI PASID is enabled alongside PCI ATS when an iommu domain is attached to the device and disabled when the device transitions to block translation mode. This approach is inappropriate as PCI PASID is a device feature independent of the type of the attached domain.

Quoted 6 lines
Quoted 3 lines

Reading through other thread, I thought we want to enable both PASID and PRI in
device probe path. Did I miss something?

PRI is different. PRI should be enabled when the first iopf-capable domain is attached to device or its PASID, and disabled when the last such domain is detached.

Right. That's what AMD driver also does (We enable it when we attach IOPF capable domain). But looking into pci_enable_pri() :

202         /*
203          * VFs must not implement the PRI Capability.  If their PF 204          * implements PRI, it is shared by the VFs, so if the PF PRI is 205          * enabled, it is also enabled for the VF.
206          */
207         if (pdev->is_virtfn) {
208                 if (pci_physfn(pdev)->pri_enabled)
209                         return 0;
210                 return -EINVAL;
211         }
212

Quoted 8 lines
Quoted 6 lines

If we try to enable PRI for VF without first enabling it in PF it will fail right?

Now if PF is attached to non-IOPF capable domain (like in AMD case attaching to domain with V1 page table) and we try to attach VF to IOPF capable domain  (say AMD v2 page table -OR- nested domain) it will fail right?

Yeah! So, the iommu driver should basically control the PRI switch on the PF whenever someone wants to use it on a VF.

PRI enable sounds like PASID enable to me.

The ATS control is per VF/PF, and PRI does nothing unless ATS returns a non-present indication.

Like PASID, it seems the purpose of PRI caps is to negotiate if the CPU can process PRI TLPs globally.

So, I'd guess that just like PASID we should turn it on at PF probe time if the IOMMU can globall handle PRI.

Enabling ATS will cause PRI TLPs to be sent.

Quoted 7 lines
Quoted 1 line

Probably more of this code should be lifted out of the iommu drivers..

Some architectures, including VT-d non-scalable mode, doesn't support ATS translation and translated requests when it is working in the IDENTITY domain mode. In that case, probably PCI ATS still need to be disabled when such domain is attached and re-enabled when the domain is detached.

Does it make sense to move both PASID/PRI enablement to probe() path? something like below :

[I am assuming ops->dev_enable_feat() interface is going away]

- Enable device side PASID/PRI during ops->probe_device()
- In device attach path (ops->attach_dev()), depending on IOMMU, device and
domain capability configure the features like PASID, IOPF and ATS. That means ATS enablement is still done at attach device path.

-Vasant

#10 lore raw
Quoted 70 lines · Vasant Hegde

Hi All,

Quoted 54 lines · Baolu Lu
Quoted 53 lines · Jason Gunthorpe
Quoted 31 lines · Baolu Lu
Quoted 30 lines · Vasant Hegde
Quoted 6 lines · Baolu Lu
Quoted 5 lines · Vasant Hegde
Quoted 4 lines · Lu Baolu

Currently, PCI PASID is enabled alongside PCI ATS when an iommu domain is attached to the device and disabled when the device transitions to block translation mode. This approach is inappropriate as PCI PASID is a device feature independent of the type of the attached domain.

Quoted 6 lines
Quoted 3 lines

Reading through other thread, I thought we want to enable both PASID and PRI in
device probe path. Did I miss something?

PRI is different. PRI should be enabled when the first iopf-capable domain is attached to device or its PASID, and disabled when the last such domain is detached.

Right. That's what AMD driver also does (We enable it when we attach IOPF capable domain). But looking into pci_enable_pri() :

202         /*
203          * VFs must not implement the PRI Capability.  If their PF 204          * implements PRI, it is shared by the VFs, so if the PF PRI is 205          * enabled, it is also enabled for the VF.
206          */
207         if (pdev->is_virtfn) {
208                 if (pci_physfn(pdev)->pri_enabled)
209                         return 0;
210                 return -EINVAL;
211         }
212

Quoted 8 lines
Quoted 6 lines

If we try to enable PRI for VF without first enabling it in PF it will fail right?

Now if PF is attached to non-IOPF capable domain (like in AMD case attaching to domain with V1 page table) and we try to attach VF to IOPF capable domain  (say AMD v2 page table -OR- nested domain) it will fail right?

Yeah! So, the iommu driver should basically control the PRI switch on the PF whenever someone wants to use it on a VF.

PRI enable sounds like PASID enable to me.

The ATS control is per VF/PF, and PRI does nothing unless ATS returns a non-present indication.

Like PASID, it seems the purpose of PRI caps is to negotiate if the CPU can process PRI TLPs globally.

So, I'd guess that just like PASID we should turn it on at PF probe time if the IOMMU can globall handle PRI.

Enabling ATS will cause PRI TLPs to be sent.

Quoted 7 lines
Quoted 1 line

Probably more of this code should be lifted out of the iommu drivers..

Some architectures, including VT-d non-scalable mode, doesn't support ATS translation and translated requests when it is working in the IDENTITY domain mode. In that case, probably PCI ATS still need to be disabled when such domain is attached and re-enabled when the domain is detached.

Does it make sense to move both PASID/PRI enablement to probe() path? something like below :

Quoted 6 lines

[I am assuming ops->dev_enable_feat() interface is going away]

- Enable device side PASID/PRI during ops->probe_device()
- In device attach path (ops->attach_dev()), depending on IOMMU, device and
domain capability configure the features like PASID, IOPF and ATS. That means ATS enablement is still done at attach device path.

Are we sure that it is ok to enable PRI before enabling ATS?

Signature · 2 lines
Regards,
Yi Liu
#11 lore raw
Quoted 1 line · Yi Liu

Are we sure that it is ok to enable PRI before enabling ATS?

I did not see language in the spec prohibiting this order.

Jason

#12 lore raw
In reply to Vasant Hegde
Quoted 3 lines · Vasant Hegde
Quoted 3 lines

Some architectures, including VT-d non-scalable mode, doesn't support ATS translation and translated requests when it is working in the IDENTITY domain mode.

ARM has a similar issue.

ATS enablement should be done when the domain is attached in those cases.

Arguably you don't want to turn ATS on anyhow for pure IDENTITY with no PASID because it is just pointless.

Quoted 6 lines

In that case, probably PCI ATS still need to be

Quoted 2 lines

disabled when such domain is attached and re-enabled when the domain is detached.

Does it make sense to move both PASID/PRI enablement to probe() path? something like below :

It makes sense.

I don't see any ordering restriction in the PCI specification.

Notice that PASID does have a specific called out restriction:

/*
* Note that PASID must be enabled before, and disabled after ATS:
* PCI Express Base 4.0r1.0 - 10.5.1.3 ATS Control Register *
* Behavior is undefined if this bit is Set and the value of the PASID
* Enable, Execute Requested Enable, or Privileged Mode Requested bits
* are changed.
*/

Quoted 1 line

[I am assuming ops->dev_enable_feat() interface is going away]

Is the plan

Quoted 1 line

- Enable device side PASID/PRI during ops->probe_device()

Yes

Quoted 3 lines

- In device attach path (ops->attach_dev()), depending on IOMMU, device and
domain capability configure the features like PASID, IOPF and ATS. That means ATS enablement is still done at attach device path.

From a PCI perspective only ATS can be changed at this point..

The SW construct of IOPF can be changed during domain attachment.

Everything that is PF-only must be setup during probe_device only otherwise SRIOV VFs will be broken insome cases.

See
https://lore.kernel.org/all/[email protected]/
for this concept applied to ATS.

This means probe_device() has to do:

- ATS properties
- PRI
- PASID properties

At a minimum.

It would be nice if the iommu core code did this setup in one place immediately after calling probe_device() but before attaching a domain.

There is no particularly good reason to have this coded in all the iommu drivers.

Jason

#13 lore raw

Jason,

Quoted 48 lines · Jason Gunthorpe
Quoted 3 lines · Vasant Hegde
Quoted 3 lines

Some architectures, including VT-d non-scalable mode, doesn't support ATS translation and translated requests when it is working in the IDENTITY domain mode.

ARM has a similar issue.

ATS enablement should be done when the domain is attached in those cases.

Arguably you don't want to turn ATS on anyhow for pure IDENTITY with no PASID because it is just pointless.

Quoted 6 lines

In that case, probably PCI ATS still need to be

Quoted 2 lines

disabled when such domain is attached and re-enabled when the domain is detached.

Does it make sense to move both PASID/PRI enablement to probe() path? something like below :

It makes sense.

I don't see any ordering restriction in the PCI specification.

Notice that PASID does have a specific called out restriction:

/*
* Note that PASID must be enabled before, and disabled after ATS:
* PCI Express Base 4.0r1.0 - 10.5.1.3 ATS Control Register *
* Behavior is undefined if this bit is Set and the value of the PASID
* Enable, Execute Requested Enable, or Privileged Mode Requested bits
* are changed.
*/

Quoted 1 line

[I am assuming ops->dev_enable_feat() interface is going away]

Is the plan

Quoted 1 line

- Enable device side PASID/PRI during ops->probe_device()

Yes

Quoted 3 lines

- In device attach path (ops->attach_dev()), depending on IOMMU, device and
domain capability configure the features like PASID, IOPF and ATS. That means ATS enablement is still done at attach device path.

Quoted 6 lines

From a PCI perspective only ATS can be changed at this point..

The SW construct of IOPF can be changed during domain attachment.

Everything that is PF-only must be setup during probe_device only otherwise SRIOV VFs will be broken insome cases.

Makes sense. I will modify AMD driver to enable PRI/PASID in probe() path.

Quoted 11 lines

See
https://lore.kernel.org/all/[email protected]/
for this concept applied to ATS.

This means probe_device() has to do:

- ATS properties
- PRI
- PASID properties

Quoted 8 lines

At a minimum.

It would be nice if the iommu core code did this setup in one place immediately after calling probe_device() but before attaching a domain.

There is no particularly good reason to have this coded in all the iommu drivers.

Yeah. that makes sense. We may have to adjust few things in driver (at least AMD driver). But its doable.

-Vasant

#14 lore raw
In reply to Vasant Hegde
Quoted 34 lines · Vasant Hegde

Hi,

Quoted 6 lines · Baolu Lu
Quoted 5 lines · Vasant Hegde
Quoted 4 lines · Lu Baolu

Currently, PCI PASID is enabled alongside PCI ATS when an iommu domain is attached to the device and disabled when the device transitions to block translation mode. This approach is inappropriate as PCI PASID is a device feature independent of the type of the attached domain.

Quoted 6 lines
Quoted 2 lines

Reading through other thread, I thought we want to enable both PASID and PRI in device probe path. Did I miss something?

PRI is different. PRI should be enabled when the first iopf-capable domain is attached to device or its PASID, and disabled when the last such domain is detached.

Right. That's what AMD driver also does (We enable it when we attach IOPF capable domain). But looking into pci_enable_pri() :

202 /*

203          * VFs must not implement the PRI Capability.  If their PF
204          * implements PRI, it is shared by the VFs, so if the PF PRI is
205          * enabled, it is also enabled for the VF.
206          */

207 if (pdev->is_virtfn) {
208 if (pci_physfn(pdev)->pri_enabled)
209 return 0;
210 return -EINVAL;
211 }
212

Quoted 5 lines

If we try to enable PRI for VF without first enabling it in PF it will fail right?

Now if PF is attached to non-IOPF capable domain (like in AMD case attaching to domain with V1 page table) and we try to attach VF to IOPF capable domain (say AMD v2 page table -OR- nested domain) it will fail right?

It looks like iommufd does not support it for VF. Baolu's iopf series disallowed it since v7. And the code in the below acts it.

+ /*
+ * Once we turn on PCI/PRI support for VF, the response failure code
+ * should not be forwarded to the hardware due to PRI being a shared
+ * resource between PF and VFs. There is no coordination for this
+ * shared capability. This waits for a vPRI reset to recover.
+ */
+ if (dev_is_pci(dev) && to_pci_dev(dev)->is_virtfn)
+ return -EINVAL;

v6: https://lore.kernel.org/linux-iommu/[email protected]/
- Refine the attach handle code by shifting the handle allocation to
the caller. The caller will then provide the allocated handle to the domain attachment interfaces.
- Add reference counter in iommufd_fault_iopf_enable/disable() helpers.
- Fix the return values of fault FD's read/write fops.
- Add IOMMU_CAP_USER_IOASID_TABLE capability and check it before roll back getting attach_handle to RID.
- Move the iopf respond queue from iommufd device to iommufd fault.
- Disallow PRI enablement on SR-IOV VF devices.

Signature · 2 lines
Regards,
Yi Liu
Reviewed-by on v1 1/1
#15 lore raw
In reply to Lu Baolu
Quoted 64 lines · Lu Baolu

Currently, PCI PASID is enabled alongside PCI ATS when an iommu domain is attached to the device and disabled when the device transitions to block translation mode. This approach is inappropriate as PCI PASID is a device feature independent of the type of the attached domain.

Enable PCI PASID during the IOMMU device probe and disables it during the release path.

---
  drivers/iommu/intel/iommu.c | 27 +++++++++++++--------------
  1 file changed, 13 insertions(+), 14 deletions(-)
drivers/iommu/intel/iommu.c +10 -14
index 9ff8b83c19a3..5a8080c71b04 100644
@@ -1322,15 +1322,6 @@ static void iommu_enable_pci_caps(struct device_domain_info *info)
  		return;
 
  	pdev = to_pci_dev(info->dev);
-
-	/* The PCIe spec, in its wisdom, declares that the behaviour of
-	   the device if you enable PASID support after ATS support is
-	   undefined. So always enable PASID support on devices which
-	   have it, even if we can't yet know if we're ever going to
-	   use it. */
-	if (info->pasid_supported && !pci_enable_pasid(pdev, info->pasid_supported & ~1))
-		info->pasid_enabled = 1;
-
  	if (info->ats_supported && pci_ats_page_aligned(pdev) &&
  	    !pci_enable_ats(pdev, VTD_PAGE_SHIFT)) {
  		info->ats_enabled = 1;
@@ -1352,11 +1343,6 @@ static void iommu_disable_pci_caps(struct device_domain_info *info)
  		info->ats_enabled = 0;
  		domain_update_iotlb(info->domain);
  	}
-
-	if (info->pasid_enabled) {
-		pci_disable_pasid(pdev);
-		info->pasid_enabled = 0;
-	}
  }
 
  static void intel_flush_iotlb_all(struct iommu_domain *domain)
@@ -4110,6 +4096,16 @@ static struct iommu_device *intel_iommu_probe_device(struct device *dev)
  		}
  	}
 
+	/*
+	 * The PCIe spec, in its wisdom, declares that the behaviour of the
+	 * device is undefined if you enable PASID support after ATS support.
+	 * So always enable PASID support on devices which have it, even if
+	 * we can't yet know if we're ever going to use it.
+	 */
+	if (info->pasid_supported &&
+	    !pci_enable_pasid(pdev, info->pasid_supported & ~1))
+		info->pasid_enabled = 1;
+
  	intel_iommu_debugfs_create_dev(info);
 
  	return &iommu->iommu;

drivers/iommu/intel/iommu.c static void intel_iommu_release_device(struct device *dev) · line 4129

Quoted 6 lines
drivers/iommu/intel/iommu.c +3 -0
@@ -4128,6 +4124,9 @@ static void intel_iommu_release_device(struct device *dev)
  	struct device_domain_info *info = dev_iommu_priv_get(dev);
  	struct intel_iommu *iommu = info->iommu;
 
+	if (info->pasid_enabled)
+		pci_disable_pasid(to_pci_dev(dev));
+

would it make sense to move this behind the intel_iommu_debugfs_remove_dev(info)? This seems to mirror the order of the intel_iommu_probe_device(). Or you may set info->pasid_enabled to 0 in case of any code uses it before info is freed if keeping this order. Otherwise, lgtm. thanks for the quick action. :)

Quoted 3 lines
drivers/iommu/intel/iommu.c +0 -0
  	mutex_lock(&iommu->iopf_lock);
  	if (dev_is_pci(dev) && pci_ats_supported(to_pci_dev(dev)))
  		device_rbtree_remove(info);
Signature · 2 lines
Regards,
Yi Liu
author maintainer
#16 lore raw
Quoted 70 lines · Yi Liu
Quoted 69 lines · Lu Baolu

Currently, PCI PASID is enabled alongside PCI ATS when an iommu domain is attached to the device and disabled when the device transitions to block translation mode. This approach is inappropriate as PCI PASID is a device feature independent of the type of the attached domain.

Enable PCI PASID during the IOMMU device probe and disables it during the release path.

---
  drivers/iommu/intel/iommu.c | 27 +++++++++++++--------------

  1 file changed, 13 insertions(+), 14 deletions(-)
drivers/iommu/intel/iommu.c +0 -9
index 9ff8b83c19a3..5a8080c71b04 100644
@@ -1322,15 +1322,6 @@ static void iommu_enable_pci_caps(struct 
 device_domain_info *info)
           return;
       pdev = to_pci_dev(info->dev);
-
-    /* The PCIe spec, in its wisdom, declares that the behaviour of
-       the device if you enable PASID support after ATS support is
-       undefined. So always enable PASID support on devices which
-       have it, even if we can't yet know if we're ever going to
-       use it. */
-    if (info->pasid_supported && !pci_enable_pasid(pdev, 
 info->pasid_supported & ~1))
-        info->pasid_enabled = 1;
-

      if (info->ats_supported && pci_ats_page_aligned(pdev) && !pci_enable_ats(pdev, VTD_PAGE_SHIFT)) { info->ats_enabled = 1;

drivers/iommu/intel/iommu.c +3 -5
@@ -1352,11 +1343,6 @@ static void iommu_disable_pci_caps(struct 
 device_domain_info *info)
           info->ats_enabled = 0;
           domain_update_iotlb(info->domain);
       }
-
-    if (info->pasid_enabled) {
-        pci_disable_pasid(pdev);
-        info->pasid_enabled = 0;
-    }
   }
   static void intel_flush_iotlb_all(struct iommu_domain *domain)
@@ -4110,6 +4096,16 @@ static struct iommu_device 
 *intel_iommu_probe_device(struct device *dev)
           }
       }
+    /*
+     * The PCIe spec, in its wisdom, declares that the behaviour of the
+     * device is undefined if you enable PASID support after ATS 

support.
+     * So always enable PASID support on devices which have it, even if +     * we can't yet know if we're ever going to use it.
+     */
+    if (info->pasid_supported &&
+        !pci_enable_pasid(pdev, info->pasid_supported & ~1)) +        info->pasid_enabled = 1;
+
      intel_iommu_debugfs_create_dev(info);
      return &iommu->iommu;

drivers/iommu/intel/iommu.c +1 -0
@@ -4128,6 +4124,9 @@ static void intel_iommu_release_device(struct 
 device *dev)
       struct device_domain_info *info = dev_iommu_priv_get(dev);
       struct intel_iommu *iommu = info->iommu;
+    if (info->pasid_enabled)
Quoted 8 lines

drivers/iommu/intel/iommu.c static void intel_iommu_release_device(struct · line 4129

Quoted 2 lines
drivers/iommu/intel/iommu.c +2 -0
+        pci_disable_pasid(to_pci_dev(dev));
+

would it make sense to move this behind the intel_iommu_debugfs_remove_dev(info)? This seems to mirror the order of the intel_iommu_probe_device(). Or you may set info->pasid_enabled to 0 in case of any code uses it before info is freed if keeping this order. Otherwise, lgtm. thanks for the quick action. 🙂

The info->pasid_enabled change should not impact the behavior of intel_iommu_debugfs_remove_dev(), and I didn't find any issue during my test.

Anyway, to make it more consistent with previous behavior, maybe I could move the part where we turn on/off pasid to the end of the probe and the start of the release.

Additional change likes below?

drivers/iommu/intel/iommu.c +2 -0
index 5a8080c71b04..76b317f1d1de 100644
@@ -4096,6 +4096,8 @@ static struct iommu_device *intel_iommu_probe_device(struct device *dev)
                }
        }
 
+       intel_iommu_debugfs_create_dev(info);
+
        /*
         * The PCIe spec, in its wisdom, declares that the behaviour of the
         * device is undefined if you enable PASID support after ATS 

support.
@@ -4106,8 +4108,6 @@ static struct iommu_device *intel_iommu_probe_device(struct device *dev)

            !pci_enable_pasid(pdev, info->pasid_supported & ~1))
                info->pasid_enabled = 1;

- intel_iommu_debugfs_create_dev(info);
-

        return &iommu->iommu;
 free_table:
        intel_pasid_free_table(dev);

Thanks,
baolu

#17 lore raw
Quoted 79 lines · Baolu Lu
Quoted 69 lines · Yi Liu
Quoted 68 lines · Lu Baolu

Currently, PCI PASID is enabled alongside PCI ATS when an iommu domain is attached to the device and disabled when the device transitions to block translation mode. This approach is inappropriate as PCI PASID is a device feature independent of the type of the attached domain.

Enable PCI PASID during the IOMMU device probe and disables it during the release path.

---
  drivers/iommu/intel/iommu.c | 27 +++++++++++++--------------

  1 file changed, 13 insertions(+), 14 deletions(-)
drivers/iommu/intel/iommu.c +0 -9
index 9ff8b83c19a3..5a8080c71b04 100644
@@ -1322,15 +1322,6 @@ static void iommu_enable_pci_caps(struct 
 device_domain_info *info)
           return;
       pdev = to_pci_dev(info->dev);
-
-    /* The PCIe spec, in its wisdom, declares that the behaviour of
-       the device if you enable PASID support after ATS support is
-       undefined. So always enable PASID support on devices which
-       have it, even if we can't yet know if we're ever going to
-       use it. */
-    if (info->pasid_supported && !pci_enable_pasid(pdev, 
 info->pasid_supported & ~1))
-        info->pasid_enabled = 1;
-

      if (info->ats_supported && pci_ats_page_aligned(pdev) && !pci_enable_ats(pdev, VTD_PAGE_SHIFT)) { info->ats_enabled = 1;

drivers/iommu/intel/iommu.c +11 -5
@@ -1352,11 +1343,6 @@ static void iommu_disable_pci_caps(struct 
 device_domain_info *info)
           info->ats_enabled = 0;
           domain_update_iotlb(info->domain);
       }
-
-    if (info->pasid_enabled) {
-        pci_disable_pasid(pdev);
-        info->pasid_enabled = 0;
-    }
   }
   static void intel_flush_iotlb_all(struct iommu_domain *domain)
@@ -4110,6 +4096,16 @@ static struct iommu_device 
 *intel_iommu_probe_device(struct device *dev)
           }
       }
+    /*
+     * The PCIe spec, in its wisdom, declares that the behaviour of the
+     * device is undefined if you enable PASID support after ATS support.
+     * So always enable PASID support on devices which have it, even if
+     * we can't yet know if we're ever going to use it.
+     */
+    if (info->pasid_supported &&
+        !pci_enable_pasid(pdev, info->pasid_supported & ~1))
+        info->pasid_enabled = 1;
+
       intel_iommu_debugfs_create_dev(info);
       return &iommu->iommu;
@@ -4128,6 +4124,9 @@ static void intel_iommu_release_device(struct 
 device *dev)
       struct device_domain_info *info = dev_iommu_priv_get(dev);
       struct intel_iommu *iommu = info->iommu;
+    if (info->pasid_enabled)
Quoted 8 lines

drivers/iommu/intel/iommu.c static void intel_iommu_release_device(struct · line 4129

Quoted 2 lines
drivers/iommu/intel/iommu.c +2 -0
+        pci_disable_pasid(to_pci_dev(dev));
+

would it make sense to move this behind the intel_iommu_debugfs_remove_dev(info)? This seems to mirror the order of the intel_iommu_probe_device(). Or you may set info->pasid_enabled to 0 in case of any code uses it before info is freed if keeping this order. Otherwise, lgtm. thanks for the quick action. 🙂

Quoted 7 lines

The info->pasid_enabled change should not impact the behavior of intel_iommu_debugfs_remove_dev(), and I didn't find any issue during my test.

Anyway, to make it more consistent with previous behavior, maybe I could move the part where we turn on/off pasid to the end of the probe and the start of the release.

yeah, this looks ok. And you may consider to clear info->pasid_enabled when it's disabled. I guess it does not affect device_rbtree_remove(), intel_pasid_teardown_sm_context(), intel_pasid_free_table() nor the intel_iommu_debugfs_remove_dev(), but good to clear it as it to reflect the status.:)

Quoted 29 lines

Additional change likes below?

drivers/iommu/intel/iommu.c +2 -0
index 5a8080c71b04..76b317f1d1de 100644
@@ -4096,6 +4096,8 @@ static struct iommu_device 
 *intel_iommu_probe_device(struct device *dev)
                 }
         }
 
+       intel_iommu_debugfs_create_dev(info);
+
         /*
          * The PCIe spec, in its wisdom, declares that the behaviour of the

         * device is undefined if you enable PASID support after ATS support.

drivers/iommu/intel/iommu.c +0 -2
@@ -4106,8 +4108,6 @@ static struct iommu_device 
 *intel_iommu_probe_device(struct device *dev)
             !pci_enable_pasid(pdev, info->pasid_supported & ~1))
                 info->pasid_enabled = 1;
 
-       intel_iommu_debugfs_create_dev(info);
-
         return &iommu->iommu;
  free_table:

        intel_pasid_free_table(dev);

Thanks,
baolu

Signature · 2 lines
Regards,
Yi Liu
author maintainer
#18 lore raw
Quoted 91 lines · Yi Liu
Quoted 82 lines · Baolu Lu
Quoted 72 lines · Yi Liu
Quoted 71 lines · Lu Baolu

Currently, PCI PASID is enabled alongside PCI ATS when an iommu domain is attached to the device and disabled when the device transitions to block translation mode. This approach is inappropriate as PCI PASID is a device feature independent of the type of the attached domain.

Enable PCI PASID during the IOMMU device probe and disables it during the release path.

---
  drivers/iommu/intel/iommu.c | 27 +++++++++++++--------------

  1 file changed, 13 insertions(+), 14 deletions(-)
drivers/iommu/intel/iommu.c +0 -9
index 9ff8b83c19a3..5a8080c71b04 100644
@@ -1322,15 +1322,6 @@ static void iommu_enable_pci_caps(struct 
 device_domain_info *info)
           return;
       pdev = to_pci_dev(info->dev);
-
-    /* The PCIe spec, in its wisdom, declares that the behaviour of
-       the device if you enable PASID support after ATS support is
-       undefined. So always enable PASID support on devices which
-       have it, even if we can't yet know if we're ever going to
-       use it. */
-    if (info->pasid_supported && !pci_enable_pasid(pdev, 
 info->pasid_supported & ~1))
-        info->pasid_enabled = 1;
-

      if (info->ats_supported && pci_ats_page_aligned(pdev) && !pci_enable_ats(pdev, VTD_PAGE_SHIFT)) { info->ats_enabled = 1;

drivers/iommu/intel/iommu.c +2 -5
@@ -1352,11 +1343,6 @@ static void iommu_disable_pci_caps(struct 
 device_domain_info *info)
           info->ats_enabled = 0;
           domain_update_iotlb(info->domain);
       }
-
-    if (info->pasid_enabled) {
-        pci_disable_pasid(pdev);
-        info->pasid_enabled = 0;
-    }
   }
   static void intel_flush_iotlb_all(struct iommu_domain *domain)
@@ -4110,6 +4096,16 @@ static struct iommu_device 
 *intel_iommu_probe_device(struct device *dev)
           }
       }
+    /*
+     * The PCIe spec, in its wisdom, declares that the behaviour of 

the
+     * device is undefined if you enable PASID support after ATS support.
+     * So always enable PASID support on devices which have it, even if
+     * we can't yet know if we're ever going to use it. +     */
+    if (info->pasid_supported &&
+        !pci_enable_pasid(pdev, info->pasid_supported & ~1)) +        info->pasid_enabled = 1;
+
      intel_iommu_debugfs_create_dev(info);
      return &iommu->iommu;

drivers/iommu/intel/iommu.c +1 -0
@@ -4128,6 +4124,9 @@ static void intel_iommu_release_device(struct 
 device *dev)
       struct device_domain_info *info = dev_iommu_priv_get(dev);
       struct intel_iommu *iommu = info->iommu;
+    if (info->pasid_enabled)
Quoted 8 lines

drivers/iommu/intel/iommu.c static void intel_iommu_release_device(struct · line 4129

Quoted 2 lines
drivers/iommu/intel/iommu.c +2 -0
+        pci_disable_pasid(to_pci_dev(dev));
+

would it make sense to move this behind the intel_iommu_debugfs_remove_dev(info)? This seems to mirror the order of the intel_iommu_probe_device(). Or you may set info->pasid_enabled to 0 in case of any code uses it before info is freed if keeping this order. Otherwise, lgtm. thanks for the quick action. 🙂

Quoted 7 lines

The info->pasid_enabled change should not impact the behavior of intel_iommu_debugfs_remove_dev(), and I didn't find any issue during my test.

Anyway, to make it more consistent with previous behavior, maybe I could move the part where we turn on/off pasid to the end of the probe and the start of the release.

Quoted 5 lines

yeah, this looks ok. And you may consider to clear info->pasid_enabled when it's disabled. I guess it does not affect device_rbtree_remove(), intel_pasid_teardown_sm_context(), intel_pasid_free_table() nor the intel_iommu_debugfs_remove_dev(), but good to clear it as it to reflect the status.:)

Done.

Thanks,
baolu