Skip to contentSkip to conversation

intel-iommu: don't disable ATS for device without page aligned request

PATCH In review Jason Wang 3 participants 6 messages Sep 9, 2020 – Sep 14, 2020

Subsystems (3)
lore
Share & tools
lore

Jason Wang

author
#0 lore raw

PATCH 1/1 intel-iommu: don't disable ATS for device without page aligned request

Commit 61363c1474b1 ("iommu/vt-d: Enable ATS only if the device uses page aligned address.") disables ATS for device that can do unaligned page request.

This looks wrong, since the commit log said it's because the page request descriptor doesn't support reporting unaligned request.

A victim is Qemu's virtio-pci which doesn't advertise the page aligned address. Fixing by disable PRI instead of ATS if device doesn't have page aligned request.

Cc Ashok Raj <[email protected]>
Cc Jacob Pan <[email protected]>
Cc Keith Busch <[email protected]>
Cc Kuppuswamy Sathyanarayanan <[email protected]>
---
 drivers/iommu/intel/iommu.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/iommu/intel/iommu.c +2 -1
index e9864e52b0e9..ef5214a8a4dd 100644
@@ -1440,10 +1440,11 @@ static void iommu_enable_dev_iotlb(struct device_domain_info *info)
 
 	if (info->pri_supported &&
 	    (info->pasid_enabled ? pci_prg_resp_pasid_required(pdev) : 1)  &&
+	    pci_ats_page_aligned(pdev) &&
 	    !pci_reset_pri(pdev) && !pci_enable_pri(pdev, 32))
 		info->pri_enabled = 1;
 #endif
-	if (info->ats_supported && pci_ats_page_aligned(pdev) &&
+	if (info->ats_supported &&
 	    !pci_enable_ats(pdev, VTD_PAGE_SHIFT)) {
 		info->ats_enabled = 1;
 		domain_update_iotlb(info->domain);
Signature · 1 line
2.20.1
#1 lore raw

Hi Jason

Quoted 3 lines · Jason Wang

Commit 61363c1474b1 ("iommu/vt-d: Enable ATS only if the device uses page aligned address.") disables ATS for device that can do unaligned page request.

Did you take a look at the PCI specification? Page Aligned Request is in the ATS capability Register.

ATS Capability Register (Offset 0x04h)

bit (5):
Page Aligned Request - If Set, indicates the Untranslated address is always aligned to 4096 byte boundary. Setting this field is recommended. This field permits software to distinguish between implemntations compatible with this specification and those compatible with an earlier version of this specification in which a Requester was permitted to supply anything in bits [11:2].

Quoted 3 lines

This looks wrong, since the commit log said it's because the page request descriptor doesn't support reporting unaligned request.

I don't think you can change the definition from ATS to PRI. Both are orthogonal feature.

Quoted 4 lines

A victim is Qemu's virtio-pci which doesn't advertise the page aligned address. Fixing by disable PRI instead of ATS if device doesn't have page aligned request.

This is a requirement for the Intel IOMMU's.

You say virtio, so is it all emulated device or you talking about some hardware that implemented virtio-pci compliant hw? If you are sure the device actually does comply with the requirement, but just not enumerating the capability, you can maybe work a quirk to overcome that?

Now PRI also has an alignment requirement, and Intel IOMMU's requires that as well. If your device supports SRIOV as well, PASID and PRI are enumerated just on the PF and not the VF. You might want to pay attension to that. We are still working on a solution for that problem.

I don't think this is the right fix for your problem.

Quoted 11 lines
Cc Ashok Raj <[email protected]>
Cc Jacob Pan <[email protected]>
Cc Keith Busch <[email protected]>
Cc Kuppuswamy Sathyanarayanan <[email protected]>
---
 drivers/iommu/intel/iommu.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

author
#2 lore raw

----- Original Message -----

Quoted 12 lines

Hi Jason

Quoted 3 lines · Jason Wang

Commit 61363c1474b1 ("iommu/vt-d: Enable ATS only if the device uses page aligned address.") disables ATS for device that can do unaligned page request.

Did you take a look at the PCI specification? Page Aligned Request is in the ATS capability Register.

ATS Capability Register (Offset 0x04h)

Quoted 7 lines

bit (5):
Page Aligned Request - If Set, indicates the Untranslated address is always aligned to 4096 byte boundary. Setting this field is recommended. This field permits software to distinguish between implemntations compatible with this specification and those compatible with an earlier version of this specification in which a Requester was permitted to supply anything in bits [11:2].

Yes, my understanding is that this is optional not mandatory.

Quoted 7 lines
Quoted 4 lines

This looks wrong, since the commit log said it's because the page request descriptor doesn't support reporting unaligned request.

I don't think you can change the definition from ATS to PRI. Both are orthogonal feature.

I may miss something, here's my understanding is that:

- page request descriptor will only be used when PRS is enabled
- ATS spec allows unaligned request

So any reason for disabling ATS for unaligned request even if PRS is not enabled?

Quoted 6 lines
Quoted 5 lines

A victim is Qemu's virtio-pci which doesn't advertise the page aligned address. Fixing by disable PRI instead of ATS if device doesn't have page aligned request.

Quoted 6 lines

This is a requirement for the Intel IOMMU's.

You say virtio, so is it all emulated device or you talking about some hardware that implemented virtio-pci compliant hw? If you are sure the device actually does comply with the requirement, but just not enumerating the capability, you can maybe work a quirk to overcome that?

So far only emulated devices. But we are helping some vendor to implement virtio hardware so we need to understand the connection between ATS alignment and page request descriptor.

Quoted 5 lines

Now PRI also has an alignment requirement, and Intel IOMMU's requires that as well. If your device supports SRIOV as well, PASID and PRI are enumerated just on the PF and not the VF. You might want to pay attension to that. We are still working on a solution for that problem.

Thanks for the reminding, but it looks to me according to the ATS spec, all PRI message is 4096 byte aligned? E.g lower bites were used for group index etc.

Thanks

Quoted 16 lines

I don't think this is the right fix for your problem.

Quoted 11 lines
Cc Ashok Raj <[email protected]>
Cc Jacob Pan <[email protected]>
Cc Keith Busch <[email protected]>
Cc Kuppuswamy Sathyanarayanan <[email protected]>
---
 drivers/iommu/intel/iommu.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#3 lore raw
Quoted 23 lines · Jason Wang

----- Original Message -----

Quoted 12 lines

Hi Jason

Quoted 3 lines · Jason Wang

Commit 61363c1474b1 ("iommu/vt-d: Enable ATS only if the device uses page aligned address.") disables ATS for device that can do unaligned page request.

Did you take a look at the PCI specification? Page Aligned Request is in the ATS capability Register.

ATS Capability Register (Offset 0x04h)

Quoted 7 lines

bit (5):
Page Aligned Request - If Set, indicates the Untranslated address is always aligned to 4096 byte boundary. Setting this field is recommended. This field permits software to distinguish between implemntations compatible with this specification and those compatible with an earlier version of this specification in which a Requester was permitted to supply anything in bits [11:2].

Quoted 1 line

Yes, my understanding is that this is optional not mandatory.

Correct, but optional on the device side. An IOMMU might *require* this for proper normal operation. Our IOMMU's do not get the low 12 bits. Which is why the spec gives SW a way to detect if the device is compatible for this IOMMU implementation.

Quoted 9 lines
Quoted 8 lines
Quoted 5 lines

This looks wrong, since the commit log said it's because the page request descriptor doesn't support reporting unaligned request.

I don't think you can change the definition from ATS to PRI. Both are orthogonal feature.

Quoted 7 lines

I may miss something, here's my understanding is that:

- page request descriptor will only be used when PRS is enabled
- ATS spec allows unaligned request

So any reason for disabling ATS for unaligned request even if PRS is not enabled?

I think you are getting confused between the 2 different PCIe features.

ATS - Address Translation Services. Used by device to simply request the Host Physical Address for some DMA operation.

When ATS response indicates failed, then the device can request a page-request (PRS this is like a device page-fault), and then IOMMU driver would work with the kernel to fault a page then respond with (Page-response) success/failure. Then the device will send a new ATS to get the new translation.

Quoted 9 lines
Quoted 9 lines
Quoted 6 lines

A victim is Qemu's virtio-pci which doesn't advertise the page aligned address. Fixing by disable PRI instead of ATS if device doesn't have page aligned request.

This is a requirement for the Intel IOMMU's.

Quoted 8 lines
Quoted 4 lines

You say virtio, so is it all emulated device or you talking about some hardware that implemented virtio-pci compliant hw? If you are sure the device actually does comply with the requirement, but just not enumerating the capability, you can maybe work a quirk to overcome that?

So far only emulated devices. But we are helping some vendor to implement virtio hardware so we need to understand the connection between ATS alignment and page request descriptor.

ATS and PRS are 2 separate orthogonal features.
PRS requires ATS, but not the other way around.

Quoted 10 lines
Quoted 6 lines

Now PRI also has an alignment requirement, and Intel IOMMU's requires that as well. If your device supports SRIOV as well, PASID and PRI are enumerated just on the PF and not the VF. You might want to pay attension to that. We are still working on a solution for that problem.

Thanks for the reminding, but it looks to me according to the ATS spec, all PRI message is 4096 byte aligned? E.g lower bites were used for group index etc.

Right, I should have been clear. The issue with PRI is we require responses to have PASID field set. There is another capability on the device that exposes that. pci_prg_resp_pasid_required(). This is required to enable PRI for a device.

Cheers,
Ashok
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

author
#4 lore raw
Quoted 21 lines · Ashok
Quoted 20 lines · Jason Wang

----- Original Message -----

Quoted 18 lines

Hi Jason

Quoted 3 lines · Jason Wang

Commit 61363c1474b1 ("iommu/vt-d: Enable ATS only if the device uses page aligned address.") disables ATS for device that can do unaligned page request.

Did you take a look at the PCI specification? Page Aligned Request is in the ATS capability Register.

ATS Capability Register (Offset 0x04h)

bit (5):
Page Aligned Request - If Set, indicates the Untranslated address is always aligned to 4096 byte boundary. Setting this field is recommended. This field permits software to distinguish between implemntations compatible with this specification and those compatible with an earlier version of this specification in which a Requester was permitted to supply anything in bits [11:2].

Quoted 5 lines
Quoted 1 line

Yes, my understanding is that this is optional not mandatory.

Correct, but optional on the device side. An IOMMU might *require* this for proper normal operation. Our IOMMU's do not get the low 12 bits. Which is why the spec gives SW a way to detect if the device is compatible for this IOMMU implementation.

I see, it's better to clarify this in the spec (or is there already had a section for this?)

Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

This looks wrong, since the commit log said it's because the page request descriptor doesn't support reporting unaligned request.

I don't think you can change the definition from ATS to PRI. Both are orthogonal feature.

Quoted 7 lines

I may miss something, here's my understanding is that:

- page request descriptor will only be used when PRS is enabled
- ATS spec allows unaligned request

So any reason for disabling ATS for unaligned request even if PRS is not enabled?

I think you are getting confused between the 2 different PCIe features.

Quoted 8 lines

ATS - Address Translation Services. Used by device to simply request the Host Physical Address for some DMA operation.

When ATS response indicates failed, then the device can request a page-request (PRS this is like a device page-fault), and then IOMMU driver would work with the kernel to fault a page then respond with (Page-response) success/failure. Then the device will send a new ATS to get the new translation.

Yes, that's my understanding as well. I think what confuses me is the commit log of 61363c1474b1 which ties a PRI queue to ATS features ...

Quoted 21 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

A victim is Qemu's virtio-pci which doesn't advertise the page aligned address. Fixing by disable PRI instead of ATS if device doesn't have page aligned request.

This is a requirement for the Intel IOMMU's.

Quoted 7 lines
Quoted 4 lines

You say virtio, so is it all emulated device or you talking about some hardware that implemented virtio-pci compliant hw? If you are sure the device actually does comply with the requirement, but just not enumerating the capability, you can maybe work a quirk to overcome that?

So far only emulated devices. But we are helping some vendor to implement virtio hardware so we need to understand the connection between ATS alignment and page request descriptor.

ATS and PRS are 2 separate orthogonal features.
PRS requires ATS, but not the other way around.

Quoted 4 lines
Quoted 4 lines

Now PRI also has an alignment requirement, and Intel IOMMU's requires that as well. If your device supports SRIOV as well, PASID and PRI are enumerated just on the PF and not the VF. You might want to pay attension to that. We are still working on a solution for that problem.

Quoted 7 lines
Quoted 3 lines

Thanks for the reminding, but it looks to me according to the ATS spec, all PRI message is 4096 byte aligned? E.g lower bites were used for group index etc.

Right, I should have been clear. The issue with PRI is we require responses to have PASID field set. There is another capability on the device that exposes that. pci_prg_resp_pasid_required(). This is required to enable PRI for a device.

Right. Thanks for the clarification, I will withdraw this patch.

Quoted 4 lines

Cheers,
Ashok

maintainer
#5 lore raw
In reply to Jason Wang

Hi

[This is an automated email]

This commit has been processed because it contains a -stable tag. The stable tag indicates that it's relevant for the following trees: all

The bot has tested the following trees: v5.8.7, v5.4.63, v4.19.143, v4.14.196, v4.9.235, v4.4.235.

v5.8.7: Build OK!
v5.4.63: Failed to apply! Possible dependencies:

    da656a042568 ("iommu/vt-d: Use pci_ats_supported()")

v4.19.143: Failed to apply! Possible dependencies:
1b84778a62ad ("iommu/vt-d: Fix PRI/PASID dependency issue.") 61363c1474b1 ("iommu/vt-d: Enable ATS only if the device uses page aligned address.") d8b859105457 ("iommu/vt-d: Disable ATS support on untrusted devices") da656a042568 ("iommu/vt-d: Use pci_ats_supported()")
fb58fdcd295b ("iommu/vt-d: Do not enable ATS for untrusted devices")

v4.14.196: Failed to apply! Possible dependencies:
1b84778a62ad ("iommu/vt-d: Fix PRI/PASID dependency issue.") 61363c1474b1 ("iommu/vt-d: Enable ATS only if the device uses page aligned address.") cef74409ea79 ("PCI: Add "pci=noats" boot parameter")
d8b859105457 ("iommu/vt-d: Disable ATS support on untrusted devices") da656a042568 ("iommu/vt-d: Use pci_ats_supported()")
fb58fdcd295b ("iommu/vt-d: Do not enable ATS for untrusted devices")

v4.9.235: Failed to apply! Possible dependencies:
1b84778a62ad ("iommu/vt-d: Fix PRI/PASID dependency issue.") 61363c1474b1 ("iommu/vt-d: Enable ATS only if the device uses page aligned address.") cef74409ea79 ("PCI: Add "pci=noats" boot parameter")
d8b859105457 ("iommu/vt-d: Disable ATS support on untrusted devices") da656a042568 ("iommu/vt-d: Use pci_ats_supported()")
fb58fdcd295b ("iommu/vt-d: Do not enable ATS for untrusted devices")

v4.4.235: Failed to apply! Possible dependencies:
0824c5920b16 ("iommu/vt-d: avoid dev iotlb logic for domains with no dev iotlbs") 0cbd4b34cda9 ("xhci: mediatek: support MTK xHCI host controller") 1b84778a62ad ("iommu/vt-d: Fix PRI/PASID dependency issue.") 61363c1474b1 ("iommu/vt-d: Enable ATS only if the device uses page aligned address.") 7e70cbffe236 ("usb: xhci: add a quirk bit for ssic port unused") 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend") cef74409ea79 ("PCI: Add "pci=noats" boot parameter")
d8b859105457 ("iommu/vt-d: Disable ATS support on untrusted devices") da656a042568 ("iommu/vt-d: Use pci_ats_supported()")
fb58fdcd295b ("iommu/vt-d: Do not enable ATS for untrusted devices")

NOTE: The patch will not be queued to stable trees until it is upstream.

How should we proceed with this patch?

Signature · 2 lines
Thanks
Sasha