iommu/amd: Fix default domain selection for PASID-capable devices
Subsystems (3)
Vasant Hegde
maintainercover iommu/amd: Fix default domain selection for PASID-capable devices
This series addresses a long-standing issue in the default domain selection path of the AMD IOMMU driver.
Currently, the driver unconditionally forces an identity domain on all devices that support PASID (Process Address Space ID).
This series introduces a more targeted approach: instead of applying identity domain assignment globally to all PASID-capable devices, quirks are added only for specific GPUs. All remaining PASID-capable devices are now allowed to boot with a paging domain, enabling proper use of page tables (v2 page table).
In addition, this series adds an ATS (Address Translation Service) quirk for the "Radeon GPU WX 4100" to ensure correct operation when booted with a paging domain.
Vasant Hegde (2):
PCI: Mark Radeon Pro WX 4100 ATS as broken iommu/amd: Force identity mode for selected GPUs only
drivers/iommu/amd/iommu.c | 56 +++++++++++++++++++++++++++++++-------- drivers/pci/quirks.c | 2 ++ 2 files changed, 47 insertions(+), 11 deletions(-)
Signature · 1 line
2.31.1
Vasant Hegde
maintainerPATCH 1/2 PCI: Mark Radeon Pro WX 4100 ATS as broken
If ATS is enabled then IOMMU logs below error and GPU fails to initialize.
[ 103.687137] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:c4:00.0 address=0x1080006060]
--- drivers/pci/quirks.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/pci/quirks.c +2 -0
@@ -5686,6 +5686,8 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATI, 0x7340, quirk_amd_harvest_no_ats);DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_ATI ,0x7341 , quirk_amd_harvest_no_ats);DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_ATI ,0x7347 , quirk_amd_harvest_no_ats);DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_ATI ,0x734f , quirk_amd_harvest_no_ats); +/* Radeon Pro WX 4100 */ +DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_ATI ,0x67e3 , quirk_amd_harvest_no_ats);/* AMD Raven platform iGPU */ DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_ATI ,0x15d8 , quirk_amd_harvest_no_ats);
Signature · 1 line
2.31.1
[email protected]
AI reviewRe: [PATCH 1/2] PCI: Mark Radeon Pro WX 4100 ATS as broken
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 9 lines
If ATS is enabled then IOMMU logs below error and GPU fails to initialize.
[ 103.687137] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:c4:00.0 address=0x1080006060]Suggested-by Alex Deucher <[email protected]>Cc Bjorn Helgaas <[email protected]>Signed-off-by Vasant Hegde <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Re: [PATCH 1/2] PCI: Mark Radeon Pro WX 4100 ATS as broken
In reply to Vasant HegdeQuoted 4 lines · Vasant Hegde
If ATS is enabled then IOMMU logs below error and GPU fails to initialize.
[ 103.687137] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:c4:00.0 address=0x1080006060]
Is this a published erratum?
Is there enough benefit to using ATS to make it worthwhile? It would have to be a pretty substantial benefit to outweigh the end-user frustration of defects like this.
If ATS isn't going to be validated by vendors, maybe we should disable it on all AMD/ATI devices? Or maybe only the GPUs?
Quoted 24 lines
Suggested-by Alex Deucher <[email protected]>Cc Bjorn Helgaas <[email protected]>Signed-off-by Vasant Hegde <[email protected]>--- drivers/pci/quirks.c | 2 ++ 1 file changed, 2 insertions(+)drivers/pci/quirks.c +2
-0@@ -5686,6 +5686,8 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATI, 0x7340, quirk_amd_harvest_no_ats);DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_ATI ,0x7341 , quirk_amd_harvest_no_ats);DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_ATI ,0x7347 , quirk_amd_harvest_no_ats);DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_ATI ,0x734f , quirk_amd_harvest_no_ats); +/* Radeon Pro WX 4100 */ +DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_ATI ,0x67e3 , quirk_amd_harvest_no_ats);/* AMD Raven platform iGPU */ DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_ATI ,0x15d8 , quirk_amd_harvest_no_ats);Signature · 1 line
2.31.1
Re: [PATCH 1/2] PCI: Mark Radeon Pro WX 4100 ATS as broken
Quoted 6 lines · Bjorn Helgaas
Quoted 4 lines · Vasant Hegde
If ATS is enabled then IOMMU logs below error and GPU fails to initialize.
[ 103.687137] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:c4:00.0 address=0x1080006060]
Quoted 8 lines
Is this a published erratum?
Is there enough benefit to using ATS to make it worthwhile? It would have to be a pretty substantial benefit to outweigh the end-user frustration of defects like this.
If ATS isn't going to be validated by vendors, maybe we should disable it on all AMD/ATI devices? Or maybe only the GPUs?
Isn't that a pretty big performance impact to have ATS disabled?
At least I don't think I would want a blanket policy without data supporting it's a good idea.
Quoted 25 lines
Quoted 25 lines
Suggested-by Alex Deucher <[email protected]>Cc Bjorn Helgaas <[email protected]>Signed-off-by Vasant Hegde <[email protected]>--- drivers/pci/quirks.c | 2 ++ 1 file changed, 2 insertions(+)drivers/pci/quirks.c +2
-0@@ -5686,6 +5686,8 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATI, 0x7340, quirk_amd_harvest_no_ats);DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_ATI ,0x7341 , quirk_amd_harvest_no_ats);DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_ATI ,0x7347 , quirk_amd_harvest_no_ats);DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_ATI ,0x734f , quirk_amd_harvest_no_ats); +/* Radeon Pro WX 4100 */ +DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_ATI ,0x67e3 , quirk_amd_harvest_no_ats);/* AMD Raven platform iGPU */ DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_ATI ,0x15d8 , quirk_amd_harvest_no_ats);Signature · 1 line
2.31.1
Vasant Hegde
maintainerRe: [PATCH 1/2] PCI: Mark Radeon Pro WX 4100 ATS as broken
Quoted 16 lines · Mario Limonciello
Quoted 13 lines · Bjorn Helgaas
Quoted 5 lines · Vasant Hegde
If ATS is enabled then IOMMU logs below error and GPU fails to initialize.
[ 103.687137] iommu ivhd0: AMD-Vi: Event logged [
IOTLB_INV_TIMEOUTdevice=0000:c4:00.0 address=0x1080006060]Is this a published erratum?
Is there enough benefit to using ATS to make it worthwhile? It would have to be a pretty substantial benefit to outweigh the end-user frustration of defects like this.
Quoted 7 lines
Quoted 2 lines
If ATS isn't going to be validated by vendors, maybe we should disable it on all AMD/ATI devices? Or maybe only the GPUs?
Isn't that a pretty big performance impact to have ATS disabled?
At least I don't think I would want a blanket policy without data supporting it's a good idea.
Ack. I will find this GPU internally and will run some more experiement and get back to you.
-Vasant
Vasant Hegde
maintainerPATCH 2/2 iommu/amd: Force identity mode for selected GPUs only
In reply to Vasant HegdeCertain AMD GPU's must always be in identity mode. Currently its enforced using PASID check. It worked fine as most GPU's has PASID feature. But this means, identity mode enforcement is done for all PASID capable devices.
Previously it made sense as domain allocation API (iommu_ops->domain_alloc()) was just passing domain type. So it couldn't check device capability and decide best suited page table type (v1 or v2). With recent enhancement to driver code, it uses domain_alloc_paging_flags() ops for all paging mode domain allocation. This can check device/flags and allocate best suited page table (v1 or v2). Hence fix amd_iommu_def_domain_type() to force identity mapping for selected GPUs only.
With this change system booted with DMA translation mode will select:
* Guest (v2) page table for PASID capable device
* Host (v1) page table for non-PASID capable device
--- drivers/iommu/amd/iommu.c | 56 +++++++++++++++++++++++++++++++-------- 1 file changed, 45 insertions(+), 11 deletions(-)
drivers/iommu/amd/iommu.c +45 -11
@@ -3112,6 +3112,32 @@ static bool amd_iommu_is_attach_deferred(struct device *dev)return dev_data->defer_attach; } +static bool quirks_force_identity_mapping (struct pci_dev *pdev) +{ +struct pci_dev *root_port; +int class = pdev->class >>8 ; + +/* AMD GPU vendor ID */ +if (pdev->vendor !=PCI_VENDOR_ID_ATI ) +return false; + +/* GPU class */ +if (class !=PCI_CLASS_DISPLAY_VGA && + class !=PCI_CLASS_DISPLAY_OTHER ) +return false; + + root_port =pcie_find_root_port (pdev); +if (!root_port) +return false; + +/* If bridge vendor is not ATI then its APU and force IDENTITY mode */ +if (root_port->vendor !=PCI_VENDOR_ID_ATI ) +return true; + +/* Rest all are dGPUs and works fine with DMA mode */ +return false; +} +static int amd_iommu_def_domain_type (struct device *dev) {struct iommu_dev_data *dev_data;
@@ -3120,20 +3146,28 @@ static int amd_iommu_def_domain_type(struct device *dev)if (!dev_data)return 0 ; +if (!dev_is_pci (dev)) +return 0 ; +/* Always use DMA domain for untrusted device */ -if (dev_is_pci (dev) &&to_pci_dev (dev)->untrusted) +if (to_pci_dev (dev)->untrusted)return IOMMU_DOMAIN_DMA ; -/* -* Do not identity map IOMMUv2 capable devices when: -* - memory encryption is active, because some of those devices -* (AMD GPUs) don't have the encryption bit in their DMA-mask -* and require remapping. -* - SNP is enabled, because it prohibits DTE[Mode]=0. -*/ -if (pdev_pasid_supported (dev_data) && - !cc_platform_has (CC_ATTR_MEM_ENCRYPT ) && - !amd_iommu_snp_en) { +/* Apply device specific quirks */ +if (quirks_force_identity_mapping (to_pci_dev (dev))) { + +/* +* When memory encryption is active, some of these devices +* don't have the encryption bit in their DMA-mask and +* require remapping. +*/ +if (cc_platform_has (CC_ATTR_MEM_ENCRYPT )) +return 0 ; + +/* DTE[Mode]=0 is prohibited when SNP is enabled */ +if (amd_iommu_snp_en) +return 0 ; +return IOMMU_DOMAIN_IDENTITY ; }
Signature · 1 line
2.31.1
[email protected]
AI reviewRe: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] quirks_force_identity_mapping() incorrectly identifies AMD discrete GPUs (dGPUs) as APUs…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] quirks_force_identity_mapping() incorrectly identifies AMD discrete GPUs (dGPUs) as APUs, inadvertently forcing them into IOMMU IDENTITY mapping mode instead of the intended DMA mode. --
--- Patch [2]: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only --- commit 43e3d4caff8faca1297d432e0146e34c6ec99227
Author: Vasant Hegde <[email protected]>
Subject: iommu/amd: Force identity mode for selected GPUs only
This commit attempts to enforce identity mapping mode specifically for APUs while keeping discrete GPUs in DMA mode by checking the upstream bridge vendor ID.
Quoted 4 lines
drivers/iommu/amd/iommu.c +0
-0
[ ... ]
Quoted 11 lines
drivers/iommu/amd/iommu.c +7
-0@@ -3122,6 +3122,32 @@ static bool amd_iommu_is_attach_deferred(struct device *dev)return dev_data->defer_attach; } +static bool quirks_force_identity_mapping (struct pci_dev *pdev) +{ +struct pci_dev *root_port; +int class = pdev->class >>8 ; + +/* AMD GPU vendor ID */ +if (pdev->vendor !=PCI_VENDOR_ID_ATI )
drivers/iommu/amd/iommu.c static bool amd_iommu_is_attach_deferred(struct device *dev) · line 3141
Quoted 10 lines
drivers/iommu/amd/iommu.c +10
-0+return false; + +/* GPU class */ +if (class !=PCI_CLASS_DISPLAY_VGA && + class !=PCI_CLASS_DISPLAY_OTHER ) +return false; + + root_port =pcie_find_root_port (pdev); +if (!root_port) +return false;
[Severity: High]
If an APU is implemented as a Root Complex Integrated Endpoint (RCiEP), pcie_find_root_port() will return NULL. Does this cause the function to return false early and leave the APU in DMA mode, which seems to be the opposite of the intended behavior?
drivers/iommu/amd/iommu.c static bool amd_iommu_is_attach_deferred(struct device *dev) · line 3135
Quoted 4 lines
drivers/iommu/amd/iommu.c +4
-0+ +/* If bridge vendor is not ATI then its APU and force IDENTITY mode */ +if (root_port->vendor !=PCI_VENDOR_ID_ATI ) +return true;
[Severity: High]
Will this correctly distinguish between discrete GPUs and APUs?
Since pcie_find_root_port() traces up to the host's root port, the vendor ID will typically match the system chipset (like AMD 0x1022 or Intel 0x8086) rather than ATI 0x1002. Doesn't this mean the condition root_port->vendor != PCI_VENDOR_ID_ATI evaluates to true for discrete GPUs, forcing them into IDENTITY mode instead of the intended DMA mode?
Quoted 4 lines
drivers/iommu/amd/iommu.c +4
-0+ +/* Rest all are dGPUs and works fine with DMA mode */ +return false; +}
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
In reply to Vasant HegdeQuoted 44 lines · Vasant Hegde
Certain AMD GPU's must always be in identity mode. Currently its enforced using PASID check. It worked fine as most GPU's has PASID feature. But this means, identity mode enforcement is done for all PASID capable devices.
Previously it made sense as domain allocation API (
iommu_ops->domain_alloc()) was just passing domain type. So it couldn't check device capability and decide best suited page table type (v1 or v2). With recent enhancement to driver code, it usesdomain_alloc_paging_flags()ops for all paging mode domain allocation. This can check device/flags and allocate best suited page table (v1 or v2). Hence fixamd_iommu_def_domain_type()to force identity mapping for selected GPUs only.With this change system booted with DMA translation mode will select:
* Guest (v2) page table for PASID capable device
* Host (v1) page table for non-PASID capable deviceCc Alex Deucher <[email protected]>Cc Mario Limonciello <[email protected]>Signed-off-by Vasant Hegde <[email protected]>Tested-by Amandeep Kaur Longia <[email protected]>--- drivers/iommu/amd/iommu.c | 56 +++++++++++++++++++++++++++++++-------- 1 file changed, 45 insertions(+), 11 deletions(-)drivers/iommu/amd/iommu.c +11
-0@@ -3112,6 +3112,32 @@ static bool amd_iommu_is_attach_deferred(struct device *dev)return dev_data->defer_attach; } +static bool quirks_force_identity_mapping (struct pci_dev *pdev) +{ +struct pci_dev *root_port; +int class = pdev->class >>8 ; + +/* AMD GPU vendor ID */ +if (pdev->vendor !=PCI_VENDOR_ID_ATI ) +return false; + +/* GPU class */ + if (class != PCI_CLASS_DISPLAY_VGA &&
drivers/iommu/amd/iommu.c static bool amd_iommu_is_attach_deferred(struct device *dev) · line 3135
Quoted 10 lines
drivers/iommu/amd/iommu.c +10
-0+ class != PCI_CLASS_DISPLAY_OTHER) +return false; + + root_port =pcie_find_root_port (pdev); +if (!root_port) +return false; + +/* If bridge vendor is not ATI then its APU and force IDENTITY mode */ +if (root_port->vendor !=PCI_VENDOR_ID_ATI ) +return true;
Hi Vasant,
For a discrete GPU, pcie_find_root_port() may walk up to the system root port. On an AMD platform that root port is vendored PCI_VENDOR_ID_AMD, which is not ATI. so this device will be forced to identity.
Is that a valid scenario?
-Ankit
Quoted 51 lines
drivers/iommu/amd/iommu.c +24
-11+ +/* Rest all are dGPUs and works fine with DMA mode */ +return false; +} +static int amd_iommu_def_domain_type (struct device *dev) {struct iommu_dev_data *dev_data;@@ -3120,20 +3146,28 @@ static int amd_iommu_def_domain_type(struct device *dev)if (!dev_data)return 0 ; +if (!dev_is_pci (dev)) +return 0 ; +/* Always use DMA domain for untrusted device */ -if (dev_is_pci (dev) &&to_pci_dev (dev)->untrusted) +if (to_pci_dev (dev)->untrusted)return IOMMU_DOMAIN_DMA ; -/* -* Do not identity map IOMMUv2 capable devices when: -* - memory encryption is active, because some of those devices -* (AMD GPUs) don't have the encryption bit in their DMA-mask -* and require remapping. -* - SNP is enabled, because it prohibits DTE[Mode]=0. -*/ -if (pdev_pasid_supported (dev_data) && - !cc_platform_has (CC_ATTR_MEM_ENCRYPT ) && - !amd_iommu_snp_en) { +/* Apply device specific quirks */ +if (quirks_force_identity_mapping (to_pci_dev (dev))) { + +/* +* When memory encryption is active, some of these devices +* don't have the encryption bit in their DMA-mask and +* require remapping. +*/ +if (cc_platform_has (CC_ATTR_MEM_ENCRYPT )) +return 0 ; + +/* DTE[Mode]=0 is prohibited when SNP is enabled */ +if (amd_iommu_snp_en) +return 0 ; +return IOMMU_DOMAIN_IDENTITY ; }Signature · 1 line
2.31.1
Vasant Hegde
maintainerRe: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
Ankit,
Quoted 58 lines · Ankit Soni
Quoted 44 lines · Vasant Hegde
Certain AMD GPU's must always be in identity mode. Currently its enforced using PASID check. It worked fine as most GPU's has PASID feature. But this means, identity mode enforcement is done for all PASID capable devices.
Previously it made sense as domain allocation API (
iommu_ops->domain_alloc()) was just passing domain type. So it couldn't check device capability and decide best suited page table type (v1 or v2). With recent enhancement to driver code, it usesdomain_alloc_paging_flags()ops for all paging mode domain allocation. This can check device/flags and allocate best suited page table (v1 or v2). Hence fixamd_iommu_def_domain_type()to force identity mapping for selected GPUs only.With this change system booted with DMA translation mode will select:
* Guest (v2) page table for PASID capable device
* Host (v1) page table for non-PASID capable deviceCc Alex Deucher <[email protected]>Cc Mario Limonciello <[email protected]>Signed-off-by Vasant Hegde <[email protected]>Tested-by Amandeep Kaur Longia <[email protected]>--- drivers/iommu/amd/iommu.c | 56 +++++++++++++++++++++++++++++++-------- 1 file changed, 45 insertions(+), 11 deletions(-)drivers/iommu/amd/iommu.c +11
-0@@ -3112,6 +3112,32 @@ static bool amd_iommu_is_attach_deferred(struct device *dev)return dev_data->defer_attach; } +static bool quirks_force_identity_mapping (struct pci_dev *pdev) +{ +struct pci_dev *root_port; +int class = pdev->class >>8 ; + +/* AMD GPU vendor ID */ +if (pdev->vendor !=PCI_VENDOR_ID_ATI ) +return false; + +/* GPU class */ + if (class != PCI_CLASS_DISPLAY_VGA &&drivers/iommu/amd/iommu.c static bool amd_iommu_is_attach_deferred(struct device *dev) · line 3135
Quoted 10 lines
drivers/iommu/amd/iommu.c +10
-0+ class != PCI_CLASS_DISPLAY_OTHER) +return false; + + root_port =pcie_find_root_port (pdev); +if (!root_port) +return false; + +/* If bridge vendor is not ATI then its APU and force IDENTITY mode */ +if (root_port->vendor !=PCI_VENDOR_ID_ATI ) +return true;Hi Vasant,
Quoted 2 lines
For a discrete GPU,
pcie_find_root_port()may walk up to the system root port. On an AMD platform that root port is vendoredPCI_VENDOR_ID_AMD, which is not ATI.
We want to filter based on vendor as ATI only. As Mario pointed out, logic got inverted. Will fix it in v2.
-Vasant
Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
In reply to Vasant HegdeQuoted 3 lines · Vasant Hegde
Certain AMD GPU's must always be in identity mode. Currently its enforced using PASID check. It worked fine as most GPU's has PASID feature. But this means, identity mode enforcement is done for all PASID capable devices.
I think it would be useful to know something about *why* these devices require identity mode. And what happens without identity mode, i.e., is there a user-visible symptom that happens when the wrong mode is used?
Since the code doesn't test any feature bits, I assume it's because these devices have some hardware defect?
s/GPU's/GPUs/ (twice)
s/Currently its/Currently it's/
s/has PASID/have PASID/
Quoted 38 lines
Previously it made sense as domain allocation API (
iommu_ops->domain_alloc()) was just passing domain type. So it couldn't check device capability and decide best suited page table type (v1 or v2). With recent enhancement to driver code, it usesdomain_alloc_paging_flags()ops for all paging mode domain allocation. This can check device/flags and allocate best suited page table (v1 or v2). Hence fixamd_iommu_def_domain_type()to force identity mapping for selected GPUs only.With this change system booted with DMA translation mode will select:
* Guest (v2) page table for PASID capable device
* Host (v1) page table for non-PASID capable deviceCc Alex Deucher <[email protected]>Cc Mario Limonciello <[email protected]>Signed-off-by Vasant Hegde <[email protected]>Tested-by Amandeep Kaur Longia <[email protected]>--- drivers/iommu/amd/iommu.c | 56 +++++++++++++++++++++++++++++++-------- 1 file changed, 45 insertions(+), 11 deletions(-)drivers/iommu/amd/iommu.c +9
-0@@ -3112,6 +3112,32 @@ static bool amd_iommu_is_attach_deferred(struct device *dev)return dev_data->defer_attach; } +static bool quirks_force_identity_mapping (struct pci_dev *pdev) +{ +struct pci_dev *root_port; +int class = pdev->class >>8 ; + +/* AMD GPU vendor ID */ +if (pdev->vendor !=PCI_VENDOR_ID_ATI ) +return false; +
drivers/iommu/amd/iommu.c static bool amd_iommu_is_attach_deferred(struct device *dev) · line 3133
Quoted 10 lines
drivers/iommu/amd/iommu.c +10
-0+/* GPU class */ +if (class !=PCI_CLASS_DISPLAY_VGA && + class !=PCI_CLASS_DISPLAY_OTHER ) +return false; + + root_port =pcie_find_root_port (pdev); +if (!root_port) +return false; + +/* If bridge vendor is not ATI then its APU and force IDENTITY mode */
s/its/it's/ "it's" == "it is"; "its" shows ownership
Quoted 11 lines
drivers/iommu/amd/iommu.c +7
-0+if (root_port->vendor !=PCI_VENDOR_ID_ATI ) +return true; + +/* Rest all are dGPUs and works fine with DMA mode */ +return false; +} +static int amd_iommu_def_domain_type (struct device *dev) {struct iommu_dev_data *dev_data;
drivers/iommu/amd/iommu.c static int amd_iommu_def_domain_type(struct device *dev) · line 3154
Quoted 10 lines
drivers/iommu/amd/iommu.c +4
-1@@ -3120,20 +3146,28 @@ static int amd_iommu_def_domain_type(struct device *dev)if (!dev_data)return 0 ; +if (!dev_is_pci (dev)) +return 0 ; +/* Always use DMA domain for untrusted device */ -if (dev_is_pci (dev) &&to_pci_dev (dev)->untrusted) +if (to_pci_dev (dev)->untrusted)return IOMMU_DOMAIN_DMA ;
This seems to apply a different rule to untrusted (e.g., removable) devices based on the topology, not the device itself. What if one of these devices is used as an eGPU and thus uses IOMMU_DOMAIN_DMA when it really needs IOMMU_DOMAIN_IDENTITY?
Quoted 31 lines
drivers/iommu/amd/iommu.c +15
-10-/* -* Do not identity map IOMMUv2 capable devices when: -* - memory encryption is active, because some of those devices -* (AMD GPUs) don't have the encryption bit in their DMA-mask -* and require remapping. -* - SNP is enabled, because it prohibits DTE[Mode]=0. -*/ -if (pdev_pasid_supported (dev_data) && - !cc_platform_has (CC_ATTR_MEM_ENCRYPT ) && - !amd_iommu_snp_en) { +/* Apply device specific quirks */ +if (quirks_force_identity_mapping (to_pci_dev (dev))) { + +/* +* When memory encryption is active, some of these devices +* don't have the encryption bit in their DMA-mask and +* require remapping. +*/ +if (cc_platform_has (CC_ATTR_MEM_ENCRYPT )) +return 0 ; + +/* DTE[Mode]=0 is prohibited when SNP is enabled */ +if (amd_iommu_snp_en) +return 0 ; +return IOMMU_DOMAIN_IDENTITY ; }Signature · 1 line
2.31.1
Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
Quoted 67 lines · Bjorn Helgaas
Quoted 3 lines · Vasant Hegde
Certain AMD GPU's must always be in identity mode. Currently its enforced using PASID check. It worked fine as most GPU's has PASID feature. But this means, identity mode enforcement is done for all PASID capable devices.
I think it would be useful to know something about *why* these devices require identity mode. And what happens without identity mode, i.e., is there a user-visible symptom that happens when the wrong mode is used?
Since the code doesn't test any feature bits, I assume it's because these devices have some hardware defect?
s/GPU's/GPUs/ (twice)
s/Currently its/Currently it's/
s/has PASID/have PASID/Quoted 38 lines
Previously it made sense as domain allocation API (
iommu_ops->domain_alloc()) was just passing domain type. So it couldn't check device capability and decide best suited page table type (v1 or v2). With recent enhancement to driver code, it usesdomain_alloc_paging_flags()ops for all paging mode domain allocation. This can check device/flags and allocate best suited page table (v1 or v2). Hence fixamd_iommu_def_domain_type()to force identity mapping for selected GPUs only.With this change system booted with DMA translation mode will select:
* Guest (v2) page table for PASID capable device
* Host (v1) page table for non-PASID capable deviceCc Alex Deucher <[email protected]>Cc Mario Limonciello <[email protected]>Signed-off-by Vasant Hegde <[email protected]>Tested-by Amandeep Kaur Longia <[email protected]>--- drivers/iommu/amd/iommu.c | 56 +++++++++++++++++++++++++++++++-------- 1 file changed, 45 insertions(+), 11 deletions(-)drivers/iommu/amd/iommu.c +9
-0@@ -3112,6 +3112,32 @@ static bool amd_iommu_is_attach_deferred(struct device *dev)return dev_data->defer_attach; } +static bool quirks_force_identity_mapping (struct pci_dev *pdev) +{ +struct pci_dev *root_port; +int class = pdev->class >>8 ; + +/* AMD GPU vendor ID */ +if (pdev->vendor !=PCI_VENDOR_ID_ATI ) +return false; +drivers/iommu/amd/iommu.c static bool amd_iommu_is_attach_deferred(struct device *dev) · line 3133
Quoted 10 lines
drivers/iommu/amd/iommu.c +10
-0+/* GPU class */ +if (class !=PCI_CLASS_DISPLAY_VGA && + class !=PCI_CLASS_DISPLAY_OTHER ) +return false; + + root_port =pcie_find_root_port (pdev); +if (!root_port) +return false; + +/* If bridge vendor is not ATI then its APU and force IDENTITY mode */ s/its/it's/ "it's" == "it is"; "its" shows ownership
Quoted 2 lines
drivers/iommu/amd/iommu.c static bool amd_iommu_is_attach_deferred(struct device *dev) · line 3135
Quoted 2 lines
drivers/iommu/amd/iommu.c +2
-0+if (root_port->vendor !=PCI_VENDOR_ID_ATI ) +return true;
This logic I believe is wrong. You're trying to look at the parent of the display device (which is an internal PCIe switch for a dGPU).
You basically want a similar implementation to amdgpu_device_find_parent() which figures out first device outside of the dGPU.
Quoted 3 lines
drivers/iommu/amd/iommu.c static bool amd_iommu_is_attach_deferred(struct device *dev) · line 3138
Quoted 3 lines
drivers/iommu/amd/iommu.c +3
-0+ +/* Rest all are dGPUs and works fine with DMA mode */ +return false;
Mostly for code flow, I think it would make sense the force_identity_mapping fallback is purely for APU.
IE something like this:
if (vendor != PCI_VENDOR_ATI)
return false; if (class != display) return false; if (pci_upstream_bridge()->vendor == PCI_VENDOR_ATI) return false;
/* rest are APUs, force identity */
return true;
Quoted 9 lines
Quoted 9 lines
drivers/iommu/amd/iommu.c +2
-0+} +static int amd_iommu_def_domain_type (struct device *dev) {struct iommu_dev_data *dev_data;@@ -3120,20 +3146,28 @@ static int amd_iommu_def_domain_type(struct device *dev)if (!dev_data)return 0 ;
Quoted 12 lines
drivers/iommu/amd/iommu.c static int amd_iommu_def_domain_type(struct device *dev) · line 3154
Quoted 7 lines
drivers/iommu/amd/iommu.c +4
-1+if (!dev_is_pci (dev)) +return 0 ; +/* Always use DMA domain for untrusted device */ -if (dev_is_pci (dev) &&to_pci_dev (dev)->untrusted) +if (to_pci_dev (dev)->untrusted)return IOMMU_DOMAIN_DMA ;This seems to apply a different rule to untrusted (e.g., removable) devices based on the topology, not the device itself. What if one of these devices is used as an eGPU and thus uses
IOMMU_DOMAIN_DMAwhen it really needsIOMMU_DOMAIN_IDENTITY?
I believe the identity domain mapping is only needed for APUs, which can't be put into an eGPU enclosure.
Quoted 32 lines
Quoted 32 lines
drivers/iommu/amd/iommu.c +15
-10-/* -* Do not identity map IOMMUv2 capable devices when: -* - memory encryption is active, because some of those devices -* (AMD GPUs) don't have the encryption bit in their DMA-mask -* and require remapping. -* - SNP is enabled, because it prohibits DTE[Mode]=0. -*/ -if (pdev_pasid_supported (dev_data) && - !cc_platform_has (CC_ATTR_MEM_ENCRYPT ) && - !amd_iommu_snp_en) { +/* Apply device specific quirks */ +if (quirks_force_identity_mapping (to_pci_dev (dev))) { + +/* +* When memory encryption is active, some of these devices +* don't have the encryption bit in their DMA-mask and +* require remapping. +*/ +if (cc_platform_has (CC_ATTR_MEM_ENCRYPT )) +return 0 ; + +/* DTE[Mode]=0 is prohibited when SNP is enabled */ +if (amd_iommu_snp_en) +return 0 ; +return IOMMU_DOMAIN_IDENTITY ; }Signature · 1 line
2.31.1
Vasant Hegde
maintainerRe: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
Mario,
Quoted 70 lines · Mario Limonciello
Quoted 67 lines · Bjorn Helgaas
Quoted 3 lines · Vasant Hegde
Certain AMD GPU's must always be in identity mode. Currently its enforced using PASID check. It worked fine as most GPU's has PASID feature. But this means, identity mode enforcement is done for all PASID capable devices.
I think it would be useful to know something about *why* these devices require identity mode. And what happens without identity mode, i.e., is there a user-visible symptom that happens when the wrong mode is used?
Since the code doesn't test any feature bits, I assume it's because these devices have some hardware defect?
s/GPU's/GPUs/ (twice)
s/Currently its/Currently it's/
s/has PASID/have PASID/Quoted 38 lines
Previously it made sense as domain allocation API (
iommu_ops->domain_alloc()) was just passing domain type. So it couldn't check device capability and decide best suited page table type (v1 or v2). With recent enhancement to driver code, it usesdomain_alloc_paging_flags()ops for all paging mode domain allocation. This can check device/flags and allocate best suited page table (v1 or v2). Hence fixamd_iommu_def_domain_type()to force identity mapping for selected GPUs only.With this change system booted with DMA translation mode will select:
* Guest (v2) page table for PASID capable device
* Host (v1) page table for non-PASID capable deviceCc Alex Deucher <[email protected]>Cc Mario Limonciello <[email protected]>Signed-off-by Vasant Hegde <[email protected]>Tested-by Amandeep Kaur Longia <[email protected]>---
drivers/iommu/amd/iommu.c| 56 +++++++++++++++++++++++++++++++--------1 file changed, 45 insertions(+), 11 deletions(-)drivers/iommu/amd/iommu.c +8
-0@@ -3112,6 +3112,32 @@ static bool amd_iommu_is_attach_deferred(struct device *dev) return dev_data->defer_attach; } +static bool quirks_force_identity_mapping(struct pci_dev *pdev) +{ + struct pci_dev *root_port; + int class = pdev->class >> 8; + + /* AMD GPU vendor ID */ + if (pdev->vendor != PCI_VENDOR_ID_ATI) + return false; +drivers/iommu/amd/iommu.c static bool amd_iommu_is_attach_deferred(struct device · line 3133
Quoted 10 lines
drivers/iommu/amd/iommu.c +10
-0+ /* GPU class */ + if (class != PCI_CLASS_DISPLAY_VGA && + class != PCI_CLASS_DISPLAY_OTHER) + return false; + + root_port = pcie_find_root_port(pdev); + if (!root_port) + return false; + + /* If bridge vendor is not ATI then its APU and force IDENTITY mode */s/its/it's/ "it's" == "it is"; "its" shows ownership
Quoted 5 lines
Quoted 2 lines
drivers/iommu/amd/iommu.c static bool amd_iommu_is_attach_deferred(struct device *dev) · line 3135
Quoted 2 lines
drivers/iommu/amd/iommu.c +2
-0+ if (root_port->vendor != PCI_VENDOR_ID_ATI) + return true;This logic I believe is wrong. You're trying to look at the parent of the display device (which is an internal PCIe switch for a dGPU).
You are right. It got inverted while I was fine tuning the code.
Quoted 3 lines
You basically want a similar implementation to
amdgpu_device_find_parent()which figures out first device outside of the dGPU.
Ack.
Quoted 17 lines
Quoted 4 lines
drivers/iommu/amd/iommu.c static bool amd_iommu_is_attach_deferred(struct device *dev) · line 3138
Quoted 4 lines
drivers/iommu/amd/iommu.c +3
-0+ + /* Rest all are dGPUs and works fine with DMA mode */ + return false;Mostly for code flow, I think it would make sense the
force_identity_mappingfallback is purely for APU.IE something like this:
if (vendor !=
PCI_VENDOR_ATI)
return false;
if (class != display)
return false;
if (pci_upstream_bridge()->vendor ==PCI_VENDOR_ATI) return false;
Quoted 2 lines
/* rest are APUs, force identity */
return true;
Makes sense. And pci_upstream_bridge() return NULL, then return true? (something like below) ?
+static bool quirks_force_identity_mapping(struct pci_dev *pdev) +{
+ struct pci_dev *root_port;
+ int class = pdev->class >> 8;
+
+ /* AMD GPU vendor ID */
+ if (pdev->vendor != PCI_VENDOR_ID_ATI)
+ return false;
+
+ /* GPU class */
+ if (class != PCI_CLASS_DISPLAY_VGA &&
+ class != PCI_CLASS_DISPLAY_OTHER)
+ return false;
+
+ if (pci_upstream_bridge(pdev) &&
+ pci_upstream_bridge(pdev)->vendor == PCI_VENDOR_ID_ATI)
+ return false;
+
+ return true;
+}
+
-Vasant
Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
Quoted 105 lines · Vasant Hegde
Mario,
Quoted 70 lines · Mario Limonciello
Quoted 67 lines · Bjorn Helgaas
Quoted 3 lines · Vasant Hegde
Certain AMD GPU's must always be in identity mode. Currently its enforced using PASID check. It worked fine as most GPU's has PASID feature. But this means, identity mode enforcement is done for all PASID capable devices.
I think it would be useful to know something about *why* these devices require identity mode. And what happens without identity mode, i.e., is there a user-visible symptom that happens when the wrong mode is used?
Since the code doesn't test any feature bits, I assume it's because these devices have some hardware defect?
s/GPU's/GPUs/ (twice)
s/Currently its/Currently it's/
s/has PASID/have PASID/Quoted 38 lines
Previously it made sense as domain allocation API (
iommu_ops->domain_alloc()) was just passing domain type. So it couldn't check device capability and decide best suited page table type (v1 or v2). With recent enhancement to driver code, it usesdomain_alloc_paging_flags()ops for all paging mode domain allocation. This can check device/flags and allocate best suited page table (v1 or v2). Hence fixamd_iommu_def_domain_type()to force identity mapping for selected GPUs only.With this change system booted with DMA translation mode will select:
* Guest (v2) page table for PASID capable device
* Host (v1) page table for non-PASID capable deviceCc Alex Deucher <[email protected]>Cc Mario Limonciello <[email protected]>Signed-off-by Vasant Hegde <[email protected]>Tested-by Amandeep Kaur Longia <[email protected]>--- drivers/iommu/amd/iommu.c | 56 +++++++++++++++++++++++++++++++-------- 1 file changed, 45 insertions(+), 11 deletions(-)drivers/iommu/amd/iommu.c +8
-0@@ -3112,6 +3112,32 @@ static bool amd_iommu_is_attach_deferred(struct device *dev) return dev_data->defer_attach; } +static bool quirks_force_identity_mapping(struct pci_dev *pdev) +{ + struct pci_dev *root_port; + int class = pdev->class >> 8; + + /* AMD GPU vendor ID */ + if (pdev->vendor != PCI_VENDOR_ID_ATI) + return false; +drivers/iommu/amd/iommu.c static bool amd_iommu_is_attach_deferred(struct device · line 3133
Quoted 10 lines
drivers/iommu/amd/iommu.c +10
-0+ /* GPU class */ + if (class != PCI_CLASS_DISPLAY_VGA && + class != PCI_CLASS_DISPLAY_OTHER) + return false; + + root_port = pcie_find_root_port(pdev); + if (!root_port) + return false; + + /* If bridge vendor is not ATI then its APU and force IDENTITY mode */s/its/it's/ "it's" == "it is"; "its" shows ownership
Quoted 5 lines
Quoted 2 lines
drivers/iommu/amd/iommu.c static bool amd_iommu_is_attach_deferred(struct device *dev) · line 3135
Quoted 2 lines
drivers/iommu/amd/iommu.c +2
-0+ if (root_port->vendor != PCI_VENDOR_ID_ATI) + return true;This logic I believe is wrong. You're trying to look at the parent of the display device (which is an internal PCIe switch for a dGPU).
You are right. It got inverted while I was fine tuning the code.
Quoted 3 lines
You basically want a similar implementation to
amdgpu_device_find_parent()which figures out first device outside of the dGPU.Ack.
Quoted 17 lines
Quoted 4 lines
drivers/iommu/amd/iommu.c static bool amd_iommu_is_attach_deferred(struct device *dev) · line 3138
Quoted 4 lines
drivers/iommu/amd/iommu.c +3
-0+ + /* Rest all are dGPUs and works fine with DMA mode */ + return false;Mostly for code flow, I think it would make sense the
force_identity_mappingfallback is purely for APU.IE something like this:
if (vendor !=
PCI_VENDOR_ATI)
return false;
if (class != display)
return false;
if (pci_upstream_bridge()->vendor ==PCI_VENDOR_ATI)
return false;
Quoted 5 lines
Quoted 2 lines
/* rest are APUs, force identity */
return true;Makes sense. And
pci_upstream_bridge()return NULL, then return true? (something like below) ?
Bascially; yeah. But one comment below.
Quoted 6 lines
drivers/iommu/amd/iommu.c +5
-0+static bool quirks_force_identity_mapping (struct pci_dev *pdev) +{ +struct pci_dev *root_port; +int class = pdev->class >>8 ; +
drivers/iommu/amd/iommu.c static bool amd_iommu_is_attach_deferred(struct device *dev)
Quoted 10 lines
drivers/iommu/amd/iommu.c +10
-0+/* AMD GPU vendor ID */ +if (pdev->vendor !=PCI_VENDOR_ID_ATI ) +return false; + +/* GPU class */ +if (class !=PCI_CLASS_DISPLAY_VGA && + class !=PCI_CLASS_DISPLAY_OTHER ) +return false; + + if (pci_upstream_bridge(pdev) &&
I don't think you need to check for pci_upstream_bridge() to be non-NULL. You already checked that it's an endpoint by looking at the class. So an endpoint will be connected to a bridge of some sort (either a switch internal to the dGPU or to a root port).
drivers/iommu/amd/iommu.c static bool amd_iommu_is_attach_deferred(struct device *dev) · line 3123
Quoted 3 lines
drivers/iommu/amd/iommu.c +3
-0+ pci_upstream_bridge(pdev)->vendor == PCI_VENDOR_ID_ATI) +return false; +
Make sure you leave a comment here about what true means (/* it is the GPU in an APU */) so that if we need to add more cases later it's obvious without having to dig up this thread again.
Quoted 6 lines
drivers/iommu/amd/iommu.c +3
-1+return true; +} + -Vasant
Vasant Hegde
maintainerRe: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
Mario,
Quoted 15 lines · Mario Limonciello
Quoted 12 lines · Vasant Hegde
Mario,
Quoted 8 lines · Mario Limonciello
Quoted 5 lines · Bjorn Helgaas
Quoted 3 lines · Vasant Hegde
Certain AMD GPU's must always be in identity mode. Currently its enforced using PASID check. It worked fine as most GPU's has PASID feature. But this means, identity mode enforcement is done for all PASID capable devices.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
I think it would be useful to know something about *why* these devices require identity mode. And what happens without identity mode, i.e., is there a user-visible symptom that happens when the wrong mode is used?
Since the code doesn't test any feature bits, I assume it's because these devices have some hardware defect?
s/GPU's/GPUs/ (twice)
.../...
Quoted 13 lines
Quoted 13 lines
drivers/iommu/amd/iommu.c +12
-0+static bool quirks_force_identity_mapping (struct pci_dev *pdev) +{ + struct pci_dev *root_port; + int class = pdev->class >> 8; + + /* AMD GPU vendor ID */ + if (pdev->vendor != PCI_VENDOR_ID_ATI) + return false; + + /* GPU class */ + if (class != PCI_CLASS_DISPLAY_VGA && + class != PCI_CLASS_DISPLAY_OTHER)
Quoted 8 lines
Quoted 3 lines
+ return false;
+
+ if (pci_upstream_bridge(pdev)&&I don't think you need to check for
pci_upstream_bridge()to be non-NULL. You already checked that it's an endpoint by looking at the class. So an endpoint will be connected to a bridge of some sort (either a switch internal to the dGPU or to a root port).
Ok will fix it .
Quoted 8 lines
Quoted 4 lines
+ pci_upstream_bridge(pdev)->vendor ==
PCI_VENDOR_ID_ATI) + return false;
+Make sure you leave a comment here about what true means (/* it is the GPU in an APU */) so that if we need to add more cases later it's obvious without having to dig up this thread again.
Sure.
-Vasant
Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
In reply to Mario LimoncielloQuoted 21 lines · Mario Limonciello
Quoted 6 lines · Vasant Hegde
Quoted 5 lines · Mario Limonciello
Quoted 4 lines · Bjorn Helgaas
Quoted 3 lines · Vasant Hegde
Certain AMD GPU's must always be in identity mode. Currently its enforced using PASID check. It worked fine as most GPU's has PASID feature. But this means, identity mode enforcement is done for all PASID capable devices.
...
Quoted 12 lines
drivers/iommu/amd/iommu.c +12
-0+static bool quirks_force_identity_mapping (struct pci_dev *pdev) +{ +struct pci_dev *root_port; +int class = pdev->class >>8 ; + +/* AMD GPU vendor ID */ +if (pdev->vendor !=PCI_VENDOR_ID_ATI ) +return false; + +/* GPU class */ +if (class !=PCI_CLASS_DISPLAY_VGA && + class !=PCI_CLASS_DISPLAY_OTHER )
Quoted 8 lines
drivers/iommu/amd/iommu.c static bool amd_iommu_is_attach_deferred(struct device *dev)
Quoted 3 lines
drivers/iommu/amd/iommu.c +3
-0+return false; + + if (pci_upstream_bridge(pdev) &&I don't think you need to check for
pci_upstream_bridge()to be non-NULL. You already checked that it's an endpoint by looking at the class. So an endpoint will be connected to a bridge of some sort (either a switch internal to the dGPU or to a root port).
I think devices can be passed through to virtualized guests with no upstream bridge visible to the guest, can't they?
Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
Quoted 31 lines · Bjorn Helgaas
Quoted 21 lines · Mario Limonciello
Quoted 6 lines · Vasant Hegde
Quoted 5 lines · Mario Limonciello
Quoted 4 lines · Bjorn Helgaas
Quoted 3 lines · Vasant Hegde
Certain AMD GPU's must always be in identity mode. Currently its enforced using PASID check. It worked fine as most GPU's has PASID feature. But this means, identity mode enforcement is done for all PASID capable devices.
...
Quoted 12 lines
drivers/iommu/amd/iommu.c +12
-0+static bool quirks_force_identity_mapping (struct pci_dev *pdev) +{ +struct pci_dev *root_port; +int class = pdev->class >>8 ; + +/* AMD GPU vendor ID */ +if (pdev->vendor !=PCI_VENDOR_ID_ATI ) +return false; + +/* GPU class */ +if (class !=PCI_CLASS_DISPLAY_VGA && + class !=PCI_CLASS_DISPLAY_OTHER )Quoted 8 lines
drivers/iommu/amd/iommu.c static bool amd_iommu_is_attach_deferred(struct device *dev)
Quoted 3 lines
drivers/iommu/amd/iommu.c +3
-0+return false; + + if (pci_upstream_bridge(pdev) &&I don't think you need to check for
pci_upstream_bridge()to be non-NULL. You already checked that it's an endpoint by looking at the class. So an endpoint will be connected to a bridge of some sort (either a switch internal to the dGPU or to a root port).
Quoted 2 lines
I think devices can be passed through to virtualized guests with no upstream bridge visible to the guest, can't they?
None of the GPUs in APUs support SRIOV, so there isn't a VF to be passed in to a guest. But I guess you could pass the whole endpoint in.
When passed into a guest doesn't it end up associated behind a root port in the guest though?
Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
[+cc Gerd, Alex]
Quoted 35 lines · Mario Limonciello
Quoted 31 lines · Bjorn Helgaas
Quoted 21 lines · Mario Limonciello
Quoted 6 lines · Vasant Hegde
Quoted 5 lines · Mario Limonciello
Quoted 4 lines · Bjorn Helgaas
Quoted 3 lines · Vasant Hegde
Certain AMD GPU's must always be in identity mode. Currently its enforced using PASID check. It worked fine as most GPU's has PASID feature. But this means, identity mode enforcement is done for all PASID capable devices.
...
Quoted 12 lines
(quoted diff) +12
-0+static bool quirks_force_identity_mapping(struct pci_dev *pdev) +{ + struct pci_dev *root_port; + int class = pdev->class >> 8; + + /* AMD GPU vendor ID */ + if (pdev->vendor != PCI_VENDOR_ID_ATI) + return false; + + /* GPU class */ + if (class != PCI_CLASS_DISPLAY_VGA && + class != PCI_CLASS_DISPLAY_OTHER)Quoted 8 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ return false; + + if (pci_upstream_bridge(pdev) &&I don't think you need to check for
pci_upstream_bridge()to be non-NULL. You already checked that it's an endpoint by looking at the class. So an endpoint will be connected to a bridge of some sort (either a switch internal to the dGPU or to a root port).Quoted 2 lines
I think devices can be passed through to virtualized guests with no upstream bridge visible to the guest, can't they?
Quoted 6 lines
None of the GPUs in APUs support SRIOV, so there isn't a VF to be passed in to a guest. But I guess you could pass the whole endpoint in.
When passed into a guest doesn't it end up associated behind a root port in the guest though?
I don't think the Root Port is always passed into the guest. From 1ae8c4ce1570 ("PCI: Enable AtomicOps only if Root Port supports them"):
On s390 and many virtualized guests, the Endpoint is visible but the Root Port is not. In this case, pci_enable_atomic_ops_to_root() previously enabled AtomicOps in the Endpoint even though it can't tell whether the Root Port supports them as a completer.
Maybe Alex will chime in and tell us more about non-s390 guests.
Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
Quoted 3 lines · Bjorn Helgaas
I don't think the Root Port is always passed into the guest. From
1ae8c4ce1570("PCI: Enable AtomicOps only if Root Port supports them"):
If this device has broken support for iommu translation entirely then vfio is unsuably broken too. The right answer is to refuse to attach a paging domain to such a device, I think this patch series should be enhanced to include this.
Like a qurik that says translation is broken and then def domain is changed and paging attach will fail
Jason
Vasant Hegde
maintainerRe: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
Quoted 11 lines · Jason Gunthorpe
Quoted 3 lines · Bjorn Helgaas
I don't think the Root Port is always passed into the guest. From
1ae8c4ce1570("PCI: Enable AtomicOps only if Root Port supports them"):If this device has broken support for iommu translation entirely then vfio is unsuably broken too. The right answer is to refuse to attach a paging domain to such a device, I think this patch series should be enhanced to include this.
Quoted 1 line
Like a qurik that says translation is broken and then def domain is
changed and paging attach will fail
Yeah. We can add check i domain allocation path as well.
-Vasant
Quoted 2 lines
Jason
Vasant Hegde
maintainerRe: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
In reply to Bjorn HelgaasBjorn,
Quoted 8 lines · Bjorn Helgaas
Quoted 3 lines · Vasant Hegde
Certain AMD GPU's must always be in identity mode. Currently its enforced using PASID check. It worked fine as most GPU's has PASID feature. But this means, identity mode enforcement is done for all PASID capable devices.
I think it would be useful to know something about *why* these devices require identity mode. And what happens without identity mode, i.e., is there a user-visible symptom that happens when the wrong mode is used?
I will have to check with Alex for the details. Will check with him.
Quoted 7 lines
Since the code doesn't test any feature bits, I assume it's because these devices have some hardware defect?
s/GPU's/GPUs/ (twice)
s/Currently its/Currently it's/
s/has PASID/have PASID/
Ack. Will fix it in v2.
Quoted 43 lines
Quoted 43 lines
Previously it made sense as domain allocation API (
iommu_ops->domain_alloc()) was just passing domain type. So it couldn't check device capability and decide best suited page table type (v1 or v2). With recent enhancement to driver code, it usesdomain_alloc_paging_flags()ops for all paging mode domain allocation. This can check device/flags and allocate best suited page table (v1 or v2). Hence fixamd_iommu_def_domain_type()to force identity mapping for selected GPUs only.With this change system booted with DMA translation mode will select:
* Guest (v2) page table for PASID capable device
* Host (v1) page table for non-PASID capable deviceCc Alex Deucher <[email protected]>Cc Mario Limonciello <[email protected]>Signed-off-by Vasant Hegde <[email protected]>Tested-by Amandeep Kaur Longia <[email protected]>--- drivers/iommu/amd/iommu.c | 56 +++++++++++++++++++++++++++++++-------- 1 file changed, 45 insertions(+), 11 deletions(-)drivers/iommu/amd/iommu.c +13
-0@@ -3112,6 +3112,32 @@ static bool amd_iommu_is_attach_deferred(struct device *dev)return dev_data->defer_attach; } +static bool quirks_force_identity_mapping (struct pci_dev *pdev) +{ +struct pci_dev *root_port; +int class = pdev->class >>8 ; + +/* AMD GPU vendor ID */ +if (pdev->vendor !=PCI_VENDOR_ID_ATI ) +return false; + +/* GPU class */ +if (class !=PCI_CLASS_DISPLAY_VGA && + class !=PCI_CLASS_DISPLAY_OTHER ) +return false;
Quoted 8 lines
drivers/iommu/amd/iommu.c static bool amd_iommu_is_attach_deferred(struct device *dev) · line 3133
Quoted 6 lines
drivers/iommu/amd/iommu.c +6
-0+ + root_port =pcie_find_root_port (pdev); +if (!root_port) +return false; + +/* If bridge vendor is not ATI then its APU and force IDENTITY mode */ s/its/it's/ "it's" == "it is"; "its" shows ownership
Ack. Will fix it in v2.
Quoted 15 lines
Quoted 15 lines
drivers/iommu/amd/iommu.c +7
-0+if (root_port->vendor !=PCI_VENDOR_ID_ATI ) +return true; + +/* Rest all are dGPUs and works fine with DMA mode */ +return false; +} +static int amd_iommu_def_domain_type (struct device *dev) {struct iommu_dev_data *dev_data;@@ -3120,20 +3146,28 @@ static int amd_iommu_def_domain_type(struct device *dev)if (!dev_data)return 0 ;
Quoted 12 lines
drivers/iommu/amd/iommu.c static int amd_iommu_def_domain_type(struct device *dev) · line 3154
Quoted 7 lines
drivers/iommu/amd/iommu.c +4
-1+if (!dev_is_pci (dev)) +return 0 ; +/* Always use DMA domain for untrusted device */ -if (dev_is_pci (dev) &&to_pci_dev (dev)->untrusted) +if (to_pci_dev (dev)->untrusted)return IOMMU_DOMAIN_DMA ;This seems to apply a different rule to untrusted (e.g., removable) devices based on the topology, not the device itself. What if one of these devices is used as an eGPU and thus uses
IOMMU_DOMAIN_DMAwhen it really needsIOMMU_DOMAIN_IDENTITY?
I tried to keep the existing order (untested followed by GPU check). As Mario commented, we need identity for APU only. So this is fine.
-Vasant
Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
In reply to Vasant Hegdedrivers/iommu/amd/iommu.c static int amd_iommu_def_domain_type(struct device *dev) · line 3154
Quoted 11 lines · Vasant Hegde
drivers/iommu/amd/iommu.c +4
-1@@ -3120,20 +3146,28 @@ static int amd_iommu_def_domain_type(struct device *dev)if (!dev_data)return 0 ; +if (!dev_is_pci (dev)) +return 0 ; +/* Always use DMA domain for untrusted device */ -if (dev_is_pci (dev) &&to_pci_dev (dev)->untrusted) +if (to_pci_dev (dev)->untrusted)return IOMMU_DOMAIN_DMA ;
The core code does this, it shouldn't be in a driver?
Quoted 10 lines
drivers/iommu/amd/iommu.c +0
-10-/* -* Do not identity map IOMMUv2 capable devices when: -* - memory encryption is active, because some of those devices -* (AMD GPUs) don't have the encryption bit in their DMA-mask -* and require remapping. -* - SNP is enabled, because it prohibits DTE[Mode]=0. -*/ -if (pdev_pasid_supported (dev_data) && - !cc_platform_has (CC_ATTR_MEM_ENCRYPT ) && - !amd_iommu_snp_en) {
drivers/iommu/amd/iommu.c static int amd_iommu_def_domain_type(struct device *dev) · line 3165
Quoted 10 lines
drivers/iommu/amd/iommu.c +10
-0+/* Apply device specific quirks */ +if (quirks_force_identity_mapping (to_pci_dev (dev))) { + +/* +* When memory encryption is active, some of these devices +* don't have the encryption bit in their DMA-mask and +* require remapping. +*/ +if (cc_platform_has (CC_ATTR_MEM_ENCRYPT )) +return 0 ;
I think this can probably be dropped now? There is no intersection of this old broken embedded GPU and a CPU that supports CC right?
drivers/iommu/amd/iommu.c static int amd_iommu_def_domain_type(struct device *dev) · line 3169
Quoted 4 lines
drivers/iommu/amd/iommu.c +4
-0+ +/* DTE[Mode]=0 is prohibited when SNP is enabled */ +if (amd_iommu_snp_en) +return 0 ;
Ditto?
Jason
Vasant Hegde
maintainerRe: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
Quoted 5 lines · Jason Gunthorpe
Quoted 4 lines · Vasant Hegde
drivers/iommu/amd/iommu.c +0
-0@@ -3120,20 +3146,28 @@ static int amd_iommu_def_domain_type(struct device *dev)if (!dev_data)return 0 ;
Quoted 9 lines
drivers/iommu/amd/iommu.c static int amd_iommu_def_domain_type(struct device *dev) · line 3154
Quoted 7 lines
drivers/iommu/amd/iommu.c +4
-1+if (!dev_is_pci (dev)) +return 0 ; +/* Always use DMA domain for untrusted device */ -if (dev_is_pci (dev) &&to_pci_dev (dev)->untrusted) +if (to_pci_dev (dev)->untrusted)return IOMMU_DOMAIN_DMA ;The core code does this, it shouldn't be in a driver?
Right. I didn't realized core has this check. I will drop this check in next version.
Quoted 16 lines
Quoted 16 lines
drivers/iommu/amd/iommu.c +5
-10-/* -* Do not identity map IOMMUv2 capable devices when: -* - memory encryption is active, because some of those devices -* (AMD GPUs) don't have the encryption bit in their DMA-mask -* and require remapping. -* - SNP is enabled, because it prohibits DTE[Mode]=0. -*/ -if (pdev_pasid_supported (dev_data) && - !cc_platform_has (CC_ATTR_MEM_ENCRYPT ) && - !amd_iommu_snp_en) { +/* Apply device specific quirks */ +if (quirks_force_identity_mapping (to_pci_dev (dev))) { + +/* +* When memory encryption is active, some of these devices
Quoted 8 lines
drivers/iommu/amd/iommu.c static int amd_iommu_def_domain_type(struct device *dev) · line 3165
Quoted 5 lines
drivers/iommu/amd/iommu.c +5
-0+* don't have the encryption bit in their DMA-mask and +* require remapping. +*/ +if (cc_platform_has (CC_ATTR_MEM_ENCRYPT )) +return 0 ;I think this can probably be dropped now? There is no intersection of this old broken embedded GPU and a CPU that supports CC right?
I did consider dropping these two checks, but I was not entirely sure. So I kept as is for now. Once these series settles and I can go with next step of dropping these two checks.
@Alex, @Mario, Any suggestions ?
-Vasant
Quoted 9 lines
drivers/iommu/amd/iommu.c static int amd_iommu_def_domain_type(struct device *dev) · line 3169
Quoted 5 lines
drivers/iommu/amd/iommu.c +4
-0+ +/* DTE[Mode]=0 is prohibited when SNP is enabled */ +if (amd_iommu_snp_en) +return 0 ;Ditto?
Jason
Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
Quoted 47 lines · Vasant Hegde
Quoted 5 lines · Jason Gunthorpe
Quoted 4 lines · Vasant Hegde
drivers/iommu/amd/iommu.c +0
-0@@ -3120,20 +3146,28 @@ static int amd_iommu_def_domain_type(struct device *dev)if (!dev_data)return 0 ;Quoted 9 lines
drivers/iommu/amd/iommu.c static int amd_iommu_def_domain_type(struct device *dev) · line 3154
Quoted 7 lines
drivers/iommu/amd/iommu.c +4
-1+if (!dev_is_pci (dev)) +return 0 ; +/* Always use DMA domain for untrusted device */ -if (dev_is_pci (dev) &&to_pci_dev (dev)->untrusted) +if (to_pci_dev (dev)->untrusted)return IOMMU_DOMAIN_DMA ;The core code does this, it shouldn't be in a driver?
Right. I didn't realized core has this check. I will drop this check in next version.
Quoted 16 lines
Quoted 16 lines
drivers/iommu/amd/iommu.c +5
-10-/* -* Do not identity map IOMMUv2 capable devices when: -* - memory encryption is active, because some of those devices -* (AMD GPUs) don't have the encryption bit in their DMA-mask -* and require remapping. -* - SNP is enabled, because it prohibits DTE[Mode]=0. -*/ -if (pdev_pasid_supported (dev_data) && - !cc_platform_has (CC_ATTR_MEM_ENCRYPT ) && - !amd_iommu_snp_en) { +/* Apply device specific quirks */ +if (quirks_force_identity_mapping (to_pci_dev (dev))) { + +/* +* When memory encryption is active, some of these devices Quoted 8 lines
drivers/iommu/amd/iommu.c static int amd_iommu_def_domain_type(struct device *dev) · line 3165
Quoted 5 lines
drivers/iommu/amd/iommu.c +5
-0+* don't have the encryption bit in their DMA-mask and +* require remapping. +*/ +if (cc_platform_has (CC_ATTR_MEM_ENCRYPT )) +return 0 ;I think this can probably be dropped now? There is no intersection of this old broken embedded GPU and a CPU that supports CC right?
Quoted 6 lines
I did consider dropping these two checks, but I was not entirely sure. So I kept as is for now. Once these series settles and I can go with next step of dropping these two checks.
@Alex, @Mario, Any suggestions ?
The background comes from this series:
https://lore.kernel.org/all/[email protected]/
Quoted 12 lines
-Vasant
Quoted 9 lines
drivers/iommu/amd/iommu.c static int amd_iommu_def_domain_type(struct device *dev) · line 3169
Quoted 5 lines
drivers/iommu/amd/iommu.c +4
-0+ +/* DTE[Mode]=0 is prohibited when SNP is enabled */ +if (amd_iommu_snp_en) +return 0 ;Ditto?
Jason
Vasant Hegde
maintainerRe: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
Mario,
Quoted 5 lines · Mario Limonciello
Quoted 2 lines · Vasant Hegde
.../...
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
I think this can probably be dropped now? There is no intersection of this old broken embedded GPU and a CPU that supports CC right?
I did consider dropping these two checks, but I was not entirely sure. So I kept as is for now. Once these series settles and I can go with next step of dropping these two checks.
Quoted 6 lines
Quoted 2 lines
@Alex, @Mario, Any suggestions ?
The background comes from this series:
Thanks for old link. So it looks like we are good to drop SME check?
Also is it safe to assume SNP is not enabled on these platform and drop that check? Anyway SNP requirement (which doesn't support identity mapping) and APU requirement (needs identity mapping) is conflicting.
-Vasant
Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
+Felix
Quoted 24 lines · Vasant Hegde
Mario,
Quoted 5 lines · Mario Limonciello
Quoted 2 lines · Vasant Hegde
.../...
Quoted 12 lines
Quoted 11 lines
Quoted 3 lines
I think this can probably be dropped now? There is no intersection of this old broken embedded GPU and a CPU that supports CC right?
I did consider dropping these two checks, but I was not entirely sure. So I kept as is for now. Once these series settles and I can go with next step of dropping these two checks.
@Alex, @Mario, Any suggestions ?
Quoted 6 lines
Quoted 3 lines
The background comes from this series:
Thanks for old link. So it looks like we are good to drop SME check?
Yeah I think so.
Quoted 5 lines
Also is it safe to assume SNP is not enabled on these platform and drop that check? Anyway SNP requirement (which doesn't support identity mapping) and APU requirement (needs identity mapping) is conflicting.
The current platforms I think this is a safe assumption.
Felix,
Can we revisit the requirement of identity mapping for APU? Does it still hold with the current hardware and software?
I'm wondering if we can just apply it to Raven and loosen up elsewhere and what gives if we do that.
Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
Quoted 42 lines · Mario Limonciello
+Felix
Quoted 24 lines · Vasant Hegde
Mario,
Quoted 5 lines · Mario Limonciello
Quoted 2 lines · Vasant Hegde
.../...
Quoted 12 lines
Quoted 11 lines
Quoted 3 lines
I think this can probably be dropped now? There is no intersection of this old broken embedded GPU and a CPU that supports CC right?
I did consider dropping these two checks, but I was not entirely sure. So I kept as is for now. Once these series settles and I can go with next step of dropping these two checks.
@Alex, @Mario, Any suggestions ?
Quoted 6 lines
Quoted 3 lines
The background comes from this series:
Thanks for old link. So it looks like we are good to drop SME check?
Yeah I think so.
Quoted 5 lines
Also is it safe to assume SNP is not enabled on these platform and drop that check? Anyway SNP requirement (which doesn't support identity mapping) and APU requirement (needs identity mapping) is conflicting.
Quoted 5 lines
The current platforms I think this is a safe assumption.
Felix,
Can we revisit the requirement of identity mapping for APU? Does it still hold with the current hardware and software?
I'm not sure what this means. Is Identity mode something like pass-through? I don't know why this is required on APUs. If it's related to IOMMUv2 support, then it's probably no longer needed. We're not using IOMMUv2 on our APUs any more. Support for that was removed from the amdgpu driver a few years ago.
Regards,
Felix
Quoted 2 lines
I'm wondering if we can just apply it to Raven and loosen up elsewhere and what gives if we do that.
Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
Quoted 43 lines · Felix
Quoted 42 lines · Mario Limonciello
+Felix
Quoted 24 lines · Vasant Hegde
Mario,
Quoted 5 lines · Mario Limonciello
Quoted 2 lines · Vasant Hegde
.../...
Quoted 12 lines
Quoted 11 lines
Quoted 3 lines
I think this can probably be dropped now? There is no intersection of this old broken embedded GPU and a CPU that supports CC right?
I did consider dropping these two checks, but I was not entirely sure. So I kept as is for now. Once these series settles and I can go with next step of dropping these two checks.
@Alex, @Mario, Any suggestions ?
Quoted 6 lines
Quoted 3 lines
The background comes from this series:
Thanks for old link. So it looks like we are good to drop SME check?
Yeah I think so.
Quoted 5 lines
Also is it safe to assume SNP is not enabled on these platform and drop that check? Anyway SNP requirement (which doesn't support identity mapping) and APU requirement (needs identity mapping) is conflicting.
Quoted 8 lines
Quoted 5 lines
The current platforms I think this is a safe assumption.
Felix,
Can we revisit the requirement of identity mapping for APU? Does it still hold with the current hardware and software?
I'm not sure what this means. Is Identity mode something like pass- through? I don't know why this is required on APUs. If it's related to IOMMUv2 support, then it's probably no longer needed. We're not using IOMMUv2 on our APUs any more. Support for that was removed from the amdgpu driver a few years ago.
Hmm, I just had a try with this on a few years old APU (PHX). It's flicker central with this (7.2-rc5~ish tree):
drivers/iommu/amd/iommu.c +0 -13
@@ -3134,19 +3134,6 @@ static int amd_iommu_def_domain_type(struct device *dev)if (dev_is_pci (dev) &&to_pci_dev (dev)->untrusted)return IOMMU_DOMAIN_DMA ; -/* -* Do not identity map IOMMUv2 capable devices when: -* - memory encryption is active, because some of those devices -* (AMD GPUs) don't have the encryption bit in their DMA-mask -* and require remapping. -* - SNP is enabled, because it prohibits DTE[Mode]=0. -*/ -if (pdev_pasid_supported (dev_data) && - !cc_platform_has (CC_ATTR_MEM_ENCRYPT ) && - !amd_iommu_snp_en) { -return IOMMU_DOMAIN_IDENTITY ; - } -return 0 ; }
The flicking is reminencent of issues moving BOs between VRAM and GTT from a few years ago.
So I don't think it can be dropped (just) yet.
Vasant Hegde
maintainerRe: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
Mario, Alex, Felix,
Quoted 8 lines · Mario Limonciello
Quoted 5 lines · Felix
Quoted 4 lines · Mario Limonciello
+Felix
Quoted 1 line · Vasant Hegde
Mario,
.../...
Quoted 31 lines
Hmm, I just had a try with this on a few years old APU (PHX). It's flicker central with this (7.2-rc5~ish tree):
drivers/iommu/amd/iommu.c +0
-13@@ -3134,19 +3134,6 @@ static int amd_iommu_def_domain_type(struct device *dev) if (dev_is_pci(dev) && to_pci_dev(dev)->untrusted) return IOMMU_DOMAIN_DMA; - /* - * Do not identity map IOMMUv2 capable devices when: - * - memory encryption is active, because some of those devices - * (AMD GPUs) don't have the encryption bit in their DMA-mask - * and require remapping. - * - SNP is enabled, because it prohibits DTE[Mode]=0. - */ - if (pdev_pasid_supported(dev_data) && - !cc_platform_has(CC_ATTR_MEM_ENCRYPT) && - !amd_iommu_snp_en) { - return IOMMU_DOMAIN_IDENTITY; - } - return 0; }The flicking is reminencent of issues moving BOs between VRAM and GTT from a few years ago.
Quoted 1 line
So I don't think it can be dropped (just) yet.
Based on discussion so far in this thread and the link Mario shared earlier [1]:
- We still need identity mapping for APUs. I will fine tune the patch based on
the discussion in this thread.
- Its fine to drop SNP check
- I am still not sure its safe to drop SME check (CC_ATTR_MEM_ENCRYPT) Because here were are putting all APUs in identity mode.
Commit ea68573d408f explicitly error out for RAVEN and some of the older APU
platform didn't support SME. But not sure about newer ones.
So I am inclined to keep SME check for now. Once this patch stabilized we can revisit the SME check?
[1] https://lore.kernel.org/all/[email protected]/
-Vasant
Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
Quoted 3 lines · Vasant Hegde
Based on discussion so far in this thread and the link Mario shared earlier [1]:
- We still need identity mapping for APUs. I will fine tune the patch based on
the discussion in this thread.
IMHO if certain HW has broken support for PAGING (eg because it can't handle the isochronous needs) then it should be quirked and blocked properly.
I thought the issue here was the PASID check forcing identity and accidently capturing these raven chips too? It should just be moved out to a normal quirk..
Quoted 5 lines
- Its fine to drop SNP check
- I am still not sure its safe to drop SME check (CC_ATTR_MEM_ENCRYPT) Because here were are putting all APUs in identity mode.
Commitea68573d408fexplicitly error out for RAVEN and some of the older APU
platform didn't support SME. But not sure about newer ones.
That's quite a hacky check to make in that commit..
Quoted 2 lines
So I am inclined to keep SME check for now. Once this patch stabilized we can revisit the SME check?
The SME check makes no sense either, but if you want to stage things I'm OK with it.
Jason
Vasant Hegde
maintainerRe: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
Jason,
Quoted 20 lines · Jason Gunthorpe
Quoted 3 lines · Vasant Hegde
Based on discussion so far in this thread and the link Mario shared earlier [1]:
- We still need identity mapping for APUs. I will fine tune the patch based on
the discussion in this thread.IMHO if certain HW has broken support for PAGING (eg because it can't handle the isochronous needs) then it should be quirked and blocked properly.
I thought the issue here was the PASID check forcing identity and accidently capturing these raven chips too? It should just be moved out to a normal quirk..
Quoted 5 lines
- Its fine to drop SNP check
- I am still not sure its safe to drop SME check (CC_ATTR_MEM_ENCRYPT) Because here were are putting all APUs in identity mode.
Commitea68573d408fexplicitly error out for RAVEN and some of the older APU
platform didn't support SME. But not sure about newer ones.
Quoted 7 lines
That's quite a hacky check to make in that commit..
Quoted 2 lines
So I am inclined to keep SME check for now. Once this patch stabilized we can revisit the SME check?
The SME check makes no sense either, but if you want to stage things I'm OK with it.
Right. Eventually it should remove SME as well. But I am concerned about any regression. Also I don't have all these devices to test it. Hence for now I will limit to APUs, keep SME check and drop SNP check. Once it settle, we can go with next step.
I will respin the patch later this week.
-Vasant
RE: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
AMD General
Quoted 45 lines
-----Original Message-----
From: Hegde, Vasant <[email protected]>
Sent: Wednesday, August 5, 2026 10:38 AM
To: Jason Gunthorpe <[email protected]>
Cc: Limonciello, Mario <[email protected]>; Kuehling, Felix <[email protected]>; Deucher, Alexander <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Suthikulpanit, Suravee <[email protected]>; [email protected]; Longia, Amandeep Kaur <[email protected]>
Subject: Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs onlyJason,
Quoted 3 lines · Jason Gunthorpe
Quoted 1 line · Vasant Hegde
Based on discussion so far in this thread and the link Mario shared earlier
[1]:
Quoted 16 lines
Quoted 2 lines
- We still need identity mapping for APUs. I will fine tune the
patch based on the discussion in this thread.IMHO if certain HW has broken support for PAGING (eg because it can't handle the isochronous needs) then it should be quirked and blocked properly.
I thought the issue here was the PASID check forcing identity and accidently capturing these raven chips too? It should just be moved out to a normal quirk..
Quoted 5 lines
- Its fine to drop SNP check
- I am still not sure its safe to drop SME check (CC_ATTR_MEM_ENCRYPT) Because here were are putting all APUs in identity mode. Commitea68573d408fexplicitly error out for RAVEN and some of
the older APU platform didn't support SME. But not sure about newerones.
Quoted 6 lines
That's quite a hacky check to make in that commit..
Quoted 2 lines
So I am inclined to keep SME check for now. Once this patch stabilized we can revisit the SME check?
Quoted 7 lines
Quoted 2 lines
The SME check makes no sense either, but if you want to stage things I'm OK with it.
Right. Eventually it should remove SME as well. But I am concerned about any regression. Also I don't have all these devices to test it. Hence for now I will limit to APUs, keep SME check and drop SNP check. Once it settle, we can go with next step.
IIRC, they supported SME, but it wasn't usable due to other limitations (e.g., the requirement for identity mappings due to the display hw bug and the latency impact of encryption for displays in system memory). Do you could effectively pick either SME or displays.
Alex
Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
Quoted 5 lines · Alexander
IIRC, they supported SME, but it wasn't usable due to other limitations (e.g., the requirement for identity mappings due to the display hw bug and the latency impact of encryption for displays in system memory). Do you could effectively pick either SME or displays.
IMHO the GPU driver should fail if it cannot work because it can't reach the IOVA space requried for SME ...
I don't think this should be a iommu problem to deal with
Jason
RE: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
Public
Quoted 20 lines
-----Original Message-----
From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, August 5, 2026 1:53 PM
To: Deucher, Alexander <[email protected]>
Cc: Hegde, Vasant <[email protected]>; Limonciello, Mario <[email protected]>; Kuehling, Felix <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Suthikulpanit, Suravee <[email protected]>; [email protected]; Longia, Amandeep Kaur <[email protected]>
Subject: Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs onlyQuoted 4 lines · Alexander
IIRC, they supported SME, but it wasn't usable due to other limitations (e.g., the requirement for identity mappings due to the display hw bug and the latency impact of encryption for displays in system memory). Do you could effectively pick either SME or displays.
Quoted 4 lines
IMHO the GPU driver should fail if it cannot work because it can't reach the IOVA space requried for SME ...
I don't think this should be a iommu problem to deal with
That means no display for the user which is a not a good user experience. Most users of systems with integrated graphics would probably prefer displays over SME.
Alex
Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
Quoted 7 lines · Alexander
Quoted 7 lines · Alexander
Quoted 5 lines
IIRC, they supported SME, but it wasn't usable due to other limitations (e.g., the requirement for identity mappings due to the display hw bug and the latency impact of encryption for displays in system memory). Do you could effectively pick either SME or displays.
Quoted 8 lines
Quoted 4 lines
IMHO the GPU driver should fail if it cannot work because it can't reach the IOVA space requried for SME ...
I don't think this should be a iommu problem to deal with
That means no display for the user which is a not a good user experience. Most users of systems with integrated graphics would probably prefer displays over SME.
You said IOVA translation doesn't work at all on these devices due to a bug, so what is the iommu even supposed to do? It can't select identity because the device doesn't work that way and it can't select paging because it doesn't work that way either.
Or are you saying paging sort of works enough to do something half-busted?
Why did AMD even offer SME in the BIOS on these systems if it doesn't even work and turn it into Linux's problem?
Jason
RE: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
AMD General
Quoted 32 lines
-----Original Message-----
From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, August 5, 2026 3:11 PM
To: Deucher, Alexander <[email protected]>
Cc: Hegde, Vasant <[email protected]>; Limonciello, Mario <[email protected]>; Kuehling, Felix <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Suthikulpanit, Suravee <[email protected]>; [email protected]; Longia, Amandeep Kaur <[email protected]>
Subject: Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs onlyQuoted 6 lines · Alexander
Quoted 6 lines · Alexander
Quoted 5 lines
IIRC, they supported SME, but it wasn't usable due to other limitations (e.g., the requirement for identity mappings due to the display hw bug and the latency impact of encryption for displays in system memory). Do you could effectively pick either SME or
displays.
Quoted 9 lines
Quoted 5 lines
IMHO the GPU driver should fail if it cannot work because it can't reach the IOVA space requried for SME ...
I don't think this should be a iommu problem to deal with
That means no display for the user which is a not a good user experience. Most users of systems with integrated graphics would probably prefer displays over SME.
Quoted 6 lines
You said IOVA translation doesn't work at all on these devices due to a bug, so what is the iommu even supposed to do? It can't select identity because the device doesn't work that way and it can't select paging because it doesn't work that way either.
Or are you saying paging sort of works enough to do something half-busted?
IOVAs worked fine for everything on the GPU (gfx, video enc/dec, etc.) except for displays. You could avoid the problem by limiting displays to carve out, but many of these platforms shipped with so little carve out that that it wasn't feasible to use carve out for displays beyond the UEFI splash screen. That's what we did on Linux initially until lots of OEMs started shipping platforms with minimal carve out and no way to change it.
Quoted 3 lines
Why did AMD even offer SME in the BIOS on these systems if it doesn't even work and turn it into Linux's problem?
These were client products running windows and windows enabled neither IOMMU nor SME at the time.
Alex
Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
Quoted 41 lines · Alexander
AMD General
Quoted 32 lines
-----Original Message-----
From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, August 5, 2026 3:11 PM
To: Deucher, Alexander <[email protected]>
Cc: Hegde, Vasant <[email protected]>; Limonciello, Mario <[email protected]>; Kuehling, Felix <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Suthikulpanit, Suravee <[email protected]>; [email protected]; Longia, Amandeep Kaur <[email protected]>
Subject: Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs onlyQuoted 6 lines · Alexander
Quoted 6 lines · Alexander
Quoted 5 lines
IIRC, they supported SME, but it wasn't usable due to other limitations (e.g., the requirement for identity mappings due to the display hw bug and the latency impact of encryption for displays in system memory). Do you could effectively pick either SME or
displays.
Quoted 9 lines
Quoted 5 lines
IMHO the GPU driver should fail if it cannot work because it can't reach the IOVA space requried for SME ...
I don't think this should be a iommu problem to deal with
That means no display for the user which is a not a good user experience. Most users of systems with integrated graphics would probably prefer displays over SME.
Quoted 6 lines
You said IOVA translation doesn't work at all on these devices due to a bug, so what is the iommu even supposed to do? It can't select identity because the device doesn't work that way and it can't select paging because it doesn't work that way either.
Or are you saying paging sort of works enough to do something half-busted?
Quoted 7 lines
IOVAs worked fine for everything on the GPU (gfx, video enc/dec, etc.) except for displays. You could avoid the problem by limiting displays to carve out, but many of these platforms shipped with so little carve out that that it wasn't feasible to use carve out for displays beyond the UEFI splash screen. That's what we did on Linux initially until lots of OEMs started shipping platforms with minimal carve out and no way to change it.
Sure, but the end result is the GPU and SME are inherently incompatible, we can't really fix it.
What's wrong with blowing up at boot with a clear message that Linux must have SME disabled on this HW printed to the UEFI console?
Jason
RE: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
Public
Quoted 67 lines
-----Original Message-----
From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, August 12, 2026 4:58 PM
To: Deucher, Alexander <[email protected]>
Cc: Hegde, Vasant <[email protected]>; Limonciello, Mario <[email protected]>; Kuehling, Felix <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Suthikulpanit, Suravee <[email protected]>; [email protected]; Longia, Amandeep Kaur <[email protected]>
Subject: Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs onlyQuoted 19 lines · Alexander
AMD General
Quoted 17 lines
-----Original Message-----
From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, August 5, 2026 3:11 PM
To: Deucher, Alexander <[email protected]>
Cc: Hegde, Vasant <[email protected]>; Limonciello, Mario <[email protected]>; Kuehling, Felix <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Suthikulpanit, Suravee <[email protected]>; [email protected]; Longia, Amandeep Kaur <[email protected]>
Subject: Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs onlyQuoted 1 line · Alexander
Quoted 1 line
On Wed, Aug 05, 2026 at 05:22:22PM +0000, Deucher, Alexander
Quoted 17 lines · wrote:
Quoted 17 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
IIRC, they supported SME, but it wasn't usable due to other limitations (e.g., the requirement for identity mappings due to the display hw bug and the latency impact of encryption for displays in system memory). Do you could effectively pick either SME or
displays.
Quoted 9 lines
Quoted 5 lines
IMHO the GPU driver should fail if it cannot work because it can't reach the IOVA space requried for SME ...
I don't think this should be a iommu problem to deal with
That means no display for the user which is a not a good user experience. Most users of systems with integrated graphics would probably prefer displays over SME.
Quoted 6 lines
Quoted 6 lines
You said IOVA translation doesn't work at all on these devices due to a bug, so what is the iommu even supposed to do? It can't select identity because the device doesn't work that way and it can't select paging because it doesn't work that way either.
Or are you saying paging sort of works enough to do something half-
busted?
Quoted 8 lines
IOVAs worked fine for everything on the GPU (gfx, video enc/dec, etc.) except for displays. You could avoid the problem by limiting displays to carve out, but many of these platforms shipped with so little carve out that that it wasn't feasible to use carve out for displays beyond the UEFI splash screen. That's what we did on Linux initially until lots of OEMs started shipping platforms with minimal carve out and no way to change it.
Quoted 5 lines
Sure, but the end result is the GPU and SME are inherently incompatible, we can't really fix it.
What's wrong with blowing up at boot with a clear message that Linux must have SME disabled on this HW printed to the UEFI console?
It's not very user friendly, plus it would break working systems. I'm not sure the average user would know what to do. You aren't likely to want SME in the first place on one of these laptops. Why not flip the logic and print a clear message that if the user wants SME they have to explicitly enable it via the kernel command line.
Alex
Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
Quoted 7 lines · Alexander
Quoted 5 lines
Sure, but the end result is the GPU and SME are inherently incompatible, we can't really fix it.
What's wrong with blowing up at boot with a clear message that Linux must have SME disabled on this HW printed to the UEFI console?
It's not very user friendly, plus it would break working systems.
Huh? You've explained that SME doesn't work with these GPUs? How are there any working systems with SME turned on already running Linux?
Quoted 1 line
I'm not sure the average user would know what to do.
Turn off SME in the bios? Doesn't it default to off?
Quoted 4 lines
You aren't likely to want SME in the first place on one of these laptops. Why not flip the logic and print a clear message that if the user wants SME they have to explicitly enable it via the kernel command line.
That seems like it would be API breaking for server users that do want SME
Jason
RE: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
AMD General
Quoted 17 lines
-----Original Message-----
From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, August 12, 2026 7:27 PM
To: Deucher, Alexander <[email protected]>
Cc: Hegde, Vasant <[email protected]>; Limonciello, Mario <[email protected]>; Kuehling, Felix <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Suthikulpanit, Suravee <[email protected]>; [email protected]; Longia, Amandeep Kaur <[email protected]>
Subject: Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs onlyQuoted 3 lines · Alexander
Quoted 3 lines
Sure, but the end result is the GPU and SME are inherently incompatible, we can't really fix it.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
What's wrong with blowing up at boot with a clear message that Linux must have SME disabled on this HW printed to the UEFI console?
It's not very user friendly, plus it would break working systems.
Huh? You've explained that SME doesn't work with these GPUs? How are there any working systems with SME turned on already running Linux?
No one noticed it because the IOMMU driver used to default to 1:1 mapping for all AMD GPUs that supported ATS and PRI for compatibility with ROCm at the time back when the amd IOMMU was the only one that supported SVA. Same reason we didn't notice the IOMMU problems with the display hardware. Linux defaulted to 1:1 and windows didn't use the IOMMU.
Quoted 4 lines
Quoted 2 lines
I'm not sure the average user would know what to do.
Turn off SME in the bios? Doesn't it default to off?
I don't remember. I suspect it defaulted to on.
Quoted 7 lines
Quoted 5 lines
You aren't likely to want SME in the first place on one of these
laptops. Why not flip the logic and print a clear message that if the user wants SME they have to explicitly enable it via the kernel command line.That seems like it would be API breaking for server users that do want SME
Only disable it by default for these specific platforms with the old iGPUs.
Alex
Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
Quoted 5 lines · Alexander
Quoted 2 lines
Huh? You've explained that SME doesn't work with these GPUs? How are there any working systems with SME turned on already running Linux?
No one noticed it because the IOMMU driver used to default to 1:1 mapping for all AMD GPUs that supported ATS and PRI for
That's back to the original point, you should just quirk this broken GPU to require 1:1 and that's the right answer
I thought you had said even that does not work with SME because of some HW reason?
Jason
RE: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
In reply to Mario LimoncielloPublic
Quoted 105 lines
-----Original Message-----
From: Limonciello, Mario <[email protected]>
Sent: Tuesday, July 28, 2026 3:58 PM
To: Kuehling, Felix <[email protected]>; Hegde, Vasant <[email protected]>; Jason Gunthorpe <[email protected]>; Deucher, Alexander <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Suthikulpanit, Suravee <[email protected]>; [email protected]; Longia, Amandeep Kaur <[email protected]>
Subject: Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs onlyQuoted 14 lines · Felix
Quoted 13 lines · Mario Limonciello
+Felix
Quoted 10 lines · Vasant Hegde
Mario,
Quoted 5 lines · Mario Limonciello
Quoted 2 lines · Vasant Hegde
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
.../...
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I think this can probably be dropped now? There is no intersection of this old broken embedded GPU and a CPU that
supports CC right?
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
I did consider dropping these two checks, but I was not entirely sure. So I kept as is for now. Once these series settles and I can go with next step of dropping these two checks.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
@Alex, @Mario, Any suggestions ?
The background comes from this series:
Quoted 18 lines
Quoted 18 lines
Quoted 3 lines
Thanks for old link. So it looks like we are good to drop SME check?
Yeah I think so.
Quoted 7 lines
Also is it safe to assume SNP is not enabled on these platform and drop that check? Anyway SNP requirement (which doesn't support identity
mapping) and APU
requirement (needs identity mapping) is conflicting.The current platforms I think this is a safe assumption.
Felix,
Quoted 9 lines
Quoted 2 lines
Can we revisit the requirement of identity mapping for APU? Does it still hold with the current hardware and software?
I'm not sure what this means. Is Identity mode something like pass- through? I don't know why this is required on APUs. If it's related to IOMMUv2 support, then it's probably no longer needed. We're not using IOMMUv2 on our APUs any more. Support for that was removed from the amdgpu driver a few years ago.
Hmm, I just had a try with this on a few years old APU (PHX). It's flicker central with this (7.2-rc5~ish tree):
drivers/iommu/amd/iommu.c +0
-13@@ -3134,19 +3134,6 @@ static int amd_iommu_def_domain_type(struct device *dev)if (dev_is_pci (dev) &&to_pci_dev (dev)->untrusted)return IOMMU_DOMAIN_DMA ; -/* -* Do not identity map IOMMUv2 capable devices when: -* - memory encryption is active, because some of those devices -* (AMD GPUs) don't have the encryption bit in their DMA-mask -* and require remapping. -* - SNP is enabled, because it prohibits DTE[Mode]=0. -*/ -if (pdev_pasid_supported (dev_data) && - !cc_platform_has (CC_ATTR_MEM_ENCRYPT ) && - !amd_iommu_snp_en) { -return IOMMU_DOMAIN_IDENTITY ; - } -return 0 ; }
Quoted 4 lines
The flicking is reminencent of issues moving BOs between VRAM and GTT from a few years ago.
So I don't think it can be dropped (just) yet.
Make sure ATS is enabled. Display buffers in system memory are really sensitive to latency. Some of the older pre-raven chips like carrizo would flicker even with an identify mapping.
Alex
RE: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only
In reply to Mario LimoncielloAMD General
Quoted 58 lines
-----Original Message-----
From: Limonciello, Mario <[email protected]>
Sent: Tuesday, July 28, 2026 1:31 AM
To: Hegde, Vasant <[email protected]>; Jason Gunthorpe <[email protected]>; Deucher, Alexander <[email protected]>; Kuehling, Felix <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Suthikulpanit, Suravee <[email protected]>; [email protected]; Longia, Amandeep Kaur <[email protected]>
Subject: Re: [PATCH 2/2] iommu/amd: Force identity mode for selected GPUs only+Felix
Quoted 24 lines · Vasant Hegde
Mario,
Quoted 5 lines · Mario Limonciello
Quoted 2 lines · Vasant Hegde
.../...
Quoted 12 lines
Quoted 11 lines
Quoted 3 lines
I think this can probably be dropped now? There is no intersection of this old broken embedded GPU and a CPU that supports CC right?
I did consider dropping these two checks, but I was not entirely sure. So I kept as is for now. Once these series settles and I can go with next step of dropping these two checks.
@Alex, @Mario, Any suggestions ?
Quoted 6 lines
Quoted 3 lines
The background comes from this series:
Thanks for old link. So it looks like we are good to drop SME check?
Yeah I think so.
Quoted 4 lines
Also is it safe to assume SNP is not enabled on these platform and drop that check? Anyway SNP requirement (which doesn't support identity mapping) and APU requirement (needs identity mapping) is
conflicting.
Quoted 1 line
The current platforms I think this is a safe assumption.
Quoted 7 lines
Felix,
Can we revisit the requirement of identity mapping for APU? Does it still hold with the current hardware and software?
I'm wondering if we can just apply it to Raven and loosen up elsewhere and what gives if we do that.
This issue only applies to the early raven/picasso families. The display hardware had some hardware addressing bug with IOVAs so it has to have IOVA = IOPA. The rest of the GPU can handle IOVAs just fine. This was fixed on Renoir IIRC.
Alex