Skip to contentSkip to conversation

x86/iommu: Drop IOMMU support when cx16 isn't supported

PATCH Superseded Teddy Astie 3 participants 8 messages Apr 8, 2024 – Apr 12, 2024

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v2 cover x86/iommu: Drop IOMMU support when cx16 isn't supported

All hardware that supports VT-d/AMD-Vi that exists also supports cx16 (aside specifically crafted virtual machines).

Some IOMMU code paths in Xen consider cases where VT-d/AMD-Vi is supported while cx16 isn't, those paths may be bugged and are barely tested, dead code in practice.

Disable IOMMU in case we have IOMMU hardware but no cx16, then cleanup no-cx16 handling logic from VT-d and AMD-Vi drivers.

Teddy

Changed in v2:

* Added cleanup no-cx16 code for x2APIC
* Fixed commit and code formatting
* Added missing Suggested-by note

Teddy Astie (3):
VT-d: Disable IOMMU if cx16 isn't supported
AMD-Vi: Disable IOMMU if cx16 isn't supported
VT-d: Cleanup MAP_SINGLE_DEVICE and related code

 xen/arch/x86/apic.c                         |  6 ++
 xen/drivers/passthrough/amd/iommu_map.c     | 42 ++++------
 xen/drivers/passthrough/amd/pci_amd_iommu.c |  6 ++
 xen/drivers/passthrough/vtd/intremap.c      | 65 ++++-----------
 xen/drivers/passthrough/vtd/iommu.c         | 92 +++++++--------------
 xen/drivers/passthrough/vtd/vtd.h           |  5 +-
 6 files changed, 71 insertions(+), 145 deletions(-)
Signature · 9 lines
2.44.0



Teddy Astie | Vates XCP-ng Intern

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#1 lore raw

PATCH v2 1/3 VT-d: Disable IOMMU if cx16 isn't supported

No hardware has VT-d support while not having cx16 support, disable IOMMU in this case to avoid potentially buggy code.

Now that IOMMU is only enabled if cx16 is supported, drop dead code that handles cases where cx16 isn't supported.

---
 xen/arch/x86/apic.c                    |  6 ++
 xen/drivers/passthrough/vtd/intremap.c | 65 +++++----------------
 xen/drivers/passthrough/vtd/iommu.c    | 80 +++++++++-----------------
 3 files changed, 46 insertions(+), 105 deletions(-)
xen/arch/x86/apic.c +6 -0
index 592b78e11e..91d7f2b248 100644
@@ -836,6 +836,12 @@ void __init x2apic_bsp_setup(void)
     if ( !cpu_has_x2apic )
         return;
 
+    if ( unlikely(!cpu_has_cx16) )
+    {
+        printk("x2APIC: CPU doesn't support CMPXCHG16B, disabling\n");
+        return;
+    }
+
     if ( !opt_x2apic )
     {
         if ( !x2apic_enabled )
xen/drivers/passthrough/vtd/intremap.c +14 -51

89 lines not on this page · raw message

xen/drivers/passthrough/vtd/iommu.c +26 -54

106 lines not on this page · raw message

Signature · 9 lines
2.44.0



Teddy Astie | Vates XCP-ng Intern

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#2 lore raw

PATCH v2 2/3 AMD-Vi: Disable IOMMU if cx16 isn't supported

In reply to Teddy Astie

No hardware has AMD-Vi support while not having cx16 support, disable IOMMU in this case to avoid potentially buggy code.

Now that IOMMU is only enabled if cx16 is supported, drop dead code that handles cases where cx16 isn't supported.

---
 xen/drivers/passthrough/amd/iommu_map.c     | 42 +++++++--------------
 xen/drivers/passthrough/amd/pci_amd_iommu.c |  6 +++
 2 files changed, 20 insertions(+), 28 deletions(-)
xen/drivers/passthrough/amd/iommu_map.c +14 -28
index e0f4fe736a..f67975e700 100644
@@ -167,15 +167,14 @@ int amd_iommu_set_root_page_table(struct amd_iommu_dte *dte,
 {
     bool valid = flags & SET_ROOT_VALID;
 
-    if ( dte->v && dte->tv &&
-         (cpu_has_cx16 || (flags & SET_ROOT_WITH_UNITY_MAP)) )
+    if ( dte->v && dte->tv )
     {
         union {
             struct amd_iommu_dte dte;
             uint64_t raw64[4];
             __uint128_t raw128[2];
         } ldte = { .dte = *dte };
-        __uint128_t old = ldte.raw128[0];
+        __uint128_t res, old = ldte.raw128[0];
         int ret = 0;
 
         ldte.dte.domain_id = domain_id;
@@ -185,33 +184,20 @@ int amd_iommu_set_root_page_table(struct amd_iommu_dte *dte,
         ldte.dte.paging_mode = paging_mode;
         ldte.dte.v = valid;
 
-        if ( cpu_has_cx16 )
-        {
-            __uint128_t res = cmpxchg16b(dte, &old, &ldte.raw128[0]);
+        res = cmpxchg16b(dte, &old, &ldte.raw128[0]);
 
-            /*
-             * Hardware does not update the DTE behind our backs, so the
-             * return value should match "old".
-             */
-            if ( res != old )
-            {
-                printk(XENLOG_ERR
-                       "Dom%d: unexpected DTE %016lx_%016lx (expected %016lx_%016lx)\n",
-                       domain_id,
-                       (uint64_t)(res >> 64), (uint64_t)res,
-                       (uint64_t)(old >> 64), (uint64_t)old);
-                ret = -EILSEQ;
-            }
-        }
-        else /* Best effort, updating domain_id last. */
+        /*
+         * Hardware does not update the DTE behind our backs, so the
+         * return value should match "old".
+         */
+        if ( res != old )
         {
-            uint64_t *ptr = (void *)dte;
-
-            write_atomic(ptr + 0, ldte.raw64[0]);
-            /* No barrier should be needed between these two. */
-            write_atomic(ptr + 1, ldte.raw64[1]);
-
-            ret = 1;
+            printk(XENLOG_ERR
+                   "Dom%d: unexpected DTE %016lx_%016lx (expected %016lx_%016lx)\n",
+                   domain_id,
+                   (uint64_t)(res >> 64), (uint64_t)res,
+                   (uint64_t)(old >> 64), (uint64_t)old);
+            ret = -EILSEQ;
         }
 
         return ret;
xen/drivers/passthrough/amd/pci_amd_iommu.c +6 -0
index f6efd88e36..656c5eda5d 100644
@@ -312,6 +312,12 @@ static int __init cf_check iov_detect(void)
         return -ENODEV;
     }
 
+    if ( unlikely(!cpu_has_cx16) )
+    {
+        printk("AMD-Vi: CPU doesn't support CMPXCHG16B, disabling\n");
+        return -ENOSYS;
+    }
+
     init_done = 1;
 
     if ( !amd_iommu_perdev_intremap )
Signature · 9 lines
2.44.0



Teddy Astie | Vates XCP-ng Intern

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#3 lore raw

PATCH v2 3/3 VT-d: Cleanup MAP_SINGLE_DEVICE and related code

In reply to Teddy Astie

This flag was only used in case cx16 is not available, as those code paths no longer exist, this flag now does basically nothing.

---
 xen/drivers/passthrough/vtd/iommu.c | 12 +++---------
 xen/drivers/passthrough/vtd/vtd.h   |  5 ++---
 2 files changed, 5 insertions(+), 12 deletions(-)
xen/drivers/passthrough/vtd/iommu.c +3 -9
index 47b56f37a9..4b15e6da79 100644
@@ -1692,15 +1692,9 @@ static int domain_context_mapping(struct domain *domain, u8 devfn,
         break;
     }
 
-    if ( domain != pdev->domain && pdev->domain != dom_io )
-    {
-        if ( pdev->domain->is_dying )
-            mode |= MAP_OWNER_DYING;
-        else if ( drhd &&
-                  !any_pdev_behind_iommu(pdev->domain, pdev, drhd->iommu) &&
-                  !pdev->phantom_stride )
-            mode |= MAP_SINGLE_DEVICE;
-    }
+    if ( domain != pdev->domain && pdev->domain != dom_io &&
+         pdev->domain->is_dying )
+        mode |= MAP_OWNER_DYING;
 
     switch ( pdev->type )
     {
xen/drivers/passthrough/vtd/vtd.h +2 -3
index cb2df76eed..43f06a353d 100644
@@ -28,9 +28,8 @@
  */
 #define MAP_WITH_RMRR         (1u << 0)
 #define MAP_OWNER_DYING       (1u << 1)
-#define MAP_SINGLE_DEVICE     (1u << 2)
-#define MAP_ERROR_RECOVERY    (1u << 3)
-#define UNMAP_ME_PHANTOM_FUNC (1u << 4)
+#define MAP_ERROR_RECOVERY    (1u << 2)
+#define UNMAP_ME_PHANTOM_FUNC (1u << 3)
 
 /* Allow for both IOAPIC and IOSAPIC. */
 #define IO_xAPIC_route_entry IO_APIC_route_entry
Signature · 9 lines
2.44.0



Teddy Astie | Vates XCP-ng Intern

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#4 lore raw
In reply to Teddy Astie
Quoted 23 lines · Teddy Astie

All hardware that supports VT-d/AMD-Vi that exists also supports cx16 (aside specifically crafted virtual machines).

Some IOMMU code paths in Xen consider cases where VT-d/AMD-Vi is supported while cx16 isn't, those paths may be bugged and are barely tested, dead code in practice.

Disable IOMMU in case we have IOMMU hardware but no cx16, then cleanup no-cx16 handling logic from VT-d and AMD-Vi drivers.

Teddy

Changed in v2:

* Added cleanup no-cx16 code for x2APIC
* Fixed commit and code formatting
* Added missing Suggested-by note

Teddy Astie (3):
VT-d: Disable IOMMU if cx16 isn't supported
AMD-Vi: Disable IOMMU if cx16 isn't supported
VT-d: Cleanup MAP_SINGLE_DEVICE and related code

Quoted 8 lines
 xen/arch/x86/apic.c                         |  6 ++
 xen/drivers/passthrough/amd/iommu_map.c     | 42 ++++------
 xen/drivers/passthrough/amd/pci_amd_iommu.c |  6 ++
 xen/drivers/passthrough/vtd/intremap.c      | 65 ++++-----------
 xen/drivers/passthrough/vtd/iommu.c         | 92 +++++++--------------
 xen/drivers/passthrough/vtd/vtd.h           |  5 +-
 6 files changed, 71 insertions(+), 145 deletions(-)

Sorry, but you've sent out two copies of each patch in this series, and it's not clear if they're identical or not.

Please could you send out another version, making sure there's only one of each patch.

Also, you need to swap ENOSYS with ENODEV, as per Jan's review on v1.

Thanks,

~Andrew

#5 lore raw
Quoted 2 lines · Andrew Cooper

Sorry, but you've sent out two copies of each patch in this series, and it's not clear if they're identical or not.

FWIW I've got just one copy.

Signature · 3 lines
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
#6 lore raw
In reply to Andrew Cooper

Le 11/04/2024 à 22:05, Andrew Cooper a écrit :

Quoted 36 lines · Teddy Astie
Quoted 23 lines

All hardware that supports VT-d/AMD-Vi that exists also supports cx16 (aside specifically crafted virtual machines).

Some IOMMU code paths in Xen consider cases where VT-d/AMD-Vi is supported while cx16 isn't, those paths may be bugged and are barely tested, dead code in practice.

Disable IOMMU in case we have IOMMU hardware but no cx16, then cleanup no-cx16 handling logic from VT-d and AMD-Vi drivers.

Teddy

Changed in v2:

* Added cleanup no-cx16 code for x2APIC
* Fixed commit and code formatting
* Added missing Suggested-by note

Teddy Astie (3):
VT-d: Disable IOMMU if cx16 isn't supported
AMD-Vi: Disable IOMMU if cx16 isn't supported
VT-d: Cleanup MAP_SINGLE_DEVICE and related code

Quoted 8 lines
  xen/arch/x86/apic.c                         |  6 ++
  xen/drivers/passthrough/amd/iommu_map.c     | 42 ++++------
  xen/drivers/passthrough/amd/pci_amd_iommu.c |  6 ++
  xen/drivers/passthrough/vtd/intremap.c      | 65 ++++-----------
  xen/drivers/passthrough/vtd/iommu.c         | 92 +++++++--------------
  xen/drivers/passthrough/vtd/vtd.h           |  5 +-
  6 files changed, 71 insertions(+), 145 deletions(-)

Sorry, but you've sent out two copies of each patch in this series, and it's not clear if they're identical or not.

Quoted 8 lines

Please could you send out another version, making sure there's only one of each patch.

Also, you need to swap ENOSYS with ENODEV, as per Jan's review on v1.

Thanks,

~Andrew

Hello,

Not entirely sure why it got sent twice, as marek said he only received it once. Will double-check next time to avoid this issue in case I wrongfully sent it twice.

Will also swap ENOSYS with ENODEV in the next version.

Thanks,

Teddy

Teddy Astie | Vates XCP-ng Intern

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

#7 lore raw
Quoted 41 lines · Teddy Astie

Le 11/04/2024 à 22:05, Andrew Cooper a écrit :

Quoted 40 lines · Teddy Astie
Quoted 23 lines

All hardware that supports VT-d/AMD-Vi that exists also supports cx16 (aside specifically crafted virtual machines).

Some IOMMU code paths in Xen consider cases where VT-d/AMD-Vi is supported while cx16 isn't, those paths may be bugged and are barely tested, dead code in practice.

Disable IOMMU in case we have IOMMU hardware but no cx16, then cleanup no-cx16 handling logic from VT-d and AMD-Vi drivers.

Teddy

Changed in v2:

* Added cleanup no-cx16 code for x2APIC
* Fixed commit and code formatting
* Added missing Suggested-by note

Teddy Astie (3):
VT-d: Disable IOMMU if cx16 isn't supported
AMD-Vi: Disable IOMMU if cx16 isn't supported
VT-d: Cleanup MAP_SINGLE_DEVICE and related code

Quoted 8 lines
  xen/arch/x86/apic.c                         |  6 ++
  xen/drivers/passthrough/amd/iommu_map.c     | 42 ++++------
  xen/drivers/passthrough/amd/pci_amd_iommu.c |  6 ++
  xen/drivers/passthrough/vtd/intremap.c      | 65 ++++-----------
  xen/drivers/passthrough/vtd/iommu.c         | 92 +++++++--------------
  xen/drivers/passthrough/vtd/vtd.h           |  5 +-
  6 files changed, 71 insertions(+), 145 deletions(-)

Sorry, but you've sent out two copies of each patch in this series, and it's not clear if they're identical or not.

Please could you send out another version, making sure there's only one of each patch.

Also, you need to swap ENOSYS with ENODEV, as per Jan's review on v1.

Quoted 8 lines
Quoted 3 lines

Thanks,

~Andrew

Hello,

Not entirely sure why it got sent twice, as marek said he only received it once. Will double-check next time to avoid this issue in case I wrongfully sent it twice.

Huh, lore agrees.  I seem to have both a direct and list copy which weren't correctly deduplicated.

Sorry for the noise.

Quoted 1 line

Will also swap ENOSYS with ENODEV in the next version.

Thanks.

~Andrew