Skip to contentSkip to conversation

x86/iommu: Drop IOMMU support when cx16 isn't supported

PATCH Superseded Teddy Astie 2 participants 7 messages Apr 12, 2024

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v3 cover x86/iommu: Drop IOMMU support when cx16 isn't supported

All hardware that supports VT-d/AMD-Vi that exists also supports cx16 (aside specifically crafted virtual machines).

Some IOMMU code paths in Xen consider cases where VT-d/AMD-Vi is supported while cx16 isn't, those paths may be bugged and are barely tested, dead code in practice.

Disable IOMMU in case we have IOMMU hardware but no cx16, then cleanup no-cx16 handling logic from VT-d and AMD-Vi drivers.

Teddy

Changed in v2:

* Added cleanup no-cx16 code for x2APIC
* Fixed commit and code formatting
* Added missing Suggested-by note

Changed in v3:

* Use -ENODEV instead of -ENOSYS.

Teddy Astie (3):
VT-d: Disable IOMMU if cx16 isn't supported
AMD-Vi: Disable IOMMU if cx16 isn't supported
VT-d: Cleanup MAP_SINGLE_DEVICE and related code

 xen/arch/x86/apic.c                         |  6 ++
 xen/drivers/passthrough/amd/iommu_map.c     | 42 ++++------
 xen/drivers/passthrough/amd/pci_amd_iommu.c |  6 ++
 xen/drivers/passthrough/vtd/intremap.c      | 65 ++++-----------
 xen/drivers/passthrough/vtd/iommu.c         | 92 +++++++--------------
 xen/drivers/passthrough/vtd/vtd.h           |  5 +-
 6 files changed, 71 insertions(+), 145 deletions(-)
Signature · 9 lines
2.44.0



Teddy Astie | Vates XCP-ng Intern

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#1 lore raw

PATCH v3 1/3 VT-d: Disable IOMMU if cx16 isn't supported

No hardware has VT-d support while not having cx16 support, disable IOMMU in this case to avoid potentially buggy code.

Now that IOMMU is only enabled if cx16 is supported, drop dead code that handles cases where cx16 isn't supported.

---
 xen/arch/x86/apic.c                    |  6 ++
 xen/drivers/passthrough/vtd/intremap.c | 65 +++++----------------
 xen/drivers/passthrough/vtd/iommu.c    | 80 +++++++++-----------------
 3 files changed, 46 insertions(+), 105 deletions(-)
xen/arch/x86/apic.c +6 -0
index 592b78e11e..91d7f2b248 100644
@@ -836,6 +836,12 @@ void __init x2apic_bsp_setup(void)
     if ( !cpu_has_x2apic )
         return;
 
+    if ( unlikely(!cpu_has_cx16) )
+    {
+        printk("x2APIC: CPU doesn't support CMPXCHG16B, disabling\n");
+        return;
+    }
+
     if ( !opt_x2apic )
     {
         if ( !x2apic_enabled )
xen/drivers/passthrough/vtd/intremap.c +14 -51

89 lines not on this page · raw message

xen/drivers/passthrough/vtd/iommu.c +26 -54

106 lines not on this page · raw message

Signature · 9 lines
2.44.0



Teddy Astie | Vates XCP-ng Intern

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#2 lore raw

PATCH v3 2/3 AMD-Vi: Disable IOMMU if cx16 isn't supported

In reply to Teddy Astie

No hardware has AMD-Vi support while not having cx16 support, disable IOMMU in this case to avoid potentially buggy code.

Now that IOMMU is only enabled if cx16 is supported, drop dead code that handles cases where cx16 isn't supported.

---
 xen/drivers/passthrough/amd/iommu_map.c     | 42 +++++++--------------
 xen/drivers/passthrough/amd/pci_amd_iommu.c |  6 +++
 2 files changed, 20 insertions(+), 28 deletions(-)
xen/drivers/passthrough/amd/iommu_map.c +14 -28
index e0f4fe736a..f67975e700 100644
@@ -167,15 +167,14 @@ int amd_iommu_set_root_page_table(struct amd_iommu_dte *dte,
 {
     bool valid = flags & SET_ROOT_VALID;
 
-    if ( dte->v && dte->tv &&
-         (cpu_has_cx16 || (flags & SET_ROOT_WITH_UNITY_MAP)) )
+    if ( dte->v && dte->tv )
     {
         union {
             struct amd_iommu_dte dte;
             uint64_t raw64[4];
             __uint128_t raw128[2];
         } ldte = { .dte = *dte };
-        __uint128_t old = ldte.raw128[0];
+        __uint128_t res, old = ldte.raw128[0];
         int ret = 0;
 
         ldte.dte.domain_id = domain_id;
@@ -185,33 +184,20 @@ int amd_iommu_set_root_page_table(struct amd_iommu_dte *dte,
         ldte.dte.paging_mode = paging_mode;
         ldte.dte.v = valid;
 
-        if ( cpu_has_cx16 )
-        {
-            __uint128_t res = cmpxchg16b(dte, &old, &ldte.raw128[0]);
+        res = cmpxchg16b(dte, &old, &ldte.raw128[0]);
 
-            /*
-             * Hardware does not update the DTE behind our backs, so the
-             * return value should match "old".
-             */
-            if ( res != old )
-            {
-                printk(XENLOG_ERR
-                       "Dom%d: unexpected DTE %016lx_%016lx (expected %016lx_%016lx)\n",
-                       domain_id,
-                       (uint64_t)(res >> 64), (uint64_t)res,
-                       (uint64_t)(old >> 64), (uint64_t)old);
-                ret = -EILSEQ;
-            }
-        }
-        else /* Best effort, updating domain_id last. */
+        /*
+         * Hardware does not update the DTE behind our backs, so the
+         * return value should match "old".
+         */
+        if ( res != old )
         {
-            uint64_t *ptr = (void *)dte;
-
-            write_atomic(ptr + 0, ldte.raw64[0]);
-            /* No barrier should be needed between these two. */
-            write_atomic(ptr + 1, ldte.raw64[1]);
-
-            ret = 1;
+            printk(XENLOG_ERR
+                   "Dom%d: unexpected DTE %016lx_%016lx (expected %016lx_%016lx)\n",
+                   domain_id,
+                   (uint64_t)(res >> 64), (uint64_t)res,
+                   (uint64_t)(old >> 64), (uint64_t)old);
+            ret = -EILSEQ;
         }
 
         return ret;
xen/drivers/passthrough/amd/pci_amd_iommu.c +6 -0
index f6efd88e36..a213dbea0b 100644
@@ -312,6 +312,12 @@ static int __init cf_check iov_detect(void)
         return -ENODEV;
     }
 
+    if ( unlikely(!cpu_has_cx16) )
+    {
+        printk("AMD-Vi: CPU doesn't support CMPXCHG16B, disabling\n");
+        return -ENODEV;
+    }
+
     init_done = 1;
 
     if ( !amd_iommu_perdev_intremap )
Signature · 9 lines
2.44.0



Teddy Astie | Vates XCP-ng Intern

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#3 lore raw

PATCH v3 3/3 VT-d: Cleanup MAP_SINGLE_DEVICE and related code

In reply to Teddy Astie

This flag was only used in case cx16 is not available, as those code paths no longer exist, this flag now does basically nothing.

---
 xen/drivers/passthrough/vtd/iommu.c | 12 +++---------
 xen/drivers/passthrough/vtd/vtd.h   |  5 ++---
 2 files changed, 5 insertions(+), 12 deletions(-)
xen/drivers/passthrough/vtd/iommu.c +3 -9
index ef9380ed6a..a1bd3c5ff6 100644
@@ -1692,15 +1692,9 @@ static int domain_context_mapping(struct domain *domain, u8 devfn,
         break;
     }
 
-    if ( domain != pdev->domain && pdev->domain != dom_io )
-    {
-        if ( pdev->domain->is_dying )
-            mode |= MAP_OWNER_DYING;
-        else if ( drhd &&
-                  !any_pdev_behind_iommu(pdev->domain, pdev, drhd->iommu) &&
-                  !pdev->phantom_stride )
-            mode |= MAP_SINGLE_DEVICE;
-    }
+    if ( domain != pdev->domain && pdev->domain != dom_io &&
+         pdev->domain->is_dying )
+        mode |= MAP_OWNER_DYING;
 
     switch ( pdev->type )
     {
xen/drivers/passthrough/vtd/vtd.h +2 -3
index cb2df76eed..43f06a353d 100644
@@ -28,9 +28,8 @@
  */
 #define MAP_WITH_RMRR         (1u << 0)
 #define MAP_OWNER_DYING       (1u << 1)
-#define MAP_SINGLE_DEVICE     (1u << 2)
-#define MAP_ERROR_RECOVERY    (1u << 3)
-#define UNMAP_ME_PHANTOM_FUNC (1u << 4)
+#define MAP_ERROR_RECOVERY    (1u << 2)
+#define UNMAP_ME_PHANTOM_FUNC (1u << 3)
 
 /* Allow for both IOAPIC and IOSAPIC. */
 #define IO_xAPIC_route_entry IO_APIC_route_entry
Signature · 9 lines
2.44.0



Teddy Astie | Vates XCP-ng Intern

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#4 lore raw
In reply to Teddy Astie
Quoted 14 lines · Teddy Astie

All hardware that supports VT-d/AMD-Vi that exists also supports cx16 (aside specifically crafted virtual machines).

Some IOMMU code paths in Xen consider cases where VT-d/AMD-Vi is supported while cx16 isn't, those paths may be bugged and are barely tested, dead code in practice.

Disable IOMMU in case we have IOMMU hardware but no cx16, then cleanup no-cx16 handling logic from VT-d and AMD-Vi drivers.

Teddy

Changed in v2:

Quoted 7 lines

* Added cleanup no-cx16 code for x2APIC
* Fixed commit and code formatting
* Added missing Suggested-by note

Changed in v3:

* Use -ENODEV instead of -ENOSYS.

Hmm - I've still got a double copy of patch 2.  No idea what's going on, but I'll make do with what b4 thinks is on the list.

A couple of things.

1) You introduced trailing whitespace in patch 1 in the middle line here:

Quoted 1 line

+ ASSERT(spin_is_locked(&iommu->intremap.lock)); + + old_ire = *entry;

2) In your commit messages, the grammar is a bit awkward.  It would be clearer to say:

"All hardware with VT-d/AMD-Vi has CMPXCHG16 support.  Check this at initialisation time, and remove the effectively-dead logic for the non-cx16 case."

just as you've done in the cover letter.

3) In patch 1, you shouldn't modify x2apic_bsp_setup() like that. x2APIC && no-IOMMU is a legal combination.

Instead, you should put a cx16 check in both driver's supports_x2apic() hooks.

4) In patch 3, you should drop the Suggested-by me.  You found that one all yourself.

~Andrew

#5 lore raw

Le 12/04/2024 à 16:49, Andrew Cooper a écrit :

Quoted 3 lines

1) You introduced trailing whitespace in patch 1 in the middle line here:

Quoted 1 line

+ ASSERT(spin_is_locked(&iommu->intremap.lock)); + + old_ire = *entry;

Good catch, will fix

Quoted 8 lines

2) In your commit messages, the grammar is a bit awkward.  It would be clearer to say:

"All hardware with VT-d/AMD-Vi has CMPXCHG16 support.  Check this at initialisation time, and remove the effectively-dead logic for the non-cx16 case."

just as you've done in the cover letter.

Yes

Quoted 5 lines

3) In patch 1, you shouldn't modify x2apic_bsp_setup() like that. x2APIC && no-IOMMU is a legal combination.

Instead, you should put a cx16 check in both driver's supports_x2apic() hooks.

In this case, you mean both intel_iommu_supports_eim and iov_supports_xt (AMD) ?

Quoted 4 lines

4) In patch 3, you should drop the Suggested-by me.  You found that one all yourself.

Will change this.

Teddy

---

Teddy Astie | Vates XCP-ng Intern

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

#6 lore raw
Quoted 8 lines · Teddy Astie

Le 12/04/2024 à 16:49, Andrew Cooper a écrit :

Quoted 5 lines

3) In patch 1, you shouldn't modify x2apic_bsp_setup() like that. x2APIC && no-IOMMU is a legal combination.

Instead, you should put a cx16 check in both driver's supports_x2apic() hooks.

In this case, you mean both intel_iommu_supports_eim and iov_supports_xt (AMD) ?

Yes, with the changes in the appropriate patch.

~Andrew