Skip to contentSkip to conversation

iommu: fixes for interrupt remapping enabling

PATCH In review Roger Pau Monne 4 participants 13 messages Oct 15, 2019 – Oct 25, 2019

lore
Share & tools
lore
#0 lore raw

v3 cover iommu: fixes for interrupt remapping enabling

Hello,

The following series contain fixes for issues found when enabling interrupt remapping and the IO-APIC already has unmasked pins. While I'm not aware of any system malfunctioning (apart from reporting IOMMU interrupt remapping faults) I think it would be nice to have those in 4.13.

The series can also be found at:

git://xenbits.xen.org/people/royger/xen.git iommu_intr_v3

Thanks, Roger.

Roger Pau Monne (4):
iommu/amd: support all delivery modes with x2APIC
x2APIC: simplify resume
x2APIC: translate IO-APIC entries when enabling the IOMMU
iommu: translate IO-APIC pins when enabling interrupt remapping

 xen/arch/x86/apic.c                           |  38 ++---
 xen/arch/x86/io_apic.c                        |   5 +-
 xen/drivers/passthrough/amd/iommu_init.c      |  11 +-
 xen/drivers/passthrough/amd/iommu_intr.c      | 141 +++++++++---------
 xen/drivers/passthrough/x86/iommu.c           |  34 ++++-
 xen/include/asm-x86/hvm/svm/amd-iommu-proto.h |   1 +
 xen/include/asm-x86/io_apic.h                 |   3 +-
 7 files changed, 121 insertions(+), 112 deletions(-)
Signature · 7 lines
2.23.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw

PATCH v3 1/4 iommu/amd: support all delivery modes with x2APIC

Current AMD IOMMU code will attempt to create remapping entries for all IO-APIC pins, regardless of the delivery mode. AMD IOMMU implementation doesn't support remapping entries for IO-APIC pins with delivery mode set to SMI, MNI, INIT or ExtINT, instead those entries are not translated provided the right bits are set in the device table entry.

This change checks whether the device table entry has the correct bits set in order to delivery the requested interrupt or a warning message is printed. It also translates the 32bit destination field into a physical or logical IO-APIC entry format. Note that if the 32bit destination cannot fit into the legacy format a message is printed and the entry is masked.

---
My AMD hardware doesn't have any of such entries, hence I've only been able to compile test this change and assert it doesn't affect the functionality of other interrupt delivery modes.

---
 xen/drivers/passthrough/amd/iommu_intr.c | 81 ++++++++++++++++++++++++
 1 file changed, 81 insertions(+)
xen/drivers/passthrough/amd/iommu_intr.c +81 -0
index fb71073c84..02dd3737c7 100644
@@ -439,6 +439,80 @@ int __init amd_iommu_setup_ioapic_remapping(void)
     return 0;
 }
 
+void setup_forced_ioapic_rte(unsigned int apic, unsigned int pin,
+                             struct amd_iommu *iommu,
+                             struct IO_APIC_route_entry *rte)
+{
+    unsigned int idx = ioapic_id_to_index(IO_APIC_ID(apic));
+    struct amd_iommu_dte *table = iommu->dev_table.buffer;
+    unsigned int req_id, dest, offset;
+    union irte_ptr entry;
+
+    ASSERT(x2apic_enabled);
+
+    if ( idx == MAX_IO_APICS )
+    {
+        rte->mask = true;
+        return;
+    }
+
+    req_id = get_intremap_requestor_id(ioapic_sbdf[idx].seg,
+                                       ioapic_sbdf[idx].bdf);
+
+    switch ( rte->delivery_mode )
+    {
+    case dest_SMI:
+        break;
+
+#define DEL_CHECK(type, dte_field)                                      \
+    case dest_ ## type:                                                 \
+        if ( !table[req_id].dte_field )                                 \
+            printk(XENLOG_WARNING                                       \
+                   STR(type) " on IO-APIC %u pin %u will be aborted\n", \
+                   apic, pin);                                          \
+        break;
+
+    DEL_CHECK(NMI, nmi_pass);
+    DEL_CHECK(INIT, init_pass);
+    DEL_CHECK(ExtINT, ext_int_pass);
+#undef DEL_CHECK
+
+    default:
+        ASSERT_UNREACHABLE();
+        return;
+    }
+
+    offset = ioapic_sbdf[idx].pin_2_idx[pin];
+    if ( offset >= INTREMAP_MAX_ENTRIES )
+    {
+        rte->mask = true;
+        return;
+    }
+
+    entry = get_intremap_entry(iommu, req_id, offset);
+    dest = get_full_dest(entry.ptr128);
+
+#define SET_DEST(name, dest_mask) {                                            \
+    if ( dest & ~(dest_mask) )                                                 \
+    {                                                                          \
+        printk(XENLOG_WARNING                                                  \
+               "IO-APIC %u pin %u " STR(name) " destination (%x) > %x\n",      \
+               apic, pin, dest, dest_mask);                                    \
+        rte->mask = true;                                                      \
+        return;                                                                \
+    }                                                                          \
+    rte->dest.name.name ## _dest = dest;                                       \
+}
+
+    if ( rte->dest_mode )
+        SET_DEST(physical, 0xf)
+    else
+        SET_DEST(logical, 0xff)
+#undef SET_DEST
+
+    return;
+}
+
 void amd_iommu_ioapic_update_ire(
     unsigned int apic, unsigned int reg, unsigned int value)
 {
@@ -482,6 +556,13 @@ void amd_iommu_ioapic_update_ire(
         *((u32 *)&new_rte) = value;
         /* read upper 32 bits from io-apic rte */
         *(((u32 *)&new_rte) + 1) = __io_apic_read(apic, reg + 1);
+
+        if ( new_rte.delivery_mode > 1 && x2apic_enabled )
+        {
+            setup_forced_ioapic_rte(apic, pin, iommu, &new_rte);
+            __ioapic_write_entry(apic, pin, true, new_rte);
+            return;
+        }
     }
     else
     {
Signature · 7 lines
2.23.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#2 lore raw

Re: [Xen-devel] [PATCH v3 1/4] iommu/amd: support all delivery modes with x2APIC

Quoted 4 lines · Roger Pau Monne

Current AMD IOMMU code will attempt to create remapping entries for all IO-APIC pins, regardless of the delivery mode. AMD IOMMU implementation doesn't support remapping entries for IO-APIC pins with delivery mode set to SMI, MNI, INIT or ExtINT, instead those entries

Nit: "NMI"

Quoted 11 lines

are not translated provided the right bits are set in the device table entry.

This change checks whether the device table entry has the correct bits set in order to delivery the requested interrupt or a warning message is printed. It also translates the 32bit destination field into a physical or logical IO-APIC entry format. Note that if the 32bit destination cannot fit into the legacy format a message is printed and the entry is masked.

For this to have an effect on firmware initialized RTEs, it also requires patch 4 aiui. In fact I think it should be _only_ this case where we allow delivery modes other than fixed and lowest priority ("arbitrated" in AMD terminology). Hence I think this patch wants to go last in the series, and the code be changed to reject runtime requests to fiddle with non-"normal" delivery modes (this may go further and actually disallow changing such RTEs at runtime alongside disallowing their production).

Quoted 8 lines
xen/drivers/passthrough/amd/iommu_intr.c +2 -0
@@ -439,6 +439,80 @@ int __init amd_iommu_setup_ioapic_remapping(void)
     return 0;
 }
 
+void setup_forced_ioapic_rte(unsigned int apic, unsigned int pin,
+                             struct amd_iommu *iommu,

xen/drivers/passthrough/amd/iommu_intr.c int __init amd_iommu_setup_ioapic_remapping(void) · line 453

Quoted 10 lines
xen/drivers/passthrough/amd/iommu_intr.c +10 -0
+                             struct IO_APIC_route_entry *rte)
+{
+    unsigned int idx = ioapic_id_to_index(IO_APIC_ID(apic));
+    struct amd_iommu_dte *table = iommu->dev_table.buffer;
+    unsigned int req_id, dest, offset;
+    union irte_ptr entry;
+
+    ASSERT(x2apic_enabled);
+
+    if ( idx == MAX_IO_APICS )

Better >= ?

xen/drivers/passthrough/amd/iommu_intr.c int __init amd_iommu_setup_ioapic_remapping(void) · line 465

Quoted 12 lines
xen/drivers/passthrough/amd/iommu_intr.c +12 -0
+    {
+        rte->mask = true;
+        return;
+    }
+
+    req_id = get_intremap_requestor_id(ioapic_sbdf[idx].seg,
+                                       ioapic_sbdf[idx].bdf);
+
+    switch ( rte->delivery_mode )
+    {
+    case dest_SMI:
+        break;

Don't you want to check the sys_mgt field here, along the lines of the other ones below?

xen/drivers/passthrough/amd/iommu_intr.c int __init amd_iommu_setup_ioapic_remapping(void) · line 473

Quoted 7 lines
xen/drivers/passthrough/amd/iommu_intr.c +7 -0
+#define DEL_CHECK(type, dte_field)                                      \
+    case dest_ ## type:                                                 \
+        if ( !table[req_id].dte_field )                                 \
+            printk(XENLOG_WARNING                                       \
+                   STR(type) " on IO-APIC %u pin %u will be aborted\n", \
+                   apic, pin);                                          \
+        break;

Please omit the final ; here, such that ...

xen/drivers/passthrough/amd/iommu_intr.c int __init amd_iommu_setup_ioapic_remapping(void) · line 477

Quoted 3 lines
xen/drivers/passthrough/amd/iommu_intr.c +3 -0
+    DEL_CHECK(NMI, nmi_pass);
+    DEL_CHECK(INIT, init_pass);
+    DEL_CHECK(ExtINT, ext_int_pass);

... the ones here become an actual requirement.

Quoted 23 lines
xen/drivers/passthrough/amd/iommu_intr.c +23 -0
+#undef DEL_CHECK
+
+    default:
+        ASSERT_UNREACHABLE();
+        return;
+    }
+
+    offset = ioapic_sbdf[idx].pin_2_idx[pin];
+    if ( offset >= INTREMAP_MAX_ENTRIES )
+    {
+        rte->mask = true;
+        return;
+    }
+
+    entry = get_intremap_entry(iommu, req_id, offset);
+    dest = get_full_dest(entry.ptr128);
+
+#define SET_DEST(name, dest_mask) {                                            \
+    if ( dest & ~(dest_mask) )                                                 \
+    {                                                                          \
+        printk(XENLOG_WARNING                                                  \
+               "IO-APIC %u pin %u " STR(name) " destination (%x) > %x\n",      \
+               apic, pin, dest, dest_mask);                                    \

xen/drivers/passthrough/amd/iommu_intr.c int __init amd_iommu_setup_ioapic_remapping(void) · line 510

Quoted 10 lines
xen/drivers/passthrough/amd/iommu_intr.c +10 -0
+        rte->mask = true;                                                      \
+        return;                                                                \
+    }                                                                          \
+    rte->dest.name.name ## _dest = dest;                                       \
+}
+
+    if ( rte->dest_mode )
+        SET_DEST(physical, 0xf)
+    else
+        SET_DEST(logical, 0xff)

This reads as if the code was broken. Please add () around the outermost {} of the macro, allowing you to put ; here. Or otherwise, just like above for DEL_CHECK(), omit the {} as well as the final semicolon.

Furthermore - are you sure about this distinction? See e.g. update_intremap_entry_from_ioapic() and amd_iommu_setup_ioapic_remapping(), which unilaterally use rte->dest.logical.logical_dest. Likewise io_apic.c's SET_DEST() users, which generally pass "logical" for as middle argument. I thought one of my half way recent patches (IRQ handling or AMD IOMMU work) had a remark in it regarding such a badly documented extension, but I can't seem to be able to find it anymore.

xen/drivers/passthrough/amd/iommu_intr.c void amd_iommu_ioapic_update_ire( · line 560

Quoted 6 lines
xen/drivers/passthrough/amd/iommu_intr.c +2 -0
@@ -482,6 +556,13 @@ void amd_iommu_ioapic_update_ire(
         *((u32 *)&new_rte) = value;
         /* read upper 32 bits from io-apic rte */
         *(((u32 *)&new_rte) + 1) = __io_apic_read(apic, reg + 1);
+
+        if ( new_rte.delivery_mode > 1 && x2apic_enabled )

The literal 1 here wants attaching of a comment. And why the x2apic_enabled part of the condition?

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#3 lore raw

PATCH v3 2/4 x2APIC: simplify resume

In reply to Roger Pau Monne

There's no need to save and restore the IO-APIC entries, the entries prior to suspension have already been saved by ioapic_suspend, and will be restored by ioapic_resume. Note that at the point where resume_x2apic gets called the IO-APIC has not yet resumed, and hence all entries should be masked.

Note this shouldn't introduce any functional change.

---
I'm Ccing Marek since I think he usually tests suspend/resume. Could you give this patch a try?
---
Cc: Marek Marczykowski-Górecki <[email protected]>
Cc: Juergen Gross <[email protected]>
---
Changes since v2:
- New in this version.

---
 xen/arch/x86/apic.c | 27 ---------------------------
 1 file changed, 27 deletions(-)
xen/arch/x86/apic.c +0 -27
index 6cdb50cf41..0607eb92a8 100644
@@ -492,35 +492,8 @@ static void __enable_x2apic(void)
 
 static void resume_x2apic(void)
 {
-    struct IO_APIC_route_entry **ioapic_entries = NULL;
-
-    ASSERT(x2apic_enabled);
-
-    ioapic_entries = alloc_ioapic_entries();
-    if ( !ioapic_entries )
-    {
-        printk("Allocate ioapic_entries failed\n");
-        goto out;
-    }
-
-    if ( save_IO_APIC_setup(ioapic_entries) )
-    {
-        printk("Saving IO-APIC state failed\n");
-        goto out;
-    }
-
-    mask_8259A();
-    mask_IO_APIC_setup(ioapic_entries);
-
     iommu_enable_x2apic();
     __enable_x2apic();
-
-    restore_IO_APIC_setup(ioapic_entries);
-    unmask_8259A();
-
-out:
-    if ( ioapic_entries )
-        free_ioapic_entries(ioapic_entries);
 }
 
 void setup_local_APIC(void)
Signature · 7 lines
2.23.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Tested-by on v3 2/4
#4 lore raw

Re: [Xen-devel] [PATCH v3 2/4] x2APIC: simplify resume

Quoted 10 lines · Roger Pau Monne

There's no need to save and restore the IO-APIC entries, the entries prior to suspension have already been saved by ioapic_suspend, and will be restored by ioapic_resume. Note that at the point where resume_x2apic gets called the IO-APIC has not yet resumed, and hence all entries should be masked.

Note this shouldn't introduce any functional change.

I've tried host suspend without any domU running and it works. I've tested just this patch without others in the series, does it matter?

Quoted 53 lines

---
I'm Ccing Marek since I think he usually tests suspend/resume. Could you give this patch a try?
---
Cc: Marek Marczykowski-Górecki <[email protected]>
Cc: Juergen Gross <[email protected]>
---
Changes since v2:
- New in this version.

---
 xen/arch/x86/apic.c | 27 ---------------------------
 1 file changed, 27 deletions(-)
xen/arch/x86/apic.c +0 -27
index 6cdb50cf41..0607eb92a8 100644
@@ -492,35 +492,8 @@ static void __enable_x2apic(void)
 
 static void resume_x2apic(void)
 {
-    struct IO_APIC_route_entry **ioapic_entries = NULL;
-
-    ASSERT(x2apic_enabled);
-
-    ioapic_entries = alloc_ioapic_entries();
-    if ( !ioapic_entries )
-    {
-        printk("Allocate ioapic_entries failed\n");
-        goto out;
-    }
-
-    if ( save_IO_APIC_setup(ioapic_entries) )
-    {
-        printk("Saving IO-APIC state failed\n");
-        goto out;
-    }
-
-    mask_8259A();
-    mask_IO_APIC_setup(ioapic_entries);
-
     iommu_enable_x2apic();
     __enable_x2apic();
-
-    restore_IO_APIC_setup(ioapic_entries);
-    unmask_8259A();
-
-out:
-    if ( ioapic_entries )
-        free_ioapic_entries(ioapic_entries);
 }
 
 void setup_local_APIC(void)
Signature · 5 lines
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#5 lore raw

Re: [Xen-devel] [PATCH v3 2/4] x2APIC: simplify resume

Quoted 7 lines · Marek Marczykowski-Górecki
Quoted 6 lines · Roger Pau Monne

There's no need to save and restore the IO-APIC entries, the entries prior to suspension have already been saved by ioapic_suspend, and will be restored by ioapic_resume. Note that at the point where resume_x2apic gets called the IO-APIC has not yet resumed, and hence all entries should be masked.

Quoted 7 lines
Quoted 4 lines

Note this shouldn't introduce any functional change.

I've tried host suspend without any domU running and it works. I've tested just this patch without others in the series, does it matter?

No that's fine.

Quoted 1 line

Thanks.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

Reviewed-by on v3 2/4
#6 lore raw

Re: [Xen-devel] [PATCH v3 2/4] x2APIC: simplify resume

In reply to Roger Pau Monne
Quoted 10 lines · Roger Pau Monne

There's no need to save and restore the IO-APIC entries, the entries prior to suspension have already been saved by ioapic_suspend, and will be restored by ioapic_resume. Note that at the point where resume_x2apic gets called the IO-APIC has not yet resumed, and hence all entries should be masked.

Note this shouldn't introduce any functional change.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#7 lore raw

PATCH v3 3/4 x2APIC: translate IO-APIC entries when enabling the IOMMU

In reply to Roger Pau Monne

When interrupt remapping is enabled as part of enabling x2APIC the IO-APIC entries also need to be translated to the new format and added to the interrupt remapping table.

This prevents IOMMU interrupt remapping faults when booting on hardware that has unmasked IO-APIC pins.

---
Cc: Juergen Gross <[email protected]>
---
Changes since v2:
- Introduce intremap_enabled.

Changes since v1:
- Remove the unneeded iommu_enabled local variable.

---
 xen/arch/x86/apic.c           | 11 ++++++++++-
 xen/arch/x86/io_apic.c        |  5 +++--
 xen/include/asm-x86/io_apic.h |  3 ++-
 3 files changed, 15 insertions(+), 4 deletions(-)
xen/arch/x86/apic.c +10 -1
index 0607eb92a8..a5f7b05d5a 100644
@@ -860,6 +860,7 @@ void __init x2apic_bsp_setup(void)
 {
     struct IO_APIC_route_entry **ioapic_entries = NULL;
     const char *orig_name;
+    bool intremap_enabled;
 
     if ( !cpu_has_x2apic )
         return;
@@ -902,11 +903,13 @@ void __init x2apic_bsp_setup(void)
     switch ( iommu_enable_x2apic() )
     {
     case 0:
+        intremap_enabled = true;
         break;
     case -ENXIO: /* ACPI_DMAR_X2APIC_OPT_OUT set */
         if ( !x2apic_enabled )
         {
             printk("Not enabling x2APIC (upon firmware request)\n");
+            intremap_enabled = false;
             goto restore_out;
         }
         /* fall through */
@@ -917,6 +920,7 @@ void __init x2apic_bsp_setup(void)
 
         printk(XENLOG_ERR
                "Failed to enable Interrupt Remapping: Will not enable x2APIC.\n");
+        intremap_enabled = false;
         goto restore_out;
     }
 
@@ -934,7 +938,12 @@ void __init x2apic_bsp_setup(void)
         printk("Switched to APIC driver %s\n", genapic.name);
 
 restore_out:
-    restore_IO_APIC_setup(ioapic_entries);
+    /*
+     * NB: do not use raw mode when restoring entries if the iommu has been
+     * enabled during the process, because the entries need to be translated
+     * and added to the remapping table in that case.
+     */
+    restore_IO_APIC_setup(ioapic_entries, !intremap_enabled);
     unmask_8259A();
 
 out:
xen/arch/x86/io_apic.c +3 -2
index 5d25862bd8..37eabc16c9 100644
@@ -379,7 +379,8 @@ void mask_IO_APIC_setup(struct IO_APIC_route_entry **ioapic_entries)
 /*
  * Restore IO APIC entries which was saved in ioapic_entries.
  */
-int restore_IO_APIC_setup(struct IO_APIC_route_entry **ioapic_entries)
+int restore_IO_APIC_setup(struct IO_APIC_route_entry **ioapic_entries,
+                          bool raw)
 {
     int apic, pin;
 
@@ -394,7 +395,7 @@ int restore_IO_APIC_setup(struct IO_APIC_route_entry **ioapic_entries)
             return -ENOMEM;
 
         for (pin = 0; pin < nr_ioapic_entries[apic]; pin++)
-	    ioapic_write_entry(apic, pin, 1, ioapic_entries[apic][pin]);
+	    ioapic_write_entry(apic, pin, raw, ioapic_entries[apic][pin]);
     }
 
     return 0;
xen/include/asm-x86/io_apic.h +2 -1
index 0b041f0565..998905186b 100644
@@ -197,7 +197,8 @@ extern struct IO_APIC_route_entry **alloc_ioapic_entries(void);
 extern void free_ioapic_entries(struct IO_APIC_route_entry **ioapic_entries);
 extern int save_IO_APIC_setup(struct IO_APIC_route_entry **ioapic_entries);
 extern void mask_IO_APIC_setup(struct IO_APIC_route_entry **ioapic_entries);
-extern int restore_IO_APIC_setup(struct IO_APIC_route_entry **ioapic_entries);
+extern int restore_IO_APIC_setup(struct IO_APIC_route_entry **ioapic_entries,
+                                 bool raw);
 
 unsigned highest_gsi(void);
 
Signature · 7 lines
2.23.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v3 3/4
#8 lore raw

Re: [Xen-devel] [PATCH v3 3/4] x2APIC: translate IO-APIC entries when enabling the IOMMU

Quoted 9 lines · Roger Pau Monne

When interrupt remapping is enabled as part of enabling x2APIC the IO-APIC entries also need to be translated to the new format and added to the interrupt remapping table.

This prevents IOMMU interrupt remapping faults when booting on hardware that has unmasked IO-APIC pins.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#9 lore raw

PATCH v3 4/4 iommu: translate IO-APIC pins when enabling interrupt remapping

In reply to Roger Pau Monne

On Intel hardware there's currently no translation of already enabled IO-APIC pins when interrupt remapping is enabled on the IOMMU, hence introduce a logic similar to the one used in x2apic_bsp_setup in order to save and mask all IO-APIC pins, and then translate and restore them after interrupt remapping has been enabled.

With this change the AMD specific logic to deal with enabled pins (amd_iommu_setup_ioapic_remapping) can be removed, thus unifying the handling of IO-APIC when enabling interrupt remapping regardless of the IOMMU vendor.

---
Cc: Juergen Gross <[email protected]>

---
 xen/drivers/passthrough/amd/iommu_init.c      | 11 ++-
 xen/drivers/passthrough/amd/iommu_intr.c      | 90 +------------------
 xen/drivers/passthrough/x86/iommu.c           | 34 ++++++-
 xen/include/asm-x86/hvm/svm/amd-iommu-proto.h |  1 +
 4 files changed, 40 insertions(+), 96 deletions(-)
xen/drivers/passthrough/amd/iommu_init.c +5 -6
index 6f53c7ec08..3c244619b9 100644
@@ -19,6 +19,7 @@
 
 #include <xen/errno.h>
 #include <xen/acpi.h>
+#include <xen/keyhandler.h>
 #include <xen/pci.h>
 #include <xen/pci_regs.h>
 #include <xen/irq.h>
@@ -1435,12 +1436,6 @@ int __init amd_iommu_init(bool xt)
     if ( rc )
         goto error_out;
 
-    /* initialize io-apic interrupt remapping entries */
-    if ( iommu_intremap )
-        rc = amd_iommu_setup_ioapic_remapping();
-    if ( rc )
-        goto error_out;
-
     /* Allocate and initialize device table(s). */
     pci_init = !xt;
     rc = iterate_ivrs_mappings(amd_iommu_setup_device_table);
@@ -1469,6 +1464,10 @@ int __init amd_iommu_init(bool xt)
             goto error_out;
     }
 
+    if ( iommu_intremap )
+        register_keyhandler('V', &amd_iommu_dump_intremap_tables,
+                            "dump IOMMU intremap tables", 0);
+
     return 0;
 
 error_out:
xen/drivers/passthrough/amd/iommu_intr.c +1 -89

114 lines not on this page · raw message

xen/drivers/passthrough/x86/iommu.c +33 -1
index 59905629e1..2cf528e760 100644
@@ -21,6 +21,7 @@
 #include <xsm/xsm.h>
 
 #include <asm/hvm/io.h>
+#include <asm/io_apic.h>
 #include <asm/setup.h>
 
 const struct iommu_init_ops *__initdata iommu_init_ops;
@@ -28,6 +29,7 @@ struct iommu_ops __read_mostly iommu_ops;
 
 int __init iommu_hardware_setup(void)
 {
+    struct IO_APIC_route_entry **ioapic_entries = NULL;
     int rc;
 
     if ( !iommu_init_ops )
@@ -43,7 +45,37 @@ int __init iommu_hardware_setup(void)
         /* x2apic setup may have previously initialised the struct. */
         ASSERT(iommu_ops.init == iommu_init_ops->ops->init);
 
-    return iommu_init_ops->setup();
+    if ( !x2apic_enabled && iommu_intremap )
+    {
+        /*
+         * If x2APIC is enabled interrupt remapping is already enabled, so
+         * there's no need to mess with the IO-APIC because the remapping
+         * entries are already correctly setup by x2apic_bsp_setup.
+         */
+        ioapic_entries = alloc_ioapic_entries();
+        if ( !ioapic_entries )
+            return -ENOMEM;
+        rc = save_IO_APIC_setup(ioapic_entries);
+        if ( rc )
+        {
+            free_ioapic_entries(ioapic_entries);
+            return rc;
+        }
+
+        mask_8259A();
+        mask_IO_APIC_setup(ioapic_entries);
+    }
+
+    rc = iommu_init_ops->setup();
+
+    if ( ioapic_entries )
+    {
+        restore_IO_APIC_setup(ioapic_entries, rc);
+        unmask_8259A();
+        free_ioapic_entries(ioapic_entries);
+    }
+
+    return rc;
 }
 
 int iommu_enable_x2apic(void)
xen/include/asm-x86/hvm/svm/amd-iommu-proto.h +1 -0
index 07d25a585d..8ed9482791 100644
@@ -114,6 +114,7 @@ int amd_iommu_msi_msg_update_ire(
 void amd_iommu_read_msi_from_ire(
     struct msi_desc *msi_desc, struct msi_msg *msg);
 int amd_setup_hpet_msi(struct msi_desc *msi_desc);
+void amd_iommu_dump_intremap_tables(unsigned char key);
 
 extern struct ioapic_sbdf {
     u16 bdf, seg;
Signature · 7 lines
2.23.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#10 lore raw
In reply to Roger Pau Monne
Quoted 11 lines · Roger Pau Monne

Hello,

The following series contain fixes for issues found when enabling interrupt remapping and the IO-APIC already has unmasked pins. While I'm not aware of any system malfunctioning (apart from reporting IOMMU interrupt remapping faults) I think it would be nice to have those in 4.13.

The series can also be found at:

git://xenbits.xen.org/people/royger/xen.git iommu_intr_v3

For the series:

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#11 lore raw
In reply to Roger Pau Monne
Quoted 12 lines · Roger Pau Monne

Hello,

The following series contain fixes for issues found when enabling interrupt remapping and the IO-APIC already has unmasked pins. While I'm not aware of any system malfunctioning (apart from reporting IOMMU interrupt remapping faults) I think it would be nice to have those in 4.13.

The series can also be found at:

git://xenbits.xen.org/people/royger/xen.git iommu_intr_v3

Quoted 7 lines

Thanks, Roger.

Roger Pau Monne (4):
iommu/amd: support all delivery modes with x2APIC
x2APIC: simplify resume
x2APIC: translate IO-APIC entries when enabling the IOMMU
iommu: translate IO-APIC pins when enabling interrupt remapping

As mentioned in reply to #1, I think that one should go last. The other three are ready to be committed, but I'd like to double check that there's no dependency of any of them on the 1st.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#12 lore raw
Quoted 21 lines · Jan Beulich
Quoted 12 lines · Roger Pau Monne

Hello,

The following series contain fixes for issues found when enabling interrupt remapping and the IO-APIC already has unmasked pins. While I'm not aware of any system malfunctioning (apart from reporting IOMMU interrupt remapping faults) I think it would be nice to have those in 4.13.

The series can also be found at:

git://xenbits.xen.org/people/royger/xen.git iommu_intr_v3

Quoted 7 lines

Thanks, Roger.

Roger Pau Monne (4):
iommu/amd: support all delivery modes with x2APIC
x2APIC: simplify resume
x2APIC: translate IO-APIC entries when enabling the IOMMU
iommu: translate IO-APIC pins when enabling interrupt remapping

Quoted 3 lines

As mentioned in reply to #1, I think that one should go last. The other three are ready to be committed, but I'd like to double check that there's no dependency of any of them on the 1st.

I think #1 is fully independent of the rest of the series, and the other three can be committed regardless of #1.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel