Skip to contentSkip to conversation

x86/ioapic: fix edge triggered interrupt migration

PATCH Superseded Roger Pau Monne 2 participants 16 messages Jul 18, 2023 – Jul 26, 2023

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover x86/ioapic: fix edge triggered interrupt migration

Hello,

Following series attempts to solve the issue with IO-APIC edge triggered interrupts seeing an inconsistent IRTE when injected while being migrated.

It's been simplified a bit from the original RFC, and does make the logic in the IOMMU RTE update handlers cleaner, as those get passed the full RTE.

When not using interrupt remapping the logic is left as-is. One further change would be to mask edge triggered interrupts, as destination and vector cannot be changed atomically when dealing with plain IO-APIC RTEs.

Last patch has also been adjusted in order to always to atomic IRTE updates on AMD.

Thanks, Roger.

Roger Pau Monne (5):
x86/ioapic: add a raw field to RTE struct x86/ioapic: RTE modifications must use ioapic_write_entry iommu/vtd: rename io_apic_read_remap_rte() local variable x86/iommu: pass full IO-APIC RTE for remapping table update amd/iommu: force atomic updates of remapping entries

 xen/arch/x86/include/asm/io_apic.h       |  65 +++++-----
 xen/arch/x86/include/asm/iommu.h         |   3 +-
 xen/arch/x86/io_apic.c                   |  40 +++---
 xen/drivers/passthrough/amd/iommu.h      |   2 +-
 xen/drivers/passthrough/amd/iommu_init.c |  10 ++
 xen/drivers/passthrough/amd/iommu_intr.c | 155 +++--------------------
 xen/drivers/passthrough/vtd/extern.h     |   2 +-
 xen/drivers/passthrough/vtd/intremap.c   | 131 ++++++++++---------
 xen/drivers/passthrough/x86/iommu.c      |   4 +-
 xen/include/xen/iommu.h                  |   3 +-
 10 files changed, 156 insertions(+), 259 deletions(-)
Signature · 1 line
2.41.0
#1 lore raw

PATCH v2 1/5 x86/ioapic: add a raw field to RTE struct

Further changes will require access to the full RTE as a single value in order to pass it to IOMMU interrupt remapping handlers.

No functional change intended.

---
Changes since v1:
- Fix initializers.
---
Tested on gitlab CI, builds on all compilers used there.

---
 xen/arch/x86/include/asm/io_apic.h       | 57 +++++++++++++-----------
 xen/arch/x86/io_apic.c                   |  2 +-
 xen/drivers/passthrough/amd/iommu_intr.c |  4 +-
 xen/drivers/passthrough/vtd/intremap.c   |  4 +-
 4 files changed, 35 insertions(+), 32 deletions(-)
xen/arch/x86/include/asm/io_apic.h +30 -27
index bd8cb95f46b1..4c4777b68a51 100644
@@ -90,35 +90,38 @@ enum ioapic_irq_destination_types {
 };
 
 struct IO_APIC_route_entry {
-    unsigned int vector:8;
-    unsigned int delivery_mode:3; /*
-                                   * 000: FIXED
-                                   * 001: lowest prio
-                                   * 111: ExtINT
-                                   */
-    unsigned int dest_mode:1;     /* 0: physical, 1: logical */
-    unsigned int delivery_status:1;
-    unsigned int polarity:1;      /* 0: low, 1: high */
-    unsigned int irr:1;
-    unsigned int trigger:1;       /* 0: edge, 1: level */
-    unsigned int mask:1;          /* 0: enabled, 1: disabled */
-    unsigned int __reserved_2:15;
-
     union {
         struct {
-            unsigned int __reserved_1:24;
-            unsigned int physical_dest:4;
-            unsigned int __reserved_2:4;
-        } physical;
-
-        struct {
-            unsigned int __reserved_1:24;
-            unsigned int logical_dest:8;
-        } logical;
-
-        /* used when Interrupt Remapping with EIM is enabled */
-        unsigned int dest32;
-    } dest;
+            unsigned int vector:8;
+            unsigned int delivery_mode:3; /*
+                                           * 000: FIXED
+                                           * 001: lowest prio
+                                           * 111: ExtINT
+                                           */
+            unsigned int dest_mode:1;     /* 0: physical, 1: logical */
+            unsigned int delivery_status:1;
+            unsigned int polarity:1;      /* 0: low, 1: high */
+            unsigned int irr:1;
+            unsigned int trigger:1;       /* 0: edge, 1: level */
+            unsigned int mask:1;          /* 0: enabled, 1: disabled */
+            unsigned int __reserved_2:15;
+
+            union {
+                struct {
+                    unsigned int __reserved_1:24;
+                    unsigned int physical_dest:4;
+                    unsigned int __reserved_2:4;
+                } physical;
+
+                struct {
+                    unsigned int __reserved_1:24;
+                    unsigned int logical_dest:8;
+                } logical;
+                unsigned int dest32;
+            } dest;
+        };
+        uint64_t raw;
+    };
 };
 
 /*
xen/arch/x86/io_apic.c +1 -1
index 25a08b1ea6c6..aada2ef96c62 100644
@@ -2360,7 +2360,7 @@ int ioapic_guest_read(unsigned long physbase, unsigned int reg, u32 *pval)
 int ioapic_guest_write(unsigned long physbase, unsigned int reg, u32 val)
 {
     int apic, pin, irq, ret, pirq;
-    struct IO_APIC_route_entry rte = { 0 };
+    struct IO_APIC_route_entry rte = { };
     unsigned long flags;
     struct irq_desc *desc;
 
xen/drivers/passthrough/amd/iommu_intr.c +2 -2
index f4de09f43180..9e6be3be3515 100644
@@ -352,8 +352,8 @@ static int update_intremap_entry_from_ioapic(
 void cf_check amd_iommu_ioapic_update_ire(
     unsigned int apic, unsigned int reg, unsigned int value)
 {
-    struct IO_APIC_route_entry old_rte = { 0 };
-    struct IO_APIC_route_entry new_rte = { 0 };
+    struct IO_APIC_route_entry old_rte = { };
+    struct IO_APIC_route_entry new_rte = { };
     unsigned int rte_lo = (reg & 1) ? reg - 1 : reg;
     unsigned int pin = (reg - 0x10) / 2;
     int seg, bdf, rc;
xen/drivers/passthrough/vtd/intremap.c +2 -2
index b39bc832821a..786388b4d9c7 100644
@@ -419,7 +419,7 @@ unsigned int cf_check io_apic_read_remap_rte(
 {
     unsigned int ioapic_pin = (reg - 0x10) / 2;
     int index;
-    struct IO_xAPIC_route_entry old_rte = { 0 };
+    struct IO_xAPIC_route_entry old_rte = { };
     int rte_upper = (reg & 1) ? 1 : 0;
     struct vtd_iommu *iommu = ioapic_to_iommu(IO_APIC_ID(apic));
 
@@ -442,7 +442,7 @@ void cf_check io_apic_write_remap_rte(
     unsigned int apic, unsigned int reg, unsigned int value)
 {
     unsigned int ioapic_pin = (reg - 0x10) / 2;
-    struct IO_xAPIC_route_entry old_rte = { 0 };
+    struct IO_xAPIC_route_entry old_rte = { };
     struct IO_APIC_route_remap_entry *remap_rte;
     unsigned int rte_upper = (reg & 1) ? 1 : 0;
     struct vtd_iommu *iommu = ioapic_to_iommu(IO_APIC_ID(apic));
Signature · 1 line
2.41.0
Acked-by on v2 1/5
#2 lore raw

Re: [PATCH v2 1/5] x86/ioapic: add a raw field to RTE struct

Quoted 6 lines · Roger Pau Monne

Further changes will require access to the full RTE as a single value in order to pass it to IOMMU interrupt remapping handlers.

No functional change intended.

As indicated for v1, I'm fine in principle, so

but I don't see this (or really any of the prereq changes) go in ahead of the actual meat of this series (patch 4 and 5, aiui).

Jan

#3 lore raw

PATCH v2 2/5 x86/ioapic: RTE modifications must use ioapic_write_entry

In reply to Roger Pau Monne

Do not allow to write to RTE registers using io_apic_write and instead require changes to RTE to be performed using ioapic_write_entry.

This is in preparation for passing the full contents of the RTE to the IOMMU interrupt remapping handlers, so remapping entries for IO-APIC RTEs can be updated atomically when possible.

While immediately this commit might expand the number of MMIO accesses in order to update an IO-APIC RTE, further changes will benefit from getting the full RTE value passed to the IOMMU handlers, as the logic is greatly simplified when the IOMMU handlers can get the complete RTE value in one go.

---
Changes since v1:
- Reinstate io_apic_modify().
- Expand commit message.

---
 xen/arch/x86/include/asm/io_apic.h       |  8 ++---
 xen/arch/x86/io_apic.c                   | 37 ++++++++++++------------
 xen/drivers/passthrough/amd/iommu_intr.c |  6 ----
 3 files changed, 22 insertions(+), 29 deletions(-)
xen/arch/x86/include/asm/io_apic.h +4 -4
index 4c4777b68a51..9165da2281ae 100644
@@ -162,8 +162,8 @@ static inline void __io_apic_write(unsigned int apic, unsigned int reg, unsigned
 
 static inline void io_apic_write(unsigned int apic, unsigned int reg, unsigned int value)
 {
-    if ( ioapic_reg_remapped(reg) )
-        return iommu_update_ire_from_apic(apic, reg, value);
+    /* RTE writes must use ioapic_write_entry. */
+    BUG_ON(reg >= 0x10);
     __io_apic_write(apic, reg, value);
 }
 
@@ -173,8 +173,8 @@ static inline void io_apic_write(unsigned int apic, unsigned int reg, unsigned i
  */
 static inline void io_apic_modify(unsigned int apic, unsigned int reg, unsigned int value)
 {
-    if ( ioapic_reg_remapped(reg) )
-        return iommu_update_ire_from_apic(apic, reg, value);
+    /* RTE writes must use ioapic_write_entry. */
+    BUG_ON(reg >= 0x10);
     *(IO_APIC_BASE(apic) + 4) = value;
 }
 
xen/arch/x86/io_apic.c +18 -19
index aada2ef96c62..85b4b4c6bc98 100644
@@ -269,15 +269,15 @@ void __ioapic_write_entry(
 {
     union entry_union eu = { .entry = e };
 
-    if ( raw )
+    if ( raw || !iommu_intremap )
     {
         __io_apic_write(apic, 0x11 + 2 * pin, eu.w2);
         __io_apic_write(apic, 0x10 + 2 * pin, eu.w1);
     }
     else
     {
-        io_apic_write(apic, 0x11 + 2 * pin, eu.w2);
-        io_apic_write(apic, 0x10 + 2 * pin, eu.w1);
+        iommu_update_ire_from_apic(apic, 0x11 + 2 * pin, eu.w2);
+        iommu_update_ire_from_apic(apic, 0x10 + 2 * pin, eu.w1);
     }
 }
 
@@ -433,16 +433,17 @@ static void modify_IO_APIC_irq(unsigned int irq, unsigned int enable,
                                unsigned int disable)
 {
     struct irq_pin_list *entry = irq_2_pin + irq;
-    unsigned int pin, reg;
 
     for (;;) {
-        pin = entry->pin;
+        unsigned int pin = entry->pin;
+        struct IO_APIC_route_entry rte;
+
         if (pin == -1)
             break;
-        reg = io_apic_read(entry->apic, 0x10 + pin*2);
-        reg &= ~disable;
-        reg |= enable;
-        io_apic_modify(entry->apic, 0x10 + pin*2, reg);
+        rte = __ioapic_read_entry(entry->apic, pin, false);
+        rte.raw &= ~(uint64_t)disable;
+        rte.raw |= enable;
+        __ioapic_write_entry(entry->apic, pin, false, rte);
         if (!entry->next)
             break;
         entry = irq_2_pin + entry->next;
@@ -584,16 +585,16 @@ set_ioapic_affinity_irq(struct irq_desc *desc, const cpumask_t *mask)
             dest = SET_APIC_LOGICAL_ID(dest);
         entry = irq_2_pin + irq;
         for (;;) {
-            unsigned int data;
+            struct IO_APIC_route_entry rte;
+
             pin = entry->pin;
             if (pin == -1)
                 break;
 
-            io_apic_write(entry->apic, 0x10 + 1 + pin*2, dest);
-            data = io_apic_read(entry->apic, 0x10 + pin*2);
-            data &= ~IO_APIC_REDIR_VECTOR_MASK;
-            data |= MASK_INSR(desc->arch.vector, IO_APIC_REDIR_VECTOR_MASK);
-            io_apic_modify(entry->apic, 0x10 + pin*2, data);
+            rte = __ioapic_read_entry(entry->apic, pin, false);
+            rte.dest.dest32 = dest;
+            rte.vector = desc->arch.vector;
+            __ioapic_write_entry(entry->apic, pin, false, rte);
 
             if (!entry->next)
                 break;
@@ -2127,10 +2128,8 @@ void ioapic_resume(void)
             reg_00.bits.ID = mp_ioapics[apic].mpc_apicid;
             __io_apic_write(apic, 0, reg_00.raw);
         }
-        for (i = 0; i < nr_ioapic_entries[apic]; i++, entry++) {
-            __io_apic_write(apic, 0x11+2*i, *(((int *)entry)+1));
-            __io_apic_write(apic, 0x10+2*i, *(((int *)entry)+0));
-        }
+        for (i = 0; i < nr_ioapic_entries[apic]; i++, entry++)
+            __ioapic_write_entry(apic, i, true, *entry);
     }
     spin_unlock_irqrestore(&ioapic_lock, flags);
 }
xen/drivers/passthrough/amd/iommu_intr.c +0 -6
index 9e6be3be3515..f32c418a7e49 100644
@@ -361,12 +361,6 @@ void cf_check amd_iommu_ioapic_update_ire(
     struct amd_iommu *iommu;
     unsigned int idx;
 
-    if ( !iommu_intremap )
-    {
-        __io_apic_write(apic, reg, value);
-        return;
-    }
-
     idx = ioapic_id_to_index(IO_APIC_ID(apic));
     if ( idx == MAX_IO_APICS )
         return;
Signature · 1 line
2.41.0
#4 lore raw

Re: [PATCH v2 2/5] x86/ioapic: RTE modifications must use ioapic_write_entry

Quoted 10 lines · Roger Pau Monne
xen/arch/x86/io_apic.c +1 -1
@@ -269,15 +269,15 @@ void __ioapic_write_entry(
 {
     union entry_union eu = { .entry = e };
 
-    if ( raw )
+    if ( raw || !iommu_intremap )
     {
         __io_apic_write(apic, 0x11 + 2 * pin, eu.w2);

xen/arch/x86/io_apic.c void __ioapic_write_entry( · line 282

Quoted 10 lines
xen/arch/x86/io_apic.c +2 -2
         __io_apic_write(apic, 0x10 + 2 * pin, eu.w1);
     }
     else
     {
-        io_apic_write(apic, 0x11 + 2 * pin, eu.w2);
-        io_apic_write(apic, 0x10 + 2 * pin, eu.w1);
+        iommu_update_ire_from_apic(apic, 0x11 + 2 * pin, eu.w2);
+        iommu_update_ire_from_apic(apic, 0x10 + 2 * pin, eu.w1);
     }
 }

I think __ioapic_read_entry() wants updating similarly, so that both remain consistent.

Quoted 11 lines
xen/arch/x86/io_apic.c +3 -2
@@ -433,16 +433,17 @@ static void modify_IO_APIC_irq(unsigned int irq, unsigned int enable,
                                unsigned int disable)
 {
     struct irq_pin_list *entry = irq_2_pin + irq;
-    unsigned int pin, reg;
 
     for (;;) {
-        pin = entry->pin;
+        unsigned int pin = entry->pin;
+        struct IO_APIC_route_entry rte;
+

xen/arch/x86/io_apic.c static void modify_IO_APIC_irq(unsigned int irq, unsigned int enable, · line 446

Quoted 10 lines
xen/arch/x86/io_apic.c +4 -4
         if (pin == -1)
             break;
-        reg = io_apic_read(entry->apic, 0x10 + pin*2);
-        reg &= ~disable;
-        reg |= enable;
-        io_apic_modify(entry->apic, 0x10 + pin*2, reg);
+        rte = __ioapic_read_entry(entry->apic, pin, false);
+        rte.raw &= ~(uint64_t)disable;
+        rte.raw |= enable;
+        __ioapic_write_entry(entry->apic, pin, false, rte);

While this isn't going to happen overly often, ...

Quoted 10 lines
xen/arch/x86/io_apic.c +2 -1
@@ -584,16 +585,16 @@ set_ioapic_affinity_irq(struct irq_desc *desc, const cpumask_t *mask)
             dest = SET_APIC_LOGICAL_ID(dest);
         entry = irq_2_pin + irq;
         for (;;) {
-            unsigned int data;
+            struct IO_APIC_route_entry rte;
+
             pin = entry->pin;
             if (pin == -1)
                 break;

xen/arch/x86/io_apic.c set_ioapic_affinity_irq(struct irq_desc *desc, const cpumask_t *mask) · line 597

Quoted 10 lines
xen/arch/x86/io_apic.c +4 -5
 
-            io_apic_write(entry->apic, 0x10 + 1 + pin*2, dest);
-            data = io_apic_read(entry->apic, 0x10 + pin*2);
-            data &= ~IO_APIC_REDIR_VECTOR_MASK;
-            data |= MASK_INSR(desc->arch.vector, IO_APIC_REDIR_VECTOR_MASK);
-            io_apic_modify(entry->apic, 0x10 + pin*2, data);
+            rte = __ioapic_read_entry(entry->apic, pin, false);
+            rte.dest.dest32 = dest;
+            rte.vector = desc->arch.vector;
+            __ioapic_write_entry(entry->apic, pin, false, rte);

... this makes me wonder whether there shouldn't be an __ioapic_modify_entry() capable of suppressing one of the two writes (but still being handed the full RTE).

Jan

#5 lore raw

Re: [PATCH v2 2/5] x86/ioapic: RTE modifications must use ioapic_write_entry

Quoted 16 lines · Jan Beulich
Quoted 15 lines · Roger Pau Monne
xen/arch/x86/io_apic.c +1 -2
@@ -269,15 +269,15 @@ void __ioapic_write_entry(
 {
     union entry_union eu = { .entry = e };
 
-    if ( raw )
+    if ( raw || !iommu_intremap )
     {
         __io_apic_write(apic, 0x11 + 2 * pin, eu.w2);
         __io_apic_write(apic, 0x10 + 2 * pin, eu.w1);
     }
     else
     {
-        io_apic_write(apic, 0x11 + 2 * pin, eu.w2);
Quoted 8 lines

xen/arch/x86/io_apic.c void __ioapic_write_entry( · line 282

Quoted 5 lines
xen/arch/x86/io_apic.c +2 -1
-        io_apic_write(apic, 0x10 + 2 * pin, eu.w1);
+        iommu_update_ire_from_apic(apic, 0x11 + 2 * pin, eu.w2);
+        iommu_update_ire_from_apic(apic, 0x10 + 2 * pin, eu.w1);
     }
 }

I think __ioapic_read_entry() wants updating similarly, so that both remain consistent.

My plan was to deal with __ioapic_read_entry() separately, as I would also like to convert iommu_read_apic_from_ire() to get passed a pin instead of a register, but I could make your requested adjustment here for consistency with __ioapic_write_entry().

Quoted 35 lines
Quoted 11 lines
xen/arch/x86/io_apic.c +3 -2
@@ -433,16 +433,17 @@ static void modify_IO_APIC_irq(unsigned int irq, unsigned int enable,
                                unsigned int disable)
 {
     struct irq_pin_list *entry = irq_2_pin + irq;
-    unsigned int pin, reg;
 
     for (;;) {
-        pin = entry->pin;
+        unsigned int pin = entry->pin;
+        struct IO_APIC_route_entry rte;
+

xen/arch/x86/io_apic.c static void modify_IO_APIC_irq(unsigned int irq, unsigned int enable, · line 446

Quoted 10 lines
xen/arch/x86/io_apic.c +4 -4
         if (pin == -1)
             break;
-        reg = io_apic_read(entry->apic, 0x10 + pin*2);
-        reg &= ~disable;
-        reg |= enable;
-        io_apic_modify(entry->apic, 0x10 + pin*2, reg);
+        rte = __ioapic_read_entry(entry->apic, pin, false);
+        rte.raw &= ~(uint64_t)disable;
+        rte.raw |= enable;
+        __ioapic_write_entry(entry->apic, pin, false, rte);

While this isn't going to happen overly often, ...

Quoted 11 lines
xen/arch/x86/io_apic.c +2 -1
@@ -584,16 +585,16 @@ set_ioapic_affinity_irq(struct irq_desc *desc, const cpumask_t *mask)
             dest = SET_APIC_LOGICAL_ID(dest);
         entry = irq_2_pin + irq;
         for (;;) {
-            unsigned int data;
+            struct IO_APIC_route_entry rte;
+
             pin = entry->pin;
             if (pin == -1)
                 break;
 
Quoted 13 lines

xen/arch/x86/io_apic.c set_ioapic_affinity_irq(struct irq_desc *desc, const cpumask_t *mask) · line 597

Quoted 9 lines
xen/arch/x86/io_apic.c +4 -5
-            io_apic_write(entry->apic, 0x10 + 1 + pin*2, dest);
-            data = io_apic_read(entry->apic, 0x10 + pin*2);
-            data &= ~IO_APIC_REDIR_VECTOR_MASK;
-            data |= MASK_INSR(desc->arch.vector, IO_APIC_REDIR_VECTOR_MASK);
-            io_apic_modify(entry->apic, 0x10 + pin*2, data);
+            rte = __ioapic_read_entry(entry->apic, pin, false);
+            rte.dest.dest32 = dest;
+            rte.vector = desc->arch.vector;
+            __ioapic_write_entry(entry->apic, pin, false, rte);

... this makes me wonder whether there shouldn't be an __ioapic_modify_entry() capable of suppressing one of the two writes (but still being handed the full RTE).

We would then need the original RTE to be passed to __ioapic_modify_entry() in order for it to decide whether one of the accesses can be eliminated (as I don't think we want to read the RTE to check for differences, as we would then perform even more accesses).

This would only be relevant for systems without IOMMU IRTEs, as otherwise the IO-APIC RTE gets modified by the IOMMU handlers.

Thanks, Roger.

#6 lore raw

Re: [PATCH v2 2/5] x86/ioapic: RTE modifications must use ioapic_write_entry

Quoted 26 lines · Roger Pau Monné
Quoted 16 lines · Jan Beulich
Quoted 15 lines · Roger Pau Monne
xen/arch/x86/io_apic.c +1 -2
@@ -269,15 +269,15 @@ void __ioapic_write_entry(
 {
     union entry_union eu = { .entry = e };
 
-    if ( raw )
+    if ( raw || !iommu_intremap )
     {
         __io_apic_write(apic, 0x11 + 2 * pin, eu.w2);
         __io_apic_write(apic, 0x10 + 2 * pin, eu.w1);
     }
     else
     {
-        io_apic_write(apic, 0x11 + 2 * pin, eu.w2);
Quoted 8 lines

xen/arch/x86/io_apic.c void __ioapic_write_entry( · line 282

Quoted 5 lines
xen/arch/x86/io_apic.c +2 -1
-        io_apic_write(apic, 0x10 + 2 * pin, eu.w1);
+        iommu_update_ire_from_apic(apic, 0x11 + 2 * pin, eu.w2);
+        iommu_update_ire_from_apic(apic, 0x10 + 2 * pin, eu.w1);
     }
 }

I think __ioapic_read_entry() wants updating similarly, so that both remain consistent.

Quoted 4 lines

My plan was to deal with __ioapic_read_entry() separately, as I would also like to convert iommu_read_apic_from_ire() to get passed a pin instead of a register, but I could make your requested adjustment here for consistency with __ioapic_write_entry().

I would indeed prefer if you did, to avoid the functions going out of sync.

Quoted 49 lines
Quoted 35 lines
Quoted 11 lines
xen/arch/x86/io_apic.c +3 -2
@@ -433,16 +433,17 @@ static void modify_IO_APIC_irq(unsigned int irq, unsigned int enable,
                                unsigned int disable)
 {
     struct irq_pin_list *entry = irq_2_pin + irq;
-    unsigned int pin, reg;
 
     for (;;) {
-        pin = entry->pin;
+        unsigned int pin = entry->pin;
+        struct IO_APIC_route_entry rte;
+

xen/arch/x86/io_apic.c static void modify_IO_APIC_irq(unsigned int irq, unsigned int enable, · line 446

Quoted 10 lines
xen/arch/x86/io_apic.c +4 -4
         if (pin == -1)
             break;
-        reg = io_apic_read(entry->apic, 0x10 + pin*2);
-        reg &= ~disable;
-        reg |= enable;
-        io_apic_modify(entry->apic, 0x10 + pin*2, reg);
+        rte = __ioapic_read_entry(entry->apic, pin, false);
+        rte.raw &= ~(uint64_t)disable;
+        rte.raw |= enable;
+        __ioapic_write_entry(entry->apic, pin, false, rte);

While this isn't going to happen overly often, ...

Quoted 11 lines
xen/arch/x86/io_apic.c +2 -1
@@ -584,16 +585,16 @@ set_ioapic_affinity_irq(struct irq_desc *desc, const cpumask_t *mask)
             dest = SET_APIC_LOGICAL_ID(dest);
         entry = irq_2_pin + irq;
         for (;;) {
-            unsigned int data;
+            struct IO_APIC_route_entry rte;
+
             pin = entry->pin;
             if (pin == -1)
                 break;
 
Quoted 13 lines

xen/arch/x86/io_apic.c set_ioapic_affinity_irq(struct irq_desc *desc, const cpumask_t *mask) · line 597

Quoted 9 lines
xen/arch/x86/io_apic.c +4 -5
-            io_apic_write(entry->apic, 0x10 + 1 + pin*2, dest);
-            data = io_apic_read(entry->apic, 0x10 + pin*2);
-            data &= ~IO_APIC_REDIR_VECTOR_MASK;
-            data |= MASK_INSR(desc->arch.vector, IO_APIC_REDIR_VECTOR_MASK);
-            io_apic_modify(entry->apic, 0x10 + pin*2, data);
+            rte = __ioapic_read_entry(entry->apic, pin, false);
+            rte.dest.dest32 = dest;
+            rte.vector = desc->arch.vector;
+            __ioapic_write_entry(entry->apic, pin, false, rte);

... this makes me wonder whether there shouldn't be an __ioapic_modify_entry() capable of suppressing one of the two writes (but still being handed the full RTE).

Quoted 5 lines

We would then need the original RTE to be passed to __ioapic_modify_entry() in order for it to decide whether one of the accesses can be eliminated (as I don't think we want to read the RTE to check for differences, as we would then perform even more accesses).

I was actually thinking of a 2-bit hint that the caller would pass:
Low half unmodified and high half unmodified.

Quoted 2 lines

This would only be relevant for systems without IOMMU IRTEs, as otherwise the IO-APIC RTE gets modified by the IOMMU handlers.

Initially yes. I think there's room there to avoid one half of the write as well.

Jan

#7 lore raw

Re: [PATCH v2 2/5] x86/ioapic: RTE modifications must use ioapic_write_entry

In reply to Jan Beulich
Quoted 14 lines · Jan Beulich
Quoted 13 lines · Roger Pau Monne
xen/arch/x86/io_apic.c +2 -1
@@ -584,16 +585,16 @@ set_ioapic_affinity_irq(struct irq_desc *desc, const cpumask_t *mask)
             dest = SET_APIC_LOGICAL_ID(dest);
         entry = irq_2_pin + irq;
         for (;;) {
-            unsigned int data;
+            struct IO_APIC_route_entry rte;
+
             pin = entry->pin;
             if (pin == -1)
                 break;
 
Quoted 13 lines

xen/arch/x86/io_apic.c set_ioapic_affinity_irq(struct irq_desc *desc, const cpumask_t *mask) · line 597

Quoted 9 lines
xen/arch/x86/io_apic.c +4 -5
-            io_apic_write(entry->apic, 0x10 + 1 + pin*2, dest);
-            data = io_apic_read(entry->apic, 0x10 + pin*2);
-            data &= ~IO_APIC_REDIR_VECTOR_MASK;
-            data |= MASK_INSR(desc->arch.vector, IO_APIC_REDIR_VECTOR_MASK);
-            io_apic_modify(entry->apic, 0x10 + pin*2, data);
+            rte = __ioapic_read_entry(entry->apic, pin, false);
+            rte.dest.dest32 = dest;
+            rte.vector = desc->arch.vector;
+            __ioapic_write_entry(entry->apic, pin, false, rte);

... this makes me wonder whether there shouldn't be an __ioapic_modify_entry() capable of suppressing one of the two writes (but still being handed the full RTE).

I've wondered about this, and I'm not sure how often can one of the two writes be suppressed here, as we modify both the low (for the vector) and the high part of the RTE (for the destination). It's unlikely that the same vector could be used on both destinations, and IMO such case doesn't warrant the introduction of the extra logic required in order to suppress one of the writes.

Am I overlooking something?

Thanks, Roger.

#8 lore raw

Re: [PATCH v2 2/5] x86/ioapic: RTE modifications must use ioapic_write_entry

Quoted 29 lines · Roger Pau Monné
Quoted 14 lines · Jan Beulich
Quoted 13 lines · Roger Pau Monne
xen/arch/x86/io_apic.c +2 -1
@@ -584,16 +585,16 @@ set_ioapic_affinity_irq(struct irq_desc *desc, const cpumask_t *mask)
             dest = SET_APIC_LOGICAL_ID(dest);
         entry = irq_2_pin + irq;
         for (;;) {
-            unsigned int data;
+            struct IO_APIC_route_entry rte;
+
             pin = entry->pin;
             if (pin == -1)
                 break;
 
Quoted 13 lines

xen/arch/x86/io_apic.c set_ioapic_affinity_irq(struct irq_desc *desc, const cpumask_t *mask) · line 597

Quoted 9 lines
xen/arch/x86/io_apic.c +4 -5
-            io_apic_write(entry->apic, 0x10 + 1 + pin*2, dest);
-            data = io_apic_read(entry->apic, 0x10 + pin*2);
-            data &= ~IO_APIC_REDIR_VECTOR_MASK;
-            data |= MASK_INSR(desc->arch.vector, IO_APIC_REDIR_VECTOR_MASK);
-            io_apic_modify(entry->apic, 0x10 + pin*2, data);
+            rte = __ioapic_read_entry(entry->apic, pin, false);
+            rte.dest.dest32 = dest;
+            rte.vector = desc->arch.vector;
+            __ioapic_write_entry(entry->apic, pin, false, rte);

... this makes me wonder whether there shouldn't be an __ioapic_modify_entry() capable of suppressing one of the two writes (but still being handed the full RTE).

Quoted 8 lines

I've wondered about this, and I'm not sure how often can one of the two writes be suppressed here, as we modify both the low (for the vector) and the high part of the RTE (for the destination). It's unlikely that the same vector could be used on both destinations, and IMO such case doesn't warrant the introduction of the extra logic required in order to suppress one of the writes.

Am I overlooking something?

Oh, no, that was me seeing the io_apic_modify() without paying attention to the earlier io_apic_write() (both in the code you replace).

Jan

#9 lore raw

PATCH v2 3/5 iommu/vtd: rename io_apic_read_remap_rte() local variable

In reply to Roger Pau Monne

Preparatory change to unify the IO-APIC pin variable name between io_apic_read_remap_rte() and amd_iommu_ioapic_update_ire(), so that the local variable can be made a function parameter with the same name across vendors.

---
Changes since v1:
- New in this version.

---
 xen/drivers/passthrough/vtd/intremap.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
xen/drivers/passthrough/vtd/intremap.c +4 -4
index 786388b4d9c7..05df6d5759b1 100644
@@ -441,14 +441,14 @@ unsigned int cf_check io_apic_read_remap_rte(
 void cf_check io_apic_write_remap_rte(
     unsigned int apic, unsigned int reg, unsigned int value)
 {
-    unsigned int ioapic_pin = (reg - 0x10) / 2;
+    unsigned int pin = (reg - 0x10) / 2;
     struct IO_xAPIC_route_entry old_rte = { };
     struct IO_APIC_route_remap_entry *remap_rte;
     unsigned int rte_upper = (reg & 1) ? 1 : 0;
     struct vtd_iommu *iommu = ioapic_to_iommu(IO_APIC_ID(apic));
     int saved_mask;
 
-    old_rte = __ioapic_read_entry(apic, ioapic_pin, true);
+    old_rte = __ioapic_read_entry(apic, pin, true);
 
     remap_rte = (struct IO_APIC_route_remap_entry *) &old_rte;
 
@@ -458,7 +458,7 @@ void cf_check io_apic_write_remap_rte(
     __io_apic_write(apic, reg & ~1, *(u32 *)&old_rte);
     remap_rte->mask = saved_mask;
 
-    if ( ioapic_rte_to_remap_entry(iommu, apic, ioapic_pin,
+    if ( ioapic_rte_to_remap_entry(iommu, apic, pin,
                                    &old_rte, rte_upper, value) )
     {
         __io_apic_write(apic, reg, value);
@@ -468,7 +468,7 @@ void cf_check io_apic_write_remap_rte(
             __io_apic_write(apic, reg & ~1, *(u32 *)&old_rte);
     }
     else
-        __ioapic_write_entry(apic, ioapic_pin, true, old_rte);
+        __ioapic_write_entry(apic, pin, true, old_rte);
 }
 
 static void set_msi_source_id(struct pci_dev *pdev, struct iremap_entry *ire)
Signature · 1 line
2.41.0
#10 lore raw

PATCH v2 4/5 x86/iommu: pass full IO-APIC RTE for remapping table update

In reply to Roger Pau Monne

So that the remapping entry can be updated atomically when possible.

Doing such update atomically will avoid Xen having to mask the IO-APIC pin prior to performing any interrupt movements (ie: changing the destination and vector fields), as the interrupt remapping entry is always consistent.

This also simplifies some of the logic on both VT-d and AMD-Vi implementations, as having the full RTE available instead of half of it avoids to possibly read and update the missing other half from hardware.

While there remove the explicit zeroing of new_ire fields in ioapic_rte_to_remap_entry() and initialize the variable at definition so all fields are zeroed. Note fields could be also initialized with final values at definition, but I found that likely too much to be done at this time.

---
Note that certain combination of changes to the RTE are impossible to handle atomically. For example changing the vector and/or destination fields together with the triggering mode is impossible to be performed atomically (as the destination and vector is set in the IRTE, but the triggering mode is set in the RTE). Xen doesn't attempt to perform such changes in a single update to the RTE anyway, so it's fine.

Naming the iommu_update_ire_from_apic() parameter RTE is not really correct, as the format of the passed value expands the destination field to be 32bits (in order to fit an x2APIC ID). Passing an IO_APIC_route_entry struct is not possible due to the circular dependency that would create between io_apic.h and iommu.h. It might be possible to move IO_APIC_route_entry declaration to a different header, but I haven't looked into it.

---
 xen/arch/x86/include/asm/iommu.h         |   3 +-
 xen/arch/x86/io_apic.c                   |   5 +-
 xen/drivers/passthrough/amd/iommu.h      |   2 +-
 xen/drivers/passthrough/amd/iommu_intr.c |  98 ++---------------
 xen/drivers/passthrough/vtd/extern.h     |   2 +-
 xen/drivers/passthrough/vtd/intremap.c   | 127 +++++++++++------------
 xen/drivers/passthrough/x86/iommu.c      |   4 +-
 xen/include/xen/iommu.h                  |   3 +-
 8 files changed, 80 insertions(+), 164 deletions(-)
xen/arch/x86/include/asm/iommu.h +2 -1
index 0540cd9faa87..eb720205e25e 100644
@@ -84,7 +84,8 @@ struct iommu_init_ops {
 
 extern const struct iommu_init_ops *iommu_init_ops;
 
-void iommu_update_ire_from_apic(unsigned int apic, unsigned int reg, unsigned int value);
+void iommu_update_ire_from_apic(unsigned int apic, unsigned int pin,
+                                uint64_t rte);
 unsigned int iommu_read_apic_from_ire(unsigned int apic, unsigned int reg);
 int iommu_setup_hpet_msi(struct msi_desc *);
 
xen/arch/x86/io_apic.c +1 -4
index 85b4b4c6bc98..255ba3e8b073 100644
@@ -275,10 +275,7 @@ void __ioapic_write_entry(
         __io_apic_write(apic, 0x10 + 2 * pin, eu.w1);
     }
     else
-    {
-        iommu_update_ire_from_apic(apic, 0x11 + 2 * pin, eu.w2);
-        iommu_update_ire_from_apic(apic, 0x10 + 2 * pin, eu.w1);
-    }
+        iommu_update_ire_from_apic(apic, pin, e.raw);
 }
 
 static void ioapic_write_entry(
xen/drivers/passthrough/amd/iommu.h +1 -1
index 8bc3c35b1bb1..aa3681a9f2aa 100644
@@ -300,7 +300,7 @@ int cf_check amd_iommu_free_intremap_table(
 unsigned int amd_iommu_intremap_table_order(
     const void *irt, const struct amd_iommu *iommu);
 void cf_check amd_iommu_ioapic_update_ire(
-    unsigned int apic, unsigned int reg, unsigned int value);
+    unsigned int apic, unsigned int pin, uint64_t raw);
 unsigned int cf_check amd_iommu_read_ioapic_from_ire(
     unsigned int apic, unsigned int reg);
 int cf_check amd_iommu_msi_msg_update_ire(
xen/drivers/passthrough/amd/iommu_intr.c +8 -90

149 lines not on this page · raw message

xen/drivers/passthrough/vtd/extern.h +1 -1
index 39602d1f88f8..032a7c3b42cd 100644
@@ -92,7 +92,7 @@ int cf_check intel_iommu_get_reserved_device_memory(
 unsigned int cf_check io_apic_read_remap_rte(
     unsigned int apic, unsigned int reg);
 void cf_check io_apic_write_remap_rte(
-    unsigned int apic, unsigned int reg, unsigned int value);
+    unsigned int apic, unsigned int ioapic_pin, uint64_t raw);
 
 struct msi_desc;
 struct msi_msg;
xen/drivers/passthrough/vtd/intremap.c +63 -64

159 lines not on this page · raw message

xen/drivers/passthrough/x86/iommu.c +2 -2
index be71a4c4641c..d290855959f2 100644
@@ -158,9 +158,9 @@ int iommu_enable_x2apic(void)
 }
 
 void iommu_update_ire_from_apic(
-    unsigned int apic, unsigned int reg, unsigned int value)
+    unsigned int apic, unsigned int pin, uint64_t rte)
 {
-    iommu_vcall(&iommu_ops, update_ire_from_apic, apic, reg, value);
+    iommu_vcall(&iommu_ops, update_ire_from_apic, apic, pin, rte);
 }
 
 unsigned int iommu_read_apic_from_ire(unsigned int apic, unsigned int reg)
xen/include/xen/iommu.h +2 -1
index 405db59971c5..9335cd074cff 100644
@@ -278,7 +278,8 @@ struct iommu_ops {
     int (*enable_x2apic)(void);
     void (*disable_x2apic)(void);
 
-    void (*update_ire_from_apic)(unsigned int apic, unsigned int reg, unsigned int value);
+    void (*update_ire_from_apic)(unsigned int apic, unsigned int pin,
+                                 uint64_t rte);
     unsigned int (*read_apic_from_ire)(unsigned int apic, unsigned int reg);
 
     int (*setup_hpet_msi)(struct msi_desc *);
Signature · 1 line
2.41.0
#11 lore raw

Re: [PATCH v2 4/5] x86/iommu: pass full IO-APIC RTE for remapping table update

Quoted 34 lines · Roger Pau Monne

So that the remapping entry can be updated atomically when possible.

Doing such update atomically will avoid Xen having to mask the IO-APIC pin prior to performing any interrupt movements (ie: changing the destination and vector fields), as the interrupt remapping entry is always consistent.

This also simplifies some of the logic on both VT-d and AMD-Vi implementations, as having the full RTE available instead of half of it avoids to possibly read and update the missing other half from hardware.

While there remove the explicit zeroing of new_ire fields in ioapic_rte_to_remap_entry() and initialize the variable at definition so all fields are zeroed. Note fields could be also initialized with final values at definition, but I found that likely too much to be done at this time.

---
Note that certain combination of changes to the RTE are impossible to handle atomically. For example changing the vector and/or destination fields together with the triggering mode is impossible to be performed atomically (as the destination and vector is set in the IRTE, but the triggering mode is set in the RTE). Xen doesn't attempt to perform such changes in a single update to the RTE anyway, so it's fine.

Naming the iommu_update_ire_from_apic() parameter RTE is not really correct, as the format of the passed value expands the destination field to be 32bits (in order to fit an x2APIC ID). Passing an IO_APIC_route_entry struct is not possible due to the circular dependency that would create between io_apic.h and iommu.h. It might be possible to move IO_APIC_route_entry declaration to a different header, but I haven't looked into it.

Quoted 10 lines
---
 xen/arch/x86/include/asm/iommu.h         |   3 +-
 xen/arch/x86/io_apic.c                   |   5 +-
 xen/drivers/passthrough/amd/iommu.h      |   2 +-
 xen/drivers/passthrough/amd/iommu_intr.c |  98 ++---------------
 xen/drivers/passthrough/vtd/extern.h     |   2 +-
 xen/drivers/passthrough/vtd/intremap.c   | 127 +++++++++++------------
 xen/drivers/passthrough/x86/iommu.c      |   4 +-
 xen/include/xen/iommu.h                  |   3 +-
 8 files changed, 80 insertions(+), 164 deletions(-)

Nice diffstat.

xen/arch/x86/include/asm/iommu.h struct iommu_init_ops { · line 88

Quoted 9 lines
xen/arch/x86/include/asm/iommu.h +2 -1
@@ -84,7 +84,8 @@ struct iommu_init_ops {
 
 extern const struct iommu_init_ops *iommu_init_ops;
 
-void iommu_update_ire_from_apic(unsigned int apic, unsigned int reg, unsigned int value);
+void iommu_update_ire_from_apic(unsigned int apic, unsigned int pin,
+                                uint64_t rte);

Much like you have it here, ...

xen/drivers/passthrough/amd/iommu.h int cf_check amd_iommu_free_intremap_table( · line 303

Quoted 8 lines
xen/drivers/passthrough/amd/iommu.h +1 -1
@@ -300,7 +300,7 @@ int cf_check amd_iommu_free_intremap_table(
 unsigned int amd_iommu_intremap_table_order(
     const void *irt, const struct amd_iommu *iommu);
 void cf_check amd_iommu_ioapic_update_ire(
-    unsigned int apic, unsigned int reg, unsigned int value);
+    unsigned int apic, unsigned int pin, uint64_t raw);

... could the changed parameter also have "rte" in its name? I don't mind if it's "raw_rte", to avoid colliding with existing variable names (in at least the VT-d counterpart).

xen/drivers/passthrough/amd/iommu.h static int update_intremap_entry_from_ioapic( · line 324

Quoted 8 lines
xen/drivers/passthrough/amd/iommu.h +1 -1
@@ -350,14 +319,11 @@ static int update_intremap_entry_from_ioapic(
 }
 
 void cf_check amd_iommu_ioapic_update_ire(
-    unsigned int apic, unsigned int reg, unsigned int value)
+    unsigned int apic, unsigned int pin, uint64_t raw)
 {
     struct IO_APIC_route_entry old_rte = { };

Looks like the initializer here now isn't needed anymore?

Quoted 51 lines
xen/drivers/passthrough/amd/iommu.h +5 -41
@@ -364,48 +363,37 @@ static int ioapic_rte_to_remap_entry(struct vtd_iommu *iommu,
 
     new_ire = *iremap_entry;
 
-    if ( rte_upper )
-    {
-        if ( x2apic_enabled )
-            new_ire.remap.dst = value;
-        else
-            new_ire.remap.dst = (value >> 24) << 8;
-    }
+    if ( x2apic_enabled )
+        new_ire.remap.dst = new_rte.dest.dest32;
     else
-    {
-        *(((u32 *)&new_rte) + 0) = value;
-        new_ire.remap.fpd = 0;
-        new_ire.remap.dm = new_rte.dest_mode;
-        new_ire.remap.tm = new_rte.trigger;
-        new_ire.remap.dlm = new_rte.delivery_mode;
-        /* Hardware require RH = 1 for LPR delivery mode */
-        new_ire.remap.rh = (new_ire.remap.dlm == dest_LowestPrio);
-        new_ire.remap.avail = 0;
-        new_ire.remap.res_1 = 0;
-        new_ire.remap.vector = new_rte.vector;
-        new_ire.remap.res_2 = 0;
-
-        set_ioapic_source_id(IO_APIC_ID(apic), &new_ire);
-        new_ire.remap.res_3 = 0;
-        new_ire.remap.res_4 = 0;
-        new_ire.remap.p = 1;     /* finally, set present bit */
-
-        /* now construct new ioapic rte entry */
-        remap_rte->vector = new_rte.vector;
-        remap_rte->delivery_mode = 0;    /* has to be 0 for remap format */
-        remap_rte->index_15 = (index >> 15) & 0x1;
-        remap_rte->index_0_14 = index & 0x7fff;
-
-        remap_rte->delivery_status = new_rte.delivery_status;
-        remap_rte->polarity = new_rte.polarity;
-        remap_rte->irr = new_rte.irr;
-        remap_rte->trigger = new_rte.trigger;
-        remap_rte->mask = new_rte.mask;
-        remap_rte->reserved = 0;
-        remap_rte->format = 1;    /* indicate remap format */
-    }
-
-    update_irte(iommu, iremap_entry, &new_ire, !init);
+        new_ire.remap.dst = (new_rte.dest.dest32 >> 24) << 8;
+
+    new_ire.remap.dm = new_rte.dest_mode;

xen/drivers/passthrough/amd/iommu.h static int ioapic_rte_to_remap_entry(struct vtd_iommu *iommu, · line 381

Quoted 10 lines
xen/drivers/passthrough/amd/iommu.h +10 -0
+    new_ire.remap.tm = new_rte.trigger;
+    new_ire.remap.dlm = new_rte.delivery_mode;
+    /* Hardware require RH = 1 for LPR delivery mode */
+    new_ire.remap.rh = (new_ire.remap.dlm == dest_LowestPrio);
+    new_ire.remap.vector = new_rte.vector;
+
+    set_ioapic_source_id(IO_APIC_ID(apic), &new_ire);
+    new_ire.remap.p = 1;     /* finally, set present bit */
+
+    /* now construct new ioapic rte entry */

Nit: Would you mind correcting comment style here as you touch (unindent) the line?

Quoted 5 lines
xen/drivers/passthrough/amd/iommu.h +5 -0
+    remap_rte->vector = new_rte.vector;
+    remap_rte->delivery_mode = 0;    /* has to be 0 for remap format */
+    remap_rte->index_15 = (index >> 15) & 0x1;
+    remap_rte->index_0_14 = index & 0x7fff;
+

xen/drivers/passthrough/amd/iommu.h static int ioapic_rte_to_remap_entry(struct vtd_iommu *iommu, · line 396

Quoted 10 lines
xen/drivers/passthrough/amd/iommu.h +10 -0
+    remap_rte->delivery_status = new_rte.delivery_status;
+    remap_rte->polarity = new_rte.polarity;
+    remap_rte->irr = new_rte.irr;
+    remap_rte->trigger = new_rte.trigger;
+    remap_rte->mask = new_rte.mask;
+    remap_rte->reserved = 0;
+    remap_rte->format = 1;    /* indicate remap format */
+
+    /* If cmpxchg16b is not available the caller must mask the IO-APIC pin. */
+    update_irte(iommu, iremap_entry, &new_ire, !init && cpu_has_cx16);

The comment suggests to me that you mean to strengthen the logic, yet the code change weakens it (the called function BUG()s when "atomic" is true). Wouldn't this better be "!init && !<old RTE was/is masked>" anyway?

Quoted 39 lines
xen/drivers/passthrough/amd/iommu.h +12 -15
@@ -439,36 +427,47 @@ unsigned int cf_check io_apic_read_remap_rte(
 }
 
 void cf_check io_apic_write_remap_rte(
-    unsigned int apic, unsigned int reg, unsigned int value)
+    unsigned int apic, unsigned int pin, uint64_t raw)
 {
-    unsigned int pin = (reg - 0x10) / 2;
+    struct IO_xAPIC_route_entry rte = { .raw = raw };
     struct IO_xAPIC_route_entry old_rte = { };
     struct IO_APIC_route_remap_entry *remap_rte;
-    unsigned int rte_upper = (reg & 1) ? 1 : 0;
     struct vtd_iommu *iommu = ioapic_to_iommu(IO_APIC_ID(apic));
-    int saved_mask;
-
-    old_rte = __ioapic_read_entry(apic, pin, true);
+    bool masked = true;
+    int rc;
 
     remap_rte = (struct IO_APIC_route_remap_entry *) &old_rte;
 
-    /* mask the interrupt while we change the intremap table */
-    saved_mask = remap_rte->mask;
-    remap_rte->mask = 1;
-    __io_apic_write(apic, reg & ~1, *(u32 *)&old_rte);
-    remap_rte->mask = saved_mask;
-
-    if ( ioapic_rte_to_remap_entry(iommu, apic, pin,
-                                   &old_rte, rte_upper, value) )
+    if ( !cpu_has_cx16 )
     {
-        __io_apic_write(apic, reg, value);
+       /*
+        * Cannot atomically update the IRTE entry: mask the IO-APIC pin to
+        * avoid interrupts seeing an inconsistent IRTE entry.
+        */
+        old_rte = __ioapic_read_entry(apic, pin, true);
+        if ( !old_rte.mask )
+        {

xen/drivers/passthrough/amd/iommu.h unsigned int cf_check io_apic_read_remap_rte( · line 456

Quoted 10 lines
xen/drivers/passthrough/amd/iommu.h +6 -3
+            masked = false;
+            old_rte.mask = 1;
+            __ioapic_write_entry(apic, pin, true, old_rte);
+        }
+    }
 
-        /* Recover the original value of 'mask' bit */
-        if ( rte_upper )
-            __io_apic_write(apic, reg & ~1, *(u32 *)&old_rte);
+    rc = ioapic_rte_to_remap_entry(iommu, apic, pin, &old_rte, rte);

I realize it has been like this before, but passing &old_rte here is odd. We already have its properly typed alias: remap_rte. All the called function does is do the same type cast again. Question is whether ...

Quoted 6 lines
xen/drivers/passthrough/amd/iommu.h +6 -0
+    if ( rc )
+    {
+        if ( !masked )
+        {
+            /* Recover the original value of 'mask' bit */
+            old_rte.mask = 0;

xen/drivers/passthrough/amd/iommu.h unsigned int cf_check io_apic_read_remap_rte( · line 470

Quoted 10 lines
xen/drivers/passthrough/amd/iommu.h +7 -2
+            __ioapic_write_entry(apic, pin, true, old_rte);
+        }
+        dprintk(XENLOG_ERR VTDPREFIX,
+                "failed to update IRTE for IO-APIC#%u pin %u: %d\n",
+                apic, pin, rc);
+        return;
     }
-    else
-        __ioapic_write_entry(apic, pin, true, old_rte);
+    __ioapic_write_entry(apic, pin, true, old_rte);

... the further uses of old_rte then won't end up yet more confusing than they already are (first and foremost again because of "old" not being applicable here).

Jan

#12 lore raw

Re: [PATCH v2 4/5] x86/iommu: pass full IO-APIC RTE for remapping table update

Quoted 73 lines · Jan Beulich
Quoted 39 lines · Roger Pau Monne
xen/drivers/passthrough/vtd/intremap.c +12 -15
@@ -439,36 +427,47 @@ unsigned int cf_check io_apic_read_remap_rte(
 }
 
 void cf_check io_apic_write_remap_rte(
-    unsigned int apic, unsigned int reg, unsigned int value)
+    unsigned int apic, unsigned int pin, uint64_t raw)
 {
-    unsigned int pin = (reg - 0x10) / 2;
+    struct IO_xAPIC_route_entry rte = { .raw = raw };
     struct IO_xAPIC_route_entry old_rte = { };
     struct IO_APIC_route_remap_entry *remap_rte;
-    unsigned int rte_upper = (reg & 1) ? 1 : 0;
     struct vtd_iommu *iommu = ioapic_to_iommu(IO_APIC_ID(apic));
-    int saved_mask;
-
-    old_rte = __ioapic_read_entry(apic, pin, true);
+    bool masked = true;
+    int rc;
 
     remap_rte = (struct IO_APIC_route_remap_entry *) &old_rte;
 
-    /* mask the interrupt while we change the intremap table */
-    saved_mask = remap_rte->mask;
-    remap_rte->mask = 1;
-    __io_apic_write(apic, reg & ~1, *(u32 *)&old_rte);
-    remap_rte->mask = saved_mask;
-
-    if ( ioapic_rte_to_remap_entry(iommu, apic, pin,
-                                   &old_rte, rte_upper, value) )
+    if ( !cpu_has_cx16 )
     {
-        __io_apic_write(apic, reg, value);
+       /*
+        * Cannot atomically update the IRTE entry: mask the IO-APIC pin to
+        * avoid interrupts seeing an inconsistent IRTE entry.
+        */
+        old_rte = __ioapic_read_entry(apic, pin, true);
+        if ( !old_rte.mask )
+        {

xen/drivers/passthrough/vtd/intremap.c unsigned int cf_check io_apic_read_remap_rte( · line 456

Quoted 10 lines
xen/drivers/passthrough/vtd/intremap.c +6 -3
+            masked = false;
+            old_rte.mask = 1;
+            __ioapic_write_entry(apic, pin, true, old_rte);
+        }
+    }
 
-        /* Recover the original value of 'mask' bit */
-        if ( rte_upper )
-            __io_apic_write(apic, reg & ~1, *(u32 *)&old_rte);
+    rc = ioapic_rte_to_remap_entry(iommu, apic, pin, &old_rte, rte);

I realize it has been like this before, but passing &old_rte here is odd. We already have its properly typed alias: remap_rte. All the called function does is do the same type cast again. Question is whether ...

Quoted 6 lines
xen/drivers/passthrough/vtd/intremap.c +6 -0
+    if ( rc )
+    {
+        if ( !masked )
+        {
+            /* Recover the original value of 'mask' bit */
+            old_rte.mask = 0;

xen/drivers/passthrough/vtd/intremap.c unsigned int cf_check io_apic_read_remap_rte( · line 470

Quoted 10 lines
xen/drivers/passthrough/vtd/intremap.c +7 -2
+            __ioapic_write_entry(apic, pin, true, old_rte);
+        }
+        dprintk(XENLOG_ERR VTDPREFIX,
+                "failed to update IRTE for IO-APIC#%u pin %u: %d\n",
+                apic, pin, rc);
+        return;
     }
-    else
-        __ioapic_write_entry(apic, pin, true, old_rte);
+    __ioapic_write_entry(apic, pin, true, old_rte);
Quoted 3 lines

... the further uses of old_rte then won't end up yet more confusing than they already are (first and foremost again because of "old" not being applicable here).

I've instead opted to remove remap_rte from io_apic_write_remap_rte(), as it was unused. I've also added a comment to clarify the usage of old_rte when ioapic_rte_to_remap_entry() returns success.

Thanks, Roger.

#13 lore raw

PATCH v2 5/5 amd/iommu: force atomic updates of remapping entries

In reply to Roger Pau Monne

Disable XT (x2APIC) support and thus 128 IRTE entries if cmpxchg16b is not available, do so in order to avoid having to disable the IRTE (and possibly loose interrupts) when updating the entry. Note this also removes the usage of a while loop in order to disable the entry, since RemapEn is no longer toggled when updating.

---
x2APIC support was added late enough on AMD that CPUs that I believe all models that have x2APIC must also have CX16.

The AMD-Vi manual contains the following advice in the "2.3.2 Making Guest Interrupt Virtualization Changes" section:

"""
If RemapEn=1 before the change, the following steps may be followed to change interrupt virtualization information:
* Set RemapEn=0 in the entry to disable interrupt virtualization; device-initiated interrupt requests for the DeviceID and vector are processed as IO_PAGE_FAULT events.
* Update the fields in the IRTE and invalidate the interrupt table (see Section 2.4.5 [INVALIDATE_INTERRUPT_TABLE]).
* Set RemapEn=1 to virtualize interrupts from the device.
"""

However if the update of the IRTE is done atomically I assume that setting RemapEn=0 is not longer necessary, and that the INVALIDATE_INTERRUPT_TABLE command can be executed after the entry has been (atomically) updated.

Note that on VT-d IRTEs always have a size of 128b, so it's not possible to use a smaller entry size if CX16 is not available in order to do atomic updates.

---
 xen/drivers/passthrough/amd/iommu_init.c | 10 +++++
 xen/drivers/passthrough/amd/iommu_intr.c | 57 +++++++-----------------
 2 files changed, 26 insertions(+), 41 deletions(-)
xen/drivers/passthrough/amd/iommu_init.c +10 -0
index af6713d2fc02..e276856507a1 100644
@@ -1417,6 +1417,16 @@ int __init amd_iommu_prepare(bool xt)
             has_xt = false;
     }
 
+    /*
+     * Prevent using 128bit IRTE format if there's no support for cmpxchg16b
+     * to update the entry atomically.
+     */
+    if ( xt && has_xt && !cpu_has_cx16 )
+    {
+        has_xt = false;
+        printk(XENLOG_WARNING "AMD-Vi: x2APIC not supported without CX16\n");
+    }
+
     if ( ivhd_type != ACPI_IVRS_TYPE_HARDWARE )
         iommuv2_enabled = true;
 
xen/drivers/passthrough/amd/iommu_intr.c +16 -41

101 lines not on this page · raw message

Signature · 1 line
2.41.0
#14 lore raw

Re: [PATCH v2 5/5] amd/iommu: force atomic updates of remapping entries

Quoted 25 lines · Roger Pau Monne

Disable XT (x2APIC) support and thus 128 IRTE entries if cmpxchg16b is not available, do so in order to avoid having to disable the IRTE (and possibly loose interrupts) when updating the entry. Note this also removes the usage of a while loop in order to disable the entry, since RemapEn is no longer toggled when updating.

---
x2APIC support was added late enough on AMD that CPUs that I believe all models that have x2APIC must also have CX16.

The AMD-Vi manual contains the following advice in the "2.3.2 Making Guest Interrupt Virtualization Changes" section:

"""
If RemapEn=1 before the change, the following steps may be followed to change interrupt virtualization information:
* Set RemapEn=0 in the entry to disable interrupt virtualization; device-initiated interrupt requests for the DeviceID and vector are processed as IO_PAGE_FAULT events.
* Update the fields in the IRTE and invalidate the interrupt table (see Section 2.4.5 [INVALIDATE_INTERRUPT_TABLE]).
* Set RemapEn=1 to virtualize interrupts from the device.
"""

Quoted 4 lines

However if the update of the IRTE is done atomically I assume that setting RemapEn=0 is not longer necessary, and that the INVALIDATE_INTERRUPT_TABLE command can be executed after the entry has been (atomically) updated.

There's one additional prereq to this: The IOMMU also needs to read 128-bit IRTEs atomically. I'm afraid we can't take this for given without it being said explicitly in the spec (I've just gone through and looked at all occurrences of "atomic", without finding anything applicable to IRTEs). If this can be resolved, I think I'm fine with the patch.

Jan

#15 lore raw

Re: [PATCH v2 5/5] amd/iommu: force atomic updates of remapping entries

Quoted 31 lines · Jan Beulich
Quoted 25 lines · Roger Pau Monne

Disable XT (x2APIC) support and thus 128 IRTE entries if cmpxchg16b is not available, do so in order to avoid having to disable the IRTE (and possibly loose interrupts) when updating the entry. Note this also removes the usage of a while loop in order to disable the entry, since RemapEn is no longer toggled when updating.

---
x2APIC support was added late enough on AMD that CPUs that I believe all models that have x2APIC must also have CX16.

The AMD-Vi manual contains the following advice in the "2.3.2 Making Guest Interrupt Virtualization Changes" section:

"""
If RemapEn=1 before the change, the following steps may be followed to change interrupt virtualization information:
* Set RemapEn=0 in the entry to disable interrupt virtualization; device-initiated interrupt requests for the DeviceID and vector are processed as IO_PAGE_FAULT events.
* Update the fields in the IRTE and invalidate the interrupt table (see Section 2.4.5 [INVALIDATE_INTERRUPT_TABLE]).
* Set RemapEn=1 to virtualize interrupts from the device.
"""

Quoted 4 lines

However if the update of the IRTE is done atomically I assume that setting RemapEn=0 is not longer necessary, and that the INVALIDATE_INTERRUPT_TABLE command can be executed after the entry has been (atomically) updated.

Quoted 6 lines

There's one additional prereq to this: The IOMMU also needs to read 128-bit IRTEs atomically. I'm afraid we can't take this for given without it being said explicitly in the spec (I've just gone through and looked at all occurrences of "atomic", without finding anything applicable to IRTEs). If this can be resolved, I think I'm fine with the patch.

Hm, indeed I was taking for granted that the IOMMU will read the entry atomically.

I was also worried about the IOMMU caching only parts of the IRTE, and thus even when doing an atomic read of the full entry it could still get inconsistent data if using previously cached parts of the entry. The text in INVALIDATE_INTERRUPT_TABLE seems to suggest that IRTE fields could be cached on a field by field basis.

I will raise a question with AMD in order to try to figure out whether the IOMMU will do atomic reads, and also cache the full entry.

Thanks, Roger.