x86/ioapic: fix edge triggered interrupt migration
v3 cover x86/ioapic: fix edge triggered interrupt migration
Hello,
Following series attempts to solve the issue with IO-APIC edge triggered interrupts seeing an inconsistent IRTE when injected while being migrated.
It's been simplified a bit from the original RFC, and does make the logic in the IOMMU RTE update handlers cleaner, as those get passed the full RTE.
When not using interrupt remapping the logic is left as-is. One further change would be to mask edge triggered interrupts, as destination and vector cannot be changed atomically when dealing with plain IO-APIC RTEs.
The previously final patch to switch AMD IOMMU to use atomic RTE updates has been drop pending feedback from AMD on whether that approach is suitable.
Thanks, Roger.
Roger Pau Monne (4):
x86/ioapic: add a raw field to RTE struct x86/ioapic: RTE modifications must use ioapic_write_entry iommu/vtd: rename io_apic_read_remap_rte() local variable x86/iommu: pass full IO-APIC RTE for remapping table update
xen/arch/x86/include/asm/io_apic.h | 65 ++++++----- xen/arch/x86/include/asm/iommu.h | 3 +- xen/arch/x86/io_apic.c | 46 ++++---- xen/drivers/passthrough/amd/iommu.h | 2 +- xen/drivers/passthrough/amd/iommu_intr.c | 106 ++--------------- xen/drivers/passthrough/vtd/extern.h | 2 +- xen/drivers/passthrough/vtd/intremap.c | 140 +++++++++++------------ xen/drivers/passthrough/x86/iommu.c | 4 +- xen/include/xen/iommu.h | 3 +- 9 files changed, 142 insertions(+), 229 deletions(-)
Signature · 1 line
2.41.0
PATCH v3 1/4 x86/ioapic: add a raw field to RTE struct
Further changes will require access to the full RTE as a single value in order to pass it to IOMMU interrupt remapping handlers.
No functional change intended.
---
Changes since v1:
- Fix initializers.
---
Tested on gitlab CI, builds on all compilers used there.
--- xen/arch/x86/include/asm/io_apic.h | 57 +++++++++++++----------- xen/arch/x86/io_apic.c | 2 +- xen/drivers/passthrough/amd/iommu_intr.c | 4 +- xen/drivers/passthrough/vtd/intremap.c | 4 +- 4 files changed, 35 insertions(+), 32 deletions(-)
xen/arch/x86/include/asm/io_apic.h +30 -27
@@ -90,35 +90,38 @@ enum ioapic_irq_destination_types { };struct IO_APIC_route_entry { -unsigned int vector:8 ; -unsigned int delivery_mode:3 ;/* -* 000: FIXED -* 001: lowest prio -* 111: ExtINT -*/ -unsigned int dest_mode:1 ;/* 0: physical, 1: logical */ -unsigned int delivery_status:1 ; -unsigned int polarity:1 ;/* 0: low, 1: high */ -unsigned int irr:1 ; -unsigned int trigger:1 ;/* 0: edge, 1: level */ -unsigned int mask:1 ;/* 0: enabled, 1: disabled */ -unsigned int __reserved_2:15 ; -union {struct { -unsigned int __reserved_1:24 ; -unsigned int physical_dest:4 ; -unsigned int __reserved_2:4 ; - } physical; - -struct { -unsigned int __reserved_1:24 ; -unsigned int logical_dest:8 ; - } logical; - -/* used when Interrupt Remapping with EIM is enabled */ -unsigned int dest32; - } dest; +unsigned int vector:8 ; +unsigned int delivery_mode:3 ;/* +* 000: FIXED +* 001: lowest prio +* 111: ExtINT +*/ +unsigned int dest_mode:1 ;/* 0: physical, 1: logical */ +unsigned int delivery_status:1 ; +unsigned int polarity:1 ;/* 0: low, 1: high */ +unsigned int irr:1 ; +unsigned int trigger:1 ;/* 0: edge, 1: level */ +unsigned int mask:1 ;/* 0: enabled, 1: disabled */ +unsigned int __reserved_2:15 ; + +union { +struct { +unsigned int __reserved_1:24 ; +unsigned int physical_dest:4 ; +unsigned int __reserved_2:4 ; + } physical; + +struct { +unsigned int __reserved_1:24 ; +unsigned int logical_dest:8 ; + } logical; +unsigned int dest32; + } dest; + }; +uint64_t raw; + }; };/*
xen/arch/x86/io_apic.c +1 -1
@@ -2360,7 +2360,7 @@ int ioapic_guest_read(unsigned long physbase, unsigned int reg, u32 *pval)int ioapic_guest_write (unsigned long physbase,unsigned int reg,u32 val) {int apic, pin, irq, ret, pirq; -struct IO_APIC_route_entry rte = {0 }; +struct IO_APIC_route_entry rte = { };unsigned long flags;struct irq_desc *desc;
xen/drivers/passthrough/amd/iommu_intr.c +2 -2
@@ -352,8 +352,8 @@ static int update_intremap_entry_from_ioapic( void cf_check amd_iommu_ioapic_update_ire(unsigned int apic,unsigned int reg,unsigned int value) { -struct IO_APIC_route_entry old_rte = {0 }; -struct IO_APIC_route_entry new_rte = {0 }; +struct IO_APIC_route_entry old_rte = { }; +struct IO_APIC_route_entry new_rte = { };unsigned int rte_lo = (reg &1 ) ? reg -1 : reg;unsigned int pin = (reg -0x10 ) /2 ;int seg, bdf, rc;
xen/drivers/passthrough/vtd/intremap.c +2 -2
@@ -419,7 +419,7 @@ unsigned int cf_check io_apic_read_remap_rte( {unsigned int ioapic_pin = (reg -0x10 ) /2 ;int index; -struct IO_xAPIC_route_entry old_rte = {0 }; +struct IO_xAPIC_route_entry old_rte = { };int rte_upper = (reg &1 ) ?1 :0 ;struct vtd_iommu *iommu =ioapic_to_iommu (IO_APIC_ID (apic));
@@ -442,7 +442,7 @@ void cf_check io_apic_write_remap_rte( unsigned int apic, unsigned int reg, unsigned int value) { unsigned int ioapic_pin = (reg - 0x10) / 2; -struct IO_xAPIC_route_entry old_rte = {0 }; +struct IO_xAPIC_route_entry old_rte = { };struct IO_APIC_route_remap_entry *remap_rte;unsigned int rte_upper = (reg &1 ) ?1 :0 ;struct vtd_iommu *iommu =ioapic_to_iommu (IO_APIC_ID (apic));
Signature · 1 line
2.41.0
PATCH v3 2/4 x86/ioapic: RTE modifications must use ioapic_write_entry
In reply to Roger Pau MonneDo not allow to write to RTE registers using io_apic_write and instead require changes to RTE to be performed using ioapic_write_entry.
This is in preparation for passing the full contents of the RTE to the IOMMU interrupt remapping handlers, so remapping entries for IO-APIC RTEs can be updated atomically when possible.
While immediately this commit might expand the number of MMIO accesses in order to update an IO-APIC RTE, further changes will benefit from getting the full RTE value passed to the IOMMU handlers, as the logic is greatly simplified when the IOMMU handlers can get the complete RTE value in one go.
---
Changes since v2:
- Also adjust __ioapic_read_entry().
Changes since v1:
- Reinstate io_apic_modify().
- Expand commit message.
--- xen/arch/x86/include/asm/io_apic.h | 8 ++--- xen/arch/x86/io_apic.c | 43 ++++++++++++------------ xen/drivers/passthrough/amd/iommu_intr.c | 6 ---- 3 files changed, 25 insertions(+), 32 deletions(-)
xen/arch/x86/include/asm/io_apic.h +4 -4
@@ -162,8 +162,8 @@ static inline void __io_apic_write(unsigned int apic, unsigned int reg, unsignedstatic inline void io_apic_write (unsigned int apic,unsigned int reg,unsigned int value) { -if (ioapic_reg_remapped (reg) ) -return iommu_update_ire_from_apic (apic, reg, value); +/* RTE writes must use ioapic_write_entry. */ +BUG_ON (reg >=0x10 );__io_apic_write (apic, reg, value); }
@@ -173,8 +173,8 @@ static inline void io_apic_write(unsigned int apic, unsigned int reg, unsigned i*/ static inline void io_apic_modify (unsigned int apic,unsigned int reg,unsigned int value) { -if (ioapic_reg_remapped (reg) ) -return iommu_update_ire_from_apic (apic, reg, value); +/* RTE writes must use ioapic_write_entry. */ +BUG_ON (reg >=0x10 ); *(IO_APIC_BASE (apic) +4 ) = value; }
xen/arch/x86/io_apic.c +21 -22
93 lines not on this page · raw message
xen/drivers/passthrough/amd/iommu_intr.c +0 -6
@@ -361,12 +361,6 @@ void cf_check amd_iommu_ioapic_update_ire(struct amd_iommu *iommu;unsigned int idx; -if ( !iommu_intremap ) - { -__io_apic_write (apic, reg, value); -return ; - } - idx =ioapic_id_to_index (IO_APIC_ID (apic));if ( idx ==MAX_IO_APICS )return ;
Signature · 1 line
2.41.0
Re: [PATCH v3 2/4] x86/ioapic: RTE modifications must use ioapic_write_entry
Quoted 7 lines · Roger Pau Monne
Do not allow to write to RTE registers using
io_apic_writeand instead require changes to RTE to be performed usingioapic_write_entry.This is in preparation for passing the full contents of the RTE to the IOMMU interrupt remapping handlers, so remapping entries for IO-APIC RTEs can be updated atomically when possible.
Quoted 7 lines
While immediately this commit might expand the number of MMIO accesses in order to update an IO-APIC RTE, further changes will benefit from getting the full RTE value passed to the IOMMU handlers, as the logic is greatly simplified when the IOMMU handlers can get the complete RTE value in one go.
Signed-off-by Roger Pau Monné <[email protected]>
PATCH v3 3/4 iommu/vtd: rename io_apic_read_remap_rte() local variable
In reply to Roger Pau MonnePreparatory change to unify the IO-APIC pin variable name between io_apic_read_remap_rte() and amd_iommu_ioapic_update_ire(), so that the local variable can be made a function parameter with the same name across vendors.
---
Changes since v1:
- New in this version.
--- xen/drivers/passthrough/vtd/intremap.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
xen/drivers/passthrough/vtd/intremap.c +4 -4
@@ -441,14 +441,14 @@ unsigned int cf_check io_apic_read_remap_rte( void cf_check io_apic_write_remap_rte(unsigned int apic,unsigned int reg,unsigned int value) { -unsigned int ioapic_pin = (reg -0x10 ) /2 ; +unsigned int pin = (reg -0x10 ) /2 ;struct IO_xAPIC_route_entry old_rte = { };struct IO_APIC_route_remap_entry *remap_rte;unsigned int rte_upper = (reg &1 ) ?1 :0 ;struct vtd_iommu *iommu =ioapic_to_iommu (IO_APIC_ID (apic));int saved_mask; - old_rte =__ioapic_read_entry (apic, ioapic_pin, true); + old_rte =__ioapic_read_entry (apic, pin, true); remap_rte = (struct IO_APIC_route_remap_entry *) &old_rte;
@@ -458,7 +458,7 @@ void cf_check io_apic_write_remap_rte(__io_apic_write (apic, reg & ~1 , *(u32 *)&old_rte); remap_rte->mask = saved_mask; -if (ioapic_rte_to_remap_entry (iommu, apic, ioapic_pin, +if (ioapic_rte_to_remap_entry (iommu, apic, pin, &old_rte, rte_upper, value) ) {__io_apic_write (apic, reg, value);
@@ -468,7 +468,7 @@ void cf_check io_apic_write_remap_rte(__io_apic_write (apic, reg & ~1 , *(u32 *)&old_rte); }else -__ioapic_write_entry (apic, ioapic_pin, true, old_rte); +__ioapic_write_entry (apic, pin, true, old_rte); }static void set_msi_source_id (struct pci_dev *pdev,struct iremap_entry *ire)
Signature · 1 line
2.41.0
RE: [PATCH v3 3/4] iommu/vtd: rename io_apic_read_remap_rte() local variable
Quoted 9 lines
From: Roger Pau Monne <[email protected]>
Sent: Wednesday, July 26, 2023 8:55 PMPreparatory change to unify the IO-APIC pin variable name between
io_apic_read_remap_rte()andamd_iommu_ioapic_update_ire(), so that the local variable can be made a function parameter with the same name across vendors.Signed-off-by Roger Pau Monné <[email protected]>
PATCH v3 4/4 x86/iommu: pass full IO-APIC RTE for remapping table update
In reply to Roger Pau MonneSo that the remapping entry can be updated atomically when possible.
Doing such update atomically will avoid Xen having to mask the IO-APIC pin prior to performing any interrupt movements (ie: changing the destination and vector fields), as the interrupt remapping entry is always consistent.
This also simplifies some of the logic on both VT-d and AMD-Vi implementations, as having the full RTE available instead of half of it avoids to possibly read and update the missing other half from hardware.
While there remove the explicit zeroing of new_ire fields in ioapic_rte_to_remap_entry() and initialize the variable at definition so all fields are zeroed. Note fields could be also initialized with final values at definition, but I found that likely too much to be done at this time.
---
Changes since v2:
- Remove unneeded initialization.
- Use 'rte' as parameter name for update_ire_from_apic() implementations.
- Fix comment style in ioapic_rte_to_remap_entry().
- Fix requirement for atomic write in update_irte() call from ioapic_rte_to_remap_entry().
- Remove remap_rte from io_apic_write_remap_rte().
---
Note that certain combination of changes to the RTE are impossible to handle atomically. For example changing the vector and/or destination fields together with the triggering mode is impossible to be performed atomically (as the destination and vector is set in the IRTE, but the triggering mode is set in the RTE). Xen doesn't attempt to perform such changes in a single update to the RTE anyway, so it's fine.
Naming the iommu_update_ire_from_apic() parameter RTE is not really correct, as the format of the passed value expands the destination field to be 32bits (in order to fit an x2APIC ID). Passing an IO_APIC_route_entry struct is not possible due to the circular dependency that would create between io_apic.h and iommu.h. It might be possible to move IO_APIC_route_entry declaration to a different header, but I haven't looked into it.
--- xen/arch/x86/include/asm/iommu.h | 3 +- xen/arch/x86/io_apic.c | 5 +- xen/drivers/passthrough/amd/iommu.h | 2 +- xen/drivers/passthrough/amd/iommu_intr.c | 100 ++--------------- xen/drivers/passthrough/vtd/extern.h | 2 +- xen/drivers/passthrough/vtd/intremap.c | 136 +++++++++++------------ xen/drivers/passthrough/x86/iommu.c | 4 +- xen/include/xen/iommu.h | 3 +- 8 files changed, 86 insertions(+), 169 deletions(-)
xen/arch/x86/include/asm/iommu.h +2 -1
@@ -84,7 +84,8 @@ struct iommu_init_ops {extern const struct iommu_init_ops *iommu_init_ops; -void iommu_update_ire_from_apic (unsigned int apic,unsigned int reg,unsigned int value); +void iommu_update_ire_from_apic (unsigned int apic,unsigned int pin, +uint64_t rte);unsigned int iommu_read_apic_from_ire (unsigned int apic,unsigned int reg);int iommu_setup_hpet_msi (struct msi_desc *);
xen/arch/x86/io_apic.c +1 -4
@@ -275,10 +275,7 @@ void __ioapic_write_entry(__io_apic_write (apic,0x10 +2 * pin, eu.w1); }else - { -iommu_update_ire_from_apic (apic,0x11 +2 * pin, eu.w2); -iommu_update_ire_from_apic (apic,0x10 +2 * pin, eu.w1); - } +iommu_update_ire_from_apic (apic, pin, e.raw); } static void ioapic_write_entry(
xen/drivers/passthrough/amd/iommu.h +1 -1
@@ -300,7 +300,7 @@ int cf_check amd_iommu_free_intremap_table(unsigned int amd_iommu_intremap_table_order (const void *irt,const struct amd_iommu *iommu); void cf_check amd_iommu_ioapic_update_ire( -unsigned int apic,unsigned int reg,unsigned int value); +unsigned int apic,unsigned int pin,uint64_t rte); unsigned int cf_check amd_iommu_read_ioapic_from_ire(unsigned int apic,unsigned int reg); int cf_check amd_iommu_msi_msg_update_ire(
xen/drivers/passthrough/amd/iommu_intr.c +9 -91
150 lines not on this page · raw message
xen/drivers/passthrough/vtd/extern.h +1 -1
@@ -92,7 +92,7 @@ int cf_check intel_iommu_get_reserved_device_memory( unsigned int cf_check io_apic_read_remap_rte(unsigned int apic,unsigned int reg); void cf_check io_apic_write_remap_rte( -unsigned int apic,unsigned int reg,unsigned int value); +unsigned int apic,unsigned int ioapic_pin,uint64_t rte);struct msi_desc ;struct msi_msg ;
xen/drivers/passthrough/vtd/intremap.c +68 -68
167 lines not on this page · raw message
xen/drivers/passthrough/x86/iommu.c +2 -2
@@ -158,9 +158,9 @@ int iommu_enable_x2apic(void) }void iommu_update_ire_from_apic ( -unsigned int apic,unsigned int reg,unsigned int value) +unsigned int apic,unsigned int pin,uint64_t rte) { -iommu_vcall (&iommu_ops, update_ire_from_apic, apic, reg, value); +iommu_vcall (&iommu_ops, update_ire_from_apic, apic, pin, rte); }unsigned int iommu_read_apic_from_ire (unsigned int apic,unsigned int reg)
xen/include/xen/iommu.h +2 -1
@@ -278,7 +278,8 @@ struct iommu_ops {int (*enable_x2apic)(void );void (*disable_x2apic)(void ); -void (*update_ire_from_apic)(unsigned int apic,unsigned int reg,unsigned int value); +void (*update_ire_from_apic)(unsigned int apic,unsigned int pin, +uint64_t rte);unsigned int (*read_apic_from_ire)(unsigned int apic,unsigned int reg);int (*setup_hpet_msi)(struct msi_desc *);
Signature · 1 line
2.41.0
Re: [PATCH v3 4/4] x86/iommu: pass full IO-APIC RTE for remapping table update
xen/drivers/passthrough/vtd/extern.h int cf_check intel_iommu_get_reserved_device_memory( · line 95
Quoted 8 lines · Roger Pau Monne
xen/drivers/passthrough/vtd/extern.h +1
-1@@ -92,7 +92,7 @@ int cf_check intel_iommu_get_reserved_device_memory( unsigned int cf_check io_apic_read_remap_rte(unsigned int apic,unsigned int reg); void cf_check io_apic_write_remap_rte( -unsigned int apic,unsigned int reg,unsigned int value); +unsigned int apic,unsigned int ioapic_pin,uint64_t rte);
Forgot to rename the middle parameter to "pin"?
Quoted 39 lines
xen/drivers/passthrough/vtd/extern.h +2
-32@@ -364,48 +363,40 @@ static int ioapic_rte_to_remap_entry(struct vtd_iommu *iommu, new_ire = *iremap_entry; -if ( rte_upper ) - { -if ( x2apic_enabled ) - new_ire.remap.dst = value; -else - new_ire.remap.dst = (value >>24 ) <<8 ; - } +if ( x2apic_enabled ) + new_ire.remap.dst = new_rte.dest.dest32;else - { - *(((u32 *)&new_rte) +0 ) = value; - new_ire.remap.fpd =0 ; - new_ire.remap.dm = new_rte.dest_mode; - new_ire.remap.tm = new_rte.trigger; - new_ire.remap.dlm = new_rte.delivery_mode; -/* Hardware require RH = 1 for LPR delivery mode */ - new_ire.remap.rh = (new_ire.remap.dlm == dest_LowestPrio); - new_ire.remap.avail =0 ; - new_ire.remap.res_1 =0 ; - new_ire.remap.vector = new_rte.vector; - new_ire.remap.res_2 =0 ; - -set_ioapic_source_id (IO_APIC_ID (apic), &new_ire); - new_ire.remap.res_3 =0 ; - new_ire.remap.res_4 =0 ; - new_ire.remap.p =1 ;/* finally, set present bit */ - -/* now construct new ioapic rte entry */ - remap_rte->vector = new_rte.vector; - remap_rte->delivery_mode =0 ;/* has to be 0 for remap format */ - remap_rte->index_15 = (index >>15 ) &0x1 ; - remap_rte->index_0_14 = index &0x7fff ; - - remap_rte->delivery_status = new_rte.delivery_status;
xen/drivers/passthrough/vtd/extern.h static int ioapic_rte_to_remap_entry(struct vtd_iommu *iommu, · line 369
Quoted 10 lines
xen/drivers/passthrough/vtd/extern.h +1
-9- remap_rte->polarity = new_rte.polarity; - remap_rte->irr = new_rte.irr; - remap_rte->trigger = new_rte.trigger; - remap_rte->mask = new_rte.mask; - remap_rte->reserved =0 ; - remap_rte->format =1 ;/* indicate remap format */ - } - -update_irte (iommu, iremap_entry, &new_ire, !init); + new_ire.remap.dst = (new_rte.dest.dest32 >>24 ) <<8 ;
I realize this was this way before, but I wonder if we couldn't use GET_xAPIC_ID() here to reduce the open-coding at least a bit.
Quoted 50 lines
xen/drivers/passthrough/vtd/extern.h +19
-21@@ -439,36 +430,45 @@ unsigned int cf_check io_apic_read_remap_rte( } void cf_check io_apic_write_remap_rte( -unsigned int apic,unsigned int reg,unsigned int value) +unsigned int apic,unsigned int pin,uint64_t rte) { -unsigned int pin = (reg -0x10 ) /2 ; +struct IO_xAPIC_route_entry new_rte = { .raw = rte };struct IO_xAPIC_route_entry old_rte = { }; -struct IO_APIC_route_remap_entry *remap_rte; -unsigned int rte_upper = (reg &1 ) ?1 :0 ;struct vtd_iommu *iommu =ioapic_to_iommu (IO_APIC_ID (apic)); -int saved_mask; - - old_rte =__ioapic_read_entry (apic, pin, true); +bool masked = true; +int rc; - remap_rte = (struct IO_APIC_route_remap_entry *) &old_rte; - -/* mask the interrupt while we change the intremap table */ - saved_mask = remap_rte->mask; - remap_rte->mask =1 ; -__io_apic_write (apic, reg & ~1 , *(u32 *)&old_rte); - remap_rte->mask = saved_mask; - -if (ioapic_rte_to_remap_entry (iommu, apic, pin, - &old_rte, rte_upper, value) ) +if ( !cpu_has_cx16 ) { -__io_apic_write (apic, reg, value); +/* +* Cannot atomically update the IRTE entry: mask the IO-APIC pin to +* avoid interrupts seeing an inconsistent IRTE entry. +*/ + old_rte =__ioapic_read_entry (apic, pin, true); +if ( !old_rte.mask ) + { + masked = false; + old_rte.mask =1 ; +__ioapic_write_entry (apic, pin, true, old_rte); + } + } -/* Recover the original value of 'mask' bit */ -if ( rte_upper ) -__io_apic_write (apic, reg & ~1 , *(u32 *)&old_rte); + rc =ioapic_rte_to_remap_entry (iommu, apic, pin, &old_rte, new_rte); +if ( rc )
xen/drivers/passthrough/vtd/extern.h unsigned int cf_check io_apic_read_remap_rte( · line 467
Quoted 10 lines
xen/drivers/passthrough/vtd/extern.h +10
-0+ { +if ( !masked ) + { +/* Recover the original value of 'mask' bit */ + old_rte.mask =0 ; +__ioapic_write_entry (apic, pin, true, old_rte); + } + dprintk(XENLOG_ERR VTDPREFIX, +"failed to update IRTE for IO-APIC#%u pin %u: %d\n" , + apic, pin, rc);
Afaics you don't alter the error behavior of ioapic_rte_to_remap_entry(), and in the sole (pre-existing) case of an error a debug log message is already being issued. Why the addition?
Jan
Re: [PATCH v3 4/4] x86/iommu: pass full IO-APIC RTE for remapping table update
Quoted 57 lines · Jan Beulich
Quoted 56 lines · Roger Pau Monne
xen/drivers/passthrough/vtd/intremap.c +25
-21@@ -439,36 +430,45 @@ unsigned int cf_check io_apic_read_remap_rte( } void cf_check io_apic_write_remap_rte( -unsigned int apic,unsigned int reg,unsigned int value) +unsigned int apic,unsigned int pin,uint64_t rte) { -unsigned int pin = (reg -0x10 ) /2 ; +struct IO_xAPIC_route_entry new_rte = { .raw = rte };struct IO_xAPIC_route_entry old_rte = { }; -struct IO_APIC_route_remap_entry *remap_rte; -unsigned int rte_upper = (reg &1 ) ?1 :0 ;struct vtd_iommu *iommu =ioapic_to_iommu (IO_APIC_ID (apic)); -int saved_mask; - - old_rte =__ioapic_read_entry (apic, pin, true); +bool masked = true; +int rc; - remap_rte = (struct IO_APIC_route_remap_entry *) &old_rte; - -/* mask the interrupt while we change the intremap table */ - saved_mask = remap_rte->mask; - remap_rte->mask =1 ; -__io_apic_write (apic, reg & ~1 , *(u32 *)&old_rte); - remap_rte->mask = saved_mask; - -if (ioapic_rte_to_remap_entry (iommu, apic, pin, - &old_rte, rte_upper, value) ) +if ( !cpu_has_cx16 ) { -__io_apic_write (apic, reg, value); +/* +* Cannot atomically update the IRTE entry: mask the IO-APIC pin to +* avoid interrupts seeing an inconsistent IRTE entry. +*/ + old_rte =__ioapic_read_entry (apic, pin, true); +if ( !old_rte.mask ) + { + masked = false; + old_rte.mask =1 ; +__ioapic_write_entry (apic, pin, true, old_rte); + } + } -/* Recover the original value of 'mask' bit */ -if ( rte_upper ) -__io_apic_write (apic, reg & ~1 , *(u32 *)&old_rte); + rc =ioapic_rte_to_remap_entry (iommu, apic, pin, &old_rte, new_rte); +if ( rc ) + { +if ( !masked ) + { +/* Recover the original value of 'mask' bit */ + old_rte.mask =0 ; +__ioapic_write_entry (apic, pin, true, old_rte);
Quoted 8 lines
xen/drivers/passthrough/vtd/intremap.c unsigned int cf_check io_apic_read_remap_rte( · line 467
Quoted 4 lines
xen/drivers/passthrough/vtd/intremap.c +4
-0+ } + dprintk(XENLOG_ERR VTDPREFIX, +"failed to update IRTE for IO-APIC#%u pin %u: %d\n" , + apic, pin, rc);Afaics you don't alter the error behavior of
ioapic_rte_to_remap_entry(), and in the sole (pre-existing) case of an error a debug log message is already being issued. Why the addition?
I think I was trying to mimic the behavior of amd_iommu_ioapic_update_ire(), which does print the errors as opposed to doing so in update_intremap_entry_from_ioapic().
I've now removed it, and adjusted the code to match the rest of your comments. Will post v4 of 4/4 only as a reply to v3, I don't think there's a need to resend the rest unless you prefer it that way.
Thanks, Roger.
Re: [PATCH v3 4/4] x86/iommu: pass full IO-APIC RTE for remapping table update
Quoted 67 lines · Roger Pau Monné
Quoted 57 lines · Jan Beulich
Quoted 56 lines · Roger Pau Monne
xen/drivers/passthrough/vtd/intremap.c +25
-21@@ -439,36 +430,45 @@ unsigned int cf_check io_apic_read_remap_rte( } void cf_check io_apic_write_remap_rte( -unsigned int apic,unsigned int reg,unsigned int value) +unsigned int apic,unsigned int pin,uint64_t rte) { -unsigned int pin = (reg -0x10 ) /2 ; +struct IO_xAPIC_route_entry new_rte = { .raw = rte };struct IO_xAPIC_route_entry old_rte = { }; -struct IO_APIC_route_remap_entry *remap_rte; -unsigned int rte_upper = (reg &1 ) ?1 :0 ;struct vtd_iommu *iommu =ioapic_to_iommu (IO_APIC_ID (apic)); -int saved_mask; - - old_rte =__ioapic_read_entry (apic, pin, true); +bool masked = true; +int rc; - remap_rte = (struct IO_APIC_route_remap_entry *) &old_rte; - -/* mask the interrupt while we change the intremap table */ - saved_mask = remap_rte->mask; - remap_rte->mask =1 ; -__io_apic_write (apic, reg & ~1 , *(u32 *)&old_rte); - remap_rte->mask = saved_mask; - -if (ioapic_rte_to_remap_entry (iommu, apic, pin, - &old_rte, rte_upper, value) ) +if ( !cpu_has_cx16 ) { -__io_apic_write (apic, reg, value); +/* +* Cannot atomically update the IRTE entry: mask the IO-APIC pin to +* avoid interrupts seeing an inconsistent IRTE entry. +*/ + old_rte =__ioapic_read_entry (apic, pin, true); +if ( !old_rte.mask ) + { + masked = false; + old_rte.mask =1 ; +__ioapic_write_entry (apic, pin, true, old_rte); + } + } -/* Recover the original value of 'mask' bit */ -if ( rte_upper ) -__io_apic_write (apic, reg & ~1 , *(u32 *)&old_rte); + rc =ioapic_rte_to_remap_entry (iommu, apic, pin, &old_rte, new_rte); +if ( rc ) + { +if ( !masked ) + { +/* Recover the original value of 'mask' bit */ + old_rte.mask =0 ; +__ioapic_write_entry (apic, pin, true, old_rte);Quoted 8 lines
xen/drivers/passthrough/vtd/intremap.c unsigned int cf_check io_apic_read_remap_rte( · line 467
Quoted 4 lines
xen/drivers/passthrough/vtd/intremap.c +4
-0+ } + dprintk(XENLOG_ERR VTDPREFIX, +"failed to update IRTE for IO-APIC#%u pin %u: %d\n" , + apic, pin, rc);Afaics you don't alter the error behavior of
ioapic_rte_to_remap_entry(), and in the sole (pre-existing) case of an error a debug log message is already being issued. Why the addition?
Quoted 7 lines
I think I was trying to mimic the behavior of
amd_iommu_ioapic_update_ire(), which does print the errors as opposed to doing so inupdate_intremap_entry_from_ioapic().I've now removed it, and adjusted the code to match the rest of your comments. Will post v4 of 4/4 only as a reply to v3, I don't think there's a need to resend the rest unless you prefer it that way.
Just this patch is going to be fine (maybe even as v3.1).
Jan
PATCH v4 4/4 x86/iommu: pass full IO-APIC RTE for remapping table update
In reply to Roger Pau MonneSo that the remapping entry can be updated atomically when possible.
Doing such update atomically will avoid Xen having to mask the IO-APIC pin prior to performing any interrupt movements (ie: changing the destination and vector fields), as the interrupt remapping entry is always consistent.
This also simplifies some of the logic on both VT-d and AMD-Vi implementations, as having the full RTE available instead of half of it avoids to possibly read and update the missing other half from hardware.
While there remove the explicit zeroing of new_ire fields in ioapic_rte_to_remap_entry() and initialize the variable at definition so all fields are zeroed. Note fields could be also initialized with final values at definition, but I found that likely too much to be done at this time.
---
Changes since v3:
- Fix prototype parameter name to match definition.
- Use GET_xAPIC_ID().
- Remove duplicated error message in io_apic_write_remap_rte().
Changes since v2:
- Remove unneeded initialization.
- Use 'rte' as parameter name for update_ire_from_apic() implementations.
- Fix comment style in ioapic_rte_to_remap_entry().
- Fix requirement for atomic write in update_irte() call from ioapic_rte_to_remap_entry().
- Remove remap_rte from io_apic_write_remap_rte().
---
Note that certain combination of changes to the RTE are impossible to handle atomically. For example changing the vector and/or destination fields together with the triggering mode is impossible to be performed atomically (as the destination and vector is set in the IRTE, but the triggering mode is set in the RTE). Xen doesn't attempt to perform such changes in a single update to the RTE anyway, so it's fine.
Naming the iommu_update_ire_from_apic() parameter RTE is not really correct, as the format of the passed value expands the destination field to be 32bits (in order to fit an x2APIC ID). Passing an IO_APIC_route_entry struct is not possible due to the circular dependency that would create between io_apic.h and iommu.h. It might be possible to move IO_APIC_route_entry declaration to a different header, but I haven't looked into it.
--- xen/arch/x86/include/asm/iommu.h | 3 +- xen/arch/x86/io_apic.c | 5 +- xen/drivers/passthrough/amd/iommu.h | 2 +- xen/drivers/passthrough/amd/iommu_intr.c | 100 ++--------------- xen/drivers/passthrough/vtd/extern.h | 2 +- xen/drivers/passthrough/vtd/intremap.c | 131 +++++++++++------------ xen/drivers/passthrough/x86/iommu.c | 4 +- xen/include/xen/iommu.h | 3 +- 8 files changed, 82 insertions(+), 168 deletions(-)
xen/arch/x86/include/asm/iommu.h +2 -1
@@ -84,7 +84,8 @@ struct iommu_init_ops {extern const struct iommu_init_ops *iommu_init_ops; -void iommu_update_ire_from_apic (unsigned int apic,unsigned int reg,unsigned int value); +void iommu_update_ire_from_apic (unsigned int apic,unsigned int pin, +uint64_t rte);unsigned int iommu_read_apic_from_ire (unsigned int apic,unsigned int reg);int iommu_setup_hpet_msi (struct msi_desc *);
xen/arch/x86/io_apic.c +1 -4
@@ -275,10 +275,7 @@ void __ioapic_write_entry(__io_apic_write (apic,0x10 +2 * pin, eu.w1); }else - { -iommu_update_ire_from_apic (apic,0x11 +2 * pin, eu.w2); -iommu_update_ire_from_apic (apic,0x10 +2 * pin, eu.w1); - } +iommu_update_ire_from_apic (apic, pin, e.raw); } static void ioapic_write_entry(
xen/drivers/passthrough/amd/iommu.h +1 -1
@@ -300,7 +300,7 @@ int cf_check amd_iommu_free_intremap_table(unsigned int amd_iommu_intremap_table_order (const void *irt,const struct amd_iommu *iommu); void cf_check amd_iommu_ioapic_update_ire( -unsigned int apic,unsigned int reg,unsigned int value); +unsigned int apic,unsigned int pin,uint64_t rte); unsigned int cf_check amd_iommu_read_ioapic_from_ire(unsigned int apic,unsigned int reg); int cf_check amd_iommu_msi_msg_update_ire(
xen/drivers/passthrough/amd/iommu_intr.c +9 -91
150 lines not on this page · raw message
xen/drivers/passthrough/vtd/extern.h +1 -1
@@ -92,7 +92,7 @@ int cf_check intel_iommu_get_reserved_device_memory( unsigned int cf_check io_apic_read_remap_rte(unsigned int apic,unsigned int reg); void cf_check io_apic_write_remap_rte( -unsigned int apic,unsigned int reg,unsigned int value); +unsigned int apic,unsigned int pin,uint64_t rte);struct msi_desc ;struct msi_msg ;
xen/drivers/passthrough/vtd/intremap.c +64 -67
163 lines not on this page · raw message
xen/drivers/passthrough/x86/iommu.c +2 -2
@@ -158,9 +158,9 @@ int iommu_enable_x2apic(void) }void iommu_update_ire_from_apic ( -unsigned int apic,unsigned int reg,unsigned int value) +unsigned int apic,unsigned int pin,uint64_t rte) { -iommu_vcall (&iommu_ops, update_ire_from_apic, apic, reg, value); +iommu_vcall (&iommu_ops, update_ire_from_apic, apic, pin, rte); }unsigned int iommu_read_apic_from_ire (unsigned int apic,unsigned int reg)
xen/include/xen/iommu.h +2 -1
@@ -278,7 +278,8 @@ struct iommu_ops {int (*enable_x2apic)(void );void (*disable_x2apic)(void ); -void (*update_ire_from_apic)(unsigned int apic,unsigned int reg,unsigned int value); +void (*update_ire_from_apic)(unsigned int apic,unsigned int pin, +uint64_t rte);unsigned int (*read_apic_from_ire)(unsigned int apic,unsigned int reg);int (*setup_hpet_msi)(struct msi_desc *);
Signature · 1 line
2.41.0
Re: [PATCH v4 4/4] x86/iommu: pass full IO-APIC RTE for remapping table update
Quoted 12 lines · Roger Pau Monne
So that the remapping entry can be updated atomically when possible.
Doing such update atomically will avoid Xen having to mask the IO-APIC pin prior to performing any interrupt movements (ie: changing the destination and vector fields), as the interrupt remapping entry is always consistent.
This also simplifies some of the logic on both VT-d and AMD-Vi implementations, as having the full RTE available instead of half of it avoids to possibly read and update the missing other half from hardware.
Quoted 7 lines
While there remove the explicit zeroing of
new_irefields inioapic_rte_to_remap_entry()and initialize the variable at definition so all fields are zeroed. Note fields could be also initialized with final values at definition, but I found that likely too much to be done at this time.Signed-off-by Roger Pau Monné <[email protected]>
RE: [PATCH v4 4/4] x86/iommu: pass full IO-APIC RTE for remapping table update
In reply to Roger Pau MonneQuoted 15 lines
From: Roger Pau Monne <[email protected]>
Sent: Friday, July 28, 2023 5:57 PMSo that the remapping entry can be updated atomically when possible.
Doing such update atomically will avoid Xen having to mask the IO-APIC pin prior to performing any interrupt movements (ie: changing the destination and vector fields), as the interrupt remapping entry is always consistent.
This also simplifies some of the logic on both VT-d and AMD-Vi implementations, as having the full RTE available instead of half of it avoids to possibly read and update the missing other half from hardware.
Quoted 7 lines
While there remove the explicit zeroing of
new_irefields inioapic_rte_to_remap_entry()and initialize the variable at definition so all fields are zeroed. Note fields could be also initialized with final values at definition, but I found that likely too much to be done at this time.Signed-off-by Roger Pau Monné <[email protected]>