iommu/vt-d: Allocate DMAR fault interrupts locally
Subsystems (3)
PATCH v2 1/1 iommu/vt-d: Allocate DMAR fault interrupts locally
The Intel IOMMU code currently tries to allocate all DMAR fault interrupt vectors on the boot cpu. On large systems with high DMAR counts this results in vector exhaustion, and most of the vectors are not initially allocated socket local.
Instead, have a cpu on each node do the vector allocation for the DMARs on that node. The boot cpu still does the allocation for its node during its boot sequence.
---
v2: per Thomas Gleixner, implement this from a DYN CPU hotplug state, though
this implementation runs in CPUHP_AP_ONLINE_DYN space rather than CPUHP_BP_PREPARE_DYN space.
drivers/iommu/amd/amd_iommu.h | 2 +- drivers/iommu/amd/init.c | 2 +- drivers/iommu/intel/dmar.c | 9 +++++++-- drivers/iommu/irq_remapping.c | 5 ++++- drivers/iommu/irq_remapping.h | 2 +- include/linux/dmar.h | 2 +- 6 files changed, 15 insertions(+), 7 deletions(-)
drivers/iommu/amd/amd_iommu.h +1 -1
@@ -33,7 +33,7 @@ int amd_iommu_prepare(void);int amd_iommu_enable (void );void amd_iommu_disable (void );int amd_iommu_reenable (int mode); -int amd_iommu_enable_faulting (void ); +int amd_iommu_enable_faulting (unsigned int cpu);extern int amd_iommu_guest_ir;extern enum io_pgtable_fmt amd_iommu_pgtable;extern int amd_iommu_gpt_level;
drivers/iommu/amd/init.c +1 -1
@@ -3389,7 +3389,7 @@ int amd_iommu_reenable(int mode)return 0 ; } -int __init amd_iommu_enable_faulting (void ) +int __init amd_iommu_enable_faulting (unsigned int cpu) {/* We enable MSI later when PCI is initialized */ return 0 ;
drivers/iommu/intel/dmar.c +7 -2
@@ -2122,7 +2122,7 @@ int dmar_set_interrupt(struct intel_iommu *iommu)return ret; } -int __init enable_drhd_fault_handling (void ) +int enable_drhd_fault_handling (unsigned int cpu) {struct dmar_drhd_unit *drhd;struct intel_iommu *iommu;
@@ -2132,7 +2132,12 @@ int __init enable_drhd_fault_handling(void)*/ for_each_iommu (iommu, drhd) {u32 fault_status; -int ret =dmar_set_interrupt (iommu); +int ret; + +if (iommu->irq || iommu->node !=cpu_to_node (cpu)) +continue ; + + ret =dmar_set_interrupt (iommu);if (ret) { pr_err("DRHD %Lx: failed to enable fault, interrupt, ret %d\n",
drivers/iommu/irq_remapping.c +4 -1
@@ -151,7 +151,10 @@ int __init irq_remap_enable_fault_handling(void)if (!remap_ops->enable_faulting)return -ENODEV ; -return remap_ops->enable_faulting (); +cpuhp_setup_state (CPUHP_AP_ONLINE_DYN ,"dmar:enable_fault_handling" , + remap_ops->enable_faulting,NULL ); + +return remap_ops->enable_faulting (smp_processor_id ()); }void panic_if_irq_remap (const char *msg)
drivers/iommu/irq_remapping.h +1 -1
@@ -41,7 +41,7 @@ struct irq_remap_ops {int (*reenable)(int );/* Enable fault handling */ -int (*enable_faulting)(void ); +int (*enable_faulting)(unsigned int ); };extern struct irq_remap_ops intel_irq_remap_ops;
include/linux/dmar.h +1 -1
@@ -117,7 +117,7 @@ extern int dmar_remove_dev_scope(struct dmar_pci_notify_info *info, int count);/* Intel IOMMU detection */ void detect_intel_iommu (void ); -extern int enable_drhd_fault_handling (void ); +extern int enable_drhd_fault_handling (unsigned int cpu);extern int dmar_device_add (acpi_handle handle);extern int dmar_device_remove (acpi_handle handle);
Signature · 1 line
2.35.3
Hi Dimitri,
Quoted 79 lines
-----Original Message-----
From: Dimitri Sivanich <[email protected]>
Sent: Friday, March 22, 2024 4:51 AM
To: Thomas Gleixner <[email protected]>; Joerg Roedel <[email protected]>; Suravee Suthikulpanit <[email protected]>; Will Deacon <[email protected]>; Robin Murphy <[email protected]>; David Woodhouse <[email protected]>; Lu Baolu <[email protected]>; Mark Rutland <[email protected]>; Peter Zijlstra <[email protected]>; Arnd Bergmann <[email protected]>; YueHaibing <[email protected]>; [email protected]; Dimitri Sivanich <[email protected]>
Cc: [email protected]; Steve Wahl <[email protected]>; Anderson, Russ <[email protected]>
Subject: [PATCH v2] iommu/vt-d: Allocate DMAR fault interrupts locallyThe Intel IOMMU code currently tries to allocate all DMAR fault interrupt vectors on the boot cpu. On large systems with high DMAR counts this results in vector exhaustion, and most of the vectors are not initially allocated socket local.
Instead, have a cpu on each node do the vector allocation for the DMARs on that node. The boot cpu still does the allocation for its node during its boot sequence.
Signed-off-by Dimitri Sivanich <[email protected]>---
v2: per Thomas Gleixner, implement this from a DYN CPU hotplug state, though
this implementation runs inCPUHP_AP_ONLINE_DYNspace rather thanCPUHP_BP_PREPARE_DYNspace.drivers/iommu/amd/amd_iommu.h | 2 +- drivers/iommu/amd/init.c | 2 +- drivers/iommu/intel/dmar.c | 9 +++++++-- drivers/iommu/irq_remapping.c | 5 ++++- drivers/iommu/irq_remapping.h | 2 +- include/linux/dmar.h | 2 +- 6 files changed, 15 insertions(+), 7 deletions(-)(quoted diff) +0
-0
b/drivers/iommu/amd/amd_iommu.hindexf482aab420f7..410c360e7e24100644drivers/iommu/amd/amd_iommu.h +0
-0@@ -33,7 +33,7 @@ int amd_iommu_prepare(void); int
amd_iommu_enable(void); voidamd_iommu_disable(void); intamd_iommu_reenable(int mode); -intamd_iommu_enable_faulting(void); +intamd_iommu_enable_faulting(unsigned int cpu);
extern intamd_iommu_guest_ir;
extern enumio_pgtable_fmtamd_iommu_pgtable; extern intamd_iommu_gpt_level; diff --gita/drivers/iommu/amd/init.cb/drivers/iommu/amd/init.cindexe7a44929f0da..4782f690ed97100644drivers/iommu/amd/init.c +1
-1@@ -3389,7 +3389,7 @@ int amd_iommu_reenable(int mode)return 0 ; } -int __init amd_iommu_enable_faulting (void ) +int __init amd_iommu_enable_faulting (unsigned int cpu) {/* We enable MSI later when PCI is initialized */ return 0 ;drivers/iommu/intel/dmar.c index renamed from drivers/iommu/intel/dmar.c +0
-0
36d7427b1202..7644a42f283c100644drivers/iommu/intel/dmar.c +1
-1@@ -2122,7 +2122,7 @@ int dmar_set_interrupt(struct intel_iommu *iommu)return ret; } -int __init enable_drhd_fault_handling (void ) +int enable_drhd_fault_handling (unsigned int cpu) {struct dmar_drhd_unit *drhd;drivers/iommu/intel/dmar.c +0
-0struct intel_iommu *iommu;
drivers/iommu/intel/dmar.c int __init enable_drhd_fault_handling(void) · line 2138
Quoted 8 lines
drivers/iommu/intel/dmar.c +4
-1@@ -2132,7 +2132,12 @@ int __init enable_drhd_fault_handling(void)*/ for_each_iommu (iommu, drhd) {u32 fault_status; -int ret =dmar_set_interrupt (iommu); +int ret; + +if (iommu->irq || iommu->node !=cpu_to_node (cpu)) +continue ;
If iommu->irq is set, current logic will clear any previous faults by accessing DMAR_FSTS_REG. However, the code change in this patch seems missing it.
The current logic:
int dmar_set_interrupt(struct intel_iommu *iommu)
{
int irq, ret;
/*
* Check if the fault interrupt is already initialized.
*/
if (iommu->irq)
return 0;
...int __init enable_drhd_fault_handling(void)
{
...
for_each_iommu(iommu, drhd) {
u32 fault_status;
int ret = dmar_set_interrupt(iommu);
if (ret) {
pr_err("DRHD %Lx: failed to enable fault, interrupt, ret %d\n",
(unsigned long long)drhd->reg_base_addr, ret);
return -1;
}
/*
* Clear any previous faults.
*/
dmar_fault(iommu->irq, iommu);
fault_status = readl(iommu->reg + DMAR_FSTS_REG);
writel(fault_status, iommu->reg + DMAR_FSTS_REG);
}
...Regards,
-Tina
Hi Tina,
Quoted 76 lines · Tina
Hi Dimitri,
Quoted 73 lines
-----Original Message-----
From: Dimitri Sivanich <[email protected]>
Sent: Friday, March 22, 2024 4:51 AM
To: Thomas Gleixner <[email protected]>; Joerg Roedel <[email protected]>; Suravee Suthikulpanit <[email protected]>; Will Deacon <[email protected]>; Robin Murphy <[email protected]>; David Woodhouse <[email protected]>; Lu Baolu <[email protected]>; Mark Rutland <[email protected]>; Peter Zijlstra <[email protected]>; Arnd Bergmann <[email protected]>; YueHaibing <[email protected]>; [email protected]; Dimitri Sivanich <[email protected]>
Cc: [email protected]; Steve Wahl <[email protected]>; Anderson, Russ <[email protected]>
Subject: [PATCH v2] iommu/vt-d: Allocate DMAR fault interrupts locallyThe Intel IOMMU code currently tries to allocate all DMAR fault interrupt vectors on the boot cpu. On large systems with high DMAR counts this results in vector exhaustion, and most of the vectors are not initially allocated socket local.
Instead, have a cpu on each node do the vector allocation for the DMARs on that node. The boot cpu still does the allocation for its node during its boot sequence.
Signed-off-by Dimitri Sivanich <[email protected]>---
v2: per Thomas Gleixner, implement this from a DYN CPU hotplug state, though
this implementation runs inCPUHP_AP_ONLINE_DYNspace rather thanCPUHP_BP_PREPARE_DYNspace.drivers/iommu/amd/amd_iommu.h | 2 +- drivers/iommu/amd/init.c | 2 +- drivers/iommu/intel/dmar.c | 9 +++++++-- drivers/iommu/irq_remapping.c | 5 ++++- drivers/iommu/irq_remapping.h | 2 +- include/linux/dmar.h | 2 +- 6 files changed, 15 insertions(+), 7 deletions(-)(quoted diff) +0
-0
b/drivers/iommu/amd/amd_iommu.hindexf482aab420f7..410c360e7e24100644drivers/iommu/amd/amd_iommu.h +0
-0@@ -33,7 +33,7 @@ int amd_iommu_prepare(void); int
amd_iommu_enable(void); voidamd_iommu_disable(void); intamd_iommu_reenable(int mode); -intamd_iommu_enable_faulting(void); +intamd_iommu_enable_faulting(unsigned int cpu);
extern intamd_iommu_guest_ir;
extern enumio_pgtable_fmtamd_iommu_pgtable; extern intamd_iommu_gpt_level; diff --gita/drivers/iommu/amd/init.cb/drivers/iommu/amd/init.cindexe7a44929f0da..4782f690ed97100644drivers/iommu/amd/init.c +1
-1@@ -3389,7 +3389,7 @@ int amd_iommu_reenable(int mode)return 0 ; } -int __init amd_iommu_enable_faulting (void ) +int __init amd_iommu_enable_faulting (unsigned int cpu) {/* We enable MSI later when PCI is initialized */ return 0 ;drivers/iommu/intel/dmar.c index renamed from drivers/iommu/intel/dmar.c +0
-0
36d7427b1202..7644a42f283c100644drivers/iommu/intel/dmar.c +0
-0@@ -2122,7 +2122,7 @@ int dmar_set_interrupt(struct intel_iommu *iommu)return ret; }
Quoted 15 lines
Quoted 6 lines
drivers/iommu/intel/dmar.c +1
-1-int __init enable_drhd_fault_handling (void ) +int enable_drhd_fault_handling (unsigned int cpu) {struct dmar_drhd_unit *drhd;drivers/iommu/intel/dmar.c +0
-0struct intel_iommu *iommu;drivers/iommu/intel/dmar.c int __init enable_drhd_fault_handling(void) · line 2138
Quoted 8 lines
drivers/iommu/intel/dmar.c +4
-1@@ -2132,7 +2132,12 @@ int __init enable_drhd_fault_handling(void)*/ for_each_iommu (iommu, drhd) {u32 fault_status; -int ret =dmar_set_interrupt (iommu); +int ret; + +if (iommu->irq || iommu->node !=cpu_to_node (cpu)) +continue ;If
iommu->irqis set, current logic will clear any previous faults by accessingDMAR_FSTS_REG. However, the code change in this patch seems missing it.
Yes, the current logic does clear the faults even in the iommu->irq!=0 case, but enable_drhd_fault_handling is currently only run once by the boot cpu, prior to startup of the other processors. So each iommu will, initially at least, only have its faults cleared once.
With this patch, enable_drhd_fault_handling will still run on the boot cpu prior to the startup of other processors, but will also run on each AP as it is brought up (hotplugged). It will now only operate on the iommus that are on its own socket, however. If we add back the fault clearing for the iommu->irq!=0 case, each iommu on a socket would end up having faults cleared once for each cpu on the socket (so 120 times for a 60-core socket with HT enabled). In addition, this would happen again each time a cpu is hot-plugged after having been hot-unplugged.
So is there a specific reason why faults would need to be cleared if the iommu->irq has already been set, since I assume they would've already been cleared during this boot?
I'm definitely willing to add back the fault clearing for the iommu->irq!=0 case, but am looking for guidance from others like yourself on this.
Quoted 38 lines
The current logic:
intdmar_set_interrupt(struct intel_iommu *iommu){ int irq, ret; /* * Check if the fault interrupt is already initialized. */ if (iommu->irq) return 0; ...int
__initenable_drhd_fault_handling(void){ ... for_each_iommu(iommu, drhd) { u32 fault_status; int ret = dmar_set_interrupt(iommu); if (ret) { pr_err("DRHD %Lx: failed to enable fault, interrupt, ret %d\n", (unsigned long long)drhd->reg_base_addr, ret); return -1; } /* * Clear any previous faults. */ dmar_fault(iommu->irq, iommu); fault_status = readl(iommu->reg + DMAR_FSTS_REG); writel(fault_status, iommu->reg + DMAR_FSTS_REG); } ...Regards,
-Tina
Hi Dimitri,
Quoted 9 lines · Dimitri Sivanich
The Intel IOMMU code currently tries to allocate all DMAR fault interrupt vectors on the boot cpu. On large systems with high DMAR counts this results in vector exhaustion, and most of the vectors are not initially allocated socket local.
Instead, have a cpu on each node do the vector allocation for the DMARs on that node. The boot cpu still does the allocation for its node during its boot sequence.
Quoted 7 lines
Signed-off-by Dimitri Sivanich <[email protected]>---
v2: per Thomas Gleixner, implement this from a DYN CPU hotplug state, though this implementation runs in
CPUHP_AP_ONLINE_DYNspace rather than
CPUHP_BP_PREPARE_DYNspace.
I tested on a dual socket system (192 core) successfully with the following:
1. After boot DMAR-MSI spread from BSP to the first CPU of the second numa node
2. Offline/Online all CPUs in the 2nd node
Code looks good to me.
Thanks,
Jacob
Quoted 96 lines
drivers/iommu/amd/amd_iommu.h | 2 +- drivers/iommu/amd/init.c | 2 +- drivers/iommu/intel/dmar.c | 9 +++++++-- drivers/iommu/irq_remapping.c | 5 ++++- drivers/iommu/irq_remapping.h | 2 +- include/linux/dmar.h | 2 +- 6 files changed, 15 insertions(+), 7 deletions(-)drivers/iommu/amd/amd_iommu.h +1
-1@@ -33,7 +33,7 @@ int amd_iommu_prepare(void);int amd_iommu_enable (void );void amd_iommu_disable (void );int amd_iommu_reenable (int mode); -int amd_iommu_enable_faulting (void ); +int amd_iommu_enable_faulting (unsigned int cpu);extern int amd_iommu_guest_ir;extern enum io_pgtable_fmt amd_iommu_pgtable;extern int amd_iommu_gpt_level;drivers/iommu/amd/init.c +1
-1@@ -3389,7 +3389,7 @@ int amd_iommu_reenable(int mode)return 0 ; } -int __init amd_iommu_enable_faulting (void ) +int __init amd_iommu_enable_faulting (unsigned int cpu) {/* We enable MSI later when PCI is initialized */ return 0 ;drivers/iommu/intel/dmar.c +7
-2@@ -2122,7 +2122,7 @@ int dmar_set_interrupt(struct intel_iommu *iommu)return ret; } -int __init enable_drhd_fault_handling (void ) +int enable_drhd_fault_handling (unsigned int cpu) {struct dmar_drhd_unit *drhd;struct intel_iommu *iommu;@@ -2132,7 +2132,12 @@ int __init enable_drhd_fault_handling(void)*/ for_each_iommu (iommu, drhd) {u32 fault_status; -int ret =dmar_set_interrupt (iommu); +int ret; + +if (iommu->irq || iommu->node !=cpu_to_node (cpu)) +continue ; + + ret =dmar_set_interrupt (iommu);if (ret) { pr_err("DRHD %Lx: failed to enable fault,interrupt, ret %d\n", diff --git
a/drivers/iommu/irq_remapping.cb/drivers/iommu/irq_remapping.cindexee59647c2050..2f7281ccc05f100644drivers/iommu/irq_remapping.c +4
-1@@ -151,7 +151,10 @@ int __init irq_remap_enable_fault_handling(void)if (!remap_ops->enable_faulting)return -ENODEV ; -return remap_ops->enable_faulting (); +cpuhp_setup_state (CPUHP_AP_ONLINE_DYN ,"dmar:enable_fault_handling" , + remap_ops->enable_faulting,NULL ); + +return remap_ops->enable_faulting (smp_processor_id ()); }void
panic_if_irq_remap(const char *msg)drivers/iommu/irq_remapping.h +1
-1@@ -41,7 +41,7 @@ struct irq_remap_ops {int (*reenable)(int );/* Enable fault handling */ -int (*enable_faulting)(void ); +int (*enable_faulting)(unsigned int ); };extern struct irq_remap_ops intel_irq_remap_ops;include/linux/dmar.h +0
-0
include/linux/dmar.h extern int dmar_remove_dev_scope(struct · line 123
Quoted 8 lines
include/linux/dmar.h +1
-1@@ -117,7 +117,7 @@ extern int dmar_remove_dev_scope(struct dmar_pci_notify_info *info, int count);/* Intel IOMMU detection */ void detect_intel_iommu (void ); -extern int enable_drhd_fault_handling (void ); +extern int enable_drhd_fault_handling (unsigned int cpu);extern int dmar_device_add (acpi_handle handle);extern int dmar_device_remove (acpi_handle handle);
Thanks,
Jacob
Quoted 8 lines
From: Dimitri Sivanich <[email protected]>
Sent: Friday, March 22, 2024 4:51 AMThe Intel IOMMU code currently tries to allocate all DMAR fault interrupt vectors on the boot cpu. On large systems with high DMAR counts this results in vector exhaustion, and most of the vectors are not initially allocated socket local.
Quoted 5 lines
Instead, have a cpu on each node do the vector allocation for the DMARs on that node. The boot cpu still does the allocation for its node during its boot sequence.
Signed-off-by Dimitri Sivanich <[email protected]>
Baolu Lu
maintainerQuoted 8 lines · Kevin
Quoted 8 lines
From: Dimitri Sivanich <[email protected]>
Sent: Friday, March 22, 2024 4:51 AMThe Intel IOMMU code currently tries to allocate all DMAR fault interrupt vectors on the boot cpu. On large systems with high DMAR counts this results in vector exhaustion, and most of the vectors are not initially allocated socket local.
Quoted 8 lines
Quoted 5 lines
Instead, have a cpu on each node do the vector allocation for the DMARs on that node. The boot cpu still does the allocation for its node during its boot sequence.
Signed-off-by Dimitri Sivanich <[email protected]>Reviewed-by Kevin Tian <[email protected]>
Kevin,
Jacob has another proposal which shares the irq among all IOMMUs.
https://lore.kernel.org/linux-iommu/[email protected]/
How do you like this?
Best regards,
baolu
Quoted 22 lines
From: Baolu Lu <[email protected]>
Sent: Monday, April 8, 2024 3:22 PMQuoted 9 lines · Kevin
Quoted 9 lines
From: Dimitri Sivanich <[email protected]>
Sent: Friday, March 22, 2024 4:51 AMThe Intel IOMMU code currently tries to allocate all DMAR fault interrupt vectors on the boot cpu. On large systems with high DMAR counts this results in vector exhaustion, and most of the vectors are not initially allocated socket local.
Instead, have a cpu on each node do the vector allocation for the DMARs
on
Quoted 7 lines
Quoted 4 lines
that node. The boot cpu still does the allocation for its node during its boot sequence.
Signed-off-by Dimitri Sivanich <[email protected]>Reviewed-by Kevin Tian <[email protected]>
Quoted 9 lines
Kevin,
Jacob has another proposal which shares the irq among all IOMMUs.
https://lore.kernel.org/linux-iommu/20240403234548.989061-1-
[email protected]/How do you like this?
I'm a bit concerning about the need of looping all IOMMU's in DMAR irqchip mask/unmask handlers. this one sounds simpler to me.
Hi Kevin,
Quoted 27 lines · Kevin
Quoted 27 lines
From: Baolu Lu <[email protected]>
Sent: Monday, April 8, 2024 3:22 PMQuoted 10 lines · Kevin
Quoted 10 lines
From: Dimitri Sivanich <[email protected]>
Sent: Friday, March 22, 2024 4:51 AMThe Intel IOMMU code currently tries to allocate all DMAR fault interrupt vectors on the boot cpu. On large systems with high DMAR counts this results in vector exhaustion, and most of the vectors are not initially allocated socket local.
Instead, have a cpu on each node do the vector allocation for the DMARs
on
Quoted 7 lines
Quoted 4 lines
that node. The boot cpu still does the allocation for its node during its boot sequence.
Signed-off-by Dimitri Sivanich <[email protected]>Reviewed-by Kevin Tian <[email protected]>Kevin,
Jacob has another proposal which shares the irq among all IOMMUs.
Quoted 8 lines
Quoted 5 lines
https://lore.kernel.org/linux-iommu/20240403234548.989061-1-
[email protected]/How do you like this?
I'm a bit concerning about the need of looping all IOMMU's in DMAR irqchip mask/unmask handlers. this one sounds simpler to me.
The difference is that with this patch, we still burn a few vectors on BSP and the leading CPU of each socket.
e.g. on sapphire rapids, we lose 8 vectors to DMAR fault IRQ on BSP.
Thanks,
Jacob
Baolu Lu
maintainerQuoted 9 lines · Dimitri Sivanich
The Intel IOMMU code currently tries to allocate all DMAR fault interrupt vectors on the boot cpu. On large systems with high DMAR counts this results in vector exhaustion, and most of the vectors are not initially allocated socket local.
Instead, have a cpu on each node do the vector allocation for the DMARs on that node. The boot cpu still does the allocation for its node during its boot sequence.
Quoted 6 lines
Signed-off-by Dimitri Sivanich<[email protected]>---
v2: per Thomas Gleixner, implement this from a DYN CPU hotplug state, though
this implementation runs inCPUHP_AP_ONLINE_DYNspace rather thanCPUHP_BP_PREPARE_DYNspace.
Patch has been queued for iommu/vt-d.
Best regards,
baolu