PCI: vmd: IRQ domain assignment to sub devices
Subsystems (3)
Nirmal Patel
maintainercover PCI: vmd: IRQ domain assignment to sub devices
Make sure VMD assigns proper IRQ domain to the child devices during device enumeration. DMAR errors were observed when interrupt remapping is enabled by intel_iommu because of the fact that VMD child devices are on different IRQ domain than all other PCI devices.
Nirmal Patel (2):
PCI: vmd: Assign VMD IRQ domain before enumeration
PCI: vmd: Revert 2565e5b69c44 ("PCI: vmd: Do not disable MSI-X
remapping if interrupt remapping is enabled by IOMMU.")
drivers/pci/controller/vmd.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
Signature · 1 line
2.26.2
Nirmal Patel
maintainerPATCH 1/2 PCI: vmd: Assign VMD IRQ domain before enumeration
VMD creates and assigns a separate IRQ domain when MSI-X remapping is enabled. For example VMD-MSI. But VMD doesn't assign IRQ domain when MSI-X remapping is disabled resulting child devices getting default PCI-MSI IRQ domain. Now when interrupt remapping is enabled by intel-iommu all the PCI devices are assigned INTEL-IR-MSI domain including VMD endpoints. But devices behind VMD get PCI-MSI IRQ domain when VMD create a root bus and configures child devices.
As a result DMAR errors were observed when interrupt remapping was enabled on Intel Icelake CPUs. For instance:
DMAR: DRHD: handling fault status reg 2
DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index 0xa00 [fault reason 0x25] Blocked a compatibility format interrupt request
--- drivers/pci/controller/vmd.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/pci/controller/vmd.c +3 -0
@@ -853,6 +853,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)vmd_attach_resources (vmd);if (vmd->irq_domain)dev_set_msi_domain (&vmd->bus->dev, vmd->irq_domain); +else +dev_set_msi_domain (&vmd->bus->dev, +dev_get_msi_domain (&vmd->dev->dev));vmd_acpi_begin ();
Signature · 1 line
2.26.2
Re: [PATCH 1/2] PCI: vmd: Assign VMD IRQ domain before enumeration
[Adding Marc, to keep an eye on IRQ domain usage]
Quoted 7 lines · Nirmal Patel
VMD creates and assigns a separate IRQ domain when MSI-X remapping is enabled. For example VMD-MSI. But VMD doesn't assign IRQ domain when MSI-X remapping is disabled resulting child devices getting default PCI-MSI IRQ domain. Now when interrupt remapping is enabled by intel-iommu all the PCI devices are assigned INTEL-IR-MSI domain including VMD endpoints. But devices behind VMD get PCI-MSI IRQ domain when VMD create a root bus and configures child devices.
I would encourage you to rewrite this log, it is unclear - granted, I don't know intel-iommu internals - but IMHO if you explain the issue and the fix thoroughly this could avoid repeating what you have to do in patch(2).
Please describe how VMD handles IRQ domains and how you are fixing that.
Thanks,
Lorenzo
Quoted 28 lines
As a result DMAR errors were observed when interrupt remapping was enabled on Intel Icelake CPUs. For instance:
DMAR: DRHD: handling fault status reg 2
DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index0xa00[fault reason0x25] Blocked a compatibility format interrupt requestSigned-off-by Nirmal Patel <[email protected]>--- drivers/pci/controller/vmd.c | 3 +++ 1 file changed, 3 insertions(+)drivers/pci/controller/vmd.c +3
-0@@ -853,6 +853,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)vmd_attach_resources (vmd);if (vmd->irq_domain)dev_set_msi_domain (&vmd->bus->dev, vmd->irq_domain); +else +dev_set_msi_domain (&vmd->bus->dev, +dev_get_msi_domain (&vmd->dev->dev));vmd_acpi_begin ();Signature · 1 line
2.26.2
Patel, Nirmal
maintainerRe: [PATCH 1/2] PCI: vmd: Assign VMD IRQ domain before enumeration
Quoted 15 lines · Lorenzo Pieralisi
[Adding Marc, to keep an eye on IRQ domain usage]
Quoted 7 lines · Nirmal Patel
VMD creates and assigns a separate IRQ domain when MSI-X remapping is enabled. For example VMD-MSI. But VMD doesn't assign IRQ domain when MSI-X remapping is disabled resulting child devices getting default PCI-MSI IRQ domain. Now when interrupt remapping is enabled by intel-iommu all the PCI devices are assigned INTEL-IR-MSI domain including VMD endpoints. But devices behind VMD get PCI-MSI IRQ domain when VMD create a root bus and configures child devices.
I would encourage you to rewrite this log, it is unclear - granted, I don't know intel-iommu internals - but IMHO if you explain the issue and the fix thoroughly this could avoid repeating what you have to do in patch(2).
Quoted 5 lines
Please describe how VMD handles IRQ domains and how you are fixing that.
Thanks,
Lorenzo
I will add more information to the commit logs.
Thanks,
nirmal
Quoted 28 lines
Quoted 28 lines
As a result DMAR errors were observed when interrupt remapping was enabled on Intel Icelake CPUs. For instance:
DMAR: DRHD: handling fault status reg 2
DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index0xa00[fault reason0x25] Blocked a compatibility format interrupt requestSigned-off-by Nirmal Patel <[email protected]>--- drivers/pci/controller/vmd.c | 3 +++ 1 file changed, 3 insertions(+)drivers/pci/controller/vmd.c +3
-0@@ -853,6 +853,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)vmd_attach_resources (vmd);if (vmd->irq_domain)dev_set_msi_domain (&vmd->bus->dev, vmd->irq_domain); +else +dev_set_msi_domain (&vmd->bus->dev, +dev_get_msi_domain (&vmd->dev->dev));vmd_acpi_begin ();Signature · 1 line
2.26.2
Nirmal Patel
maintainerPATCH 2/2 PCI: vmd: Revert 2565e5b69c44 ("PCI: vmd: Do not disable MSI-X remapping if interrupt remapping is enabled by IOMMU.")
In reply to Nirmal PatelRevert 2565e5b69c44 ("PCI: vmd: Do not disable MSI-X remapping if interrupt remapping is enabled by IOMMU.")
The commit 2565e5b69c44 was added as a workaround to enable MSI-X remapping if IOMMU enables interrupt remapping. VMD does not assign proper IRQ domain to child devices when MSI-X remapping is disabled. There is no dependency between MSI-X remapping by VMD and interrupt remapping by IOMMU.
--- drivers/pci/controller/vmd.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
drivers/pci/controller/vmd.c +1 -3
@@ -6,7 +6,6 @@#include <linux/device.h> #include <linux/interrupt.h> -#include <linux/iommu.h> #include <linux/irq.h> #include <linux/kernel.h> #include <linux/module.h>
@@ -813,8 +812,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)* acceptable because the guest is usually CPU-limited and MSI * remapping doesn't become a performance bottleneck. */ -if (iommu_capable (vmd->dev->dev.bus,IOMMU_CAP_INTR_REMAP ) || - !(features &VMD_FEAT_CAN_BYPASS_MSI_REMAP ) || +if (!(features &VMD_FEAT_CAN_BYPASS_MSI_REMAP ) || offset[0 ] || offset[1 ]) { ret =vmd_alloc_irqs (vmd);if (ret)
Signature · 1 line
2.26.2