Skip to contentSkip to conversation

PCI: vmd: IRQ domain assignment to sub devices

PATCH Applied Nirmal Patel 2 participants 4 messages May 11, 2022 – May 12, 2022

Subsystems (3)
lore
Share & tools
lore

Nirmal Patel

author maintainer
#0 lore raw

v2 cover PCI: vmd: IRQ domain assignment to sub devices

Host OS fails to boot and DMAR errors were observed when interrupt remapping is enabled by intel_iommu because of the fact that VMD child devices are on different IRQ domain than all other PCI devices. Make sure VMD assigns proper IRQ domain to the child devices during device enumeration.

Nirmal Patel (2):
PCI: vmd: Assign VMD IRQ domain before enumeration
PCI: vmd: Revert 2565e5b69c44 ("PCI: vmd: Do not disable MSI-X
remapping if interrupt remapping is enabled by IOMMU.")

 drivers/pci/controller/vmd.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
Signature · 1 line
2.26.2
author maintainer
#1 lore raw

PATCH v2 1/2 PCI: vmd: Assign VMD IRQ domain before enumeration

During the boot process all the PCI devices are assigned default PCI-MSI IRQ domain including VMD endpoint devices. If interrupt-remapping is enabled by IOMMU, the PCI devices except VMD get new INTEL-IR-MSI IRQ domain. And VMD is supposed to create and assign a separate VMD-MSI IRQ domain for its child devices in order to support MSI-X remapping capabilities.

Now when MSI-X remapping in VMD is disabled in order to improve performance, VMD skips VMD-MSI IRQ domain assignment process to its child devices. Thus the devices behind VMD get default PCI-MSI IRQ domain instead of INTEL-IR-MSI IRQ domain when VMD creates root bus and configures child devices.

As a result host OS fails to boot and DMAR errors were observed when interrupt remapping was enabled on Intel Icelake CPUs. For instance:

DMAR: DRHD: handling fault status reg 2
DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index 0xa00 [fault reason 0x25] Blocked a compatibility format interrupt request

To fix this issue, dev_msi_info struct in dev struct maintains correct value of IRQ domain. VMD will use this information to assign proper IRQ domain to its child devices when it doesn't create a separate IRQ domain.

---
v1->v2: Adding more information to commit log.

---
 drivers/pci/controller/vmd.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/pci/controller/vmd.c +3 -0
index eb05cceab964..5015adc04d19 100644
@@ -853,6 +853,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	vmd_attach_resources(vmd);
 	if (vmd->irq_domain)
 		dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
+	else
+		dev_set_msi_domain(&vmd->bus->dev,
+				   dev_get_msi_domain(&vmd->dev->dev));
 
 	vmd_acpi_begin();
 
Signature · 1 line
2.26.2
author maintainer
#2 lore raw

PATCH v2 2/2 PCI: vmd: Revert 2565e5b69c44 ("PCI: vmd: Do not disable MSI-X remapping if interrupt remapping is enabled by IOMMU.")

In reply to Nirmal Patel

Revert 2565e5b69c44 ("PCI: vmd: Do not disable MSI-X remapping if interrupt remapping is enabled by IOMMU.")

The commit 2565e5b69c44 was added as a workaround to keep MSI-X remapping enabled if IOMMU enables interrupt remapping. VMD would keep running in low performance mode. There is no dependency between MSI-X remapping by VMD and interrupt remapping by IOMMU.

---
v1->v2: Add more information to the commit log.

---
 drivers/pci/controller/vmd.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
drivers/pci/controller/vmd.c +1 -3
index 5015adc04d19..94a14a3d7e55 100644
@@ -6,7 +6,6 @@
 
 #include <linux/device.h>
 #include <linux/interrupt.h>
-#include <linux/iommu.h>
 #include <linux/irq.h>
 #include <linux/kernel.h>
 #include <linux/module.h>
@@ -813,8 +812,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	 * acceptable because the guest is usually CPU-limited and MSI
 	 * remapping doesn't become a performance bottleneck.
 	 */
-	if (iommu_capable(vmd->dev->dev.bus, IOMMU_CAP_INTR_REMAP) ||
-	    !(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) ||
+	if (!(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) ||
 	    offset[0] || offset[1]) {
 		ret = vmd_alloc_irqs(vmd);
 		if (ret)
Signature · 1 line
2.26.2
#3 lore raw
In reply to Nirmal Patel
Quoted 6 lines · Nirmal Patel

Host OS fails to boot and DMAR errors were observed when interrupt remapping is enabled by intel_iommu because of the fact that VMD child devices are on different IRQ domain than all other PCI devices. Make sure VMD assigns proper IRQ domain to the child devices during device enumeration.

Quoted 6 lines

Nirmal Patel (2):
PCI: vmd: Assign VMD IRQ domain before enumeration
PCI: vmd: Revert 2565e5b69c44 ("PCI: vmd: Do not disable MSI-X
remapping if interrupt remapping is enabled by IOMMU.")

[...]

Applied to pci/vmd, thanks!

[1/2] PCI: vmd: Assign VMD IRQ domain before enumeration

      https://git.kernel.org/lpieralisi/pci/c/886e67100b
[2/2] PCI: vmd: Revert 2565e5b69c44 ("PCI: vmd: Do not disable MSI-X remapping if interrupt remapping is enabled by IOMMU.")
      https://git.kernel.org/lpieralisi/pci/c/c94f732e80

Thanks,
Lorenzo