Skip to contentSkip to conversation

PCI: vmd: IRQ domain assignment to sub devices

PATCH Superseded Nirmal Patel 2 participants 5 messages May 2, 2022 – May 11, 2022

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Nirmal Patel

author maintainer
#0 lore raw

cover PCI: vmd: IRQ domain assignment to sub devices

Make sure VMD assigns proper IRQ domain to the child devices during device enumeration. DMAR errors were observed when interrupt remapping is enabled by intel_iommu because of the fact that VMD child devices are on different IRQ domain than all other PCI devices.

Nirmal Patel (2):
PCI: vmd: Assign VMD IRQ domain before enumeration
PCI: vmd: Revert 2565e5b69c44 ("PCI: vmd: Do not disable MSI-X
remapping if interrupt remapping is enabled by IOMMU.")

 drivers/pci/controller/vmd.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
Signature · 1 line
2.26.2
author maintainer
#1 lore raw

PATCH 1/2 PCI: vmd: Assign VMD IRQ domain before enumeration

VMD creates and assigns a separate IRQ domain when MSI-X remapping is enabled. For example VMD-MSI. But VMD doesn't assign IRQ domain when MSI-X remapping is disabled resulting child devices getting default PCI-MSI IRQ domain. Now when interrupt remapping is enabled by intel-iommu all the PCI devices are assigned INTEL-IR-MSI domain including VMD endpoints. But devices behind VMD get PCI-MSI IRQ domain when VMD create a root bus and configures child devices.

As a result DMAR errors were observed when interrupt remapping was enabled on Intel Icelake CPUs. For instance:

DMAR: DRHD: handling fault status reg 2
DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index 0xa00 [fault reason 0x25] Blocked a compatibility format interrupt request

---
 drivers/pci/controller/vmd.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/pci/controller/vmd.c +3 -0
index eb05cceab964..5015adc04d19 100644
@@ -853,6 +853,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	vmd_attach_resources(vmd);
 	if (vmd->irq_domain)
 		dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
+	else
+		dev_set_msi_domain(&vmd->bus->dev,
+				   dev_get_msi_domain(&vmd->dev->dev));
 
 	vmd_acpi_begin();
 
Signature · 1 line
2.26.2
#2 lore raw

Re: [PATCH 1/2] PCI: vmd: Assign VMD IRQ domain before enumeration

[Adding Marc, to keep an eye on IRQ domain usage]

Quoted 7 lines · Nirmal Patel

VMD creates and assigns a separate IRQ domain when MSI-X remapping is enabled. For example VMD-MSI. But VMD doesn't assign IRQ domain when MSI-X remapping is disabled resulting child devices getting default PCI-MSI IRQ domain. Now when interrupt remapping is enabled by intel-iommu all the PCI devices are assigned INTEL-IR-MSI domain including VMD endpoints. But devices behind VMD get PCI-MSI IRQ domain when VMD create a root bus and configures child devices.

I would encourage you to rewrite this log, it is unclear - granted, I don't know intel-iommu internals - but IMHO if you explain the issue and the fix thoroughly this could avoid repeating what you have to do in patch(2).

Please describe how VMD handles IRQ domains and how you are fixing that.

Thanks,
Lorenzo

Quoted 28 lines

As a result DMAR errors were observed when interrupt remapping was enabled on Intel Icelake CPUs. For instance:

DMAR: DRHD: handling fault status reg 2
DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index 0xa00 [fault reason 0x25] Blocked a compatibility format interrupt request

---
 drivers/pci/controller/vmd.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/pci/controller/vmd.c +3 -0
index eb05cceab964..5015adc04d19 100644
@@ -853,6 +853,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	vmd_attach_resources(vmd);
 	if (vmd->irq_domain)
 		dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
+	else
+		dev_set_msi_domain(&vmd->bus->dev,
+				   dev_get_msi_domain(&vmd->dev->dev));
 
 	vmd_acpi_begin();
 
Signature · 1 line
2.26.2
author maintainer
#3 lore raw

Re: [PATCH 1/2] PCI: vmd: Assign VMD IRQ domain before enumeration

Quoted 15 lines · Lorenzo Pieralisi

[Adding Marc, to keep an eye on IRQ domain usage]

Quoted 7 lines · Nirmal Patel

VMD creates and assigns a separate IRQ domain when MSI-X remapping is enabled. For example VMD-MSI. But VMD doesn't assign IRQ domain when MSI-X remapping is disabled resulting child devices getting default PCI-MSI IRQ domain. Now when interrupt remapping is enabled by intel-iommu all the PCI devices are assigned INTEL-IR-MSI domain including VMD endpoints. But devices behind VMD get PCI-MSI IRQ domain when VMD create a root bus and configures child devices.

I would encourage you to rewrite this log, it is unclear - granted, I don't know intel-iommu internals - but IMHO if you explain the issue and the fix thoroughly this could avoid repeating what you have to do in patch(2).

Quoted 5 lines

Please describe how VMD handles IRQ domains and how you are fixing that.

Thanks,
Lorenzo

I will add more information to the commit logs.

Thanks,
nirmal

Quoted 28 lines
Quoted 28 lines

As a result DMAR errors were observed when interrupt remapping was enabled on Intel Icelake CPUs. For instance:

DMAR: DRHD: handling fault status reg 2
DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index 0xa00 [fault reason 0x25] Blocked a compatibility format interrupt request

---
 drivers/pci/controller/vmd.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/pci/controller/vmd.c +3 -0
index eb05cceab964..5015adc04d19 100644
@@ -853,6 +853,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	vmd_attach_resources(vmd);
 	if (vmd->irq_domain)
 		dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
+	else
+		dev_set_msi_domain(&vmd->bus->dev,
+				   dev_get_msi_domain(&vmd->dev->dev));
 
 	vmd_acpi_begin();
 
Signature · 1 line
2.26.2
author maintainer
#4 lore raw

PATCH 2/2 PCI: vmd: Revert 2565e5b69c44 ("PCI: vmd: Do not disable MSI-X remapping if interrupt remapping is enabled by IOMMU.")

In reply to Nirmal Patel

Revert 2565e5b69c44 ("PCI: vmd: Do not disable MSI-X remapping if interrupt remapping is enabled by IOMMU.")

The commit 2565e5b69c44 was added as a workaround to enable MSI-X remapping if IOMMU enables interrupt remapping. VMD does not assign proper IRQ domain to child devices when MSI-X remapping is disabled. There is no dependency between MSI-X remapping by VMD and interrupt remapping by IOMMU.

---
 drivers/pci/controller/vmd.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
drivers/pci/controller/vmd.c +1 -3
index 5015adc04d19..94a14a3d7e55 100644
@@ -6,7 +6,6 @@
 
 #include <linux/device.h>
 #include <linux/interrupt.h>
-#include <linux/iommu.h>
 #include <linux/irq.h>
 #include <linux/kernel.h>
 #include <linux/module.h>
@@ -813,8 +812,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	 * acceptable because the guest is usually CPU-limited and MSI
 	 * remapping doesn't become a performance bottleneck.
 	 */
-	if (iommu_capable(vmd->dev->dev.bus, IOMMU_CAP_INTR_REMAP) ||
-	    !(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) ||
+	if (!(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) ||
 	    offset[0] || offset[1]) {
 		ret = vmd_alloc_irqs(vmd);
 		if (ret)
Signature · 1 line
2.26.2