Skip to contentSkip to conversation

PCI: vmd: Assign vmd irq domain before enumeration

PATCH Patel, Nirmal 3 participants 4 messages Mar 2, 2022 – Mar 14, 2022 1 message not in the archive PCI

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 40 lines · Nirmal Patel

vmd creates and assigns separate irq domain only when MSI remapping is enabled. For example VMD-MSI. But vmd doesn't assign irq domain when MSI remapping is disabled resulting child devices getting default PCI-MSI irq domain. Now when Interrupt remapping is enabled all the pci devices are assigned INTEL-IR-MSI domain including vmd endpoints. But devices behind vmd gets PCI-MSI irq domain when vmd creates root bus and configures child devices.

As a result DMAR errors were observed when interrupt remapping was enabled on Intel Icelake CPUs.
For instance:
DMAR: DRHD: handling fault status reg 2
DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index 0xa00 [fault reason 0x25] Blocked a compatibility format interrupt request

So make sure vmd assigns proper irq domain. This patch also removes a placeholder patch 2565e5b69c44 (PCI: vmd: Do not disable MSI-X remapping if interrupt remapping is enabled by IOMMU.) MSI remapping should be enabled or disabled with and without Interrupt remap.

---
 drivers/pci/controller/vmd.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
drivers/pci/controller/vmd.c +1 -2
index cc166c683638..c8d73d75a1c0 100644
@@ -813,8 +813,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	 * acceptable because the guest is usually CPU-limited and MSI
 	 * remapping doesn't become a performance bottleneck.
 	 */
-	if (iommu_capable(vmd->dev->dev.bus, IOMMU_CAP_INTR_REMAP) ||
-	    !(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) ||
+	if (!(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) ||
 	    offset[0] || offset[1]) {
 		ret = vmd_alloc_irqs(vmd);
 		if (ret)

drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 860

Quoted 10 lines
drivers/pci/controller/vmd.c +3 -1
@@ -853,7 +852,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	vmd_attach_resources(vmd);
 	if (vmd->irq_domain)
 		dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
-
+	else
+		dev_set_msi_domain(&vmd->bus->dev, vmd->dev->dev.msi.domain);
+	
 	vmd_acpi_begin();
 
 	pci_scan_child_bus(vmd->bus);

Gentle ping!

Thanks

nirmal

#2 lore raw

Hey Nirmal,

Sorry I didn't catch this earlier.

Quoted 8 lines · Nirmal
Quoted 7 lines · Nirmal Patel

vmd creates and assigns separate irq domain only when MSI remapping is enabled. For example VMD-MSI. But vmd doesn't assign irq domain when MSI remapping is disabled resulting child devices getting default PCI-MSI irq domain. Now when Interrupt remapping is enabled all the pci devices are assigned INTEL-IR-MSI domain including vmd endpoints. But devices behind vmd gets PCI-MSI irq domain when vmd creates root bus and configures child devices.

can you capitalize VMD for consistency?

Quoted 8 lines
Quoted 8 lines

As a result DMAR errors were observed when interrupt remapping was enabled on Intel Icelake CPUs.
For instance:
DMAR: DRHD: handling fault status reg 2
DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index 0xa00 [fault reason 0x25] Blocked a compatibility format interrupt request

Quoted 5 lines
Quoted 5 lines

So make sure vmd assigns proper irq domain. This patch also removes a placeholder patch 2565e5b69c44 (PCI: vmd: Do not disable MSI-X remapping if interrupt remapping is enabled by IOMMU.) MSI remapping should be enabled or disabled with and without Interrupt remap.

So this keeps the performance path working with remapping? Great!

Quoted 18 lines
Quoted 18 lines
---
  drivers/pci/controller/vmd.c | 7 ++++---
  1 file changed, 4 insertions(+), 3 deletions(-)
drivers/pci/controller/vmd.c +1 -2
index cc166c683638..c8d73d75a1c0 100644
@@ -813,8 +813,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
  	 * acceptable because the guest is usually CPU-limited and MSI
  	 * remapping doesn't become a performance bottleneck.
  	 */
-	if (iommu_capable(vmd->dev->dev.bus, IOMMU_CAP_INTR_REMAP) ||
-	    !(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) ||
+	if (!(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) ||
  	    offset[0] || offset[1]) {
  		ret = vmd_alloc_irqs(vmd);
Quoted 8 lines

drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 856

Quoted 8 lines
drivers/pci/controller/vmd.c +2 -1
  		if (ret)
@@ -853,7 +852,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
  	vmd_attach_resources(vmd);
  	if (vmd->irq_domain)
  		dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
-
+	else
+		dev_set_msi_domain(&vmd->bus->dev, vmd->dev->dev.msi.domain);

how about dev_get_msi_domain(vmd->dev) ?

Quoted 11 lines

drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 860

Quoted 4 lines
drivers/pci/controller/vmd.c +1 -0
+	
  	vmd_acpi_begin();
 
  	pci_scan_child_bus(vmd->bus);

Gentle ping!

Thanks

nirmal

#3 lore raw
Quoted 6 lines · Jonathan Derrick

Hey Nirmal,

Sorry I didn't catch this earlier.

Quoted 1 line · Nirmal

On 2/23/2022 12:52 AM, Nirmal Patel wrote:

Quoted 8 lines
Quoted 7 lines
Quoted 7 lines

vmd creates and assigns separate irq domain only when MSI remapping is enabled. For example VMD-MSI. But vmd doesn't assign irq domain when MSI remapping is disabled resulting child devices getting default PCI-MSI irq domain. Now when Interrupt remapping is enabled all the pci devices are assigned INTEL-IR-MSI domain including vmd endpoints. But devices behind vmd gets PCI-MSI irq domain when vmd creates root bus and configures child devices.

can you capitalize VMD for consistency?

And IRQ, while you're at it. Both occur several times above (and below). And PCI.

Also, s/Interrupt remapping/interrupt remapping/ above, since there's no reason to capitalize "Interrupt" there.

s/behind vmd gets/behind VMD get/

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

As a result DMAR errors were observed when interrupt remapping was enabled on Intel Icelake CPUs.
For instance:
DMAR: DRHD: handling fault status reg 2
DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index 0xa00 [fault reason 0x25] Blocked a compatibility format interrupt request

Quote material as in 2565e5b69c44, e.g.,

As a result DMAR errors were observed ... For instance:

DMAR: DRHD: handling fault status reg 2
DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index 0xa00 [fault reason 0x25] Blocked a compatibility format interrupt request

(Blank line before and after quoted material, indent quote two spaces, don't break messages over lines, so they're easy to grep for.)

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

So make sure vmd assigns proper irq domain. This patch also removes a placeholder patch 2565e5b69c44 (PCI: vmd: Do not disable MSI-X remapping if interrupt remapping is enabled by IOMMU.)

Usual commit citation style is

2565e5b69c44 ("PCI: vmd: Do not disable MSI-X remapping if interrupt remapping is enabled by IOMMU")

Why is this revert part of this patch? Could it be a separate patch? Either way, we should explain why we can now revert it.

Quoted 45 lines
Quoted 3 lines
Quoted 3 lines

MSI remapping
should be enabled or disabled with and without Interrupt remap.

So this keeps the performance path working with remapping? Great!

Quoted 18 lines
Quoted 18 lines
---
  drivers/pci/controller/vmd.c | 7 ++++---
  1 file changed, 4 insertions(+), 3 deletions(-)
drivers/pci/controller/vmd.c +1 -2
index cc166c683638..c8d73d75a1c0 100644
@@ -813,8 +813,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
  	 * acceptable because the guest is usually CPU-limited and MSI
  	 * remapping doesn't become a performance bottleneck.
  	 */
-	if (iommu_capable(vmd->dev->dev.bus, IOMMU_CAP_INTR_REMAP) ||
-	    !(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) ||
+	if (!(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) ||
  	    offset[0] || offset[1]) {
  		ret = vmd_alloc_irqs(vmd);
Quoted 8 lines

drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 856

Quoted 8 lines
drivers/pci/controller/vmd.c +2 -1
  		if (ret)
@@ -853,7 +852,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
  	vmd_attach_resources(vmd);
  	if (vmd->irq_domain)
  		dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
-
+	else
+		dev_set_msi_domain(&vmd->bus->dev, vmd->dev->dev.msi.domain);

how about dev_get_msi_domain(vmd->dev) ?

Quoted 10 lines

drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 859

Quoted 3 lines
drivers/pci/controller/vmd.c +1 -0
+	
  	vmd_acpi_begin();
  	pci_scan_child_bus(vmd->bus);

Gentle ping!

Thanks

nirmal

#4 lore raw
Quoted 14 lines · Bjorn Helgaas
Quoted 13 lines · Jonathan Derrick

Hey Nirmal,

Sorry I didn't catch this earlier.

Quoted 8 lines · Nirmal
Quoted 7 lines · Nirmal Patel

vmd creates and assigns separate irq domain only when MSI remapping is enabled. For example VMD-MSI. But vmd doesn't assign irq domain when MSI remapping is disabled resulting child devices getting default PCI-MSI irq domain. Now when Interrupt remapping is enabled all the pci devices are assigned INTEL-IR-MSI domain including vmd endpoints. But devices behind vmd gets PCI-MSI irq domain when vmd creates root bus and configures child devices.

Quoted 8 lines
Quoted 1 line

can you capitalize VMD for consistency?

And IRQ, while you're at it. Both occur several times above (and below). And PCI.

Also, s/Interrupt remapping/interrupt remapping/ above, since there's no reason to capitalize "Interrupt" there.

s/behind vmd gets/behind VMD get/

Understood.

Quoted 20 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

As a result DMAR errors were observed when interrupt remapping was enabled on Intel Icelake CPUs.
For instance:
DMAR: DRHD: handling fault status reg 2
DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index 0xa00 [fault reason 0x25] Blocked a compatibility format interrupt request

Quote material as in 2565e5b69c44, e.g.,

As a result DMAR errors were observed ... For instance:

DMAR: DRHD: handling fault status reg 2
DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index 0xa00 [fault reason 0x25] Blocked a compatibility format interrupt request

(Blank line before and after quoted material, indent quote two spaces, don't break messages over lines, so they're easy to grep for.)

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

So make sure vmd assigns proper irq domain. This patch also removes a placeholder patch 2565e5b69c44 (PCI: vmd: Do not disable MSI-X remapping if interrupt remapping is enabled by IOMMU.)

Quoted 7 lines

Usual commit citation style is

2565e5b69c44 ("PCI: vmd: Do not disable MSI-X remapping if interrupt remapping is enabled by IOMMU")

Why is this revert part of this patch? Could it be a separate patch? Either way, we should explain why we can now revert it.

I will add some more details.

Quoted 27 lines
Quoted 27 lines
Quoted 4 lines
Quoted 4 lines

MSI remapping
should be enabled or disabled with and without Interrupt remap.

So this keeps the performance path working with remapping? Great!

Quoted 19 lines
Quoted 19 lines
---
  drivers/pci/controller/vmd.c | 7 ++++---
  1 file changed, 4 insertions(+), 3 deletions(-)
drivers/pci/controller/vmd.c +1 -2
index cc166c683638..c8d73d75a1c0 100644
@@ -813,8 +813,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
  	 * acceptable because the guest is usually CPU-limited and MSI
  	 * remapping doesn't become a performance bottleneck.
  	 */
-	if (iommu_capable(vmd->dev->dev.bus, IOMMU_CAP_INTR_REMAP) ||
-	    !(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) ||
+	if (!(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) ||
  	    offset[0] || offset[1]) {
  		ret = vmd_alloc_irqs(vmd);
  		if (ret)
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 856

Quoted 7 lines
drivers/pci/controller/vmd.c +2 -1
@@ -853,7 +852,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
  	vmd_attach_resources(vmd);
  	if (vmd->irq_domain)
  		dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
-
+	else
+		dev_set_msi_domain(&vmd->bus->dev, vmd->dev->dev.msi.domain);

how about dev_get_msi_domain(vmd->dev) ?

Good idea.

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 859

Quoted 4 lines
drivers/pci/controller/vmd.c +1 -0
+	
  	vmd_acpi_begin();
  	pci_scan_child_bus(vmd->bus);

Gentle ping!

Thanks

nirmal