Skip to contentSkip to conversation

PCI: vmd: Assign VMD IRQ domain before enumeration

PATCH Patel, Nirmal 3 participants 9 messages Mar 29, 2022 – Mar 30, 2022 2 messages not in the archive PCI

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 27 lines · Nirmal Patel

From: Nirmal Patel <[email protected]>

VMD creates and assigns a separate IRQ domain only when MSI remapping is enabled. For example VMD-MSI. But VMD doesn't assign IRQ domain when MSI remapping is disabled resulting child devices getting default PCI-MSI IRQ domain. Now when interrupt remapping is enabled by intel-iommu all the PCI devices are assigned INTEL-IR-MSI domain including VMD endpoints. But devices behind VMD get PCI-MSI IRQ domain when VMD create a root bus and configures child devices.

As a result DMAR errors were observed when interrupt remapping was enabled on Intel Icelake CPUs. For instance:

DMAR: DRHD: handling fault status reg 2
DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index 0xa00 [fault reason 0x25] Blocked a compatibility format interrupt request

---
 drivers/pci/controller/vmd.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/pci/controller/vmd.c +0 -0
index cc166c683638..3a6570e5b765 100644

drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 860

Quoted 8 lines
drivers/pci/controller/vmd.c +2 -0
@@ -853,6 +853,8 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	vmd_attach_resources(vmd);
 	if (vmd->irq_domain)
 		dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
+	else
+		dev_set_msi_domain(&vmd->bus->dev, dev_get_msi_domain(&vmd->dev->dev));
 
 	vmd_acpi_begin();
 

Gentle ping!

Thanks
nirmal

#2 lore raw
Quoted 35 lines · Nirmal
Quoted 33 lines · Nirmal Patel

From: Nirmal Patel <[email protected]>

VMD creates and assigns a separate IRQ domain only when MSI remapping is enabled. For example VMD-MSI. But VMD doesn't assign IRQ domain when MSI remapping is disabled resulting child devices getting default PCI-MSI IRQ domain. Now when interrupt remapping is enabled by intel-iommu all the PCI devices are assigned INTEL-IR-MSI domain including VMD endpoints. But devices behind VMD get PCI-MSI IRQ domain when VMD create a root bus and configures child devices.

As a result DMAR errors were observed when interrupt remapping was enabled on Intel Icelake CPUs. For instance:

DMAR: DRHD: handling fault status reg 2
DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index 0xa00 [fault reason 0x25] Blocked a compatibility format interrupt request

---
 drivers/pci/controller/vmd.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/pci/controller/vmd.c +2 -0
index cc166c683638..3a6570e5b765 100644
@@ -853,6 +853,8 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
      vmd_attach_resources(vmd);
      if (vmd->irq_domain)
              dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
+     else
+             dev_set_msi_domain(&vmd->bus->dev, dev_get_msi_domain(&vmd->dev->dev));
 
Quoted 4 lines

drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 860

Quoted 2 lines
drivers/pci/controller/vmd.c +0 -0
      vmd_acpi_begin();
 

Gentle ping!

It helps to be explicit when you send a patch and a follow-up ping. Are you asking Lorenzo to take this? Is this urgent such that Bjorn should consider taking it directly? The changelog notes what happens, but not the severity of end user visible impact. The merge window is presently open so the natural inclination is to just wait until that closes to circle back to outstanding patches.

#3 lore raw
Quoted 39 lines · Dan Williams
Quoted 34 lines · Nirmal
Quoted 33 lines · Nirmal Patel

From: Nirmal Patel <[email protected]>

VMD creates and assigns a separate IRQ domain only when MSI remapping is enabled. For example VMD-MSI. But VMD doesn't assign IRQ domain when MSI remapping is disabled resulting child devices getting default PCI-MSI IRQ domain. Now when interrupt remapping is enabled by intel-iommu all the PCI devices are assigned INTEL-IR-MSI domain including VMD endpoints. But devices behind VMD get PCI-MSI IRQ domain when VMD create a root bus and configures child devices.

As a result DMAR errors were observed when interrupt remapping was enabled on Intel Icelake CPUs. For instance:

DMAR: DRHD: handling fault status reg 2
DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index 0xa00 [fault reason 0x25] Blocked a compatibility format interrupt request

---
 drivers/pci/controller/vmd.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/pci/controller/vmd.c +2 -0
index cc166c683638..3a6570e5b765 100644
@@ -853,6 +853,8 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
      vmd_attach_resources(vmd);
      if (vmd->irq_domain)
              dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
+     else
+             dev_set_msi_domain(&vmd->bus->dev, dev_get_msi_domain(&vmd->dev->dev));
 
Quoted 3 lines

drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 860

Quoted 2 lines
drivers/pci/controller/vmd.c +0 -0
      vmd_acpi_begin();
 

Gentle ping!

Quoted 6 lines

It helps to be explicit when you send a patch and a follow-up ping. Are you asking Lorenzo to take this? Is this urgent such that Bjorn should consider taking it directly? The changelog notes what happens, but not the severity of end user visible impact. The merge window is presently open so the natural inclination is to just wait until that closes to circle back to outstanding patches.

This patch removes a flag that bypasses MSI disable feature of VMD and improves the performance. So it would be nice if the patch gets accepted sooner. I tend to send follow-up ping after a week or so if I do not get any feedback and to allow it to get accepted in time.

Thanks.

#4 lore raw
Quoted 47 lines · Nirmal
Quoted 39 lines · Dan Williams
Quoted 34 lines · Nirmal
Quoted 33 lines · Nirmal Patel

From: Nirmal Patel <[email protected]>

VMD creates and assigns a separate IRQ domain only when MSI remapping is enabled. For example VMD-MSI. But VMD doesn't assign IRQ domain when MSI remapping is disabled resulting child devices getting default PCI-MSI IRQ domain. Now when interrupt remapping is enabled by intel-iommu all the PCI devices are assigned INTEL-IR-MSI domain including VMD endpoints. But devices behind VMD get PCI-MSI IRQ domain when VMD create a root bus and configures child devices.

As a result DMAR errors were observed when interrupt remapping was enabled on Intel Icelake CPUs. For instance:

DMAR: DRHD: handling fault status reg 2
DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index 0xa00 [fault reason 0x25] Blocked a compatibility format interrupt request

---
 drivers/pci/controller/vmd.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/pci/controller/vmd.c +2 -0
index cc166c683638..3a6570e5b765 100644
@@ -853,6 +853,8 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
      vmd_attach_resources(vmd);
      if (vmd->irq_domain)
              dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
+     else
+             dev_set_msi_domain(&vmd->bus->dev, dev_get_msi_domain(&vmd->dev->dev));
 
Quoted 3 lines

drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 860

Quoted 2 lines
drivers/pci/controller/vmd.c +0 -0
      vmd_acpi_begin();
 

Gentle ping!

Quoted 6 lines

It helps to be explicit when you send a patch and a follow-up ping. Are you asking Lorenzo to take this? Is this urgent such that Bjorn should consider taking it directly? The changelog notes what happens, but not the severity of end user visible impact. The merge window is presently open so the natural inclination is to just wait until that closes to circle back to outstanding patches.

Quoted 4 lines

This patch removes a flag that bypasses MSI disable feature of VMD and improves the performance. So it would be nice if the patch gets accepted sooner. I tend to send follow-up ping after a week or so if I do not get any feedback and to allow it to get accepted in time.

There are only a few days left in the v5.18 merge window, so unless it's an emergency, this would be v5.19 material.

This claims to be a v2, but I missed the v1, and the lore archives [1] seem incomplete. Maybe the v1 (and maybe the cover letter?) were HTML or got lost for some other reason?

Bjorn

[1] https://lore.kernel.org/all/?q=f%3Anirmal.patel

#5 lore raw
Quoted 54 lines · Bjorn Helgaas
Quoted 46 lines · Nirmal
Quoted 39 lines · Dan Williams
Quoted 34 lines · Nirmal
Quoted 33 lines · Nirmal Patel

From: Nirmal Patel <[email protected]>

VMD creates and assigns a separate IRQ domain only when MSI remapping is enabled. For example VMD-MSI. But VMD doesn't assign IRQ domain when MSI remapping is disabled resulting child devices getting default PCI-MSI IRQ domain. Now when interrupt remapping is enabled by intel-iommu all the PCI devices are assigned INTEL-IR-MSI domain including VMD endpoints. But devices behind VMD get PCI-MSI IRQ domain when VMD create a root bus and configures child devices.

As a result DMAR errors were observed when interrupt remapping was enabled on Intel Icelake CPUs. For instance:

DMAR: DRHD: handling fault status reg 2
DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index 0xa00 [fault reason 0x25] Blocked a compatibility format interrupt request

---
 drivers/pci/controller/vmd.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/pci/controller/vmd.c +2 -0
index cc166c683638..3a6570e5b765 100644
@@ -853,6 +853,8 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
      vmd_attach_resources(vmd);
      if (vmd->irq_domain)
              dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
+     else
+             dev_set_msi_domain(&vmd->bus->dev, dev_get_msi_domain(&vmd->dev->dev));
 
Quoted 3 lines

drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 860

Quoted 2 lines
drivers/pci/controller/vmd.c +0 -0
      vmd_acpi_begin();
 

Gentle ping!

Quoted 6 lines

It helps to be explicit when you send a patch and a follow-up ping. Are you asking Lorenzo to take this? Is this urgent such that Bjorn should consider taking it directly? The changelog notes what happens, but not the severity of end user visible impact. The merge window is presently open so the natural inclination is to just wait until that closes to circle back to outstanding patches.

Quoted 4 lines

This patch removes a flag that bypasses MSI disable feature of VMD and improves the performance. So it would be nice if the patch gets accepted sooner. I tend to send follow-up ping after a week or so if I do not get any feedback and to allow it to get accepted in time.

There are only a few days left in the v5.18 merge window, so unless it's an emergency, this would be v5.19 material.

Quoted 7 lines

This claims to be a v2, but I missed the v1, and the lore archives [1] seem incomplete. Maybe the v1 (and maybe the cover letter?) were HTML or got lost for some other reason?

Bjorn

[1] https://lore.kernel.org/all/?q=f%3Anirmal.patel

Initially I created one patch [1] and I was advised to create two separate patches.

[1] https://lore.kernel.org/all/[email protected]/

#6 lore raw
Quoted 7 lines · Nirmal

On 3/30/2022 10:54 AM, Bjorn Helgaas wrote:
...

Quoted 4 lines

This claims to be a v2, but I missed the v1, and the lore archives [1] seem incomplete. Maybe the v1 (and maybe the cover letter?) were HTML or got lost for some other reason?

Quoted 7 lines
Quoted 3 lines

Bjorn

[1] https://lore.kernel.org/all/?q=f%3Anirmal.patel

Initially I created one patch [1] and I was advised to create two separate patches.

[1] https://lore.kernel.org/all/[email protected]/

The point is that
https://lore.kernel.org/all/[email protected]/
is a reply to something that didn't make it into the archives.

It says:

In-Reply-To: <[email protected]>

but [email protected] is unknown to lore.

You can't fix this now; I'm just pointing it out in case there's some problem with the way you're sending patches.

Bjorn

A message that is not in the archive (sent off-list or not yet received): [email protected]

#8 lore raw

Re: [PATCH v2 2/2] Allow VMD to disable MSIX remapping with interrupt remapping enabled.

Quoted 26 lines · Nirmal Patel

This patch removes a placeholder patch 2565e5b69c44 ("PCI: vmd: Do not disable MSI-X remapping if interrupt remapping is enabled by IOMMU.") This patch was added as a workaround to disable MSI remapping if iommu enables interrupt remapping. VMD does not assign proper IRQ domain to child devices when MSIX is disabled. There is no dependency between MSI remapping by VMD and interrupt remapping by iommu. MSI remapping can be enabled or disabled with and without interrupt remap.

---
 drivers/pci/controller/vmd.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
drivers/pci/controller/vmd.c +0 -1
index 3a6570e5b765..91bc1b40d40c 100644
@@ -6,7 +6,6 @@
 
 #include <linux/device.h>
 #include <linux/interrupt.h>
-#include <linux/iommu.h>
 #include <linux/irq.h>
 #include <linux/kernel.h>
 #include <linux/module.h>

drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 818

Quoted 9 lines
drivers/pci/controller/vmd.c +1 -2
@@ -813,8 +812,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	 * acceptable because the guest is usually CPU-limited and MSI
 	 * remapping doesn't become a performance bottleneck.
 	 */
-	if (iommu_capable(vmd->dev->dev.bus, IOMMU_CAP_INTR_REMAP) ||
-	    !(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) ||
+	if (!(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) ||
 	    offset[0] || offset[1]) {
 		ret = vmd_alloc_irqs(vmd);
 		if (ret)

Gentle ping!

Thanks
nirmal

#9 lore raw

Re: [PATCH v2 2/2] Allow VMD to disable MSIX remapping with interrupt remapping enabled.

Quoted 20 lines · Nirmal
Quoted 19 lines · Nirmal Patel

This patch removes a placeholder patch 2565e5b69c44 ("PCI: vmd: Do not disable MSI-X remapping if interrupt remapping is enabled by IOMMU.") This patch was added as a workaround to disable MSI remapping if iommu enables interrupt remapping. VMD does not assign proper IRQ domain to child devices when MSIX is disabled. There is no dependency between MSI remapping by VMD and interrupt remapping by iommu. MSI remapping can be enabled or disabled with and without interrupt remap.

---
 drivers/pci/controller/vmd.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
drivers/pci/controller/vmd.c +0 -0
index 3a6570e5b765..91bc1b40d40c 100644
@@ -6,7 +6,6 @@
 
Quoted 16 lines
Quoted 7 lines
drivers/pci/controller/vmd.c +0 -1
 #include <linux/device.h>
 #include <linux/interrupt.h>
-#include <linux/iommu.h>
 #include <linux/irq.h>
 #include <linux/kernel.h>
 #include <linux/module.h>

drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 818

Quoted 9 lines
drivers/pci/controller/vmd.c +1 -2
@@ -813,8 +812,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	 * acceptable because the guest is usually CPU-limited and MSI
 	 * remapping doesn't become a performance bottleneck.
 	 */
-	if (iommu_capable(vmd->dev->dev.bus, IOMMU_CAP_INTR_REMAP) ||
-	    !(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) ||
+	if (!(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) ||
 	    offset[0] || offset[1]) {
 		ret = vmd_alloc_irqs(vmd);
 		if (ret)

If/when you repost this, please update the subject and commit log to use "MSI-X" consistently instead of the current mix of "MSI-X" and "MSIX".

Also s/iommu/IOMMU/.

In subject line, add "PCI: vmd: " prefix and drop trailing period.

Also rewrite commit log in imperative mood, e.g., "Revert 2565e5b69c44 ..." instead of "This patch removes ..." It's 100% clear that the commit log refers to *this* patch, so it's pointless to include that.

It's further confusing that "This patch was added ..." refers to *2565e5b69c44*, not this revert.

This reverts 2565e5b69c44 (but doesn't remove the #include <linux/iommu.h>" added by 2565e5b69c44).

2565e5b69c44 fixed a problem. If that fix is no longer necessary because of some other change, the commit log should mention that change. Otherwise somebody will backport this fix too far and reintroduce the problem solved by 2565e5b69c44.

Bjorn

#10 lore raw

Re: [PATCH v2 2/2] Allow VMD to disable MSIX remapping with interrupt remapping enabled.

Quoted 55 lines · Bjorn Helgaas
Quoted 20 lines · Nirmal
Quoted 19 lines · Nirmal Patel

This patch removes a placeholder patch 2565e5b69c44 ("PCI: vmd: Do not disable MSI-X remapping if interrupt remapping is enabled by IOMMU.") This patch was added as a workaround to disable MSI remapping if iommu enables interrupt remapping. VMD does not assign proper IRQ domain to child devices when MSIX is disabled. There is no dependency between MSI remapping by VMD and interrupt remapping by iommu. MSI remapping can be enabled or disabled with and without interrupt remap.

---
 drivers/pci/controller/vmd.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
drivers/pci/controller/vmd.c +0 -0
index 3a6570e5b765..91bc1b40d40c 100644
@@ -6,7 +6,6 @@
 
Quoted 16 lines
Quoted 7 lines
drivers/pci/controller/vmd.c +0 -1
 #include <linux/device.h>
 #include <linux/interrupt.h>
-#include <linux/iommu.h>
 #include <linux/irq.h>
 #include <linux/kernel.h>
 #include <linux/module.h>

drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 818

Quoted 9 lines
drivers/pci/controller/vmd.c +1 -2
@@ -813,8 +812,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	 * acceptable because the guest is usually CPU-limited and MSI
 	 * remapping doesn't become a performance bottleneck.
 	 */
-	if (iommu_capable(vmd->dev->dev.bus, IOMMU_CAP_INTR_REMAP) ||
-	    !(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) ||
+	if (!(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) ||
 	    offset[0] || offset[1]) {
 		ret = vmd_alloc_irqs(vmd);
 		if (ret)

If/when you repost this, please update the subject and commit log to use "MSI-X" consistently instead of the current mix of "MSI-X" and "MSIX".

Also s/iommu/IOMMU/.

In subject line, add "PCI: vmd: " prefix and drop trailing period.

Also rewrite commit log in imperative mood, e.g., "Revert 2565e5b69c44 ..." instead of "This patch removes ..." It's 100% clear that the commit log refers to *this* patch, so it's pointless to include that.

It's further confusing that "This patch was added ..." refers to *2565e5b69c44*, not this revert.

This reverts 2565e5b69c44 (but doesn't remove the #include <linux/iommu.h>" added by 2565e5b69c44).

Quoted 6 lines

2565e5b69c44 fixed a problem. If that fix is no longer necessary because of some other change, the commit log should mention that change. Otherwise somebody will backport this fix too far and reintroduce the problem solved by 2565e5b69c44.

Bjorn

I will apply these changes.

Thank you
Nirmal