PCI: vmd: Assign VMD IRQ domain before enumeration
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 27 lines · Nirmal Patel
From: Nirmal Patel <[email protected]>
VMD creates and assigns a separate IRQ domain only when MSI remapping is enabled. For example VMD-MSI. But VMD doesn't assign IRQ domain when MSI remapping is disabled resulting child devices getting default PCI-MSI IRQ domain. Now when interrupt remapping is enabled by intel-iommu all the PCI devices are assigned INTEL-IR-MSI domain including VMD endpoints. But devices behind VMD get PCI-MSI IRQ domain when VMD create a root bus and configures child devices.
As a result DMAR errors were observed when interrupt remapping was enabled on Intel Icelake CPUs. For instance:
DMAR: DRHD: handling fault status reg 2
DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index0xa00[fault reason0x25] Blocked a compatibility format interrupt requestAcked-by Dan Williams <[email protected]>Signed-off-by Nirmal Patel <[email protected]>--- drivers/pci/controller/vmd.c | 2 ++ 1 file changed, 2 insertions(+)drivers/pci/controller/vmd.c +0
-0
drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 860
Quoted 8 lines
drivers/pci/controller/vmd.c +2
-0@@ -853,6 +853,8 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)vmd_attach_resources (vmd);if (vmd->irq_domain)dev_set_msi_domain (&vmd->bus->dev, vmd->irq_domain); +else +dev_set_msi_domain (&vmd->bus->dev,dev_get_msi_domain (&vmd->dev->dev));vmd_acpi_begin ();
Gentle ping!
Thanks
nirmal
Quoted 35 lines · Nirmal
Quoted 33 lines · Nirmal Patel
From: Nirmal Patel <[email protected]>
VMD creates and assigns a separate IRQ domain only when MSI remapping is enabled. For example VMD-MSI. But VMD doesn't assign IRQ domain when MSI remapping is disabled resulting child devices getting default PCI-MSI IRQ domain. Now when interrupt remapping is enabled by intel-iommu all the PCI devices are assigned INTEL-IR-MSI domain including VMD endpoints. But devices behind VMD get PCI-MSI IRQ domain when VMD create a root bus and configures child devices.
As a result DMAR errors were observed when interrupt remapping was enabled on Intel Icelake CPUs. For instance:
DMAR: DRHD: handling fault status reg 2
DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index0xa00[fault reason0x25] Blocked a compatibility format interrupt requestAcked-by Dan Williams <[email protected]>Signed-off-by Nirmal Patel <[email protected]>--- drivers/pci/controller/vmd.c | 2 ++ 1 file changed, 2 insertions(+)drivers/pci/controller/vmd.c +2
-0@@ -853,6 +853,8 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)vmd_attach_resources (vmd);if (vmd->irq_domain)dev_set_msi_domain (&vmd->bus->dev, vmd->irq_domain); +else +dev_set_msi_domain (&vmd->bus->dev,dev_get_msi_domain (&vmd->dev->dev));
Quoted 4 lines
drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 860
Quoted 2 lines
drivers/pci/controller/vmd.c +0
-0vmd_acpi_begin ();Gentle ping!
It helps to be explicit when you send a patch and a follow-up ping. Are you asking Lorenzo to take this? Is this urgent such that Bjorn should consider taking it directly? The changelog notes what happens, but not the severity of end user visible impact. The merge window is presently open so the natural inclination is to just wait until that closes to circle back to outstanding patches.
Quoted 39 lines · Dan Williams
Quoted 34 lines · Nirmal
Quoted 33 lines · Nirmal Patel
From: Nirmal Patel <[email protected]>
VMD creates and assigns a separate IRQ domain only when MSI remapping is enabled. For example VMD-MSI. But VMD doesn't assign IRQ domain when MSI remapping is disabled resulting child devices getting default PCI-MSI IRQ domain. Now when interrupt remapping is enabled by intel-iommu all the PCI devices are assigned INTEL-IR-MSI domain including VMD endpoints. But devices behind VMD get PCI-MSI IRQ domain when VMD create a root bus and configures child devices.
As a result DMAR errors were observed when interrupt remapping was enabled on Intel Icelake CPUs. For instance:
DMAR: DRHD: handling fault status reg 2
DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index0xa00[fault reason0x25] Blocked a compatibility format interrupt requestAcked-by Dan Williams <[email protected]>Signed-off-by Nirmal Patel <[email protected]>--- drivers/pci/controller/vmd.c | 2 ++ 1 file changed, 2 insertions(+)drivers/pci/controller/vmd.c +2
-0@@ -853,6 +853,8 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)vmd_attach_resources (vmd);if (vmd->irq_domain)dev_set_msi_domain (&vmd->bus->dev, vmd->irq_domain); +else +dev_set_msi_domain (&vmd->bus->dev,dev_get_msi_domain (&vmd->dev->dev));Quoted 3 lines
drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 860
Quoted 2 lines
drivers/pci/controller/vmd.c +0
-0vmd_acpi_begin ();Gentle ping!
Quoted 6 lines
It helps to be explicit when you send a patch and a follow-up ping. Are you asking Lorenzo to take this? Is this urgent such that Bjorn should consider taking it directly? The changelog notes what happens, but not the severity of end user visible impact. The merge window is presently open so the natural inclination is to just wait until that closes to circle back to outstanding patches.
This patch removes a flag that bypasses MSI disable feature of VMD and improves the performance. So it would be nice if the patch gets accepted sooner. I tend to send follow-up ping after a week or so if I do not get any feedback and to allow it to get accepted in time.
Thanks.
Quoted 47 lines · Nirmal
Quoted 39 lines · Dan Williams
Quoted 34 lines · Nirmal
Quoted 33 lines · Nirmal Patel
From: Nirmal Patel <[email protected]>
VMD creates and assigns a separate IRQ domain only when MSI remapping is enabled. For example VMD-MSI. But VMD doesn't assign IRQ domain when MSI remapping is disabled resulting child devices getting default PCI-MSI IRQ domain. Now when interrupt remapping is enabled by intel-iommu all the PCI devices are assigned INTEL-IR-MSI domain including VMD endpoints. But devices behind VMD get PCI-MSI IRQ domain when VMD create a root bus and configures child devices.
As a result DMAR errors were observed when interrupt remapping was enabled on Intel Icelake CPUs. For instance:
DMAR: DRHD: handling fault status reg 2
DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index0xa00[fault reason0x25] Blocked a compatibility format interrupt requestAcked-by Dan Williams <[email protected]>Signed-off-by Nirmal Patel <[email protected]>--- drivers/pci/controller/vmd.c | 2 ++ 1 file changed, 2 insertions(+)drivers/pci/controller/vmd.c +2
-0@@ -853,6 +853,8 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)vmd_attach_resources (vmd);if (vmd->irq_domain)dev_set_msi_domain (&vmd->bus->dev, vmd->irq_domain); +else +dev_set_msi_domain (&vmd->bus->dev,dev_get_msi_domain (&vmd->dev->dev));Quoted 3 lines
drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 860
Quoted 2 lines
drivers/pci/controller/vmd.c +0
-0vmd_acpi_begin ();Gentle ping!
Quoted 6 lines
It helps to be explicit when you send a patch and a follow-up ping. Are you asking Lorenzo to take this? Is this urgent such that Bjorn should consider taking it directly? The changelog notes what happens, but not the severity of end user visible impact. The merge window is presently open so the natural inclination is to just wait until that closes to circle back to outstanding patches.
Quoted 4 lines
This patch removes a flag that bypasses MSI disable feature of VMD and improves the performance. So it would be nice if the patch gets accepted sooner. I tend to send follow-up ping after a week or so if I do not get any feedback and to allow it to get accepted in time.
There are only a few days left in the v5.18 merge window, so unless it's an emergency, this would be v5.19 material.
This claims to be a v2, but I missed the v1, and the lore archives [1] seem incomplete. Maybe the v1 (and maybe the cover letter?) were HTML or got lost for some other reason?
Bjorn
Quoted 54 lines · Bjorn Helgaas
Quoted 46 lines · Nirmal
Quoted 39 lines · Dan Williams
Quoted 34 lines · Nirmal
Quoted 33 lines · Nirmal Patel
From: Nirmal Patel <[email protected]>
VMD creates and assigns a separate IRQ domain only when MSI remapping is enabled. For example VMD-MSI. But VMD doesn't assign IRQ domain when MSI remapping is disabled resulting child devices getting default PCI-MSI IRQ domain. Now when interrupt remapping is enabled by intel-iommu all the PCI devices are assigned INTEL-IR-MSI domain including VMD endpoints. But devices behind VMD get PCI-MSI IRQ domain when VMD create a root bus and configures child devices.
As a result DMAR errors were observed when interrupt remapping was enabled on Intel Icelake CPUs. For instance:
DMAR: DRHD: handling fault status reg 2
DMAR: [INTR-REMAP] Request device [0xe2:0x00.0] fault index0xa00[fault reason0x25] Blocked a compatibility format interrupt requestAcked-by Dan Williams <[email protected]>Signed-off-by Nirmal Patel <[email protected]>--- drivers/pci/controller/vmd.c | 2 ++ 1 file changed, 2 insertions(+)drivers/pci/controller/vmd.c +2
-0@@ -853,6 +853,8 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)vmd_attach_resources (vmd);if (vmd->irq_domain)dev_set_msi_domain (&vmd->bus->dev, vmd->irq_domain); +else +dev_set_msi_domain (&vmd->bus->dev,dev_get_msi_domain (&vmd->dev->dev));Quoted 3 lines
drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 860
Quoted 2 lines
drivers/pci/controller/vmd.c +0
-0vmd_acpi_begin ();Gentle ping!
Quoted 6 lines
It helps to be explicit when you send a patch and a follow-up ping. Are you asking Lorenzo to take this? Is this urgent such that Bjorn should consider taking it directly? The changelog notes what happens, but not the severity of end user visible impact. The merge window is presently open so the natural inclination is to just wait until that closes to circle back to outstanding patches.
Quoted 4 lines
This patch removes a flag that bypasses MSI disable feature of VMD and improves the performance. So it would be nice if the patch gets accepted sooner. I tend to send follow-up ping after a week or so if I do not get any feedback and to allow it to get accepted in time.
There are only a few days left in the v5.18 merge window, so unless it's an emergency, this would be v5.19 material.
Quoted 7 lines
This claims to be a v2, but I missed the v1, and the lore archives [1] seem incomplete. Maybe the v1 (and maybe the cover letter?) were HTML or got lost for some other reason?
Bjorn
Initially I created one patch [1] and I was advised to create two separate patches.
Quoted 7 lines · Nirmal
On 3/30/2022 10:54 AM, Bjorn Helgaas wrote:
...Quoted 4 lines
This claims to be a v2, but I missed the v1, and the lore archives [1] seem incomplete. Maybe the v1 (and maybe the cover letter?) were HTML or got lost for some other reason?
Quoted 7 lines
Quoted 3 lines
Bjorn
Initially I created one patch [1] and I was advised to create two separate patches.
The point is that
https://lore.kernel.org/all/[email protected]/
is a reply to something that didn't make it into the archives.
It says:
In-Reply-To: <[email protected]>
but [email protected] is unknown to lore.
You can't fix this now; I'm just pointing it out in case there's some problem with the way you're sending patches.
Bjorn
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH v2 2/2] Allow VMD to disable MSIX remapping with interrupt remapping enabled.
Quoted 26 lines · Nirmal Patel
This patch removes a placeholder patch
2565e5b69c44("PCI: vmd: Do not disable MSI-X remapping if interrupt remapping is enabled by IOMMU.") This patch was added as a workaround to disable MSI remapping if iommu enables interrupt remapping. VMD does not assign proper IRQ domain to child devices when MSIX is disabled. There is no dependency between MSI remapping by VMD and interrupt remapping by iommu. MSI remapping can be enabled or disabled with and without interrupt remap.Signed-off-by Nirmal Patel <[email protected]>--- drivers/pci/controller/vmd.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)drivers/pci/controller/vmd.c +0
-1@@ -6,7 +6,6 @@#include <linux/device.h> #include <linux/interrupt.h> -#include <linux/iommu.h> #include <linux/irq.h> #include <linux/kernel.h> #include <linux/module.h>
drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 818
Quoted 9 lines
drivers/pci/controller/vmd.c +1
-2@@ -813,8 +812,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)* acceptable because the guest is usually CPU-limited and MSI * remapping doesn't become a performance bottleneck. */ -if (iommu_capable (vmd->dev->dev.bus,IOMMU_CAP_INTR_REMAP ) || - !(features &VMD_FEAT_CAN_BYPASS_MSI_REMAP ) || +if (!(features &VMD_FEAT_CAN_BYPASS_MSI_REMAP ) || offset[0 ] || offset[1 ]) { ret =vmd_alloc_irqs (vmd);if (ret)
Gentle ping!
Thanks
nirmal
Re: [PATCH v2 2/2] Allow VMD to disable MSIX remapping with interrupt remapping enabled.
Quoted 20 lines · Nirmal
Quoted 19 lines · Nirmal Patel
This patch removes a placeholder patch
2565e5b69c44("PCI: vmd: Do not disable MSI-X remapping if interrupt remapping is enabled by IOMMU.") This patch was added as a workaround to disable MSI remapping if iommu enables interrupt remapping. VMD does not assign proper IRQ domain to child devices when MSIX is disabled. There is no dependency between MSI remapping by VMD and interrupt remapping by iommu. MSI remapping can be enabled or disabled with and without interrupt remap.Signed-off-by Nirmal Patel <[email protected]>--- drivers/pci/controller/vmd.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)drivers/pci/controller/vmd.c +0
-0@@ -6,7 +6,6 @@
Quoted 16 lines
Quoted 7 lines
drivers/pci/controller/vmd.c +0
-1#include <linux/device.h> #include <linux/interrupt.h> -#include <linux/iommu.h> #include <linux/irq.h> #include <linux/kernel.h> #include <linux/module.h> drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 818
Quoted 9 lines
drivers/pci/controller/vmd.c +1
-2@@ -813,8 +812,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)* acceptable because the guest is usually CPU-limited and MSI * remapping doesn't become a performance bottleneck. */ -if (iommu_capable (vmd->dev->dev.bus,IOMMU_CAP_INTR_REMAP ) || - !(features &VMD_FEAT_CAN_BYPASS_MSI_REMAP ) || +if (!(features &VMD_FEAT_CAN_BYPASS_MSI_REMAP ) || offset[0 ] || offset[1 ]) { ret =vmd_alloc_irqs (vmd);if (ret)
If/when you repost this, please update the subject and commit log to use "MSI-X" consistently instead of the current mix of "MSI-X" and "MSIX".
Also s/iommu/IOMMU/.
In subject line, add "PCI: vmd: " prefix and drop trailing period.
Also rewrite commit log in imperative mood, e.g., "Revert 2565e5b69c44 ..." instead of "This patch removes ..." It's 100% clear that the commit log refers to *this* patch, so it's pointless to include that.
It's further confusing that "This patch was added ..." refers to *2565e5b69c44*, not this revert.
This reverts 2565e5b69c44 (but doesn't remove the #include <linux/iommu.h>" added by 2565e5b69c44).
2565e5b69c44 fixed a problem. If that fix is no longer necessary because of some other change, the commit log should mention that change. Otherwise somebody will backport this fix too far and reintroduce the problem solved by 2565e5b69c44.
Bjorn
Re: [PATCH v2 2/2] Allow VMD to disable MSIX remapping with interrupt remapping enabled.
Quoted 55 lines · Bjorn Helgaas
Quoted 20 lines · Nirmal
Quoted 19 lines · Nirmal Patel
This patch removes a placeholder patch
2565e5b69c44("PCI: vmd: Do not disable MSI-X remapping if interrupt remapping is enabled by IOMMU.") This patch was added as a workaround to disable MSI remapping if iommu enables interrupt remapping. VMD does not assign proper IRQ domain to child devices when MSIX is disabled. There is no dependency between MSI remapping by VMD and interrupt remapping by iommu. MSI remapping can be enabled or disabled with and without interrupt remap.Signed-off-by Nirmal Patel <[email protected]>--- drivers/pci/controller/vmd.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)drivers/pci/controller/vmd.c +0
-0@@ -6,7 +6,6 @@Quoted 16 lines
Quoted 7 lines
drivers/pci/controller/vmd.c +0
-1#include <linux/device.h> #include <linux/interrupt.h> -#include <linux/iommu.h> #include <linux/irq.h> #include <linux/kernel.h> #include <linux/module.h> drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 818
Quoted 9 lines
drivers/pci/controller/vmd.c +1
-2@@ -813,8 +812,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)* acceptable because the guest is usually CPU-limited and MSI * remapping doesn't become a performance bottleneck. */ -if (iommu_capable (vmd->dev->dev.bus,IOMMU_CAP_INTR_REMAP ) || - !(features &VMD_FEAT_CAN_BYPASS_MSI_REMAP ) || +if (!(features &VMD_FEAT_CAN_BYPASS_MSI_REMAP ) || offset[0 ] || offset[1 ]) { ret =vmd_alloc_irqs (vmd);if (ret)If/when you repost this, please update the subject and commit log to use "MSI-X" consistently instead of the current mix of "MSI-X" and "MSIX".
Also s/iommu/IOMMU/.
In subject line, add "PCI: vmd: " prefix and drop trailing period.
Also rewrite commit log in imperative mood, e.g., "Revert
2565e5b69c44..." instead of "This patch removes ..." It's 100% clear that the commit log refers to *this* patch, so it's pointless to include that.It's further confusing that "This patch was added ..." refers to *2565e5b69c44*, not this revert.
This reverts
2565e5b69c44(but doesn't remove the #include <linux/iommu.h>" added by2565e5b69c44).
Quoted 6 lines
2565e5b69c44fixed a problem. If that fix is no longer necessary because of some other change, the commit log should mention that change. Otherwise somebody will backport this fix too far and reintroduce the problem solved by2565e5b69c44.Bjorn
I will apply these changes.
Thank you
Nirmal