Skip to contentSkip to conversation

PCI: vmd: Disable MSI remapping after suspend

PATCH In review [email protected] 3 participants 3 messages Nov 7, 2022 – Nov 8, 2022

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 PCI: vmd: Disable MSI remapping after suspend

From: Nirmal Patel <[email protected]>

MSI remapping is disbaled by VMD driver for intel's icelake and newer systems in order to improve performance by setting MSI_RMP_DIS bit. By design MSI_RMP_DIS bit of VMCONFIG registers is cleared. The same register gets cleared when system is put in S3 power state. VMD driver needs to set this register again in order to avoid interrupt issues with devices behind VMD if MSI remapping was disabled before.

---
 drivers/pci/controller/vmd.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/pci/controller/vmd.c +3 -0
index e06e9f4fc50f..247012b343fd 100644
@@ -980,6 +980,9 @@ static int vmd_resume(struct device *dev)
 	struct vmd_dev *vmd = pci_get_drvdata(pdev);
 	int err, i;
 
+	if (!vmd->irq_domain)
+		vmd_set_msi_remapping(vmd, false);
+
 	for (i = 0; i < vmd->msix_count; i++) {
 		err = devm_request_irq(dev, vmd->irqs[i].virq,
 				       vmd_irq, IRQF_NO_THREAD,
Signature · 1 line
2.34.1
#1 lore raw
Quoted 3 lines · On Mon, Nov 07, 2022 at 10:27:35AM -0800, [email protected] wrote:

From: Nirmal Patel <[email protected]>

MSI remapping is disbaled by VMD driver for intel's icelake and

disabled
Intel's
Ice Lake (at least per intel.com)

Quoted 2 lines

newer systems in order to improve performance by setting MSI_RMP_DIS bit. By design MSI_RMP_DIS bit of VMCONFIG registers is cleared.

register is

"MSI_RMP_DIS" doesn't appear in the kernel source. Is it the same as VMCONFIG_MSI_REMAP?

Quoted 4 lines

The same register gets cleared when system is put in S3 power state. VMD driver needs to set this register again in order to avoid interrupt issues with devices behind VMD if MSI remapping was disabled before.

Should there be a Fixes: tag here? I guess the failure symptom is that a driver doesn't see interrupts it expects?

Quoted 11 lines
---
 drivers/pci/controller/vmd.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/pci/controller/vmd.c +0 -0
index e06e9f4fc50f..247012b343fd 100644

drivers/pci/controller/vmd.c static int vmd_resume(struct device *dev) · line 984

Quoted 5 lines
drivers/pci/controller/vmd.c +2 -0
@@ -980,6 +980,9 @@ static int vmd_resume(struct device *dev)
 	struct vmd_dev *vmd = pci_get_drvdata(pdev);
 	int err, i;
 
+	if (!vmd->irq_domain)
+		vmd_set_msi_remapping(vmd, false);

I suppose this is functionally equivalent to this code in vmd_enable_domain():

        if (!(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) ||
            offset[0] || offset[1]) {
                ret = vmd_create_irq_domain(vmd);
        } else {
                vmd_set_msi_remapping(vmd, false);

because vmd->irq_domain will be NULL if we disabled MSI remapping at probe-time.

Maybe the vmd_enable_domain() code could be rearranged a little bit to make it obvious that we're doing the same thing both at probe-time and resume-time?

Should the vmd_resume() code *enable* MSI remapping in the other case, e.g.,

  if (vmd->irq_domain)
    vmd_set_msi_remapping(vmd, true);
  else
    vmd_set_msi_remapping(vmd, false);

I don't know what clears PCI_REG_VMCONFIG (which enabled MSI remapping IIUC). If it's cleared by firmware, the current patch depends on that firmware behavior, so maybe it would be good to remove that dependency?

Quoted 6 lines
drivers/pci/controller/vmd.c +0 -0
 	for (i = 0; i < vmd->msix_count; i++) {
 		err = devm_request_irq(dev, vmd->irqs[i].virq,
 				       vmd_irq, IRQF_NO_THREAD,
Signature · 1 line
2.34.1
maintainer
#2 lore raw
Quoted 7 lines · Bjorn Helgaas
Quoted 3 lines · On Mon, Nov 07, 2022 at 10:27:35AM -0800, [email protected] wrote:

From: Nirmal Patel <[email protected]>

MSI remapping is disbaled by VMD driver for intel's icelake and

disabled
Intel's
Ice Lake (at least per intel.com)

ACK

Quoted 7 lines
Quoted 3 lines

newer systems in order to improve performance by setting MSI_RMP_DIS bit. By design MSI_RMP_DIS bit of VMCONFIG registers is cleared.

register is

"MSI_RMP_DIS" doesn't appear in the kernel source. Is it the same as VMCONFIG_MSI_REMAP?

Yes. Using VMCONFIG_MSI_REMAP makes more sense here.

Quoted 48 lines
Quoted 5 lines

The same register gets cleared when system is put in S3 power state. VMD driver needs to set this register again in order to avoid interrupt issues with devices behind VMD if MSI remapping was disabled before.

Should there be a Fixes: tag here? I guess the failure symptom is that a driver doesn't see interrupts it expects?

Quoted 11 lines
---
 drivers/pci/controller/vmd.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/pci/controller/vmd.c +0 -0
index e06e9f4fc50f..247012b343fd 100644

drivers/pci/controller/vmd.c static int vmd_resume(struct device *dev) · line 984

Quoted 5 lines
drivers/pci/controller/vmd.c +2 -0
@@ -980,6 +980,9 @@ static int vmd_resume(struct device *dev)
 	struct vmd_dev *vmd = pci_get_drvdata(pdev);
 	int err, i;
 
+	if (!vmd->irq_domain)
+		vmd_set_msi_remapping(vmd, false);

I suppose this is functionally equivalent to this code in vmd_enable_domain():

        if (!(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) ||
            offset[0] || offset[1]) {
                ret = vmd_create_irq_domain(vmd);
        } else {
                vmd_set_msi_remapping(vmd, false);

because vmd->irq_domain will be NULL if we disabled MSI remapping at probe-time.

Maybe the vmd_enable_domain() code could be rearranged a little bit to make it obvious that we're doing the same thing both at probe-time and resume-time?

Should the vmd_resume() code *enable* MSI remapping in the other case, e.g.,

  if (vmd->irq_domain)
    vmd_set_msi_remapping(vmd, true);
  else
    vmd_set_msi_remapping(vmd, false);
Quoted 4 lines

I don't know what clears PCI_REG_VMCONFIG (which enabled MSI remapping IIUC). If it's cleared by firmware, the current patch depends on that firmware behavior, so maybe it would be good to remove that dependency?

Good suggestion. I will make changes for both enabling and disabling MSI remapping. That will help avoid dependency from firmware. Thanks.

Quoted 7 lines
Quoted 7 lines
drivers/pci/controller/vmd.c +0 -0
 	for (i = 0; i < vmd->msix_count; i++) {
 		err = devm_request_irq(dev, vmd->irqs[i].virq,
 				       vmd_irq, IRQF_NO_THREAD,
Signature · 1 line
2.34.1