Skip to contentSkip to conversation

PCI: dwc: Fix MSI not work after resume

PATCH RESEND In review Jisheng Zhang 3 participants 6 messages Mar 1, 2021 – Mar 23, 2021

Subsystems (3)

v1 of 2 In review Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH RESEND 1/1 PCI: dwc: Fix MSI not work after resume

After we move dw_pcie_msi_init() into core -- dw_pcie_host_init(), the MSI stops working after resume. Because dw_pcie_host_init() is only called once during probe. To fix this issue, we move dw_pcie_msi_init() to dw_pcie_setup_rc().

Fixes 59fbab1ae40e ("PCI: dwc: Move dw_pcie_msi_init() into core")

---
Since v1:
- collect Reviewed-by tag

 drivers/pci/controller/dwc/pcie-designware-host.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/pci/controller/dwc/pcie-designware-host.c +2 -1
index 7e55b2b66182..e6c274f4485c 100644
@@ -400,7 +400,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
 	}
 
 	dw_pcie_setup_rc(pp);
-	dw_pcie_msi_init(pp);
 
 	if (!dw_pcie_link_up(pci) && pci->ops && pci->ops->start_link) {
 		ret = pci->ops->start_link(pci);
@@ -551,6 +550,8 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
 		}
 	}
 
+	dw_pcie_msi_init(pp);
+
 	/* Setup RC BARs */
 	dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_0, 0x00000004);
 	dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_1, 0x00000000);
Signature · 1 line
2.30.1
#1 lore raw
Quoted 8 lines · Jisheng Zhang

After we move dw_pcie_msi_init() into core -- dw_pcie_host_init(), the MSI stops working after resume. Because dw_pcie_host_init() is only called once during probe. To fix this issue, we move dw_pcie_msi_init() to dw_pcie_setup_rc().

Fixes 59fbab1ae40e ("PCI: dwc: Move dw_pcie_msi_init() into core")

Oops, sorry, looks like this fell through the cracks. Since 59fbab1ae40e appeared in v5.11, I think we should add:

Cc [email protected] # v5.11+

I'm sure Lorenzo will add it when applying, so no need to repost just for that.

Quoted 31 lines

---
Since v1:
- collect Reviewed-by tag

 drivers/pci/controller/dwc/pcie-designware-host.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/pci/controller/dwc/pcie-designware-host.c +2 -1
index 7e55b2b66182..e6c274f4485c 100644
@@ -400,7 +400,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
 	}
 
 	dw_pcie_setup_rc(pp);
-	dw_pcie_msi_init(pp);
 
 	if (!dw_pcie_link_up(pci) && pci->ops && pci->ops->start_link) {
 		ret = pci->ops->start_link(pci);
@@ -551,6 +550,8 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
 		}
 	}
 
+	dw_pcie_msi_init(pp);
+
 	/* Setup RC BARs */
 	dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_0, 0x00000004);
 	dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_1, 0x00000000);
Signature · 1 line
2.30.1
#2 lore raw
In reply to Jisheng Zhang

[+cc Kishon, Richard, Lucas, Dilip]

Quoted 4 lines · Jisheng Zhang

After we move dw_pcie_msi_init() into core -- dw_pcie_host_init(), the MSI stops working after resume. Because dw_pcie_host_init() is only called once during probe. To fix this issue, we move dw_pcie_msi_init() to dw_pcie_setup_rc().

This patch looks fine, but I don't think the commit log tells the whole story.

Prior to 59fbab1ae40e, it looks like the only dwc-based drivers with resume functions were dra7xx, imx6, intel-gw, and tegra [1].

Only tegra called dw_pcie_msi_init() in the resume path, and I do think 59fbab1ae40e broke MSI after resume because it removed the dw_pcie_msi_init() call from tegra_pcie_enable_msi_interrupts().

I'm not convinced this patch fixes it reliably, though. The call chain looks like this:

tegra_pcie_dw_resume_noirq

    tegra_pcie_dw_start_link
      if (dw_pcie_wait_for_link(pci))
        dw_pcie_setup_rc

dw_pcie_wait_for_link() returns 0 if the link is up, so we only call dw_pcie_setup_rc() in the case where the link *didn't* come up. If the link comes up nicely without retry, we won't call dw_pcie_setup_rc() and hence won't call dw_pcie_msi_init().

Since then, exynos added a resume function. My guess is MSI never worked after resume for dra7xx, exynos, imx6, and intel-gw because they don't call dw_pcie_msi_init() in their resume functions.

This patch looks like it should fix MSI after resume for exynos, imx6, and intel-gw because they *do* call dw_pcie_setup_rc() from their resume functions [2], and after this patch, dw_pcie_msi_init() will be called from there.

I suspect MSI after resume still doesn't work on dra7xx.

[1] git grep -A20 -e "static.*resume_noirq" 59fbab1ae40e^:drivers/pci/controller/dwc
[2] git grep -A20 -e "static.*resume_noirq" drivers/pci/controller/dwc

Quoted 34 lines
Fixes 59fbab1ae40e ("PCI: dwc: Move dw_pcie_msi_init() into core")

---
Since v1:
- collect Reviewed-by tag

 drivers/pci/controller/dwc/pcie-designware-host.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/pci/controller/dwc/pcie-designware-host.c +2 -1
index 7e55b2b66182..e6c274f4485c 100644
@@ -400,7 +400,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
 	}
 
 	dw_pcie_setup_rc(pp);
-	dw_pcie_msi_init(pp);
 
 	if (!dw_pcie_link_up(pci) && pci->ops && pci->ops->start_link) {
 		ret = pci->ops->start_link(pci);
@@ -551,6 +550,8 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
 		}
 	}
 
+	dw_pcie_msi_init(pp);
+
 	/* Setup RC BARs */
 	dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_0, 0x00000004);
 	dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_1, 0x00000000);
Signature · 1 line
2.30.1
#3 lore raw

Hi

Quoted 23 lines · On Mon, 22 Mar 2021 20:24:41 -0500 Bjorn Helgaas wrote:

[+cc Kishon, Richard, Lucas, Dilip]

Quoted 4 lines · Jisheng Zhang

After we move dw_pcie_msi_init() into core -- dw_pcie_host_init(), the MSI stops working after resume. Because dw_pcie_host_init() is only called once during probe. To fix this issue, we move dw_pcie_msi_init() to dw_pcie_setup_rc().

This patch looks fine, but I don't think the commit log tells the whole story.

Prior to 59fbab1ae40e, it looks like the only dwc-based drivers with resume functions were dra7xx, imx6, intel-gw, and tegra [1].

Only tegra called dw_pcie_msi_init() in the resume path, and I do think 59fbab1ae40e broke MSI after resume because it removed the dw_pcie_msi_init() call from tegra_pcie_enable_msi_interrupts().

I'm not convinced this patch fixes it reliably, though. The call chain looks like this:

tegra_pcie_dw_resume_noirq

Quoted 8 lines
    tegra_pcie_dw_start_link
      if (dw_pcie_wait_for_link(pci))
        dw_pcie_setup_rc

dw_pcie_wait_for_link() returns 0 if the link is up, so we only call dw_pcie_setup_rc() in the case where the link *didn't* come up. If the link comes up nicely without retry, we won't call dw_pcie_setup_rc() and hence won't call dw_pcie_msi_init().

The v1 version patch was sent before commit 275e88b06a (PCI: tegra: Fix host link initialization"). At that time, the resume path looks like this:

tegra_pcie_dw_resume_noirq
tegra_pcie_dw_host_init
tegra_pcie_prepare_host
dw_pcie_setup_rc

so after patch, dw_pcie_msi_init() will be called. But now it seems that the tegra version needs one more fix for the resume.

So could I sent a new patch to update the commit-msg a bit?

Quoted 11 lines

Since then, exynos added a resume function. My guess is MSI never worked after resume for dra7xx, exynos, imx6, and intel-gw because they don't call dw_pcie_msi_init() in their resume functions.

This patch looks like it should fix MSI after resume for exynos, imx6, and intel-gw because they *do* call dw_pcie_setup_rc() from their resume functions [2], and after this patch, dw_pcie_msi_init() will be called from there.

I suspect MSI after resume still doesn't work on dra7xx.

I checked the dra7xx history, I'm afraid that the resume never works from the beginning if the host lost power during suspend, I guess the platform never power off the host but only the phy?

Quoted 38 lines

[1] git grep -A20 -e "static.*resume_noirq" 59fbab1ae40e^:drivers/pci/controller/dwc
[2] git grep -A20 -e "static.*resume_noirq" drivers/pci/controller/dwc

Quoted 34 lines
Fixes 59fbab1ae40e ("PCI: dwc: Move dw_pcie_msi_init() into core")

---
Since v1:
- collect Reviewed-by tag

 drivers/pci/controller/dwc/pcie-designware-host.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/pci/controller/dwc/pcie-designware-host.c +2 -2
index 7e55b2b66182..e6c274f4485c 100644
@@ -400,7 +400,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
      }
 
      dw_pcie_setup_rc(pp);
-     dw_pcie_msi_init(pp);
 
      if (!dw_pcie_link_up(pci) && pci->ops && pci->ops->start_link) {
              ret = pci->ops->start_link(pci);
@@ -551,6 +550,8 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
              }
      }
 
+     dw_pcie_msi_init(pp);
+
      /* Setup RC BARs */
      dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_0, 0x00000004);
      dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_1, 0x00000000);
--

2.30.1

#4 lore raw

[-cc Dilip (mail to him bounced)]

Quoted 41 lines · Jisheng Zhang
Quoted 23 lines · On Mon, 22 Mar 2021 20:24:41 -0500 Bjorn Helgaas wrote:

[+cc Kishon, Richard, Lucas, Dilip]

Quoted 4 lines · Jisheng Zhang

After we move dw_pcie_msi_init() into core -- dw_pcie_host_init(), the MSI stops working after resume. Because dw_pcie_host_init() is only called once during probe. To fix this issue, we move dw_pcie_msi_init() to dw_pcie_setup_rc().

This patch looks fine, but I don't think the commit log tells the whole story.

Prior to 59fbab1ae40e, it looks like the only dwc-based drivers with resume functions were dra7xx, imx6, intel-gw, and tegra [1].

Only tegra called dw_pcie_msi_init() in the resume path, and I do think 59fbab1ae40e broke MSI after resume because it removed the dw_pcie_msi_init() call from tegra_pcie_enable_msi_interrupts().

I'm not convinced this patch fixes it reliably, though. The call chain looks like this:

tegra_pcie_dw_resume_noirq

Quoted 8 lines
    tegra_pcie_dw_start_link
      if (dw_pcie_wait_for_link(pci))
        dw_pcie_setup_rc

dw_pcie_wait_for_link() returns 0 if the link is up, so we only call dw_pcie_setup_rc() in the case where the link *didn't* come up. If the link comes up nicely without retry, we won't call dw_pcie_setup_rc() and hence won't call dw_pcie_msi_init().

The v1 version patch was sent before commit 275e88b06a (PCI: tegra: Fix host link initialization"). At that time, the resume path looks like this:

tegra_pcie_dw_resume_noirq
tegra_pcie_dw_host_init
tegra_pcie_prepare_host
dw_pcie_setup_rc

Quoted 4 lines

so after patch, dw_pcie_msi_init() will be called. But now it seems that the tegra version needs one more fix for the resume.

So could I sent a new patch to update the commit-msg a bit?

This patch only touches the dwc core, and the commit log says generically that it fixes MSI after resume, so one could assume that it applies to all dwc-based drivers. But I don't think it's that simple, so I'd like to know *which* drivers are fixed and which commits are related. I don't see how 59fbab1ae40e breaks anything except tegra.

Quoted 9 lines
Quoted 9 lines

Since then, exynos added a resume function. My guess is MSI never worked after resume for dra7xx, exynos, imx6, and intel-gw because they don't call dw_pcie_msi_init() in their resume functions.

This patch looks like it should fix MSI after resume for exynos, imx6, and intel-gw because they *do* call dw_pcie_setup_rc() from their resume functions [2], and after this patch, dw_pcie_msi_init() will be called from there.

Quoted 5 lines
Quoted 1 line

I suspect MSI after resume still doesn't work on dra7xx.

I checked the dra7xx history, I'm afraid that the resume never works from the beginning if the host lost power during suspend, I guess the platform never power off the host but only the phy?

Sounds like that would make sense.

Quoted 38 lines
Quoted 37 lines

[1] git grep -A20 -e "static.*resume_noirq" 59fbab1ae40e^:drivers/pci/controller/dwc
[2] git grep -A20 -e "static.*resume_noirq" drivers/pci/controller/dwc

Quoted 34 lines
Fixes 59fbab1ae40e ("PCI: dwc: Move dw_pcie_msi_init() into core")

---
Since v1:
- collect Reviewed-by tag

 drivers/pci/controller/dwc/pcie-designware-host.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/pci/controller/dwc/pcie-designware-host.c +2 -2
index 7e55b2b66182..e6c274f4485c 100644
@@ -400,7 +400,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
      }
 
      dw_pcie_setup_rc(pp);
-     dw_pcie_msi_init(pp);
 
      if (!dw_pcie_link_up(pci) && pci->ops && pci->ops->start_link) {
              ret = pci->ops->start_link(pci);
@@ -551,6 +550,8 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
              }
      }
 
+     dw_pcie_msi_init(pp);
+
      /* Setup RC BARs */
      dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_0, 0x00000004);
      dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_1, 0x00000000);
--

2.30.1

#5 lore raw

Hi,

Quoted 47 lines · Bjorn Helgaas

[-cc Dilip (mail to him bounced)]

Quoted 44 lines · Jisheng Zhang
Quoted 23 lines · On Mon, 22 Mar 2021 20:24:41 -0500 Bjorn Helgaas wrote:

[+cc Kishon, Richard, Lucas, Dilip]

Quoted 4 lines · Jisheng Zhang

After we move dw_pcie_msi_init() into core -- dw_pcie_host_init(), the MSI stops working after resume. Because dw_pcie_host_init() is only called once during probe. To fix this issue, we move dw_pcie_msi_init() to dw_pcie_setup_rc().

This patch looks fine, but I don't think the commit log tells the whole story.

Prior to 59fbab1ae40e, it looks like the only dwc-based drivers with resume functions were dra7xx, imx6, intel-gw, and tegra [1].

Only tegra called dw_pcie_msi_init() in the resume path, and I do think 59fbab1ae40e broke MSI after resume because it removed the dw_pcie_msi_init() call from tegra_pcie_enable_msi_interrupts().

I'm not convinced this patch fixes it reliably, though. The call chain looks like this:

tegra_pcie_dw_resume_noirq

Quoted 8 lines
    tegra_pcie_dw_start_link
      if (dw_pcie_wait_for_link(pci))
        dw_pcie_setup_rc

dw_pcie_wait_for_link() returns 0 if the link is up, so we only call dw_pcie_setup_rc() in the case where the link *didn't* come up. If the link comes up nicely without retry, we won't call dw_pcie_setup_rc() and hence won't call dw_pcie_msi_init().

The v1 version patch was sent before commit 275e88b06a (PCI: tegra: Fix host link initialization"). At that time, the resume path looks like this:

tegra_pcie_dw_resume_noirq
tegra_pcie_dw_host_init
tegra_pcie_prepare_host
dw_pcie_setup_rc

so after patch, dw_pcie_msi_init() will be called. But now it seems that the tegra version needs one more fix for the resume.

Quoted 23 lines
Quoted 1 line

So could I sent a new patch to update the commit-msg a bit?

This patch only touches the dwc core, and the commit log says generically that it fixes MSI after resume, so one could assume that it applies to all dwc-based drivers. But I don't think it's that simple, so I'd like to know *which* drivers are fixed and which commits are related. I don't see how 59fbab1ae40e breaks anything except tegra.

Quoted 14 lines
Quoted 10 lines

Since then, exynos added a resume function. My guess is MSI never worked after resume for dra7xx, exynos, imx6, and intel-gw because they don't call dw_pcie_msi_init() in their resume functions.

This patch looks like it should fix MSI after resume for exynos, imx6, and intel-gw because they *do* call dw_pcie_setup_rc() from their resume functions [2], and after this patch, dw_pcie_msi_init() will be called from there.

I suspect MSI after resume still doesn't work on dra7xx.

I checked the dra7xx history, I'm afraid that the resume never works from the beginning if the host lost power during suspend, I guess the platform never power off the host but only the phy?

Suspend on dra7xx disabled clocks and powered off phy (at-least while suspend/resume hooks were merged) and resume enabled clocks and phy. However suspend/resume is broken in dra7xx system and is not validated. I'll send a patch to remove the suspend/resume hooks in dra7xx.

Thanks
Kishon

Quoted 41 lines

Sounds like that would make sense.

Quoted 38 lines
Quoted 37 lines

[1] git grep -A20 -e "static.*resume_noirq" 59fbab1ae40e^:drivers/pci/controller/dwc
[2] git grep -A20 -e "static.*resume_noirq" drivers/pci/controller/dwc

Quoted 34 lines
Fixes 59fbab1ae40e ("PCI: dwc: Move dw_pcie_msi_init() into core")

---
Since v1:
- collect Reviewed-by tag

 drivers/pci/controller/dwc/pcie-designware-host.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/pci/controller/dwc/pcie-designware-host.c +2 -2
index 7e55b2b66182..e6c274f4485c 100644
@@ -400,7 +400,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
      }
 
      dw_pcie_setup_rc(pp);
-     dw_pcie_msi_init(pp);
 
      if (!dw_pcie_link_up(pci) && pci->ops && pci->ops->start_link) {
              ret = pci->ops->start_link(pci);
@@ -551,6 +550,8 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
              }
      }
 
+     dw_pcie_msi_init(pp);
+
      /* Setup RC BARs */
      dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_0, 0x00000004);
      dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_1, 0x00000000);
--

2.30.1