Skip to contentSkip to conversation

PCI: Re-enable downstream port LTR if it was previously enabled

PATCH In review Mika Westerberg 3 participants 6 messages Jan 19, 2021 – Jan 25, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 PCI: Re-enable downstream port LTR if it was previously enabled

PCIe r5.0, sec 7.5.3.16 says that the downstream ports must reset the LTR enable bit if the link goes down (port goes DL_Down status). Now, if we had LTR previously enabled and the PCIe endpoint gets hot-removed and then hot-added back the ->ltr_path of the downstream port is still set but the port now does not have the LTR enable bit set anymore.

For this reason check if the bridge upstream had LTR enabled previously and re-enable it before enabling LTR for the endpoint.

---
Previous version can be found here:

https://lore.kernel.org/linux-pci/[email protected]/

Changes from the previous version:

* Corrected typos in the commit message
* No need to call pcie_downstream_port()

 drivers/pci/probe.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
drivers/pci/probe.c +16 -1
index 0eb68b47354f..a4a8c0305fb9 100644
@@ -2153,7 +2153,7 @@ static void pci_configure_ltr(struct pci_dev *dev)
 {
 #ifdef CONFIG_PCIEASPM
 	struct pci_host_bridge *host = pci_find_host_bridge(dev->bus);
-	struct pci_dev *bridge;
+	struct pci_dev *bridge = NULL;
 	u32 cap, ctl;
 
 	if (!pci_is_pcie(dev))
@@ -2191,6 +2191,21 @@ static void pci_configure_ltr(struct pci_dev *dev)
 	if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT ||
 	    ((bridge = pci_upstream_bridge(dev)) &&
 	      bridge->ltr_path)) {
+		/*
+		 * Downstream ports reset the LTR enable bit when the
+		 * link goes down (e.g on hot-remove) so re-enable the
+		 * bit here if not set anymore.
+		 * PCIe r5.0, sec 7.5.3.16.
+		 */
+		if (bridge) {
+			pcie_capability_read_dword(bridge, PCI_EXP_DEVCTL2, &ctl);
+			if (!(ctl & PCI_EXP_DEVCTL2_LTR_EN)) {
+				pci_dbg(bridge, "re-enabling LTR\n");
+				pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2,
+							 PCI_EXP_DEVCTL2_LTR_EN);
+			}
+		}
+		pci_dbg(dev, "enabling LTR\n");
 		pcie_capability_set_word(dev, PCI_EXP_DEVCTL2,
 					 PCI_EXP_DEVCTL2_LTR_EN);
 		dev->ltr_path = 1;
Signature · 1 line
2.29.2
#1 lore raw

[+cc Alex and Mingchuang et al from
https://lore.kernel.org/r/[email protected]]

Quoted 11 lines · Mika Westerberg

PCIe r5.0, sec 7.5.3.16 says that the downstream ports must reset the LTR enable bit if the link goes down (port goes DL_Down status). Now, if we had LTR previously enabled and the PCIe endpoint gets hot-removed and then hot-added back the ->ltr_path of the downstream port is still set but the port now does not have the LTR enable bit set anymore.

For this reason check if the bridge upstream had LTR enabled previously and re-enable it before enabling LTR for the endpoint.

I think this and Mingchuang's patch, which is essentially identical, are right and solves the problem for hot-remove/hot-add. In that scenario we call pci_configure_ltr() on the hot-added device, and with this patch, we'll re-enable LTR on the bridge leading to the new device before enabling LTR on the new device itself.

But don't we have a similar problem if we simply do a Fundamental Reset on a device? I think the reset path will restore the device's state, including PCI_EXP_DEVCTL2, but it doesn't do anything with the upstream bridge, does it?

So if a bridge and a device below it both have LTR enabled, can't we have the following:

- bridge LTR enabled
- device LTR enabled
- reset device, e.g., via Secondary Bus Reset
- link goes down, bridge disables LTR
- link comes back up, LTR disabled in both bridge and device
- restore device state, including LTR enable
- device sends LTR message
- bridge reports Unsupported Request

Quoted 51 lines

---
Previous version can be found here:

https://lore.kernel.org/linux-pci/[email protected]/

Changes from the previous version:

* Corrected typos in the commit message
* No need to call pcie_downstream_port()

 drivers/pci/probe.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
drivers/pci/probe.c +16 -1
index 0eb68b47354f..a4a8c0305fb9 100644
@@ -2153,7 +2153,7 @@ static void pci_configure_ltr(struct pci_dev *dev)
 {
 #ifdef CONFIG_PCIEASPM
 	struct pci_host_bridge *host = pci_find_host_bridge(dev->bus);
-	struct pci_dev *bridge;
+	struct pci_dev *bridge = NULL;
 	u32 cap, ctl;
 
 	if (!pci_is_pcie(dev))
@@ -2191,6 +2191,21 @@ static void pci_configure_ltr(struct pci_dev *dev)
 	if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT ||
 	    ((bridge = pci_upstream_bridge(dev)) &&
 	      bridge->ltr_path)) {
+		/*
+		 * Downstream ports reset the LTR enable bit when the
+		 * link goes down (e.g on hot-remove) so re-enable the
+		 * bit here if not set anymore.
+		 * PCIe r5.0, sec 7.5.3.16.
+		 */
+		if (bridge) {
+			pcie_capability_read_dword(bridge, PCI_EXP_DEVCTL2, &ctl);
+			if (!(ctl & PCI_EXP_DEVCTL2_LTR_EN)) {
+				pci_dbg(bridge, "re-enabling LTR\n");
+				pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2,
+							 PCI_EXP_DEVCTL2_LTR_EN);
+			}
+		}
+		pci_dbg(dev, "enabling LTR\n");
 		pcie_capability_set_word(dev, PCI_EXP_DEVCTL2,
 					 PCI_EXP_DEVCTL2_LTR_EN);
 		dev->ltr_path = 1;
Signature · 1 line
2.29.2

_______________________________________________
Linux-mediatek mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-mediatek

#2 lore raw
Quoted 22 lines · Bjorn Helgaas

[+cc Alex and Mingchuang et al from
https://lore.kernel.org/r/[email protected]]

Quoted 11 lines · Mika Westerberg

PCIe r5.0, sec 7.5.3.16 says that the downstream ports must reset the LTR enable bit if the link goes down (port goes DL_Down status). Now, if we had LTR previously enabled and the PCIe endpoint gets hot-removed and then hot-added back the ->ltr_path of the downstream port is still set but the port now does not have the LTR enable bit set anymore.

For this reason check if the bridge upstream had LTR enabled previously and re-enable it before enabling LTR for the endpoint.

I think this and Mingchuang's patch, which is essentially identical, are right and solves the problem for hot-remove/hot-add. In that scenario we call pci_configure_ltr() on the hot-added device, and with this patch, we'll re-enable LTR on the bridge leading to the new device before enabling LTR on the new device itself.

Quoted 5 lines

But don't we have a similar problem if we simply do a Fundamental Reset on a device? I think the reset path will restore the device's state, including PCI_EXP_DEVCTL2, but it doesn't do anything with the upstream bridge, does it?

Yes. I think the same problem exists under such scenario, and that’s the issue my patch intends to resolve.
I also prepared a v2 patch for review(update the patch description). Shall I submit the v2 patch for review?

Quoted 55 lines

So if a bridge and a device below it both have LTR enabled, can't we have the following:

- bridge LTR enabled
- device LTR enabled
- reset device, e.g., via Secondary Bus Reset
- link goes down, bridge disables LTR
- link comes back up, LTR disabled in both bridge and device
- restore device state, including LTR enable
- device sends LTR message
- bridge reports Unsupported Request

Quoted 43 lines

---
Previous version can be found here:

https://lore.kernel.org/linux-pci/[email protected]/

Changes from the previous version:

* Corrected typos in the commit message
* No need to call pcie_downstream_port()

 drivers/pci/probe.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
drivers/pci/probe.c +14 -1
index 0eb68b47354f..a4a8c0305fb9 100644
@@ -2153,7 +2153,7 @@ static void pci_configure_ltr(struct pci_dev *dev)
 {
 #ifdef CONFIG_PCIEASPM
 	struct pci_host_bridge *host = pci_find_host_bridge(dev->bus);
-	struct pci_dev *bridge;
+	struct pci_dev *bridge = NULL;
 	u32 cap, ctl;
 
 	if (!pci_is_pcie(dev))
@@ -2191,6 +2191,21 @@ static void pci_configure_ltr(struct pci_dev *dev)
 	if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT ||
 	    ((bridge = pci_upstream_bridge(dev)) &&
 	      bridge->ltr_path)) {
+		/*
+		 * Downstream ports reset the LTR enable bit when the
+		 * link goes down (e.g on hot-remove) so re-enable the
+		 * bit here if not set anymore.
+		 * PCIe r5.0, sec 7.5.3.16.
+		 */
+		if (bridge) {
+			pcie_capability_read_dword(bridge, PCI_EXP_DEVCTL2, &ctl);
+			if (!(ctl & PCI_EXP_DEVCTL2_LTR_EN)) {
+				pci_dbg(bridge, "re-enabling LTR\n");
+				pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2,
+							 PCI_EXP_DEVCTL2_LTR_EN);
+			}
Quoted 8 lines
Quoted 8 lines
drivers/pci/probe.c +2 -0
+		}
+		pci_dbg(dev, "enabling LTR\n");
 		pcie_capability_set_word(dev, PCI_EXP_DEVCTL2,
 					 PCI_EXP_DEVCTL2_LTR_EN);
 		dev->ltr_path = 1;
Signature · 1 line
2.29.2

_______________________________________________
Linux-mediatek mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-mediatek

#3 lore raw

Hi,

Quoted 29 lines · Mingchuang Qiao
Quoted 22 lines · Bjorn Helgaas

[+cc Alex and Mingchuang et al from
https://lore.kernel.org/r/[email protected]]

Quoted 11 lines · Mika Westerberg

PCIe r5.0, sec 7.5.3.16 says that the downstream ports must reset the LTR enable bit if the link goes down (port goes DL_Down status). Now, if we had LTR previously enabled and the PCIe endpoint gets hot-removed and then hot-added back the ->ltr_path of the downstream port is still set but the port now does not have the LTR enable bit set anymore.

For this reason check if the bridge upstream had LTR enabled previously and re-enable it before enabling LTR for the endpoint.

I think this and Mingchuang's patch, which is essentially identical, are right and solves the problem for hot-remove/hot-add. In that scenario we call pci_configure_ltr() on the hot-added device, and with this patch, we'll re-enable LTR on the bridge leading to the new device before enabling LTR on the new device itself.

Quoted 5 lines

But don't we have a similar problem if we simply do a Fundamental Reset on a device? I think the reset path will restore the device's state, including PCI_EXP_DEVCTL2, but it doesn't do anything with the upstream bridge, does it?

Quoted 4 lines

Yes. I think the same problem exists under such scenario, and that’s the issue my patch intends to resolve.
I also prepared a v2 patch for review(update the patch description). Shall I submit the v2 patch for review?

I looked at your patch and indeed it is essentially doing the same as this one. So let's forget this patch and go forward with yours :)

Would you like to expand your patch to handle the reset case too that Bjorn desribes below?

Quoted 11 lines
Quoted 11 lines

So if a bridge and a device below it both have LTR enabled, can't we have the following:

- bridge LTR enabled
- device LTR enabled
- reset device, e.g., via Secondary Bus Reset
- link goes down, bridge disables LTR
- link comes back up, LTR disabled in both bridge and device
- restore device state, including LTR enable
- device sends LTR message
- bridge reports Unsupported Request

_______________________________________________
Linux-mediatek mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-mediatek

#4 lore raw
In reply to Mingchuang Qiao
Quoted 28 lines · Mingchuang Qiao
Quoted 22 lines · Bjorn Helgaas

[+cc Alex and Mingchuang et al from
https://lore.kernel.org/r/[email protected]]

Quoted 11 lines · Mika Westerberg

PCIe r5.0, sec 7.5.3.16 says that the downstream ports must reset the LTR enable bit if the link goes down (port goes DL_Down status). Now, if we had LTR previously enabled and the PCIe endpoint gets hot-removed and then hot-added back the ->ltr_path of the downstream port is still set but the port now does not have the LTR enable bit set anymore.

For this reason check if the bridge upstream had LTR enabled previously and re-enable it before enabling LTR for the endpoint.

I think this and Mingchuang's patch, which is essentially identical, are right and solves the problem for hot-remove/hot-add. In that scenario we call pci_configure_ltr() on the hot-added device, and with this patch, we'll re-enable LTR on the bridge leading to the new device before enabling LTR on the new device itself.

Quoted 4 lines

But don't we have a similar problem if we simply do a Fundamental Reset on a device? I think the reset path will restore the device's state, including PCI_EXP_DEVCTL2, but it doesn't do anything with the upstream bridge, does it?

Quoted 4 lines

Yes. I think the same problem exists under such scenario, and that’s the issue my patch intends to resolve.
I also prepared a v2 patch for review(update the patch description). Shall I submit the v2 patch for review?

How does your patch solve this for the reset path? I don't think we call pci_configure_ltr() when we reset a device.

Quoted 55 lines
Quoted 55 lines

So if a bridge and a device below it both have LTR enabled, can't we have the following:

- bridge LTR enabled
- device LTR enabled
- reset device, e.g., via Secondary Bus Reset
- link goes down, bridge disables LTR
- link comes back up, LTR disabled in both bridge and device
- restore device state, including LTR enable
- device sends LTR message
- bridge reports Unsupported Request

Quoted 43 lines

---
Previous version can be found here:

https://lore.kernel.org/linux-pci/[email protected]/

Changes from the previous version:

* Corrected typos in the commit message
* No need to call pcie_downstream_port()

 drivers/pci/probe.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
drivers/pci/probe.c +14 -1
index 0eb68b47354f..a4a8c0305fb9 100644
@@ -2153,7 +2153,7 @@ static void pci_configure_ltr(struct pci_dev *dev)
 {
 #ifdef CONFIG_PCIEASPM
 	struct pci_host_bridge *host = pci_find_host_bridge(dev->bus);
-	struct pci_dev *bridge;
+	struct pci_dev *bridge = NULL;
 	u32 cap, ctl;
 
 	if (!pci_is_pcie(dev))
@@ -2191,6 +2191,21 @@ static void pci_configure_ltr(struct pci_dev *dev)
 	if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT ||
 	    ((bridge = pci_upstream_bridge(dev)) &&
 	      bridge->ltr_path)) {
+		/*
+		 * Downstream ports reset the LTR enable bit when the
+		 * link goes down (e.g on hot-remove) so re-enable the
+		 * bit here if not set anymore.
+		 * PCIe r5.0, sec 7.5.3.16.
+		 */
+		if (bridge) {
+			pcie_capability_read_dword(bridge, PCI_EXP_DEVCTL2, &ctl);
+			if (!(ctl & PCI_EXP_DEVCTL2_LTR_EN)) {
+				pci_dbg(bridge, "re-enabling LTR\n");
+				pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2,
+							 PCI_EXP_DEVCTL2_LTR_EN);
+			}
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
drivers/pci/probe.c +2 -0
+		}
+		pci_dbg(dev, "enabling LTR\n");
 		pcie_capability_set_word(dev, PCI_EXP_DEVCTL2,
 					 PCI_EXP_DEVCTL2_LTR_EN);
 		dev->ltr_path = 1;
Signature · 1 line
2.29.2

_______________________________________________
Linux-mediatek mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-mediatek

#5 lore raw
Quoted 29 lines · Bjorn Helgaas
Quoted 28 lines · Mingchuang Qiao
Quoted 26 lines · Bjorn Helgaas

[+cc Alex and Mingchuang et al from
https://lore.kernel.org/r/[email protected]]

Quoted 11 lines · Mika Westerberg

PCIe r5.0, sec 7.5.3.16 says that the downstream ports must reset the LTR enable bit if the link goes down (port goes DL_Down status). Now, if we had LTR previously enabled and the PCIe endpoint gets hot-removed and then hot-added back the ->ltr_path of the downstream port is still set but the port now does not have the LTR enable bit set anymore.

For this reason check if the bridge upstream had LTR enabled previously and re-enable it before enabling LTR for the endpoint.

I think this and Mingchuang's patch, which is essentially identical, are right and solves the problem for hot-remove/hot-add. In that scenario we call pci_configure_ltr() on the hot-added device, and with this patch, we'll re-enable LTR on the bridge leading to the new device before enabling LTR on the new device itself.

But don't we have a similar problem if we simply do a Fundamental Reset on a device? I think the reset path will restore the device's state, including PCI_EXP_DEVCTL2, but it doesn't do anything with the upstream bridge, does it?

Quoted 8 lines
Quoted 4 lines

Yes. I think the same problem exists under such scenario, and that’s the issue my patch intends to resolve.
I also prepared a v2 patch for review(update the patch description). Shall I submit the v2 patch for review?

How does your patch solve this for the reset path? I don't think we call pci_configure_ltr() when we reset a device.

Sorry, I misunderstand the reset path. When we do a Fundamental Reset on a device, we can trigger device removal and rescan flow to restore the device. In device rescan flow, pci_configure_ltr() will be invoked. I regard the "remove and rescan flow" as a part of reset path for this case :)
If we restore device just with pci_restore_state() in device driver after device resets, the LTR problem also exists due to pci_restore_state() does nothing with upstream bridge. In next patch, I would like to re-enable LTR for upstream bridge before restoring device's PCI_EXP_DEVCTL2 if it is needed.

Quoted 56 lines
Quoted 56 lines
Quoted 56 lines

So if a bridge and a device below it both have LTR enabled, can't we have the following:

- bridge LTR enabled
- device LTR enabled
- reset device, e.g., via Secondary Bus Reset
- link goes down, bridge disables LTR
- link comes back up, LTR disabled in both bridge and device
- restore device state, including LTR enable
- device sends LTR message
- bridge reports Unsupported Request

Quoted 44 lines

---
Previous version can be found here:

https://lore.kernel.org/linux-pci/[email protected]/

Changes from the previous version:

* Corrected typos in the commit message
* No need to call pcie_downstream_port()

 drivers/pci/probe.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
drivers/pci/probe.c +15 -1
index 0eb68b47354f..a4a8c0305fb9 100644
@@ -2153,7 +2153,7 @@ static void pci_configure_ltr(struct pci_dev *dev)
 {
 #ifdef CONFIG_PCIEASPM
 	struct pci_host_bridge *host = pci_find_host_bridge(dev->bus);
-	struct pci_dev *bridge;
+	struct pci_dev *bridge = NULL;
 	u32 cap, ctl;
 
 	if (!pci_is_pcie(dev))
@@ -2191,6 +2191,21 @@ static void pci_configure_ltr(struct pci_dev *dev)
 	if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT ||
 	    ((bridge = pci_upstream_bridge(dev)) &&
 	      bridge->ltr_path)) {
+		/*
+		 * Downstream ports reset the LTR enable bit when the
+		 * link goes down (e.g on hot-remove) so re-enable the
+		 * bit here if not set anymore.
+		 * PCIe r5.0, sec 7.5.3.16.
+		 */
+		if (bridge) {
+			pcie_capability_read_dword(bridge, PCI_EXP_DEVCTL2, &ctl);
+			if (!(ctl & PCI_EXP_DEVCTL2_LTR_EN)) {
+				pci_dbg(bridge, "re-enabling LTR\n");
+				pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2,
+							 PCI_EXP_DEVCTL2_LTR_EN);
+			}
+		}
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
drivers/pci/probe.c +1 -0
+		pci_dbg(dev, "enabling LTR\n");
 		pcie_capability_set_word(dev, PCI_EXP_DEVCTL2,
 					 PCI_EXP_DEVCTL2_LTR_EN);
 		dev->ltr_path = 1;
Signature · 1 line
2.29.2

_______________________________________________
Linux-mediatek mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-mediatek