Skip to contentSkip to conversation

PCI: Do not use pcie_get_speed_cap() to determine when to start waiting

PATCH In review Mika Westerberg 3 participants 6 messages May 14, 2020 – May 16, 2020 PCI

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 PCI: Do not use pcie_get_speed_cap() to determine when to start waiting

Kai-Heng Feng reported that it takes long time (>1s) to resume Thunderbolt connected PCIe devices from both runtime suspend and system sleep (s2idle).

These PCIe downstream ports the second link capability (PCI_EXP_LNKCAP2) announces support for speeds > 5 GT/s but it is then capped by the second link control (PCI_EXP_LNKCTL2) register to 2.5 GT/s. This possiblity was not considered in pci_bridge_wait_for_secondary_bus() so it ended up waiting for 1100 ms as these ports do not support active link layer reporting either.

PCIe spec 5.0 section 6.6.1 mandates that we must wait minimum of 100 ms before sending configuration request to the device below, if the port does not support speeds > 5 GT/s, and if it does we first need to wait for the data link layer to become active before waiting for that 100 ms.

PCIe spec 5.0 section 7.5.3.6 further says that all downstream ports that support speeds > 5 GT/s must support active link layer reporting so instead of looking for the speed we can check for the active link layer reporting capability and determine how to wait based on that (as they go hand in hand).

While there restructure the code a bit so that the delay is always issued in pci_bridge_wait_for_secondary_bus() by passing value of 0 to pcie_wait_for_link_delay().

---
v2: Restructured the code a bit so that it should be more readable now

Previous version can be found here:

https://lore.kernel.org/linux-pci/[email protected]/

@Kai-heng, since this patch is slightly different than what you tried, I wonder if you could check that it still solves the and does not break anything? I tested it myself but it would be nice to get your Tested-by to make sure it still works.

 drivers/pci/pci.c | 23 ++++++++++++++---------
 1 file changed, 14 insertions(+), 9 deletions(-)
drivers/pci/pci.c +14 -9
index 595fcf59843f..590c73dc7e0d 100644
@@ -4660,7 +4660,8 @@ static int pci_pm_reset(struct pci_dev *dev, int probe)
  * pcie_wait_for_link_delay - Wait until link is active or inactive
  * @pdev: Bridge device
  * @active: waiting for active or inactive?
- * @delay: Delay to wait after link has become active (in ms)
+ * @delay: Delay to wait after link has become active (in ms). Specify %0
+ *	   for no delay.
  *
  * Use this to wait till link becomes active or inactive.
  */
@@ -4701,7 +4702,7 @@ static bool pcie_wait_for_link_delay(struct pci_dev *pdev, bool active,
 		msleep(10);
 		timeout -= 10;
 	}
-	if (active && ret)
+	if (active && ret && delay)
 		msleep(delay);
 	else if (ret != active)
 		pci_info(pdev, "Data Link Layer Link Active not %s in 1000 msec\n",
@@ -4822,17 +4823,21 @@ void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev)
 	if (!pcie_downstream_port(dev))
 		return;
 
-	if (pcie_get_speed_cap(dev) <= PCIE_SPEED_5_0GT) {
-		pci_dbg(dev, "waiting %d ms for downstream link\n", delay);
-		msleep(delay);
-	} else {
-		pci_dbg(dev, "waiting %d ms for downstream link, after activation\n",
-			delay);
-		if (!pcie_wait_for_link_delay(dev, true, delay)) {
+	/*
+	 * Since PCIe spec mandates that all downstream ports that support
+	 * speeds greater than 5 GT/s must support data link layer active
+	 * reporting so if it is supported we poll for the link to become
+	 * active before issuing the mandatory delay.
+	 */
+	if (dev->link_active_reporting) {
+		pci_dbg(dev, "waiting for link to train\n");
+		if (!pcie_wait_for_link_delay(dev, true, 0)) {
 			/* Did not train, no need to wait any further */
 			return;
 		}
 	}
+	pci_dbg(child, "waiting %d ms to become accessible\n", delay);
+	msleep(delay);
 
 	if (!pci_device_is_present(child)) {
 		pci_dbg(child, "waiting additional %d ms to become accessible\n", delay);
Signature · 1 line
2.26.2
#1 lore raw
Quoted 10 lines · Mika Westerberg

Kai-Heng Feng reported that it takes long time (>1s) to resume Thunderbolt connected PCIe devices from both runtime suspend and system sleep (s2idle).

These PCIe downstream ports the second link capability (PCI_EXP_LNKCAP2) announces support for speeds > 5 GT/s but it is then capped by the second link control (PCI_EXP_LNKCTL2) register to 2.5 GT/s. This possiblity was not considered in pci_bridge_wait_for_secondary_bus() so it ended up waiting for 1100 ms as these ports do not support active link layer reporting either.

I don't think PCI_EXP_LNKCTL2 is relevant here. I think the lack of Data Link Layer Link Active is just a chip erratum. Is that documented anywhere?

Quoted 48 lines

PCIe spec 5.0 section 6.6.1 mandates that we must wait minimum of 100 ms before sending configuration request to the device below, if the port does not support speeds > 5 GT/s, and if it does we first need to wait for the data link layer to become active before waiting for that 100 ms.

PCIe spec 5.0 section 7.5.3.6 further says that all downstream ports that support speeds > 5 GT/s must support active link layer reporting so instead of looking for the speed we can check for the active link layer reporting capability and determine how to wait based on that (as they go hand in hand).

While there restructure the code a bit so that the delay is always issued in pci_bridge_wait_for_secondary_bus() by passing value of 0 to pcie_wait_for_link_delay().

---
v2: Restructured the code a bit so that it should be more readable now

Previous version can be found here:

https://lore.kernel.org/linux-pci/[email protected]/

@Kai-heng, since this patch is slightly different than what you tried, I wonder if you could check that it still solves the and does not break anything? I tested it myself but it would be nice to get your Tested-by to make sure it still works.

 drivers/pci/pci.c | 23 ++++++++++++++---------
 1 file changed, 14 insertions(+), 9 deletions(-)
drivers/pci/pci.c +2 -1
index 595fcf59843f..590c73dc7e0d 100644
@@ -4660,7 +4660,8 @@ static int pci_pm_reset(struct pci_dev *dev, int probe)
  * pcie_wait_for_link_delay - Wait until link is active or inactive
  * @pdev: Bridge device
  * @active: waiting for active or inactive?
- * @delay: Delay to wait after link has become active (in ms)
+ * @delay: Delay to wait after link has become active (in ms). Specify %0
+ *	   for no delay.
  *
  * Use this to wait till link becomes active or inactive.
  */

drivers/pci/pci.c static bool pcie_wait_for_link_delay(struct pci_dev *pdev, bool active, · line 4704

Quoted 3 lines
drivers/pci/pci.c +0 -0
@@ -4701,7 +4702,7 @@ static bool pcie_wait_for_link_delay(struct pci_dev *pdev, bool active,
 		msleep(10);
 		timeout -= 10;
 	}

I think maybe the code above (not included in the context here) should say:

  if (!pdev->link_active_reporting) {
    msleep(timeout + delay);
    return true;
  }

to match the rest of 4827d63891b6 ("PCI/PM: Add pcie_wait_for_link_delay()"). What do you think? If you agree, I'd make that a separate patch because it's not related to the current fix.

Quoted 21 lines
drivers/pci/pci.c +6 -8
-	if (active && ret)
+	if (active && ret && delay)
 		msleep(delay);
 	else if (ret != active)
 		pci_info(pdev, "Data Link Layer Link Active not %s in 1000 msec\n",
@@ -4822,17 +4823,21 @@ void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev)
 	if (!pcie_downstream_port(dev))
 		return;
 
-	if (pcie_get_speed_cap(dev) <= PCIE_SPEED_5_0GT) {
-		pci_dbg(dev, "waiting %d ms for downstream link\n", delay);
-		msleep(delay);
-	} else {
-		pci_dbg(dev, "waiting %d ms for downstream link, after activation\n",
-			delay);
-		if (!pcie_wait_for_link_delay(dev, true, delay)) {
+	/*
+	 * Since PCIe spec mandates that all downstream ports that support
+	 * speeds greater than 5 GT/s must support data link layer active
+	 * reporting so if it is supported we poll for the link to become
+	 * active before issuing the mandatory delay.

drivers/pci/pci.c void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev) · line 4840

Quoted 10 lines
drivers/pci/pci.c +6 -0
+	 */
+	if (dev->link_active_reporting) {
+		pci_dbg(dev, "waiting for link to train\n");
+		if (!pcie_wait_for_link_delay(dev, true, 0)) {
 			/* Did not train, no need to wait any further */
 			return;
 		}
 	}
+	pci_dbg(child, "waiting %d ms to become accessible\n", delay);
+	msleep(delay);

Nice solution, I like this a lot.

Quoted 5 lines
drivers/pci/pci.c +0 -0
 	if (!pci_device_is_present(child)) {
 		pci_dbg(child, "waiting additional %d ms to become accessible\n", delay);
Signature · 1 line
2.26.2
#2 lore raw
Quoted 12 lines · Bjorn Helgaas
Quoted 10 lines · Mika Westerberg

Kai-Heng Feng reported that it takes long time (>1s) to resume Thunderbolt connected PCIe devices from both runtime suspend and system sleep (s2idle).

These PCIe downstream ports the second link capability (PCI_EXP_LNKCAP2) announces support for speeds > 5 GT/s but it is then capped by the second link control (PCI_EXP_LNKCTL2) register to 2.5 GT/s. This possiblity was not considered in pci_bridge_wait_for_secondary_bus() so it ended up waiting for 1100 ms as these ports do not support active link layer reporting either.

Quoted 3 lines

I don't think PCI_EXP_LNKCTL2 is relevant here. I think the lack of Data Link Layer Link Active is just a chip erratum. Is that documented anywhere?

I think it is relevant because if you hard-code (hardware) LNKCTL2 to always target 2.5GT/s then it effectively does not need to implement data link layer active because the link speed never goes higher than that.

I don't think these devices have public documentation.

Quoted 60 lines
Quoted 48 lines

PCIe spec 5.0 section 6.6.1 mandates that we must wait minimum of 100 ms before sending configuration request to the device below, if the port does not support speeds > 5 GT/s, and if it does we first need to wait for the data link layer to become active before waiting for that 100 ms.

PCIe spec 5.0 section 7.5.3.6 further says that all downstream ports that support speeds > 5 GT/s must support active link layer reporting so instead of looking for the speed we can check for the active link layer reporting capability and determine how to wait based on that (as they go hand in hand).

While there restructure the code a bit so that the delay is always issued in pci_bridge_wait_for_secondary_bus() by passing value of 0 to pcie_wait_for_link_delay().

---
v2: Restructured the code a bit so that it should be more readable now

Previous version can be found here:

https://lore.kernel.org/linux-pci/[email protected]/

@Kai-heng, since this patch is slightly different than what you tried, I wonder if you could check that it still solves the and does not break anything? I tested it myself but it would be nice to get your Tested-by to make sure it still works.

 drivers/pci/pci.c | 23 ++++++++++++++---------
 1 file changed, 14 insertions(+), 9 deletions(-)
drivers/pci/pci.c +2 -1
index 595fcf59843f..590c73dc7e0d 100644
@@ -4660,7 +4660,8 @@ static int pci_pm_reset(struct pci_dev *dev, int probe)
  * pcie_wait_for_link_delay - Wait until link is active or inactive
  * @pdev: Bridge device
  * @active: waiting for active or inactive?
- * @delay: Delay to wait after link has become active (in ms)
+ * @delay: Delay to wait after link has become active (in ms). Specify %0
+ *	   for no delay.
  *
  * Use this to wait till link becomes active or inactive.
  */

drivers/pci/pci.c static bool pcie_wait_for_link_delay(struct pci_dev *pdev, bool active, · line 4704

Quoted 3 lines
drivers/pci/pci.c +0 -0
@@ -4701,7 +4702,7 @@ static bool pcie_wait_for_link_delay(struct pci_dev *pdev, bool active,
 		msleep(10);
 		timeout -= 10;
 	}

I think maybe the code above (not included in the context here) should say:

  if (!pdev->link_active_reporting) {
    msleep(timeout + delay);
    return true;
  }
Quoted 4 lines

to match the rest of 4827d63891b6 ("PCI/PM: Add pcie_wait_for_link_delay()"). What do you think? If you agree, I'd make that a separate patch because it's not related to the current fix.

Yes, I agree.

Quoted 25 lines
Quoted 25 lines
drivers/pci/pci.c +10 -8
-	if (active && ret)
+	if (active && ret && delay)
 		msleep(delay);
 	else if (ret != active)
 		pci_info(pdev, "Data Link Layer Link Active not %s in 1000 msec\n",
@@ -4822,17 +4823,21 @@ void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev)
 	if (!pcie_downstream_port(dev))
 		return;
 
-	if (pcie_get_speed_cap(dev) <= PCIE_SPEED_5_0GT) {
-		pci_dbg(dev, "waiting %d ms for downstream link\n", delay);
-		msleep(delay);
-	} else {
-		pci_dbg(dev, "waiting %d ms for downstream link, after activation\n",
-			delay);
-		if (!pcie_wait_for_link_delay(dev, true, delay)) {
+	/*
+	 * Since PCIe spec mandates that all downstream ports that support
+	 * speeds greater than 5 GT/s must support data link layer active
+	 * reporting so if it is supported we poll for the link to become
+	 * active before issuing the mandatory delay.
+	 */
+	if (dev->link_active_reporting) {
+		pci_dbg(dev, "waiting for link to train\n");
+		if (!pcie_wait_for_link_delay(dev, true, 0)) {
Quoted 8 lines

drivers/pci/pci.c void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev) · line 4840

Quoted 6 lines
drivers/pci/pci.c +2 -0
 			/* Did not train, no need to wait any further */
 			return;
 		}
 	}
+	pci_dbg(child, "waiting %d ms to become accessible\n", delay);
+	msleep(delay);

Nice solution, I like this a lot.

Thanks :)

#3 lore raw
Quoted 13 lines · Mika Westerberg
Quoted 12 lines · Bjorn Helgaas
Quoted 10 lines · Mika Westerberg

Kai-Heng Feng reported that it takes long time (>1s) to resume Thunderbolt connected PCIe devices from both runtime suspend and system sleep (s2idle).

These PCIe downstream ports the second link capability (PCI_EXP_LNKCAP2) announces support for speeds > 5 GT/s but it is then capped by the second link control (PCI_EXP_LNKCTL2) register to 2.5 GT/s. This possiblity was not considered in pci_bridge_wait_for_secondary_bus() so it ended up waiting for 1100 ms as these ports do not support active link layer reporting either.

Quoted 8 lines
Quoted 3 lines

I don't think PCI_EXP_LNKCTL2 is relevant here. I think the lack of Data Link Layer Link Active is just a chip erratum. Is that documented anywhere?

I think it is relevant because if you hard-code (hardware) LNKCTL2 to always target 2.5GT/s then it effectively does not need to implement data link layer active because the link speed never goes higher than that.

I don't think it's reasonable to expect software to check Link Capabilities 2, then try to write Link Control 2 and figure out whether the target speed is hard-wired. I think these devices are just broken (at least per spec).

Quoted 13 lines
Quoted 13 lines

drivers/pci/pci.c static bool pcie_wait_for_link_delay(struct pci_dev *pdev, bool active, · line 4704

Quoted 4 lines
drivers/pci/pci.c +0 -0
@@ -4701,7 +4702,7 @@ static bool pcie_wait_for_link_delay(struct pci_dev *pdev, bool active,
 		msleep(10);
 		timeout -= 10;
 	}

I think maybe the code above (not included in the context here) should say:

  if (!pdev->link_active_reporting) {
    msleep(timeout + delay);
    return true;
  }
Quoted 6 lines
Quoted 4 lines

to match the rest of 4827d63891b6 ("PCI/PM: Add pcie_wait_for_link_delay()"). What do you think? If you agree, I'd make that a separate patch because it's not related to the current fix.

Yes, I agree.

OK, I added a patch to do this and applied both to pci/pm for v5.8. Thanks!

Bjorn

#4 lore raw
Quoted 23 lines · Bjorn Helgaas
Quoted 13 lines · Mika Westerberg
Quoted 12 lines · Bjorn Helgaas
Quoted 10 lines · Mika Westerberg

Kai-Heng Feng reported that it takes long time (>1s) to resume Thunderbolt connected PCIe devices from both runtime suspend and system sleep (s2idle).

These PCIe downstream ports the second link capability (PCI_EXP_LNKCAP2) announces support for speeds > 5 GT/s but it is then capped by the second link control (PCI_EXP_LNKCTL2) register to 2.5 GT/s. This possiblity was not considered in pci_bridge_wait_for_secondary_bus() so it ended up waiting for 1100 ms as these ports do not support active link layer reporting either.

Quoted 8 lines
Quoted 3 lines

I don't think PCI_EXP_LNKCTL2 is relevant here. I think the lack of Data Link Layer Link Active is just a chip erratum. Is that documented anywhere?

I think it is relevant because if you hard-code (hardware) LNKCTL2 to always target 2.5GT/s then it effectively does not need to implement data link layer active because the link speed never goes higher than that.

Quoted 4 lines

I don't think it's reasonable to expect software to check Link Capabilities 2, then try to write Link Control 2 and figure out whether the target speed is hard-wired. I think these devices are just broken (at least per spec).

Software does not need to figure that out. It needs to check this field if it needs to know the "actual" supported link speed.

Spec specifically allows this:

The default value of this field is the highest Link speed supported by the component (as reported in the Max Link Speed field of the Link Capabilities Register) unless the corresponding platform/form factor requires a different default value.

and it even allows hardwiring this:

Components that support only the 2.5 GT/s speed are permitted to hardwire this field to 0000b.

Tested-by on v2 1/1
#5 lore raw
In reply to Mika Westerberg
Quoted 26 lines · Mika Westerberg

Kai-Heng Feng reported that it takes long time (>1s) to resume Thunderbolt connected PCIe devices from both runtime suspend and system sleep (s2idle).

These PCIe downstream ports the second link capability (PCI_EXP_LNKCAP2) announces support for speeds > 5 GT/s but it is then capped by the second link control (PCI_EXP_LNKCTL2) register to 2.5 GT/s. This possiblity was not considered in pci_bridge_wait_for_secondary_bus() so it ended up waiting for 1100 ms as these ports do not support active link layer reporting either.

PCIe spec 5.0 section 6.6.1 mandates that we must wait minimum of 100 ms before sending configuration request to the device below, if the port does not support speeds > 5 GT/s, and if it does we first need to wait for the data link layer to become active before waiting for that 100 ms.

PCIe spec 5.0 section 7.5.3.6 further says that all downstream ports that support speeds > 5 GT/s must support active link layer reporting so instead of looking for the speed we can check for the active link layer reporting capability and determine how to wait based on that (as they go hand in hand).

Quoted 7 lines

While there restructure the code a bit so that the delay is always issued in pci_bridge_wait_for_secondary_bus() by passing value of 0 to pcie_wait_for_link_delay().

Thanks!

Quoted 70 lines

---
v2: Restructured the code a bit so that it should be more readable now

Previous version can be found here:

https://lore.kernel.org/linux-pci/[email protected]/

@Kai-heng, since this patch is slightly different than what you tried, I wonder if you could check that it still solves the and does not break anything? I tested it myself but it would be nice to get your Tested-by to make sure it still works.

drivers/pci/pci.c | 23 ++++++++++++++---------

1 file changed, 14 insertions(+), 9 deletions(-)
drivers/pci/pci.c +14 -9
index 595fcf59843f..590c73dc7e0d 100644
@@ -4660,7 +4660,8 @@ static int pci_pm_reset(struct pci_dev *dev, int probe)
 * pcie_wait_for_link_delay - Wait until link is active or inactive
 * @pdev: Bridge device
 * @active: waiting for active or inactive?
- * @delay: Delay to wait after link has become active (in ms)
+ * @delay: Delay to wait after link has become active (in ms). Specify %0
+ *	   for no delay.
 *
 * Use this to wait till link becomes active or inactive.
 */
@@ -4701,7 +4702,7 @@ static bool pcie_wait_for_link_delay(struct pci_dev *pdev, bool active,
 		msleep(10);
 		timeout -= 10;
 	}
-	if (active && ret)
+	if (active && ret && delay)
 		msleep(delay);
 	else if (ret != active)
 		pci_info(pdev, "Data Link Layer Link Active not %s in 1000 msec\n",
@@ -4822,17 +4823,21 @@ void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev)
 	if (!pcie_downstream_port(dev))
 		return;
 
-	if (pcie_get_speed_cap(dev) <= PCIE_SPEED_5_0GT) {
-		pci_dbg(dev, "waiting %d ms for downstream link\n", delay);
-		msleep(delay);
-	} else {
-		pci_dbg(dev, "waiting %d ms for downstream link, after activation\n",
-			delay);
-		if (!pcie_wait_for_link_delay(dev, true, delay)) {
+	/*
+	 * Since PCIe spec mandates that all downstream ports that support
+	 * speeds greater than 5 GT/s must support data link layer active
+	 * reporting so if it is supported we poll for the link to become
+	 * active before issuing the mandatory delay.
+	 */
+	if (dev->link_active_reporting) {
+		pci_dbg(dev, "waiting for link to train\n");
+		if (!pcie_wait_for_link_delay(dev, true, 0)) {
 			/* Did not train, no need to wait any further */
 			return;
 		}
 	}
+	pci_dbg(child, "waiting %d ms to become accessible\n", delay);
+	msleep(delay);
 
 	if (!pci_device_is_present(child)) {
 		pci_dbg(child, "waiting additional %d ms to become accessible\n", delay);
Signature · 1 line
2.26.2