PCI: Do not use pcie_get_speed_cap() to determine when to start waiting
PATCH v2 1/1 PCI: Do not use pcie_get_speed_cap() to determine when to start waiting
Kai-Heng Feng reported that it takes long time (>1s) to resume Thunderbolt connected PCIe devices from both runtime suspend and system sleep (s2idle).
These PCIe downstream ports the second link capability (PCI_EXP_LNKCAP2) announces support for speeds > 5 GT/s but it is then capped by the second link control (PCI_EXP_LNKCTL2) register to 2.5 GT/s. This possiblity was not considered in pci_bridge_wait_for_secondary_bus() so it ended up waiting for 1100 ms as these ports do not support active link layer reporting either.
PCIe spec 5.0 section 6.6.1 mandates that we must wait minimum of 100 ms before sending configuration request to the device below, if the port does not support speeds > 5 GT/s, and if it does we first need to wait for the data link layer to become active before waiting for that 100 ms.
PCIe spec 5.0 section 7.5.3.6 further says that all downstream ports that support speeds > 5 GT/s must support active link layer reporting so instead of looking for the speed we can check for the active link layer reporting capability and determine how to wait based on that (as they go hand in hand).
While there restructure the code a bit so that the delay is always issued in pci_bridge_wait_for_secondary_bus() by passing value of 0 to pcie_wait_for_link_delay().
---
v2: Restructured the code a bit so that it should be more readable now
Previous version can be found here:
https://lore.kernel.org/linux-pci/[email protected]/
@Kai-heng, since this patch is slightly different than what you tried, I wonder if you could check that it still solves the and does not break anything? I tested it myself but it would be nice to get your Tested-by to make sure it still works.
drivers/pci/pci.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-)
drivers/pci/pci.c +14 -9
@@ -4660,7 +4660,8 @@ static int pci_pm_reset(struct pci_dev *dev, int probe)* pcie_wait_for_link_delay - Wait until link is active or inactive * @pdev: Bridge device * @active: waiting for active or inactive? -* @delay: Delay to wait after link has become active (in ms) +* @delay: Delay to wait after link has become active (in ms). Specify %0 +* for no delay. * * Use this to wait till link becomes active or inactive. */
@@ -4701,7 +4702,7 @@ static bool pcie_wait_for_link_delay(struct pci_dev *pdev, bool active,msleep (10 ); timeout -=10 ; } -if (active && ret) +if (active && ret && delay)msleep (delay);else if (ret != active) pci_info(pdev, "Data Link Layer Link Active not %s in 1000 msec\n",
@@ -4822,17 +4823,21 @@ void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev)if (!pcie_downstream_port (dev))return ; -if (pcie_get_speed_cap (dev) <=PCIE_SPEED_5_0GT ) { -pci_dbg (dev,"waiting %d ms for downstream link\n" , delay); -msleep (delay); - }else { -pci_dbg (dev,"waiting %d ms for downstream link, after activation\n" , - delay); -if (!pcie_wait_for_link_delay (dev, true, delay)) { +/* +* Since PCIe spec mandates that all downstream ports that support +* speeds greater than 5 GT/s must support data link layer active +* reporting so if it is supported we poll for the link to become +* active before issuing the mandatory delay. +*/ +if (dev->link_active_reporting) { +pci_dbg (dev,"waiting for link to train\n" ); +if (!pcie_wait_for_link_delay (dev, true,0 )) {/* Did not train, no need to wait any further */ return ; } } +pci_dbg (child,"waiting %d ms to become accessible\n" , delay); +msleep (delay);if (!pci_device_is_present (child)) {pci_dbg (child,"waiting additional %d ms to become accessible\n" , delay);
Signature · 1 line
2.26.2
Quoted 10 lines · Mika Westerberg
Kai-Heng Feng reported that it takes long time (>1s) to resume Thunderbolt connected PCIe devices from both runtime suspend and system sleep (s2idle).
These PCIe downstream ports the second link capability (
PCI_EXP_LNKCAP2) announces support for speeds > 5 GT/s but it is then capped by the second link control (PCI_EXP_LNKCTL2) register to 2.5 GT/s. This possiblity was not considered inpci_bridge_wait_for_secondary_bus()so it ended up waiting for 1100 ms as these ports do not support active link layer reporting either.
I don't think PCI_EXP_LNKCTL2 is relevant here. I think the lack of Data Link Layer Link Active is just a chip erratum. Is that documented anywhere?
Quoted 48 lines
PCIe spec 5.0 section 6.6.1 mandates that we must wait minimum of 100 ms before sending configuration request to the device below, if the port does not support speeds > 5 GT/s, and if it does we first need to wait for the data link layer to become active before waiting for that 100 ms.
PCIe spec 5.0 section 7.5.3.6 further says that all downstream ports that support speeds > 5 GT/s must support active link layer reporting so instead of looking for the speed we can check for the active link layer reporting capability and determine how to wait based on that (as they go hand in hand).
While there restructure the code a bit so that the delay is always issued in
pci_bridge_wait_for_secondary_bus()by passing value of 0 topcie_wait_for_link_delay().Reported-by Kai-Heng Feng <[email protected]>Signed-off-by Mika Westerberg <[email protected]>---
v2: Restructured the code a bit so that it should be more readable nowPrevious version can be found here:
https://lore.kernel.org/linux-pci/[email protected]/
@Kai-heng, since this patch is slightly different than what you tried, I wonder if you could check that it still solves the and does not break anything? I tested it myself but it would be nice to get your Tested-by to make sure it still works.
drivers/pci/pci.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-)drivers/pci/pci.c +2
-1@@ -4660,7 +4660,8 @@ static int pci_pm_reset(struct pci_dev *dev, int probe)* pcie_wait_for_link_delay - Wait until link is active or inactive * @pdev: Bridge device * @active: waiting for active or inactive? -* @delay: Delay to wait after link has become active (in ms) +* @delay: Delay to wait after link has become active (in ms). Specify %0 +* for no delay. * * Use this to wait till link becomes active or inactive. */
drivers/pci/pci.c static bool pcie_wait_for_link_delay(struct pci_dev *pdev, bool active, · line 4704
Quoted 3 lines
drivers/pci/pci.c +0
-0@@ -4701,7 +4702,7 @@ static bool pcie_wait_for_link_delay(struct pci_dev *pdev, bool active,msleep (10 ); timeout -=10 ; }
I think maybe the code above (not included in the context here) should say:
if (!pdev->link_active_reporting) {
msleep(timeout + delay);
return true;
}to match the rest of 4827d63891b6 ("PCI/PM: Add pcie_wait_for_link_delay()"). What do you think? If you agree, I'd make that a separate patch because it's not related to the current fix.
Quoted 21 lines
drivers/pci/pci.c +6
-8-if (active && ret) +if (active && ret && delay)msleep (delay);else if (ret != active) pci_info(pdev, "Data Link Layer Link Active not %s in 1000 msec\n",@@ -4822,17 +4823,21 @@ void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev)if (!pcie_downstream_port (dev))return ; -if (pcie_get_speed_cap (dev) <=PCIE_SPEED_5_0GT ) { -pci_dbg (dev,"waiting %d ms for downstream link\n" , delay); -msleep (delay); - }else { -pci_dbg (dev,"waiting %d ms for downstream link, after activation\n" , - delay); -if (!pcie_wait_for_link_delay (dev, true, delay)) { +/* +* Since PCIe spec mandates that all downstream ports that support +* speeds greater than 5 GT/s must support data link layer active +* reporting so if it is supported we poll for the link to become +* active before issuing the mandatory delay.
drivers/pci/pci.c void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev) · line 4840
Quoted 10 lines
drivers/pci/pci.c +6
-0+*/ +if (dev->link_active_reporting) { +pci_dbg (dev,"waiting for link to train\n" ); +if (!pcie_wait_for_link_delay (dev, true,0 )) {/* Did not train, no need to wait any further */ return ; } } +pci_dbg (child,"waiting %d ms to become accessible\n" , delay); +msleep (delay);
Nice solution, I like this a lot.
Quoted 5 lines
drivers/pci/pci.c +0
-0if (!pci_device_is_present (child)) {pci_dbg (child,"waiting additional %d ms to become accessible\n" , delay);Signature · 1 line
2.26.2
Quoted 26 lines · Mika Westerberg
Kai-Heng Feng reported that it takes long time (>1s) to resume Thunderbolt connected PCIe devices from both runtime suspend and system sleep (s2idle).
These PCIe downstream ports the second link capability (
PCI_EXP_LNKCAP2) announces support for speeds > 5 GT/s but it is then capped by the second link control (PCI_EXP_LNKCTL2) register to 2.5 GT/s. This possiblity was not considered inpci_bridge_wait_for_secondary_bus()so it ended up waiting for 1100 ms as these ports do not support active link layer reporting either.PCIe spec 5.0 section 6.6.1 mandates that we must wait minimum of 100 ms before sending configuration request to the device below, if the port does not support speeds > 5 GT/s, and if it does we first need to wait for the data link layer to become active before waiting for that 100 ms.
PCIe spec 5.0 section 7.5.3.6 further says that all downstream ports that support speeds > 5 GT/s must support active link layer reporting so instead of looking for the speed we can check for the active link layer reporting capability and determine how to wait based on that (as they go hand in hand).
Quoted 7 lines
While there restructure the code a bit so that the delay is always issued in
pci_bridge_wait_for_secondary_bus()by passing value of 0 topcie_wait_for_link_delay().Reported-by Kai-Heng Feng <[email protected]>Signed-off-by Mika Westerberg <[email protected]>
Thanks!
Quoted 70 lines
---
v2: Restructured the code a bit so that it should be more readable nowPrevious version can be found here:
https://lore.kernel.org/linux-pci/[email protected]/
@Kai-heng, since this patch is slightly different than what you tried, I wonder if you could check that it still solves the and does not break anything? I tested it myself but it would be nice to get your Tested-by to make sure it still works.
drivers/pci/pci.c| 23 ++++++++++++++---------1 file changed, 14 insertions(+), 9 deletions(-)drivers/pci/pci.c +14
-9@@ -4660,7 +4660,8 @@ static int pci_pm_reset(struct pci_dev *dev, int probe)* pcie_wait_for_link_delay - Wait until link is active or inactive * @pdev: Bridge device * @active: waiting for active or inactive? -* @delay: Delay to wait after link has become active (in ms) +* @delay: Delay to wait after link has become active (in ms). Specify %0 +* for no delay. * * Use this to wait till link becomes active or inactive. */ @@ -4701,7 +4702,7 @@ static bool pcie_wait_for_link_delay(struct pci_dev *pdev, bool active,msleep (10 ); timeout -=10 ; } -if (active && ret) +if (active && ret && delay)msleep (delay);else if (ret != active) pci_info(pdev, "Data Link Layer Link Active not %s in 1000 msec\n",@@ -4822,17 +4823,21 @@ void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev)if (!pcie_downstream_port (dev))return ; -if (pcie_get_speed_cap (dev) <=PCIE_SPEED_5_0GT ) { -pci_dbg (dev,"waiting %d ms for downstream link\n" , delay); -msleep (delay); - }else { -pci_dbg (dev,"waiting %d ms for downstream link, after activation\n" , - delay); -if (!pcie_wait_for_link_delay (dev, true, delay)) { +/* +* Since PCIe spec mandates that all downstream ports that support +* speeds greater than 5 GT/s must support data link layer active +* reporting so if it is supported we poll for the link to become +* active before issuing the mandatory delay. +*/ +if (dev->link_active_reporting) { +pci_dbg (dev,"waiting for link to train\n" ); +if (!pcie_wait_for_link_delay (dev, true,0 )) {/* Did not train, no need to wait any further */ return ; } } +pci_dbg (child,"waiting %d ms to become accessible\n" , delay); +msleep (delay);if (!pci_device_is_present (child)) {pci_dbg (child,"waiting additional %d ms to become accessible\n" , delay);Signature · 1 line
2.26.2
Quoted 12 lines · Bjorn Helgaas
Quoted 10 lines · Mika Westerberg
Kai-Heng Feng reported that it takes long time (>1s) to resume Thunderbolt connected PCIe devices from both runtime suspend and system sleep (s2idle).
These PCIe downstream ports the second link capability (
PCI_EXP_LNKCAP2) announces support for speeds > 5 GT/s but it is then capped by the second link control (PCI_EXP_LNKCTL2) register to 2.5 GT/s. This possiblity was not considered inpci_bridge_wait_for_secondary_bus()so it ended up waiting for 1100 ms as these ports do not support active link layer reporting either.
Quoted 3 lines
I don't think
PCI_EXP_LNKCTL2is relevant here. I think the lack of Data Link Layer Link Active is just a chip erratum. Is that documented anywhere?
I think it is relevant because if you hard-code (hardware) LNKCTL2 to always target 2.5GT/s then it effectively does not need to implement data link layer active because the link speed never goes higher than that.
I don't think these devices have public documentation.
Quoted 60 lines
Quoted 48 lines
PCIe spec 5.0 section 6.6.1 mandates that we must wait minimum of 100 ms before sending configuration request to the device below, if the port does not support speeds > 5 GT/s, and if it does we first need to wait for the data link layer to become active before waiting for that 100 ms.
PCIe spec 5.0 section 7.5.3.6 further says that all downstream ports that support speeds > 5 GT/s must support active link layer reporting so instead of looking for the speed we can check for the active link layer reporting capability and determine how to wait based on that (as they go hand in hand).
While there restructure the code a bit so that the delay is always issued in
pci_bridge_wait_for_secondary_bus()by passing value of 0 topcie_wait_for_link_delay().Reported-by Kai-Heng Feng <[email protected]>Signed-off-by Mika Westerberg <[email protected]>---
v2: Restructured the code a bit so that it should be more readable nowPrevious version can be found here:
https://lore.kernel.org/linux-pci/[email protected]/
@Kai-heng, since this patch is slightly different than what you tried, I wonder if you could check that it still solves the and does not break anything? I tested it myself but it would be nice to get your Tested-by to make sure it still works.
drivers/pci/pci.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-)drivers/pci/pci.c +2
-1@@ -4660,7 +4660,8 @@ static int pci_pm_reset(struct pci_dev *dev, int probe)* pcie_wait_for_link_delay - Wait until link is active or inactive * @pdev: Bridge device * @active: waiting for active or inactive? -* @delay: Delay to wait after link has become active (in ms) +* @delay: Delay to wait after link has become active (in ms). Specify %0 +* for no delay. * * Use this to wait till link becomes active or inactive. */ drivers/pci/pci.c static bool pcie_wait_for_link_delay(struct pci_dev *pdev, bool active, · line 4704
Quoted 3 lines
drivers/pci/pci.c +0
-0@@ -4701,7 +4702,7 @@ static bool pcie_wait_for_link_delay(struct pci_dev *pdev, bool active,msleep (10 ); timeout -=10 ; }I think maybe the code above (not included in the context here) should say:
if (!pdev->link_active_reporting) { msleep(timeout + delay); return true; }
Quoted 4 lines
to match the rest of
4827d63891b6("PCI/PM: Addpcie_wait_for_link_delay()"). What do you think? If you agree, I'd make that a separate patch because it's not related to the current fix.
Yes, I agree.
Quoted 25 lines
Quoted 25 lines
drivers/pci/pci.c +10
-8-if (active && ret) +if (active && ret && delay)msleep (delay);else if (ret != active) pci_info(pdev, "Data Link Layer Link Active not %s in 1000 msec\n",@@ -4822,17 +4823,21 @@ void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev)if (!pcie_downstream_port (dev))return ; -if (pcie_get_speed_cap (dev) <=PCIE_SPEED_5_0GT ) { -pci_dbg (dev,"waiting %d ms for downstream link\n" , delay); -msleep (delay); - }else { -pci_dbg (dev,"waiting %d ms for downstream link, after activation\n" , - delay); -if (!pcie_wait_for_link_delay (dev, true, delay)) { +/* +* Since PCIe spec mandates that all downstream ports that support +* speeds greater than 5 GT/s must support data link layer active +* reporting so if it is supported we poll for the link to become +* active before issuing the mandatory delay. +*/ +if (dev->link_active_reporting) { +pci_dbg (dev,"waiting for link to train\n" ); +if (!pcie_wait_for_link_delay (dev, true,0 )) {
Quoted 8 lines
drivers/pci/pci.c void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev) · line 4840
Quoted 6 lines
drivers/pci/pci.c +2
-0/* Did not train, no need to wait any further */ return ; } } +pci_dbg (child,"waiting %d ms to become accessible\n" , delay); +msleep (delay);Nice solution, I like this a lot.
Thanks :)
Quoted 13 lines · Mika Westerberg
Quoted 12 lines · Bjorn Helgaas
Quoted 10 lines · Mika Westerberg
Kai-Heng Feng reported that it takes long time (>1s) to resume Thunderbolt connected PCIe devices from both runtime suspend and system sleep (s2idle).
These PCIe downstream ports the second link capability (
PCI_EXP_LNKCAP2) announces support for speeds > 5 GT/s but it is then capped by the second link control (PCI_EXP_LNKCTL2) register to 2.5 GT/s. This possiblity was not considered inpci_bridge_wait_for_secondary_bus()so it ended up waiting for 1100 ms as these ports do not support active link layer reporting either.
Quoted 8 lines
Quoted 3 lines
I don't think
PCI_EXP_LNKCTL2is relevant here. I think the lack of Data Link Layer Link Active is just a chip erratum. Is that documented anywhere?I think it is relevant because if you hard-code (hardware) LNKCTL2 to always target 2.5GT/s then it effectively does not need to implement data link layer active because the link speed never goes higher than that.
I don't think it's reasonable to expect software to check Link Capabilities 2, then try to write Link Control 2 and figure out whether the target speed is hard-wired. I think these devices are just broken (at least per spec).
Quoted 13 lines
Quoted 13 lines
drivers/pci/pci.c static bool pcie_wait_for_link_delay(struct pci_dev *pdev, bool active, · line 4704
Quoted 4 lines
drivers/pci/pci.c +0
-0@@ -4701,7 +4702,7 @@ static bool pcie_wait_for_link_delay(struct pci_dev *pdev, bool active,msleep (10 ); timeout -=10 ; }I think maybe the code above (not included in the context here) should say:
if (!pdev->link_active_reporting) { msleep(timeout + delay); return true; }
Quoted 6 lines
Quoted 4 lines
to match the rest of
4827d63891b6("PCI/PM: Addpcie_wait_for_link_delay()"). What do you think? If you agree, I'd make that a separate patch because it's not related to the current fix.Yes, I agree.
OK, I added a patch to do this and applied both to pci/pm for v5.8. Thanks!
Bjorn
Quoted 23 lines · Bjorn Helgaas
Quoted 13 lines · Mika Westerberg
Quoted 12 lines · Bjorn Helgaas
Quoted 10 lines · Mika Westerberg
Kai-Heng Feng reported that it takes long time (>1s) to resume Thunderbolt connected PCIe devices from both runtime suspend and system sleep (s2idle).
These PCIe downstream ports the second link capability (
PCI_EXP_LNKCAP2) announces support for speeds > 5 GT/s but it is then capped by the second link control (PCI_EXP_LNKCTL2) register to 2.5 GT/s. This possiblity was not considered inpci_bridge_wait_for_secondary_bus()so it ended up waiting for 1100 ms as these ports do not support active link layer reporting either.Quoted 8 lines
Quoted 3 lines
I don't think
PCI_EXP_LNKCTL2is relevant here. I think the lack of Data Link Layer Link Active is just a chip erratum. Is that documented anywhere?I think it is relevant because if you hard-code (hardware) LNKCTL2 to always target 2.5GT/s then it effectively does not need to implement data link layer active because the link speed never goes higher than that.
Quoted 4 lines
I don't think it's reasonable to expect software to check Link Capabilities 2, then try to write Link Control 2 and figure out whether the target speed is hard-wired. I think these devices are just broken (at least per spec).
Software does not need to figure that out. It needs to check this field if it needs to know the "actual" supported link speed.
Spec specifically allows this:
The default value of this field is the highest Link speed supported by the component (as reported in the Max Link Speed field of the Link Capabilities Register) unless the corresponding platform/form factor requires a different default value.
and it even allows hardwiring this:
Components that support only the 2.5 GT/s speed are permitted to hardwire this field to 0000b.