Skip to contentSkip to conversation

Wait for device readiness after D3hot -> D0uninitialized transition

PATCH Applied Bjorn Helgaas 3 participants 8 messages May 18 – May 22 PCI

lore
Share & tools
lore
#0 lore raw

v2 cover Wait for device readiness after D3hot -> D0uninitialized transition

In pci_power_up(), wait for device to become ready after a soft reset (D3hot->D0uninitialized transition). We already wait 10 ms, but even after that, the device is permitted to respond to config requests with Request Retry Status (RRS) completion status. Use pci_dev_wait() to wait for successful completion.

Bjorn Helgaas (2):
PCI: Log device readiness timeouts as errors
PCI: Wait for device readiness after D3hot -> D0uninitialized
transition

 drivers/pci/pci.c | 28 ++++++++++++++++++++++++----
 1 file changed, 24 insertions(+), 4 deletions(-)

Changes since v1:
- Log device readiness timeouts as errors, not warnings
- On timeout, set dev->current_state to D3cold and return -EIO

v1: https://lore.kernel.org/all/[email protected]/

#1 lore raw

PATCH v2 1/2 PCI: Log device readiness timeouts as errors

pci_dev_wait() waits for a device to be Configuration-Ready after a reset, such as a Function-Level Reset (FLR), a soft reset during a D3hot-> D0uninitialized transition when No_Soft_Reset == 0), or a power-up sequence from D3cold->D0uninitialized.

If pci_dev_wait() returns success, the device is guaranteed to respond to configuration requests with Successful Completion status. If it times out, device is completely non-responsive.

Upgrade the log level from pci_warn() to pci_err() to reflect this failure state.

---
 drivers/pci/pci.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/pci/pci.c +2 -2
index 8f7cfcc00090..5a9af0bb2c71 100644
@@ -1253,8 +1253,8 @@ static int pci_dev_wait(struct pci_dev *dev, char *reset_type, int timeout)
 		}
 
 		if (delay > timeout) {
-			pci_warn(dev, "not ready %dms after %s; giving up\n",
-				 delay - 1, reset_type);
+			pci_err(dev, "not ready %dms after %s; giving up\n",
+				delay - 1, reset_type);
 			return -ENOTTY;
 		}
 
Signature · 1 line
2.51.0
#2 lore raw

PATCH v2 2/2 PCI: Wait for device readiness after D3hot -> D0uninitialized transition

In reply to Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

For a device that advertises No_Soft_Reset == 0, a transition from D3hot to D0uninitialized is a soft reset, and the resulting internal device state is undefined.

Per PCIe r7.0, sec 2.3.1, a transition from D3hot to D0uninitialized mandates a minimum 10 ms delay before accessing the device. Following this delay, the device is permitted to respond to initial configuration requests with a Request Retry Status (RRS) completion status if it needs more time to initialize.

Call pci_dev_wait() after pci_power_up() performs a D3hot->D0uninitialized transition to ensure the device is ready to accept config accesses, as is done after the similar transition in pci_pm_reset().

If the device is already ready, this is essentially a no-op except for one additional config read.

---
 drivers/pci/pci.c | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
drivers/pci/pci.c +22 -2
index 5a9af0bb2c71..8228d2782f95 100644
@@ -1300,7 +1300,18 @@ int pci_power_up(struct pci_dev *dev)
 	bool need_restore;
 	pci_power_t state;
 	u16 pmcsr;
+	int ret;
 
+	/*
+	 * When setting power state to D0, platform_pci_set_power_state()
+	 * ensures main power is on.  If it puts the device in D0, it also
+	 * completes any required delays after the transition; if it leaves
+	 * the device in D1, D2, or D3hot, we use the PM Capability to
+	 * transition to D0.
+	 *
+	 * In all cases, the device is either Configuration-Ready or
+	 * inaccessible upon return.
+	 */
 	platform_pci_set_power_state(dev, PCI_D0);
 
 	if (!dev->pm_cap) {
@@ -1341,10 +1352,19 @@ int pci_power_up(struct pci_dev *dev)
 	pci_write_config_word(dev, dev->pm_cap + PCI_PM_CTRL, 0);
 
 	/* Mandatory transition delays; see PCI PM 1.2. */
-	if (state == PCI_D3hot)
+	if (state == PCI_D3hot) {
 		pci_dev_d3_sleep(dev);
-	else if (state == PCI_D2)
+		if (!(pmcsr & PCI_PM_CTRL_NO_SOFT_RESET)) {
+			ret = pci_dev_wait(dev, "power up D3hot->D0uninitialized",
+					   PCIE_RESET_READY_POLL_MS);
+			if (ret) {
+				dev->current_state = PCI_D3cold;
+				return -EIO;
+			}
+		}
+	} else if (state == PCI_D2) {
 		udelay(PCI_PM_D2_DELAY);
+	}
 
 end:
 	dev->current_state = PCI_D0;
Signature · 1 line
2.51.0
#3 lore raw

Re: [PATCH v2 2/2] PCI: Wait for device readiness after D3hot -> D0uninitialized transition

Quoted 49 lines · Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

For a device that advertises No_Soft_Reset == 0, a transition from D3hot to D0uninitialized is a soft reset, and the resulting internal device state is undefined.

Per PCIe r7.0, sec 2.3.1, a transition from D3hot to D0uninitialized mandates a minimum 10 ms delay before accessing the device. Following this delay, the device is permitted to respond to initial configuration requests with a Request Retry Status (RRS) completion status if it needs more time to initialize.

Call pci_dev_wait() after pci_power_up() performs a D3hot->D0uninitialized transition to ensure the device is ready to accept config accesses, as is done after the similar transition in pci_pm_reset().

If the device is already ready, this is essentially a no-op except for one additional config read.

---
 drivers/pci/pci.c | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
drivers/pci/pci.c +11 -0
index 5a9af0bb2c71..8228d2782f95 100644
@@ -1300,7 +1300,18 @@ int pci_power_up(struct pci_dev *dev)
        bool need_restore;
        pci_power_t state;
        u16 pmcsr;
+       int ret;
 
+       /*
+        * When setting power state to D0, platform_pci_set_power_state()
+        * ensures main power is on.  If it puts the device in D0, it also
+        * completes any required delays after the transition; if it leaves
+        * the device in D1, D2, or D3hot, we use the PM Capability to
+        * transition to D0.
+        *
+        * In all cases, the device is either Configuration-Ready or
+        * inaccessible upon return.
+        */
        platform_pci_set_power_state(dev, PCI_D0);
 
        if (!dev->pm_cap) {

drivers/pci/pci.c int pci_power_up(struct pci_dev *dev) · line 1357

Quoted 8 lines
drivers/pci/pci.c +2 -2
@@ -1341,10 +1352,19 @@ int pci_power_up(struct pci_dev *dev)
        pci_write_config_word(dev, dev->pm_cap + PCI_PM_CTRL, 0);
 
        /* Mandatory transition delays; see PCI PM 1.2. */
-       if (state == PCI_D3hot)
+       if (state == PCI_D3hot) {
                pci_dev_d3_sleep(dev);
-       else if (state == PCI_D2)
+               if (!(pmcsr & PCI_PM_CTRL_NO_SOFT_RESET)) {

Hmm. The pmcsr value here is from before the pci_write_config_word(), so AFAICS it should be updated after the write.

This means an additional config space access and I wonder if it can be added to pci_dev_wait()?

Quoted 14 lines
drivers/pci/pci.c +9 -1
+                       ret = pci_dev_wait(dev, "power up D3hot->D0uninitialized",
+                                          PCIE_RESET_READY_POLL_MS);
+                       if (ret) {
+                               dev->current_state = PCI_D3cold;
+                               return -EIO;
+                       }
+               }
+       } else if (state == PCI_D2) {
                udelay(PCI_PM_D2_DELAY);
+       }
 
 end:
        dev->current_state = PCI_D0;
--
#4 lore raw

Re: [PATCH v2 2/2] PCI: Wait for device readiness after D3hot -> D0uninitialized transition

Quoted 5 lines · Rafael J. Wysocki
Quoted 4 lines · Bjorn Helgaas

+++ b/drivers/pci/pci.c

drivers/pci/pci.c +0 -0
@@ -1341,10 +1352,19 @@ int pci_power_up(struct pci_dev *dev)
        pci_write_config_word(dev, dev->pm_cap + PCI_PM_CTRL, 0);
 
Quoted 9 lines

drivers/pci/pci.c int pci_power_up(struct pci_dev *dev) · line 1357

Quoted 6 lines
drivers/pci/pci.c +2 -2
        /* Mandatory transition delays; see PCI PM 1.2. */
-       if (state == PCI_D3hot)
+       if (state == PCI_D3hot) {
                pci_dev_d3_sleep(dev);
-       else if (state == PCI_D2)
+               if (!(pmcsr & PCI_PM_CTRL_NO_SOFT_RESET)) {

Hmm. The pmcsr value here is from before the pci_write_config_word(), so AFAICS it should be updated after the write.

The No_Soft_Reset bit is RO (PCIe r7.0 sec 7.5.2.2), so its value should never change when writing zero to the Power Management Control/Status Register.

Thanks,

Lukas

#5 lore raw

Re: [PATCH v2 2/2] PCI: Wait for device readiness after D3hot -> D0uninitialized transition

Quoted 17 lines · Lukas Wunner
Quoted 5 lines · Rafael J. Wysocki
Quoted 4 lines · Bjorn Helgaas

+++ b/drivers/pci/pci.c

drivers/pci/pci.c +0 -0
@@ -1341,10 +1352,19 @@ int pci_power_up(struct pci_dev *dev)
        pci_write_config_word(dev, dev->pm_cap + PCI_PM_CTRL, 0);
 
Quoted 9 lines

drivers/pci/pci.c int pci_power_up(struct pci_dev *dev) · line 1357

Quoted 6 lines
drivers/pci/pci.c +2 -2
        /* Mandatory transition delays; see PCI PM 1.2. */
-       if (state == PCI_D3hot)
+       if (state == PCI_D3hot) {
                pci_dev_d3_sleep(dev);
-       else if (state == PCI_D2)
+               if (!(pmcsr & PCI_PM_CTRL_NO_SOFT_RESET)) {

Hmm. The pmcsr value here is from before the pci_write_config_word(), so AFAICS it should be updated after the write.

Quoted 3 lines

The No_Soft_Reset bit is RO (PCIe r7.0 sec 7.5.2.2), so its value should never change when writing zero to the Power Management Control/Status Register.

Oh right, I confused it with something else.

Reviewed-by on the v2 series
#6 lore raw
In reply to Bjorn Helgaas
Quoted 16 lines · Bjorn Helgaas

In pci_power_up(), wait for device to become ready after a soft reset (D3hot->D0uninitialized transition). We already wait 10 ms, but even after that, the device is permitted to respond to config requests with Request Retry Status (RRS) completion status. Use pci_dev_wait() to wait for successful completion.

Bjorn Helgaas (2):
PCI: Log device readiness timeouts as errors
PCI: Wait for device readiness after D3hot -> D0uninitialized
transition

 drivers/pci/pci.c | 28 ++++++++++++++++++++++++----
 1 file changed, 24 insertions(+), 4 deletions(-)
Quoted 5 lines

Changes since v1:
- Log device readiness timeouts as errors, not warnings
- On timeout, set dev->current_state to D3cold and return -EIO

v1: https://lore.kernel.org/all/[email protected]/

For the series:

author Applied
#7 lore raw
In reply to Bjorn Helgaas
Quoted 15 lines · Bjorn Helgaas

In pci_power_up(), wait for device to become ready after a soft reset (D3hot->D0uninitialized transition). We already wait 10 ms, but even after that, the device is permitted to respond to config requests with Request Retry Status (RRS) completion status. Use pci_dev_wait() to wait for successful completion.

Bjorn Helgaas (2):
PCI: Log device readiness timeouts as errors
PCI: Wait for device readiness after D3hot -> D0uninitialized
transition

 drivers/pci/pci.c | 28 ++++++++++++++++++++++++----
 1 file changed, 24 insertions(+), 4 deletions(-)
Quoted 5 lines

Changes since v1:
- Log device readiness timeouts as errors, not warnings
- On timeout, set dev->current_state to D3cold and return -EIO

v1: https://lore.kernel.org/all/[email protected]/

I applied these to pci/reset for v7.2.