Skip to contentSkip to conversation

PCI: handle CRS response following Hot Reset and D3hot->D0

PATCH Superseded Sinan Kaya 3 participants 19 messages Nov 27, 2017 – Jan 2, 2018

Subsystems (3)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Sinan Kaya

author
#0 lore raw

v2 cover PCI: handle CRS response following Hot Reset and D3hot->D0

Rev 3.1 Sec 2.3.1 Request Handling Rules:
Valid reset conditions after which a device is permitted to return CRS are:
* Cold, Warm, and Hot Resets,
* FLR
* A reset initiated in response to a D3hot to D0 uninitialized

Try to reuse FLR implementation towards other reset types.

Changes from v1:
* PCI: add a return type for pci_reset_bridge_secondary_bus()
* set waittime as a global variable and have a kernel command line to
* override.
* get rid of initial sleep parameter

Sinan Kaya (7):
PCI: protect restore with device lock to be consistent
PCI: handle FLR failure and allow other reset types
PCI: make pci_flr_wait() generic and rename to pci_dev_wait()
PCI: wait device ready after pci_pm_reset()
PCI: add a return type for pci_reset_bridge_secondary_bus()
PCI: add device wait after slot and bus reset
PCI: make reset poll time adjustable

 Documentation/admin-guide/kernel-parameters.txt |  2 +
 drivers/pci/pci.c                               | 72 ++++++++++++++++---------
 include/linux/pci.h                             |  4 +-
 3 files changed, 51 insertions(+), 27 deletions(-)
Signature · 1 line
1.9.1
author
#1 lore raw

PATCH v2 1/7 PCI: protect restore with device lock to be consistent

Commit b014e96d1abb ("PCI: Protect pci_error_handlers->reset_notify() usage with device_lock()") added protection around pci_dev_restore() function so that device specific remove callback does not cause a race condition against hotplug.

pci_dev_lock() usage has been forgotten in two different places in the code. Adding locks for pci_slot_restore() and moving pci_dev_restore() inside the locks for pci_try_reset_function().

---
 drivers/pci/pci.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/pci/pci.c +3 -1
index 6078dfc..23681f4 100644
@@ -4344,9 +4344,9 @@ int pci_try_reset_function(struct pci_dev *dev)
 
 	pci_dev_save_and_disable(dev);
 	rc = __pci_reset_function_locked(dev);
+	pci_dev_restore(dev);
 	pci_dev_unlock(dev);
 
-	pci_dev_restore(dev);
 	return rc;
 }
 EXPORT_SYMBOL_GPL(pci_try_reset_function);
@@ -4546,7 +4546,9 @@ static void pci_slot_restore(struct pci_slot *slot)
 	list_for_each_entry(dev, &slot->bus->devices, bus_list) {
 		if (!dev->slot || dev->slot != slot)
 			continue;
+		pci_dev_lock(dev);
 		pci_dev_restore(dev);
+		pci_dev_unlock(dev);
 		if (dev->subordinate)
 			pci_bus_restore(dev->subordinate);
 	}
Signature · 1 line
1.9.1
Reviewed-by on v2 1/7
#2 lore raw

Re: [PATCH V2 1/7] PCI: protect restore with device lock to be consistent

Quoted 10 lines · Sinan Kaya

Commit b014e96d1abb ("PCI: Protect pci_error_handlers->reset_notify() usage with device_lock()") added protection around pci_dev_restore() function so that device specific remove callback does not cause a race condition against hotplug.

pci_dev_lock() usage has been forgotten in two different places in the code. Adding locks for pci_slot_restore() and moving pci_dev_restore() inside the locks for pci_try_reset_function().

Looks good,

author
#3 lore raw

PATCH v2 2/7 PCI: handle FLR failure and allow other reset types

In reply to Sinan Kaya

pci_flr_wait() and pci_af_flr() functions assume graceful return even though the device is inaccessible under error conditions.

Return -ENOTTY in error cases so that __pci_reset_function_locked() can try other reset types if AF_FLR/FLR reset fails.

---
 drivers/pci/pci.c   | 18 ++++++++++--------
 include/linux/pci.h |  2 +-
 2 files changed, 11 insertions(+), 9 deletions(-)
drivers/pci/pci.c +10 -8
index 23681f4..27ec45d 100644
@@ -3820,7 +3820,7 @@ int pci_wait_for_pending_transaction(struct pci_dev *dev)
 }
 EXPORT_SYMBOL(pci_wait_for_pending_transaction);
 
-static void pci_flr_wait(struct pci_dev *dev)
+static int pci_flr_wait(struct pci_dev *dev)
 {
 	int delay = 1, timeout = 60000;
 	u32 id;
@@ -3849,7 +3849,7 @@ static void pci_flr_wait(struct pci_dev *dev)
 		if (delay > timeout) {
 			dev_warn(&dev->dev, "not ready %dms after FLR; giving up\n",
 				 100 + delay - 1);
-			return;
+			return -ENOTTY;
 		}
 
 		if (delay > 1000)
@@ -3863,6 +3863,8 @@ static void pci_flr_wait(struct pci_dev *dev)
 
 	if (delay > 1000)
 		dev_info(&dev->dev, "ready %dms after FLR\n", 100 + delay - 1);
+
+	return 0;
 }
 
 /**
@@ -3891,13 +3893,13 @@ static bool pcie_has_flr(struct pci_dev *dev)
  * device supports FLR before calling this function, e.g. by using the
  * pcie_has_flr() helper.
  */
-void pcie_flr(struct pci_dev *dev)
+int pcie_flr(struct pci_dev *dev)
 {
 	if (!pci_wait_for_pending_transaction(dev))
 		dev_err(&dev->dev, "timed out waiting for pending transaction; performing function level reset anyway\n");
 
 	pcie_capability_set_word(dev, PCI_EXP_DEVCTL, PCI_EXP_DEVCTL_BCR_FLR);
-	pci_flr_wait(dev);
+	return pci_flr_wait(dev);
 }
 EXPORT_SYMBOL_GPL(pcie_flr);
 
@@ -3930,8 +3932,7 @@ static int pci_af_flr(struct pci_dev *dev, int probe)
 		dev_err(&dev->dev, "timed out waiting for pending transaction; performing AF function level reset anyway\n");
 
 	pci_write_config_byte(dev, pos + PCI_AF_CTRL, PCI_AF_CTRL_FLR);
-	pci_flr_wait(dev);
-	return 0;
+	return pci_flr_wait(dev);
 }
 
 /**
@@ -4203,8 +4204,9 @@ int __pci_reset_function_locked(struct pci_dev *dev)
 	if (rc != -ENOTTY)
 		return rc;
 	if (pcie_has_flr(dev)) {
-		pcie_flr(dev);
-		return 0;
+		rc = pcie_flr(dev);
+		if (rc != -ENOTTY)
+			return rc;
 	}
 	rc = pci_af_flr(dev, 0);
 	if (rc != -ENOTTY)
include/linux/pci.h +1 -1
index d16a7c0..354b018 100644
@@ -1089,7 +1089,7 @@ static inline int pci_is_managed(struct pci_dev *pdev)
 int pcie_set_mps(struct pci_dev *dev, int mps);
 int pcie_get_minimum_link(struct pci_dev *dev, enum pci_bus_speed *speed,
 			  enum pcie_link_width *width);
-void pcie_flr(struct pci_dev *dev);
+int pcie_flr(struct pci_dev *dev);
 int __pci_reset_function(struct pci_dev *dev);
 int __pci_reset_function_locked(struct pci_dev *dev);
 int pci_reset_function(struct pci_dev *dev);
Signature · 1 line
1.9.1
Reviewed-by on v2 2/7
#4 lore raw

Re: [PATCH V2 2/7] PCI: handle FLR failure and allow other reset types

Quoted 5 lines · Sinan Kaya

pci_flr_wait() and pci_af_flr() functions assume graceful return even though the device is inaccessible under error conditions.

Return -ENOTTY in error cases so that __pci_reset_function_locked() can try other reset types if AF_FLR/FLR reset fails.

Looks good,

author
#5 lore raw

PATCH v2 3/7 PCI: make pci_flr_wait() generic and rename to pci_dev_wait()

In reply to Sinan Kaya

Rev 3.1 Sec 2.3.1 Request Handling Rules:
Valid reset conditions after which a device is permitted to return CRS are:
* Cold, Warm, and Hot Resets,
* FLR
* A reset initiated in response to a D3hot to D0 uninitialized

Try to reuse FLR implementation towards other reset types.

---
 drivers/pci/pci.c | 47 ++++++++++++++++++++++++++++++-----------------
 1 file changed, 30 insertions(+), 17 deletions(-)
drivers/pci/pci.c +30 -17
index 27ec45d..ed3c3bc 100644
@@ -125,6 +125,9 @@ static int __init pcie_port_pm_setup(char *str)
 }
 __setup("pcie_port_pm=", pcie_port_pm_setup);
 
+/* time to wait after a reset for device to become responsive */
+#define PCIE_RESET_READY_POLL_MS 60000
+
 /**
  * pci_bus_max_busnr - returns maximum PCI bus number of given bus' children
  * @bus: pointer to PCI bus structure to search
@@ -3820,20 +3823,13 @@ int pci_wait_for_pending_transaction(struct pci_dev *dev)
 }
 EXPORT_SYMBOL(pci_wait_for_pending_transaction);
 
-static int pci_flr_wait(struct pci_dev *dev)
+static int pci_dev_wait(struct pci_dev *dev, char *reset_type, int timeout)
 {
-	int delay = 1, timeout = 60000;
+	int delay = 1;
 	u32 id;
 
 	/*
-	 * Per PCIe r3.1, sec 6.6.2, a device must complete an FLR within
-	 * 100ms, but may silently discard requests while the FLR is in
-	 * progress.  Wait 100ms before trying to access the device.
-	 */
-	msleep(100);
-
-	/*
-	 * After 100ms, the device should not silently discard config
+	 * After reset, the device should not silently discard config
 	 * requests, but it may still indicate that it needs more time by
 	 * responding to them with CRS completions.  The Root Port will
 	 * generally synthesize ~0 data to complete the read (except when
@@ -3847,14 +3843,14 @@ static int pci_flr_wait(struct pci_dev *dev)
 	pci_read_config_dword(dev, PCI_COMMAND, &id);
 	while (id == ~0) {
 		if (delay > timeout) {
-			dev_warn(&dev->dev, "not ready %dms after FLR; giving up\n",
-				 100 + delay - 1);
+			dev_warn(&dev->dev, "not ready %dms after %s; giving up\n",
+				 delay - 1, reset_type);
 			return -ENOTTY;
 		}
 
 		if (delay > 1000)
-			dev_info(&dev->dev, "not ready %dms after FLR; waiting\n",
-				 100 + delay - 1);
+			dev_info(&dev->dev, "not ready %dms after %s; waiting\n",
+				 delay - 1, reset_type);
 
 		msleep(delay);
 		delay *= 2;
@@ -3862,7 +3858,8 @@ static int pci_flr_wait(struct pci_dev *dev)
 	}
 
 	if (delay > 1000)
-		dev_info(&dev->dev, "ready %dms after FLR\n", 100 + delay - 1);
+		dev_info(&dev->dev, "ready %dms after %s\n", delay - 1,
+			 reset_type);
 
 	return 0;
 }
@@ -3899,7 +3896,15 @@ int pcie_flr(struct pci_dev *dev)
 		dev_err(&dev->dev, "timed out waiting for pending transaction; performing function level reset anyway\n");
 
 	pcie_capability_set_word(dev, PCI_EXP_DEVCTL, PCI_EXP_DEVCTL_BCR_FLR);
-	return pci_flr_wait(dev);
+
+	/*
+	 * Per PCIe r3.1, sec 6.6.2, a device must complete an FLR within
+	 * 100ms, but may silently discard requests while the FLR is in
+	 * progress.  Wait 100ms before trying to access the device.
+	 */
+	msleep(100);
+
+	return pci_dev_wait(dev, "FLR", PCIE_RESET_READY_POLL_MS);
 }
 EXPORT_SYMBOL_GPL(pcie_flr);
 
@@ -3932,7 +3937,15 @@ static int pci_af_flr(struct pci_dev *dev, int probe)
 		dev_err(&dev->dev, "timed out waiting for pending transaction; performing AF function level reset anyway\n");
 
 	pci_write_config_byte(dev, pos + PCI_AF_CTRL, PCI_AF_CTRL_FLR);
-	return pci_flr_wait(dev);
+
+	/*
+	 * Per PCIe r3.1, sec 6.6.2, a device must complete an FLR within
+	 * 100ms, but may silently discard requests while the FLR is in
+	 * progress.  Wait 100ms before trying to access the device.
+	 */
+	msleep(100);
+
+	return pci_dev_wait(dev, "AF_FLR", PCIE_RESET_READY_POLL_MS);
 }
 
 /**
Signature · 1 line
1.9.1
#6 lore raw

Re: [PATCH V2 3/7] PCI: make pci_flr_wait() generic and rename to pci_dev_wait()

Quoted 8 lines · Sinan Kaya

Rev 3.1 Sec 2.3.1 Request Handling Rules:
Valid reset conditions after which a device is permitted to return CRS are:
* Cold, Warm, and Hot Resets,
* FLR
* A reset initiated in response to a D3hot to D0 uninitialized

Try to reuse FLR implementation towards other reset types.

Should we keep a helper for the FLR wait? If not I guess the reference for AF_FLR should be the PCI spec, not the PCIE spec.

author
#7 lore raw

Re: [PATCH V2 3/7] PCI: make pci_flr_wait() generic and rename to pci_dev_wait()

Quoted 8 lines · Christoph Hellwig
Quoted 7 lines · Sinan Kaya

Rev 3.1 Sec 2.3.1 Request Handling Rules:
Valid reset conditions after which a device is permitted to return CRS are:
* Cold, Warm, and Hot Resets,
* FLR
* A reset initiated in response to a D3hot to D0 uninitialized

Quoted 5 lines
Quoted 1 line

Try to reuse FLR implementation towards other reset types.

Should we keep a helper for the FLR wait? If not I guess the reference for AF_FLR should be the PCI spec, not the PCIE spec.

Bjorn,

what is your opinion?

Sinan

Signature · 3 lines
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
#8 lore raw

Re: [PATCH V2 3/7] PCI: make pci_flr_wait() generic and rename to pci_dev_wait()

In reply to Sinan Kaya
Quoted 99 lines · Sinan Kaya

Rev 3.1 Sec 2.3.1 Request Handling Rules:
Valid reset conditions after which a device is permitted to return CRS are:
* Cold, Warm, and Hot Resets,
* FLR
* A reset initiated in response to a D3hot to D0 uninitialized

Try to reuse FLR implementation towards other reset types.

---
 drivers/pci/pci.c | 47 ++++++++++++++++++++++++++++++-----------------
 1 file changed, 30 insertions(+), 17 deletions(-)
drivers/pci/pci.c +21 -16
index 27ec45d..ed3c3bc 100644
@@ -125,6 +125,9 @@ static int __init pcie_port_pm_setup(char *str)
 }
 __setup("pcie_port_pm=", pcie_port_pm_setup);
 
+/* time to wait after a reset for device to become responsive */
+#define PCIE_RESET_READY_POLL_MS 60000
+
 /**
  * pci_bus_max_busnr - returns maximum PCI bus number of given bus' children
  * @bus: pointer to PCI bus structure to search
@@ -3820,20 +3823,13 @@ int pci_wait_for_pending_transaction(struct pci_dev *dev)
 }
 EXPORT_SYMBOL(pci_wait_for_pending_transaction);
 
-static int pci_flr_wait(struct pci_dev *dev)
+static int pci_dev_wait(struct pci_dev *dev, char *reset_type, int timeout)
 {
-	int delay = 1, timeout = 60000;
+	int delay = 1;
 	u32 id;
 
 	/*
-	 * Per PCIe r3.1, sec 6.6.2, a device must complete an FLR within
-	 * 100ms, but may silently discard requests while the FLR is in
-	 * progress.  Wait 100ms before trying to access the device.
-	 */
-	msleep(100);
-
-	/*
-	 * After 100ms, the device should not silently discard config
+	 * After reset, the device should not silently discard config
 	 * requests, but it may still indicate that it needs more time by
 	 * responding to them with CRS completions.  The Root Port will
 	 * generally synthesize ~0 data to complete the read (except when
@@ -3847,14 +3843,14 @@ static int pci_flr_wait(struct pci_dev *dev)
 	pci_read_config_dword(dev, PCI_COMMAND, &id);
 	while (id == ~0) {
 		if (delay > timeout) {
-			dev_warn(&dev->dev, "not ready %dms after FLR; giving up\n",
-				 100 + delay - 1);
+			dev_warn(&dev->dev, "not ready %dms after %s; giving up\n",
+				 delay - 1, reset_type);
 			return -ENOTTY;
 		}
 
 		if (delay > 1000)
-			dev_info(&dev->dev, "not ready %dms after FLR; waiting\n",
-				 100 + delay - 1);
+			dev_info(&dev->dev, "not ready %dms after %s; waiting\n",
+				 delay - 1, reset_type);
 
 		msleep(delay);
 		delay *= 2;
@@ -3862,7 +3858,8 @@ static int pci_flr_wait(struct pci_dev *dev)
 	}
 
 	if (delay > 1000)
-		dev_info(&dev->dev, "ready %dms after FLR\n", 100 + delay - 1);
+		dev_info(&dev->dev, "ready %dms after %s\n", delay - 1,
+			 reset_type);
 
 	return 0;
 }
@@ -3899,7 +3896,15 @@ int pcie_flr(struct pci_dev *dev)
 		dev_err(&dev->dev, "timed out waiting for pending transaction; performing function level reset anyway\n");
 
 	pcie_capability_set_word(dev, PCI_EXP_DEVCTL, PCI_EXP_DEVCTL_BCR_FLR);
-	return pci_flr_wait(dev);
+
+	/*
+	 * Per PCIe r3.1, sec 6.6.2, a device must complete an FLR within
+	 * 100ms, but may silently discard requests while the FLR is in
+	 * progress.  Wait 100ms before trying to access the device.
+	 */
+	msleep(100);
+
+	return pci_dev_wait(dev, "FLR", PCIE_RESET_READY_POLL_MS);
 }
 EXPORT_SYMBOL_GPL(pcie_flr);
 

drivers/pci/pci.c static int pci_af_flr(struct pci_dev *dev, int probe) · line 3942

Quoted 7 lines
drivers/pci/pci.c +3 -1
@@ -3932,7 +3937,15 @@ static int pci_af_flr(struct pci_dev *dev, int probe)
 		dev_err(&dev->dev, "timed out waiting for pending transaction; performing AF function level reset anyway\n");
 
 	pci_write_config_byte(dev, pos + PCI_AF_CTRL, PCI_AF_CTRL_FLR);
-	return pci_flr_wait(dev);
+
+	/*
+	 * Per PCIe r3.1, sec 6.6.2, a device must complete an FLR within

I think this should reference the "Advanced Capabilities for Conventional PCI" ECN, shouldn't it? The one I see is dated 13 April 2006, updated 27 July 2006, and I don't see a PCI spec that includes it.

drivers/pci/pci.c static int pci_af_flr(struct pci_dev *dev, int probe) · line 3948

Quoted 6 lines
drivers/pci/pci.c +6 -0
+	 * 100ms, but may silently discard requests while the FLR is in
+	 * progress.  Wait 100ms before trying to access the device.
+	 */
+	msleep(100);
+
+	return pci_dev_wait(dev, "AF_FLR", PCIE_RESET_READY_POLL_MS);

CRS is not applicable to conventional PCI. The ECN mentions waiting 100ms. I don't see anything about polling after that, but I guess it probably doesn't hurt anything.

Quoted 11 lines
drivers/pci/pci.c +0 -0
 }
 
 /**
Signature · 7 lines
1.9.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
author
#9 lore raw

Re: [PATCH V2 3/7] PCI: make pci_flr_wait() generic and rename to pci_dev_wait()

Quoted 7 lines · Bjorn Helgaas

drivers/pci/pci.c int pcie_flr(struct pci_dev *dev) · line 3901

Quoted 3 lines
drivers/pci/pci.c +3 -0
+
+	/*
+	 * Per PCIe r3.1, sec 6.6.2, a device must complete an FLR within

I think this should reference the "Advanced Capabilities for Conventional PCI" ECN, shouldn't it? The one I see is dated 13 April 2006, updated 27 July 2006, and I don't see a PCI spec that includes it.

I'll add reference to PCI spec as well. Any other comments on the series?

Quoted 10 lines

drivers/pci/pci.c static int pci_af_flr(struct pci_dev *dev, int probe) · line 3948

Quoted 7 lines
drivers/pci/pci.c +6 -0
+	 * 100ms, but may silently discard requests while the FLR is in
+	 * progress.  Wait 100ms before trying to access the device.
+	 */
+	msleep(100);
+
+	return pci_dev_wait(dev, "AF_FLR", PCIE_RESET_READY_POLL_MS);

CRS is not applicable to conventional PCI. The ECN mentions waiting 100ms. I don't see anything about polling after that, but I guess it probably doesn't hurt anything.

ok, I'll keep it as it is.

Quoted 2 lines
Quoted 2 lines

}

Signature · 3 lines
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
author
#10 lore raw

PATCH v2 4/7 PCI: wait device ready after pci_pm_reset()

In reply to Sinan Kaya

Rev 3.1 Sec 2.3.1 Request Handling Rules says a device can issue CRS following a D3hot->D0 transition. Add pci_dev_wait() call to see if device is available before returning.

---
 drivers/pci/pci.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/pci/pci.c +1 -1
index ed3c3bc..87e4688 100644
@@ -3990,7 +3990,7 @@ static int pci_pm_reset(struct pci_dev *dev, int probe)
 	pci_write_config_word(dev, dev->pm_cap + PCI_PM_CTRL, csr);
 	pci_dev_d3_sleep(dev);
 
-	return 0;
+	return pci_dev_wait(dev, "PM D3->D0", PCIE_RESET_READY_POLL_MS);
 }
 
 void pci_reset_secondary_bus(struct pci_dev *dev)
Signature · 1 line
1.9.1
author
#12 lore raw

PATCH v2 5/7 PCI: add a return type for pci_reset_bridge_secondary_bus()

In reply to Sinan Kaya

Getting ready to return an error from pci_reset_bridge_secondary_bus() when device is unreachable.

---
 drivers/pci/pci.c   | 4 +++-
 include/linux/pci.h | 2 +-
 2 files changed, 4 insertions(+), 2 deletions(-)
drivers/pci/pci.c +3 -1
index 87e4688..0a9a696 100644
@@ -4031,9 +4031,11 @@ void __weak pcibios_reset_secondary_bus(struct pci_dev *dev)
  * Use the bridge control register to assert reset on the secondary bus.
  * Devices on the secondary bus are left in power-on state.
  */
-void pci_reset_bridge_secondary_bus(struct pci_dev *dev)
+int pci_reset_bridge_secondary_bus(struct pci_dev *dev)
 {
 	pcibios_reset_secondary_bus(dev);
+
+	return 0;
 }
 EXPORT_SYMBOL_GPL(pci_reset_bridge_secondary_bus);
 
include/linux/pci.h +1 -1
index 354b018..539b3af 100644
@@ -1103,7 +1103,7 @@ int pcie_get_minimum_link(struct pci_dev *dev, enum pci_bus_speed *speed,
 int pci_try_reset_bus(struct pci_bus *bus);
 void pci_reset_secondary_bus(struct pci_dev *dev);
 void pcibios_reset_secondary_bus(struct pci_dev *dev);
-void pci_reset_bridge_secondary_bus(struct pci_dev *dev);
+int pci_reset_bridge_secondary_bus(struct pci_dev *dev);
 void pci_update_resource(struct pci_dev *dev, int resno);
 int __must_check pci_assign_resource(struct pci_dev *dev, int i);
 int __must_check pci_reassign_resource(struct pci_dev *dev, int i, resource_size_t add_size, resource_size_t align);
Signature · 1 line
1.9.1
author
#14 lore raw

PATCH v2 6/7 PCI: add device wait after slot and bus reset

In reply to Sinan Kaya

Rev 3.1 Sec 2.3.1 Request Handling Rules indicates that a device can issue CRS following secondary bus reset. Handle device presence gracefully.

---
 drivers/pci/pci.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/pci/pci.c +1 -1
index 0a9a696..8472c24 100644
@@ -4035,7 +4035,7 @@ int pci_reset_bridge_secondary_bus(struct pci_dev *dev)
 {
 	pcibios_reset_secondary_bus(dev);
 
-	return 0;
+	return pci_dev_wait(dev, "bus reset", PCIE_RESET_READY_POLL_MS);
 }
 EXPORT_SYMBOL_GPL(pci_reset_bridge_secondary_bus);
 
Signature · 1 line
1.9.1
author
#16 lore raw

PATCH v2 7/7 PCI: make reset poll time adjustable

In reply to Sinan Kaya

Introduce pci=resetpolltime= argument to override 60 seconds poll time in units of milliseconds.

---
 Documentation/admin-guide/kernel-parameters.txt |  2 ++
 drivers/pci/pci.c                               | 13 ++++++++-----
 2 files changed, 10 insertions(+), 5 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +2 -0
index 0549662..a07d4f5 100644
@@ -3071,6 +3071,8 @@
 		pcie_scan_all	Scan all possible PCIe devices.  Otherwise we
 				only look for one device below a PCIe downstream
 				port.
+		resetpolltime=	Adjusts the default poll time following hot reset
+				and D3->D0 transition.
 
 	pcie_aspm=	[PCIE] Forcibly enable or disable PCIe Active State Power
 			Management.
drivers/pci/pci.c +8 -5
index 8472c24..a6c3e25 100644
@@ -127,7 +127,7 @@ static int __init pcie_port_pm_setup(char *str)
 
 /* time to wait after a reset for device to become responsive */
 #define PCIE_RESET_READY_POLL_MS 60000
-
+static unsigned long pci_reset_polltime = PCIE_RESET_READY_POLL_MS;
 /**
  * pci_bus_max_busnr - returns maximum PCI bus number of given bus' children
  * @bus: pointer to PCI bus structure to search
@@ -3904,7 +3904,7 @@ int pcie_flr(struct pci_dev *dev)
 	 */
 	msleep(100);
 
-	return pci_dev_wait(dev, "FLR", PCIE_RESET_READY_POLL_MS);
+	return pci_dev_wait(dev, "FLR", pci_reset_polltime);
 }
 EXPORT_SYMBOL_GPL(pcie_flr);
 
@@ -3945,7 +3945,7 @@ static int pci_af_flr(struct pci_dev *dev, int probe)
 	 */
 	msleep(100);
 
-	return pci_dev_wait(dev, "AF_FLR", PCIE_RESET_READY_POLL_MS);
+	return pci_dev_wait(dev, "AF_FLR", pci_reset_polltime);
 }
 
 /**
@@ -3990,7 +3990,7 @@ static int pci_pm_reset(struct pci_dev *dev, int probe)
 	pci_write_config_word(dev, dev->pm_cap + PCI_PM_CTRL, csr);
 	pci_dev_d3_sleep(dev);
 
-	return pci_dev_wait(dev, "PM D3->D0", PCIE_RESET_READY_POLL_MS);
+	return pci_dev_wait(dev, "PM D3->D0", pci_reset_polltime);
 }
 
 void pci_reset_secondary_bus(struct pci_dev *dev)
@@ -4035,7 +4035,7 @@ int pci_reset_bridge_secondary_bus(struct pci_dev *dev)
 {
 	pcibios_reset_secondary_bus(dev);
 
-	return pci_dev_wait(dev, "bus reset", PCIE_RESET_READY_POLL_MS);
+	return pci_dev_wait(dev, "bus reset", pci_reset_polltime);
 }
 EXPORT_SYMBOL_GPL(pci_reset_bridge_secondary_bus);
 
@@ -5528,6 +5528,9 @@ static int __init pci_setup(char *str)
 				pcie_bus_config = PCIE_BUS_PEER2PEER;
 			} else if (!strncmp(str, "pcie_scan_all", 13)) {
 				pci_add_flags(PCI_SCAN_ALL_PCIE_DEVS);
+			} else if (!strncmp(str, "resetpolltime=", 14)) {
+				pci_reset_polltime =
+					simple_strtoul(str + 14, &str, 0);
 			} else {
 				printk(KERN_ERR "PCI: Unknown option `%s'\n",
 						str);
Signature · 1 line
1.9.1
#17 lore raw

Re: [PATCH V2 7/7] PCI: make reset poll time adjustable

Quoted 2 lines · Sinan Kaya

Introduce pci=resetpolltime= argument to override 60 seconds poll time in units of milliseconds.

I resist adding kernel parameters because they really complicate the user experience. Obviously you added this for a reason, but I don't know what that is. If we really need it, is there any way we could automatically figure out in the kernel when we need different poll times?

Quoted 81 lines
---
 Documentation/admin-guide/kernel-parameters.txt |  2 ++
 drivers/pci/pci.c                               | 13 ++++++++-----
 2 files changed, 10 insertions(+), 5 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +2 -0
index 0549662..a07d4f5 100644
@@ -3071,6 +3071,8 @@
 		pcie_scan_all	Scan all possible PCIe devices.  Otherwise we
 				only look for one device below a PCIe downstream
 				port.
+		resetpolltime=	Adjusts the default poll time following hot reset
+				and D3->D0 transition.
 
 	pcie_aspm=	[PCIE] Forcibly enable or disable PCIe Active State Power
 			Management.
drivers/pci/pci.c +8 -5
index 8472c24..a6c3e25 100644
@@ -127,7 +127,7 @@ static int __init pcie_port_pm_setup(char *str)
 
 /* time to wait after a reset for device to become responsive */
 #define PCIE_RESET_READY_POLL_MS 60000
-
+static unsigned long pci_reset_polltime = PCIE_RESET_READY_POLL_MS;
 /**
  * pci_bus_max_busnr - returns maximum PCI bus number of given bus' children
  * @bus: pointer to PCI bus structure to search
@@ -3904,7 +3904,7 @@ int pcie_flr(struct pci_dev *dev)
 	 */
 	msleep(100);
 
-	return pci_dev_wait(dev, "FLR", PCIE_RESET_READY_POLL_MS);
+	return pci_dev_wait(dev, "FLR", pci_reset_polltime);
 }
 EXPORT_SYMBOL_GPL(pcie_flr);
 
@@ -3945,7 +3945,7 @@ static int pci_af_flr(struct pci_dev *dev, int probe)
 	 */
 	msleep(100);
 
-	return pci_dev_wait(dev, "AF_FLR", PCIE_RESET_READY_POLL_MS);
+	return pci_dev_wait(dev, "AF_FLR", pci_reset_polltime);
 }
 
 /**
@@ -3990,7 +3990,7 @@ static int pci_pm_reset(struct pci_dev *dev, int probe)
 	pci_write_config_word(dev, dev->pm_cap + PCI_PM_CTRL, csr);
 	pci_dev_d3_sleep(dev);
 
-	return pci_dev_wait(dev, "PM D3->D0", PCIE_RESET_READY_POLL_MS);
+	return pci_dev_wait(dev, "PM D3->D0", pci_reset_polltime);
 }
 
 void pci_reset_secondary_bus(struct pci_dev *dev)
@@ -4035,7 +4035,7 @@ int pci_reset_bridge_secondary_bus(struct pci_dev *dev)
 {
 	pcibios_reset_secondary_bus(dev);
 
-	return pci_dev_wait(dev, "bus reset", PCIE_RESET_READY_POLL_MS);
+	return pci_dev_wait(dev, "bus reset", pci_reset_polltime);
 }
 EXPORT_SYMBOL_GPL(pci_reset_bridge_secondary_bus);
 
@@ -5528,6 +5528,9 @@ static int __init pci_setup(char *str)
 				pcie_bus_config = PCIE_BUS_PEER2PEER;
 			} else if (!strncmp(str, "pcie_scan_all", 13)) {
 				pci_add_flags(PCI_SCAN_ALL_PCIE_DEVS);
+			} else if (!strncmp(str, "resetpolltime=", 14)) {
+				pci_reset_polltime =
+					simple_strtoul(str + 14, &str, 0);
 			} else {
 				printk(KERN_ERR "PCI: Unknown option `%s'\n",
 						str);
Signature · 1 line
1.9.1
author
#18 lore raw

Re: [PATCH V2 7/7] PCI: make reset poll time adjustable

Quoted 8 lines · Bjorn Helgaas
Quoted 2 lines

Introduce pci=resetpolltime= argument to override 60 seconds poll time in units of milliseconds.

I resist adding kernel parameters because they really complicate the user experience. Obviously you added this for a reason, but I don't know what that is. If we really need it, is there any way we could automatically figure out in the kernel when we need different poll times?

No, we can drop this patch if we want to . This was mostly a scalability concern.

I worried adding 60 seconds would be too much for some use case. We can hold onto this change until that use case happens if you want.

Signature · 3 lines
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.