Skip to contentSkip to conversation

nvme-pci: try function level reset on init failure

PATCH Applied Keith Busch 5 participants 8 messages Jul 15, 2025 – Jul 17, 2025 NVMe

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 nvme-pci: try function level reset on init failure

From: Keith Busch <[email protected]>

NVMe devices from multiple vendors appear to get stuck in a reset state that we can't get out of with an NVMe level Controller Reset. The kernel would report these with messages that look like:

Device not ready; aborting reset, CSTS=0x1

These have historically required a power cycle to make them usable again, but in many cases, a PCIe FLR is sufficient to restart operation without a power cycle. Try it if the initial controller reset fails during any nvme reset attempt.

Cc Chaitanya Kulkarni <[email protected]>

---
v1->v2:

Added code comment explaining whe escalation

Add an informational kernel message that this event occured

Use the "pcie_reset_flr()" API instead of "pcie_flr()" since that one checks for quirks and capabilities before writing FLR config bits. Note, NVMe PCI Trasnsport Spec mandates FLR capability, so the latter should not apply to any compliant device, but you never know...

 drivers/nvme/host/pci.c | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
drivers/nvme/host/pci.c +22 -2
index 4cf87fb5d8573..f8f8cb6a4786a 100644
@@ -2067,8 +2067,28 @@ static int nvme_pci_configure_admin_queue(struct nvme_dev *dev)
 	 * might be pointing at!
 	 */
 	result = nvme_disable_ctrl(&dev->ctrl, false);
-	if (result < 0)
-		return result;
+	if (result < 0) {
+		struct pci_dev *pdev = to_pci_dev(dev->dev);
+
+		/*
+		 * The NVMe Controller Reset method did not get an expected
+		 * CSTS.RDY transition, so something with the device appears to
+		 * be stuck. Use the lower level and bigger hammer PCIe
+		 * Function Level Reset to attempt restoring the device to its
+		 * initial state, and try again.
+		 */
+		result = pcie_reset_flr(pdev, false);
+		if (result < 0)
+			return result;
+
+		pci_restore_state(pdev);
+		result = nvme_disable_ctrl(&dev->ctrl, false);
+		if (result < 0)
+			return result;
+
+		dev_info(&dev->ctrl.device,
+			"controller reset completed after pcie flr\n");
+	}
 
 	result = nvme_alloc_queue(dev, 0, NVME_AQ_DEPTH);
 	if (result)
Signature · 1 line
2.47.1
Reviewed-by on v2 1/1
#1 lore raw
Quoted 8 lines · Keith Busch

From: Keith Busch<[email protected]>

NVMe devices from multiple vendors appear to get stuck in a reset state that we can't get out of with an NVMe level Controller Reset. The kernel would report these with messages that look like:

Device not ready; aborting reset, CSTS=0x1

Quoted 7 lines

These have historically required a power cycle to make them usable again, but in many cases, a PCIe FLR is sufficient to restart operation without a power cycle. Try it if the initial controller reset fails during any nvme reset attempt.

Cc Chaitanya Kulkarni<[email protected]>

Looks good.

-ck

Reviewed-by on v2 1/1
#2 lore raw
In reply to Keith Busch
Quoted 22 lines · Keith Busch

From: Keith Busch <[email protected]>

NVMe devices from multiple vendors appear to get stuck in a reset state that we can't get out of with an NVMe level Controller Reset. The kernel would report these with messages that look like:

Device not ready; aborting reset, CSTS=0x1

These have historically required a power cycle to make them usable again, but in many cases, a PCIe FLR is sufficient to restart operation without a power cycle. Try it if the initial controller reset fails during any nvme reset attempt.

Cc Chaitanya Kulkarni <[email protected]>

---
v1->v2:

Added code comment explaining whe escalation

Add an informational kernel message that this event occured

Quoted 8 lines

Use the "pcie_reset_flr()" API instead of "pcie_flr()" since that one checks for quirks and capabilities before writing FLR config bits. Note, NVMe PCI Trasnsport Spec mandates FLR capability, so the latter should not apply to any compliant device, but you never know...

drivers/nvme/host/pci.c | 24 ++++++++++++++++++++++--

1 file changed, 22 insertions(+), 2 deletions(-)
maintainer
#4 lore raw
In reply to Keith Busch

Review

Quoted 2 lines · Keith Busch
(quoted diff) +2 -0
+		dev_info(&dev->ctrl.device,
+			"controller reset completed after pcie flr\n");

Urg, sorry, this was the wrong diff: that '&' shouldn't be there, and build bot will probably flag this.

Christoph, could you squash this in?

-- >8 --

drivers/nvme/host/pci.c +1 -1
index a7aa8ba4951f4..73d5a5298822a 100644
@@ -2083,7 +2083,7 @@ static int nvme_pci_configure_admin_queue(struct nvme_dev *dev)
 		if (result < 0)
 			return result;
 
-		dev_info(&dev->ctrl.device,
+		dev_info(dev->ctrl.device,
 			"controller reset completed after pcie flr\n");
 	}
 

--

#5 lore raw
Quoted 8 lines · Keith Busch

Review

Quoted 2 lines · Keith Busch
(quoted diff) +2 -0
+		dev_info(&dev->ctrl.device,
+			"controller reset completed after pcie flr\n");

Urg, sorry, this was the wrong diff: that '&' shouldn't be there, and build bot will probably flag this.

Christoph, could you squash this in?

In fact I already did, but forgot to push out the result or tell anyone..

maintainer
#6 lore raw
Quoted 5 lines · Christoph Hellwig
Quoted 1 line · Keith Busch

Christoph, could you squash this in?

In fact I already did, but forgot to push out the result or tell anyone..

Thanks, but the new push appears to have incorporated some unrelated changes to fs/verity/verify.c

#7 lore raw
Quoted 9 lines · Keith Busch
Quoted 5 lines · Christoph Hellwig
Quoted 1 line · Keith Busch

Christoph, could you squash this in?

In fact I already did, but forgot to push out the result or tell anyone..

Thanks, but the new push appears to have incorporated some unrelated changes to fs/verity/verify.c

Fixes. That's what you for not commiting thing immediately before rushing to a meeting.