nvme-pci: try function level reset on init failure
PATCH v2 1/1 nvme-pci: try function level reset on init failure
From: Keith Busch <[email protected]>
NVMe devices from multiple vendors appear to get stuck in a reset state that we can't get out of with an NVMe level Controller Reset. The kernel would report these with messages that look like:
Device not ready; aborting reset, CSTS=0x1
These have historically required a power cycle to make them usable again, but in many cases, a PCIe FLR is sufficient to restart operation without a power cycle. Try it if the initial controller reset fails during any nvme reset attempt.
---v1->v2:
Added code comment explaining whe escalation
Add an informational kernel message that this event occured
Use the "pcie_reset_flr()" API instead of "pcie_flr()" since that one checks for quirks and capabilities before writing FLR config bits. Note, NVMe PCI Trasnsport Spec mandates FLR capability, so the latter should not apply to any compliant device, but you never know...
drivers/nvme/host/pci.c | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-)
drivers/nvme/host/pci.c +22 -2
@@ -2067,8 +2067,28 @@ static int nvme_pci_configure_admin_queue(struct nvme_dev *dev)* might be pointing at! */ result =nvme_disable_ctrl (&dev->ctrl, false); -if (result <0 ) -return result; +if (result <0 ) { +struct pci_dev *pdev =to_pci_dev (dev->dev); + +/* +* The NVMe Controller Reset method did not get an expected +* CSTS.RDY transition, so something with the device appears to +* be stuck. Use the lower level and bigger hammer PCIe +* Function Level Reset to attempt restoring the device to its +* initial state, and try again. +*/ + result =pcie_reset_flr (pdev, false); +if (result <0 ) +return result; + +pci_restore_state (pdev); + result =nvme_disable_ctrl (&dev->ctrl, false); +if (result <0 ) +return result; + +dev_info (&dev->ctrl.device, +"controller reset completed after pcie flr\n" ); + } result =nvme_alloc_queue (dev,0 ,NVME_AQ_DEPTH );if (result)
Signature · 1 line
2.47.1
Quoted 8 lines · Keith Busch
From: Keith Busch<[email protected]>
NVMe devices from multiple vendors appear to get stuck in a reset state that we can't get out of with an NVMe level Controller Reset. The kernel would report these with messages that look like:
Device not ready; aborting reset, CSTS=0x1
Quoted 7 lines
These have historically required a power cycle to make them usable again, but in many cases, a PCIe FLR is sufficient to restart operation without a power cycle. Try it if the initial controller reset fails during any nvme reset attempt.
Cc Chaitanya Kulkarni<[email protected]>Signed-off-by Keith Busch<[email protected]>
Looks good.
-ck
Quoted 22 lines · Keith Busch
From: Keith Busch <[email protected]>
NVMe devices from multiple vendors appear to get stuck in a reset state that we can't get out of with an NVMe level Controller Reset. The kernel would report these with messages that look like:
Device not ready; aborting reset, CSTS=0x1
These have historically required a power cycle to make them usable again, but in many cases, a PCIe FLR is sufficient to restart operation without a power cycle. Try it if the initial controller reset fails during any nvme reset attempt.
Cc Chaitanya Kulkarni <[email protected]>Signed-off-by Keith Busch <[email protected]>---
v1->v2:Added code comment explaining whe escalation
Add an informational kernel message that this event occured
Quoted 8 lines
Use the "
pcie_reset_flr()" API instead of "pcie_flr()" since that one checks for quirks and capabilities before writing FLR config bits. Note, NVMe PCI Trasnsport Spec mandates FLR capability, so the latter should not apply to any compliant device, but you never know...
drivers/nvme/host/pci.c| 24 ++++++++++++++++++++++--1 file changed, 22 insertions(+), 2 deletions(-)
Thanks,
applied to nvme-6.17.
Keith Busch
maintainerReview
Quoted 2 lines · Keith Busch
(quoted diff) +2
-0+ dev_info(&dev->ctrl.device, + "controller reset completed after pcie flr\n");
Urg, sorry, this was the wrong diff: that '&' shouldn't be there, and build bot will probably flag this.
Christoph, could you squash this in?
-- >8 --
drivers/nvme/host/pci.c +1 -1
@@ -2083,7 +2083,7 @@ static int nvme_pci_configure_admin_queue(struct nvme_dev *dev)if (result <0 )return result; -dev_info (&dev->ctrl.device, +dev_info (dev->ctrl.device,"controller reset completed after pcie flr\n" ); }
--
Christoph Hellwig
maintainerQuoted 8 lines · Keith Busch
Review
Quoted 2 lines · Keith Busch
(quoted diff) +2
-0+ dev_info(&dev->ctrl.device, + "controller reset completed after pcie flr\n");Urg, sorry, this was the wrong diff: that '&' shouldn't be there, and build bot will probably flag this.
Christoph, could you squash this in?
In fact I already did, but forgot to push out the result or tell anyone..
Keith Busch
maintainerQuoted 5 lines · Christoph Hellwig
Quoted 1 line · Keith Busch
Christoph, could you squash this in?
In fact I already did, but forgot to push out the result or tell anyone..
Thanks, but the new push appears to have incorporated some unrelated changes to fs/verity/verify.c
Christoph Hellwig
maintainerQuoted 9 lines · Keith Busch
Quoted 5 lines · Christoph Hellwig
Quoted 1 line · Keith Busch
Christoph, could you squash this in?
In fact I already did, but forgot to push out the result or tell anyone..
Thanks, but the new push appears to have incorporated some unrelated changes to
fs/verity/verify.c
Fixes. That's what you for not commiting thing immediately before rushing to a meeting.