Skip to content

nvme-pci: recover from NVM subsystem reset

Nilay Shroff linux-nvme 3 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v3 Latest1 patch 9 repliesIn review
  2. v21 patch 1 replySuperseded
  3. v11 patch 1 replySuperseded

Changes in v3 from the cover letter

- Formatting cleanup
- Updated commit changelog to better describe the issue
- Added the cover later to add more details about nvme subsystem reset and error recovery(EEH)

Patches in v3 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/1nvme-pci : Fix EEH failure on ppc after subsystem resetdrivers/nvme/host/core.c drivers/nvme/host/pci.c+19 -39 replies

Reviews trailers per patch

The patch has a Reviewed-by or Acked-by.

PatchHannes Reinecke
1/1R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v3 thread

Cover letter [PATCH v3 0/1] nvme-pci: recover from NVM subsystem reset

Hi Keith,

My previous attempt to get attention for this patch didn't garner enough eyeballs and so I thought to rewrite the text and tried including some more background on this. For those interested, I have also copied below the link to the previous email where we had some discussions about this patch.

The NVM subsystem reset command might be needed for activating nvme controller firmware image after the image is committed to a slot or in some cases to recover from the controller fatal error. The NVM subsystem reset when executed, may cause the loss of communication with NVMe controller. And the only way to re-establish communication with NVMe adapter is to either re-enumerate the pci bus or hotplug NVMe disk or reboot the OS. Fortunately, the PPC architecture supports extended PCI capability which could help recover the loss of PCI adapter communication. The EEH (Enhanced Error Handling) hardware features on PPC machine allow PCI bus errors to be cleared and a PCI card to be "rebooted", without actually having to reboot the OS or re-enumerating PCI bus or hotplugging NVMe disk.

In the current implementation, when user executes NVM subsystem reset command, kernel programs the nvme subsystem register (NSSR) and then initiates the nvme reset work. The nvme reset work first shuts down the controller and that requires access to PCIe config space. As programming to NSSR typically causes the loss of communication with NVMe controller, the nvme reset work which is immediately followed after that would fail to read/write to PCIe config space and that causes the nvme driver to believe that controller is dead and so driver cleanup all resources associated with that NVMe controller and marks the controller dead. So the PCI error recovery (EEH on PPC) doesn't get chance to try recover device from the adapter communication lost.

This patch helps to detect the case if the communication with the NVMe adapter is lost and the PCI error recovery has been initiated by the platform then allow error recovery to forward progress and thus contain the nvme reset work (which has been initiated post NVM subsystem reset) from marking the controller dead. If in case pci error recovery is unable to recover the device then it sets the pci channel state to "permanent failure" and help removes the device.

I have tested the following cases with this patch applied,
1. NVM subsystem reset while no IO is running
2. NVM subsystem reset while IO is ongoing
3. Inject PCI error while reset work is scheduled and no IO is running
4. Inject PCI error while reset work is scheduled and IO is ongoing

For all above cases (1-4), verified that pci error recovery could successfully recover the nvme disk.

5. NVM subsystem reset and then immediately hot remove the NVMe disk:
In this case though pci error recovery is initiated it couldn't forward progress (as disk is hot removed) and so controller is deleted and it's all associated resources are freed.

6. NVM subsystem reset and PCI error recovery is unable to recover the device:
In this case controller is deleted and it's all associated resources are freed.

7. NVM subsystem reset on a platform which doesn't support PCI error recovery:
In this case nvme reset work frees resources associated with the controller and mark it dead.

Changelog:
==========
Changes from v2:
- Formatting cleanup
- Updated commit changelog to better describe the issue
- Added the cover later to add more details about nvme subsystem reset and error recovery(EEH)

Changes from v1:
- Allow a controller to move from CONNECTING state to RESETTING state (Keith)

- Fix race condition between reset work and pci error handler code which may contain reset work and pci recovery from forward progress (Keith)

Nilay Shroff (1):
nvme-pci : Fix EEH failure on ppc after subsystem reset

 drivers/nvme/host/core.c |  1 +
 drivers/nvme/host/pci.c  | 20 +++++++++++++++++---
 2 files changed, 18 insertions(+), 3 deletions(-)
Signature · 1 line
2.45.1