nvme: fix PCIe subsystem reset controller state transition
Subsystems (2)
PATCH 1/1 nvme: fix PCIe subsystem reset controller state transition
The commit d2fe192348f9 (“nvme: only allow entering LIVE from CONNECTING state”) disallows controller state transitions directly from RESETTING to LIVE. However, the NVMe PCIe subsystem reset path relies on this transition to recover the controller on PowerPC (PPC) systems.
On PPC systems, issuing a subsystem reset causes a temporary loss of communication with the NVMe adapter. A subsequent PCIe MMIO read then triggers EEH recovery, which restores the PCIe link and brings the controller back online. For EEH recovery to proceed correctly, the controller must transition back to the LIVE state.
Due to the changes introduced by commit d2fe192348f9 (“nvme: only allow entering LIVE from CONNECTING state”), the controller can no longer transition directly from RESETTING to LIVE. As a result, EEH recovery exits prematurely, leaving the controller stuck in the RESETTING state.
Fix this by explicitly transitioning the controller state from RESETTING to CONNECTING and then to LIVE. This satisfies the updated state transition rules and allows the controller to be successfully recovered on PPC systems following a PCIe subsystem reset.
--- drivers/nvme/host/pci.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/nvme/host/pci.c +4 -1
@@ -1532,7 +1532,10 @@ static int nvme_pci_subsystem_reset(struct nvme_ctrl *ctrl) }writel (NVME_SUBSYS_RESET , dev->bar +NVME_REG_NSSR ); -nvme_change_ctrl_state (ctrl,NVME_CTRL_LIVE ); + +if (!nvme_change_ctrl_state (ctrl,NVME_CTRL_CONNECTING ) || + !nvme_change_ctrl_state (ctrl,NVME_CTRL_LIVE )) + goto unlock;/* * Read controller status to flush the previous write and trigger a
Signature · 1 line
2.52.0
Quoted 16 lines · Nilay Shroff
The commit
d2fe192348f9(“nvme: only allow entering LIVE from CONNECTING state”) disallows controller state transitions directly from RESETTING to LIVE. However, the NVMe PCIe subsystem reset path relies on this transition to recover the controller on PowerPC (PPC) systems.On PPC systems, issuing a subsystem reset causes a temporary loss of communication with the NVMe adapter. A subsequent PCIe MMIO read then triggers EEH recovery, which restores the PCIe link and brings the controller back online. For EEH recovery to proceed correctly, the controller must transition back to the LIVE state.
Due to the changes introduced by commit
d2fe192348f9(“nvme: only allow entering LIVE from CONNECTING state”), the controller can no longer transition directly from RESETTING to LIVE. As a result, EEH recovery exits prematurely, leaving the controller stuck in the RESETTING state.
Quoted 8 lines
Fix this by explicitly transitioning the controller state from RESETTING to CONNECTING and then to LIVE. This satisfies the updated state transition rules and allows the controller to be successfully recovered on PPC systems following a PCIe subsystem reset.
Fixes d2fe192348f9 ("nvme: only allow entering LIVE from CONNECTING state")Signed-off-by Nilay Shroff <[email protected]>
Thanks for fixing this.
Quoted 16 lines · Nilay Shroff
The commit
d2fe192348f9("nvme: only allow entering LIVE from CONNECTING state") disallows controller state transitions directly from RESETTING to LIVE. However, the NVMe PCIe subsystem reset path relies on this transition to recover the controller on PowerPC (PPC) systems.On PPC systems, issuing a subsystem reset causes a temporary loss of communication with the NVMe adapter. A subsequent PCIe MMIO read then triggers EEH recovery, which restores the PCIe link and brings the controller back online. For EEH recovery to proceed correctly, the controller must transition back to the LIVE state.
Due to the changes introduced by commit
d2fe192348f9("nvme: only allow entering LIVE from CONNECTING state"), the controller can no longer transition directly from RESETTING to LIVE. As a result, EEH recovery exits prematurely, leaving the controller stuck in the RESETTING state.
Quoted 4 lines
Fix this by explicitly transitioning the controller state from RESETTING to CONNECTING and then to LIVE. This satisfies the updated state transition rules and allows the controller to be successfully recovered on PPC systems following a PCIe subsystem reset.
Thanks, applied to nvme-6.19.
I've tested this patch with 6.19.0-rc5+ on both x86_64 and ppc64le platforms.
Note that, on the PowerPC platform the following additional patch was needed:
/John
[root@rdma-cert-03-lp10 ~]# uname -a
Linux rdma-cert-03-lp10.rdma.lab.eng.rdu2.redhat.com 6.19.0-rc5+ #1 SMP Thu Jan 15 14:14:45 EST 2026 ppc64le GNU/Linux [root@rdma-cert-03-lp10 ~]# dmesg -C
[root@rdma-cert-03-lp10 ~]# dmesg -Tw&
[1] 15696
[root@rdma-cert-03-lp10 ~]# nvme list-subsys
nvme-subsys0 - NQN=nqn.1994-11.com.samsung:nvme:PM1735:HHHL:S4WANA0R400032
hostnqn=nqn.2014-08.org.nvmexpress:uuid:1654a627-93b6-4650-ba90-f4dc7a2fd3ee
iopolicy=numa\
+- nvme0 pcie 0018:01:00.0 live
[root@rdma-cert-03-lp10 ~]# [Thu Jan 15 14:24:11 2026] block nvme0n1: No UUID available providing old NGUID
[root@rdma-cert-03-lp10 ~]# nvme subsystem-reset /dev/nvme0 [root@rdma-cert-03-lp10 ~]# nvme list-subsys /dev/nvme0n1
[Thu Jan 15 14:25:29 2026] EEH: Recovering PHB#18-PE#10000 [Thu Jan 15 14:25:29 2026] EEH: PE location: N/A, PHB location: N/A [Thu Jan 15 14:25:29 2026] EEH: Frozen PHB#18-PE#10000 detected [Thu Jan 15 14:25:29 2026] EEH: Call Trace: [Thu Jan 15 14:25:29 2026] EEH: [00000000d71b2d94] __eeh_send_failure_event+0x78/0x160 [Thu Jan 15 14:25:29 2026] EEH: [00000000465f9a5f] eeh_dev_check_failure+0x2c0/0x700 [Thu Jan 15 14:25:29 2026] EEH: [0000000015f9541b] nvme_timeout+0x274/0x690 [nvme] [Thu Jan 15 14:25:29 2026] EEH: [00000000231862b5] blk_mq_handle_expired+0xb0/0x130 [Thu Jan 15 14:25:29 2026] EEH: [0000000080ea6b3b] bt_iter+0xf8/0x140 [Thu Jan 15 14:25:29 2026] EEH: [00000000b79a498f] blk_mq_queue_tag_busy_iter+0x35c/0x700 [Thu Jan 15 14:25:29 2026] EEH: [0000000077a6dbdc] blk_mq_timeout_work+0x1a8/0x240 [Thu Jan 15 14:25:29 2026] EEH: [000000000ef3edde] process_one_work+0x1f4/0x500 [Thu Jan 15 14:25:29 2026] EEH: [0000000039a4e1cd] worker_thread+0x33c/0x510 [Thu Jan 15 14:25:29 2026] EEH: [000000005a866bd1] kthread+0x154/0x170 [Thu Jan 15 14:25:29 2026] EEH: [0000000077e75258] start_kernel_thread+0x14/0x18 [Thu Jan 15 14:25:29 2026] EEH: This PCI device has failed 1 times in the last hour and will be permanently disabled after 5 failures. [Thu Jan 15 14:25:29 2026] EEH: Notify device drivers to shutdown [Thu Jan 15 14:25:29 2026] EEH: Beginning: 'error_detected(IO frozen)' [Thu Jan 15 14:25:29 2026] PCI 0018:01:00.0#10000: EEH: Invoking nvme->error_detected(IO frozen) [Thu Jan 15 14:25:29 2026] nvme nvme0: frozen state error detected, reset controller [Thu Jan 15 14:25:29 2026] block nvme0n1: no usable path - requeuing I/O [Thu Jan 15 14:25:29 2026] block nvme0n1: no usable path - requeuing I/O [Thu Jan 15 14:25:29 2026] block nvme0n1: no usable path - requeuing I/O [Thu Jan 15 14:25:29 2026] block nvme0n1: no usable path - requeuing I/O [Thu Jan 15 14:25:29 2026] nvme nvme0: Identify namespace failed (-4) [Thu Jan 15 14:25:29 2026] PCI 0018:01:00.0#10000: EEH: nvme driver reports: 'need reset' [Thu Jan 15 14:25:29 2026] EEH: Finished:'error_detected(IO frozen)' with aggregate recovery state:'need reset' [Thu Jan 15 14:25:29 2026] EEH: Collect temporary log [Thu Jan 15 14:25:29 2026] EEH: of node=0018:01:00.0 [Thu Jan 15 14:25:29 2026] EEH: PCI device/vendor: a824144d [Thu Jan 15 14:25:29 2026] EEH: PCI cmd/status register: 00100140 [Thu Jan 15 14:25:29 2026] EEH: PCI-E capabilities and status follow: [Thu Jan 15 14:25:29 2026] EEH: PCI-E 00: 0002b010 10008fe2 00002910 00437084 [Thu Jan 15 14:25:29 2026] EEH: PCI-E 10: 10840000 00000000 00000000 00000000 [Thu Jan 15 14:25:29 2026] EEH: PCI-E 20: 00000000 [Thu Jan 15 14:25:29 2026] EEH: PCI-E AER capability register set follows: [Thu Jan 15 14:25:29 2026] EEH: PCI-E AER 00: 14820001 00000000 00400000 00462030 [Thu Jan 15 14:25:29 2026] EEH: PCI-E AER 10: 00000000 0000e000 000002a0 00000000 [Thu Jan 15 14:25:29 2026] EEH: PCI-E AER 20: 00000000 00000000 00000000 00000000 [Thu Jan 15 14:25:29 2026] EEH: PCI-E AER 30: 00000000 00000000 [Thu Jan 15 14:25:29 2026] RTAS: event: 3, Type: Platform Error (224), Severity: 2 [Thu Jan 15 14:25:29 2026] EEH: Reset without hotplug activity [Thu Jan 15 14:25:29 2026] block nvme0n1: no usable path - requeuing I/O [Thu Jan 15 14:25:29 2026] block nvme0n1: no usable path - requeuing I/O [Thu Jan 15 14:25:31 2026] EEH: Beginning: 'slot_reset' [Thu Jan 15 14:25:31 2026] PCI 0018:01:00.0#10000: EEH: Invoking nvme->slot_reset() [Thu Jan 15 14:25:31 2026] nvme nvme0: restart after slot reset [Thu Jan 15 14:25:31 2026] PCI 0018:01:00.0#10000: EEH: nvme driver reports: 'recovered' [Thu Jan 15 14:25:31 2026] EEH: Finished:'slot_reset' with aggregate recovery state:'recovered' [Thu Jan 15 14:25:31 2026] EEH: Notify device driver to resume [Thu Jan 15 14:25:31 2026] EEH: Beginning: 'resume' [Thu Jan 15 14:25:31 2026] PCI 0018:01:00.0#10000: EEH: Invoking nvme->resume() [Thu Jan 15 14:25:31 2026] nvme nvme0: D3 entry latency set to 10 seconds [Thu Jan 15 14:25:31 2026] nvme nvme0: 16/0/0 default/read/poll queues [Thu Jan 15 14:25:31 2026] PCI 0018:01:00.0#10000: EEH: nvme driver reports: 'none' [Thu Jan 15 14:25:31 2026] EEH: Finished:'resume' [Thu Jan 15 14:25:31 2026] EEH: Recovery successful.
[root@rdma-cert-03-lp10 ~]# nvme list-subsys /dev/nvme0n1 nvme-subsys0 - NQN=nqn.1994-11.com.samsung:nvme:PM1735:HHHL:S4WANA0R400032
hostnqn=nqn.2014-08.org.nvmexpress:uuid:1654a627-93b6-4650-ba90-f4dc7a2fd3ee
iopolicy=numa\
+- nvme0 pcie 0018:01:00.0 live optimized
Quoted 44 lines · Nilay Shroff
The commit
d2fe192348f9(“nvme: only allow entering LIVE from CONNECTING state”) disallows controller state transitions directly from RESETTING to LIVE. However, the NVMe PCIe subsystem reset path relies on this transition to recover the controller on PowerPC (PPC) systems.On PPC systems, issuing a subsystem reset causes a temporary loss of communication with the NVMe adapter. A subsequent PCIe MMIO read then triggers EEH recovery, which restores the PCIe link and brings the controller back online. For EEH recovery to proceed correctly, the controller must transition back to the LIVE state.
Due to the changes introduced by commit
d2fe192348f9(“nvme: only allow entering LIVE from CONNECTING state”), the controller can no longer transition directly from RESETTING to LIVE. As a result, EEH recovery exits prematurely, leaving the controller stuck in the RESETTING state.Fix this by explicitly transitioning the controller state from RESETTING to CONNECTING and then to LIVE. This satisfies the updated state transition rules and allows the controller to be successfully recovered on PPC systems following a PCIe subsystem reset.
Fixes d2fe192348f9 ("nvme: only allow entering LIVE from CONNECTING state")Signed-off-by Nilay Shroff <[email protected]>--- drivers/nvme/host/pci.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)drivers/nvme/host/pci.c +4
-1@@ -1532,7 +1532,10 @@ static int nvme_pci_subsystem_reset(struct nvme_ctrl *ctrl) }writel (NVME_SUBSYS_RESET , dev->bar +NVME_REG_NSSR ); -nvme_change_ctrl_state (ctrl,NVME_CTRL_LIVE ); + +if (!nvme_change_ctrl_state (ctrl,NVME_CTRL_CONNECTING ) || + !nvme_change_ctrl_state (ctrl,NVME_CTRL_LIVE )) + goto unlock;/* * Read controller status to flush the previous write and trigger a