nvme: do not reset controllers in NVME_CTRL_NEW state
PATCH 1/1 nvme: do not reset controllers in NVME_CTRL_NEW state
During NVMe controller creation, the controller is exposed to sysfs via nvme_add_ctrl() while it is still in the NVME_CTRL_NEW state. This creates a narrow race window where a userspace process can write to the reset_controller sysfs node before the initialization thread transitions the state to NVME_CTRL_CONNECTING.
If a reset is triggered during this window, the state machine allows the transition from NVME_CTRL_NEW to NVME_CTRL_RESETTING, and the reset work is queued. However, the original creation thread continues its execution, subsequently moving the state to NVME_CTRL_CONNECTING and finally to NVME_CTRL_LIVE.
When the delayed reset work finally executes, it attempts to tear down the controller and transition the state to NVME_CTRL_CONNECTING. Because the state is now NVME_CTRL_LIVE, this transition fails, triggering a WARN_ON in the reset work.
Fix this by removing NVME_CTRL_NEW from the allowed prior states for the NVME_CTRL_RESETTING transition.
--- drivers/nvme/host/core.c | 1 - 1 file changed, 1 deletion(-)
drivers/nvme/host/core.c +0 -1
@@ -598,7 +598,6 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,break ;case NVME_CTRL_RESETTING :switch (old_state) { -case NVME_CTRL_NEW :case NVME_CTRL_LIVE : changed = true;atomic_long_inc (&ctrl->nr_reset);
Signature · 1 line
2.55.0
Quoted 17 lines · Maurizio Lombardi
During NVMe controller creation, the controller is exposed to sysfs via
nvme_add_ctrl()while it is still in theNVME_CTRL_NEWstate. This creates a narrow race window where a userspace process can write to thereset_controllersysfs node before the initialization thread transitions the state toNVME_CTRL_CONNECTING.If a reset is triggered during this window, the state machine allows the transition from
NVME_CTRL_NEWtoNVME_CTRL_RESETTING, and the reset work is queued. However, the original creation thread continues its execution, subsequently moving the state toNVME_CTRL_CONNECTINGand finally toNVME_CTRL_LIVE.When the delayed reset work finally executes, it attempts to tear down the controller and transition the state to
NVME_CTRL_CONNECTING. Because the state is nowNVME_CTRL_LIVE, this transition fails, triggering aWARN_ONin the reset work.
Quoted 7 lines
Fix this by removing
NVME_CTRL_NEWfrom the allowed prior states for theNVME_CTRL_RESETTINGtransition.Fixes 8bfc3b4c6f9d ("nvmet: switch loopback target state to connecting when resetting")Reported-by [email protected]Signed-off-by Maurizio Lombardi <[email protected] or why it wasthere it there in the first place
Looks reasonable. It might be worth explaining why we don't need to handle the NEW -> RESETTING state transition or why it was there in the first place.
Quoted 21 lines · Daniel Wagner
Quoted 20 lines · Maurizio Lombardi
During NVMe controller creation, the controller is exposed to sysfs via
nvme_add_ctrl()while it is still in theNVME_CTRL_NEWstate. This creates a narrow race window where a userspace process can write to thereset_controllersysfs node before the initialization thread transitions the state toNVME_CTRL_CONNECTING.If a reset is triggered during this window, the state machine allows the transition from
NVME_CTRL_NEWtoNVME_CTRL_RESETTING, and the reset work is queued. However, the original creation thread continues its execution, subsequently moving the state toNVME_CTRL_CONNECTINGand finally toNVME_CTRL_LIVE.When the delayed reset work finally executes, it attempts to tear down the controller and transition the state to
NVME_CTRL_CONNECTING. Because the state is nowNVME_CTRL_LIVE, this transition fails, triggering aWARN_ONin the reset work.Fix this by removing
NVME_CTRL_NEWfrom the allowed prior states for theNVME_CTRL_RESETTINGtransition.
Quoted 8 lines
Quoted 4 lines
Fixes 8bfc3b4c6f9d ("nvmet: switch loopback target state to connecting when resetting")Reported-by [email protected]Signed-off-by Maurizio Lombardi <[email protected] or why it wasthere it there in the first place
Looks reasonable. It might be worth explaining why we don't need to handle the NEW -> RESETTING state transition or why it was there in the first place.
Maybe the last paragraph could be expanded in the following way:
"
Fix this by removing NVME_CTRL_NEW from the allowed prior states for the NVME_CTRL_RESETTING transition. It is safe to drop this transition entirely because a controller in the NVME_CTRL_NEW state is merely allocated and not yet initialized, a reset operation at this stage is meaningless.
"
For what concernes the "why it was there in the first place" question, I think that the reason is that in ancient times a CONNECTING state didn't exist. The controller initialization was done via a reset; so the transition was NEW -> RESETTING -> LIVE.
Maurizio
Quoted 8 lines · Maurizio Lombardi
"
Fix this by removingNVME_CTRL_NEWfrom the allowed prior states for theNVME_CTRL_RESETTINGtransition. It is safe to drop this transition entirely because a controller in theNVME_CTRL_NEWstate is merely allocated and not yet initialized, a reset operation at this stage is meaningless.
"
Quoted 4 lines
For what concernes the "why it was there in the first place" question, I think that the reason is that in ancient times a CONNECTING state didn't exist. The controller initialization was done via a reset; so the transition was NEW -> RESETTING -> LIVE.
This rings a bell. With the introduction of the fabrics drivers, the CONNECTING state was introduced and the PCI driver didn't use the CONNECTING state for while. But due to various reasons, the PCI transport follows the same state transitions as the fabrics ones now.
I am just a bit careful with the state machine, even obvious correct changes can introduce subtle regressions.
Keith Busch
maintainerQuoted 14 lines · Daniel Wagner
Quoted 8 lines · Maurizio Lombardi
"
Fix this by removingNVME_CTRL_NEWfrom the allowed prior states for theNVME_CTRL_RESETTINGtransition. It is safe to drop this transition entirely because a controller in theNVME_CTRL_NEWstate is merely allocated and not yet initialized, a reset operation at this stage is meaningless.
"Quoted 4 lines
For what concernes the "why it was there in the first place" question, I think that the reason is that in ancient times a CONNECTING state didn't exist. The controller initialization was done via a reset; so the transition was NEW -> RESETTING -> LIVE.
Quoted 4 lines
This rings a bell. With the introduction of the fabrics drivers, the CONNECTING state was introduced and the PCI driver didn't use the CONNECTING state for while. But due to various reasons, the PCI transport follows the same state transitions as the fabrics ones now.
Yep, nvme pci used to always kick most of the initialization to the reset_work, so that transport wanted the NEW->RESETTING transition. With asynchronous probe support added, we do the initial setup without reset_work.
Quoted 12 lines · Maurizio Lombardi
During NVMe controller creation, the controller is exposed to sysfs via
nvme_add_ctrl()while it is still in theNVME_CTRL_NEWstate. This creates a narrow race window where a userspace process can write to thereset_controllersysfs node before the initialization thread transitions the state toNVME_CTRL_CONNECTING.If a reset is triggered during this window, the state machine allows the transition from
NVME_CTRL_NEWtoNVME_CTRL_RESETTING, and the reset work is queued. However, the original creation thread continues its execution, subsequently moving the state toNVME_CTRL_CONNECTINGand finally toNVME_CTRL_LIVE.
Quoted 7 lines
When the delayed reset work finally executes, it attempts to tear down the controller and transition the state to
NVME_CTRL_CONNECTING. Because the state is nowNVME_CTRL_LIVE, this transition fails, triggering aWARN_ONin the reset work.Fix this by removing
NVME_CTRL_NEWfrom the allowed prior states for theNVME_CTRL_RESETTINGtransition.
Thanks, applied to nvme-7.3.
Quoted 32 lines · Maurizio Lombardi
During NVMe controller creation, the controller is exposed to sysfs via
nvme_add_ctrl()while it is still in theNVME_CTRL_NEWstate. This creates a narrow race window where a userspace process can write to thereset_controllersysfs node before the initialization thread transitions the state toNVME_CTRL_CONNECTING.If a reset is triggered during this window, the state machine allows the transition from
NVME_CTRL_NEWtoNVME_CTRL_RESETTING, and the reset work is queued. However, the original creation thread continues its execution, subsequently moving the state toNVME_CTRL_CONNECTINGand finally toNVME_CTRL_LIVE.When the delayed reset work finally executes, it attempts to tear down the controller and transition the state to
NVME_CTRL_CONNECTING. Because the state is nowNVME_CTRL_LIVE, this transition fails, triggering aWARN_ONin the reset work.Fix this by removing
NVME_CTRL_NEWfrom the allowed prior states for theNVME_CTRL_RESETTINGtransition.Fixes 8bfc3b4c6f9d ("nvmet: switch loopback target state to connecting when resetting")Reported-by [email protected]Signed-off-by Maurizio Lombardi <[email protected]>--- drivers/nvme/host/core.c | 1 - 1 file changed, 1 deletion(-)drivers/nvme/host/core.c +0
-0
drivers/nvme/host/core.c bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl, · line 603
Quoted 7 lines
drivers/nvme/host/core.c +0
-1@@ -598,7 +598,6 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,break ;case NVME_CTRL_RESETTING :switch (old_state) { -case NVME_CTRL_NEW :case NVME_CTRL_LIVE : changed = true;atomic_long_inc (&ctrl->nr_reset);
Hi!
I was testing something with linux-next 2026-10-02 and ran into an issue with the nvme-apple driver where /dev/nvme0 exists but none of the namespaces appear, and the dmesg is awfully quiet about the nvme device. I bisected it down to this patch, which as far as I can tell leads to the nvme-apple being stuck in NVME_CTRL_NEW state. Without being familiar with the matter, this looks like the nvme-apple driver was using this transition as part of its initialization procedure. If one of the NVMe people who know how these transitions are supposed to work could have a look, that would be much appreciated, as otherwise this will break nvme-apple in 7.3-rc6.
Thanks,
- Yureka
Quoted 56 lines · Yureka Lilian
Quoted 32 lines · Maurizio Lombardi
During NVMe controller creation, the controller is exposed to sysfs via
nvme_add_ctrl()while it is still in theNVME_CTRL_NEWstate. This creates a narrow race window where a userspace process can write to thereset_controllersysfs node before the initialization thread transitions the state toNVME_CTRL_CONNECTING.If a reset is triggered during this window, the state machine allows the transition from
NVME_CTRL_NEWtoNVME_CTRL_RESETTING, and the reset work is queued. However, the original creation thread continues its execution, subsequently moving the state toNVME_CTRL_CONNECTINGand finally toNVME_CTRL_LIVE.When the delayed reset work finally executes, it attempts to tear down the controller and transition the state to
NVME_CTRL_CONNECTING. Because the state is nowNVME_CTRL_LIVE, this transition fails, triggering aWARN_ONin the reset work.Fix this by removing
NVME_CTRL_NEWfrom the allowed prior states for theNVME_CTRL_RESETTINGtransition.Fixes 8bfc3b4c6f9d ("nvmet: switch loopback target state to connecting when resetting")Reported-by [email protected]Signed-off-by Maurizio Lombardi <[email protected]>--- drivers/nvme/host/core.c | 1 - 1 file changed, 1 deletion(-)drivers/nvme/host/core.c +0
-0drivers/nvme/host/core.c bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl, · line 603
Quoted 7 lines
drivers/nvme/host/core.c +0
-1@@ -598,7 +598,6 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,break ;case NVME_CTRL_RESETTING :switch (old_state) { -case NVME_CTRL_NEW :case NVME_CTRL_LIVE : changed = true;atomic_long_inc (&ctrl->nr_reset);Hi!
I was testing something with linux-next 2026-10-02 and ran into an issue with the
nvme-apple driver where/dev/nvme0exists but none of the namespaces appear, and the dmesg is awfully quiet about the nvme device. I bisected it down to this patch, which as far as I can tell leads to the nvme-apple being stuck inNVME_CTRL_NEWstate. Without being familiar with the matter, this looks like the nvme-apple driver was using this transition as part of its initialization
procedure. If one of the NVMe people who know how these transitions are supposed to work could have a look, that would be much appreciated, as otherwise this will break nvme-apple in 7.3-rc6.
Quoted 1 line
Thanks,
Thanks for reporting this,
it seems that nvme-apple is the only driver that still initializes the controller by performing a reset directly from a NEW state.
This should be easy to fix by moving the state to CONNECTING in apple_nvme_probe, just before nvme_reset_ctrl() is called.
Would you be able to test the patch if I prepare it?
Maurizio
Quoted 6 lines · Maurizio Lombardi
Thanks for reporting this,
it seems that nvme-apple is the only driver that still initializes the controller by performing a reset directly from a NEW state.This should be easy to fix by moving the state to CONNECTING in
apple_nvme_probe, just beforenvme_reset_ctrl()is called.
Ops, this won't work, as we can enter the resetting state only when the controller is live.
Keith, I think we have no other option but revert this patch to unbreak apple-nvme until I find a proper fix.
Maurizio
Quoted 7 lines · Maurizio Lombardi
Quoted 6 lines · Maurizio Lombardi
Thanks for reporting this,
it seems that nvme-apple is the only driver that still initializes the controller by performing a reset directly from a NEW state.This should be easy to fix by moving the state to CONNECTING in
apple_nvme_probe, just beforenvme_reset_ctrl()is called.
Quoted 7 lines
Ops, this won't work, as we can enter the resetting state only when the controller is live.
Keith, I think we have no other option but revert this patch to unbreak apple-nvme until I find a proper fix.
Maurizio
When you eventually propose a new variant of this, I can certainly test it with nvme-apple.
- Yureka
Keith Busch
maintainerQuoted 11 lines · Yureka Lilian
Quoted 10 lines · Maurizio Lombardi
Quoted 6 lines · Maurizio Lombardi
Thanks for reporting this,
it seems that nvme-apple is the only driver that still initializes the controller by performing a reset directly from a NEW state.This should be easy to fix by moving the state to CONNECTING in
apple_nvme_probe, just beforenvme_reset_ctrl()is called.Ops, this won't work, as we can enter the resetting state only when the controller is live.
Quoted 8 lines
Quoted 4 lines
Keith, I think we have no other option but revert this patch to unbreak apple-nvme until I find a proper fix.
Maurizio
When you eventually propose a new variant of this, I can certainly test it with nvme-apple.
Thanks for the notice. Patch reverted committed to nvme-7.3 and will send a pull request shortly.