Skip to contentSkip to conversation

nvme: do not reset controllers in NVME_CTRL_NEW state

PATCH Applied Maurizio Lombardi 5 participants 11 messages Sep 15 – Oct 5 NVMe

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 nvme: do not reset controllers in NVME_CTRL_NEW state

During NVMe controller creation, the controller is exposed to sysfs via nvme_add_ctrl() while it is still in the NVME_CTRL_NEW state. This creates a narrow race window where a userspace process can write to the reset_controller sysfs node before the initialization thread transitions the state to NVME_CTRL_CONNECTING.

If a reset is triggered during this window, the state machine allows the transition from NVME_CTRL_NEW to NVME_CTRL_RESETTING, and the reset work is queued. However, the original creation thread continues its execution, subsequently moving the state to NVME_CTRL_CONNECTING and finally to NVME_CTRL_LIVE.

When the delayed reset work finally executes, it attempts to tear down the controller and transition the state to NVME_CTRL_CONNECTING. Because the state is now NVME_CTRL_LIVE, this transition fails, triggering a WARN_ON in the reset work.

Fix this by removing NVME_CTRL_NEW from the allowed prior states for the NVME_CTRL_RESETTING transition.

Fixes 8bfc3b4c6f9d ("nvmet: switch loopback target state to connecting when resetting")
Reported-by [email protected]
---
 drivers/nvme/host/core.c | 1 -
 1 file changed, 1 deletion(-)
drivers/nvme/host/core.c +0 -1
index beea23d04a70..7ea8dd27d08d 100644
@@ -598,7 +598,6 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,
 		break;
 	case NVME_CTRL_RESETTING:
 		switch (old_state) {
-		case NVME_CTRL_NEW:
 		case NVME_CTRL_LIVE:
 			changed = true;
 			atomic_long_inc(&ctrl->nr_reset);
Signature · 1 line
2.55.0
Reviewed-by on v1 1/1
#1 lore raw
Quoted 17 lines · Maurizio Lombardi

During NVMe controller creation, the controller is exposed to sysfs via nvme_add_ctrl() while it is still in the NVME_CTRL_NEW state. This creates a narrow race window where a userspace process can write to the reset_controller sysfs node before the initialization thread transitions the state to NVME_CTRL_CONNECTING.

If a reset is triggered during this window, the state machine allows the transition from NVME_CTRL_NEW to NVME_CTRL_RESETTING, and the reset work is queued. However, the original creation thread continues its execution, subsequently moving the state to NVME_CTRL_CONNECTING and finally to NVME_CTRL_LIVE.

When the delayed reset work finally executes, it attempts to tear down the controller and transition the state to NVME_CTRL_CONNECTING. Because the state is now NVME_CTRL_LIVE, this transition fails, triggering a WARN_ON in the reset work.

Quoted 7 lines

Fix this by removing NVME_CTRL_NEW from the allowed prior states for the NVME_CTRL_RESETTING transition.

Fixes 8bfc3b4c6f9d ("nvmet: switch loopback target state to connecting when resetting")
Reported-by [email protected]

there it there in the first place

Looks reasonable. It might be worth explaining why we don't need to handle the NEW -> RESETTING state transition or why it was there in the first place.

#2 lore raw
Quoted 21 lines · Daniel Wagner
Quoted 20 lines · Maurizio Lombardi

During NVMe controller creation, the controller is exposed to sysfs via nvme_add_ctrl() while it is still in the NVME_CTRL_NEW state. This creates a narrow race window where a userspace process can write to the reset_controller sysfs node before the initialization thread transitions the state to NVME_CTRL_CONNECTING.

If a reset is triggered during this window, the state machine allows the transition from NVME_CTRL_NEW to NVME_CTRL_RESETTING, and the reset work is queued. However, the original creation thread continues its execution, subsequently moving the state to NVME_CTRL_CONNECTING and finally to NVME_CTRL_LIVE.

When the delayed reset work finally executes, it attempts to tear down the controller and transition the state to NVME_CTRL_CONNECTING. Because the state is now NVME_CTRL_LIVE, this transition fails, triggering a WARN_ON in the reset work.

Fix this by removing NVME_CTRL_NEW from the allowed prior states for the NVME_CTRL_RESETTING transition.

Quoted 8 lines
Quoted 4 lines
Fixes 8bfc3b4c6f9d ("nvmet: switch loopback target state to connecting when resetting")
Reported-by [email protected]

there it there in the first place

Looks reasonable. It might be worth explaining why we don't need to handle the NEW -> RESETTING state transition or why it was there in the first place.

Maybe the last paragraph could be expanded in the following way:

"
Fix this by removing NVME_CTRL_NEW from the allowed prior states for the NVME_CTRL_RESETTING transition. It is safe to drop this transition entirely because a controller in the NVME_CTRL_NEW state is merely allocated and not yet initialized, a reset operation at this stage is meaningless.
"

For what concernes the "why it was there in the first place" question, I think that the reason is that in ancient times a CONNECTING state didn't exist. The controller initialization was done via a reset; so the transition was NEW -> RESETTING -> LIVE.

Maurizio

#3 lore raw
Quoted 8 lines · Maurizio Lombardi

"
Fix this by removing NVME_CTRL_NEW from the allowed prior states for the NVME_CTRL_RESETTING transition. It is safe to drop this transition entirely because a controller in the NVME_CTRL_NEW state is merely allocated and not yet initialized, a reset operation at this stage is meaningless.
"

Quoted 4 lines

For what concernes the "why it was there in the first place" question, I think that the reason is that in ancient times a CONNECTING state didn't exist. The controller initialization was done via a reset; so the transition was NEW -> RESETTING -> LIVE.

This rings a bell. With the introduction of the fabrics drivers, the CONNECTING state was introduced and the PCI driver didn't use the CONNECTING state for while. But due to various reasons, the PCI transport follows the same state transitions as the fabrics ones now.

I am just a bit careful with the state machine, even obvious correct changes can introduce subtle regressions.

maintainer
#4 lore raw
Quoted 14 lines · Daniel Wagner
Quoted 8 lines · Maurizio Lombardi

"
Fix this by removing NVME_CTRL_NEW from the allowed prior states for the NVME_CTRL_RESETTING transition. It is safe to drop this transition entirely because a controller in the NVME_CTRL_NEW state is merely allocated and not yet initialized, a reset operation at this stage is meaningless.
"

Quoted 4 lines

For what concernes the "why it was there in the first place" question, I think that the reason is that in ancient times a CONNECTING state didn't exist. The controller initialization was done via a reset; so the transition was NEW -> RESETTING -> LIVE.

Quoted 4 lines

This rings a bell. With the introduction of the fabrics drivers, the CONNECTING state was introduced and the PCI driver didn't use the CONNECTING state for while. But due to various reasons, the PCI transport follows the same state transitions as the fabrics ones now.

Yep, nvme pci used to always kick most of the initialization to the reset_work, so that transport wanted the NEW->RESETTING transition. With asynchronous probe support added, we do the initial setup without reset_work.

maintainer Applied
#5 lore raw
In reply to Maurizio Lombardi
Quoted 12 lines · Maurizio Lombardi

During NVMe controller creation, the controller is exposed to sysfs via nvme_add_ctrl() while it is still in the NVME_CTRL_NEW state. This creates a narrow race window where a userspace process can write to the reset_controller sysfs node before the initialization thread transitions the state to NVME_CTRL_CONNECTING.

If a reset is triggered during this window, the state machine allows the transition from NVME_CTRL_NEW to NVME_CTRL_RESETTING, and the reset work is queued. However, the original creation thread continues its execution, subsequently moving the state to NVME_CTRL_CONNECTING and finally to NVME_CTRL_LIVE.

Quoted 7 lines

When the delayed reset work finally executes, it attempts to tear down the controller and transition the state to NVME_CTRL_CONNECTING. Because the state is now NVME_CTRL_LIVE, this transition fails, triggering a WARN_ON in the reset work.

Fix this by removing NVME_CTRL_NEW from the allowed prior states for the NVME_CTRL_RESETTING transition.

Thanks, applied to nvme-7.3.

#6 lore raw
In reply to Maurizio Lombardi
Quoted 32 lines · Maurizio Lombardi

During NVMe controller creation, the controller is exposed to sysfs via nvme_add_ctrl() while it is still in the NVME_CTRL_NEW state. This creates a narrow race window where a userspace process can write to the reset_controller sysfs node before the initialization thread transitions the state to NVME_CTRL_CONNECTING.

If a reset is triggered during this window, the state machine allows the transition from NVME_CTRL_NEW to NVME_CTRL_RESETTING, and the reset work is queued. However, the original creation thread continues its execution, subsequently moving the state to NVME_CTRL_CONNECTING and finally to NVME_CTRL_LIVE.

When the delayed reset work finally executes, it attempts to tear down the controller and transition the state to NVME_CTRL_CONNECTING. Because the state is now NVME_CTRL_LIVE, this transition fails, triggering a WARN_ON in the reset work.

Fix this by removing NVME_CTRL_NEW from the allowed prior states for the NVME_CTRL_RESETTING transition.

Fixes 8bfc3b4c6f9d ("nvmet: switch loopback target state to connecting when resetting")
Reported-by [email protected]
---
  drivers/nvme/host/core.c | 1 -
  1 file changed, 1 deletion(-)
drivers/nvme/host/core.c +0 -0
index beea23d04a70..7ea8dd27d08d 100644

drivers/nvme/host/core.c bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl, · line 603

Quoted 7 lines
drivers/nvme/host/core.c +0 -1
@@ -598,7 +598,6 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,
  		break;
  	case NVME_CTRL_RESETTING:
  		switch (old_state) {
-		case NVME_CTRL_NEW:
  		case NVME_CTRL_LIVE:
  			changed = true;
  			atomic_long_inc(&ctrl->nr_reset);

Hi!

I was testing something with linux-next 2026-10-02 and ran into an issue with the nvme-apple driver where /dev/nvme0 exists but none of the namespaces appear, and the dmesg is awfully quiet about the nvme device. I bisected it down to this patch, which as far as I can tell leads to the nvme-apple being stuck in NVME_CTRL_NEW state. Without being familiar with the matter, this looks like the nvme-apple driver was using this transition as part of its initialization procedure.  If one of the NVMe people who know how these transitions are supposed to work could have a look, that would be much appreciated, as otherwise this will break nvme-apple in 7.3-rc6.

Thanks,

- Yureka

#7 lore raw
Quoted 56 lines · Yureka Lilian
Quoted 32 lines · Maurizio Lombardi

During NVMe controller creation, the controller is exposed to sysfs via nvme_add_ctrl() while it is still in the NVME_CTRL_NEW state. This creates a narrow race window where a userspace process can write to the reset_controller sysfs node before the initialization thread transitions the state to NVME_CTRL_CONNECTING.

If a reset is triggered during this window, the state machine allows the transition from NVME_CTRL_NEW to NVME_CTRL_RESETTING, and the reset work is queued. However, the original creation thread continues its execution, subsequently moving the state to NVME_CTRL_CONNECTING and finally to NVME_CTRL_LIVE.

When the delayed reset work finally executes, it attempts to tear down the controller and transition the state to NVME_CTRL_CONNECTING. Because the state is now NVME_CTRL_LIVE, this transition fails, triggering a WARN_ON in the reset work.

Fix this by removing NVME_CTRL_NEW from the allowed prior states for the NVME_CTRL_RESETTING transition.

Fixes 8bfc3b4c6f9d ("nvmet: switch loopback target state to connecting when resetting")
Reported-by [email protected]
---
  drivers/nvme/host/core.c | 1 -
  1 file changed, 1 deletion(-)
drivers/nvme/host/core.c +0 -0
index beea23d04a70..7ea8dd27d08d 100644

drivers/nvme/host/core.c bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl, · line 603

Quoted 7 lines
drivers/nvme/host/core.c +0 -1
@@ -598,7 +598,6 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,
  		break;
  	case NVME_CTRL_RESETTING:
  		switch (old_state) {
-		case NVME_CTRL_NEW:
  		case NVME_CTRL_LIVE:
  			changed = true;
  			atomic_long_inc(&ctrl->nr_reset);

Hi!

I was testing something with linux-next 2026-10-02 and ran into an issue with the
nvme-apple driver where /dev/nvme0 exists but none of the namespaces appear, and the dmesg is awfully quiet about the nvme device. I bisected it down to this patch, which as far as I can tell leads to the nvme-apple being stuck in
NVME_CTRL_NEW state. Without being familiar with the matter, this looks like the nvme-apple driver was using this transition as part of its initialization
procedure.  If one of the NVMe people who know how these transitions are supposed to work could have a look, that would be much appreciated, as otherwise this will break nvme-apple in 7.3-rc6.

Quoted 1 line

Thanks,

Thanks for reporting this,
it seems that nvme-apple is the only driver that still initializes the controller by performing a reset directly from a NEW state.

This should be easy to fix by moving the state to CONNECTING in apple_nvme_probe, just before nvme_reset_ctrl() is called.

Would you be able to test the patch if I prepare it?

Maurizio

#8 lore raw
Quoted 6 lines · Maurizio Lombardi

Thanks for reporting this,
it seems that nvme-apple is the only driver that still initializes the controller by performing a reset directly from a NEW state.

This should be easy to fix by moving the state to CONNECTING in apple_nvme_probe, just before nvme_reset_ctrl() is called.

Ops, this won't work, as we can enter the resetting state only when the controller is live.

Keith, I think we have no other option but revert this patch to unbreak apple-nvme until I find a proper fix.

Maurizio

#9 lore raw
Quoted 7 lines · Maurizio Lombardi
Quoted 6 lines · Maurizio Lombardi

Thanks for reporting this,
it seems that nvme-apple is the only driver that still initializes the controller by performing a reset directly from a NEW state.

This should be easy to fix by moving the state to CONNECTING in apple_nvme_probe, just before nvme_reset_ctrl() is called.

Quoted 7 lines

Ops, this won't work, as we can enter the resetting state only when the controller is live.

Keith, I think we have no other option but revert this patch to unbreak apple-nvme until I find a proper fix.

Maurizio

When you eventually propose a new variant of this, I can certainly test it with nvme-apple.

- Yureka

maintainer
#10 lore raw
Quoted 11 lines · Yureka Lilian
Quoted 10 lines · Maurizio Lombardi
Quoted 6 lines · Maurizio Lombardi

Thanks for reporting this,
it seems that nvme-apple is the only driver that still initializes the controller by performing a reset directly from a NEW state.

This should be easy to fix by moving the state to CONNECTING in apple_nvme_probe, just before nvme_reset_ctrl() is called.

Ops, this won't work, as we can enter the resetting state only when the controller is live.

Quoted 8 lines
Quoted 4 lines

Keith, I think we have no other option but revert this patch to unbreak apple-nvme until I find a proper fix.

Maurizio

When you eventually propose a new variant of this, I can certainly test it with nvme-apple.

Thanks for the notice. Patch reverted committed to nvme-7.3 and will send a pull request shortly.