Skip to contentSkip to conversation

nvme: delay failover by command quiesce timeout

RFC PATCH Jiewei Ke 3 participants 3 messages Apr 10, 2025 – Apr 15, 2025 NVMe

lore
Share & tools
lore

Jiewei Ke

author
#0 lore raw

Hi Daniel,

I just noticed that your patchset addresses a similar issue to the one I ’m trying to solve with my recently submitted patchset [1]. Compared to your approach, mine differs in a few key aspects:

1. Only aborted requests are delayed for retry. In the current implementation, nvmf_complete_timed_out_request and nvme_cancel_request set the request status to NVME_SC_HOST_ABORTED_CMD. These requests are usually already sent to the target, but may have timed out or been canceled before a response is received. Since the target may still be processing them, the host needs to delay retrying to ensure the target has completed or cleaned up the stale requests. On the other hand, requests that are not aborted — such as those that never got submitted due to no usable path (e.g., from nvme_ns_head_submit_bio), or those that already received an ANA error from the target — do not need delayed retry.

2. The host explicitly disconnects and stops KeepAlive before delay scheduling retrying requests. This aligns with Section 9.6 "Communication Loss Handling" of the NVMe Base Specification 2.1. Once the host disconnects, the target may take up to the KATO interval to detect the lost connection and begin cleaning up any remaining requests. Retrying too early may still lead to data corruption issues.

@@ -2178,6 +2180,7 @@ static void nvme_rdma_shutdown_ctrl(struct nvme_rdma_ctrl *ctrl, bool shutdown)
nvme_quiesce_admin_queue(&ctrl->ctrl);
nvme_disable_ctrl(&ctrl->ctrl, shutdown); nvme_rdma_teardown_admin_queue(ctrl, shutdown);
+ nvme_delay_kick_retry_lists(&ctrl->ctrl); <<< delay kick retry after teardown all queues
}

3. Delayed retry of aborted requests is supported across multiple scenarios, including error recovery work, controller reset, controller deletion, and controller reconnect failure handling. Here's the relevant code for reference.

drivers/nvme/host/tcp.c +2 -0
index 9109d5476417..f07b3960df7c 100644
@@ -2449,6 +2449,7 @@ static int nvme_tcp_setup_ctrl(struct nvme_ctrl *ctrl, bool new)
 destroy_admin:
 	nvme_stop_keep_alive(ctrl);
 	nvme_tcp_teardown_admin_queue(ctrl, new);
+	nvme_delay_kick_retry_lists(ctrl); <<< requests may be timed out when ctrl reconnects
 	return ret;
 }
 
@@ -2494,6 +2495,7 @@ static void nvme_tcp_error_recovery_work(struct work_struct *work)
 	nvme_tcp_teardown_admin_queue(ctrl, false);
 	nvme_unquiesce_admin_queue(ctrl);
 	nvme_auth_stop(ctrl);
+	nvme_delay_kick_retry_lists(ctrl); <<< retry_lists may contain timed out or cancelled requests
 
 	if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_CONNECTING)) {
 		/* state change failure is ok if we started ctrl delete */

@@ -2513,6 +2515,7 @@ static void nvme_tcp_teardown_ctrl(struct nvme_ctrl *ctrl, bool shutdown)
nvme_quiesce_admin_queue(ctrl);
nvme_disable_ctrl(ctrl, shutdown);
nvme_tcp_teardown_admin_queue(ctrl, shutdown);
+ nvme_delay_kick_retry_lists(ctrl); <<< retry_lists may contain timed out or cancelled requests when ctrl reset or delete
}

Besides, in nvme_tcp_error_recovery_work, the delayed retry must occur after nvme_tcp_teardown_io_queues, because the teardown cancels requests that may need to be retried too.

One limitation of my patchset is that it does not yet include full CQT support, and due to testing environment constraints, only nvme_tcp and nvme_rdma are currently covered.

I’d be happy to discuss the pros and cons of both approaches -- perhaps we can combine the best aspects.

Looking forward to your thoughts.

Thanks,
Jiewei

[1] https://lore.kernel.org/linux-nvme/[email protected]/

#1 lore raw

Hi Jiewei,

Quoted 4 lines · Jiewei Ke

I just noticed that your patchset addresses a similar issue to the one I ’m trying to solve with my recently submitted patchset [1]. Compared to your approach, mine differs in a few key aspects:

Quoted 10 lines

1. Only aborted requests are delayed for retry. In the current implementation, nvmf_complete_timed_out_request and nvme_cancel_request set the request status to NVME_SC_HOST_ABORTED_CMD. These requests are usually already sent to the target, but may have timed out or been canceled before a response is received. Since the target may still be processing them, the host needs to delay retrying to ensure the target has completed or cleaned up the stale requests. On the other hand, requests that are not aborted — such as those that never got submitted due to no usable path (e.g., from nvme_ns_head_submit_bio), or those that already received an ANA error from the target — do not need delayed retry.

If I understand you correctly, you are concerned about delaying all commands even these which are not transmitted yet. If there are no garantees on ordering it would be possible to failover these commands immediately. Sure something which could be improved in my series.

Quoted 6 lines

2. The host explicitly disconnects and stops KeepAlive before delay scheduling retrying requests. This aligns with Section 9.6 "Communication Loss Handling" of the NVMe Base Specification 2.1. Once the host disconnects, the target may take up to the KATO interval to detect the lost connection and begin cleaning up any remaining requests. Retrying too early may still lead to data corruption issues.

The host error handler is executed thus all communication is stopped. This part hasn't changed. Not sure what you are referring too. The only thing which changes in this series is when we enqueue the failed commands again.

Review static void nvme_rdma_shutdown_ctrl(struct nvme_rdma_ctrl *ctrl, bool shutdown) · line 2184

Quoted 6 lines
(quoted diff) +1 -0
@@ -2178,6 +2180,7 @@ static void nvme_rdma_shutdown_ctrl(struct nvme_rdma_ctrl *ctrl, bool shutdown)
 	nvme_quiesce_admin_queue(&ctrl->ctrl);
 	nvme_disable_ctrl(&ctrl->ctrl, shutdown);
 	nvme_rdma_teardown_admin_queue(ctrl, shutdown);
+	nvme_delay_kick_retry_lists(&ctrl->ctrl); <<< delay kick retry
 	after teardown all queues

Without the kick it hangs. The admin has explicitly removed the ctrl.

As I already said, this is a RFC for the sake to figure out if this approach is a good idea. We all agreed, we should first try to sort this out before introducing a new queue. There are many problems which it will introduce, like the one from above 'why delaying not send requests?', 'What happens when we have several short disconnects/connects?', ...

BTW, there is already a hack in disconnect/connect state transition. Ideally we solve this in a more generic manner.

Thanks,
Daniel

#2 lore raw

Hi Daniel,

Quoted 20 lines

2025年4月15日 20:34,Daniel Wagner <[email protected]> 写道:

Hi Jiewei,

Quoted 4 lines · Jiewei Ke

I just noticed that your patchset addresses a similar issue to the one I ’m trying to solve with my recently submitted patchset [1]. Compared to your approach, mine differs in a few key aspects:

Quoted 10 lines

1. Only aborted requests are delayed for retry. In the current implementation, nvmf_complete_timed_out_request and nvme_cancel_request set the request status to NVME_SC_HOST_ABORTED_CMD. These requests are usually already sent to the target, but may have timed out or been canceled before a response is received. Since the target may still be processing them, the host needs to delay retrying to ensure the target has completed or cleaned up the stale requests. On the other hand, requests that are not aborted — such as those that never got submitted due to no usable path (e.g., from nvme_ns_head_submit_bio), or those that already received an ANA error from the target — do not need delayed retry.

Quoted 4 lines

If I understand you correctly, you are concerned about delaying all commands even these which are not transmitted yet. If there are no garantees on ordering it would be possible to failover these commands immediately. Sure something which could be improved in my series.

Yes, your patchset delays all commands. In fact, some distinction needs to be made:
commands with path errors and those that haven't been sent yet can be failed over immediately. You might consider applying the delay only to commands with status NVME_SC_HOST_ABORTED_CMD.

Quoted 11 lines
Quoted 6 lines

2. The host explicitly disconnects and stops KeepAlive before delay scheduling retrying requests. This aligns with Section 9.6 "Communication Loss Handling" of the NVMe Base Specification 2.1. Once the host disconnects, the target may take up to the KATO interval to detect the lost connection and begin cleaning up any remaining requests. Retrying too early may still lead to data corruption issues.

The host error handler is executed thus all communication is stopped. This part hasn't changed. Not sure what you are referring too. The only thing which changes in this series is when we enqueue the failed commands again.

In your patch as shown below, I think nvme_schedule_failover should be called after nvme_rdma_teardown_admin_queue for two reasons:

* nvme_rdma_teardown_io_queues will call nvme_cancel_tagset, and then the inflight commands will be added to the ctrl's request_list. So nvme_schedule_failover should be after nvme_rdma_teardown_io_queues to kick them.

* This does not follow the rule of starting the delay timer(aka nvme_schedule_failover) after all communication is stopped.

Quoted 26 lines
(quoted diff) +1 -0
@@ -2153,6 +2154,7 @@ static const struct blk_mq_ops nvme_rdma_admin_mq_ops = {
 
 static void nvme_rdma_shutdown_ctrl(struct nvme_rdma_ctrl *ctrl, bool shutdown)
 {
+ nvme_schedule_failover(&ctrl->ctrl);
 nvme_rdma_teardown_io_queues(ctrl, shutdown);
 nvme_quiesce_admin_queue(&ctrl->ctrl);
 nvme_disable_ctrl(&ctrl->ctrl, shutdown);

nvme_rdma_teardown_admin_queue(ctrl, shutdown);

Quoted 6 lines
(quoted diff) +1 -0
@@ -2178,6 +2180,7 @@ static void nvme_rdma_shutdown_ctrl(struct nvme_rdma_ctrl *ctrl, bool shutdown)
 nvme_quiesce_admin_queue(&ctrl->ctrl);
 nvme_disable_ctrl(&ctrl->ctrl, shutdown);
 nvme_rdma_teardown_admin_queue(ctrl, shutdown);
+ nvme_delay_kick_retry_lists(&ctrl->ctrl); <<< delay kick retry

after teardown all queues

Without the kick it hangs. The admin has explicitly removed the ctrl.

As I already said, this is a RFC for the sake to figure out if this approach is a good idea. We all agreed, we should first try to sort this out before introducing a new queue. There are many problems which it will introduce, like the one from above 'why delaying not send requests?', 'What happens when we have several short disconnects/connects?', ...

Quoted 2 lines

BTW, there is already a hack in disconnect/connect state transition. Ideally we solve this in a more generic manner.

I agree that introducing a controller-level queue is necessary, these commands pending retry should be uniformly delayed after the controller finishes disconnecting.

Regarding the issue of multiple connections and disconnections, in my patchset I avoided using delayed_work, and instead used a synchronous msleep approach. This prevents premature retries.

If we use delayed_work, the following issue may arise: during a controller reset, suppose an inflight command (command 1) is canceled and added to the controller’s request_list, then the connection is torn down and nvme_schedule_failover is called. Command 1 will be scheduled to retry after 30 seconds. And then the reconnection completes, but command 2 encounters an I/O timeout and is also added to the request_list. At this point, the 30-second delay expires, and command 2 might be mistakenly retried immediately.

The current state machine cannot avoid this issue if using delayed_work.

Thanks,
Jiewei