nvme-core: fix deadlock when reconnect failed due to nvme_set_queue_count timeout
Subsystems (2)
PATCH 2/3 nvme-core: fix deadlock when reconnect failed due to nvme_set_queue_count timeout
A deadlock happens When we test nvme over roce with link blink. The
reason: link blink will cause error recovery, and then reconnect.If reconnect fail due to nvme_set_queue_count timeout, the reconnect process will set the queue count as 0 and continue , and then nvme_start_ctrl will call nvme_enable_aen, and deadlock happens because the admin queue is quiesced.
log:
Aug 3 22:47:24 localhost kernel: nvme nvme2: I/O 22 QID 0 timeout Aug 3 22:47:24 localhost kernel: nvme nvme2: Could not set queue count
(881)
stack:
root 23848 0.0 0.0 0 0 ? D Aug03 0:00
[kworker/u12:4+nvme-wq]
[<0>] blk_execute_rq+0x69/0xa0 [<0>] __nvme_submit_sync_cmd+0xaf/0x1b0 [nvme_core] [<0>] nvme_features+0x73/0xb0 [nvme_core] [<0>] nvme_start_ctrl+0xa4/0x100 [nvme_core] [<0>] nvme_rdma_setup_ctrl+0x438/0x700 [nvme_rdma] [<0>] nvme_rdma_reconnect_ctrl_work+0x22/0x30 [nvme_rdma] [<0>] process_one_work+0x1a7/0x370 [<0>] worker_thread+0x30/0x380 [<0>] kthread+0x112/0x130 [<0>] ret_from_fork+0x35/0x40
Many functions which call __nvme_submit_sync_cmd treat error code in two
modes: If error code less than 0, treat as command failed. If erroe code more than 0, treat as target not support or other and continue. NVME_SC_HOST_ABORTED_CMD and NVME_SC_HOST_PATH_ERROR both are cancled io by host, is not the real error code return from target. So we need set the flag:NVME_REQ_CANCELLED. Thus __nvme_submit_sync_cmd translate the error to INTR, nvme_set_queue_count will return error, reconnect process will terminate instead of continue.
--- drivers/nvme/host/core.c | 1 + drivers/nvme/host/fabrics.c | 1 + 2 files changed, 2 insertions(+)
drivers/nvme/host/core.c +1 -0
@@ -307,6 +307,7 @@ bool nvme_cancel_request(struct request *req, void *data, bool reserved)if (blk_mq_request_completed (req))return true; +nvme_req (req)->flags |=NVME_REQ_CANCELLED ;nvme_req (req)->status =NVME_SC_HOST_ABORTED_CMD ;blk_mq_complete_request (req);return true;
drivers/nvme/host/fabrics.c +1 -0
@@ -552,6 +552,7 @@ blk_status_t nvmf_fail_nonready_command(struct nvme_ctrl *ctrl, !blk_noretry_request(rq) && !(rq->cmd_flags & REQ_NVME_MPATH))return BLK_STS_RESOURCE ; +nvme_req (rq)->flags |=NVME_REQ_CANCELLED ;nvme_req (rq)->status =NVME_SC_HOST_PATH_ERROR ;blk_mq_start_request (rq);nvme_complete_rq (rq);
Signature · 1 line
2.16.4
Christoph Hellwig
maintainerQuoted 25 lines · Chao Leng
A deadlock happens When we test nvme over roce with link blink. The
reason: link blink will cause error recovery, and then reconnect.If reconnect fail due tonvme_set_queue_counttimeout, the reconnect process will set the queue count as 0 and continue , and thennvme_start_ctrlwill callnvme_enable_aen, and deadlock happens because the admin queue is quiesced.log:
Aug 3 22:47:24 localhost kernel: nvme nvme2: I/O 22 QID 0 timeout Aug 3 22:47:24 localhost kernel: nvme nvme2: Could not set queue count(881)
stack:
root 23848 0.0 0.0 0 0 ? D Aug03 0:00
[kworker/u12:4+nvme-wq][<0>] blk_execute_rq+0x69/0xa0 [<0>] __nvme_submit_sync_cmd+0xaf/0x1b0 [nvme_core] [<0>] nvme_features+0x73/0xb0 [nvme_core] [<0>] nvme_start_ctrl+0xa4/0x100 [nvme_core] [<0>] nvme_rdma_setup_ctrl+0x438/0x700 [nvme_rdma] [<0>] nvme_rdma_reconnect_ctrl_work+0x22/0x30 [nvme_rdma] [<0>] process_one_work+0x1a7/0x370 [<0>] worker_thread+0x30/0x380 [<0>] kthread+0x112/0x130 [<0>] ret_from_fork+0x35/0x40
Quoted 8 lines
Many functions which call
__nvme_submit_sync_cmdtreat error code in two
modes: If error code less than 0, treat as command failed. If erroe code more than 0, treat as target not support or other and continue.NVME_SC_HOST_ABORTED_CMDandNVME_SC_HOST_PATH_ERRORboth are cancled io by host, is not the real error code return from target. So we need set the flag:NVME_REQ_CANCELLED. Thus__nvme_submit_sync_cmdtranslate the error to INTR,nvme_set_queue_countwill return error, reconnect process will terminate instead of continue.
But we could still race with a real completion. I suspect the right answer is to translate NVME_SC_HOST_ABORTED_CMD and NVME_SC_HOST_PATH_ERROR to a negative error code in __nvme_submit_sync_cmd.
Sagi Grimberg
maintainerQuoted 35 lines
Quoted 26 lines
A deadlock happens When we test nvme over roce with link blink. The
reason: link blink will cause error recovery, and then reconnect.If reconnect fail due tonvme_set_queue_counttimeout, the reconnect process will set the queue count as 0 and continue , and thennvme_start_ctrlwill callnvme_enable_aen, and deadlock happens because the admin queue is quiesced.log:
Aug 3 22:47:24 localhost kernel: nvme nvme2: I/O 22 QID 0 timeout Aug 3 22:47:24 localhost kernel: nvme nvme2: Could not set queue count(881)
stack:
root 23848 0.0 0.0 0 0 ? D Aug03 0:00
[kworker/u12:4+nvme-wq][<0>] blk_execute_rq+0x69/0xa0 [<0>] __nvme_submit_sync_cmd+0xaf/0x1b0 [nvme_core] [<0>] nvme_features+0x73/0xb0 [nvme_core] [<0>] nvme_start_ctrl+0xa4/0x100 [nvme_core] [<0>] nvme_rdma_setup_ctrl+0x438/0x700 [nvme_rdma] [<0>] nvme_rdma_reconnect_ctrl_work+0x22/0x30 [nvme_rdma] [<0>] process_one_work+0x1a7/0x370 [<0>] worker_thread+0x30/0x380 [<0>] kthread+0x112/0x130 [<0>] ret_from_fork+0x35/0x40Quoted 8 lines
Many functions which call
__nvme_submit_sync_cmdtreat error code in two
modes: If error code less than 0, treat as command failed. If erroe code more than 0, treat as target not support or other and continue.NVME_SC_HOST_ABORTED_CMDandNVME_SC_HOST_PATH_ERRORboth are cancled io by host, is not the real error code return from target. So we need set the flag:NVME_REQ_CANCELLED. Thus__nvme_submit_sync_cmdtranslate the error to INTR,nvme_set_queue_countwill return error, reconnect process will terminate instead of continue.
Quoted 4 lines
But we could still race with a real completion. I suspect the right answer is to translate
NVME_SC_HOST_ABORTED_CMDandNVME_SC_HOST_PATH_ERRORto a negative error code in__nvme_submit_sync_cmd.
So the scheme you suggest is:
- treat any negative status or !DNR as "we never made it to the target"
- Any positive status with DNR is a "controller generated status"
This will need a careful audit of all the call-sites we place such assumptions...
Christoph Hellwig
maintainerQuoted 14 lines · Sagi Grimberg
Quoted 9 lines
Quoted 8 lines
Many functions which call
__nvme_submit_sync_cmdtreat error code in two
modes: If error code less than 0, treat as command failed. If erroe code more than 0, treat as target not support or other and continue.NVME_SC_HOST_ABORTED_CMDandNVME_SC_HOST_PATH_ERRORboth are cancled io by host, is not the real error code return from target. So we need set the flag:NVME_REQ_CANCELLED. Thus__nvme_submit_sync_cmdtranslate the error to INTR,nvme_set_queue_countwill return error, reconnect process will terminate instead of continue.Quoted 4 lines
But we could still race with a real completion. I suspect the right answer is to translate
NVME_SC_HOST_ABORTED_CMDandNVME_SC_HOST_PATH_ERRORto a negative error code in__nvme_submit_sync_cmd.
Quoted 7 lines
So the scheme you suggest is:
- treat any negative status or !DNR as "we never made it to the target"
- Any positive status with DNR is a "controller generated status"This will need a careful audit of all the call-sites we place such assumptions...
No. negative error means never made it to the controller, and we need to map the magic NVME_SC_HOST_ABORTED_CMD and NVME_SC_HOST_PATH_ERROR errors to negative error codes if we want to keep using them (and IIRC we started because it solved an issue, by my memory is foggy). All the real NVMe status codes come from the controller, so the commmand must have made it there.