nvme-core: fix deadlock when delete ctrl due to reconnect fail
PATCH 1/1 nvme-core: fix deadlock when delete ctrl due to reconnect fail
A deadlock happens when test link blink for nvme over roce. If time out in reconneting process, nvme_rdma_timeout->nvme_rdma_teardown_io_queues will quiesce the io queues, and then the ctrl will be deleted after reconnect times exceed max_reconnects. If run fdisk from the time when the queue is quiesced to the time when the ctrl is deleted, delete ctrl will deadlock, the process: nvme_do_delete_ctrl-> nvme_remove_namespaces->nvme_ns_remove->blk_cleanup_queue-> blk_freeze_queue->blk_mq_freeze_queue_wait, blk_mq_freeze_queue_wait will wait until q_usage_counter of queue become 0, but the queue is quiesced, can not clean any request.
Solution: nvme_rdma_timeout should call nvme_start_queues after call nvme_rdma_teardown_io_queues. further more, we need start queues regardless of whether the remove flag is set, after cancel requests in nvme_rdma_teardown_io_queues.
--- drivers/nvme/host/rdma.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
drivers/nvme/host/rdma.c +1 -3
@@ -989,8 +989,7 @@ static void nvme_rdma_teardown_io_queues(struct nvme_rdma_ctrl *ctrl, nvme_cancel_request, &ctrl->ctrl);blk_mq_tagset_wait_completed_request (ctrl->ctrl.tagset); } -if (remove) -nvme_start_queues (&ctrl->ctrl); +nvme_start_queues (&ctrl->ctrl);nvme_rdma_destroy_io_queues (ctrl, remove); } }
@@ -1128,7 +1127,6 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)nvme_stop_keep_alive (&ctrl->ctrl);nvme_rdma_teardown_io_queues (ctrl, false); -nvme_start_queues (&ctrl->ctrl);nvme_rdma_teardown_admin_queue (ctrl, false);blk_mq_unquiesce_queue (ctrl->ctrl.admin_q);
Signature · 7 lines
2.16.4 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Title is error. Ignore this email, see next email.
Quoted 37 lines · Chao Leng
A deadlock happens when test link blink for nvme over roce. If time out in reconneting process,
nvme_rdma_timeout->nvme_rdma_teardown_io_queueswill quiesce the io queues, and then the ctrl will be deleted after reconnect times exceedmax_reconnects. If run fdisk from the time when the queue is quiesced to the time when the ctrl is deleted, delete ctrl will deadlock, the process: nvme_do_delete_ctrl-> nvme_remove_namespaces->nvme_ns_remove->blk_cleanup_queue->blk_freeze_queue->blk_mq_freeze_queue_wait,blk_mq_freeze_queue_waitwill wait untilq_usage_counterof queue become 0, but the queue is quiesced, can not clean any request.Solution:
nvme_rdma_timeoutshould callnvme_start_queuesafter callnvme_rdma_teardown_io_queues. further more, we need start queues regardless of whether the remove flag is set, after cancel requests innvme_rdma_teardown_io_queues.Signed-off-by Chao Leng <[email protected]>--- drivers/nvme/host/rdma.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)drivers/nvme/host/rdma.c +1
-2@@ -989,8 +989,7 @@ static void nvme_rdma_teardown_io_queues(struct nvme_rdma_ctrl *ctrl, nvme_cancel_request, &ctrl->ctrl);blk_mq_tagset_wait_completed_request (ctrl->ctrl.tagset); } -if (remove) -nvme_start_queues (&ctrl->ctrl); +nvme_start_queues (&ctrl->ctrl);nvme_rdma_destroy_io_queues (ctrl, remove); } }
drivers/nvme/host/rdma.c static void nvme_rdma_error_recovery_work(struct work_struct *work) · line 1132
Quoted 7 lines
drivers/nvme/host/rdma.c +0
-1@@ -1128,7 +1127,6 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)nvme_stop_keep_alive (&ctrl->ctrl);nvme_rdma_teardown_io_queues (ctrl, false); -nvme_start_queues (&ctrl->ctrl);nvme_rdma_teardown_admin_queue (ctrl, false);blk_mq_unquiesce_queue (ctrl->ctrl.admin_q);
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme