nvme-rdma: fix controller reset hang during traffic
Sagi Grimberg
maintainerPATCH v2 2/2 nvme-rdma: fix controller reset hang during traffic
commit fe35ec58f0d3 ("block: update hctx map when use multiple maps") exposed an issue where we may hang trying to wait for queue freeze during I/O. We call blk_mq_update_nr_hw_queues which in case of multiple queue maps (which we have now for default/read/poll) is attempting to freeze the queue. However we never started queue freeze when starting the reset, which means that we have inflight pending requests that entered the queue that we will not complete once the queue is quiesced.
So start a freeze before we quiesce the queue, and unfreeze the queue after we successfully connected the I/O queues (and make sure to call blk_mq_update_nr_hw_queues only after we are sure that the queue was already frozen).
This follows to how the pci driver handles resets.
---
Changes from v1:
- fix silly compilation errors
drivers/nvme/host/rdma.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
drivers/nvme/host/rdma.c +9 -3
@@ -967,15 +967,20 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new) ret =PTR_ERR (ctrl->ctrl.connect_q); goto out_free_tag_set; } - }else { -blk_mq_update_nr_hw_queues (&ctrl->tag_set, - ctrl->ctrl.queue_count -1 ); } ret =nvme_rdma_start_io_queues (ctrl);if (ret) goto out_cleanup_connect_q; +if (!new) { +nvme_start_queues (&ctrl->ctrl); +nvme_wait_freeze (&ctrl->ctrl); +blk_mq_update_nr_hw_queues (ctrl->ctrl.tagset, + ctrl->ctrl.queue_count -1 ); +nvme_unfreeze (&ctrl->ctrl); + } +return 0 ; out_cleanup_connect_q:
@@ -1008,6 +1013,7 @@ static void nvme_rdma_teardown_io_queues(struct nvme_rdma_ctrl *ctrl, bool remove) {if (ctrl->ctrl.queue_count >1 ) { +nvme_start_freeze (&ctrl->ctrl);nvme_stop_queues (&ctrl->ctrl);nvme_rdma_stop_io_queues (ctrl);if (ctrl->ctrl.tagset) {
Signature · 7 lines
2.25.1 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Christoph Hellwig
maintainerWhy is this a 2/2 v2, where is the 1/2?
Quoted 61 lines · Sagi Grimberg
commit
fe35ec58f0d3("block: update hctx map when use multiple maps") exposed an issue where we may hang trying to wait for queue freeze during I/O. We callblk_mq_update_nr_hw_queueswhich in case of multiple queue maps (which we have now for default/read/poll) is attempting to freeze the queue. However we never started queue freeze when starting the reset, which means that we have inflight pending requests that entered the queue that we will not complete once the queue is quiesced.So start a freeze before we quiesce the queue, and unfreeze the queue after we successfully connected the I/O queues (and make sure to call
blk_mq_update_nr_hw_queuesonly after we are sure that the queue was already frozen).This follows to how the pci driver handles resets.
Signed-off-by Sagi Grimberg <[email protected]>---
Changes from v1:
- fix silly compilation errorsdrivers/nvme/host/rdma.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)drivers/nvme/host/rdma.c +9
-3@@ -967,15 +967,20 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new) ret =PTR_ERR (ctrl->ctrl.connect_q); goto out_free_tag_set; } - }else { -blk_mq_update_nr_hw_queues (&ctrl->tag_set, - ctrl->ctrl.queue_count -1 ); } ret =nvme_rdma_start_io_queues (ctrl);if (ret) goto out_cleanup_connect_q; +if (!new) { +nvme_start_queues (&ctrl->ctrl); +nvme_wait_freeze (&ctrl->ctrl); +blk_mq_update_nr_hw_queues (ctrl->ctrl.tagset, + ctrl->ctrl.queue_count -1 ); +nvme_unfreeze (&ctrl->ctrl); + } +return 0 ; out_cleanup_connect_q:@@ -1008,6 +1013,7 @@ static void nvme_rdma_teardown_io_queues(struct nvme_rdma_ctrl *ctrl, bool remove) {if (ctrl->ctrl.queue_count >1 ) { +nvme_start_freeze (&ctrl->ctrl);nvme_stop_queues (&ctrl->ctrl);nvme_rdma_stop_io_queues (ctrl);if (ctrl->ctrl.tagset) {Signature · 1 line
2.25.1
---end quoted text---
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerQuoted 2 lines
Why is this a 2/2 v2, where is the 1/2?
Sent on 7/24/20, 3:10 PM (PST)
Along with:
[PATCH 1/2] nvme-tcp: fix controller reset hang during traffic
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Applied to nvme-5.9 and added a fixes tag.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme