Skip to contentSkip to conversation

nvme-rdma: fix deadlock when delete ctrl due to reconnect fail

PATCH In review Chao Leng 2 participants 6 messages Jul 27, 2020 – Jul 28, 2020 1 message not in the archive NVMe

lore
Share & tools
lore

Chao Leng

author
#0 lore raw

PATCH 1/1 nvme-rdma: fix deadlock when delete ctrl due to reconnect fail

A deadlock happens when test link blink for nvme over roce. If time out in reconneting process, nvme_rdma_timeout->nvme_rdma_teardown_io_queues will quiesce the io queues, and then the ctrl will be deleted after reconnect times exceed max_reconnects. If run fdisk from the time when the queue is quiesced to the time when the ctrl is deleted, delete ctrl will deadlock, the process: nvme_do_delete_ctrl-> nvme_remove_namespaces->nvme_ns_remove->blk_cleanup_queue-> blk_freeze_queue->blk_mq_freeze_queue_wait, blk_mq_freeze_queue_wait will wait until q_usage_counter of queue become 0, but the queue is quiesced, can not clean any request.

Solution: nvme_rdma_timeout should call nvme_start_queues after call nvme_rdma_teardown_io_queues. further more, we need start queues regardless of whether the remove flag is set, after cancel requests in nvme_rdma_teardown_io_queues.

---
 drivers/nvme/host/rdma.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
drivers/nvme/host/rdma.c +1 -3
index f8f856dc0c67..b381e2cde50a 100644
@@ -989,8 +989,7 @@ static void nvme_rdma_teardown_io_queues(struct nvme_rdma_ctrl *ctrl,
 				nvme_cancel_request, &ctrl->ctrl);
 			blk_mq_tagset_wait_completed_request(ctrl->ctrl.tagset);
 		}
-		if (remove)
-			nvme_start_queues(&ctrl->ctrl);
+		nvme_start_queues(&ctrl->ctrl);
 		nvme_rdma_destroy_io_queues(ctrl, remove);
 	}
 }
@@ -1128,7 +1127,6 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)
 
 	nvme_stop_keep_alive(&ctrl->ctrl);
 	nvme_rdma_teardown_io_queues(ctrl, false);
-	nvme_start_queues(&ctrl->ctrl);
 	nvme_rdma_teardown_admin_queue(ctrl, false);
 	blk_mq_unquiesce_queue(ctrl->ctrl.admin_q);
 
Signature · 7 lines
2.16.4


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
maintainer
#1 lore raw
Quoted 27 lines

A deadlock happens when test link blink for nvme over roce. If time out in reconneting process, nvme_rdma_timeout->nvme_rdma_teardown_io_queues will quiesce the io queues, and then the ctrl will be deleted after reconnect times exceed max_reconnects. If run fdisk from the time when the queue is quiesced to the time when the ctrl is deleted, delete ctrl will deadlock, the process: nvme_do_delete_ctrl-> nvme_remove_namespaces->nvme_ns_remove->blk_cleanup_queue-> blk_freeze_queue->blk_mq_freeze_queue_wait, blk_mq_freeze_queue_wait will wait until q_usage_counter of queue become 0, but the queue is quiesced, can not clean any request.

Solution: nvme_rdma_timeout should call nvme_start_queues after call nvme_rdma_teardown_io_queues. further more, we need start queues regardless of whether the remove flag is set, after cancel requests in nvme_rdma_teardown_io_queues.

---
  drivers/nvme/host/rdma.c | 4 +---
  1 file changed, 1 insertion(+), 3 deletions(-)
drivers/nvme/host/rdma.c +0 -0
index f8f856dc0c67..b381e2cde50a 100644

drivers/nvme/host/rdma.c static void nvme_rdma_teardown_io_queues(struct nvme_rdma_ctrl *ctrl, · line 992

Quoted 6 lines
drivers/nvme/host/rdma.c +1 -2
@@ -989,8 +989,7 @@ static void nvme_rdma_teardown_io_queues(struct nvme_rdma_ctrl *ctrl,
  				nvme_cancel_request, &ctrl->ctrl);
  			blk_mq_tagset_wait_completed_request(ctrl->ctrl.tagset);
  		}
-		if (remove)
-			nvme_start_queues(&ctrl->ctrl);
+		nvme_start_queues(&ctrl->ctrl);

This will fail I/O during controller reset, so nak on this.

drivers/nvme/host/rdma.c static void nvme_rdma_error_recovery_work(struct work_struct *work) · line 1132

Quoted 12 lines
drivers/nvme/host/rdma.c +0 -1
  		nvme_rdma_destroy_io_queues(ctrl, remove);
  	}
  }
@@ -1128,7 +1127,6 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)
 
  	nvme_stop_keep_alive(&ctrl->ctrl);
  	nvme_rdma_teardown_io_queues(ctrl, false);
-	nvme_start_queues(&ctrl->ctrl);
  	nvme_rdma_teardown_admin_queue(ctrl, false);
  	blk_mq_unquiesce_queue(ctrl->ctrl.admin_q);
 

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#2 lore raw
Quoted 15 lines

drivers/nvme/host/rdma.c static void nvme_rdma_teardown_io_queues(struct · line 993

Quoted 13 lines
drivers/nvme/host/rdma.c +1 -2
index f8f856dc0c67..b381e2cde50a 100644
@@ -989,8 +989,7 @@ static void nvme_rdma_teardown_io_queues(struct 
 nvme_rdma_ctrl *ctrl,
                   nvme_cancel_request, &ctrl->ctrl);
               blk_mq_tagset_wait_completed_request(ctrl->ctrl.tagset);
           }
-        if (remove)
-            nvme_start_queues(&ctrl->ctrl);
+        nvme_start_queues(&ctrl->ctrl);

This will fail I/O during controller reset, so nak on this.

Can you try this:
--

drivers/nvme/host/rdma.c +5 -0
index d58231636d11..96c0d664fe9b 100644
@@ -1149,6 +1149,11 @@ static void nvme_rdma_reconnect_ctrl_work(struct work_struct *work)
        return;
 
 requeue:
+       /*
+        * make sure queues are not quiesced due to a reconnect
+        * sequence that failed after creating some I/O queues
+        */
+       nvme_start_queues(ctrl);
        dev_info(ctrl->ctrl.device, "Failed reconnect attempt %d\n",
                        ctrl->ctrl.nr_reconnects);
        nvme_rdma_reconnect_or_remove(ctrl);

--

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

author
#3 lore raw
Quoted 14 lines · Sagi Grimberg
Quoted 14 lines

drivers/nvme/host/rdma.c static void nvme_rdma_teardown_io_queues(struct nvme_rdma_ctrl *ctrl, · line 992

Quoted 12 lines
drivers/nvme/host/rdma.c +1 -2
index f8f856dc0c67..b381e2cde50a 100644
@@ -989,8 +989,7 @@ static void nvme_rdma_teardown_io_queues(struct nvme_rdma_ctrl *ctrl,
                   nvme_cancel_request, &ctrl->ctrl);
               blk_mq_tagset_wait_completed_request(ctrl->ctrl.tagset);
           }
-        if (remove)
-            nvme_start_queues(&ctrl->ctrl);
+        nvme_start_queues(&ctrl->ctrl);

This will fail I/O during controller reset, so nak on this.

The io will do not fail. If work with native multipath or dm-multipath, nvme_rdma_queue_rq will return io error, and then multipath will fail over to other path and retry io, this is we expected. If work without multipath, nvme_rdma_queue_rq will return BLK_STS_RESOURCE, and then the upper layer will requeue and retry. Surely there is a weakness:the io will retry repeated every BLK_MQ_RESOURCE_DELAY(3ms) while reconnecting. Because controller reset may need long time, and nvme over roce is mainly used with multipath software, so when controller reset we expect fail over to other path and retry io,, just like error recovery. If work without multipath, we tolerate repeated I/O retries during error recovery or controller reset.

Quoted 21 lines

Can you try this:

drivers/nvme/host/rdma.c +5 -0
index d58231636d11..96c0d664fe9b 100644
@@ -1149,6 +1149,11 @@ static void nvme_rdma_reconnect_ctrl_work(struct work_struct *work)
         return;
 
  requeue:
+       /*
+        * make sure queues are not quiesced due to a reconnect
+        * sequence that failed after creating some I/O queues
+        */
+       nvme_start_queues(ctrl);
         dev_info(ctrl->ctrl.device, "Failed reconnect attempt %d\n",
                         ctrl->ctrl.nr_reconnects);
         nvme_rdma_reconnect_or_remove(ctrl);
Signature · 1 line
.Surely this can solve the problem, but when controller reset in

multipath environment we expect fail over to other path and retry io, just like error recovery. Both for controller reset and error recovery, we need unquiesce the queue. So nvme_rdma_teardown_io_queues should directly unquiesce queues after cancel request, do not need care the parameter:remove.
For this solution, if nvme_rdma_setup_ctrl failed in nvme_rdma_reset_ctrl_work, will start nvme_rdma_reconnect_ctrl_work, thus no difference.
If we do not tolerate repeated I/O retries during controller reset, we can do like this:

---
 drivers/nvme/host/rdma.c | 1 +
 1 file changed, 1 insertion(+)
drivers/nvme/host/rdma.c +1 -0
index b381e2cde50a..ac79c4d294c2 100644
@@ -1936,6 +1936,7 @@ nvme_rdma_timeout(struct request *rq, bool reserved)
 		 */
 		flush_work(&ctrl->err_work);
 		nvme_rdma_teardown_io_queues(ctrl, false);
+		nvme_start_queues(&ctrl->ctrl);
 		nvme_rdma_teardown_admin_queue(ctrl, false);
 		return BLK_EH_DONE;
 	}
Signature · 7 lines
2.16.4


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
maintainer
#4 lore raw
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

drivers/nvme/host/rdma.c static void nvme_rdma_teardown_io_queues(struct · line 993

Quoted 13 lines
drivers/nvme/host/rdma.c +1 -2
index f8f856dc0c67..b381e2cde50a 100644
@@ -989,8 +989,7 @@ static void nvme_rdma_teardown_io_queues(struct 
 nvme_rdma_ctrl *ctrl,
                   nvme_cancel_request, &ctrl->ctrl);
               blk_mq_tagset_wait_completed_request(ctrl->ctrl.tagset);
           }
-        if (remove)
-            nvme_start_queues(&ctrl->ctrl);
+        nvme_start_queues(&ctrl->ctrl);

This will fail I/O during controller reset, so nak on this.

Quoted 11 lines

The io will do not fail. If work with native multipath or dm-multipath, nvme_rdma_queue_rq will return io error, and then multipath will fail over to other path and retry io, this is we expected. If work without multipath, nvme_rdma_queue_rq will return BLK_STS_RESOURCE, and then the upper layer will requeue and retry. Surely there is a weakness:the io will retry repeated every BLK_MQ_RESOURCE_DELAY(3ms) while reconnecting. Because controller reset may need long time, and nvme over roce is mainly used with multipath software, so when controller reset we expect fail over to other path and retry io,, just like error recovery. If work without multipath, we tolerate repeated I/O retries during error recovery or controller reset.

I/O should not fail during reset, mpath or not, period.

Quoted 39 lines
Quoted 11 lines

Can you try this:

drivers/nvme/host/rdma.c +0 -0
index d58231636d11..96c0d664fe9b 100644
@@ -1149,6 +1149,11 @@ static void 
 nvme_rdma_reconnect_ctrl_work(struct work_struct *work)
          return;
 

  requeue:

Quoted 10 lines

+       /*
+        * make sure queues are not quiesced due to a reconnect +        * sequence that failed after creating some I/O queues +        */
+       nvme_start_queues(ctrl); dev_info(ctrl->ctrl.device, "Failed reconnect attempt %d\n", ctrl->ctrl.nr_reconnects); nvme_rdma_reconnect_or_remove(ctrl);

Signature · 1 line
.Surely this can solve the problem, but when controller reset in

multipath environment we expect fail over to other path and retry io, just like error recovery. Both for controller reset and error recovery, we need unquiesce the queue. So nvme_rdma_teardown_io_queues should directly unquiesce queues after cancel request, do not need care the parameter:remove.
For this solution, if nvme_rdma_setup_ctrl failed in nvme_rdma_reset_ctrl_work, will start nvme_rdma_reconnect_ctrl_work, thus no difference.
If we do not tolerate repeated I/O retries during controller reset, we can do like this:

---
  drivers/nvme/host/rdma.c | 1 +
  1 file changed, 1 insertion(+)
drivers/nvme/host/rdma.c +0 -0
index b381e2cde50a..ac79c4d294c2 100644

drivers/nvme/host/rdma.c nvme_rdma_timeout(struct request *rq, bool reserved) · line 1942

Quoted 7 lines
drivers/nvme/host/rdma.c +1 -0
@@ -1936,6 +1936,7 @@ nvme_rdma_timeout(struct request *rq, bool reserved)
           */
          flush_work(&ctrl->err_work);
          nvme_rdma_teardown_io_queues(ctrl, false);
+        nvme_start_queues(&ctrl->ctrl);
          nvme_rdma_teardown_admin_queue(ctrl, false);
          return BLK_EH_DONE;
      }

That is fine too, just do not unquiesce the queues in a normal reset.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

A message that is not in the archive (sent off-list or not yet received): [email protected]

maintainer
#6 lore raw
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines

The io will do not fail. If work with native multipath or dm-multipath, nvme_rdma_queue_rq will return io error, and then multipath will fail over to other path and retry io, this is we expected. If work without multipath, nvme_rdma_queue_rq will return BLK_STS_RESOURCE, and then the upper layer will requeue and retry. Surely there is a weakness:the io will retry repeated every BLK_MQ_RESOURCE_DELAY(3ms) while reconnecting. Because controller reset may need long time, and nvme over roce is mainly used with multipath software, so when controller reset we expect fail over to other path and retry io,, just like error recovery. If work without multipath, we tolerate repeated I/O retries during error recovery or controller reset.

Quoted 3 lines
Quoted 1 line

I/O should not fail during reset, mpath or not, period.

except when marked as an internal io (one used for reconnect, or maybe an ioctl) or marked for mpath.

I meant normal fs I/O, from the user perspective.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme