Skip to contentSkip to conversation

nvme fabircs: fix io hang in error recovery vs. removal

PATCH In review Ming Lei 4 participants 9 messages Jul 11, 2023 – Jul 21, 2023

Subsystems (2)
lore
Share & tools
lore

Ming Lei

author
#0 lore raw

cover nvme fabircs: fix io hang in error recovery vs. removal

Hello,

The 1st patch fixes io hang when queues are left as quiesced.

The other 2 patches fixes io hang when queues are left as frozen.

All are triggered in error recovery vs. removal.

Ming Lei (3):
nvme: fix possible hang when removing a controller during error
recovery
nvme-tcp: fix potential unbalanced freeze & unfreeze
nvme-rdma: fix potential unbalanced freeze & unfreeze

 drivers/nvme/host/core.c | 10 +++++++---
 drivers/nvme/host/rdma.c |  3 ++-
 drivers/nvme/host/tcp.c  |  3 ++-
 3 files changed, 11 insertions(+), 5 deletions(-)
Signature · 1 line
2.40.1
author
#1 lore raw

PATCH v2 1/3 nvme: fix possible hang when removing a controller during error recovery

Error recovery can be interrupted by controller removal, then the controller is left as quiesced, and IO hang can be caused.

Fix the issue by unquiescing controller unconditionally when removing namespaces.

This way is reasonable and safe given forward progress can be made when removing namespaces.

---
 drivers/nvme/host/core.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
drivers/nvme/host/core.c +7 -3
index 47d7ba2827ff..98fa8315bc65 100644
@@ -3903,6 +3903,12 @@ void nvme_remove_namespaces(struct nvme_ctrl *ctrl)
 	 */
 	nvme_mpath_clear_ctrl_paths(ctrl);
 
+	/*
+	 * Unquiesce io queues so any pending IO won't hang, especially
+	 * those submitted from scan work
+	 */
+	nvme_unquiesce_io_queues(ctrl);
+
 	/* prevent racing with ns scanning */
 	flush_work(&ctrl->scan_work);
 
@@ -3912,10 +3918,8 @@ void nvme_remove_namespaces(struct nvme_ctrl *ctrl)
 	 * removing the namespaces' disks; fail all the queues now to avoid
 	 * potentially having to clean up the failed sync later.
 	 */
-	if (ctrl->state == NVME_CTRL_DEAD) {
+	if (ctrl->state == NVME_CTRL_DEAD)
 		nvme_mark_namespaces_dead(ctrl);
-		nvme_unquiesce_io_queues(ctrl);
-	}
 
 	/* this is a no-op when called from the controller reset handler */
 	nvme_change_ctrl_state(ctrl, NVME_CTRL_DELETING_NOIO);
Signature · 1 line
2.40.1
author
#2 lore raw

PATCH v2 2/3 nvme-tcp: fix potential unbalanced freeze & unfreeze

In reply to Ming Lei

Move start_freeze into nvme_tcp_configure_io_queues(), and there is at least two benefits:

1) fix unbalanced freeze and unfreeze, since re-connection work may fail or be broken by removal

2) IO during error recovery can be failfast quickly because nvme fabrics unquiesces queues after teardown.

One side-effect is that !mpath request may timeout during connecting because of queue topo change, but that looks not one big deal:

1) same problem exists with current code base

2) compared with !mpath, mpath use case is dominant

Fixes 2875b0aecabe ("nvme-tcp: fix controller reset hang during traffic")
---
 drivers/nvme/host/tcp.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/host/tcp.c +2 -1
index 3e7dd6f91832..fb24cd8ac46c 100644
@@ -1868,6 +1868,7 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
 		goto out_cleanup_connect_q;
 
 	if (!new) {
+		nvme_start_freeze(ctrl);
 		nvme_unquiesce_io_queues(ctrl);
 		if (!nvme_wait_freeze_timeout(ctrl, NVME_IO_TIMEOUT)) {
 			/*
@@ -1876,6 +1877,7 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
 			 * to be safe.
 			 */
 			ret = -ENODEV;
+			nvme_unfreeze(ctrl);
 			goto out_wait_freeze_timed_out;
 		}
 		blk_mq_update_nr_hw_queues(ctrl->tagset,
@@ -1980,7 +1982,6 @@ static void nvme_tcp_teardown_io_queues(struct nvme_ctrl *ctrl,
 	if (ctrl->queue_count <= 1)
 		return;
 	nvme_quiesce_admin_queue(ctrl);
-	nvme_start_freeze(ctrl);
 	nvme_quiesce_io_queues(ctrl);
 	nvme_sync_io_queues(ctrl);
 	nvme_tcp_stop_io_queues(ctrl);
Signature · 1 line
2.40.1
Tested-by on v2 2/3
#4 lore raw

Re: [PATCH V2 2/3] nvme-tcp: fix potential unbalanced freeze & unfreeze

In reply to Ming Lei

Verified it with the nvme/tcp scenario, Thanks Ming

Quoted 55 lines · Ming Lei

Move start_freeze into nvme_tcp_configure_io_queues(), and there is at least two benefits:

1) fix unbalanced freeze and unfreeze, since re-connection work may fail or be broken by removal

2) IO during error recovery can be failfast quickly because nvme fabrics unquiesces queues after teardown.

One side-effect is that !mpath request may timeout during connecting because of queue topo change, but that looks not one big deal:

1) same problem exists with current code base

2) compared with !mpath, mpath use case is dominant

Fixes 2875b0aecabe ("nvme-tcp: fix controller reset hang during traffic")
---
 drivers/nvme/host/tcp.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/host/tcp.c +2 -2
index 3e7dd6f91832..fb24cd8ac46c 100644
@@ -1868,6 +1868,7 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
                goto out_cleanup_connect_q;
 
        if (!new) {
+               nvme_start_freeze(ctrl);
                nvme_unquiesce_io_queues(ctrl);
                if (!nvme_wait_freeze_timeout(ctrl, NVME_IO_TIMEOUT)) {
                        /*
@@ -1876,6 +1877,7 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
                         * to be safe.
                         */
                        ret = -ENODEV;
+                       nvme_unfreeze(ctrl);
                        goto out_wait_freeze_timed_out;
                }
                blk_mq_update_nr_hw_queues(ctrl->tagset,
@@ -1980,7 +1982,6 @@ static void nvme_tcp_teardown_io_queues(struct nvme_ctrl *ctrl,
        if (ctrl->queue_count <= 1)
                return;
        nvme_quiesce_admin_queue(ctrl);
-       nvme_start_freeze(ctrl);
        nvme_quiesce_io_queues(ctrl);
        nvme_sync_io_queues(ctrl);
        nvme_tcp_stop_io_queues(ctrl);
--

2.40.1

Signature · 2 lines
Best Regards,
  Yi Zhang
author
#5 lore raw

PATCH v2 3/3 nvme-rdma: fix potential unbalanced freeze & unfreeze

In reply to Ming Lei

Move start_freeze into nvme_rdma_configure_io_queues(), and there is at least two benefits:

1) fix unbalanced freeze and unfreeze, since re-connection work may fail or be broken by removal

2) IO during error recovery can be failfast quickly because nvme fabrics unquiesces queues after teardown.

One side-effect is that !mpath request may timeout during connecting because of queue topo change, but that looks not one big deal:

1) same problem exists with current code base

2) compared with !mpath, mpath use case is dominant

Fixes 9f98772ba307 ("nvme-rdma: fix controller reset hang during traffic")
---
 drivers/nvme/host/rdma.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/host/rdma.c +2 -1
index d433b2ec07a6..337a624a537c 100644
@@ -883,6 +883,7 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)
 		goto out_cleanup_tagset;
 
 	if (!new) {
+		nvme_start_freeze(&ctrl->ctrl);
 		nvme_unquiesce_io_queues(&ctrl->ctrl);
 		if (!nvme_wait_freeze_timeout(&ctrl->ctrl, NVME_IO_TIMEOUT)) {
 			/*
@@ -891,6 +892,7 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)
 			 * to be safe.
 			 */
 			ret = -ENODEV;
+			nvme_unfreeze(&ctrl->ctrl);
 			goto out_wait_freeze_timed_out;
 		}
 		blk_mq_update_nr_hw_queues(ctrl->ctrl.tagset,
@@ -940,7 +942,6 @@ static void nvme_rdma_teardown_io_queues(struct nvme_rdma_ctrl *ctrl,
 		bool remove)
 {
 	if (ctrl->ctrl.queue_count > 1) {
-		nvme_start_freeze(&ctrl->ctrl);
 		nvme_quiesce_io_queues(&ctrl->ctrl);
 		nvme_sync_io_queues(&ctrl->ctrl);
 		nvme_rdma_stop_io_queues(ctrl);
Signature · 1 line
2.40.1
Tested-by on v2 3/3
#7 lore raw

Re: [PATCH V2 3/3] nvme-rdma: fix potential unbalanced freeze & unfreeze

In reply to Ming Lei

Verified it with the nvme/rdma scenario, Thanks Ming

Quoted 55 lines · Ming Lei

Move start_freeze into nvme_rdma_configure_io_queues(), and there is at least two benefits:

1) fix unbalanced freeze and unfreeze, since re-connection work may fail or be broken by removal

2) IO during error recovery can be failfast quickly because nvme fabrics unquiesces queues after teardown.

One side-effect is that !mpath request may timeout during connecting because of queue topo change, but that looks not one big deal:

1) same problem exists with current code base

2) compared with !mpath, mpath use case is dominant

Fixes 9f98772ba307 ("nvme-rdma: fix controller reset hang during traffic")
---
 drivers/nvme/host/rdma.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/host/rdma.c +2 -2
index d433b2ec07a6..337a624a537c 100644
@@ -883,6 +883,7 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)
                goto out_cleanup_tagset;
 
        if (!new) {
+               nvme_start_freeze(&ctrl->ctrl);
                nvme_unquiesce_io_queues(&ctrl->ctrl);
                if (!nvme_wait_freeze_timeout(&ctrl->ctrl, NVME_IO_TIMEOUT)) {
                        /*
@@ -891,6 +892,7 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)
                         * to be safe.
                         */
                        ret = -ENODEV;
+                       nvme_unfreeze(&ctrl->ctrl);
                        goto out_wait_freeze_timed_out;
                }
                blk_mq_update_nr_hw_queues(ctrl->ctrl.tagset,
@@ -940,7 +942,6 @@ static void nvme_rdma_teardown_io_queues(struct nvme_rdma_ctrl *ctrl,
                bool remove)
 {
        if (ctrl->ctrl.queue_count > 1) {
-               nvme_start_freeze(&ctrl->ctrl);
                nvme_quiesce_io_queues(&ctrl->ctrl);
                nvme_sync_io_queues(&ctrl->ctrl);
                nvme_rdma_stop_io_queues(ctrl);
--

2.40.1

Signature · 2 lines
Best Regards,
  Yi Zhang