Skip to contentSkip to conversation

nvme: don't wait freeze during resetting

PATCH Superseded Ming Lei 3 participants 7 messages Sep 20, 2022 – Sep 22, 2022

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Ming Lei

author
#0 lore raw

PATCH 1/1 nvme: don't wait freeze during resetting

First it isn't necessary to call nvme_wait_freeze during reset. For nvme-pci, if tagset isn't allocated, there can't be any inflight IOs; otherwise blk_mq_update_nr_hw_queues can freeze & wait queues.

Second, since commit bdd6316094e0 ("block: Allow unfreezing of a queue while requests are in progress"), it is fine to unfreeze queue without draining inflight IOs.

Also both nvme-rdma and nvme-tcp's timeout handler provides forward progress if the controller state isn't LIVE, so it is fine to drop the timeout function of nvme_wait_freeze_timeout().

Cc Sagi Grimberg <[email protected]>
Cc Chao Leng <[email protected]>
Cc Keith Busch <[email protected]>
---
 drivers/nvme/host/apple.c |  1 -
 drivers/nvme/host/pci.c   |  1 -
 drivers/nvme/host/rdma.c  | 13 -------------
 drivers/nvme/host/tcp.c   | 13 -------------
 4 files changed, 28 deletions(-)
drivers/nvme/host/apple.c +0 -1
index 5fc5ea196b40..9cd02b57fc85 100644
@@ -1126,7 +1126,6 @@ static void apple_nvme_reset_work(struct work_struct *work)
 	anv->ctrl.queue_count = nr_io_queues + 1;
 
 	nvme_start_queues(&anv->ctrl);
-	nvme_wait_freeze(&anv->ctrl);
 	blk_mq_update_nr_hw_queues(&anv->tagset, 1);
 	nvme_unfreeze(&anv->ctrl);
 
drivers/nvme/host/pci.c +0 -1
index 98864b853eef..985b216907fc 100644
@@ -2910,7 +2910,6 @@ static void nvme_reset_work(struct work_struct *work)
 		nvme_free_tagset(dev);
 	} else {
 		nvme_start_queues(&dev->ctrl);
-		nvme_wait_freeze(&dev->ctrl);
 		if (!dev->ctrl.tagset)
 			nvme_pci_alloc_tag_set(dev);
 		else
drivers/nvme/host/rdma.c +0 -13
index 3100643be299..beb0d1a6a84d 100644
@@ -986,15 +986,6 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)
 
 	if (!new) {
 		nvme_start_queues(&ctrl->ctrl);
-		if (!nvme_wait_freeze_timeout(&ctrl->ctrl, NVME_IO_TIMEOUT)) {
-			/*
-			 * If we timed out waiting for freeze we are likely to
-			 * be stuck.  Fail the controller initialization just
-			 * to be safe.
-			 */
-			ret = -ENODEV;
-			goto out_wait_freeze_timed_out;
-		}
 		blk_mq_update_nr_hw_queues(ctrl->ctrl.tagset,
 			ctrl->ctrl.queue_count - 1);
 		nvme_unfreeze(&ctrl->ctrl);
@@ -1002,10 +993,6 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)
 
 	return 0;
 
-out_wait_freeze_timed_out:
-	nvme_stop_queues(&ctrl->ctrl);
-	nvme_sync_io_queues(&ctrl->ctrl);
-	nvme_rdma_stop_io_queues(ctrl);
 out_cleanup_connect_q:
 	nvme_cancel_tagset(&ctrl->ctrl);
 	if (new)
drivers/nvme/host/tcp.c +0 -13
index d5871fd6f769..49d9bef806f9 100644
@@ -1920,15 +1920,6 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
 
 	if (!new) {
 		nvme_start_queues(ctrl);
-		if (!nvme_wait_freeze_timeout(ctrl, NVME_IO_TIMEOUT)) {
-			/*
-			 * If we timed out waiting for freeze we are likely to
-			 * be stuck.  Fail the controller initialization just
-			 * to be safe.
-			 */
-			ret = -ENODEV;
-			goto out_wait_freeze_timed_out;
-		}
 		blk_mq_update_nr_hw_queues(ctrl->tagset,
 			ctrl->queue_count - 1);
 		nvme_unfreeze(ctrl);
@@ -1936,10 +1927,6 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
 
 	return 0;
 
-out_wait_freeze_timed_out:
-	nvme_stop_queues(ctrl);
-	nvme_sync_io_queues(ctrl);
-	nvme_tcp_stop_io_queues(ctrl);
 out_cleanup_connect_q:
 	nvme_cancel_tagset(ctrl);
 	if (new)
Signature · 1 line
2.31.1
maintainer
#1 lore raw
Quoted 5 lines

First it isn't necessary to call nvme_wait_freeze during reset. For nvme-pci, if tagset isn't allocated, there can't be any inflight IOs; otherwise blk_mq_update_nr_hw_queues can freeze & wait queues.

Quoted 7 lines

Second, since commit bdd6316094e0 ("block: Allow unfreezing of a queue while requests are in progress"), it is fine to unfreeze queue without draining inflight IOs.

Also both nvme-rdma and nvme-tcp's timeout handler provides forward progress if the controller state isn't LIVE, so it is fine to drop the timeout function of nvme_wait_freeze_timeout().

The rdma/tcp should probably be split to separate patches.

Quoted 43 lines
Cc Sagi Grimberg <[email protected]>
Cc Chao Leng <[email protected]>
Cc Keith Busch <[email protected]>
---
  drivers/nvme/host/apple.c |  1 -
  drivers/nvme/host/pci.c   |  1 -
  drivers/nvme/host/rdma.c  | 13 -------------
  drivers/nvme/host/tcp.c   | 13 -------------
  4 files changed, 28 deletions(-)
drivers/nvme/host/apple.c +0 -1
index 5fc5ea196b40..9cd02b57fc85 100644
@@ -1126,7 +1126,6 @@ static void apple_nvme_reset_work(struct work_struct *work)
  	anv->ctrl.queue_count = nr_io_queues + 1;
 
  	nvme_start_queues(&anv->ctrl);
-	nvme_wait_freeze(&anv->ctrl);
  	blk_mq_update_nr_hw_queues(&anv->tagset, 1);
  	nvme_unfreeze(&anv->ctrl);
 
drivers/nvme/host/pci.c +0 -1
index 98864b853eef..985b216907fc 100644
@@ -2910,7 +2910,6 @@ static void nvme_reset_work(struct work_struct *work)
  		nvme_free_tagset(dev);
  	} else {
  		nvme_start_queues(&dev->ctrl);
-		nvme_wait_freeze(&dev->ctrl);
  		if (!dev->ctrl.tagset)
  			nvme_pci_alloc_tag_set(dev);
  		else
drivers/nvme/host/rdma.c +0 -0
index 3100643be299..beb0d1a6a84d 100644
@@ -986,15 +986,6 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)
 
  	if (!new) {

drivers/nvme/host/rdma.c static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new) · line 997

Quoted 10 lines
drivers/nvme/host/rdma.c +0 -9
  		nvme_start_queues(&ctrl->ctrl);
-		if (!nvme_wait_freeze_timeout(&ctrl->ctrl, NVME_IO_TIMEOUT)) {
-			/*
-			 * If we timed out waiting for freeze we are likely to
-			 * be stuck.  Fail the controller initialization just
-			 * to be safe.
-			 */
-			ret = -ENODEV;
-			goto out_wait_freeze_timed_out;
-		}

So here is the description from the patch that introduced this:
--
nvme-rdma: fix reset hang if controller died in the middle of a reset

If the controller becomes unresponsive in the middle of a reset, we will hang because we are waiting for the freeze to complete, but that cannot happen since we have commands that are inflight holding the q_usage_counter, and we can't blindly fail requests that times out.

So give a timeout and if we cannot wait for queue freeze before unfreezing, fail and have the error handling take care how to proceed (either schedule a reconnect of remove the controller). --

So if between nvme_start_queues() and the freeze (with a full wait) that is done in blk_mq_update_nr_hw_queues() the controller becomes non responsive, in this case we may hang blocking on I/O that was pending and requeued after nvme_start_queues().

The problem is, that we cannot do any error recovery because the controller is in the middle of a reset/reconnect...
So the code that you deleted was designed to detect this state, and reschedule another reconnect if the controller became non responsive.

What is preventing this from happening now?

I wish we had a test for this... It is very difficult to hit because the controller needs to become non-responsive exactly at this point in the reset...

Quoted 45 lines
drivers/nvme/host/rdma.c +0 -4
  		blk_mq_update_nr_hw_queues(ctrl->ctrl.tagset,
  			ctrl->ctrl.queue_count - 1);
  		nvme_unfreeze(&ctrl->ctrl);
@@ -1002,10 +993,6 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)
 
  	return 0;
 
-out_wait_freeze_timed_out:
-	nvme_stop_queues(&ctrl->ctrl);
-	nvme_sync_io_queues(&ctrl->ctrl);
-	nvme_rdma_stop_io_queues(ctrl);
  out_cleanup_connect_q:
  	nvme_cancel_tagset(&ctrl->ctrl);
  	if (new)
drivers/nvme/host/tcp.c +0 -13
index d5871fd6f769..49d9bef806f9 100644
@@ -1920,15 +1920,6 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
 
  	if (!new) {
  		nvme_start_queues(ctrl);
-		if (!nvme_wait_freeze_timeout(ctrl, NVME_IO_TIMEOUT)) {
-			/*
-			 * If we timed out waiting for freeze we are likely to
-			 * be stuck.  Fail the controller initialization just
-			 * to be safe.
-			 */
-			ret = -ENODEV;
-			goto out_wait_freeze_timed_out;
-		}
  		blk_mq_update_nr_hw_queues(ctrl->tagset,
  			ctrl->queue_count - 1);
  		nvme_unfreeze(ctrl);
@@ -1936,10 +1927,6 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
 
  	return 0;
 
-out_wait_freeze_timed_out:
-	nvme_stop_queues(ctrl);
-	nvme_sync_io_queues(ctrl);
-	nvme_tcp_stop_io_queues(ctrl);
  out_cleanup_connect_q:
  	nvme_cancel_tagset(ctrl);
  	if (new)
author
#2 lore raw
Quoted 85 lines · Sagi Grimberg
Quoted 5 lines

First it isn't necessary to call nvme_wait_freeze during reset. For nvme-pci, if tagset isn't allocated, there can't be any inflight IOs; otherwise blk_mq_update_nr_hw_queues can freeze & wait queues.

Quoted 7 lines

Second, since commit bdd6316094e0 ("block: Allow unfreezing of a queue while requests are in progress"), it is fine to unfreeze queue without draining inflight IOs.

Also both nvme-rdma and nvme-tcp's timeout handler provides forward progress if the controller state isn't LIVE, so it is fine to drop the timeout function of nvme_wait_freeze_timeout().

The rdma/tcp should probably be split to separate patches.

Quoted 40 lines
Cc Sagi Grimberg <[email protected]>
Cc Chao Leng <[email protected]>
Cc Keith Busch <[email protected]>
---
  drivers/nvme/host/apple.c |  1 -
  drivers/nvme/host/pci.c   |  1 -
  drivers/nvme/host/rdma.c  | 13 -------------
  drivers/nvme/host/tcp.c   | 13 -------------
  4 files changed, 28 deletions(-)
drivers/nvme/host/apple.c +0 -1
index 5fc5ea196b40..9cd02b57fc85 100644
@@ -1126,7 +1126,6 @@ static void apple_nvme_reset_work(struct work_struct *work)
  	anv->ctrl.queue_count = nr_io_queues + 1;
  	nvme_start_queues(&anv->ctrl);
-	nvme_wait_freeze(&anv->ctrl);
  	blk_mq_update_nr_hw_queues(&anv->tagset, 1);
  	nvme_unfreeze(&anv->ctrl);
drivers/nvme/host/pci.c +0 -1
index 98864b853eef..985b216907fc 100644
@@ -2910,7 +2910,6 @@ static void nvme_reset_work(struct work_struct *work)
  		nvme_free_tagset(dev);
  	} else {
  		nvme_start_queues(&dev->ctrl);
-		nvme_wait_freeze(&dev->ctrl);
  		if (!dev->ctrl.tagset)
  			nvme_pci_alloc_tag_set(dev);
  		else
drivers/nvme/host/rdma.c +0 -0
index 3100643be299..beb0d1a6a84d 100644
@@ -986,15 +986,6 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)
  	if (!new) {

drivers/nvme/host/rdma.c static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new) · line 996

Quoted 10 lines
drivers/nvme/host/rdma.c +0 -9
  		nvme_start_queues(&ctrl->ctrl);
-		if (!nvme_wait_freeze_timeout(&ctrl->ctrl, NVME_IO_TIMEOUT)) {
-			/*
-			 * If we timed out waiting for freeze we are likely to
-			 * be stuck.  Fail the controller initialization just
-			 * to be safe.
-			 */
-			ret = -ENODEV;
-			goto out_wait_freeze_timed_out;
-		}

So here is the description from the patch that introduced this:
--
nvme-rdma: fix reset hang if controller died in the middle of a reset

If the controller becomes unresponsive in the middle of a reset, we will hang because we are waiting for the freeze to complete, but that cannot happen since we have commands that are inflight holding the q_usage_counter, and we can't blindly fail requests that times out.

So give a timeout and if we cannot wait for queue freeze before unfreezing, fail and have the error handling take care how to proceed (either schedule a reconnect of remove the controller). --

So if between nvme_start_queues() and the freeze (with a full wait) that is done in blk_mq_update_nr_hw_queues() the controller becomes non responsive, in this case we may hang blocking on I/O that was pending and requeued after nvme_start_queues().

Quoted 6 lines

The problem is, that we cannot do any error recovery because the controller is in the middle of a reset/reconnect...
So the code that you deleted was designed to detect this state, and reschedule another reconnect if the controller became non responsive.

What is preventing this from happening now?

Please see nvme_rdma_timeout() & nvme_tcp_timeout(), if controller state isn't live, request will be aborted.

Thanks,
Ming

maintainer
#3 lore raw
Quoted 91 lines · Ming Lei
Quoted 90 lines · Sagi Grimberg
Quoted 5 lines

First it isn't necessary to call nvme_wait_freeze during reset. For nvme-pci, if tagset isn't allocated, there can't be any inflight IOs; otherwise blk_mq_update_nr_hw_queues can freeze & wait queues.

Quoted 7 lines

Second, since commit bdd6316094e0 ("block: Allow unfreezing of a queue while requests are in progress"), it is fine to unfreeze queue without draining inflight IOs.

Also both nvme-rdma and nvme-tcp's timeout handler provides forward progress if the controller state isn't LIVE, so it is fine to drop the timeout function of nvme_wait_freeze_timeout().

The rdma/tcp should probably be split to separate patches.

Quoted 40 lines
Cc Sagi Grimberg <[email protected]>
Cc Chao Leng <[email protected]>
Cc Keith Busch <[email protected]>
---
   drivers/nvme/host/apple.c |  1 -
   drivers/nvme/host/pci.c   |  1 -
   drivers/nvme/host/rdma.c  | 13 -------------
   drivers/nvme/host/tcp.c   | 13 -------------
   4 files changed, 28 deletions(-)
drivers/nvme/host/apple.c +0 -1
index 5fc5ea196b40..9cd02b57fc85 100644
@@ -1126,7 +1126,6 @@ static void apple_nvme_reset_work(struct work_struct *work)
   	anv->ctrl.queue_count = nr_io_queues + 1;
   	nvme_start_queues(&anv->ctrl);
-	nvme_wait_freeze(&anv->ctrl);
   	blk_mq_update_nr_hw_queues(&anv->tagset, 1);
   	nvme_unfreeze(&anv->ctrl);
drivers/nvme/host/pci.c +0 -1
index 98864b853eef..985b216907fc 100644
@@ -2910,7 +2910,6 @@ static void nvme_reset_work(struct work_struct *work)
   		nvme_free_tagset(dev);
   	} else {
   		nvme_start_queues(&dev->ctrl);
-		nvme_wait_freeze(&dev->ctrl);
   		if (!dev->ctrl.tagset)
   			nvme_pci_alloc_tag_set(dev);
   		else
drivers/nvme/host/rdma.c +0 -0
index 3100643be299..beb0d1a6a84d 100644
@@ -986,15 +986,6 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)
   	if (!new) {

drivers/nvme/host/rdma.c static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new) · line 996

Quoted 10 lines
drivers/nvme/host/rdma.c +0 -9
   		nvme_start_queues(&ctrl->ctrl);
-		if (!nvme_wait_freeze_timeout(&ctrl->ctrl, NVME_IO_TIMEOUT)) {
-			/*
-			 * If we timed out waiting for freeze we are likely to
-			 * be stuck.  Fail the controller initialization just
-			 * to be safe.
-			 */
-			ret = -ENODEV;
-			goto out_wait_freeze_timed_out;
-		}

So here is the description from the patch that introduced this:
--
nvme-rdma: fix reset hang if controller died in the middle of a reset

If the controller becomes unresponsive in the middle of a reset, we will hang because we are waiting for the freeze to complete, but that cannot happen since we have commands that are inflight holding the q_usage_counter, and we can't blindly fail requests that times out.

So give a timeout and if we cannot wait for queue freeze before unfreezing, fail and have the error handling take care how to proceed (either schedule a reconnect of remove the controller). --

So if between nvme_start_queues() and the freeze (with a full wait) that is done in blk_mq_update_nr_hw_queues() the controller becomes non responsive, in this case we may hang blocking on I/O that was pending and requeued after nvme_start_queues().

The problem is, that we cannot do any error recovery because the controller is in the middle of a reset/reconnect...
So the code that you deleted was designed to detect this state, and reschedule another reconnect if the controller became non responsive.

Quoted 4 lines
Quoted 1 line

What is preventing this from happening now?

Please see nvme_rdma_timeout() & nvme_tcp_timeout(), if controller state isn't live, request will be aborted.

I agree with you. However non-mpath devices will most likely retry the
command and not fail it like in the multipath case (see nvme_decide_disposition) and will cause the I/O to block.

While it is arguable if non-mpath fabrics devices are important in any capacity, the design was that IO is not completed until the controller either successfully reconnects (and retried), or it disconnects (failed), or fast_io_fail_tmo expires.

Hence for non-mpath controllers, the request(s) will timeout, and aborted, but nvme will opt to retry them instead of completing them with a failure (at least until fast_io_fail_tmo expires, but that can be arbitrarily long).

author
#4 lore raw
Quoted 108 lines · Sagi Grimberg
Quoted 91 lines · Ming Lei
Quoted 90 lines · Sagi Grimberg
Quoted 5 lines

First it isn't necessary to call nvme_wait_freeze during reset. For nvme-pci, if tagset isn't allocated, there can't be any inflight IOs; otherwise blk_mq_update_nr_hw_queues can freeze & wait queues.

Quoted 7 lines

Second, since commit bdd6316094e0 ("block: Allow unfreezing of a queue while requests are in progress"), it is fine to unfreeze queue without draining inflight IOs.

Also both nvme-rdma and nvme-tcp's timeout handler provides forward progress if the controller state isn't LIVE, so it is fine to drop the timeout function of nvme_wait_freeze_timeout().

The rdma/tcp should probably be split to separate patches.

Quoted 40 lines
Cc Sagi Grimberg <[email protected]>
Cc Chao Leng <[email protected]>
Cc Keith Busch <[email protected]>
---
   drivers/nvme/host/apple.c |  1 -
   drivers/nvme/host/pci.c   |  1 -
   drivers/nvme/host/rdma.c  | 13 -------------
   drivers/nvme/host/tcp.c   | 13 -------------
   4 files changed, 28 deletions(-)
drivers/nvme/host/apple.c +0 -1
index 5fc5ea196b40..9cd02b57fc85 100644
@@ -1126,7 +1126,6 @@ static void apple_nvme_reset_work(struct work_struct *work)
   	anv->ctrl.queue_count = nr_io_queues + 1;
   	nvme_start_queues(&anv->ctrl);
-	nvme_wait_freeze(&anv->ctrl);
   	blk_mq_update_nr_hw_queues(&anv->tagset, 1);
   	nvme_unfreeze(&anv->ctrl);
drivers/nvme/host/pci.c +0 -1
index 98864b853eef..985b216907fc 100644
@@ -2910,7 +2910,6 @@ static void nvme_reset_work(struct work_struct *work)
   		nvme_free_tagset(dev);
   	} else {
   		nvme_start_queues(&dev->ctrl);
-		nvme_wait_freeze(&dev->ctrl);
   		if (!dev->ctrl.tagset)
   			nvme_pci_alloc_tag_set(dev);
   		else
drivers/nvme/host/rdma.c +0 -0
index 3100643be299..beb0d1a6a84d 100644
@@ -986,15 +986,6 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)
   	if (!new) {

drivers/nvme/host/rdma.c static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new) · line 996

Quoted 10 lines
drivers/nvme/host/rdma.c +0 -9
   		nvme_start_queues(&ctrl->ctrl);
-		if (!nvme_wait_freeze_timeout(&ctrl->ctrl, NVME_IO_TIMEOUT)) {
-			/*
-			 * If we timed out waiting for freeze we are likely to
-			 * be stuck.  Fail the controller initialization just
-			 * to be safe.
-			 */
-			ret = -ENODEV;
-			goto out_wait_freeze_timed_out;
-		}

So here is the description from the patch that introduced this:
--
nvme-rdma: fix reset hang if controller died in the middle of a reset

If the controller becomes unresponsive in the middle of a reset, we will hang because we are waiting for the freeze to complete, but that cannot happen since we have commands that are inflight holding the q_usage_counter, and we can't blindly fail requests that times out.

So give a timeout and if we cannot wait for queue freeze before unfreezing, fail and have the error handling take care how to proceed (either schedule a reconnect of remove the controller). --

So if between nvme_start_queues() and the freeze (with a full wait) that is done in blk_mq_update_nr_hw_queues() the controller becomes non responsive, in this case we may hang blocking on I/O that was pending and requeued after nvme_start_queues().

The problem is, that we cannot do any error recovery because the controller is in the middle of a reset/reconnect...
So the code that you deleted was designed to detect this state, and reschedule another reconnect if the controller became non responsive.

Quoted 4 lines
Quoted 1 line

What is preventing this from happening now?

Please see nvme_rdma_timeout() & nvme_tcp_timeout(), if controller state isn't live, request will be aborted.

I agree with you. However non-mpath devices will most likely retry the command and not fail it like in the multipath case (see nvme_decide_disposition) and will cause the I/O to block.

While it is arguable if non-mpath fabrics devices are important in any capacity, the design was that IO is not completed until the controller either successfully reconnects (and retried), or it disconnects (failed), or fast_io_fail_tmo expires.

Quoted 4 lines

Hence for non-mpath controllers, the request(s) will timeout, and aborted, but nvme will opt to retry them instead of completing them with a failure (at least until fast_io_fail_tmo expires, but that can be arbitrarily long).

OK, I think it is better to change the behavior for non-mpath rdma/tcp, will remove it in next version.

thanks,
Ming

#6 lore raw
Quoted 3 lines · Christoph Hellwig

Thanks,

applied for nvme-6.1.

Sorry, I replied to the wrong mail, so strike this.