nvme: don't wait freeze during resetting
Subsystems (2)
PATCH 1/1 nvme: don't wait freeze during resetting
First it isn't necessary to call nvme_wait_freeze during reset. For nvme-pci, if tagset isn't allocated, there can't be any inflight IOs; otherwise blk_mq_update_nr_hw_queues can freeze & wait queues.
Second, since commit bdd6316094e0 ("block: Allow unfreezing of a queue while requests are in progress"), it is fine to unfreeze queue without draining inflight IOs.
Also both nvme-rdma and nvme-tcp's timeout handler provides forward progress if the controller state isn't LIVE, so it is fine to drop the timeout function of nvme_wait_freeze_timeout().
--- drivers/nvme/host/apple.c | 1 - drivers/nvme/host/pci.c | 1 - drivers/nvme/host/rdma.c | 13 ------------- drivers/nvme/host/tcp.c | 13 ------------- 4 files changed, 28 deletions(-)
drivers/nvme/host/apple.c +0 -1
@@ -1126,7 +1126,6 @@ static void apple_nvme_reset_work(struct work_struct *work) anv->ctrl.queue_count = nr_io_queues +1 ;nvme_start_queues (&anv->ctrl); -nvme_wait_freeze (&anv->ctrl);blk_mq_update_nr_hw_queues (&anv->tagset,1 );nvme_unfreeze (&anv->ctrl);
drivers/nvme/host/pci.c +0 -1
@@ -2910,7 +2910,6 @@ static void nvme_reset_work(struct work_struct *work)nvme_free_tagset (dev); }else {nvme_start_queues (&dev->ctrl); -nvme_wait_freeze (&dev->ctrl);if (!dev->ctrl.tagset)nvme_pci_alloc_tag_set (dev);else
drivers/nvme/host/rdma.c +0 -13
@@ -986,15 +986,6 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)if (!new) {nvme_start_queues (&ctrl->ctrl); -if (!nvme_wait_freeze_timeout (&ctrl->ctrl,NVME_IO_TIMEOUT )) { -/* -* If we timed out waiting for freeze we are likely to -* be stuck. Fail the controller initialization just -* to be safe. -*/ - ret = -ENODEV ; - goto out_wait_freeze_timed_out; - }blk_mq_update_nr_hw_queues (ctrl->ctrl.tagset, ctrl->ctrl.queue_count -1 );nvme_unfreeze (&ctrl->ctrl);
@@ -1002,10 +993,6 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)return 0 ; -out_wait_freeze_timed_out: -nvme_stop_queues (&ctrl->ctrl); -nvme_sync_io_queues (&ctrl->ctrl); -nvme_rdma_stop_io_queues (ctrl); out_cleanup_connect_q:nvme_cancel_tagset (&ctrl->ctrl);if (new)
drivers/nvme/host/tcp.c +0 -13
@@ -1920,15 +1920,6 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)if (!new) {nvme_start_queues (ctrl); -if (!nvme_wait_freeze_timeout (ctrl,NVME_IO_TIMEOUT )) { -/* -* If we timed out waiting for freeze we are likely to -* be stuck. Fail the controller initialization just -* to be safe. -*/ - ret = -ENODEV ; - goto out_wait_freeze_timed_out; - }blk_mq_update_nr_hw_queues (ctrl->tagset, ctrl->queue_count -1 );nvme_unfreeze (ctrl);
@@ -1936,10 +1927,6 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)return 0 ; -out_wait_freeze_timed_out: -nvme_stop_queues (ctrl); -nvme_sync_io_queues (ctrl); -nvme_tcp_stop_io_queues (ctrl); out_cleanup_connect_q:nvme_cancel_tagset (ctrl);if (new)
Signature · 1 line
2.31.1
Sagi Grimberg
maintainerQuoted 5 lines
First it isn't necessary to call
nvme_wait_freezeduring reset. For nvme-pci, if tagset isn't allocated, there can't be any inflight IOs; otherwiseblk_mq_update_nr_hw_queuescan freeze & wait queues.
Quoted 7 lines
Second, since commit
bdd6316094e0("block: Allow unfreezing of a queue while requests are in progress"), it is fine to unfreeze queue without draining inflight IOs.Also both nvme-rdma and nvme-tcp's timeout handler provides forward progress if the controller state isn't LIVE, so it is fine to drop the timeout function of
nvme_wait_freeze_timeout().
The rdma/tcp should probably be split to separate patches.
Quoted 43 lines
Cc Sagi Grimberg <[email protected]>Cc Chao Leng <[email protected]>Cc Keith Busch <[email protected]>Signed-off-by Ming Lei <[email protected]>--- drivers/nvme/host/apple.c | 1 - drivers/nvme/host/pci.c | 1 - drivers/nvme/host/rdma.c | 13 ------------- drivers/nvme/host/tcp.c | 13 ------------- 4 files changed, 28 deletions(-)drivers/nvme/host/apple.c +0
-1@@ -1126,7 +1126,6 @@ static void apple_nvme_reset_work(struct work_struct *work) anv->ctrl.queue_count = nr_io_queues +1 ;nvme_start_queues (&anv->ctrl); -nvme_wait_freeze (&anv->ctrl);blk_mq_update_nr_hw_queues (&anv->tagset,1 );nvme_unfreeze (&anv->ctrl);drivers/nvme/host/pci.c +0
-1@@ -2910,7 +2910,6 @@ static void nvme_reset_work(struct work_struct *work)nvme_free_tagset (dev); }else {nvme_start_queues (&dev->ctrl); -nvme_wait_freeze (&dev->ctrl);if (!dev->ctrl.tagset)nvme_pci_alloc_tag_set (dev);else drivers/nvme/host/rdma.c +0
-0@@ -986,15 +986,6 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)if (!new) {
drivers/nvme/host/rdma.c static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new) · line 997
Quoted 10 lines
drivers/nvme/host/rdma.c +0
-9nvme_start_queues (&ctrl->ctrl); -if (!nvme_wait_freeze_timeout (&ctrl->ctrl,NVME_IO_TIMEOUT )) { -/* -* If we timed out waiting for freeze we are likely to -* be stuck. Fail the controller initialization just -* to be safe. -*/ - ret = -ENODEV ; - goto out_wait_freeze_timed_out; - }
So here is the description from the patch that introduced this:
--
nvme-rdma: fix reset hang if controller died in the middle of a reset
If the controller becomes unresponsive in the middle of a reset, we will hang because we are waiting for the freeze to complete, but that cannot happen since we have commands that are inflight holding the q_usage_counter, and we can't blindly fail requests that times out.
So give a timeout and if we cannot wait for queue freeze before unfreezing, fail and have the error handling take care how to proceed (either schedule a reconnect of remove the controller). --
So if between nvme_start_queues() and the freeze (with a full wait) that is done in blk_mq_update_nr_hw_queues() the controller becomes non responsive, in this case we may hang blocking on I/O that was pending and requeued after nvme_start_queues().
The problem is, that we cannot do any error recovery because the controller is in the middle of a reset/reconnect...
So the code that you deleted was designed to detect this state, and reschedule another reconnect if the controller became non responsive.
What is preventing this from happening now?
I wish we had a test for this... It is very difficult to hit because the controller needs to become non-responsive exactly at this point in the reset...
Quoted 45 lines
drivers/nvme/host/rdma.c +0
-4blk_mq_update_nr_hw_queues (ctrl->ctrl.tagset, ctrl->ctrl.queue_count -1 );nvme_unfreeze (&ctrl->ctrl);@@ -1002,10 +993,6 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)return 0 ; -out_wait_freeze_timed_out: -nvme_stop_queues (&ctrl->ctrl); -nvme_sync_io_queues (&ctrl->ctrl); -nvme_rdma_stop_io_queues (ctrl); out_cleanup_connect_q:nvme_cancel_tagset (&ctrl->ctrl);if (new)drivers/nvme/host/tcp.c +0
-13@@ -1920,15 +1920,6 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)if (!new) {nvme_start_queues (ctrl); -if (!nvme_wait_freeze_timeout (ctrl,NVME_IO_TIMEOUT )) { -/* -* If we timed out waiting for freeze we are likely to -* be stuck. Fail the controller initialization just -* to be safe. -*/ - ret = -ENODEV ; - goto out_wait_freeze_timed_out; - }blk_mq_update_nr_hw_queues (ctrl->tagset, ctrl->queue_count -1 );nvme_unfreeze (ctrl);@@ -1936,10 +1927,6 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)return 0 ; -out_wait_freeze_timed_out: -nvme_stop_queues (ctrl); -nvme_sync_io_queues (ctrl); -nvme_tcp_stop_io_queues (ctrl); out_cleanup_connect_q:nvme_cancel_tagset (ctrl);if (new)
Quoted 85 lines · Sagi Grimberg
Quoted 5 lines
First it isn't necessary to call
nvme_wait_freezeduring reset. For nvme-pci, if tagset isn't allocated, there can't be any inflight IOs; otherwiseblk_mq_update_nr_hw_queuescan freeze & wait queues.Quoted 7 lines
Second, since commit
bdd6316094e0("block: Allow unfreezing of a queue while requests are in progress"), it is fine to unfreeze queue without draining inflight IOs.Also both nvme-rdma and nvme-tcp's timeout handler provides forward progress if the controller state isn't LIVE, so it is fine to drop the timeout function of
nvme_wait_freeze_timeout().The rdma/tcp should probably be split to separate patches.
Quoted 40 lines
Cc Sagi Grimberg <[email protected]>Cc Chao Leng <[email protected]>Cc Keith Busch <[email protected]>Signed-off-by Ming Lei <[email protected]>--- drivers/nvme/host/apple.c | 1 - drivers/nvme/host/pci.c | 1 - drivers/nvme/host/rdma.c | 13 ------------- drivers/nvme/host/tcp.c | 13 ------------- 4 files changed, 28 deletions(-)drivers/nvme/host/apple.c +0
-1@@ -1126,7 +1126,6 @@ static void apple_nvme_reset_work(struct work_struct *work) anv->ctrl.queue_count = nr_io_queues +1 ;nvme_start_queues (&anv->ctrl); -nvme_wait_freeze (&anv->ctrl);blk_mq_update_nr_hw_queues (&anv->tagset,1 );nvme_unfreeze (&anv->ctrl);drivers/nvme/host/pci.c +0
-1@@ -2910,7 +2910,6 @@ static void nvme_reset_work(struct work_struct *work)nvme_free_tagset (dev); }else {nvme_start_queues (&dev->ctrl); -nvme_wait_freeze (&dev->ctrl);if (!dev->ctrl.tagset)nvme_pci_alloc_tag_set (dev);else drivers/nvme/host/rdma.c +0
-0@@ -986,15 +986,6 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)if (!new) {drivers/nvme/host/rdma.c static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new) · line 996
Quoted 10 lines
drivers/nvme/host/rdma.c +0
-9nvme_start_queues (&ctrl->ctrl); -if (!nvme_wait_freeze_timeout (&ctrl->ctrl,NVME_IO_TIMEOUT )) { -/* -* If we timed out waiting for freeze we are likely to -* be stuck. Fail the controller initialization just -* to be safe. -*/ - ret = -ENODEV ; - goto out_wait_freeze_timed_out; - }So here is the description from the patch that introduced this:
--
nvme-rdma: fix reset hang if controller died in the middle of a resetIf the controller becomes unresponsive in the middle of a reset, we will hang because we are waiting for the freeze to complete, but that cannot happen since we have commands that are inflight holding the
q_usage_counter, and we can't blindly fail requests that times out.So give a timeout and if we cannot wait for queue freeze before unfreezing, fail and have the error handling take care how to proceed (either schedule a reconnect of remove the controller). --
So if between
nvme_start_queues()and the freeze (with a full wait) that is done inblk_mq_update_nr_hw_queues()the controller becomes non responsive, in this case we may hang blocking on I/O that was pending and requeued afternvme_start_queues().
Quoted 6 lines
The problem is, that we cannot do any error recovery because the controller is in the middle of a reset/reconnect...
So the code that you deleted was designed to detect this state, and reschedule another reconnect if the controller became non responsive.What is preventing this from happening now?
Please see nvme_rdma_timeout() & nvme_tcp_timeout(), if controller state isn't live, request will be aborted.
Thanks,
Ming
Sagi Grimberg
maintainerQuoted 91 lines · Ming Lei
Quoted 90 lines · Sagi Grimberg
Quoted 5 lines
First it isn't necessary to call
nvme_wait_freezeduring reset. For nvme-pci, if tagset isn't allocated, there can't be any inflight IOs; otherwiseblk_mq_update_nr_hw_queuescan freeze & wait queues.Quoted 7 lines
Second, since commit
bdd6316094e0("block: Allow unfreezing of a queue while requests are in progress"), it is fine to unfreeze queue without draining inflight IOs.Also both nvme-rdma and nvme-tcp's timeout handler provides forward progress if the controller state isn't LIVE, so it is fine to drop the timeout function of
nvme_wait_freeze_timeout().The rdma/tcp should probably be split to separate patches.
Quoted 40 lines
Cc Sagi Grimberg <[email protected]>Cc Chao Leng <[email protected]>Cc Keith Busch <[email protected]>Signed-off-by Ming Lei <[email protected]>--- drivers/nvme/host/apple.c | 1 - drivers/nvme/host/pci.c | 1 - drivers/nvme/host/rdma.c | 13 ------------- drivers/nvme/host/tcp.c | 13 ------------- 4 files changed, 28 deletions(-)drivers/nvme/host/apple.c +0
-1@@ -1126,7 +1126,6 @@ static void apple_nvme_reset_work(struct work_struct *work) anv->ctrl.queue_count = nr_io_queues +1 ;nvme_start_queues (&anv->ctrl); -nvme_wait_freeze (&anv->ctrl);blk_mq_update_nr_hw_queues (&anv->tagset,1 );nvme_unfreeze (&anv->ctrl);drivers/nvme/host/pci.c +0
-1@@ -2910,7 +2910,6 @@ static void nvme_reset_work(struct work_struct *work)nvme_free_tagset (dev); }else {nvme_start_queues (&dev->ctrl); -nvme_wait_freeze (&dev->ctrl);if (!dev->ctrl.tagset)nvme_pci_alloc_tag_set (dev);else drivers/nvme/host/rdma.c +0
-0@@ -986,15 +986,6 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)if (!new) {drivers/nvme/host/rdma.c static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new) · line 996
Quoted 10 lines
drivers/nvme/host/rdma.c +0
-9nvme_start_queues (&ctrl->ctrl); -if (!nvme_wait_freeze_timeout (&ctrl->ctrl,NVME_IO_TIMEOUT )) { -/* -* If we timed out waiting for freeze we are likely to -* be stuck. Fail the controller initialization just -* to be safe. -*/ - ret = -ENODEV ; - goto out_wait_freeze_timed_out; - }So here is the description from the patch that introduced this:
--
nvme-rdma: fix reset hang if controller died in the middle of a resetIf the controller becomes unresponsive in the middle of a reset, we will hang because we are waiting for the freeze to complete, but that cannot happen since we have commands that are inflight holding the
q_usage_counter, and we can't blindly fail requests that times out.So give a timeout and if we cannot wait for queue freeze before unfreezing, fail and have the error handling take care how to proceed (either schedule a reconnect of remove the controller). --
So if between
nvme_start_queues()and the freeze (with a full wait) that is done inblk_mq_update_nr_hw_queues()the controller becomes non responsive, in this case we may hang blocking on I/O that was pending and requeued afternvme_start_queues().The problem is, that we cannot do any error recovery because the controller is in the middle of a reset/reconnect...
So the code that you deleted was designed to detect this state, and reschedule another reconnect if the controller became non responsive.
Quoted 4 lines
Quoted 1 line
What is preventing this from happening now?
Please see
nvme_rdma_timeout()&nvme_tcp_timeout(), if controller state isn't live, request will be aborted.
I agree with you. However non-mpath devices will most likely retry the
command and not fail it like in the multipath case (see nvme_decide_disposition) and will cause the I/O to block.
While it is arguable if non-mpath fabrics devices are important in any capacity, the design was that IO is not completed until the controller either successfully reconnects (and retried), or it disconnects (failed), or fast_io_fail_tmo expires.
Hence for non-mpath controllers, the request(s) will timeout, and aborted, but nvme will opt to retry them instead of completing them with a failure (at least until fast_io_fail_tmo expires, but that can be arbitrarily long).
Quoted 108 lines · Sagi Grimberg
Quoted 91 lines · Ming Lei
Quoted 90 lines · Sagi Grimberg
Quoted 5 lines
First it isn't necessary to call
nvme_wait_freezeduring reset. For nvme-pci, if tagset isn't allocated, there can't be any inflight IOs; otherwiseblk_mq_update_nr_hw_queuescan freeze & wait queues.Quoted 7 lines
Second, since commit
bdd6316094e0("block: Allow unfreezing of a queue while requests are in progress"), it is fine to unfreeze queue without draining inflight IOs.Also both nvme-rdma and nvme-tcp's timeout handler provides forward progress if the controller state isn't LIVE, so it is fine to drop the timeout function of
nvme_wait_freeze_timeout().The rdma/tcp should probably be split to separate patches.
Quoted 40 lines
Cc Sagi Grimberg <[email protected]>Cc Chao Leng <[email protected]>Cc Keith Busch <[email protected]>Signed-off-by Ming Lei <[email protected]>--- drivers/nvme/host/apple.c | 1 - drivers/nvme/host/pci.c | 1 - drivers/nvme/host/rdma.c | 13 ------------- drivers/nvme/host/tcp.c | 13 ------------- 4 files changed, 28 deletions(-)drivers/nvme/host/apple.c +0
-1@@ -1126,7 +1126,6 @@ static void apple_nvme_reset_work(struct work_struct *work) anv->ctrl.queue_count = nr_io_queues +1 ;nvme_start_queues (&anv->ctrl); -nvme_wait_freeze (&anv->ctrl);blk_mq_update_nr_hw_queues (&anv->tagset,1 );nvme_unfreeze (&anv->ctrl);drivers/nvme/host/pci.c +0
-1@@ -2910,7 +2910,6 @@ static void nvme_reset_work(struct work_struct *work)nvme_free_tagset (dev); }else {nvme_start_queues (&dev->ctrl); -nvme_wait_freeze (&dev->ctrl);if (!dev->ctrl.tagset)nvme_pci_alloc_tag_set (dev);else drivers/nvme/host/rdma.c +0
-0@@ -986,15 +986,6 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)if (!new) {drivers/nvme/host/rdma.c static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new) · line 996
Quoted 10 lines
drivers/nvme/host/rdma.c +0
-9nvme_start_queues (&ctrl->ctrl); -if (!nvme_wait_freeze_timeout (&ctrl->ctrl,NVME_IO_TIMEOUT )) { -/* -* If we timed out waiting for freeze we are likely to -* be stuck. Fail the controller initialization just -* to be safe. -*/ - ret = -ENODEV ; - goto out_wait_freeze_timed_out; - }So here is the description from the patch that introduced this:
--
nvme-rdma: fix reset hang if controller died in the middle of a resetIf the controller becomes unresponsive in the middle of a reset, we will hang because we are waiting for the freeze to complete, but that cannot happen since we have commands that are inflight holding the
q_usage_counter, and we can't blindly fail requests that times out.So give a timeout and if we cannot wait for queue freeze before unfreezing, fail and have the error handling take care how to proceed (either schedule a reconnect of remove the controller). --
So if between
nvme_start_queues()and the freeze (with a full wait) that is done inblk_mq_update_nr_hw_queues()the controller becomes non responsive, in this case we may hang blocking on I/O that was pending and requeued afternvme_start_queues().The problem is, that we cannot do any error recovery because the controller is in the middle of a reset/reconnect...
So the code that you deleted was designed to detect this state, and reschedule another reconnect if the controller became non responsive.Quoted 4 lines
Quoted 1 line
What is preventing this from happening now?
Please see
nvme_rdma_timeout()&nvme_tcp_timeout(), if controller state isn't live, request will be aborted.I agree with you. However non-mpath devices will most likely retry the command and not fail it like in the multipath case (see
nvme_decide_disposition) and will cause the I/O to block.While it is arguable if non-mpath fabrics devices are important in any capacity, the design was that IO is not completed until the controller either successfully reconnects (and retried), or it disconnects (failed), or
fast_io_fail_tmoexpires.
Quoted 4 lines
Hence for non-mpath controllers, the request(s) will timeout, and aborted, but nvme will opt to retry them instead of completing them with a failure (at least until
fast_io_fail_tmoexpires, but that can be arbitrarily long).
OK, I think it is better to change the behavior for non-mpath rdma/tcp, will remove it in next version.
thanks,
Ming
Christoph Hellwig
maintainerThanks,
applied for nvme-6.1.
Christoph Hellwig
maintainerQuoted 3 lines · Christoph Hellwig
Thanks,
applied for nvme-6.1.
Sorry, I replied to the wrong mail, so strike this.