nvme fabircs: fix io hang in error recovery vs. removal
Subsystems (2)
cover nvme fabircs: fix io hang in error recovery vs. removal
Hello,
The 1st patch fixes io hang when queues are left as quiesced.
The other 2 patches fixes io hang when queues are left as frozen.
All are triggered in error recovery vs. removal.
Ming Lei (3):
nvme: fix possible hang when removing a controller during error
recovery
nvme-tcp: fix potential unbalanced freeze & unfreeze
nvme-rdma: fix potential unbalanced freeze & unfreeze
drivers/nvme/host/core.c | 10 +++++++--- drivers/nvme/host/rdma.c | 3 ++- drivers/nvme/host/tcp.c | 3 ++- 3 files changed, 11 insertions(+), 5 deletions(-)
Signature · 1 line
2.40.1
PATCH v2 1/3 nvme: fix possible hang when removing a controller during error recovery
Error recovery can be interrupted by controller removal, then the controller is left as quiesced, and IO hang can be caused.
Fix the issue by unquiescing controller unconditionally when removing namespaces.
This way is reasonable and safe given forward progress can be made when removing namespaces.
--- drivers/nvme/host/core.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
drivers/nvme/host/core.c +7 -3
@@ -3903,6 +3903,12 @@ void nvme_remove_namespaces(struct nvme_ctrl *ctrl)*/ nvme_mpath_clear_ctrl_paths (ctrl); +/* +* Unquiesce io queues so any pending IO won't hang, especially +* those submitted from scan work +*/ +nvme_unquiesce_io_queues (ctrl); +/* prevent racing with ns scanning */ flush_work (&ctrl->scan_work);
@@ -3912,10 +3918,8 @@ void nvme_remove_namespaces(struct nvme_ctrl *ctrl)* removing the namespaces' disks; fail all the queues now to avoid * potentially having to clean up the failed sync later. */ -if (ctrl->state ==NVME_CTRL_DEAD ) { +if (ctrl->state ==NVME_CTRL_DEAD )nvme_mark_namespaces_dead (ctrl); -nvme_unquiesce_io_queues (ctrl); - }/* this is a no-op when called from the controller reset handler */ nvme_change_ctrl_state (ctrl,NVME_CTRL_DELETING_NOIO );
Signature · 1 line
2.40.1
PATCH v2 2/3 nvme-tcp: fix potential unbalanced freeze & unfreeze
In reply to Ming LeiMove start_freeze into nvme_tcp_configure_io_queues(), and there is at least two benefits:
1) fix unbalanced freeze and unfreeze, since re-connection work may fail or be broken by removal
2) IO during error recovery can be failfast quickly because nvme fabrics unquiesces queues after teardown.
One side-effect is that !mpath request may timeout during connecting because of queue topo change, but that looks not one big deal:
1) same problem exists with current code base
2) compared with !mpath, mpath use case is dominant
--- drivers/nvme/host/tcp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/host/tcp.c +2 -1
@@ -1868,6 +1868,7 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) goto out_cleanup_connect_q;if (!new) { +nvme_start_freeze (ctrl);nvme_unquiesce_io_queues (ctrl);if (!nvme_wait_freeze_timeout (ctrl,NVME_IO_TIMEOUT )) {/*
@@ -1876,6 +1877,7 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)* to be safe. */ ret = -ENODEV ; +nvme_unfreeze (ctrl); goto out_wait_freeze_timed_out; } blk_mq_update_nr_hw_queues(ctrl->tagset,
@@ -1980,7 +1982,6 @@ static void nvme_tcp_teardown_io_queues(struct nvme_ctrl *ctrl,if (ctrl->queue_count <=1 )return ;nvme_quiesce_admin_queue (ctrl); -nvme_start_freeze (ctrl);nvme_quiesce_io_queues (ctrl);nvme_sync_io_queues (ctrl);nvme_tcp_stop_io_queues (ctrl);
Signature · 1 line
2.40.1
Re: [PATCH V2 2/3] nvme-tcp: fix potential unbalanced freeze & unfreeze
Re: [PATCH V2 2/3] nvme-tcp: fix potential unbalanced freeze & unfreeze
In reply to Ming LeiVerified it with the nvme/tcp scenario, Thanks Ming
Quoted 55 lines · Ming Lei
Move
start_freezeintonvme_tcp_configure_io_queues(), and there is at least two benefits:1) fix unbalanced freeze and unfreeze, since re-connection work may fail or be broken by removal
2) IO during error recovery can be failfast quickly because nvme fabrics unquiesces queues after teardown.
One side-effect is that !mpath request may timeout during connecting because of queue topo change, but that looks not one big deal:
1) same problem exists with current code base
2) compared with !mpath, mpath use case is dominant
Fixes 2875b0aecabe ("nvme-tcp: fix controller reset hang during traffic")Signed-off-by Ming Lei <[email protected]>--- drivers/nvme/host/tcp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/nvme/host/tcp.c +2
-2@@ -1868,6 +1868,7 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) goto out_cleanup_connect_q;if (!new) { +nvme_start_freeze (ctrl);nvme_unquiesce_io_queues (ctrl);if (!nvme_wait_freeze_timeout (ctrl,NVME_IO_TIMEOUT )) {/* @@ -1876,6 +1877,7 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)* to be safe. */ ret = -ENODEV ; +nvme_unfreeze (ctrl); goto out_wait_freeze_timed_out; } blk_mq_update_nr_hw_queues(ctrl->tagset,@@ -1980,7 +1982,6 @@ static void nvme_tcp_teardown_io_queues(struct nvme_ctrl *ctrl,if (ctrl->queue_count <=1 )return ;nvme_quiesce_admin_queue (ctrl); -nvme_start_freeze (ctrl);nvme_quiesce_io_queues (ctrl);nvme_sync_io_queues (ctrl);nvme_tcp_stop_io_queues (ctrl); --2.40.1
Signature · 2 lines
Best Regards, Yi Zhang
PATCH v2 3/3 nvme-rdma: fix potential unbalanced freeze & unfreeze
In reply to Ming LeiMove start_freeze into nvme_rdma_configure_io_queues(), and there is at least two benefits:
1) fix unbalanced freeze and unfreeze, since re-connection work may fail or be broken by removal
2) IO during error recovery can be failfast quickly because nvme fabrics unquiesces queues after teardown.
One side-effect is that !mpath request may timeout during connecting because of queue topo change, but that looks not one big deal:
1) same problem exists with current code base
2) compared with !mpath, mpath use case is dominant
--- drivers/nvme/host/rdma.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/host/rdma.c +2 -1
@@ -883,6 +883,7 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new) goto out_cleanup_tagset;if (!new) { +nvme_start_freeze (&ctrl->ctrl);nvme_unquiesce_io_queues (&ctrl->ctrl);if (!nvme_wait_freeze_timeout (&ctrl->ctrl,NVME_IO_TIMEOUT )) {/*
@@ -891,6 +892,7 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)* to be safe. */ ret = -ENODEV ; +nvme_unfreeze (&ctrl->ctrl); goto out_wait_freeze_timed_out; } blk_mq_update_nr_hw_queues(ctrl->ctrl.tagset,
@@ -940,7 +942,6 @@ static void nvme_rdma_teardown_io_queues(struct nvme_rdma_ctrl *ctrl, bool remove) {if (ctrl->ctrl.queue_count >1 ) { -nvme_start_freeze (&ctrl->ctrl);nvme_quiesce_io_queues (&ctrl->ctrl);nvme_sync_io_queues (&ctrl->ctrl);nvme_rdma_stop_io_queues (ctrl);
Signature · 1 line
2.40.1
Re: [PATCH V2 3/3] nvme-rdma: fix potential unbalanced freeze & unfreeze
Re: [PATCH V2 3/3] nvme-rdma: fix potential unbalanced freeze & unfreeze
In reply to Ming LeiVerified it with the nvme/rdma scenario, Thanks Ming
Quoted 55 lines · Ming Lei
Move
start_freezeintonvme_rdma_configure_io_queues(), and there is at least two benefits:1) fix unbalanced freeze and unfreeze, since re-connection work may fail or be broken by removal
2) IO during error recovery can be failfast quickly because nvme fabrics unquiesces queues after teardown.
One side-effect is that !mpath request may timeout during connecting because of queue topo change, but that looks not one big deal:
1) same problem exists with current code base
2) compared with !mpath, mpath use case is dominant
Fixes 9f98772ba307 ("nvme-rdma: fix controller reset hang during traffic")Signed-off-by Ming Lei <[email protected]>--- drivers/nvme/host/rdma.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/nvme/host/rdma.c +2
-2@@ -883,6 +883,7 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new) goto out_cleanup_tagset;if (!new) { +nvme_start_freeze (&ctrl->ctrl);nvme_unquiesce_io_queues (&ctrl->ctrl);if (!nvme_wait_freeze_timeout (&ctrl->ctrl,NVME_IO_TIMEOUT )) {/* @@ -891,6 +892,7 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)* to be safe. */ ret = -ENODEV ; +nvme_unfreeze (&ctrl->ctrl); goto out_wait_freeze_timed_out; } blk_mq_update_nr_hw_queues(ctrl->ctrl.tagset,@@ -940,7 +942,6 @@ static void nvme_rdma_teardown_io_queues(struct nvme_rdma_ctrl *ctrl, bool remove) {if (ctrl->ctrl.queue_count >1 ) { -nvme_start_freeze (&ctrl->ctrl);nvme_quiesce_io_queues (&ctrl->ctrl);nvme_sync_io_queues (&ctrl->ctrl);nvme_rdma_stop_io_queues (ctrl); --2.40.1
Signature · 2 lines
Best Regards, Yi Zhang
Keith Busch
maintainerSeries applied for nvme-6.5. Thanks!