nvme-core: flush namespace scanning work just before removing namespaces
Sagi Grimberg
maintainerPATCH 1/1 nvme-core: flush namespace scanning work just before removing namespaces
nvme_stop_ctrl can be called also for reset flow and there is no need to flush the scan_work as namespaces are not being removed. This can cause deadlock in rdma, fc and loop drivers since nvme_stop_ctrl barriers before controller teardown (and specifically I/O cancellation of the scan_work itself) takes place, but the scan_work will be blocked anyways so there is no need to flush it.
Instead, move scan_work flush to nvme_remove_namespaces() where it really needs to flush.
---
This is a stable candidate...
drivers/nvme/host/core.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +3 -1
@@ -3327,6 +3327,9 @@ void nvme_remove_namespaces(struct nvme_ctrl *ctrl)struct nvme_ns *ns, *next;LIST_HEAD (ns_list); +/* prevent racing with ns scanning */ +flush_work (&ctrl->scan_work); +/* * The dead states indicates the controller was not gracefully * disconnected. In that case, we won't be able to flush any data while
@@ -3489,7 +3492,6 @@ void nvme_stop_ctrl(struct nvme_ctrl *ctrl)nvme_mpath_stop (ctrl);nvme_stop_keep_alive (ctrl);flush_work (&ctrl->async_event_work); -flush_work (&ctrl->scan_work);cancel_work_sync (&ctrl->fw_act_work);if (ctrl->ops->stop_ctrl) ctrl->ops->stop_ctrl (ctrl);
Signature · 1 line
2.17.1
PATCH 1/1 nvme-core: flush namespace scanning work just before removing namespaces
Quoted 33 lines · Sagi Grimberg
nvme_stop_ctrlcan be called also for reset flow and there is no need to flush thescan_workas namespaces are not being removed. This can cause deadlock in rdma, fc and loop drivers sincenvme_stop_ctrlbarriers before controller teardown (and specifically I/O cancellation of thescan_workitself) takes place, but thescan_workwill be blocked anyways so there is no need to flush it.Instead, move
scan_workflush tonvme_remove_namespaces()where it really needs to flush.Reported-by Ming Lei <ming.lei at redhat.com>Signed-off-by Sagi Grimberg <sagi at grimberg.me>---
This is a stable candidate...drivers/nvme/host/core.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/nvme/host/core.c +3
-0@@ -3327,6 +3327,9 @@ void nvme_remove_namespaces(struct nvme_ctrl *ctrl)struct nvme_ns *ns, *next;LIST_HEAD (ns_list); +/* prevent racing with ns scanning */ +flush_work (&ctrl->scan_work); +/* * The dead states indicates the controller was not gracefully * disconnected. In that case, we won't be able to flush any data while
drivers/nvme/host/core.c void nvme_stop_ctrl(struct nvme_ctrl *ctrl) · line 3497
Quoted 7 lines
drivers/nvme/host/core.c +0
-1@@ -3489,7 +3492,6 @@ void nvme_stop_ctrl(struct nvme_ctrl *ctrl)nvme_mpath_stop (ctrl);nvme_stop_keep_alive (ctrl);flush_work (&ctrl->async_event_work); -flush_work (&ctrl->scan_work);cancel_work_sync (&ctrl->fw_act_work);if (ctrl->ops->stop_ctrl) ctrl->ops->stop_ctrl (ctrl);
With this change, I can no longer reproduce the NVMe fabrics reset hangs I was seeing (e.g. after a few hundred resets). A longer-duration fabric connection bounce test survived as well, this change looks good to me.
Thanks for fixing this, Sagi.
PATCH 1/1 nvme-core: flush namespace scanning work just before removing namespaces
In reply to Sagi GrimbergQuoted 7 lines · Sagi Grimberg
nvme_stop_ctrlcan be called also for reset flow and there is no need to flush thescan_workas namespaces are not being removed. This can cause deadlock in rdma, fc and loop drivers sincenvme_stop_ctrlbarriers before controller teardown (and specifically I/O cancellation of thescan_workitself) takes place, but thescan_workwill be blocked anyways so there is no need to flush it.
Quoted 5 lines
Instead, move
scan_workflush tonvme_remove_namespaces()where it really needs to flush.Reported-by Ming Lei <ming.lei at redhat.com>Signed-off-by Sagi Grimberg <sagi at grimberg.me>
Looks good to me.
Quoted 21 lines
drivers/nvme/host/core.c +3
-1@@ -3327,6 +3327,9 @@ void nvme_remove_namespaces(struct nvme_ctrl *ctrl)struct nvme_ns *ns, *next;LIST_HEAD (ns_list); +/* prevent racing with ns scanning */ +flush_work (&ctrl->scan_work); +/* * The dead states indicates the controller was not gracefully * disconnected. In that case, we won't be able to flush any data while @@ -3489,7 +3492,6 @@ void nvme_stop_ctrl(struct nvme_ctrl *ctrl)nvme_mpath_stop (ctrl);nvme_stop_keep_alive (ctrl);flush_work (&ctrl->async_event_work); -flush_work (&ctrl->scan_work);cancel_work_sync (&ctrl->fw_act_work);if (ctrl->ops->stop_ctrl) ctrl->ops->stop_ctrl (ctrl);
PATCH 1/1 nvme-core: flush namespace scanning work just before removing namespaces
In reply to Sagi GrimbergQuoted 10 lines · Sagi Grimberg
nvme_stop_ctrlcan be called also for reset flow and there is no need to flush thescan_workas namespaces are not being removed. This can cause deadlock in rdma, fc and loop drivers sincenvme_stop_ctrlbarriers before controller teardown (and specifically I/O cancellation of thescan_workitself) takes place, but thescan_workwill be blocked anyways so there is no need to flush it.Instead, move
scan_workflush tonvme_remove_namespaces()where it really needs to flush.
Quoted 8 lines
Reported-by Ming Lei <ming.lei at redhat.com>Signed-off-by Sagi Grimberg <sagi at grimberg.me>---
This is a stable candidate...drivers/nvme/host/core.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
Looks good to me.
Reviewed by:? James Smart? <jsmart2021 at gmail.com>
-- james
PATCH 1/1 nvme-core: flush namespace scanning work just before removing namespaces
In reply to Sagi GrimbergThanks,
applied to nvme-4.20.