nvme-multipath: fix possible io hang after ctrl reconnect
PATCH 1/1 nvme-multipath: fix possible io hang after ctrl reconnect
The following scenario results in an IO hang:
1) ctrl completes a request with NVME_SC_ANA_TRANSITION. NVME_NS_ANA_PENDING bit in ns->flags is set and ana_work is triggered.
2) ana_work: nvme_read_ana_log() tries to get the ANA log page from the ctrl. This fails because ctrl disconnects.
Therefore nvme_update_ns_ana_state() is not called
and NVME_NS_ANA_PENDING bit in ns->flags is not cleared.
3) ctrl reconnects: nvme_mpath_init(ctrl,...) calls nvme_read_ana_log(ctrl, groups_only=true).
However, nvme_update_ana_state() does not update namespaces because nr_nsids = 0 (due to groups_only mode).
4) scan_work calls nvme_validate_ns() finds the ns and re-validates OK.
Result:
The ctrl is now live but NVME_NS_ANA_PENDING bit in ns->flags is still set. Consequently ctrl will never be considered a viable path by __nvme_find_path(). IO will hang if ctrl is the only or the last path to the namespace.
More generally, while ctrl is reconnecting, its ANA state may change. And because nvme_mpath_init() requests ANA log in groups_only mode, these changes are not propagated to the existing ctrl namespaces. This may result in a mal-function or an IO hang.
Solution:nvme_mpath_init() will nvme_read_ana_log() with groups_only set to false. This will not harm the new ctrl case (no namespaces present), and will make sure the ANA state of namespaces gets updated after reconnect. Leaving the groups_only capability in nvme_read_ana_log() in place, although it is unused for now (NVME_ANA_LOG_RGO is defined in spec).
Another option would be for nvme_mpath_init() to invoke nvme_parse_ana_log(..., nvme_set_ns_ana_state) for each existing namespace.
--- drivers/nvme/host/multipath.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/multipath.c +1 -1
@@ -715,7 +715,7 @@ int nvme_mpath_init(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id) goto out; } - error =nvme_read_ana_log (ctrl, true); + error =nvme_read_ana_log (ctrl, false);if (error) goto out_free_ana_log_buf;return 0 ;
Signature · 7 lines
2.14.1 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 45 lines · Anton Eidelman
The following scenario results in an IO hang:
1) ctrl completes a request withNVME_SC_ANA_TRANSITION.NVME_NS_ANA_PENDINGbit inns->flagsis set andana_workis triggered.
2)ana_work:nvme_read_ana_log()tries to get the ANA log page from the ctrl. This fails because ctrl disconnects.
Thereforenvme_update_ns_ana_state()is not called
andNVME_NS_ANA_PENDINGbit inns->flagsis not cleared.
3) ctrl reconnects:nvme_mpath_init(ctrl,...)callsnvme_read_ana_log(ctrl, groups_only=true).
However,nvme_update_ana_state()does not update namespaces becausenr_nsids= 0 (due togroups_onlymode).
4)scan_workcallsnvme_validate_ns()finds the ns and re-validates OK.Result:
The ctrl is now live butNVME_NS_ANA_PENDINGbit inns->flagsis still set. Consequently ctrl will never be considered a viable path by__nvme_find_path(). IO will hang if ctrl is the only or the last path to the namespace.More generally, while ctrl is reconnecting, its ANA state may change. And because
nvme_mpath_init()requests ANA log ingroups_onlymode, these changes are not propagated to the existing ctrl namespaces. This may result in a mal-function or an IO hang.Solution:
nvme_mpath_init()willnvme_read_ana_log()withgroups_onlyset to false. This will not harm the new ctrl case (no namespaces present), and will make sure the ANA state of namespaces gets updated after reconnect. Leaving thegroups_onlycapability innvme_read_ana_log()in place, although it is unused for now (NVME_ANA_LOG_RGOis defined in spec).Another option would be for
nvme_mpath_init()to invokenvme_parse_ana_log(..., nvme_set_ns_ana_state)for each existing namespace.Signed-off-by Anton Eidelman <[email protected]>--- drivers/nvme/host/multipath.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/nvme/host/multipath.c +0
-0
drivers/nvme/host/multipath.c int nvme_mpath_init(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id) · line 721
Quoted 8 lines
drivers/nvme/host/multipath.c +1
-1@@ -715,7 +715,7 @@ int nvme_mpath_init(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id) goto out; } - error =nvme_read_ana_log (ctrl, true); + error =nvme_read_ana_log (ctrl, false);if (error) goto out_free_ana_log_buf;return 0 ;
Actually I think we should drop the RGO handling in nvme_read_ana_log(), seeing the issues it has caused already.
But this can be done in a new patch.
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Teamlead Storage & Networking [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 247165 (AG München), GF: Felix Imendörffer _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Christoph Hellwig
maintainerQuoted 11 lines · Anton Eidelman
The following scenario results in an IO hang:
1) ctrl completes a request withNVME_SC_ANA_TRANSITION.NVME_NS_ANA_PENDINGbit inns->flagsis set andana_workis triggered.
2)ana_work:nvme_read_ana_log()tries to get the ANA log page from the ctrl. This fails because ctrl disconnects.
Thereforenvme_update_ns_ana_state()is not called
andNVME_NS_ANA_PENDINGbit inns->flagsis not cleared.
3) ctrl reconnects:nvme_mpath_init(ctrl,...)callsnvme_read_ana_log(ctrl, groups_only=true).
However,nvme_update_ana_state()does not update namespaces becausenr_nsids= 0 (due togroups_onlymode).
How do you end up calling into nvme_init_identify for a reconnect? That should always hit the !new path in nvme_tcp_configure_admin_queue.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
As far as I can see nvme_init_identify() is called from nvme_xxx_configure_admin_queue() in tcp.c as well as in rdma.c regardless of the value of "new".
This makes sense as over reconnect things in the identify might change for a dynamic controller, e.g. the cntlid.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Christoph Hellwig
maintainerQuoted 5 lines · Anton Eidelman
As far as I can see
nvme_init_identify()is called fromnvme_xxx_configure_admin_queue()intcp.cas well as inrdma.cregardless of the value of "new".
This makes sense as over reconnect things in the identify might change for a dynamic controller, e.g. the cntlid.
True. I guess the patch is fine, but please remove the now dead groups_only code.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v2 1/2 nvme-multipath: fix possible io hang after ctrl reconnect
The following scenario results in an IO hang:
1) ctrl completes a request with NVME_SC_ANA_TRANSITION. NVME_NS_ANA_PENDING bit in ns->flags is set and ana_work is triggered.
2) ana_work: nvme_read_ana_log() tries to get the ANA log page from the ctrl. This fails because ctrl disconnects.
Therefore nvme_update_ns_ana_state() is not called
and NVME_NS_ANA_PENDING bit in ns->flags is not cleared.
3) ctrl reconnects: nvme_mpath_init(ctrl,...) calls nvme_read_ana_log(ctrl, groups_only=true).
However, nvme_update_ana_state() does not update namespaces because nr_nsids = 0 (due to groups_only mode).
4) scan_work calls nvme_validate_ns() finds the ns and re-validates OK.
Result:
The ctrl is now live but NVME_NS_ANA_PENDING bit in ns->flags is still set. Consequently ctrl will never be considered a viable path by __nvme_find_path(). IO will hang if ctrl is the only or the last path to the namespace.
More generally, while ctrl is reconnecting, its ANA state may change. And because nvme_mpath_init() requests ANA log in groups_only mode, these changes are not propagated to the existing ctrl namespaces. This may result in a mal-function or an IO hang.
Solution:nvme_mpath_init() will nvme_read_ana_log() with groups_only set to false. This will not harm the new ctrl case (no namespaces present), and will make sure the ANA state of namespaces gets updated after reconnect.
Note: Another option would be for nvme_mpath_init() to invoke nvme_parse_ana_log(..., nvme_set_ns_ana_state) for each existing namespace.
--- drivers/nvme/host/multipath.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/multipath.c +1 -1
@@ -715,7 +715,7 @@ int nvme_mpath_init(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id) goto out; } - error =nvme_read_ana_log (ctrl, true); + error =nvme_read_ana_log (ctrl, false);if (error) goto out_free_ana_log_buf;return 0 ;
Signature · 7 lines
2.14.1 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v2 2/2 nvme-multipath: remove unused groups_only mode in ana log
groups_only mode in nvme_read_ana_log() is no longer used: remove it.
--- drivers/nvme/host/multipath.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
drivers/nvme/host/multipath.c +4 -5
@@ -522,14 +522,13 @@ static int nvme_update_ana_state(struct nvme_ctrl *ctrl,return 0 ; } -static int nvme_read_ana_log (struct nvme_ctrl *ctrl,bool groups_only) +static int nvme_read_ana_log (struct nvme_ctrl *ctrl) {u32 nr_change_groups =0 ;int error;mutex_lock (&ctrl->ana_lock); - error =nvme_get_log (ctrl,NVME_NSID_ALL ,NVME_LOG_ANA , - groups_only ?NVME_ANA_LOG_RGO :0 , + error =nvme_get_log (ctrl,NVME_NSID_ALL ,NVME_LOG_ANA ,0 , ctrl->ana_log_buf, ctrl->ana_log_size,0 );if (error) {dev_warn (ctrl->device,"Failed to get ANA log: %d\n" , error);
@@ -565,7 +564,7 @@ static void nvme_ana_work(struct work_struct *work) { struct nvme_ctrl *ctrl = container_of(work, struct nvme_ctrl, ana_work); -nvme_read_ana_log (ctrl, false); +nvme_read_ana_log (ctrl); }static void nvme_anatt_timeout (struct timer_list *t)
@@ -715,7 +714,7 @@ int nvme_mpath_init(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id) goto out; } - error =nvme_read_ana_log (ctrl, false); + error =nvme_read_ana_log (ctrl);if (error) goto out_free_ana_log_buf;return 0 ;
Signature · 7 lines
2.14.1 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH v2 2/2] nvme-multipath: remove unused groups_only mode in ana log
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith can you apply these to nvme-5.4-rc?
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerQuoted 3 lines · Sagi Grimberg
Reviewed-by Sagi Grimberg <[email protected]>Keith can you apply these to nvme-5.4-rc?
Yep, applied to nvme-5.4. Thanks!
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme