nvme-fc: fix double-free scenarios on hw queues
Subsystems (2)
PATCH 1/1 nvme-fc: fix double-free scenarios on hw queues
If an error occurs on one of the ios used for creating an association, the creating routine has error paths that are invoked by the command failure and the error paths will free up the controller resources created to that point.
But... the io was ultimately determined by an asynchronous completion routine that detected the error and which unconditionally invokes the error_recovery path which calls delete_association. Delete association deletes all outstanding io then tears down the controller resources. So the create_association thread can be running in parallel with the error_recovery thread. What was seen was the LLDD received a call to delete a queue, causing the LLDD to do a free of a resource, then the transport called the delete queue again causing the driver to repeat the free call. The second free routine corrupted the allocator. The transport shouldn't be making the duplicate call, and the delete queue is just one of the resources being freed.
To fix, it is realized that the create_association path is completely serialized with one command at a time. So the failed io completion will always be seen by the create_association path and as of the failure, there are no ios to terminate and there is no reason to be manipulating queue freeze states, etc. The serialized condition stays true until the controller is transitioned to the LIVE state. Thus the fix is to change the error recovery path to check the controller state and only invoke the teardown path if not already in the CONNECTING state.
--- drivers/nvme/host/fc.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-)
drivers/nvme/host/fc.c +15 -3
@@ -2910,10 +2910,22 @@ nvme_fc_reconnect_or_delete(struct nvme_fc_ctrl *ctrl, int status)static void __nvme_fc_terminate_io (struct nvme_fc_ctrl *ctrl) { -nvme_stop_keep_alive (&ctrl->ctrl); +/* +* if state is connecting - the error occurred as part of a +* reconnect attempt. The create_association error paths will +* clean up any outstanding io. +* +* if it's a different state - ensure all pending io is +* terminated. Given this can delay while waiting for the +* aborted io to return, we recheck adapter state below +* before changing state. +*/ +if (ctrl->ctrl.state !=NVME_CTRL_CONNECTING ) { +nvme_stop_keep_alive (&ctrl->ctrl); -/* will block will waiting for io to terminate */ -nvme_fc_delete_association (ctrl); +/* will block will waiting for io to terminate */ +nvme_fc_delete_association (ctrl); + }if (ctrl->ctrl.state !=NVME_CTRL_CONNECTING && !nvme_change_ctrl_state (&ctrl->ctrl,NVME_CTRL_CONNECTING ))
Signature · 7 lines
2.13.7 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 54 lines · James Smart
If an error occurs on one of the ios used for creating an association, the creating routine has error paths that are invoked by the command failure and the error paths will free up the controller resources created to that point.
But... the io was ultimately determined by an asynchronous completion routine that detected the error and which unconditionally invokes the
error_recoverypath which callsdelete_association. Delete association deletes all outstanding io then tears down the controller resources. So thecreate_associationthread can be running in parallel with theerror_recoverythread. What was seen was the LLDD received a call to delete a queue, causing the LLDD to do a free of a resource, then the transport called the delete queue again causing the driver to repeat the free call. The second free routine corrupted the allocator. The transport shouldn't be making the duplicate call, and the delete queue is just one of the resources being freed.To fix, it is realized that the
create_associationpath is completely serialized with one command at a time. So the failed io completion will always be seen by thecreate_associationpath and as of the failure, there are no ios to terminate and there is no reason to be manipulating queue freeze states, etc. The serialized condition stays true until the controller is transitioned to the LIVE state. Thus the fix is to change the error recovery path to check the controller state and only invoke the teardown path if not already in the CONNECTING state.Signed-off-by James Smart <[email protected]>--- drivers/nvme/host/fc.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-)drivers/nvme/host/fc.c +11
-1@@ -2910,10 +2910,22 @@ nvme_fc_reconnect_or_delete(struct nvme_fc_ctrl *ctrl, int status)static void __nvme_fc_terminate_io (struct nvme_fc_ctrl *ctrl) { -nvme_stop_keep_alive (&ctrl->ctrl); +/* +* if state is connecting - the error occurred as part of a +* reconnect attempt. The create_association error paths will +* clean up any outstanding io. +* +* if it's a different state - ensure all pending io is +* terminated. Given this can delay while waiting for the +* aborted io to return, we recheck adapter state below +* before changing state. +*/ +if (ctrl->ctrl.state !=NVME_CTRL_CONNECTING ) {
drivers/nvme/host/fc.c nvme_fc_reconnect_or_delete(struct nvme_fc_ctrl *ctrl, int status) · line 2931
Quoted 10 lines
drivers/nvme/host/fc.c +4
-2+nvme_stop_keep_alive (&ctrl->ctrl); -/* will block will waiting for io to terminate */ -nvme_fc_delete_association (ctrl); +/* will block will waiting for io to terminate */ +nvme_fc_delete_association (ctrl); + }if (ctrl->ctrl.state !=NVME_CTRL_CONNECTING && !nvme_change_ctrl_state (&ctrl->ctrl,NVME_CTRL_CONNECTING ))
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
On 11/21/19, 12:00 PM, "Linux-nvme on behalf of James Smart" <[email protected] on behalf of [email protected]> wrote:
If an error occurs on one of the ios used for creating an association, the creating routine has error paths that are invoked by the command failure and the error paths will free up the controller resources created to that point.
But... the io was ultimately determined by an asynchronous completion routine that detected the error and which unconditionally invokes the error_recovery path which calls delete_association. Delete association deletes all outstanding io then tears down the controller resources. So the create_association thread can be running in parallel with the error_recovery thread. What was seen was the LLDD received a call to delete a queue, causing the LLDD to do a free of a resource, then the transport called the delete queue again causing the driver to repeat the free call. The second free routine corrupted the allocator. The transport shouldn't be making the duplicate call, and the delete queue is just one of the resources being freed.
To fix, it is realized that the create_association path is completely serialized with one command at a time. So the failed io completion will always be seen by the create_association path and as of the failure, there are no ios to terminate and there is no reason to be manipulating queue freeze states, etc. The serialized condition stays true until the controller is transitioned to the LIVE state. Thus the fix is to change the error recovery path to check the controller state and only invoke the teardown path if not already in the CONNECTING state.
---
drivers/nvme/host/fc.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-)
diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 679a721ae229..2acb850bf9f4 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2910,10 +2910,22 @@ nvme_fc_reconnect_or_delete(struct nvme_fc_ctrl *ctrl, int status)
static void
__nvme_fc_terminate_io(struct nvme_fc_ctrl *ctrl)
{
- nvme_stop_keep_alive(&ctrl->ctrl);
+ /*
+ * if state is connecting - the error occurred as part of a
+ * reconnect attempt. The create_association error paths will
+ * clean up any outstanding io.
+ *
+ * if it's a different state - ensure all pending io is
+ * terminated. Given this can delay while waiting for the
+ * aborted io to return, we recheck adapter state below
+ * before changing state.
+ */
+ if (ctrl->ctrl.state != NVME_CTRL_CONNECTING) {
+ nvme_stop_keep_alive(&ctrl->ctrl);
- /* will block will waiting for io to terminate */
- nvme_fc_delete_association(ctrl);
+ /* will block will waiting for io to terminate */
+ nvme_fc_delete_association(ctrl);
+ }
if (ctrl->ctrl.state != NVME_CTRL_CONNECTING &&
!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_CONNECTING))
--
2.13.7
_______________________________________________
Linux-nvme mailing list
[email protected]
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.infradead.org_mailman_listinfo_linux-2Dnvme&d=DwICAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=lanuADUiz2ynuLzGAQkJPTKDNod8UMh5Vi117R7DgS4&m=Vk2A_OK6OxwR1xqGsZrj6snG3gFAGnqmbBuHm9pi_E8&s=wFqks65MM_XSxeEc_PsYTElI1efUDTmkWe_pX3pTGZU&e=
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Applied to nvme/for-5.5
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme