: nvme-fc: fix race with connectivity loss with nvme_fc_create_association
Subsystems (2)
cover : nvme-fc: fix race with connectivity loss with nvme_fc_create_association
We got a bug report that a controller was stuck in the connected state after an association dropped. It turns out that nvme_fc_create_association can succeed even though some operation do fail. This is on purpose to handle other corner cases, thus we can't really make nvme_fc_create_assocication bullet proof.
Given this, I think the second best option is to ensure that we always schedule a reset whenever a connectivity loss is observed in CONNECTING state. This is not perfect (possible back to back reset) but at least it is correct, which wins in my books.
Daniel Wagner (2):
nvme-fc: go straight to connecting state when initializing
nvme-fc: fix race with connectivity loss with
nvme_fc_create_association
drivers/nvme/host/fc.c | 27 ++++++++++++++++++++++----- 1 file changed, 22 insertions(+), 5 deletions(-)
Signature · 1 line
2.45.2
PATCH 1/2 nvme-fc: go straight to connecting state when initializing
The initial controller initialization mimiks the reconnect loop behavior by switching from NEW to RESETTING and then to CONNECTING.
The transition from NEW to CONNECTING is a valid transition, so there is no point entering the RESETTING state. TCP and RDMA also transition directly to CONNECTING state.
--- drivers/nvme/host/fc.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/nvme/host/fc.c +1 -2
@@ -3551,8 +3551,7 @@ nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts,list_add_tail (&ctrl->ctrl_list, &rport->ctrl_list);spin_unlock_irqrestore (&rport->lock, flags); -if (!nvme_change_ctrl_state (&ctrl->ctrl,NVME_CTRL_RESETTING ) || - !nvme_change_ctrl_state (&ctrl->ctrl,NVME_CTRL_CONNECTING )) { +if (!nvme_change_ctrl_state (&ctrl->ctrl,NVME_CTRL_CONNECTING )) {dev_err (ctrl->ctrl.device,"NVME-FC{%d}: failed to init ctrl state\n" , ctrl->cnum); goto fail_ctrl;
Signature · 1 line
2.45.2
Re: [PATCH v1 1/2] nvme-fc: go straight to connecting state when initializing
Re: [PATCH v1 1/2] nvme-fc: go straight to connecting state when initializing
In reply to Daniel WagnerQuoted 17 lines · Daniel Wagner
The initial controller initialization mimiks the reconnect loop behavior by switching from NEW to RESETTING and then to CONNECTING.
The transition from NEW to CONNECTING is a valid transition, so there is no point entering the RESETTING state. TCP and RDMA also transition directly to CONNECTING state.
Signed-off-by Daniel Wagner <[email protected]>--- drivers/nvme/host/fc.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)drivers/nvme/host/fc.c +0
-0
drivers/nvme/host/fc.c nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts, · line 3557
Quoted 9 lines
drivers/nvme/host/fc.c +1
-2@@ -3551,8 +3551,7 @@ nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts,list_add_tail (&ctrl->ctrl_list, &rport->ctrl_list);spin_unlock_irqrestore (&rport->lock, flags); -if (!nvme_change_ctrl_state (&ctrl->ctrl,NVME_CTRL_RESETTING ) || - !nvme_change_ctrl_state (&ctrl->ctrl,NVME_CTRL_CONNECTING )) { +if (!nvme_change_ctrl_state (&ctrl->ctrl,NVME_CTRL_CONNECTING )) {dev_err (ctrl->ctrl.device,"NVME-FC{%d}: failed to init ctrl state\n" , ctrl->cnum); goto fail_ctrl;
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Ivo Totev, Andrew McDonald, Werner Knoblich
PATCH 2/2 nvme-fc: fix race with connectivity loss with nvme_fc_create_association
In reply to Daniel WagnerWe are racing with nvme_fc_create_association here. We wouldn't have to do anything in nvme_fc_ctrl_connectivity_loss IF nvme_fc_create_association would reliable detect that the setup failed. In reality, nvme_fc_create_association is able to return success even though some of the operation fail during the setup (and this is even on purpose):
nvme nvme10: NVME-FC{10}: create association : ...
nvme nvme10: NVME-FC{10}: controller connectivity lost. Awaiting Reconnect nvme nvme10: queue_size 128 > ctrl maxcmd 32, reducing to maxcmd nvme nvme10: Could not set queue count (880)
nvme nvme10: Failed to configure AEN (cfg 900)
nvme nvme10: NVME-FC{10}: controller connect complete
Thus we transition to the CONNECTED state and don't start the reconnect loop. Thus we should unconditionally schedule a reset.
If a reset has already been schedule it's either a no-op or we observe back to back reset. But we can't really distinguish between host side double scheduled reset work or a connectivity loss event right after a reset. Thus we have to live with this corner case. The system will enter the correct state eventually.
--- drivers/nvme/host/fc.c | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)
drivers/nvme/host/fc.c +21 -3
@@ -169,6 +169,7 @@ struct nvme_fc_ctrl {struct work_struct ioerr_work;struct delayed_work connect_work; +struct work_struct fc_reset_work;struct kref ref;unsigned long flags;
@@ -779,6 +780,14 @@ nvme_fc_abort_lsops(struct nvme_fc_rport *rport)return 0 ; } +static void nvme_fc_defer_reset_work (struct work_struct *work) +{ +struct nvme_fc_ctrl *ctrl = + container_of(work, struct nvme_fc_ctrl, fc_reset_work); + +nvme_reset_ctrl (&ctrl->ctrl); +} +static void nvme_fc_ctrl_connectivity_loss (struct nvme_fc_ctrl *ctrl) {
@@ -806,12 +815,20 @@ nvme_fc_ctrl_connectivity_loss(struct nvme_fc_ctrl *ctrl)case NVME_CTRL_CONNECTING :/* -* The association has already been terminated and the -* controller is attempting reconnects. No need to do anything -* futher. Reconnects will be attempted until either the +* We are racing with nvme_fc_create_association here. Ensure we +* never lose the connectivity loss event and unconditionally +* schedule a reset. +* +* The reconnects will be attempted until either the * ctlr_loss_tmo (max_retries * connect_delay) expires or the * remoteport's dev_loss_tmo expires. +* +* We can't sleep in this context (e.g flush_delayed_work) but +* we should execute reset after the connect work item has +* finished. Thus defer the reset trigger to the nvme_wq where +* the connect is executed. */ +queue_work (nvme_wq, &ctrl->fc_reset_work);break ;case NVME_CTRL_RESETTING :
@@ -3505,6 +3522,7 @@ nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts,INIT_WORK (&ctrl->ctrl.reset_work, nvme_fc_reset_ctrl_work);INIT_DELAYED_WORK (&ctrl->connect_work, nvme_fc_connect_ctrl_work);INIT_WORK (&ctrl->ioerr_work, nvme_fc_ctrl_ioerr_work); +INIT_WORK (&ctrl->fc_reset_work, nvme_fc_defer_reset_work);spin_lock_init (&ctrl->lock);/* io queue count */
Signature · 1 line
2.45.2
Sagi Grimberg
maintainerRe: [PATCH v1 2/2] nvme-fc: fix race with connectivity loss with nvme_fc_create_association
Quoted 41 lines · Daniel Wagner
We are racing with
nvme_fc_create_associationhere. We wouldn't have to do anything innvme_fc_ctrl_connectivity_lossIFnvme_fc_create_associationwould reliable detect that the setup failed. In reality,nvme_fc_create_associationis able to return success even though some of the operation fail during the setup (and this is even on purpose):nvme nvme10: NVME-FC{10}: create association : ...
nvme nvme10: NVME-FC{10}: controller connectivity lost. Awaiting Reconnect nvme nvme10:queue_size128 > ctrl maxcmd 32, reducing to maxcmd nvme nvme10: Could not set queue count (880)
nvme nvme10: Failed to configure AEN (cfg 900)
nvme nvme10: NVME-FC{10}: controller connect completeThus we transition to the CONNECTED state and don't start the reconnect loop. Thus we should unconditionally schedule a reset.
If a reset has already been schedule it's either a no-op or we observe back to back reset. But we can't really distinguish between host side double scheduled reset work or a connectivity loss event right after a reset. Thus we have to live with this corner case. The system will enter the correct state eventually.
Signed-off-by Daniel Wagner <[email protected]>--- drivers/nvme/host/fc.c | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)drivers/nvme/host/fc.c +1
-0@@ -169,6 +169,7 @@ struct nvme_fc_ctrl {struct work_struct ioerr_work;struct delayed_work connect_work; +struct work_struct fc_reset_work;struct kref ref;unsigned long flags;
drivers/nvme/host/fc.c nvme_fc_abort_lsops(struct nvme_fc_rport *rport) · line 789
Quoted 10 lines
drivers/nvme/host/fc.c +7
-0@@ -779,6 +780,14 @@ nvme_fc_abort_lsops(struct nvme_fc_rport *rport)return 0 ; } +static void nvme_fc_defer_reset_work (struct work_struct *work) +{ +struct nvme_fc_ctrl *ctrl = + container_of(work, struct nvme_fc_ctrl, fc_reset_work); + +nvme_reset_ctrl (&ctrl->ctrl); +}
I'm not entirely sure I understand what you're trying to solve here, but scheduling a work that in turn schedules a work looks bogus.
Re: [PATCH v1 2/2] nvme-fc: fix race with connectivity loss with nvme_fc_create_association
Quoted 10 lines · Sagi Grimberg
drivers/nvme/host/fc.c nvme_fc_abort_lsops(struct nvme_fc_rport *rport) · line 789
Quoted 7 lines
drivers/nvme/host/fc.c +7
-0+static void nvme_fc_defer_reset_work (struct work_struct *work) +{ +struct nvme_fc_ctrl *ctrl = + container_of(work, struct nvme_fc_ctrl, fc_reset_work); + +nvme_reset_ctrl (&ctrl->ctrl); +}I'm not entirely sure I understand what you're trying to solve here, but scheduling a work that in turn schedules a work looks bogus.
nvme_fc_ctrl_connectivity_loss is called from a interrupt context. If we are in the connecting state we know that connect work job is still executing. I figured we should wait until it has finished (see next paragraph) and then issue an reset. If nvme_fc_ctrl_connectivity_loss was called from normal context a flushed_delayed_work would do the trick. The nvme_fc_defer_reset_work is queued/executed on the normal nvme_wq thus the connecting work has finished.
The other idea I had was to issue the reset immediately and handle the fallouts in the connect code path. But it turns out the connect path is somewhat expecting errors and just ignores them on purpose and thus still reaches the connected state.
I agree, it's dodgy but I haven't found any better solution so far. Any good idea how to solve this are highly appreciated.
Re: [PATCH v1 2/2] nvme-fc: fix race with connectivity loss with nvme_fc_create_association
In reply to Daniel WagnerQuoted 23 lines · Daniel Wagner
We are racing with
nvme_fc_create_associationhere. We wouldn't have to do anything innvme_fc_ctrl_connectivity_lossIFnvme_fc_create_associationwould reliable detect that the setup failed. In reality,nvme_fc_create_associationis able to return success even though some of the operation fail during the setup (and this is even on purpose):nvme nvme10: NVME-FC{10}: create association : ...
nvme nvme10: NVME-FC{10}: controller connectivity lost. Awaiting Reconnect nvme nvme10:queue_size128 > ctrl maxcmd 32, reducing to maxcmd nvme nvme10: Could not set queue count (880)
nvme nvme10: Failed to configure AEN (cfg 900)
nvme nvme10: NVME-FC{10}: controller connect completeThus we transition to the CONNECTED state and don't start the reconnect loop. Thus we should unconditionally schedule a reset.
If a reset has already been schedule it's either a no-op or we observe back to back reset. But we can't really distinguish between host side double scheduled reset work or a connectivity loss event right after a reset. Thus we have to live with this corner case. The system will enter the correct state eventually.
Quoted 5 lines
Signed-off-by Daniel Wagner <[email protected]>--- drivers/nvme/host/fc.c | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Ivo Totev, Andrew McDonald, Werner Knoblich
A message that is not in the archive (sent off-list or not yet received): CABPRKS_rC3eXiDGTORhRVUpFLXZWPrmd8kRwVL8FdAvX3aVu-A@mail.gmail.com
Re: [PATCH v1 2/2] nvme-fc: fix race with connectivity loss with nvme_fc_create_association
Quoted 98 lines
From: Daniel Wagner <[email protected]>
Date: Tue, Jun 11, 2024 at 12:15 PM
Subject: [PATCH v1 2/2] nvme-fc: fix race with connectivity loss withnvme_fc_create_association
To: James Smart <[email protected]>
Cc: Keith Busch <[email protected]>, Christoph Hellwig <[email protected]>, Sagi Grimberg <[email protected]>, <[email protected]>, Hannes Reinecke <[email protected]>, Daniel Wagner <[email protected]>We are racing with
nvme_fc_create_associationhere. We wouldn't have to do anything innvme_fc_ctrl_connectivity_lossIFnvme_fc_create_associationwould reliable detect that the setup failed. In reality,nvme_fc_create_associationis able to return success even though some of the operation fail during the setup (and this is even on purpose):nvme nvme10: NVME-FC{10}: create association : ...
nvme nvme10: NVME-FC{10}: controller connectivity lost. Awaiting Reconnect nvme nvme10:queue_size128 > ctrl maxcmd 32, reducing to maxcmd nvme nvme10: Could not set queue count (880)
nvme nvme10: Failed to configure AEN (cfg 900)
nvme nvme10: NVME-FC{10}: controller connect completeThus we transition to the CONNECTED state and don't start the reconnect loop. Thus we should unconditionally schedule a reset.
If a reset has already been schedule it's either a no-op or we observe back to back reset. But we can't really distinguish between host side double scheduled reset work or a connectivity loss event right after a reset. Thus we have to live with this corner case. The system will enter the correct state eventually.
Signed-off-by Daniel Wagner <[email protected]>--- drivers/nvme/host/fc.c | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)drivers/nvme/host/fc.c +21
-3@@ -169,6 +169,7 @@ struct nvme_fc_ctrl {struct work_struct ioerr_work;struct delayed_work connect_work; +struct work_struct fc_reset_work;struct kref ref;unsigned long flags;@@ -779,6 +780,14 @@ nvme_fc_abort_lsops(struct nvme_fc_rport *rport)return 0 ; } +static void nvme_fc_defer_reset_work (struct work_struct *work) +{ +struct nvme_fc_ctrl *ctrl = + container_of(work, struct nvme_fc_ctrl, fc_reset_work); + +nvme_reset_ctrl (&ctrl->ctrl); +} +static void nvme_fc_ctrl_connectivity_loss (struct nvme_fc_ctrl *ctrl) {@@ -806,12 +815,20 @@ nvme_fc_ctrl_connectivity_loss(struct nvme_fc_ctrl *ctrl)case NVME_CTRL_CONNECTING :/* -* The association has already been terminated and the -* controller is attempting reconnects. No need to do anything -* futher. Reconnects will be attempted until either the +* We are racing with nvme_fc_create_association here. Ensure we +* never lose the connectivity loss event and unconditionally +* schedule a reset. +* +* The reconnects will be attempted until either the * ctlr_loss_tmo (max_retries * connect_delay) expires or the * remoteport's dev_loss_tmo expires. +* +* We can't sleep in this context (e.g flush_delayed_work) but +* we should execute reset after the connect work item has +* finished. Thus defer the reset trigger to the nvme_wq where +* the connect is executed. */ +queue_work (nvme_wq, &ctrl->fc_reset_work);break ;case NVME_CTRL_RESETTING :@@ -3505,6 +3522,7 @@ nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts,INIT_WORK (&ctrl->ctrl.reset_work, nvme_fc_reset_ctrl_work);INIT_DELAYED_WORK (&ctrl->connect_work, nvme_fc_connect_ctrl_work);INIT_WORK (&ctrl->ioerr_work, nvme_fc_ctrl_ioerr_work); +INIT_WORK (&ctrl->fc_reset_work, nvme_fc_defer_reset_work);spin_lock_init (&ctrl->lock);/* io queue count */
Quoted 2 lines
--
2.45.2
I tested this change with multiple NVME target vendors, multiple subsystems and namespaces
and fault injection with IO running. This test reproduced the controllers stuck in "Connecting"
state 100% of the time and now run clean for the test duration.
Signature · 11 lines
This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.
Quoted 9 lines · Daniel Wagner
We got a bug report that a controller was stuck in the connected state after an association dropped. It turns out that
nvme_fc_create_associationcan succeed even though some operation do fail. This is on purpose to handle other corner cases, thus we can't really makenvme_fc_create_associcationbullet proof.Given this, I think the second best option is to ensure that we always schedule a reset whenever a connectivity loss is observed in CONNECTING state. This is not perfect (possible back to back reset) but at least it is correct, which wins in my books.
Any feedback?