Skip to contentSkip to conversation

: nvme-fc: fix race with connectivity loss with nvme_fc_create_association

PATCH In review Daniel Wagner 4 participants 10 messages Jun 11, 2024 – Jul 26, 2024 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover : nvme-fc: fix race with connectivity loss with nvme_fc_create_association

We got a bug report that a controller was stuck in the connected state after an association dropped. It turns out that nvme_fc_create_association can succeed even though some operation do fail. This is on purpose to handle other corner cases, thus we can't really make nvme_fc_create_assocication bullet proof.

Given this, I think the second best option is to ensure that we always schedule a reset whenever a connectivity loss is observed in CONNECTING state. This is not perfect (possible back to back reset) but at least it is correct, which wins in my books.

Daniel Wagner (2):
nvme-fc: go straight to connecting state when initializing
nvme-fc: fix race with connectivity loss with
nvme_fc_create_association

 drivers/nvme/host/fc.c | 27 ++++++++++++++++++++++-----
 1 file changed, 22 insertions(+), 5 deletions(-)
Signature · 1 line
2.45.2
#1 lore raw

PATCH 1/2 nvme-fc: go straight to connecting state when initializing

The initial controller initialization mimiks the reconnect loop behavior by switching from NEW to RESETTING and then to CONNECTING.

The transition from NEW to CONNECTING is a valid transition, so there is no point entering the RESETTING state. TCP and RDMA also transition directly to CONNECTING state.

---
 drivers/nvme/host/fc.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/nvme/host/fc.c +1 -2
index f0b081332749..6ff28d59b169 100644
@@ -3551,8 +3551,7 @@ nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts,
 	list_add_tail(&ctrl->ctrl_list, &rport->ctrl_list);
 	spin_unlock_irqrestore(&rport->lock, flags);
 
-	if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_RESETTING) ||
-	    !nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_CONNECTING)) {
+	if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_CONNECTING)) {
 		dev_err(ctrl->ctrl.device,
 			"NVME-FC{%d}: failed to init ctrl state\n", ctrl->cnum);
 		goto fail_ctrl;
Signature · 1 line
2.45.2
Reviewed-by on v1 1/2
#3 lore raw

Re: [PATCH v1 1/2] nvme-fc: go straight to connecting state when initializing

In reply to Daniel Wagner
Quoted 17 lines · Daniel Wagner

The initial controller initialization mimiks the reconnect loop behavior by switching from NEW to RESETTING and then to CONNECTING.

The transition from NEW to CONNECTING is a valid transition, so there is no point entering the RESETTING state. TCP and RDMA also transition directly to CONNECTING state.

---
  drivers/nvme/host/fc.c | 3 +--
  1 file changed, 1 insertion(+), 2 deletions(-)
drivers/nvme/host/fc.c +0 -0
index f0b081332749..6ff28d59b169 100644

drivers/nvme/host/fc.c nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts, · line 3557

Quoted 9 lines
drivers/nvme/host/fc.c +1 -2
@@ -3551,8 +3551,7 @@ nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts,
  	list_add_tail(&ctrl->ctrl_list, &rport->ctrl_list);
  	spin_unlock_irqrestore(&rport->lock, flags);
 
-	if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_RESETTING) ||
-	    !nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_CONNECTING)) {
+	if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_CONNECTING)) {
  		dev_err(ctrl->ctrl.device,
  			"NVME-FC{%d}: failed to init ctrl state\n", ctrl->cnum);
  		goto fail_ctrl;

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Ivo Totev, Andrew McDonald,
Werner Knoblich
#4 lore raw

PATCH 2/2 nvme-fc: fix race with connectivity loss with nvme_fc_create_association

In reply to Daniel Wagner

We are racing with nvme_fc_create_association here. We wouldn't have to do anything in nvme_fc_ctrl_connectivity_loss IF nvme_fc_create_association would reliable detect that the setup failed. In reality, nvme_fc_create_association is able to return success even though some of the operation fail during the setup (and this is even on purpose):

nvme nvme10: NVME-FC{10}: create association : ...
nvme nvme10: NVME-FC{10}: controller connectivity lost. Awaiting Reconnect nvme nvme10: queue_size 128 > ctrl maxcmd 32, reducing to maxcmd nvme nvme10: Could not set queue count (880)
nvme nvme10: Failed to configure AEN (cfg 900)
nvme nvme10: NVME-FC{10}: controller connect complete

Thus we transition to the CONNECTED state and don't start the reconnect loop. Thus we should unconditionally schedule a reset.

If a reset has already been schedule it's either a no-op or we observe back to back reset. But we can't really distinguish between host side double scheduled reset work or a connectivity loss event right after a reset. Thus we have to live with this corner case. The system will enter the correct state eventually.

---
 drivers/nvme/host/fc.c | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)
drivers/nvme/host/fc.c +21 -3
index 6ff28d59b169..10dc456928a2 100644
@@ -169,6 +169,7 @@ struct nvme_fc_ctrl {
 
 	struct work_struct	ioerr_work;
 	struct delayed_work	connect_work;
+	struct work_struct	fc_reset_work;
 
 	struct kref		ref;
 	unsigned long		flags;
@@ -779,6 +780,14 @@ nvme_fc_abort_lsops(struct nvme_fc_rport *rport)
 	return 0;
 }
 
+static void nvme_fc_defer_reset_work(struct work_struct *work)
+{
+	struct nvme_fc_ctrl *ctrl =
+		container_of(work, struct nvme_fc_ctrl, fc_reset_work);
+
+	nvme_reset_ctrl(&ctrl->ctrl);
+}
+
 static void
 nvme_fc_ctrl_connectivity_loss(struct nvme_fc_ctrl *ctrl)
 {
@@ -806,12 +815,20 @@ nvme_fc_ctrl_connectivity_loss(struct nvme_fc_ctrl *ctrl)
 
 	case NVME_CTRL_CONNECTING:
 		/*
-		 * The association has already been terminated and the
-		 * controller is attempting reconnects.  No need to do anything
-		 * futher.  Reconnects will be attempted until either the
+		 * We are racing with nvme_fc_create_association here. Ensure we
+		 * never lose the connectivity loss event and unconditionally
+		 * schedule a reset.
+		 *
+		 * The reconnects will be attempted until either the
 		 * ctlr_loss_tmo (max_retries * connect_delay) expires or the
 		 * remoteport's dev_loss_tmo expires.
+		 *
+		 * We can't sleep in this context (e.g flush_delayed_work) but
+		 * we should execute reset after the connect work item has
+		 * finished. Thus defer the reset trigger to the nvme_wq where
+		 * the connect is executed.
 		 */
+		queue_work(nvme_wq, &ctrl->fc_reset_work);
 		break;
 
 	case NVME_CTRL_RESETTING:
@@ -3505,6 +3522,7 @@ nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts,
 	INIT_WORK(&ctrl->ctrl.reset_work, nvme_fc_reset_ctrl_work);
 	INIT_DELAYED_WORK(&ctrl->connect_work, nvme_fc_connect_ctrl_work);
 	INIT_WORK(&ctrl->ioerr_work, nvme_fc_ctrl_ioerr_work);
+	INIT_WORK(&ctrl->fc_reset_work, nvme_fc_defer_reset_work);
 	spin_lock_init(&ctrl->lock);
 
 	/* io queue count */
Signature · 1 line
2.45.2
maintainer
#5 lore raw

Re: [PATCH v1 2/2] nvme-fc: fix race with connectivity loss with nvme_fc_create_association

Quoted 41 lines · Daniel Wagner

We are racing with nvme_fc_create_association here. We wouldn't have to do anything in nvme_fc_ctrl_connectivity_loss IF nvme_fc_create_association would reliable detect that the setup failed. In reality, nvme_fc_create_association is able to return success even though some of the operation fail during the setup (and this is even on purpose):

nvme nvme10: NVME-FC{10}: create association : ...
nvme nvme10: NVME-FC{10}: controller connectivity lost. Awaiting Reconnect nvme nvme10: queue_size 128 > ctrl maxcmd 32, reducing to maxcmd nvme nvme10: Could not set queue count (880)
nvme nvme10: Failed to configure AEN (cfg 900)
nvme nvme10: NVME-FC{10}: controller connect complete

Thus we transition to the CONNECTED state and don't start the reconnect loop. Thus we should unconditionally schedule a reset.

If a reset has already been schedule it's either a no-op or we observe back to back reset. But we can't really distinguish between host side double scheduled reset work or a connectivity loss event right after a reset. Thus we have to live with this corner case. The system will enter the correct state eventually.

---
  drivers/nvme/host/fc.c | 24 +++++++++++++++++++++---
  1 file changed, 21 insertions(+), 3 deletions(-)
drivers/nvme/host/fc.c +1 -0
index 6ff28d59b169..10dc456928a2 100644
@@ -169,6 +169,7 @@ struct nvme_fc_ctrl {
 
  	struct work_struct	ioerr_work;
  	struct delayed_work	connect_work;
+	struct work_struct	fc_reset_work;
 
  	struct kref		ref;
  	unsigned long		flags;

drivers/nvme/host/fc.c nvme_fc_abort_lsops(struct nvme_fc_rport *rport) · line 789

Quoted 10 lines
drivers/nvme/host/fc.c +7 -0
@@ -779,6 +780,14 @@ nvme_fc_abort_lsops(struct nvme_fc_rport *rport)
  	return 0;
  }
 
+static void nvme_fc_defer_reset_work(struct work_struct *work)
+{
+	struct nvme_fc_ctrl *ctrl =
+		container_of(work, struct nvme_fc_ctrl, fc_reset_work);
+
+	nvme_reset_ctrl(&ctrl->ctrl);
+}

I'm not entirely sure I understand what you're trying to solve here, but scheduling a work that in turn schedules a work looks bogus.

#6 lore raw

Re: [PATCH v1 2/2] nvme-fc: fix race with connectivity loss with nvme_fc_create_association

Quoted 10 lines · Sagi Grimberg

drivers/nvme/host/fc.c nvme_fc_abort_lsops(struct nvme_fc_rport *rport) · line 789

Quoted 7 lines
drivers/nvme/host/fc.c +7 -0
+static void nvme_fc_defer_reset_work(struct work_struct *work)
+{
+	struct nvme_fc_ctrl *ctrl =
+		container_of(work, struct nvme_fc_ctrl, fc_reset_work);
+
+	nvme_reset_ctrl(&ctrl->ctrl);
+}

I'm not entirely sure I understand what you're trying to solve here, but scheduling a work that in turn schedules a work looks bogus.

nvme_fc_ctrl_connectivity_loss is called from a interrupt context. If we are in the connecting state we know that connect work job is still executing. I figured we should wait until it has finished (see next paragraph) and then issue an reset. If nvme_fc_ctrl_connectivity_loss was called from normal context a flushed_delayed_work would do the trick. The nvme_fc_defer_reset_work is queued/executed on the normal nvme_wq thus the connecting work has finished.

The other idea I had was to issue the reset immediately and handle the fallouts in the connect code path. But it turns out the connect path is somewhat expecting errors and just ignores them on purpose and thus still reaches the connected state.

I agree, it's dodgy but I haven't found any better solution so far. Any good idea how to solve this are highly appreciated.

Reviewed-by on v1 2/2
#7 lore raw

Re: [PATCH v1 2/2] nvme-fc: fix race with connectivity loss with nvme_fc_create_association

In reply to Daniel Wagner
Quoted 23 lines · Daniel Wagner

We are racing with nvme_fc_create_association here. We wouldn't have to do anything in nvme_fc_ctrl_connectivity_loss IF nvme_fc_create_association would reliable detect that the setup failed. In reality, nvme_fc_create_association is able to return success even though some of the operation fail during the setup (and this is even on purpose):

nvme nvme10: NVME-FC{10}: create association : ...
nvme nvme10: NVME-FC{10}: controller connectivity lost. Awaiting Reconnect nvme nvme10: queue_size 128 > ctrl maxcmd 32, reducing to maxcmd nvme nvme10: Could not set queue count (880)
nvme nvme10: Failed to configure AEN (cfg 900)
nvme nvme10: NVME-FC{10}: controller connect complete

Thus we transition to the CONNECTED state and don't start the reconnect loop. Thus we should unconditionally schedule a reset.

If a reset has already been schedule it's either a no-op or we observe back to back reset. But we can't really distinguish between host side double scheduled reset work or a connectivity loss event right after a reset. Thus we have to live with this corner case. The system will enter the correct state eventually.

Quoted 5 lines
---
  drivers/nvme/host/fc.c | 24 +++++++++++++++++++++---
  1 file changed, 21 insertions(+), 3 deletions(-)

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Ivo Totev, Andrew McDonald,
Werner Knoblich

A message that is not in the archive (sent off-list or not yet received): CABPRKS_rC3eXiDGTORhRVUpFLXZWPrmd8kRwVL8FdAvX3aVu-A@mail.gmail.com

maintainer Reviewed-by
#9 lore raw

Re: [PATCH v1 2/2] nvme-fc: fix race with connectivity loss with nvme_fc_create_association

Quoted 98 lines

From: Daniel Wagner <[email protected]>
Date: Tue, Jun 11, 2024 at 12:15 PM
Subject: [PATCH v1 2/2] nvme-fc: fix race with connectivity loss with nvme_fc_create_association
To: James Smart <[email protected]>
Cc: Keith Busch <[email protected]>, Christoph Hellwig <[email protected]>, Sagi Grimberg <[email protected]>, <[email protected]>, Hannes Reinecke <[email protected]>, Daniel Wagner <[email protected]>

We are racing with nvme_fc_create_association here. We wouldn't have to do anything in nvme_fc_ctrl_connectivity_loss IF nvme_fc_create_association would reliable detect that the setup failed. In reality, nvme_fc_create_association is able to return success even though some of the operation fail during the setup (and this is even on purpose):

nvme nvme10: NVME-FC{10}: create association : ...
nvme nvme10: NVME-FC{10}: controller connectivity lost. Awaiting Reconnect nvme nvme10: queue_size 128 > ctrl maxcmd 32, reducing to maxcmd nvme nvme10: Could not set queue count (880)
nvme nvme10: Failed to configure AEN (cfg 900)
nvme nvme10: NVME-FC{10}: controller connect complete

Thus we transition to the CONNECTED state and don't start the reconnect loop. Thus we should unconditionally schedule a reset.

If a reset has already been schedule it's either a no-op or we observe back to back reset. But we can't really distinguish between host side double scheduled reset work or a connectivity loss event right after a reset. Thus we have to live with this corner case. The system will enter the correct state eventually.

---
 drivers/nvme/host/fc.c | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)
drivers/nvme/host/fc.c +21 -3
index 6ff28d59b169..10dc456928a2 100644
@@ -169,6 +169,7 @@ struct nvme_fc_ctrl {
 
        struct work_struct      ioerr_work;
        struct delayed_work     connect_work;
+       struct work_struct      fc_reset_work;
 
        struct kref             ref;
        unsigned long           flags;
@@ -779,6 +780,14 @@ nvme_fc_abort_lsops(struct nvme_fc_rport *rport)
        return 0;
 }
 
+static void nvme_fc_defer_reset_work(struct work_struct *work)
+{
+       struct nvme_fc_ctrl *ctrl =
+               container_of(work, struct nvme_fc_ctrl, fc_reset_work);
+
+       nvme_reset_ctrl(&ctrl->ctrl);
+}
+
 static void
 nvme_fc_ctrl_connectivity_loss(struct nvme_fc_ctrl *ctrl)
 {
@@ -806,12 +815,20 @@ nvme_fc_ctrl_connectivity_loss(struct nvme_fc_ctrl *ctrl)
 
        case NVME_CTRL_CONNECTING:
                /*
-                * The association has already been terminated and the
-                * controller is attempting reconnects.  No need to do anything
-                * futher.  Reconnects will be attempted until either the
+                * We are racing with nvme_fc_create_association here. Ensure we
+                * never lose the connectivity loss event and unconditionally
+                * schedule a reset.
+                *
+                * The reconnects will be attempted until either the
                 * ctlr_loss_tmo (max_retries * connect_delay) expires or the
                 * remoteport's dev_loss_tmo expires.
+                *
+                * We can't sleep in this context (e.g flush_delayed_work) but
+                * we should execute reset after the connect work item has
+                * finished. Thus defer the reset trigger to the nvme_wq where
+                * the connect is executed.
                 */
+               queue_work(nvme_wq, &ctrl->fc_reset_work);
                break;
 
        case NVME_CTRL_RESETTING:
@@ -3505,6 +3522,7 @@ nvme_fc_init_ctrl(struct device *dev, struct
 nvmf_ctrl_options *opts,
        INIT_WORK(&ctrl->ctrl.reset_work, nvme_fc_reset_ctrl_work);
        INIT_DELAYED_WORK(&ctrl->connect_work, nvme_fc_connect_ctrl_work);
        INIT_WORK(&ctrl->ioerr_work, nvme_fc_ctrl_ioerr_work);
+       INIT_WORK(&ctrl->fc_reset_work, nvme_fc_defer_reset_work);
        spin_lock_init(&ctrl->lock);
 
        /* io queue count */
Quoted 2 lines

--
2.45.2

I tested this change with multiple NVME target vendors, multiple subsystems and namespaces
and fault injection with IO running. This test reproduced the controllers stuck in "Connecting"
state 100% of the time and now run clean for the test duration.

Signature · 11 lines
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.
#10 lore raw
In reply to Daniel Wagner
Quoted 9 lines · Daniel Wagner

We got a bug report that a controller was stuck in the connected state after an association dropped. It turns out that nvme_fc_create_association can succeed even though some operation do fail. This is on purpose to handle other corner cases, thus we can't really make nvme_fc_create_assocication bullet proof.

Given this, I think the second best option is to ensure that we always schedule a reset whenever a connectivity loss is observed in CONNECTING state. This is not perfect (possible back to back reset) but at least it is correct, which wins in my books.

Any feedback?