Skip to contentSkip to conversation

Handle update hardware queues and queue freeze more carefully

PATCH RESEND Superseded Daniel Wagner 6 participants 29 messages Aug 2, 2021 – Aug 18, 2021

Subsystems (2)

v4 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v4 cover Handle update hardware queues and queue freeze more carefully

[forgot to add the linux-nvme mailing list]

Hi,

This update version makes sure the unfreeze call is done when recreating the queues. I was able to reproduce hanging I/Os when we go into error recovery mode for FC and TCP [1]. Unfortunatly, I don't have access to a RDMA setup to verify but as the code is identically to the TCP, RDMA looks to like to suffer from the same problem.

Thanks,
Daniel

[1] https://lore.kernel.org/linux-nvme/[email protected]/

    https://lore.kernel.org/linux-nvme/[email protected]/

v1:
- https://lore.kernel.org/linux-nvme/[email protected]/
v2:
- https://lore.kernel.org/linux-nvme/[email protected]/
- reviewed tags collected
- added 'update hardware queues' for all transport
- added fix for fc hanger in nvme_wait_freeze_timeout
v3:
- https://lore.kernel.org/linux-nvme/[email protected]/
- dropped 'nvme-fc: Freeze queues before destroying them'
- added James' two patches
v4:
- added 'nvme-*: Unfreeze queues on reconnect'
- added Hannes' reviewed tags

Daniel Wagner (5):
nvme-fc: Update hardware queues before using them
nvme-rdma: Update number of hardware queues before using them
nvme-fc: Wait with a timeout for queue to freeze
nvme-tcp: Unfreeze queues on reconnect
nvme-rdma: Unfreeze queues on reconnect

Hannes Reinecke (1):
nvme-tcp: Update number of hardware queues before using them

James Smart (2):
nvme-fc: avoid race between time out and tear down
nvme-fc: fix controller reset hang during traffic

 drivers/nvme/host/fc.c   | 28 +++++++++++++++++++---------
 drivers/nvme/host/rdma.c | 15 ++++++++-------
 drivers/nvme/host/tcp.c  | 18 +++++++++---------
 3 files changed, 36 insertions(+), 25 deletions(-)
Signature · 1 line
2.29.2
#1 lore raw

PATCH v4 1/8 nvme-fc: Update hardware queues before using them

In case the number of hardware queues changes, do the update the tagset and ctx to hctx first before using the mapping to recreate and connnect the IO queues.

---
 drivers/nvme/host/fc.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)
drivers/nvme/host/fc.c +8 -8
index 7f462af1b02a..8a903769364f 100644
@@ -2951,14 +2951,6 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
 	if (ctrl->ctrl.queue_count == 1)
 		return 0;
 
-	ret = nvme_fc_create_hw_io_queues(ctrl, ctrl->ctrl.sqsize + 1);
-	if (ret)
-		goto out_free_io_queues;
-
-	ret = nvme_fc_connect_io_queues(ctrl, ctrl->ctrl.sqsize + 1);
-	if (ret)
-		goto out_delete_hw_queues;
-
 	if (prior_ioq_cnt != nr_io_queues) {
 		dev_info(ctrl->ctrl.device,
 			"reconnect: revising io queue count from %d to %d\n",
@@ -2968,6 +2960,14 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
 		nvme_unfreeze(&ctrl->ctrl);
 	}
 
+	ret = nvme_fc_create_hw_io_queues(ctrl, ctrl->ctrl.sqsize + 1);
+	if (ret)
+		goto out_free_io_queues;
+
+	ret = nvme_fc_connect_io_queues(ctrl, ctrl->ctrl.sqsize + 1);
+	if (ret)
+		goto out_delete_hw_queues;
+
 	return 0;
 
 out_delete_hw_queues:
Signature · 1 line
2.29.2
Reviewed-by on v4 1/8
#2 lore raw

Re: [PATCH v4 1/8] nvme-fc: Update hardware queues before using them

Quoted 37 lines · Daniel Wagner

In case the number of hardware queues changes, do the update the tagset and ctx to hctx first before using the mapping to recreate and connnect the IO queues.

---
  drivers/nvme/host/fc.c | 16 ++++++++--------
  1 file changed, 8 insertions(+), 8 deletions(-)
drivers/nvme/host/fc.c +1 -8
index 7f462af1b02a..8a903769364f 100644
@@ -2951,14 +2951,6 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
  	if (ctrl->ctrl.queue_count == 1)
  		return 0;
 
-	ret = nvme_fc_create_hw_io_queues(ctrl, ctrl->ctrl.sqsize + 1);
-	if (ret)
-		goto out_free_io_queues;
-
-	ret = nvme_fc_connect_io_queues(ctrl, ctrl->ctrl.sqsize + 1);
-	if (ret)
-		goto out_delete_hw_queues;
-
  	if (prior_ioq_cnt != nr_io_queues) {
  		dev_info(ctrl->ctrl.device,
  			"reconnect: revising io queue count from %d to %d\n",
@@ -2968,6 +2960,14 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
  		nvme_unfreeze(&ctrl->ctrl);
  	}
 
+	ret = nvme_fc_create_hw_io_queues(ctrl, ctrl->ctrl.sqsize + 1);

drivers/nvme/host/fc.c nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl) · line 2973

Quoted 10 lines
drivers/nvme/host/fc.c +7 -0
+	if (ret)
+		goto out_free_io_queues;
+
+	ret = nvme_fc_connect_io_queues(ctrl, ctrl->ctrl.sqsize + 1);
+	if (ret)
+		goto out_delete_hw_queues;
+
  	return 0;
 
  out_delete_hw_queues:
Signature · 6 lines
Himanshu Madhani                                Oracle Linux Engineering

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#3 lore raw

PATCH v4 2/8 nvme-tcp: Update number of hardware queues before using them

In reply to Daniel Wagner

From: Hannes Reinecke <[email protected]>

When the number of hardware queues changes during resetting we should update the tagset first before using it.

---
 drivers/nvme/host/tcp.c | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)
drivers/nvme/host/tcp.c +6 -8
index 0a97ba02f61e..32268f24f62a 100644
@@ -1789,6 +1789,7 @@ static void nvme_tcp_destroy_io_queues(struct nvme_ctrl *ctrl, bool remove)
 static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
 {
 	int ret;
+	u32 prior_q_cnt = ctrl->queue_count;
 
 	ret = nvme_tcp_alloc_io_queues(ctrl);
 	if (ret)
@@ -1806,14 +1807,7 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
 			ret = PTR_ERR(ctrl->connect_q);
 			goto out_free_tag_set;
 		}
-	}
-
-	ret = nvme_tcp_start_io_queues(ctrl);
-	if (ret)
-		goto out_cleanup_connect_q;
-
-	if (!new) {
-		nvme_start_queues(ctrl);
+	} else if (prior_q_cnt != ctrl->queue_count) {
 		if (!nvme_wait_freeze_timeout(ctrl, NVME_IO_TIMEOUT)) {
 			/*
 			 * If we timed out waiting for freeze we are likely to
@@ -1828,6 +1822,10 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
 		nvme_unfreeze(ctrl);
 	}
 
+	ret = nvme_tcp_start_io_queues(ctrl);
+	if (ret)
+		goto out_cleanup_connect_q;
+
 	return 0;
 
 out_wait_freeze_timed_out:
Signature · 1 line
2.29.2
maintainer
#4 lore raw

Re: [PATCH v4 2/8] nvme-tcp: Update number of hardware queues before using them

Quoted 27 lines

From: Hannes Reinecke <[email protected]>

When the number of hardware queues changes during resetting we should update the tagset first before using it.

---
  drivers/nvme/host/tcp.c | 14 ++++++--------
  1 file changed, 6 insertions(+), 8 deletions(-)
drivers/nvme/host/tcp.c +1 -0
index 0a97ba02f61e..32268f24f62a 100644
@@ -1789,6 +1789,7 @@ static void nvme_tcp_destroy_io_queues(struct nvme_ctrl *ctrl, bool remove)
  static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
  {
  	int ret;
+	u32 prior_q_cnt = ctrl->queue_count;
 
  	ret = nvme_tcp_alloc_io_queues(ctrl);
  	if (ret)
@@ -1806,14 +1807,7 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
  			ret = PTR_ERR(ctrl->connect_q);
  			goto out_free_tag_set;

drivers/nvme/host/tcp.c static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) · line 1810

Quoted 10 lines
drivers/nvme/host/tcp.c +1 -8
  		}
-	}
-
-	ret = nvme_tcp_start_io_queues(ctrl);
-	if (ret)
-		goto out_cleanup_connect_q;
-
-	if (!new) {
-		nvme_start_queues(ctrl);
+	} else if (prior_q_cnt != ctrl->queue_count) {

So if the queue count did not change we don't wait to make sure the queue g_usage_counter ref made it to zero? What guarantees that it did?

drivers/nvme/host/tcp.c static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) · line 1828

Quoted 11 lines
drivers/nvme/host/tcp.c +4 -0
  		if (!nvme_wait_freeze_timeout(ctrl, NVME_IO_TIMEOUT)) {
  			/*
  			 * If we timed out waiting for freeze we are likely to
@@ -1828,6 +1822,10 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
  		nvme_unfreeze(ctrl);
  	}
 
+	ret = nvme_tcp_start_io_queues(ctrl);
+	if (ret)
+		goto out_cleanup_connect_q;
+

Did you test this with both heavy I/O, reset loop and ifdown/ifup loop?

If we unquiesce and unfreeze before we start the queues the pending I/Os may resume before the connect and not allow the connect to make forward progress.

Quoted 4 lines
drivers/nvme/host/tcp.c +0 -0
  	return 0;
 
  out_wait_freeze_timed_out:
#5 lore raw

Re: [PATCH v4 2/8] nvme-tcp: Update number of hardware queues before using them

Hi Sagi,

Quoted 11 lines · Sagi Grimberg

drivers/nvme/host/tcp.c static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) · line 1810

Quoted 7 lines
drivers/nvme/host/tcp.c +1 -6
-	ret = nvme_tcp_start_io_queues(ctrl);
-	if (ret)
-		goto out_cleanup_connect_q;
-
-	if (!new) {
-		nvme_start_queues(ctrl);
+	} else if (prior_q_cnt != ctrl->queue_count) {

So if the queue count did not change we don't wait to make sure the queue g_usage_counter ref made it to zero? What guarantees that it did?

Hmm, good point. we should always call nvme_wait_freeze_timeout() for !new queues. Is this what you are implying?

Quoted 5 lines
Quoted 5 lines

if (!nvme_wait_freeze_timeout(ctrl, NVME_IO_TIMEOUT)) {
/*
* If we timed out waiting for freeze we are likely to

drivers/nvme/host/tcp.c +0 -0
@@ -1828,6 +1822,10 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
  		nvme_unfreeze(ctrl);
Quoted 8 lines

drivers/nvme/host/tcp.c static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) · line 1828

Quoted 5 lines
drivers/nvme/host/tcp.c +4 -0
  	}
+	ret = nvme_tcp_start_io_queues(ctrl);
+	if (ret)
+		goto out_cleanup_connect_q;
+

Did you test this with both heavy I/O, reset loop and ifdown/ifup loop?

Not sure if this classifies as heavy I/O (on 80 CPU machine)

fio --rw=readwrite --name=test --filename=/dev/nvme16n1 --size=50M \
    --direct=1 --bs=4k --numjobs=40 --group_reporting --runtime=4h \
    --time_based

and then I installed iptables rules to block the traffic on the controller side. With this test it is pretty easily to get the host hanging. Let me know what test you would like to see from me. I am glad to try to get them running.

Quoted 3 lines

If we unquiesce and unfreeze before we start the queues the pending I/Os may resume before the connect and not allow the connect to make forward progress.

So the unfreeze should happen after the connect call? What about the newly created queues? Do they not suffer from the same problem? Isn't the NVME_TCP_Q_LIVE flag not enough?

Daniel

maintainer
#6 lore raw

Re: [PATCH v4 2/8] nvme-tcp: Update number of hardware queues before using them

Quoted 15 lines · Daniel Wagner

Hi Sagi,

Quoted 11 lines · Sagi Grimberg

drivers/nvme/host/tcp.c static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) · line 1810

Quoted 7 lines
drivers/nvme/host/tcp.c +1 -6
-	ret = nvme_tcp_start_io_queues(ctrl);
-	if (ret)
-		goto out_cleanup_connect_q;
-
-	if (!new) {
-		nvme_start_queues(ctrl);
+	} else if (prior_q_cnt != ctrl->queue_count) {

So if the queue count did not change we don't wait to make sure the queue g_usage_counter ref made it to zero? What guarantees that it did?

Quoted 2 lines

Hmm, good point. we should always call nvme_wait_freeze_timeout() for !new queues. Is this what you are implying?

I think we should always wait for the freeze to complete.

Quoted 18 lines
Quoted 15 lines

drivers/nvme/host/tcp.c static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) · line 1827

Quoted 12 lines

   		if (!nvme_wait_freeze_timeout(ctrl, NVME_IO_TIMEOUT)) {
   			/*
   			 * If we timed out waiting for freeze we are likely to
drivers/nvme/host/tcp.c +4 -0
@@ -1828,6 +1822,10 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
   		nvme_unfreeze(ctrl);
   	}
+	ret = nvme_tcp_start_io_queues(ctrl);
+	if (ret)
+		goto out_cleanup_connect_q;
+

Did you test this with both heavy I/O, reset loop and ifdown/ifup loop?

Not sure if this classifies as heavy I/O (on 80 CPU machine)

Quoted 8 lines
fio --rw=readwrite --name=test --filename=/dev/nvme16n1 --size=50M \
     --direct=1 --bs=4k --numjobs=40 --group_reporting --runtime=4h \
     --time_based

and then I installed iptables rules to block the traffic on the controller side. With this test it is pretty easily to get the host hanging. Let me know what test you would like to see from me. I am glad to try to get them running.

Lets add iodepth=128

Quoted 7 lines
Quoted 3 lines

If we unquiesce and unfreeze before we start the queues the pending I/Os may resume before the connect and not allow the connect to make forward progress.

So the unfreeze should happen after the connect call? What about the newly created queues? Do they not suffer from the same problem? Isn't the NVME_TCP_Q_LIVE flag not enough?

Q_LIVE will protect against the transport itself from queueing, however when multipath is not used, the transport will return BLK_STS_RESOURCE which will immediately trigger re-submission, in an endless loop, and that can prevent forward progress. It is also consistent with what nvme-pci does.

maintainer
#7 lore raw

Re: [PATCH v4 2/8] nvme-tcp: Update number of hardware queues before using them

Quoted 18 lines · Sagi Grimberg
Quoted 15 lines · Daniel Wagner

Hi Sagi,

Quoted 11 lines · Sagi Grimberg

drivers/nvme/host/tcp.c static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) · line 1810

Quoted 7 lines
drivers/nvme/host/tcp.c +1 -6
-	ret = nvme_tcp_start_io_queues(ctrl);
-	if (ret)
-		goto out_cleanup_connect_q;
-
-	if (!new) {
-		nvme_start_queues(ctrl);
+	} else if (prior_q_cnt != ctrl->queue_count) {

So if the queue count did not change we don't wait to make sure the queue g_usage_counter ref made it to zero? What guarantees that it did?

Quoted 4 lines
Quoted 2 lines

Hmm, good point. we should always call nvme_wait_freeze_timeout() for !new queues. Is this what you are implying?

I think we should always wait for the freeze to complete.

Don't the queues need to be started in order for the freeze to complete? Any enqueued requests on the quiesced queues will never complete this way, so the wait_freeze() will be stuck, right? If so, I think the nvme_start_queues() was in the correct place already.

maintainer
#8 lore raw

Re: [PATCH v4 2/8] nvme-tcp: Update number of hardware queues before using them

Quoted 20 lines
Quoted 20 lines · Daniel Wagner
Quoted 17 lines

Hi Sagi,

Quoted 11 lines · Sagi Grimberg

drivers/nvme/host/tcp.c static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) · line 1810

Quoted 7 lines
drivers/nvme/host/tcp.c +1 -6
-	ret = nvme_tcp_start_io_queues(ctrl);
-	if (ret)
-		goto out_cleanup_connect_q;
-
-	if (!new) {
-		nvme_start_queues(ctrl);
+	} else if (prior_q_cnt != ctrl->queue_count) {

So if the queue count did not change we don't wait to make sure the queue g_usage_counter ref made it to zero? What guarantees that it did?

Hmm, good point. we should always call nvme_wait_freeze_timeout() for !new queues. Is this what you are implying?

Quoted 6 lines
Quoted 1 line

I think we should always wait for the freeze to complete.

Don't the queues need to be started in order for the freeze to complete? Any enqueued requests on the quiesced queues will never complete this way, so the wait_freeze() will be stuck, right? If so, I think the nvme_start_queues() was in the correct place already.

Exactly what I was trying to point out (poorly though)

#9 lore raw

Re: [PATCH v4 2/8] nvme-tcp: Update number of hardware queues before using them

Quoted 8 lines · Sagi Grimberg
Quoted 6 lines
Quoted 1 line

I think we should always wait for the freeze to complete.

Don't the queues need to be started in order for the freeze to complete? Any enqueued requests on the quiesced queues will never complete this way, so the wait_freeze() will be stuck, right? If so, I think the nvme_start_queues() was in the correct place already.

Exactly what I was trying to point out (poorly though)

Thanks for explaining. I think I got the general idea what the different states are doing and what the transitions are now. (famous last words).

Anyway, the first three patches are the result of debugging the case of 'prior_ioq_cnt != nr_io_queues'. Starting the queues before updating the number of queues lookes strange.

I suppose in the case 'prior_ioq_cnt > nr_io_queues', nvme_tcp_start_io_queues() should be successful and we do the blk_mq_update_nr_hw_queues(). In the other case we should land in the error recovery.

Wouldn't it make sense to always exercise the error recovery path if we detect 'prior_ioq_cnt != nr_io_queues' and reduce the number of things which can go wrong?

Daniel

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#10 lore raw

PATCH v4 3/8 nvme-rdma: Update number of hardware queues before using them

In reply to Daniel Wagner

When the number of hardware queues changes during resetting we should update the tagset first before using it.

---
 drivers/nvme/host/rdma.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
drivers/nvme/host/rdma.c +6 -7
index 69ae67652f38..de2a8950d282 100644
@@ -965,6 +965,7 @@ static void nvme_rdma_destroy_io_queues(struct nvme_rdma_ctrl *ctrl,
 static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)
 {
 	int ret;
+	u32 prior_q_cnt = ctrl->ctrl.queue_count;
 
 	ret = nvme_rdma_alloc_io_queues(ctrl);
 	if (ret)
@@ -982,13 +983,7 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)
 			ret = PTR_ERR(ctrl->ctrl.connect_q);
 			goto out_free_tag_set;
 		}
-	}
-
-	ret = nvme_rdma_start_io_queues(ctrl);
-	if (ret)
-		goto out_cleanup_connect_q;
-
-	if (!new) {
+	} else if (prior_q_cnt != ctrl->ctrl.queue_count) {
 		nvme_start_queues(&ctrl->ctrl);
 		if (!nvme_wait_freeze_timeout(&ctrl->ctrl, NVME_IO_TIMEOUT)) {
 			/*
@@ -1004,6 +999,10 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)
 		nvme_unfreeze(&ctrl->ctrl);
 	}
 
+	ret = nvme_rdma_start_io_queues(ctrl);
+	if (ret)
+		goto out_cleanup_connect_q;
+
 	return 0;
 
 out_wait_freeze_timed_out:
Signature · 1 line
2.29.2
#11 lore raw

PATCH v4 4/8 nvme-fc: Wait with a timeout for queue to freeze

In reply to Daniel Wagner

Do not wait indifinitly for all queues to freeze. Instead use a timeout and abort the operation if we get stuck.

---
 drivers/nvme/host/fc.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/nvme/host/fc.c +8 -1
index 8a903769364f..dbb8ad816df8 100644
@@ -2955,7 +2955,14 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
 		dev_info(ctrl->ctrl.device,
 			"reconnect: revising io queue count from %d to %d\n",
 			prior_ioq_cnt, nr_io_queues);
-		nvme_wait_freeze(&ctrl->ctrl);
+		if (!nvme_wait_freeze_timeout(&ctrl->ctrl, NVME_IO_TIMEOUT)) {
+			/*
+			 * If we timed out waiting for freeze we are likely to
+			 * be stuck.  Fail the controller initialization just
+			 * to be safe.
+			 */
+			return -ENODEV;
+		}
 		blk_mq_update_nr_hw_queues(&ctrl->tag_set, nr_io_queues);
 		nvme_unfreeze(&ctrl->ctrl);
 	}
Signature · 1 line
2.29.2
Reviewed-by on v4 4/8
#12 lore raw

Re: [PATCH v4 4/8] nvme-fc: Wait with a timeout for queue to freeze

Quoted 21 lines · Daniel Wagner

Do not wait indifinitly for all queues to freeze. Instead use a timeout and abort the operation if we get stuck.

---
  drivers/nvme/host/fc.c | 9 ++++++++-
  1 file changed, 8 insertions(+), 1 deletion(-)
drivers/nvme/host/fc.c +1 -1
index 8a903769364f..dbb8ad816df8 100644
@@ -2955,7 +2955,14 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
  		dev_info(ctrl->ctrl.device,
  			"reconnect: revising io queue count from %d to %d\n",
  			prior_ioq_cnt, nr_io_queues);
-		nvme_wait_freeze(&ctrl->ctrl);
+		if (!nvme_wait_freeze_timeout(&ctrl->ctrl, NVME_IO_TIMEOUT)) {

drivers/nvme/host/fc.c nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl) · line 2968

Quoted 10 lines
drivers/nvme/host/fc.c +7 -0
+			/*
+			 * If we timed out waiting for freeze we are likely to
+			 * be stuck.  Fail the controller initialization just
+			 * to be safe.
+			 */
+			return -ENODEV;
+		}
  		blk_mq_update_nr_hw_queues(&ctrl->tag_set, nr_io_queues);
  		nvme_unfreeze(&ctrl->ctrl);
  	}
Signature · 6 lines
Himanshu Madhani                                Oracle Linux Engineering

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#13 lore raw

PATCH v4 5/8 nvme-fc: avoid race between time out and tear down

In reply to Daniel Wagner

From: James Smart <[email protected]>

To avoid race between time out and tear down, in tear down process, first we quiesce the queue, and then delete the timer and cancel the time out work for the queue.

This patch merges the admin and io sync ops into the queue teardown logic as shown in the RDMA patch 3017013dcc "nvme-rdma: avoid race between time out and tear down". There is no teardown_lock in nvme-fc.

CC Chao Leng <[email protected]>
[dwagner: updated commit id referenced in commit message]
---
 drivers/nvme/host/fc.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/nvme/host/fc.c +2 -0
index dbb8ad816df8..133b87db4f1d 100644
@@ -2487,6 +2487,7 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues)
 	 */
 	if (ctrl->ctrl.queue_count > 1) {
 		nvme_stop_queues(&ctrl->ctrl);
+		nvme_sync_io_queues(&ctrl->ctrl);
 		blk_mq_tagset_busy_iter(&ctrl->tag_set,
 				nvme_fc_terminate_exchange, &ctrl->ctrl);
 		blk_mq_tagset_wait_completed_request(&ctrl->tag_set);
@@ -2510,6 +2511,7 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues)
 	 * clean up the admin queue. Same thing as above.
 	 */
 	blk_mq_quiesce_queue(ctrl->ctrl.admin_q);
+	blk_sync_queue(ctrl->ctrl.admin_q);
 	blk_mq_tagset_busy_iter(&ctrl->admin_tag_set,
 				nvme_fc_terminate_exchange, &ctrl->ctrl);
 	blk_mq_tagset_wait_completed_request(&ctrl->admin_tag_set);
Signature · 1 line
2.29.2
Reviewed-by on v4 5/8
#14 lore raw

Re: [PATCH v4 5/8] nvme-fc: avoid race between time out and tear down

Quoted 34 lines · Daniel Wagner

From: James Smart <[email protected]>

To avoid race between time out and tear down, in tear down process, first we quiesce the queue, and then delete the timer and cancel the time out work for the queue.

This patch merges the admin and io sync ops into the queue teardown logic as shown in the RDMA patch 3017013dcc "nvme-rdma: avoid race between time out and tear down". There is no teardown_lock in nvme-fc.

CC Chao Leng <[email protected]>
[dwagner: updated commit id referenced in commit message]
---
  drivers/nvme/host/fc.c | 2 ++
  1 file changed, 2 insertions(+)
drivers/nvme/host/fc.c +1 -0
index dbb8ad816df8..133b87db4f1d 100644
@@ -2487,6 +2487,7 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues)
  	 */
  	if (ctrl->ctrl.queue_count > 1) {
  		nvme_stop_queues(&ctrl->ctrl);
+		nvme_sync_io_queues(&ctrl->ctrl);
  		blk_mq_tagset_busy_iter(&ctrl->tag_set,
  				nvme_fc_terminate_exchange, &ctrl->ctrl);
  		blk_mq_tagset_wait_completed_request(&ctrl->tag_set);

drivers/nvme/host/fc.c __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues) · line 2517

Quoted 7 lines
drivers/nvme/host/fc.c +1 -0
@@ -2510,6 +2511,7 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues)
  	 * clean up the admin queue. Same thing as above.
  	 */
  	blk_mq_quiesce_queue(ctrl->ctrl.admin_q);
+	blk_sync_queue(ctrl->ctrl.admin_q);
  	blk_mq_tagset_busy_iter(&ctrl->admin_tag_set,
  				nvme_fc_terminate_exchange, &ctrl->ctrl);
  	blk_mq_tagset_wait_completed_request(&ctrl->admin_tag_set);
Signature · 1 line
Himanshu Madhani                                Oracle Linux Engineering
#15 lore raw

PATCH v4 6/8 nvme-fc: fix controller reset hang during traffic

In reply to Daniel Wagner

From: James Smart <[email protected]>

commit fe35ec58f0d3 ("block: update hctx map when use multiple maps") exposed an issue where we may hang trying to wait for queue freeze during I/O. We call blk_mq_update_nr_hw_queues which may attempt to freeze the queue. However we never started queue freeze when starting the reset, which means that we have inflight pending requests that entered the queue that we will not complete once the queue is quiesced.

So start a freeze before we quiesce the queue, and unfreeze the queue after we successfully connected the I/O queues (the unfreeze is already present in the code). blk_mq_update_nr_hw_queues will be called only after we are sure that the queue was already frozen.

This follows to how the pci driver handles resets.

This patch added logic introduced in commit 9f98772ba307 "nvme-rdma: fix controller reset hang during traffic".

[dwagner: call nvme_unfreeze() unconditionally in
nvme_fc_recreate_io_queues() to match the nvme_start_freeze()]

---
 drivers/nvme/host/fc.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/host/fc.c +2 -1
index 133b87db4f1d..b292af0fd655 100644
@@ -2486,6 +2486,7 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues)
 	 * (but with error status).
 	 */
 	if (ctrl->ctrl.queue_count > 1) {
+		nvme_start_freeze(&ctrl->ctrl);
 		nvme_stop_queues(&ctrl->ctrl);
 		nvme_sync_io_queues(&ctrl->ctrl);
 		blk_mq_tagset_busy_iter(&ctrl->tag_set,
@@ -2966,8 +2967,8 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
 			return -ENODEV;
 		}
 		blk_mq_update_nr_hw_queues(&ctrl->tag_set, nr_io_queues);
-		nvme_unfreeze(&ctrl->ctrl);
 	}
+	nvme_unfreeze(&ctrl->ctrl);
 
 	ret = nvme_fc_create_hw_io_queues(ctrl, ctrl->ctrl.sqsize + 1);
 	if (ret)
Signature · 1 line
2.29.2
Reviewed-by on v4 6/8
#16 lore raw

Re: [PATCH v4 6/8] nvme-fc: fix controller reset hang during traffic

Quoted 43 lines · Daniel Wagner

From: James Smart <[email protected]>

commit fe35ec58f0d3 ("block: update hctx map when use multiple maps") exposed an issue where we may hang trying to wait for queue freeze during I/O. We call blk_mq_update_nr_hw_queues which may attempt to freeze the queue. However we never started queue freeze when starting the reset, which means that we have inflight pending requests that entered the queue that we will not complete once the queue is quiesced.

So start a freeze before we quiesce the queue, and unfreeze the queue after we successfully connected the I/O queues (the unfreeze is already present in the code). blk_mq_update_nr_hw_queues will be called only after we are sure that the queue was already frozen.

This follows to how the pci driver handles resets.

This patch added logic introduced in commit 9f98772ba307 "nvme-rdma: fix controller reset hang during traffic".

[dwagner: call nvme_unfreeze() unconditionally in
nvme_fc_recreate_io_queues() to match the nvme_start_freeze()]

---
  drivers/nvme/host/fc.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/host/fc.c +1 -0
index 133b87db4f1d..b292af0fd655 100644
@@ -2486,6 +2486,7 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues)
  	 * (but with error status).
  	 */
  	if (ctrl->ctrl.queue_count > 1) {
+		nvme_start_freeze(&ctrl->ctrl);
  		nvme_stop_queues(&ctrl->ctrl);
  		nvme_sync_io_queues(&ctrl->ctrl);
  		blk_mq_tagset_busy_iter(&ctrl->tag_set,

drivers/nvme/host/fc.c nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl) · line 2974

Quoted 9 lines
drivers/nvme/host/fc.c +1 -1
@@ -2966,8 +2967,8 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
  			return -ENODEV;
  		}
  		blk_mq_update_nr_hw_queues(&ctrl->tag_set, nr_io_queues);
-		nvme_unfreeze(&ctrl->ctrl);
  	}
+	nvme_unfreeze(&ctrl->ctrl);
 
  	ret = nvme_fc_create_hw_io_queues(ctrl, ctrl->ctrl.sqsize + 1);
  	if (ret)
Signature · 6 lines
Himanshu Madhani                                Oracle Linux Engineering

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#17 lore raw

Re: [PATCH v4 6/8] nvme-fc: fix controller reset hang during traffic

In reply to Daniel Wagner
Quoted 43 lines · Daniel Wagner

From: James Smart <[email protected]>

commit fe35ec58f0d3 ("block: update hctx map when use multiple maps") exposed an issue where we may hang trying to wait for queue freeze during I/O. We call blk_mq_update_nr_hw_queues which may attempt to freeze the queue. However we never started queue freeze when starting the reset, which means that we have inflight pending requests that entered the queue that we will not complete once the queue is quiesced.

So start a freeze before we quiesce the queue, and unfreeze the queue after we successfully connected the I/O queues (the unfreeze is already present in the code). blk_mq_update_nr_hw_queues will be called only after we are sure that the queue was already frozen.

This follows to how the pci driver handles resets.

This patch added logic introduced in commit 9f98772ba307 "nvme-rdma: fix controller reset hang during traffic".

[dwagner: call nvme_unfreeze() unconditionally in
nvme_fc_recreate_io_queues() to match the nvme_start_freeze()]

---
 drivers/nvme/host/fc.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/host/fc.c +1 -0
index 133b87db4f1d..b292af0fd655 100644
@@ -2486,6 +2486,7 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues)
 	 * (but with error status).
 	 */
 	if (ctrl->ctrl.queue_count > 1) {
+		nvme_start_freeze(&ctrl->ctrl);
 		nvme_stop_queues(&ctrl->ctrl);
 		nvme_sync_io_queues(&ctrl->ctrl);
 		blk_mq_tagset_busy_iter(&ctrl->tag_set,

drivers/nvme/host/fc.c nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl) · line 2974

Quoted 9 lines
drivers/nvme/host/fc.c +1 -1
@@ -2966,8 +2967,8 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
 			return -ENODEV;
 		}
 		blk_mq_update_nr_hw_queues(&ctrl->tag_set, nr_io_queues);
-		nvme_unfreeze(&ctrl->ctrl);
 	}
+	nvme_unfreeze(&ctrl->ctrl);
 
 	ret = nvme_fc_create_hw_io_queues(ctrl, ctrl->ctrl.sqsize + 1);
 	if (ret)

There still is now an imbalance, as we're always calling 'nvme_unfreeze()' (irrespective on the number of queues), but will only call 'nvme_start_freeze()' if we have more than one queue.

This might lead to an imbalance on the mq_freeze_depth counter. Wouldn't it be better to move the call to 'nvme_start_freeze()' out of the if() condition to avoid the imbalance?

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke		           Kernel Storage Architect
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer
#18 lore raw

Re: [PATCH v4 6/8] nvme-fc: fix controller reset hang during traffic

Quoted 7 lines · Hannes Reinecke

There still is now an imbalance, as we're always calling 'nvme_unfreeze()' (irrespective on the number of queues), but will only call 'nvme_start_freeze()' if we have more than one queue.

This might lead to an imbalance on the mq_freeze_depth counter. Wouldn't it be better to move the call to 'nvme_start_freeze()' out of the if() condition to avoid the imbalance?

What about something like nmve_is_frozen() which would avoid the tracking of the queue state open coded as it is right?

maintainer
#19 lore raw

Re: [PATCH v4 6/8] nvme-fc: fix controller reset hang during traffic

Quoted 6 lines · Hannes Reinecke
Quoted 4 lines

There still is now an imbalance, as we're always calling 'nvme_unfreeze()' (irrespective on the number of queues), but will only call 'nvme_start_freeze()' if we have more than one queue.

Quoted 6 lines
Quoted 3 lines

This might lead to an imbalance on the mq_freeze_depth counter. Wouldn't it be better to move the call to 'nvme_start_freeze()' out of the if() condition to avoid the imbalance?

What about something like nmve_is_frozen() which would avoid the tracking of the queue state open coded as it is right?

Why is there a conditional freeze?

#20 lore raw

Re: [PATCH v4 6/8] nvme-fc: fix controller reset hang during traffic

Quoted 1 line · Sagi Grimberg

Why is there a conditional freeze?

The freeze only happens if I/O queues have been created/started. If I understand this correctly, this is the corner case we were only able to establish the admin queue and fail later to setup the I/O queues.

#21 lore raw

Re: [PATCH v4 6/8] nvme-fc: fix controller reset hang during traffic

In reply to Hannes Reinecke
Quoted 57 lines · Hannes Reinecke
Quoted 43 lines · Daniel Wagner

From: James Smart <[email protected]>

commit fe35ec58f0d3 ("block: update hctx map when use multiple maps") exposed an issue where we may hang trying to wait for queue freeze during I/O. We call blk_mq_update_nr_hw_queues which may attempt to freeze the queue. However we never started queue freeze when starting the reset, which means that we have inflight pending requests that entered the queue that we will not complete once the queue is quiesced.

So start a freeze before we quiesce the queue, and unfreeze the queue after we successfully connected the I/O queues (the unfreeze is already present in the code). blk_mq_update_nr_hw_queues will be called only after we are sure that the queue was already frozen.

This follows to how the pci driver handles resets.

This patch added logic introduced in commit 9f98772ba307 "nvme-rdma: fix controller reset hang during traffic".

[dwagner: call nvme_unfreeze() unconditionally in
nvme_fc_recreate_io_queues() to match the nvme_start_freeze()]

---
  drivers/nvme/host/fc.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/host/fc.c +1 -0
index 133b87db4f1d..b292af0fd655 100644
@@ -2486,6 +2486,7 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues)
  	 * (but with error status).
  	 */
  	if (ctrl->ctrl.queue_count > 1) {
+		nvme_start_freeze(&ctrl->ctrl);
  		nvme_stop_queues(&ctrl->ctrl);
  		nvme_sync_io_queues(&ctrl->ctrl);
  		blk_mq_tagset_busy_iter(&ctrl->tag_set,

drivers/nvme/host/fc.c nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl) · line 2974

Quoted 9 lines
drivers/nvme/host/fc.c +1 -1
@@ -2966,8 +2967,8 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
  			return -ENODEV;
  		}
  		blk_mq_update_nr_hw_queues(&ctrl->tag_set, nr_io_queues);
-		nvme_unfreeze(&ctrl->ctrl);
  	}
+	nvme_unfreeze(&ctrl->ctrl);
 
  	ret = nvme_fc_create_hw_io_queues(ctrl, ctrl->ctrl.sqsize + 1);
  	if (ret)

There still is now an imbalance, as we're always calling 'nvme_unfreeze()' (irrespective on the number of queues), but will only call 'nvme_start_freeze()' if we have more than one queue.

Quoted 5 lines

This might lead to an imbalance on the mq_freeze_depth counter. Wouldn't it be better to move the call to 'nvme_start_freeze()' out of the if() condition to avoid the imbalance?

Cheers,

Daniel,

try this. Moves the location of the freeze and will always pair with the unfreeze.

fc.c.new renamed from fc.c +5 -1
@@ -2965,9 +2965,10 @@ nvme_fc_recreate_io_queues(struct nvme_f
 			prior_ioq_cnt, nr_io_queues);
 		nvme_wait_freeze(&ctrl->ctrl);
 		blk_mq_update_nr_hw_queues(&ctrl->tag_set, nr_io_queues);
-		nvme_unfreeze(&ctrl->ctrl);
 	}
 
+	nvme_unfreeze(&ctrl->ctrl);
+
 	return 0;
 
 out_delete_hw_queues:
@@ -3206,6 +3207,9 @@ nvme_fc_delete_association(struct nvme_f
 	ctrl->iocnt = 0;
 	spin_unlock_irqrestore(&ctrl->lock, flags);
 
+	if (ctrl->ctrl.queue_count > 1)
+		nvme_start_freeze(&ctrl->ctrl);
+
 	__nvme_fc_abort_outstanding_ios(ctrl, false);
 
 	/* kill the aens as they are a separate path */
#22 lore raw

Re: [PATCH v4 6/8] nvme-fc: fix controller reset hang during traffic

Quoted 2 lines · James Smart

try this. Moves the location of the freeze and will always pair with the unfreeze.

Yes, this works. Do you send a proper patch out or should I just pick up this patch and add a commit message?

#23 lore raw

Re: [PATCH v4 6/8] nvme-fc: fix controller reset hang during traffic

Quoted 6 lines · Daniel Wagner
Quoted 2 lines · James Smart

try this. Moves the location of the freeze and will always pair with the unfreeze.

Yes, this works. Do you send a proper patch out or should I just pick up this patch and add a commit message?

Forget it, the commit message is still the same. Let me respin this series.

#24 lore raw

PATCH v4 7/8 nvme-tcp: Unfreeze queues on reconnect

In reply to Daniel Wagner

During the queue teardown in nvme_tcp_teardown_io_queues() freeze is called unconditionally. When we reconnect we need to pair the freeze with an unfreeze to avoid hanging I/Os. For newly created connection this is not needed.

Fixes 2875b0aecabe ("nvme-tcp: fix controller reset hang during traffic")
---
 drivers/nvme/host/tcp.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/nvme/host/tcp.c +3 -1
index 32268f24f62a..097f7dd10ed3 100644
@@ -1819,9 +1819,11 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
 		}
 		blk_mq_update_nr_hw_queues(ctrl->tagset,
 			ctrl->queue_count - 1);
-		nvme_unfreeze(ctrl);
 	}
 
+	if (!new)
+		nvme_unfreeze(ctrl);
+
 	ret = nvme_tcp_start_io_queues(ctrl);
 	if (ret)
 		goto out_cleanup_connect_q;
Signature · 1 line
2.29.2
#25 lore raw

PATCH v4 8/8 nvme-rdma: Unfreeze queues on reconnect

In reply to Daniel Wagner

During the queue teardown in nvme_rdma_teardown_io_queues() freeze is called unconditionally. When we reconnect we need to pair the freeze with an unfreeze to avoid hanging I/Os. For newly created connection this is not needed.

Fixes 9f98772ba307 ("nvme-rdma: fix controller reset hang during traffic")
---
 drivers/nvme/host/rdma.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/nvme/host/rdma.c +2 -0
index de2a8950d282..21a8a5353af0 100644
@@ -901,6 +901,8 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl,
 			error = PTR_ERR(ctrl->ctrl.admin_q);
 			goto out_cleanup_fabrics_q;
 		}
+	} else {
+		nvme_unfreeze(&ctrl->ctrl);
 	}
 
 	error = nvme_rdma_start_queue(ctrl, 0);
Signature · 1 line
2.29.2
Reviewed-by on v4 8/8
#26 lore raw

Re: [PATCH v4 8/8] nvme-rdma: Unfreeze queues on reconnect

Quoted 17 lines · Daniel Wagner

During the queue teardown in nvme_rdma_teardown_io_queues() freeze is called unconditionally. When we reconnect we need to pair the freeze with an unfreeze to avoid hanging I/Os. For newly created connection this is not needed.

Fixes 9f98772ba307 ("nvme-rdma: fix controller reset hang during traffic")
---
 drivers/nvme/host/rdma.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/nvme/host/rdma.c +0 -0
index de2a8950d282..21a8a5353af0 100644

drivers/nvme/host/rdma.c static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, · line 908

Quoted 8 lines
drivers/nvme/host/rdma.c +2 -0
@@ -901,6 +901,8 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl,
 			error = PTR_ERR(ctrl->ctrl.admin_q);
 			goto out_cleanup_fabrics_q;
 		}
+	} else {
+		nvme_unfreeze(&ctrl->ctrl);
 	}
 
 	error = nvme_rdma_start_queue(ctrl, 0);

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke		           Kernel Storage Architect
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer
maintainer
#27 lore raw

Re: [PATCH v4 8/8] nvme-rdma: Unfreeze queues on reconnect

In reply to Daniel Wagner
Quoted 17 lines

During the queue teardown in nvme_rdma_teardown_io_queues() freeze is called unconditionally. When we reconnect we need to pair the freeze with an unfreeze to avoid hanging I/Os. For newly created connection this is not needed.

Fixes 9f98772ba307 ("nvme-rdma: fix controller reset hang during traffic")
---
  drivers/nvme/host/rdma.c | 2 ++
  1 file changed, 2 insertions(+)
drivers/nvme/host/rdma.c +0 -0
index de2a8950d282..21a8a5353af0 100644

drivers/nvme/host/rdma.c static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, · line 905

Quoted 5 lines
drivers/nvme/host/rdma.c +2 -0
@@ -901,6 +901,8 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl,
  			error = PTR_ERR(ctrl->ctrl.admin_q);
  			goto out_cleanup_fabrics_q;
  		}
+	} else {
+		nvme_unfreeze(&ctrl->ctrl);

That seems misplaced.. unfreezing the I/O queues when setting up the admin queue?

#28 lore raw

Re: [PATCH v4 8/8] nvme-rdma: Unfreeze queues on reconnect

Hi Sagi,

Quoted 12 lines · Sagi Grimberg
Quoted 12 lines

During the queue teardown in nvme_rdma_teardown_io_queues() freeze is called unconditionally. When we reconnect we need to pair the freeze with an unfreeze to avoid hanging I/Os. For newly created connection this is not needed.

Fixes 9f98772ba307 ("nvme-rdma: fix controller reset hang during traffic")
---
  drivers/nvme/host/rdma.c | 2 ++
  1 file changed, 2 insertions(+)
Quoted 13 lines

drivers/nvme/host/rdma.c static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, · line 905

Quoted 10 lines
drivers/nvme/host/rdma.c +2 -0
index de2a8950d282..21a8a5353af0 100644
@@ -901,6 +901,8 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl,
  			error = PTR_ERR(ctrl->ctrl.admin_q);
  			goto out_cleanup_fabrics_q;
  		}
+	} else {
+		nvme_unfreeze(&ctrl->ctrl);

That seems misplaced.. unfreezing the I/O queues when setting up the admin queue?

Indeed. After looking again on it, this should be almost identically to the tcp.c fix in nvme_rdma_configure_io_queues.

BTW, I am working on getting a RDMA test setup running. Hopefully I have all the right licenses on the array.

Daniel