Skip to contentSkip to conversation

nvmet: avoid circular locking warning

PATCH Superseded Hannes Reinecke 3 participants 6 messages Nov 1, 2023

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#4 lore raw

PATCH 2/2 nvmet-tcp: avoid circular locking dependency on install_queue()

In reply to Hannes Reinecke

nvmet_tcp_install_queue() is driven from the ->io_work workqueue function, but will call flush_workqueue() which might trigger ->release_work() which in itself calls flush_work on ->io_work.

To avoid that check for pending queue in disconnecting status, and return 'controller busy' until all disconnects are completed.

---
 drivers/nvme/target/tcp.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/nvme/target/tcp.c +12 -2
index b3c1cc6690f6..ecc9ccfac823 100644
@@ -2121,8 +2121,18 @@ static u16 nvmet_tcp_install_queue(struct nvmet_sq *sq)
 		container_of(sq, struct nvmet_tcp_queue, nvme_sq);
 
 	if (sq->qid == 0) {
-		/* Let inflight controller teardown complete */
-		flush_workqueue(nvmet_wq);
+		struct nvmet_tcp_queue *q;
+		int pending = 0;
+
+		mutex_lock(&nvmet_tcp_queue_mutex);
+		list_for_each_entry(q, &nvmet_tcp_queue_list, queue_list) {
+			if (q->state == NVMET_TCP_Q_DISCONNECTING)
+				pending++;
+		}
+		mutex_unlock(&nvmet_tcp_queue_mutex);
+		/* Retry for pending controller teardown */
+		if (pending)
+			return NVME_SC_CONNECT_CTRL_BUSY;
 	}
 
 	queue->nr_cmds = sq->size * 2;
Signature · 1 line
2.35.3
#0 lore raw

cover nvmet: avoid circular locking warning

nvmet-rdma and nvmet-tcp trigger a circular locking warning when tearing down; reason is a call to 'flush_workqueue' when creating a new controller which tries to cover for the fact that old controller instances might be in the process of tearing down.
However, this is pure speculation as we don't know (and don't check) if there really _are_ controllers in shutdown.
And even if there were, that should be short-lived, and would have been resolved by connecting just a tad later.
So this patch returns 'controller busy' if we really find ourselves in this situation, allowing the caller to reconnect later.

Hannes Reinecke (2):
nvmet-rdma: avoid circular locking dependency on install_queue()
nvmet-tcp: avoid circular locking dependency on install_queue()

 drivers/nvme/target/rdma.c | 14 ++++++++++++--
 drivers/nvme/target/tcp.c  | 14 ++++++++++++--
 2 files changed, 24 insertions(+), 4 deletions(-)
Signature · 1 line
2.35.3
#1 lore raw

PATCH 1/2 nvmet-rdma: avoid circular locking dependency on install_queue()

nvmet_rdma_install_queue() is driven from the ->io_work workqueue function, but will call flush_workqueue() which might trigger ->release_work() which in itself calls flush_work on ->io_work.

To avoid that check for pending queue in disconnecting status, and return 'controller busy' until all disconnects are completed.

---
 drivers/nvme/target/rdma.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/nvme/target/rdma.c +12 -2
index 4597bca43a6d..eaeb94a9e863 100644
@@ -1583,8 +1583,18 @@ static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id,
 	}
 
 	if (queue->host_qid == 0) {
-		/* Let inflight controller teardown complete */
-		flush_workqueue(nvmet_wq);
+		struct nvmet_rdma_queue *q;
+		int pending = 0;
+
+		mutex_lock(&nvmet_rdma_queue_mutex);
+		list_for_each_entry(q, &nvmet_rdma_queue_list, queue_list) {
+			if (q->state == NVMET_RDMA_Q_DISCONNECTING)
+				pending++;
+		}
+		mutex_unlock(&nvmet_rdma_queue_mutex);
+		/* Retry for pending controller teardown */
+		if (pending)
+			return NVME_SC_CONNECT_CTRL_BUSY;
 	}
 
 	ret = nvmet_rdma_cm_accept(cm_id, queue, &event->param.conn);
Signature · 1 line
2.35.3
Tested-by on the v1 series
#5 lore raw
In reply to Hannes Reinecke
Quoted 10 lines · Hannes Reinecke

nvmet-rdma and nvmet-tcp trigger a circular locking warning when tearing down; reason is a call to 'flush_workqueue' when creating a new controller which tries to cover for the fact that old controller instances might be in the process of tearing down.
However, this is pure speculation as we don't know (and don't check) if there really _are_ controllers in shutdown.
And even if there were, that should be short-lived, and would have been resolved by connecting just a tad later.
So this patch returns 'controller busy' if we really find ourselves in this situation, allowing the caller to reconnect later.

I confirmed this series avoids the failure that I reported for blktests test case nvme/003 with trtype=rdma or tcp. For the series,

maintainer
#2 lore raw

Re: [PATCH 1/2] nvmet-rdma: avoid circular locking dependency on install_queue()

In reply to Hannes Reinecke
Quoted 24 lines · Hannes Reinecke

nvmet_rdma_install_queue() is driven from the ->io_work workqueue function, but will call flush_workqueue() which might trigger ->release_work() which in itself calls flush_work on ->io_work.

To avoid that check for pending queue in disconnecting status, and return 'controller busy' until all disconnects are completed.

---
 drivers/nvme/target/rdma.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/nvme/target/rdma.c +2 -2
index 4597bca43a6d..eaeb94a9e863 100644
@@ -1583,8 +1583,18 @@ static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id,
 	}
 
 	if (queue->host_qid == 0) {
-		/* Let inflight controller teardown complete */
-		flush_workqueue(nvmet_wq);
+		struct nvmet_rdma_queue *q;
+		int pending = 0;

drivers/nvme/target/rdma.c static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id, · line 1597

Quoted 10 lines
drivers/nvme/target/rdma.c +10 -0
+
+		mutex_lock(&nvmet_rdma_queue_mutex);
+		list_for_each_entry(q, &nvmet_rdma_queue_list, queue_list) {
+			if (q->state == NVMET_RDMA_Q_DISCONNECTING)
+				pending++;
+		}
+		mutex_unlock(&nvmet_rdma_queue_mutex);
+		/* Retry for pending controller teardown */
+		if (pending)
+			return NVME_SC_CONNECT_CTRL_BUSY;

Not sure if it's worth turning this into a helper since both patches do the same thing. Probably not, since you'd need to pass in the mutex and state too. In any case, why not just break if you hit NVMET_RDMA_Q_DISCONNECTING rather than keep looping? You don't care about the exact count, jsut whether it's non-zero or not.

Signature · 1 line
Jens Axboe
#3 lore raw

Re: [PATCH 1/2] nvmet-rdma: avoid circular locking dependency on install_queue()

Quoted 33 lines · Jens Axboe
Quoted 32 lines · Hannes Reinecke

nvmet_rdma_install_queue() is driven from the ->io_work workqueue function, but will call flush_workqueue() which might trigger ->release_work() which in itself calls flush_work on ->io_work.

To avoid that check for pending queue in disconnecting status, and return 'controller busy' until all disconnects are completed.

---
  drivers/nvme/target/rdma.c | 14 ++++++++++++--
  1 file changed, 12 insertions(+), 2 deletions(-)
drivers/nvme/target/rdma.c +10 -2
index 4597bca43a6d..eaeb94a9e863 100644
@@ -1583,8 +1583,18 @@ static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id,
  	}
 
  	if (queue->host_qid == 0) {
-		/* Let inflight controller teardown complete */
-		flush_workqueue(nvmet_wq);
+		struct nvmet_rdma_queue *q;
+		int pending = 0;
+
+		mutex_lock(&nvmet_rdma_queue_mutex);
+		list_for_each_entry(q, &nvmet_rdma_queue_list, queue_list) {
+			if (q->state == NVMET_RDMA_Q_DISCONNECTING)
+				pending++;
+		}
+		mutex_unlock(&nvmet_rdma_queue_mutex);
+		/* Retry for pending controller teardown */
Quoted 9 lines

drivers/nvme/target/rdma.c static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id, · line 1597

Quoted 2 lines
drivers/nvme/target/rdma.c +2 -0
+		if (pending)
+			return NVME_SC_CONNECT_CTRL_BUSY;

Not sure if it's worth turning this into a helper since both patches do the same thing. Probably not, since you'd need to pass in the mutex and state too. In any case, why not just break if you hit NVMET_RDMA_Q_DISCONNECTING rather than keep looping? You don't care about the exact count, jsut whether it's non-zero or not.

True, that's easier. Will be updating the patches.

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman