Skip to contentSkip to conversation

nvmet-tcp: avoid circular locking dependency on install_queue()

PATCH In review Hannes Reinecke 2 participants 6 messages Aug 10, 2023 – Aug 17, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 nvmet-tcp: avoid circular locking dependency on install_queue()

nvmet_tcp_install_queue() is driven from the ->io_work workqueue function, but will call flush_workqueue() which might trigger ->release_work() which in itself calls flush_work on ->io_work.

To avoid that check for pending queue in disconnecting status, and return 'controller busy' until all disconnects are completed.

---
 drivers/nvme/target/tcp.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/nvme/target/tcp.c +12 -2
index f131b504aa01..97d07488072d 100644
@@ -1845,8 +1845,18 @@ static u16 nvmet_tcp_install_queue(struct nvmet_sq *sq)
 		container_of(sq, struct nvmet_tcp_queue, nvme_sq);
 
 	if (sq->qid == 0) {
-		/* Let inflight controller teardown complete */
-		flush_workqueue(nvmet_wq);
+		struct nvmet_tcp_queue *q;
+		int pending = 0;
+
+		mutex_lock(&nvmet_tcp_queue_mutex);
+		list_for_each_entry(q, &nvmet_tcp_queue_list, queue_list) {
+			if (q->state == NVMET_TCP_Q_DISCONNECTING)
+				pending++;
+		}
+		mutex_unlock(&nvmet_tcp_queue_mutex);
+		/* Retry for pending controller teardown */
+		if (pending)
+			return NVME_SC_CONNECT_CTRL_BUSY;
 	}
 
 	queue->nr_cmds = sq->size * 2;
Signature · 1 line
2.35.3
maintainer
#1 lore raw
Quoted 24 lines · Hannes Reinecke

nvmet_tcp_install_queue() is driven from the ->io_work workqueue function, but will call flush_workqueue() which might trigger ->release_work() which in itself calls flush_work on ->io_work.

To avoid that check for pending queue in disconnecting status, and return 'controller busy' until all disconnects are completed.

---
  drivers/nvme/target/tcp.c | 14 ++++++++++++--
  1 file changed, 12 insertions(+), 2 deletions(-)
drivers/nvme/target/tcp.c +2 -2
index f131b504aa01..97d07488072d 100644
@@ -1845,8 +1845,18 @@ static u16 nvmet_tcp_install_queue(struct nvmet_sq *sq)
  		container_of(sq, struct nvmet_tcp_queue, nvme_sq);
 
  	if (sq->qid == 0) {
-		/* Let inflight controller teardown complete */
-		flush_workqueue(nvmet_wq);
+		struct nvmet_tcp_queue *q;
+		int pending = 0;

drivers/nvme/target/tcp.c static u16 nvmet_tcp_install_queue(struct nvmet_sq *sq) · line 1859

Quoted 10 lines
drivers/nvme/target/tcp.c +10 -0
+
+		mutex_lock(&nvmet_tcp_queue_mutex);
+		list_for_each_entry(q, &nvmet_tcp_queue_list, queue_list) {
+			if (q->state == NVMET_TCP_Q_DISCONNECTING)
+				pending++;
+		}
+		mutex_unlock(&nvmet_tcp_queue_mutex);
+		/* Retry for pending controller teardown */
+		if (pending)
+			return NVME_SC_CONNECT_CTRL_BUSY;

Overall I'm fine with the approach.

That will most likely be the case for a controller reset or a "reset storm". Maybe it is better to keep it at some reasonably allowed pending to say 10 or something... The whole point is to not let the removals accumulate and continue creating new ones...

Can you please test this with a host that does a loop like:
while true; do nvme reset /dev/nvme0; done

And let it run for some time...
Or even better create a test that does this for 100 iterations or a log/short variants of this?

The same would be needed by nvmet-rdma btw.

#2 lore raw
Quoted 45 lines · Sagi Grimberg
Quoted 24 lines · Hannes Reinecke

nvmet_tcp_install_queue() is driven from the ->io_work workqueue function, but will call flush_workqueue() which might trigger ->release_work() which in itself calls flush_work on ->io_work.

To avoid that check for pending queue in disconnecting status, and return 'controller busy' until all disconnects are completed.

---
  drivers/nvme/target/tcp.c | 14 ++++++++++++--

  1 file changed, 12 insertions(+), 2 deletions(-)
drivers/nvme/target/tcp.c +2 -2
index f131b504aa01..97d07488072d 100644
@@ -1845,8 +1845,18 @@ static u16 nvmet_tcp_install_queue(struct 
 nvmet_sq *sq)
           container_of(sq, struct nvmet_tcp_queue, nvme_sq);
       if (sq->qid == 0) {
-        /* Let inflight controller teardown complete */
-        flush_workqueue(nvmet_wq);
+        struct nvmet_tcp_queue *q;
+        int pending = 0;

drivers/nvme/target/tcp.c static u16 nvmet_tcp_install_queue(struct · line 1859

Quoted 10 lines
drivers/nvme/target/tcp.c +10 -0
+
+        mutex_lock(&nvmet_tcp_queue_mutex);
+        list_for_each_entry(q, &nvmet_tcp_queue_list, queue_list) {
+            if (q->state == NVMET_TCP_Q_DISCONNECTING)
+                pending++;
+        }
+        mutex_unlock(&nvmet_tcp_queue_mutex);
+        /* Retry for pending controller teardown */
+        if (pending)
+            return NVME_SC_CONNECT_CTRL_BUSY;

Overall I'm fine with the approach.

That will most likely be the case for a controller reset or a "reset storm". Maybe it is better to keep it at some reasonably allowed pending to say 10 or something... The whole point is to not let the removals accumulate and continue creating new ones...

Quoted 8 lines

Can you please test this with a host that does a loop like:
while true; do nvme reset /dev/nvme0; done

And let it run for some time...
Or even better create a test that does this for 100 iterations or a log/short variants of this?

The same would be needed by nvmet-rdma btw.

Retested with the reset loop; no issues found after 327 iterations.

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman
maintainer
#3 lore raw
Quoted 47 lines
Quoted 47 lines · Hannes Reinecke
Quoted 24 lines

nvmet_tcp_install_queue() is driven from the ->io_work workqueue function, but will call flush_workqueue() which might trigger ->release_work() which in itself calls flush_work on ->io_work.

To avoid that check for pending queue in disconnecting status, and return 'controller busy' until all disconnects are completed.

---
  drivers/nvme/target/tcp.c | 14 ++++++++++++--

  1 file changed, 12 insertions(+), 2 deletions(-)
drivers/nvme/target/tcp.c +2 -2
index f131b504aa01..97d07488072d 100644
@@ -1845,8 +1845,18 @@ static u16 nvmet_tcp_install_queue(struct 
 nvmet_sq *sq)
           container_of(sq, struct nvmet_tcp_queue, nvme_sq);
       if (sq->qid == 0) {
-        /* Let inflight controller teardown complete */
-        flush_workqueue(nvmet_wq);
+        struct nvmet_tcp_queue *q;
+        int pending = 0;

drivers/nvme/target/tcp.c static u16 nvmet_tcp_install_queue(struct · line 1859

Quoted 10 lines
drivers/nvme/target/tcp.c +10 -0
+
+        mutex_lock(&nvmet_tcp_queue_mutex);
+        list_for_each_entry(q, &nvmet_tcp_queue_list, queue_list) {
+            if (q->state == NVMET_TCP_Q_DISCONNECTING)
+                pending++;
+        }
+        mutex_unlock(&nvmet_tcp_queue_mutex);
+        /* Retry for pending controller teardown */
+        if (pending)
+            return NVME_SC_CONNECT_CTRL_BUSY;

Overall I'm fine with the approach.

That will most likely be the case for a controller reset or a "reset storm". Maybe it is better to keep it at some reasonably allowed pending to say 10 or something... The whole point is to not let the removals accumulate and continue creating new ones...

Can you please test this with a host that does a loop like:
while true; do nvme reset /dev/nvme0; done

Quoted 7 lines
Quoted 5 lines

And let it run for some time...
Or even better create a test that does this for 100 iterations or a log/short variants of this?

The same would be needed by nvmet-rdma btw.

Retested with the reset loop; no issues found after 327 iterations.

Interesting, did the host see a NVME_SC_CONNECT_CTRL_BUSY during the test?

#4 lore raw
Quoted 51 lines · Sagi Grimberg
Quoted 51 lines
Quoted 51 lines · Hannes Reinecke
Quoted 24 lines

nvmet_tcp_install_queue() is driven from the ->io_work workqueue function, but will call flush_workqueue() which might trigger ->release_work() which in itself calls flush_work on ->io_work.

To avoid that check for pending queue in disconnecting status, and return 'controller busy' until all disconnects are completed.

---
  drivers/nvme/target/tcp.c | 14 ++++++++++++--

  1 file changed, 12 insertions(+), 2 deletions(-)
drivers/nvme/target/tcp.c +2 -2
index f131b504aa01..97d07488072d 100644
@@ -1845,8 +1845,18 @@ static u16 nvmet_tcp_install_queue(struct 
 nvmet_sq *sq)
           container_of(sq, struct nvmet_tcp_queue, nvme_sq);
       if (sq->qid == 0) {
-        /* Let inflight controller teardown complete */
-        flush_workqueue(nvmet_wq);
+        struct nvmet_tcp_queue *q;
+        int pending = 0;

drivers/nvme/target/tcp.c static u16 nvmet_tcp_install_queue(struct · line 1859

Quoted 10 lines
drivers/nvme/target/tcp.c +10 -0
+
+        mutex_lock(&nvmet_tcp_queue_mutex);
+        list_for_each_entry(q, &nvmet_tcp_queue_list, queue_list) {
+            if (q->state == NVMET_TCP_Q_DISCONNECTING)
+                pending++;
+        }
+        mutex_unlock(&nvmet_tcp_queue_mutex);
+        /* Retry for pending controller teardown */
+        if (pending)
+            return NVME_SC_CONNECT_CTRL_BUSY;

Overall I'm fine with the approach.

That will most likely be the case for a controller reset or a "reset storm". Maybe it is better to keep it at some reasonably allowed pending to say 10 or something... The whole point is to not let the removals accumulate and continue creating new ones...

Can you please test this with a host that does a loop like:
while true; do nvme reset /dev/nvme0; done

And let it run for some time...
Or even better create a test that does this for 100 iterations or a log/short variants of this?

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

The same would be needed by nvmet-rdma btw.

Retested with the reset loop; no issues found after 327 iterations.

Interesting, did the host see a NVME_SC_CONNECT_CTRL_BUSY  during the test?

Nope.

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman
maintainer
#5 lore raw
Quoted 50 lines
Quoted 50 lines
Quoted 50 lines
Quoted 50 lines
Quoted 25 lines

nvmet_tcp_install_queue() is driven from the ->io_work workqueue function, but will call flush_workqueue() which might trigger ->release_work() which in itself calls flush_work on ->io_work.

To avoid that check for pending queue in disconnecting status, and return 'controller busy' until all disconnects are completed.

---
  drivers/nvme/target/tcp.c | 14 ++++++++++++--

  1 file changed, 12 insertions(+), 2 deletions(-)
drivers/nvme/target/tcp.c +2 -2
index f131b504aa01..97d07488072d 100644
@@ -1845,8 +1845,18 @@ static u16 nvmet_tcp_install_queue(struct 
 nvmet_sq *sq)
           container_of(sq, struct nvmet_tcp_queue, nvme_sq);
       if (sq->qid == 0) {
-        /* Let inflight controller teardown complete */
-        flush_workqueue(nvmet_wq);
+        struct nvmet_tcp_queue *q;
+        int pending = 0;

drivers/nvme/target/tcp.c static u16 nvmet_tcp_install_queue(struct · line 1859

Quoted 10 lines
drivers/nvme/target/tcp.c +10 -0
+
+        mutex_lock(&nvmet_tcp_queue_mutex);
+        list_for_each_entry(q, &nvmet_tcp_queue_list, queue_list) {
+            if (q->state == NVMET_TCP_Q_DISCONNECTING)
+                pending++;
+        }
+        mutex_unlock(&nvmet_tcp_queue_mutex);
+        /* Retry for pending controller teardown */
+        if (pending)
+            return NVME_SC_CONNECT_CTRL_BUSY;

Overall I'm fine with the approach.

That will most likely be the case for a controller reset or a "reset storm". Maybe it is better to keep it at some reasonably allowed pending to say 10 or something... The whole point is to not let the removals accumulate and continue creating new ones...

Can you please test this with a host that does a loop like:
while true; do nvme reset /dev/nvme0; done

And let it run for some time...
Or even better create a test that does this for 100 iterations or a log/short variants of this?

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

The same would be needed by nvmet-rdma btw.

Retested with the reset loop; no issues found after 327 iterations.

Interesting, did the host see a NVME_SC_CONNECT_CTRL_BUSY  during the test?

Nope.

So you are not stressing enough?

Before (5+ years ago), multiple reporters, including myself ran a similar test and got to OOM after enough time.

If you are not exercising the busy return, you are not getting to a point where you need to back-pressure...