nvmet-tcp: avoid circular locking dependency on install_queue()
Subsystems (2)
PATCH 1/1 nvmet-tcp: avoid circular locking dependency on install_queue()
nvmet_tcp_install_queue() is driven from the ->io_work workqueue function, but will call flush_workqueue() which might trigger ->release_work() which in itself calls flush_work on ->io_work.
To avoid that check for pending queue in disconnecting status, and return 'controller busy' until all disconnects are completed.
--- drivers/nvme/target/tcp.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/nvme/target/tcp.c +12 -2
@@ -1845,8 +1845,18 @@ static u16 nvmet_tcp_install_queue(struct nvmet_sq *sq) container_of(sq, struct nvmet_tcp_queue, nvme_sq);if (sq->qid ==0 ) { -/* Let inflight controller teardown complete */ -flush_workqueue (nvmet_wq); +struct nvmet_tcp_queue *q; +int pending =0 ; + +mutex_lock (&nvmet_tcp_queue_mutex); +list_for_each_entry (q, &nvmet_tcp_queue_list, queue_list) { +if (q->state ==NVMET_TCP_Q_DISCONNECTING ) + pending++; + } +mutex_unlock (&nvmet_tcp_queue_mutex); +/* Retry for pending controller teardown */ +if (pending) +return NVME_SC_CONNECT_CTRL_BUSY ; } queue->nr_cmds = sq->size *2 ;
Signature · 1 line
2.35.3
Sagi Grimberg
maintainerQuoted 24 lines · Hannes Reinecke
nvmet_tcp_install_queue()is driven from the ->io_work workqueue function, but will callflush_workqueue()which might trigger ->release_work() which in itself callsflush_workon ->io_work.To avoid that check for pending queue in disconnecting status, and return 'controller busy' until all disconnects are completed.
Signed-off-by Hannes Reinecke <[email protected]>--- drivers/nvme/target/tcp.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)drivers/nvme/target/tcp.c +2
-2@@ -1845,8 +1845,18 @@ static u16 nvmet_tcp_install_queue(struct nvmet_sq *sq) container_of(sq, struct nvmet_tcp_queue, nvme_sq);if (sq->qid ==0 ) { -/* Let inflight controller teardown complete */ -flush_workqueue (nvmet_wq); +struct nvmet_tcp_queue *q; +int pending =0 ;
drivers/nvme/target/tcp.c static u16 nvmet_tcp_install_queue(struct nvmet_sq *sq) · line 1859
Quoted 10 lines
drivers/nvme/target/tcp.c +10
-0+ +mutex_lock (&nvmet_tcp_queue_mutex); +list_for_each_entry (q, &nvmet_tcp_queue_list, queue_list) { +if (q->state ==NVMET_TCP_Q_DISCONNECTING ) + pending++; + } +mutex_unlock (&nvmet_tcp_queue_mutex); +/* Retry for pending controller teardown */ +if (pending) +return NVME_SC_CONNECT_CTRL_BUSY ;
Overall I'm fine with the approach.
That will most likely be the case for a controller reset or a "reset storm". Maybe it is better to keep it at some reasonably allowed pending to say 10 or something... The whole point is to not let the removals accumulate and continue creating new ones...
Can you please test this with a host that does a loop like:
while true; do nvme reset /dev/nvme0; done
And let it run for some time...
Or even better create a test that does this for 100 iterations or a log/short variants of this?
The same would be needed by nvmet-rdma btw.
Quoted 45 lines · Sagi Grimberg
Quoted 24 lines · Hannes Reinecke
nvmet_tcp_install_queue()is driven from the ->io_work workqueue function, but will callflush_workqueue()which might trigger ->release_work() which in itself callsflush_workon ->io_work.To avoid that check for pending queue in disconnecting status, and return 'controller busy' until all disconnects are completed.
Signed-off-by Hannes Reinecke <[email protected]>---
drivers/nvme/target/tcp.c| 14 ++++++++++++--1 file changed, 12 insertions(+), 2 deletions(-)drivers/nvme/target/tcp.c +2
-2@@ -1845,8 +1845,18 @@ static u16 nvmet_tcp_install_queue(struct nvmet_sq *sq) container_of(sq, struct nvmet_tcp_queue, nvme_sq); if (sq->qid == 0) { - /* Let inflight controller teardown complete */ - flush_workqueue(nvmet_wq); + struct nvmet_tcp_queue *q; + int pending = 0;drivers/nvme/target/tcp.c static u16 nvmet_tcp_install_queue(struct · line 1859
Quoted 10 lines
drivers/nvme/target/tcp.c +10
-0+ + mutex_lock(&nvmet_tcp_queue_mutex); + list_for_each_entry(q, &nvmet_tcp_queue_list, queue_list) { + if (q->state == NVMET_TCP_Q_DISCONNECTING) + pending++; + } + mutex_unlock(&nvmet_tcp_queue_mutex); + /* Retry for pending controller teardown */ + if (pending) + return NVME_SC_CONNECT_CTRL_BUSY;Overall I'm fine with the approach.
That will most likely be the case for a controller reset or a "reset storm". Maybe it is better to keep it at some reasonably allowed pending to say 10 or something... The whole point is to not let the removals accumulate and continue creating new ones...
Quoted 8 lines
Can you please test this with a host that does a loop like:
while true; do nvme reset/dev/nvme0; doneAnd let it run for some time...
Or even better create a test that does this for 100 iterations or a log/short variants of this?The same would be needed by nvmet-rdma btw.
Retested with the reset loop; no issues found after 327 iterations.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Sagi Grimberg
maintainerQuoted 47 lines
Quoted 47 lines · Hannes Reinecke
Quoted 24 lines
nvmet_tcp_install_queue()is driven from the ->io_work workqueue function, but will callflush_workqueue()which might trigger ->release_work() which in itself callsflush_workon ->io_work.To avoid that check for pending queue in disconnecting status, and return 'controller busy' until all disconnects are completed.
Signed-off-by Hannes Reinecke <[email protected]>---
drivers/nvme/target/tcp.c| 14 ++++++++++++--1 file changed, 12 insertions(+), 2 deletions(-)drivers/nvme/target/tcp.c +2
-2@@ -1845,8 +1845,18 @@ static u16 nvmet_tcp_install_queue(struct nvmet_sq *sq) container_of(sq, struct nvmet_tcp_queue, nvme_sq); if (sq->qid == 0) { - /* Let inflight controller teardown complete */ - flush_workqueue(nvmet_wq); + struct nvmet_tcp_queue *q; + int pending = 0;drivers/nvme/target/tcp.c static u16 nvmet_tcp_install_queue(struct · line 1859
Quoted 10 lines
drivers/nvme/target/tcp.c +10
-0+ + mutex_lock(&nvmet_tcp_queue_mutex); + list_for_each_entry(q, &nvmet_tcp_queue_list, queue_list) { + if (q->state == NVMET_TCP_Q_DISCONNECTING) + pending++; + } + mutex_unlock(&nvmet_tcp_queue_mutex); + /* Retry for pending controller teardown */ + if (pending) + return NVME_SC_CONNECT_CTRL_BUSY;Overall I'm fine with the approach.
That will most likely be the case for a controller reset or a "reset storm". Maybe it is better to keep it at some reasonably allowed pending to say 10 or something... The whole point is to not let the removals accumulate and continue creating new ones...
Can you please test this with a host that does a loop like:
while true; do nvme reset/dev/nvme0; done
Quoted 7 lines
Quoted 5 lines
And let it run for some time...
Or even better create a test that does this for 100 iterations or a log/short variants of this?The same would be needed by nvmet-rdma btw.
Retested with the reset loop; no issues found after 327 iterations.
Interesting, did the host see a NVME_SC_CONNECT_CTRL_BUSY during the test?
Quoted 51 lines · Sagi Grimberg
Quoted 51 lines
Quoted 51 lines · Hannes Reinecke
Quoted 24 lines
nvmet_tcp_install_queue()is driven from the ->io_work workqueue function, but will callflush_workqueue()which might trigger ->release_work() which in itself callsflush_workon ->io_work.To avoid that check for pending queue in disconnecting status, and return 'controller busy' until all disconnects are completed.
Signed-off-by Hannes Reinecke <[email protected]>---
drivers/nvme/target/tcp.c| 14 ++++++++++++--1 file changed, 12 insertions(+), 2 deletions(-)drivers/nvme/target/tcp.c +2
-2@@ -1845,8 +1845,18 @@ static u16 nvmet_tcp_install_queue(struct nvmet_sq *sq) container_of(sq, struct nvmet_tcp_queue, nvme_sq); if (sq->qid == 0) { - /* Let inflight controller teardown complete */ - flush_workqueue(nvmet_wq); + struct nvmet_tcp_queue *q; + int pending = 0;drivers/nvme/target/tcp.c static u16 nvmet_tcp_install_queue(struct · line 1859
Quoted 10 lines
drivers/nvme/target/tcp.c +10
-0+ + mutex_lock(&nvmet_tcp_queue_mutex); + list_for_each_entry(q, &nvmet_tcp_queue_list, queue_list) { + if (q->state == NVMET_TCP_Q_DISCONNECTING) + pending++; + } + mutex_unlock(&nvmet_tcp_queue_mutex); + /* Retry for pending controller teardown */ + if (pending) + return NVME_SC_CONNECT_CTRL_BUSY;Overall I'm fine with the approach.
That will most likely be the case for a controller reset or a "reset storm". Maybe it is better to keep it at some reasonably allowed pending to say 10 or something... The whole point is to not let the removals accumulate and continue creating new ones...
Can you please test this with a host that does a loop like:
while true; do nvme reset/dev/nvme0; doneAnd let it run for some time...
Or even better create a test that does this for 100 iterations or a log/short variants of this?
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
The same would be needed by nvmet-rdma btw.
Retested with the reset loop; no issues found after 327 iterations.
Interesting, did the host see a NVME_SC_CONNECT_CTRL_BUSY during the test?
Nope.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Sagi Grimberg
maintainerQuoted 50 lines
Quoted 50 lines
Quoted 50 lines
Quoted 50 lines
Quoted 25 lines
nvmet_tcp_install_queue()is driven from the ->io_work workqueue function, but will callflush_workqueue()which might trigger ->release_work() which in itself callsflush_workon ->io_work.To avoid that check for pending queue in disconnecting status, and return 'controller busy' until all disconnects are completed.
Signed-off-by Hannes Reinecke <[email protected]>---
drivers/nvme/target/tcp.c| 14 ++++++++++++--1 file changed, 12 insertions(+), 2 deletions(-)drivers/nvme/target/tcp.c +2
-2@@ -1845,8 +1845,18 @@ static u16 nvmet_tcp_install_queue(struct nvmet_sq *sq) container_of(sq, struct nvmet_tcp_queue, nvme_sq); if (sq->qid == 0) { - /* Let inflight controller teardown complete */ - flush_workqueue(nvmet_wq); + struct nvmet_tcp_queue *q; + int pending = 0;drivers/nvme/target/tcp.c static u16 nvmet_tcp_install_queue(struct · line 1859
Quoted 10 lines
drivers/nvme/target/tcp.c +10
-0+ + mutex_lock(&nvmet_tcp_queue_mutex); + list_for_each_entry(q, &nvmet_tcp_queue_list, queue_list) { + if (q->state == NVMET_TCP_Q_DISCONNECTING) + pending++; + } + mutex_unlock(&nvmet_tcp_queue_mutex); + /* Retry for pending controller teardown */ + if (pending) + return NVME_SC_CONNECT_CTRL_BUSY;Overall I'm fine with the approach.
That will most likely be the case for a controller reset or a "reset storm". Maybe it is better to keep it at some reasonably allowed pending to say 10 or something... The whole point is to not let the removals accumulate and continue creating new ones...
Can you please test this with a host that does a loop like:
while true; do nvme reset/dev/nvme0; doneAnd let it run for some time...
Or even better create a test that does this for 100 iterations or a log/short variants of this?
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
The same would be needed by nvmet-rdma btw.
Retested with the reset loop; no issues found after 327 iterations.
Interesting, did the host see a NVME_SC_CONNECT_CTRL_BUSY during the test?
Nope.
So you are not stressing enough?
Before (5+ years ago), multiple reporters, including myself ran a similar test and got to OOM after enough time.
If you are not exercising the busy return, you are not getting to a point where you need to back-pressure...