nvmet: avoid circular locking warning
Subsystems (2)
cover nvmet: avoid circular locking warning
nvmet-rdma and nvmet-tcp trigger a circular locking warning when tearing down; reason is a call to 'flush_workqueue' when creating a new controller which tries to cover for the fact that old controller instances might be in the process of tearing down.
However, this is pure speculation as we don't know (and don't check) if there really _are_ controllers in shutdown.
And even if there were, that should be short-lived, and would have been resolved by connecting just a tad later.
So this patch returns 'controller busy' if we really find ourselves in this situation, allowing the caller to reconnect later.
Hannes Reinecke (2):
nvmet-rdma: avoid circular locking dependency on install_queue()
nvmet-tcp: avoid circular locking dependency on install_queue()
drivers/nvme/target/rdma.c | 14 ++++++++++++-- drivers/nvme/target/tcp.c | 14 ++++++++++++-- 2 files changed, 24 insertions(+), 4 deletions(-)
Signature · 1 line
2.35.3
PATCH 1/2 nvmet-rdma: avoid circular locking dependency on install_queue()
nvmet_rdma_install_queue() is driven from the ->io_work workqueue function, but will call flush_workqueue() which might trigger ->release_work() which in itself calls flush_work on ->io_work.
To avoid that check for pending queue in disconnecting status, and return 'controller busy' until all disconnects are completed.
--- drivers/nvme/target/rdma.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/nvme/target/rdma.c +12 -2
@@ -1583,8 +1583,18 @@ static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id, }if (queue->host_qid ==0 ) { -/* Let inflight controller teardown complete */ -flush_workqueue (nvmet_wq); +struct nvmet_rdma_queue *q; +int pending =0 ; + +mutex_lock (&nvmet_rdma_queue_mutex); +list_for_each_entry (q, &nvmet_rdma_queue_list, queue_list) { +if (q->state ==NVMET_RDMA_Q_DISCONNECTING ) + pending++; + } +mutex_unlock (&nvmet_rdma_queue_mutex); +/* Retry for pending controller teardown */ +if (pending) +return NVME_SC_CONNECT_CTRL_BUSY ; } ret =nvmet_rdma_cm_accept (cm_id, queue, &event->param.conn);
Signature · 1 line
2.35.3
Jens Axboe
maintainerRe: [PATCH 1/2] nvmet-rdma: avoid circular locking dependency on install_queue()
Quoted 24 lines · Hannes Reinecke
nvmet_rdma_install_queue()is driven from the ->io_work workqueue function, but will callflush_workqueue()which might trigger ->release_work() which in itself callsflush_workon ->io_work.To avoid that check for pending queue in disconnecting status, and return 'controller busy' until all disconnects are completed.
Signed-off-by Hannes Reinecke <[email protected]>--- drivers/nvme/target/rdma.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)drivers/nvme/target/rdma.c +2
-2@@ -1583,8 +1583,18 @@ static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id, }if (queue->host_qid ==0 ) { -/* Let inflight controller teardown complete */ -flush_workqueue (nvmet_wq); +struct nvmet_rdma_queue *q; +int pending =0 ;
drivers/nvme/target/rdma.c static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id, · line 1597
Quoted 10 lines
drivers/nvme/target/rdma.c +10
-0+ +mutex_lock (&nvmet_rdma_queue_mutex); +list_for_each_entry (q, &nvmet_rdma_queue_list, queue_list) { +if (q->state ==NVMET_RDMA_Q_DISCONNECTING ) + pending++; + } +mutex_unlock (&nvmet_rdma_queue_mutex); +/* Retry for pending controller teardown */ +if (pending) +return NVME_SC_CONNECT_CTRL_BUSY ;
Not sure if it's worth turning this into a helper since both patches do the same thing. Probably not, since you'd need to pass in the mutex and state too. In any case, why not just break if you hit NVMET_RDMA_Q_DISCONNECTING rather than keep looping? You don't care about the exact count, jsut whether it's non-zero or not.
Signature · 1 line
Jens Axboe
Re: [PATCH 1/2] nvmet-rdma: avoid circular locking dependency on install_queue()
Quoted 33 lines · Jens Axboe
Quoted 32 lines · Hannes Reinecke
nvmet_rdma_install_queue()is driven from the ->io_work workqueue function, but will callflush_workqueue()which might trigger ->release_work() which in itself callsflush_workon ->io_work.To avoid that check for pending queue in disconnecting status, and return 'controller busy' until all disconnects are completed.
Signed-off-by Hannes Reinecke <[email protected]>--- drivers/nvme/target/rdma.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)drivers/nvme/target/rdma.c +10
-2@@ -1583,8 +1583,18 @@ static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id, }if (queue->host_qid ==0 ) { -/* Let inflight controller teardown complete */ -flush_workqueue (nvmet_wq); +struct nvmet_rdma_queue *q; +int pending =0 ; + +mutex_lock (&nvmet_rdma_queue_mutex); +list_for_each_entry (q, &nvmet_rdma_queue_list, queue_list) { +if (q->state ==NVMET_RDMA_Q_DISCONNECTING ) + pending++; + } +mutex_unlock (&nvmet_rdma_queue_mutex); +/* Retry for pending controller teardown */
Quoted 9 lines
drivers/nvme/target/rdma.c static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id, · line 1597
Quoted 2 lines
drivers/nvme/target/rdma.c +2
-0+if (pending) +return NVME_SC_CONNECT_CTRL_BUSY ;Not sure if it's worth turning this into a helper since both patches do the same thing. Probably not, since you'd need to pass in the mutex and state too. In any case, why not just break if you hit
NVMET_RDMA_Q_DISCONNECTINGrather than keep looping? You don't care about the exact count, jsut whether it's non-zero or not.
True, that's easier. Will be updating the patches.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
PATCH 2/2 nvmet-tcp: avoid circular locking dependency on install_queue()
In reply to Hannes Reineckenvmet_tcp_install_queue() is driven from the ->io_work workqueue function, but will call flush_workqueue() which might trigger ->release_work() which in itself calls flush_work on ->io_work.
To avoid that check for pending queue in disconnecting status, and return 'controller busy' until all disconnects are completed.
--- drivers/nvme/target/tcp.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/nvme/target/tcp.c +12 -2
@@ -2121,8 +2121,18 @@ static u16 nvmet_tcp_install_queue(struct nvmet_sq *sq) container_of(sq, struct nvmet_tcp_queue, nvme_sq);if (sq->qid ==0 ) { -/* Let inflight controller teardown complete */ -flush_workqueue (nvmet_wq); +struct nvmet_tcp_queue *q; +int pending =0 ; + +mutex_lock (&nvmet_tcp_queue_mutex); +list_for_each_entry (q, &nvmet_tcp_queue_list, queue_list) { +if (q->state ==NVMET_TCP_Q_DISCONNECTING ) + pending++; + } +mutex_unlock (&nvmet_tcp_queue_mutex); +/* Retry for pending controller teardown */ +if (pending) +return NVME_SC_CONNECT_CTRL_BUSY ; } queue->nr_cmds = sq->size *2 ;
Signature · 1 line
2.35.3
Quoted 10 lines · Hannes Reinecke
nvmet-rdma and nvmet-tcp trigger a circular locking warning when tearing down; reason is a call to '
flush_workqueue' when creating a new controller which tries to cover for the fact that old controller instances might be in the process of tearing down.
However, this is pure speculation as we don't know (and don't check) if there really _are_ controllers in shutdown.
And even if there were, that should be short-lived, and would have been resolved by connecting just a tad later.
So this patch returns 'controller busy' if we really find ourselves in this situation, allowing the caller to reconnect later.
I confirmed this series avoids the failure that I reported for blktests test case nvme/003 with trtype=rdma or tcp. For the series,