nvmet: avoid circular locking warning
Subsystems (2)
v3 cover nvmet: avoid circular locking warning
From: Hannes Reinecke <[email protected]>
nvmet-rdma and nvmet-tcp trigger a circular locking warning when tearing down; reason is a call to 'flush_workqueue' when creating a new controller which tries to cover for the fact that old controller instances might be in the process of tearing down.
However, this is pure speculation as we don't know (and don't check) if there really _are_ controllers in shutdown.
And even if there were, that should be short-lived, and would have been resolved by connecting just a tad later.
So this patch returns 'controller busy' if we really find ourselves in this situation, allowing the caller to reconnect later.
Changes to v2:
- Implement backlog as suggested by Sagi
Changes to the original version:
- Update the rdma patch to implement 'install_queue()'
- Include suggestions from Jens Axboe
Hannes Reinecke (2):
nvmet-tcp: avoid circular locking dependency on install_queue()
nvmet-rdma: avoid circular locking dependency on install_queue()
drivers/nvme/target/rdma.c | 19 ++++++++++++++++--- drivers/nvme/target/tcp.c | 18 +++++++++++++++--- 2 files changed, 31 insertions(+), 6 deletions(-)
Signature · 1 line
2.35.3
PATCH v3 1/2 nvmet-tcp: avoid circular locking dependency on install_queue()
From: Hannes Reinecke <[email protected]>
nvmet_tcp_install_queue() is driven from the ->io_work workqueue function, but will call flush_workqueue() which might trigger ->release_work() which in itself calls flush_work on ->io_work.
To avoid that check for pending queue in disconnecting status, and return 'controller busy' when we reached a certain threshold.
--- drivers/nvme/target/tcp.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-)
drivers/nvme/target/tcp.c +15 -3
@@ -24,6 +24,7 @@#include "nvmet.h" #define NVMET_TCP_DEF_INLINE_DATA_SIZE (4 * PAGE_SIZE) +#define NVMET_TCP_BACKLOG 128static int param_store_val (const char *str,int *val,int min,int max) {
@@ -2053,7 +2054,7 @@ static int nvmet_tcp_add_port(struct nvmet_port *nport) goto err_sock; } - ret =kernel_listen (port->sock,128 ); + ret =kernel_listen (port->sock,NVMET_TCP_BACKLOG );if (ret) {pr_err ("failed to listen %d on port sock\n" , ret); goto err_sock;
@@ -2119,8 +2120,19 @@ static u16 nvmet_tcp_install_queue(struct nvmet_sq *sq) container_of(sq, struct nvmet_tcp_queue, nvme_sq);if (sq->qid ==0 ) { -/* Let inflight controller teardown complete */ -flush_workqueue (nvmet_wq); +struct nvmet_tcp_queue *q; +int pending =0 ; + +/* Check for pending controller teardown */ +mutex_lock (&nvmet_tcp_queue_mutex); +list_for_each_entry (q, &nvmet_tcp_queue_list, queue_list) { +if (q->nvme_sq.ctrl == sq->ctrl && + q->state ==NVMET_TCP_Q_DISCONNECTING ) + pending++; + } +mutex_unlock (&nvmet_tcp_queue_mutex); +if (pending >NVMET_TCP_BACKLOG ) +return NVME_SC_CONNECT_CTRL_BUSY ; } queue->nr_cmds = sq->size *2 ;
Signature · 1 line
2.35.3
Re: [PATCH 1/2] nvmet-tcp: avoid circular locking dependency on install_queue()
PATCH v3 2/2 nvmet-rdma: avoid circular locking dependency on install_queue()
In reply to [email protected]From: Hannes Reinecke <[email protected]>
nvmet_rdma_install_queue() is driven from the ->io_work workqueue function, but will call flush_workqueue() which might trigger ->release_work() which in itself calls flush_work on ->io_work.
To avoid that check for pending queue in disconnecting status, and return 'controller busy' when we reached a certain threshold.
--- drivers/nvme/target/rdma.c | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-)
drivers/nvme/target/rdma.c +16 -3
@@ -37,6 +37,8 @@#define NVMET_RDMA_MAX_MDTS 8#define NVMET_RDMA_MAX_METADATA_MDTS 5 +#define NVMET_RDMA_BACKLOG 128 +struct nvmet_rdma_srq ;struct nvmet_rdma_cmd {
@@ -1583,8 +1585,19 @@ static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id, }if (queue->host_qid ==0 ) { -/* Let inflight controller teardown complete */ -flush_workqueue (nvmet_wq); +struct nvmet_rdma_queue *q; +int pending =0 ; + +/* Check for pending controller teardown */ +mutex_lock (&nvmet_rdma_queue_mutex); +list_for_each_entry (q, &nvmet_rdma_queue_list, queue_list) { +if (q->nvme_sq.ctrl == queue->nvme_sq.ctrl && + q->state ==NVMET_RDMA_Q_DISCONNECTING ) + pending++; + } +mutex_unlock (&nvmet_rdma_queue_mutex); +if (pending >NVMET_RDMA_BACKLOG ) +return NVME_SC_CONNECT_CTRL_BUSY ; } ret =nvmet_rdma_cm_accept (cm_id, queue, &event->param.conn);
@@ -1880,7 +1893,7 @@ static int nvmet_rdma_enable_port(struct nvmet_rdma_port *port) goto out_destroy_id; } - ret =rdma_listen (cm_id,128 ); + ret =rdma_listen (cm_id,NVMET_RDMA_BACKLOG );if (ret) {pr_err ("listening to %pISpcs failed (%d)\n" , addr, ret); goto out_destroy_id;
Signature · 1 line
2.35.3
Re: [PATCH 2/2] nvmet-rdma: avoid circular locking dependency on install_queue()
A message that is not in the archive (sent off-list or not yet received): [email protected]
Sagi Grimberg
maintainerRe: [PATCH 2/2] nvmet-rdma: avoid circular locking dependency on install_queue()
Quoted 51 lines · Max Gurtovoy
Hi Hannes/Sagi,
sorry for the late response on this one.
Quoted 3 lines · On 08/12/2023 14:53, [email protected] wrote:
From: Hannes Reinecke<[email protected]>
nvmet_rdma_install_queue()is driven from the ->io_work workqueue
nvmet_rdma_install_queuecallback is not implemented in the driver. And RDMA doesn't useio_workworkqueue..Quoted 26 lines
function, but will call
flush_workqueue()which might trigger ->release_work() which in itself callsflush_workon ->io_work.To avoid that check for pending queue in disconnecting status, and return 'controller busy' when we reached a certain threshold.
Signed-off-by Hannes Reinecke<[email protected]>Tested-by Shin'ichiro Kawasaki<[email protected]>---
drivers/nvme/target/rdma.c| 19 ++++++++++++++++---1 file changed, 16 insertions(+), 3 deletions(-)drivers/nvme/target/rdma.c +0
-0@@ -37,6 +37,8 @@ #define NVMET_RDMA_MAX_MDTS 8 #define NVMET_RDMA_MAX_METADATA_MDTS 5+#define
NVMET_RDMA_BACKLOG128
+
structnvmet_rdma_srq;
structnvmet_rdma_cmd{drivers/nvme/target/rdma.c +0
-0@@ -1583,8 +1585,19 @@ static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id, }drivers/nvme/target/rdma.c static int nvmet_rdma_queue_connect(struct · line 1594
Quoted 10 lines
drivers/nvme/target/rdma.c +7
-2if (queue->host_qid == 0) { - /* Let inflight controller teardown complete */ - flush_workqueue(nvmet_wq); + struct nvmet_rdma_queue *q; + int pending = 0; + + /* Check for pending controller teardown */ + mutex_lock(&nvmet_rdma_queue_mutex); + list_for_each_entry(q, &nvmet_rdma_queue_list, queue_list) { + if (q->nvme_sq.ctrl == queue->nvme_sq.ctrl &&
nvme_sq->ctrlpointer is set uponFABRICS_CONNECTduringnvmet_install_queue().
Quoted 1 line
obviously this check is not relevant since
queue->nvme_sq.ctrlis always NULL, isn't it ?
I think that the goal was to look at all the queues that were scheduled for teardown (queue->release_work) so sq->ctrl should be valid for those queues afaict.
Quoted 7 lines
So, I wonder what does this check is doing ?
Was the intention to check that
port->cm_id(listener) is not overloaded ?Also the return value is "FABRICS" return code and not the value that
rdma_cmis looking for.
Quoted 7 lines
We probably should introduce
nvmet_rdma_install_queue()and check the status of pending disconnections associated withport->cm_itwhich is the one with the configured backlog.I don't understand the solution also for the
nvmet_tcp.I think this series only removed the "
flush_workqueue(nvmet_wq);" actually.Is there something I miss in this commit ?
IIRC the goal was to get rid of the circular locking lockdep complaint. nvmet needs a back-pressure mechanism for incoming connections because queue teardown is async and nvmet may hit OOM in connect-disconnect storm scenario. The goal here was to detect such scenario, by counting the number of queues that are queued for teardown, and if they exceed a certain threshold, return a BUSY status to the incoming fabrics connect.
Before, nvmet flushed the pending queue teardowns, but that triggered a lockdep complaint and hannes attempted to address it.
Quoted 18 lines
Quoted 18 lines
drivers/nvme/target/rdma.c +7
-0+ q->state == NVMET_RDMA_Q_DISCONNECTING) + pending++; + } + mutex_unlock(&nvmet_rdma_queue_mutex); + if (pending > NVMET_RDMA_BACKLOG) + return NVME_SC_CONNECT_CTRL_BUSY; } ret = nvmet_rdma_cm_accept(cm_id, queue, &event->param.conn);@@ -1880,7 +1893,7 @@ static int nvmet_rdma_enable_port(struct nvmet_rdma_port *port) goto out_destroy_id; } - ret = rdma_listen(cm_id, 128); + ret = rdma_listen(cm_id, NVMET_RDMA_BACKLOG); if (ret) { pr_err("listening to %pISpcs failed (%d)\n", addr, ret);goto
out_destroy_id;
Quoted 12 lines · On Dec 08, 2023 / 13:53, [email protected] wrote:
From: Hannes Reinecke <[email protected]>
nvmet-rdma and nvmet-tcp trigger a circular locking warning when tearing down; reason is a call to '
flush_workqueue' when creating a new controller which tries to cover for the fact that old controller instances might be in the process of tearing down.
However, this is pure speculation as we don't know (and don't check) if there really _are_ controllers in shutdown.
And even if there were, that should be short-lived, and would have been resolved by connecting just a tad later.
So this patch returns 'controller busy' if we really find ourselves in this situation, allowing the caller to reconnect later.
Thanks Hannes. I reconfirmed the patches avoid the circular lockdep WARN with rdma and tcp transports using v6.7-rc5 kernel. And I observed no regression in my blktests runs. Looks good.
Quoted 12 lines · On Dec 08, 2023 / 13:53, [email protected] wrote:
From: Hannes Reinecke <[email protected]>
nvmet-rdma and nvmet-tcp trigger a circular locking warning when tearing down; reason is a call to '
flush_workqueue' when creating a new controller which tries to cover for the fact that old controller instances might be in the process of tearing down.
However, this is pure speculation as we don't know (and don't check) if there really _are_ controllers in shutdown.
And even if there were, that should be short-lived, and would have been resolved by connecting just a tad later.
So this patch returns 'controller busy' if we really find ourselves in this situation, allowing the caller to reconnect later.
Keith,
Could you consider to upstream these patches for kernel v6.8-rcX? They fix lockdep WARNs observed with blktests nvme test groups and rdma/tcp transport.
Keith Busch
maintainerQuoted 14 lines · Shinichiro Kawasaki
Quoted 12 lines · On Dec 08, 2023 / 13:53, [email protected] wrote:
From: Hannes Reinecke <[email protected]>
nvmet-rdma and nvmet-tcp trigger a circular locking warning when tearing down; reason is a call to '
flush_workqueue' when creating a new controller which tries to cover for the fact that old controller instances might be in the process of tearing down.
However, this is pure speculation as we don't know (and don't check) if there really _are_ controllers in shutdown.
And even if there were, that should be short-lived, and would have been resolved by connecting just a tad later.
So this patch returns 'controller busy' if we really find ourselves in this situation, allowing the caller to reconnect later.
Quoted 4 lines
Keith,
Could you consider to upstream these patches for kernel v6.8-rcX? They fix lockdep WARNs observed with blktests nvme test groups and rdma/tcp transport.
Definitely, I knew I was forgetting something. Patches applied now for 6.8 and our 2nd merge window pull request will be issued today.