nvme-tcp: parallelize I/O setup queue in connect
v3 cover nvme-tcp: parallelize I/O setup queue in connect
Similar to commit 2a8513091d2f ("nvme-rdma: parallelize I/O queue allocation and startup"), I/O queue allocation and start steps (including authentication) can be parallelized for nvme-tcp as well.
Patch 1 fixes a pre-existing bug: nvme_tcp_alloc_queue() used current->nsproxy->net_ns to create sockets during initial connect while it used kernel's init_net for reconnect and error recovery paths. The fix captures the caller's network namespace at controller allocation time.
Patch 2 is a preparatory refactor: it splits the queue-count negotiation and TLS PSK checks into helpers so that the individual steps can be called directly from nvme_tcp_configure_io_queues(). No functional change.
Patch 3 has the async implementation: it fans the per-queue alloc and start out over an async domain, serializes the per-cpu queue-count in nvme_tcp_set_queue_io_cpu() that can now run concurrently, and uses ctrl->net for all socket creation.
Testing on a 64-core host with 64 IO-queues shows nvme-tcp connection time reduced from 61ms to 11ms.
---
v3:
- Added patch 1 fixing a network namespace bug: reconnect and error recovery paths were creating sockets in init_net instead of the controller's namespace. Patch 3 now uses ctrl->net (from patch 1) instead of hardcoding init_net.
- Patches renumbered accordingly (v2 patch 1->2, patch 2->3).
v2:
- Resending the entire series to fix missing numbering in the original patch 2 subject line. No functional code changes.
v1: https://lore.kernel.org/[email protected]/
v2: https://lore.kernel.org/[email protected]/
---
Surabhi Gogte (3):
nvme-tcp: store and use the caller's network namespace
nvme-tcp: refactor I/O queue setup path
nvme-tcp: parallelize I/O queue allocation and startup
drivers/nvme/host/tcp.c | 144 +++++++++++++++++++++++++++------------- 1 file changed, 98 insertions(+), 46 deletions(-)
Signature · 1 line
2.55.0
PATCH v3 1/3 nvme-tcp: store and use the caller's network namespace
nvme_tcp_alloc_queue() creates each socket using current->nsproxy->net_ns, which is correct for the initial connect that runs in userspace context. However, the reconnect and error recovery paths run from a workqueue where current netns is a kernel thread in init_net, so sockets created during reconnect end up in a different namespace than the one the controller was originally connected from.
Fix this by capturing the caller's network namespace at controller allocation time. Use ctrl->net for all socket creation so that every queue, on initial connect, reset, or reconnect, is created in the same namespace.
--- drivers/nvme/host/tcp.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
drivers/nvme/host/tcp.c +6 -1
@@ -171,6 +171,7 @@ struct nvme_tcp_ctrl {struct delayed_work connect_work;struct nvme_tcp_request async_req;u32 io_queues[HCTX_MAX_TYPES ]; +struct net *net; };static struct workqueue_struct *nvme_tcp_wq;
@@ -1846,7 +1847,7 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid, queue->cmnd_capsule_len =sizeof (struct nvme_command ) +NVME_TCP_ADMIN_CCSZ ; - ret =sock_create_kern (current->nsproxy->net_ns, + ret =sock_create_kern (ctrl->net, ctrl->addr.ss_family,SOCK_STREAM ,IPPROTO_TCP , &queue->sock);if (ret) {
@@ -2638,6 +2639,7 @@ static void nvme_tcp_free_ctrl(struct nvme_ctrl *nctrl)nvmf_free_options (nctrl->opts); free_ctrl: +put_net (ctrl->net);kfree (ctrl->queues);kfree (ctrl); }
@@ -3036,12 +3038,15 @@ static struct nvme_tcp_ctrl *nvme_tcp_alloc_ctrl(struct device *dev, goto out_free_ctrl; } + ctrl->net =get_net (current->nsproxy->net_ns); + ret =nvme_init_ctrl (&ctrl->ctrl, dev, &nvme_tcp_ctrl_ops,0 );if (ret) goto out_kfree_queues;return ctrl; out_kfree_queues: +put_net (ctrl->net);kfree (ctrl->queues); out_free_ctrl:kfree (ctrl);
Signature · 1 line
2.55.0
Re: [PATCH v3 1/3] nvme-tcp: store and use the caller's network namespace
Re: [PATCH v3 1/3] nvme-tcp: store and use the caller's network namespace
In reply to Surabhi GogteQuoted 53 lines · Surabhi Gogte
nvme_tcp_alloc_queue()creates each socket usingcurrent->nsproxy->net_ns, which is correct for the initial connect that runs in userspace context. However, the reconnect and error recovery paths run from a workqueue where current netns is a kernel thread ininit_net, so sockets created during reconnect end up in a different namespace than the one the controller was originally connected from.Fix this by capturing the caller's network namespace at controller allocation time. Use
ctrl->netfor all socket creation so that every queue, on initial connect, reset, or reconnect, is created in the same namespace.Fixes 3f2304f8c6d6 ("nvme-tcp: add NVMe over TCP host driver")Signed-off-by Surabhi Gogte <[email protected]>--- drivers/nvme/host/tcp.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)drivers/nvme/host/tcp.c +5
-1@@ -171,6 +171,7 @@ struct nvme_tcp_ctrl {struct delayed_work connect_work;struct nvme_tcp_request async_req;u32 io_queues[HCTX_MAX_TYPES ]; +struct net *net; };static struct workqueue_struct *nvme_tcp_wq;@@ -1846,7 +1847,7 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid, queue->cmnd_capsule_len =sizeof (struct nvme_command ) +NVME_TCP_ADMIN_CCSZ ; - ret =sock_create_kern (current->nsproxy->net_ns, + ret =sock_create_kern (ctrl->net, ctrl->addr.ss_family,SOCK_STREAM ,IPPROTO_TCP , &queue->sock);if (ret) {@@ -2638,6 +2639,7 @@ static void nvme_tcp_free_ctrl(struct nvme_ctrl *nctrl)nvmf_free_options (nctrl->opts); free_ctrl: +put_net (ctrl->net);kfree (ctrl->queues);kfree (ctrl); }@@ -3036,12 +3038,15 @@ static struct nvme_tcp_ctrl *nvme_tcp_alloc_ctrl(struct device *dev, goto out_free_ctrl; } + ctrl->net =get_net (current->nsproxy->net_ns); +
drivers/nvme/host/tcp.c static struct nvme_tcp_ctrl *nvme_tcp_alloc_ctrl(struct device *dev, · line 3052
Quoted 10 lines
drivers/nvme/host/tcp.c +1
-0ret =nvme_init_ctrl (&ctrl->ctrl, dev, &nvme_tcp_ctrl_ops,0 );if (ret) goto out_kfree_queues;return ctrl; out_kfree_queues: +put_net (ctrl->net);kfree (ctrl->queues); out_free_ctrl:kfree (ctrl);
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
PATCH v3 2/3 nvme-tcp: refactor I/O queue setup path
In reply to Surabhi GogteSplit the I/O queue setup helpers apart so that the individual steps can be called directly from nvme_tcp_configure_io_queues():
- Queue count negotiation moves out of nvme_tcp_alloc_io_queues() into a new nvme_tcp_set_io_queue_count(), leaving the allocator with just the per-queue allocation loop.
- TLS PSK validation moves out of __nvme_tcp_alloc_io_queues() into a new nvme_tcp_tls_check_psk().
- nvme_tcp_configure_io_queues() now calls the three steps directly in the same order as before.
--- drivers/nvme/host/tcp.c | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-)
drivers/nvme/host/tcp.c +19 -6
@@ -2172,10 +2172,8 @@ static int nvme_tcp_alloc_admin_queue(struct nvme_ctrl *ctrl)return ret; } -static int __nvme_tcp_alloc_io_queues (struct nvme_ctrl *ctrl) +static int nvme_tcp_tls_check_psk (struct nvme_ctrl *ctrl) { -int i, ret; -if (nvme_tcp_tls_configured (ctrl)) {if (ctrl->opts->concat) {/*
@@ -2197,6 +2195,13 @@ static int __nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl) } } +return 0 ; +} + +static int __nvme_tcp_alloc_io_queues (struct nvme_ctrl *ctrl) +{ +int i, ret; +for (i =1 ; i < ctrl->queue_count; i++) { ret =nvme_tcp_alloc_queue (ctrl, i, ctrl->tls_pskid);
@@ -2213,7 +2218,7 @@ static int __nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl)return ret; } -static int nvme_tcp_alloc_io_queues (struct nvme_ctrl *ctrl) +static int nvme_tcp_set_io_queue_count (struct nvme_ctrl *ctrl) {unsigned int nr_io_queues;int ret;
@@ -2235,14 +2240,22 @@ static int nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl)nvmf_set_io_queues (ctrl->opts, nr_io_queues,to_tcp_ctrl (ctrl)->io_queues); -return __nvme_tcp_alloc_io_queues (ctrl); +return 0 ; }static int nvme_tcp_configure_io_queues (struct nvme_ctrl *ctrl,bool new) {int ret, nr_queues; - ret =nvme_tcp_alloc_io_queues (ctrl); + ret =nvme_tcp_set_io_queue_count (ctrl); +if (ret) +return ret; + + ret =nvme_tcp_tls_check_psk (ctrl); +if (ret) +return ret; + + ret =__nvme_tcp_alloc_io_queues (ctrl);if (ret)return ret;
Signature · 1 line
2.55.0
Re: [PATCH v3 2/3] nvme-tcp: refactor I/O queue setup path
Re: [PATCH v3 2/3] nvme-tcp: refactor I/O queue setup path
In reply to Surabhi GogteQuoted 47 lines · Surabhi Gogte
Split the I/O queue setup helpers apart so that the individual steps can be called directly from
nvme_tcp_configure_io_queues():- Queue count negotiation moves out of
nvme_tcp_alloc_io_queues()into a newnvme_tcp_set_io_queue_count(), leaving the allocator with just the per-queue allocation loop.
- TLS PSK validation moves out of__nvme_tcp_alloc_io_queues()into a newnvme_tcp_tls_check_psk().
-nvme_tcp_configure_io_queues()now calls the three steps directly in the same order as before.Signed-off-by Surabhi Gogte <[email protected]>--- drivers/nvme/host/tcp.c | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-)drivers/nvme/host/tcp.c +8
-3@@ -2172,10 +2172,8 @@ static int nvme_tcp_alloc_admin_queue(struct nvme_ctrl *ctrl)return ret; } -static int __nvme_tcp_alloc_io_queues (struct nvme_ctrl *ctrl) +static int nvme_tcp_tls_check_psk (struct nvme_ctrl *ctrl) { -int i, ret; -if (nvme_tcp_tls_configured (ctrl)) {if (ctrl->opts->concat) {/* @@ -2197,6 +2195,13 @@ static int __nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl) } } +return 0 ; +} + +static int __nvme_tcp_alloc_io_queues (struct nvme_ctrl *ctrl) +{ +int i, ret; +for (i =1 ; i < ctrl->queue_count; i++) { ret =nvme_tcp_alloc_queue (ctrl, i, ctrl->tls_pskid);
drivers/nvme/host/tcp.c static int __nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl) · line 2224
Quoted 8 lines
drivers/nvme/host/tcp.c +1
-1@@ -2213,7 +2218,7 @@ static int __nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl)return ret; } -static int nvme_tcp_alloc_io_queues (struct nvme_ctrl *ctrl) +static int nvme_tcp_set_io_queue_count (struct nvme_ctrl *ctrl) {unsigned int nr_io_queues;int ret;
We are not only set the queue count, but also allocating the queues themselves. Why not keep the original name?
Quoted 15 lines
drivers/nvme/host/tcp.c +3
-2@@ -2235,14 +2240,22 @@ static int nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl)nvmf_set_io_queues (ctrl->opts, nr_io_queues,to_tcp_ctrl (ctrl)->io_queues); -return __nvme_tcp_alloc_io_queues (ctrl); +return 0 ; }static int nvme_tcp_configure_io_queues (struct nvme_ctrl *ctrl,bool new) {int ret, nr_queues; - ret =nvme_tcp_alloc_io_queues (ctrl); + ret =nvme_tcp_set_io_queue_count (ctrl); +if (ret)
drivers/nvme/host/tcp.c static int nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl) · line 2261
Quoted 10 lines
drivers/nvme/host/tcp.c +7
-0+return ret; + + ret =nvme_tcp_tls_check_psk (ctrl); +if (ret) +return ret; + + ret =__nvme_tcp_alloc_io_queues (ctrl);if (ret)return ret;
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
PATCH v3 3/3 nvme-tcp: parallelize I/O queue allocation and startup
In reply to Surabhi GogteSimilar to commit 2a8513091d2f ("nvme-rdma: parallelize I/O queue allocation and startup"), refactor nvme tcp I/O queue setup to use async API, combining allocation and startup into a single parallel operation per queue. This reduces connection and reconnection setup time when there are delays in establishing connections, which is especially important for high-core-count hosts.
Key changes:
- Use async API to facilitate parallel calls for io queue setup.
- Add nvme_tcp_queue_setup_ctx for propagating errors from async workers.
- Remove nvme_tcp_start_io_queues() and __nvme_tcp_alloc_io_queues(); their logic is folded into nvme_tcp_setup_io_queues() and nvme_tcp_configure_io_queues().
- Allocate the io tag set before the queues so that the queue range is known, and only set up the reconnect grow case if the queue count actually increased.
- Serialize the cpu scan and claim in nvme_tcp_set_queue_io_cpu() with a spinlock, as concurrent callers would otherwise select the same cpu. The per-cpu counters no longer need to be atomics.
Testing on a 64-core host with 64 IO-queues shows nvme-tcp connection time reduced from 61ms to 11ms.
--- drivers/nvme/host/tcp.c | 124 +++++++++++++++++++++++++--------------- 1 file changed, 79 insertions(+), 45 deletions(-)
drivers/nvme/host/tcp.c +79 -45
@@ -7,6 +7,7 @@#include <linux/module.h> #include <linux/init.h> #include <linux/slab.h> +#include <linux/async.h> #include <linux/err.h> #include <linux/crc32.h> #include <linux/nvme-tcp.h>
@@ -54,7 +55,8 @@ MODULE_PARM_DESC(tls_handshake_timeout, "nvme TLS handshake timeout in seconds (default 10)");#endif -static atomic_t nvme_tcp_cpu_queues[NR_CPUS ]; +static int nvme_tcp_cpu_queues[NR_CPUS ]; +static DEFINE_SPINLOCK (nvme_tcp_cpu_queues_lock);enum nvme_tcp_send_state {NVME_TCP_SEND_CMD_PDU =0 ,
@@ -154,6 +156,12 @@ struct nvme_tcp_queue {static DEFINE_MUTEX (nvme_tcp_ctrl_mutex); static LIST_HEAD_GUARDED(nvme_tcp_ctrl_list, nvme_tcp_ctrl_mutex); +struct nvme_tcp_queue_setup_ctx { +struct nvme_ctrl *ctrl; +int qid; +int *err; +}; +struct nvme_tcp_ctrl {/* read only in the hot path */ struct nvme_tcp_queue *queues;
@@ -1719,9 +1727,10 @@ static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) goto out;/* Search for the least used cpu from the mq_map */ +spin_lock (&nvme_tcp_cpu_queues_lock); io_cpu =WORK_CPU_UNBOUND ;for_each_online_cpu (cpu) { -int num_queues =atomic_read (&nvme_tcp_cpu_queues[cpu]); +int num_queues = nvme_tcp_cpu_queues[cpu];if (mq_map[cpu] != qid)continue ;
@@ -1732,9 +1741,10 @@ static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) }if (io_cpu !=WORK_CPU_UNBOUND ) { queue->io_cpu = io_cpu; -atomic_inc (&nvme_tcp_cpu_queues[io_cpu]); + nvme_tcp_cpu_queues[io_cpu]++;set_bit (NVME_TCP_Q_IO_CPU_SET , &queue->flags); } +spin_unlock (&nvme_tcp_cpu_queues_lock); out:dev_dbg (ctrl->ctrl.device,"queue %d: using cpu %d\n" , qid, queue->io_cpu);
@@ -2011,8 +2021,11 @@ static void nvme_tcp_stop_queue_nowait(struct nvme_ctrl *nctrl, int qid)if (!test_bit (NVME_TCP_Q_ALLOCATED , &queue->flags))return ; -if (test_and_clear_bit (NVME_TCP_Q_IO_CPU_SET , &queue->flags)) -atomic_dec (&nvme_tcp_cpu_queues[queue->io_cpu]); +if (test_and_clear_bit (NVME_TCP_Q_IO_CPU_SET , &queue->flags)) { +spin_lock (&nvme_tcp_cpu_queues_lock); + nvme_tcp_cpu_queues[queue->io_cpu]--; +spin_unlock (&nvme_tcp_cpu_queues_lock); + }mutex_lock (&queue->queue_lock);if (test_and_clear_bit (NVME_TCP_Q_LIVE , &queue->flags))
@@ -2119,25 +2132,6 @@ static void nvme_tcp_stop_io_queues(struct nvme_ctrl *ctrl)nvme_tcp_wait_queue (ctrl, i); } -static int nvme_tcp_start_io_queues (struct nvme_ctrl *ctrl, -int first,int last) -{ -int i, ret; - -for (i = first; i < last; i++) { - ret =nvme_tcp_start_queue (ctrl, i); -if (ret) - goto out_stop_queues; - } - -return 0 ; - -out_stop_queues: -for (i--; i >= first; i--) -nvme_tcp_stop_queue (ctrl, i); -return ret; -} -static int nvme_tcp_alloc_admin_queue (struct nvme_ctrl *ctrl) {int ret;
@@ -2198,22 +2192,64 @@ static int nvme_tcp_tls_check_psk(struct nvme_ctrl *ctrl)return 0 ; } -static int __nvme_tcp_alloc_io_queues (struct nvme_ctrl *ctrl) +static void nvme_tcp_setup_queue_async (void *data,async_cookie_t cookie) { -int i, ret; +struct nvme_tcp_queue_setup_ctx *ctx = data; +struct nvme_ctrl *ctrl = ctx->ctrl; +int ret; -for (i =1 ; i < ctrl->queue_count; i++) { - ret =nvme_tcp_alloc_queue (ctrl, i, - ctrl->tls_pskid); -if (ret) - goto out_free_queues; + ret =nvme_tcp_alloc_queue (ctrl, ctx->qid, ctrl->tls_pskid); +if (ret) + goto out_err; + + ret =nvme_tcp_start_queue (ctrl, ctx->qid); +if (ret) + goto out_err; + +return ; + +out_err: +WRITE_ONCE (*ctx->err, ret); +} + +static int nvme_tcp_setup_io_queues (struct nvme_ctrl *ctrl,unsigned int first, +unsigned int last) +{ +ASYNC_DOMAIN_EXCLUSIVE (queue_domain); +struct nvme_tcp_queue_setup_ctx *ctxs; +int nr_queues = last - first; +int err =0 , i, ret; + + ctxs =kmalloc_objs (*ctxs, nr_queues); +if (!ctxs) +return -ENOMEM ; + +for (i =0 ; i < nr_queues; i++) { + ctxs[i].ctrl = ctrl; + ctxs[i].qid = first + i; + ctxs[i].err = &err; +async_schedule_domain (nvme_tcp_setup_queue_async, &ctxs[i], + &queue_domain); } +async_synchronize_full_domain (&queue_domain); +kfree (ctxs); + + ret =READ_ONCE (err); +if (ret) + goto out_free_queues; +return 0 ; out_free_queues: -for (i--; i >=1 ; i--) -nvme_tcp_free_queue (ctrl, i); +for (i = first; i < last; i++) { +struct nvme_tcp_queue *queue = &to_tcp_ctrl (ctrl)->queues[i]; + +if (test_bit (NVME_TCP_Q_LIVE , &queue->flags)) +nvme_tcp_stop_queue (ctrl, i); +if (test_bit (NVME_TCP_Q_ALLOCATED , &queue->flags)) +nvme_tcp_free_queue (ctrl, i); + }return ret; }
@@ -2255,10 +2291,6 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)if (ret)return ret; - ret =__nvme_tcp_alloc_io_queues (ctrl); -if (ret) -return ret; -if (new) { ret = nvme_alloc_io_tag_set(ctrl, &to_tcp_ctrl(ctrl)->tag_set, &nvme_tcp_mq_ops,
@@ -2269,12 +2301,12 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) }/* -* Only start IO queues for which we have allocated the tagset +* Only setup IO queues for which we have allocated the tagset * and limited it to the available queues. On reconnects, the * queue number might have changed. */ nr_queues =min (ctrl->tagset->nr_hw_queues +1 , ctrl->queue_count); - ret =nvme_tcp_start_io_queues (ctrl,1 , nr_queues); + ret =nvme_tcp_setup_io_queues (ctrl,1 , nr_queues);if (ret) goto out_cleanup_connect_q;
@@ -2298,12 +2330,14 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)/* * If the number of queues has increased (reconnect case) -* start all new queues now. +* setup all new queues now. */ - ret =nvme_tcp_start_io_queues (ctrl, nr_queues, - ctrl->tagset->nr_hw_queues +1 ); -if (ret) - goto out_wait_freeze_timed_out; +if (ctrl->tagset->nr_hw_queues +1 > nr_queues) { + ret =nvme_tcp_setup_io_queues (ctrl, nr_queues, + ctrl->tagset->nr_hw_queues +1 ); +if (ret) + goto out_wait_freeze_timed_out; + }return 0 ;
@@ -3145,7 +3179,7 @@ static int __init nvme_tcp_init_module(void)return -ENOMEM ;for_each_possible_cpu (cpu) -atomic_set (&nvme_tcp_cpu_queues[cpu],0 ); + nvme_tcp_cpu_queues[cpu] =0 ;nvmf_register_transport (&nvme_tcp_transport);return 0 ;
Signature · 1 line
2.55.0
Sagi Grimberg
maintainerRe: [PATCH v3 3/3] nvme-tcp: parallelize I/O queue allocation and startup
Quoted 235 lines · Surabhi Gogte
Similar to commit
2a8513091d2f("nvme-rdma: parallelize I/O queue allocation and startup"), refactor nvme tcp I/O queue setup to use async API, combining allocation and startup into a single parallel operation per queue. This reduces connection and reconnection setup time when there are delays in establishing connections, which is especially important for high-core-count hosts.Key changes:
- Use async API to facilitate parallel calls for io queue setup.
- Addnvme_tcp_queue_setup_ctxfor propagating errors from async workers.
- Removenvme_tcp_start_io_queues()and__nvme_tcp_alloc_io_queues(); their logic is folded intonvme_tcp_setup_io_queues()andnvme_tcp_configure_io_queues().
- Allocate the io tag set before the queues so that the queue range is known, and only set up the reconnect grow case if the queue count actually increased.
- Serialize the cpu scan and claim innvme_tcp_set_queue_io_cpu()with a spinlock, as concurrent callers would otherwise select the same cpu. The per-cpu counters no longer need to be atomics.Testing on a 64-core host with 64 IO-queues shows nvme-tcp connection time reduced from 61ms to 11ms.
Signed-off-by Surabhi Gogte <[email protected]>--- drivers/nvme/host/tcp.c | 124 +++++++++++++++++++++++++--------------- 1 file changed, 79 insertions(+), 45 deletions(-)drivers/nvme/host/tcp.c +72
-40@@ -7,6 +7,7 @@#include <linux/module.h> #include <linux/init.h> #include <linux/slab.h> +#include <linux/async.h> #include <linux/err.h> #include <linux/crc32.h> #include <linux/nvme-tcp.h> @@ -54,7 +55,8 @@ MODULE_PARM_DESC(tls_handshake_timeout, "nvme TLS handshake timeout in seconds (default 10)");#endif -static atomic_t nvme_tcp_cpu_queues[NR_CPUS ]; +static int nvme_tcp_cpu_queues[NR_CPUS ]; +static DEFINE_SPINLOCK (nvme_tcp_cpu_queues_lock);enum nvme_tcp_send_state {NVME_TCP_SEND_CMD_PDU =0 ,@@ -154,6 +156,12 @@ struct nvme_tcp_queue {static DEFINE_MUTEX (nvme_tcp_ctrl_mutex); static LIST_HEAD_GUARDED(nvme_tcp_ctrl_list, nvme_tcp_ctrl_mutex); +struct nvme_tcp_queue_setup_ctx { +struct nvme_ctrl *ctrl; +int qid; +int *err; +}; +struct nvme_tcp_ctrl {/* read only in the hot path */ struct nvme_tcp_queue *queues;@@ -1719,9 +1727,10 @@ static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) goto out;/* Search for the least used cpu from the mq_map */ +spin_lock (&nvme_tcp_cpu_queues_lock); io_cpu =WORK_CPU_UNBOUND ;for_each_online_cpu (cpu) { -int num_queues =atomic_read (&nvme_tcp_cpu_queues[cpu]); +int num_queues = nvme_tcp_cpu_queues[cpu];if (mq_map[cpu] != qid)continue ;@@ -1732,9 +1741,10 @@ static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) }if (io_cpu !=WORK_CPU_UNBOUND ) { queue->io_cpu = io_cpu; -atomic_inc (&nvme_tcp_cpu_queues[io_cpu]); + nvme_tcp_cpu_queues[io_cpu]++;set_bit (NVME_TCP_Q_IO_CPU_SET , &queue->flags); } +spin_unlock (&nvme_tcp_cpu_queues_lock); out:dev_dbg (ctrl->ctrl.device,"queue %d: using cpu %d\n" , qid, queue->io_cpu);@@ -2011,8 +2021,11 @@ static void nvme_tcp_stop_queue_nowait(struct nvme_ctrl *nctrl, int qid)if (!test_bit (NVME_TCP_Q_ALLOCATED , &queue->flags))return ; -if (test_and_clear_bit (NVME_TCP_Q_IO_CPU_SET , &queue->flags)) -atomic_dec (&nvme_tcp_cpu_queues[queue->io_cpu]); +if (test_and_clear_bit (NVME_TCP_Q_IO_CPU_SET , &queue->flags)) { +spin_lock (&nvme_tcp_cpu_queues_lock); + nvme_tcp_cpu_queues[queue->io_cpu]--; +spin_unlock (&nvme_tcp_cpu_queues_lock); + }mutex_lock (&queue->queue_lock);if (test_and_clear_bit (NVME_TCP_Q_LIVE , &queue->flags))@@ -2119,25 +2132,6 @@ static void nvme_tcp_stop_io_queues(struct nvme_ctrl *ctrl)nvme_tcp_wait_queue (ctrl, i); } -static int nvme_tcp_start_io_queues (struct nvme_ctrl *ctrl, -int first,int last) -{ -int i, ret; - -for (i = first; i < last; i++) { - ret =nvme_tcp_start_queue (ctrl, i); -if (ret) - goto out_stop_queues; - } - -return 0 ; - -out_stop_queues: -for (i--; i >= first; i--) -nvme_tcp_stop_queue (ctrl, i); -return ret; -} -static int nvme_tcp_alloc_admin_queue (struct nvme_ctrl *ctrl) {int ret;@@ -2198,22 +2192,64 @@ static int nvme_tcp_tls_check_psk(struct nvme_ctrl *ctrl)return 0 ; } -static int __nvme_tcp_alloc_io_queues (struct nvme_ctrl *ctrl) +static void nvme_tcp_setup_queue_async (void *data,async_cookie_t cookie) { -int i, ret; +struct nvme_tcp_queue_setup_ctx *ctx = data; +struct nvme_ctrl *ctrl = ctx->ctrl; +int ret; -for (i =1 ; i < ctrl->queue_count; i++) { - ret =nvme_tcp_alloc_queue (ctrl, i, - ctrl->tls_pskid); -if (ret) - goto out_free_queues; + ret =nvme_tcp_alloc_queue (ctrl, ctx->qid, ctrl->tls_pskid); +if (ret) + goto out_err; + + ret =nvme_tcp_start_queue (ctrl, ctx->qid); +if (ret) + goto out_err; + +return ; + +out_err: +WRITE_ONCE (*ctx->err, ret); +} + +static int nvme_tcp_setup_io_queues (struct nvme_ctrl *ctrl,unsigned int first, +unsigned int last) +{ +ASYNC_DOMAIN_EXCLUSIVE (queue_domain); +struct nvme_tcp_queue_setup_ctx *ctxs; +int nr_queues = last - first; +int err =0 , i, ret; + + ctxs =kmalloc_objs (*ctxs, nr_queues); +if (!ctxs) +return -ENOMEM ; + +for (i =0 ; i < nr_queues; i++) { + ctxs[i].ctrl = ctrl; + ctxs[i].qid = first + i; + ctxs[i].err = &err; +async_schedule_domain (nvme_tcp_setup_queue_async, &ctxs[i], + &queue_domain); } +async_synchronize_full_domain (&queue_domain); +kfree (ctxs); + + ret =READ_ONCE (err); +if (ret) + goto out_free_queues; +return 0 ; out_free_queues: -for (i--; i >=1 ; i--) -nvme_tcp_free_queue (ctrl, i); +for (i = first; i < last; i++) { +struct nvme_tcp_queue *queue = &to_tcp_ctrl (ctrl)->queues[i]; + +if (test_bit (NVME_TCP_Q_LIVE , &queue->flags)) +nvme_tcp_stop_queue (ctrl, i); +if (test_bit (NVME_TCP_Q_ALLOCATED , &queue->flags)) +nvme_tcp_free_queue (ctrl, i); + }return ret; }@@ -2255,10 +2291,6 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)if (ret)return ret; - ret =__nvme_tcp_alloc_io_queues (ctrl); -if (ret) -return ret; -if (new) { ret = nvme_alloc_io_tag_set(ctrl, &to_tcp_ctrl(ctrl)->tag_set, &nvme_tcp_mq_ops,@@ -2269,12 +2301,12 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) }/* -* Only start IO queues for which we have allocated the tagset +* Only setup IO queues for which we have allocated the tagset * and limited it to the available queues. On reconnects, the * queue number might have changed. */ nr_queues =min (ctrl->tagset->nr_hw_queues +1 , ctrl->queue_count); - ret =nvme_tcp_start_io_queues (ctrl,1 , nr_queues); + ret =nvme_tcp_setup_io_queues (ctrl,1 , nr_queues);if (ret) goto out_cleanup_connect_q;@@ -2298,12 +2330,14 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)/* * If the number of queues has increased (reconnect case) -* start all new queues now. +* setup all new queues now. */
drivers/nvme/host/tcp.c static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) · line 2340
Quoted 10 lines
drivers/nvme/host/tcp.c +6
-4- ret =nvme_tcp_start_io_queues (ctrl, nr_queues, - ctrl->tagset->nr_hw_queues +1 ); -if (ret) - goto out_wait_freeze_timed_out; +if (ctrl->tagset->nr_hw_queues +1 > nr_queues) { + ret =nvme_tcp_setup_io_queues (ctrl, nr_queues, + ctrl->tagset->nr_hw_queues +1 ); +if (ret) + goto out_wait_freeze_timed_out; + }
This if condition can probably move into nvme_tcp_setup_io_queues() and exit early if both are the same. Other than that, patch looks solid. Can you please also run blktests to see that nothing broke. There may be some tests that have subtle race conditions. rdma and tcp btw.
Re: [PATCH v3 3/3] nvme-tcp: parallelize I/O queue allocation and startup
Quoted 246 lines · Sagi Grimberg
Quoted 242 lines · Surabhi Gogte
Similar to commit
2a8513091d2f("nvme-rdma: parallelize I/O queue allocation and startup"), refactor nvme tcp I/O queue setup to use async API, combining allocation and startup into a single parallel operation per queue. This reduces connection and reconnection setup time when there are delays in establishing connections, which is especially important for high-core-count hosts.Key changes:
- Use async API to facilitate parallel calls for io queue setup.
- Addnvme_tcp_queue_setup_ctxfor propagating errors from async workers.
- Removenvme_tcp_start_io_queues()and__nvme_tcp_alloc_io_queues(); their logic is folded intonvme_tcp_setup_io_queues()andnvme_tcp_configure_io_queues().
- Allocate the io tag set before the queues so that the queue range is known, and only set up the reconnect grow case if the queue count actually increased.
- Serialize the cpu scan and claim innvme_tcp_set_queue_io_cpu()with a spinlock, as concurrent callers would otherwise select the same cpu. The per-cpu counters no longer need to be atomics.Testing on a 64-core host with 64 IO-queues shows nvme-tcp connection time reduced from 61ms to 11ms.
Signed-off-by Surabhi Gogte <[email protected]>--- drivers/nvme/host/tcp.c | 124 +++++++++++++++++++++++++--------------- 1 file changed, 79 insertions(+), 45 deletions(-)drivers/nvme/host/tcp.c +75
-44@@ -7,6 +7,7 @@#include <linux/module.h> #include <linux/init.h> #include <linux/slab.h> +#include <linux/async.h> #include <linux/err.h> #include <linux/crc32.h> #include <linux/nvme-tcp.h> @@ -54,7 +55,8 @@ MODULE_PARM_DESC(tls_handshake_timeout, "nvme TLS handshake timeout in seconds (default 10)");#endif -static atomic_t nvme_tcp_cpu_queues[NR_CPUS ]; +static int nvme_tcp_cpu_queues[NR_CPUS ]; +static DEFINE_SPINLOCK (nvme_tcp_cpu_queues_lock);enum nvme_tcp_send_state {NVME_TCP_SEND_CMD_PDU =0 ,@@ -154,6 +156,12 @@ struct nvme_tcp_queue {static DEFINE_MUTEX (nvme_tcp_ctrl_mutex); static LIST_HEAD_GUARDED(nvme_tcp_ctrl_list, nvme_tcp_ctrl_mutex); +struct nvme_tcp_queue_setup_ctx { +struct nvme_ctrl *ctrl; +int qid; +int *err; +}; +struct nvme_tcp_ctrl {/* read only in the hot path */ struct nvme_tcp_queue *queues;@@ -1719,9 +1727,10 @@ static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) goto out;/* Search for the least used cpu from the mq_map */ +spin_lock (&nvme_tcp_cpu_queues_lock); io_cpu =WORK_CPU_UNBOUND ;for_each_online_cpu (cpu) { -int num_queues =atomic_read (&nvme_tcp_cpu_queues[cpu]); +int num_queues = nvme_tcp_cpu_queues[cpu];if (mq_map[cpu] != qid)continue ;@@ -1732,9 +1741,10 @@ static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) }if (io_cpu !=WORK_CPU_UNBOUND ) { queue->io_cpu = io_cpu; -atomic_inc (&nvme_tcp_cpu_queues[io_cpu]); + nvme_tcp_cpu_queues[io_cpu]++;set_bit (NVME_TCP_Q_IO_CPU_SET , &queue->flags); } +spin_unlock (&nvme_tcp_cpu_queues_lock); out:dev_dbg (ctrl->ctrl.device,"queue %d: using cpu %d\n" , qid, queue->io_cpu);@@ -2011,8 +2021,11 @@ static void nvme_tcp_stop_queue_nowait(struct nvme_ctrl *nctrl, int qid)if (!test_bit (NVME_TCP_Q_ALLOCATED , &queue->flags))return ; -if (test_and_clear_bit (NVME_TCP_Q_IO_CPU_SET , &queue->flags)) -atomic_dec (&nvme_tcp_cpu_queues[queue->io_cpu]); +if (test_and_clear_bit (NVME_TCP_Q_IO_CPU_SET , &queue->flags)) { +spin_lock (&nvme_tcp_cpu_queues_lock); + nvme_tcp_cpu_queues[queue->io_cpu]--; +spin_unlock (&nvme_tcp_cpu_queues_lock); + }mutex_lock (&queue->queue_lock);if (test_and_clear_bit (NVME_TCP_Q_LIVE , &queue->flags))@@ -2119,25 +2132,6 @@ static void nvme_tcp_stop_io_queues(struct nvme_ctrl *ctrl)nvme_tcp_wait_queue (ctrl, i); } -static int nvme_tcp_start_io_queues (struct nvme_ctrl *ctrl, -int first,int last) -{ -int i, ret; - -for (i = first; i < last; i++) { - ret =nvme_tcp_start_queue (ctrl, i); -if (ret) - goto out_stop_queues; - } - -return 0 ; - -out_stop_queues: -for (i--; i >= first; i--) -nvme_tcp_stop_queue (ctrl, i); -return ret; -} -static int nvme_tcp_alloc_admin_queue (struct nvme_ctrl *ctrl) {int ret;@@ -2198,22 +2192,64 @@ static int nvme_tcp_tls_check_psk(struct nvme_ctrl *ctrl)return 0 ; } -static int __nvme_tcp_alloc_io_queues (struct nvme_ctrl *ctrl) +static void nvme_tcp_setup_queue_async (void *data,async_cookie_t cookie) { -int i, ret; +struct nvme_tcp_queue_setup_ctx *ctx = data; +struct nvme_ctrl *ctrl = ctx->ctrl; +int ret; -for (i =1 ; i < ctrl->queue_count; i++) { - ret =nvme_tcp_alloc_queue (ctrl, i, - ctrl->tls_pskid); -if (ret) - goto out_free_queues; + ret =nvme_tcp_alloc_queue (ctrl, ctx->qid, ctrl->tls_pskid); +if (ret) + goto out_err; + + ret =nvme_tcp_start_queue (ctrl, ctx->qid); +if (ret) + goto out_err; + +return ; + +out_err: +WRITE_ONCE (*ctx->err, ret); +} + +static int nvme_tcp_setup_io_queues (struct nvme_ctrl *ctrl,unsigned int first, +unsigned int last) +{ +ASYNC_DOMAIN_EXCLUSIVE (queue_domain); +struct nvme_tcp_queue_setup_ctx *ctxs; +int nr_queues = last - first; +int err =0 , i, ret; + + ctxs =kmalloc_objs (*ctxs, nr_queues); +if (!ctxs) +return -ENOMEM ; + +for (i =0 ; i < nr_queues; i++) { + ctxs[i].ctrl = ctrl; + ctxs[i].qid = first + i; + ctxs[i].err = &err; +async_schedule_domain (nvme_tcp_setup_queue_async, &ctxs[i], + &queue_domain); } +async_synchronize_full_domain (&queue_domain); +kfree (ctxs); + + ret =READ_ONCE (err); +if (ret) + goto out_free_queues; +return 0 ; out_free_queues: -for (i--; i >=1 ; i--) -nvme_tcp_free_queue (ctrl, i); +for (i = first; i < last; i++) { +struct nvme_tcp_queue *queue = &to_tcp_ctrl (ctrl)->queues[i]; + +if (test_bit (NVME_TCP_Q_LIVE , &queue->flags)) +nvme_tcp_stop_queue (ctrl, i); +if (test_bit (NVME_TCP_Q_ALLOCATED , &queue->flags)) +nvme_tcp_free_queue (ctrl, i); + }return ret; }@@ -2255,10 +2291,6 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)if (ret)return ret; - ret =__nvme_tcp_alloc_io_queues (ctrl); -if (ret) -return ret; -if (new) { ret = nvme_alloc_io_tag_set(ctrl, &to_tcp_ctrl(ctrl)->tag_set, &nvme_tcp_mq_ops,@@ -2269,12 +2301,12 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) }/* -* Only start IO queues for which we have allocated the tagset +* Only setup IO queues for which we have allocated the tagset * and limited it to the available queues. On reconnects, the * queue number might have changed. */ nr_queues =min (ctrl->tagset->nr_hw_queues +1 , ctrl->queue_count); - ret =nvme_tcp_start_io_queues (ctrl,1 , nr_queues); + ret =nvme_tcp_setup_io_queues (ctrl,1 , nr_queues);if (ret) goto out_cleanup_connect_q;@@ -2298,12 +2330,14 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)/* * If the number of queues has increased (reconnect case) -* start all new queues now. +* setup all new queues now. */ - ret =nvme_tcp_start_io_queues (ctrl, nr_queues, - ctrl->tagset->nr_hw_queues +1 ); -if (ret) - goto out_wait_freeze_timed_out; +if (ctrl->tagset->nr_hw_queues +1 > nr_queues) { + ret =nvme_tcp_setup_io_queues (ctrl, nr_queues, + ctrl->tagset->nr_hw_queues +1 );
Quoted 8 lines
drivers/nvme/host/tcp.c static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) · line 2340
Quoted 3 lines
drivers/nvme/host/tcp.c +3
-0+if (ret) + goto out_wait_freeze_timed_out; + }This if condition can probably move into
nvme_tcp_setup_io_queues()and exit early if both are the same. Other than that, patch looks solid. Can you please also run blktests to see that nothing broke. There may be some tests that have subtle race conditions. rdma and tcp btw.
Thanks. Sending a v4 version with this change. Also, I ran blktests for tcp and rdma. The blktests nvme/062 "Create TLS-encrypted connections" and nvme/063 "Create authenticated TCP connections with secure concatenation" related to tls need ktls-utils-1.4.0 (c63c04b) (tlshd) linked against gnutls 3.8.5. rdma also passed fine.
Re: [PATCH v3 3/3] nvme-tcp: parallelize I/O queue allocation and startup
In reply to Surabhi GogteQuoted 21 lines · Surabhi Gogte
Similar to commit
2a8513091d2f("nvme-rdma: parallelize I/O queue allocation and startup"), refactor nvme tcp I/O queue setup to use async API, combining allocation and startup into a single parallel operation per queue. This reduces connection and reconnection setup time when there are delays in establishing connections, which is especially important for high-core-count hosts.Key changes:
- Use async API to facilitate parallel calls for io queue setup.
- Addnvme_tcp_queue_setup_ctxfor propagating errors from async workers.
- Removenvme_tcp_start_io_queues()and__nvme_tcp_alloc_io_queues(); their logic is folded intonvme_tcp_setup_io_queues()andnvme_tcp_configure_io_queues().
- Allocate the io tag set before the queues so that the queue range is known, and only set up the reconnect grow case if the queue count actually increased.
- Serialize the cpu scan and claim innvme_tcp_set_queue_io_cpu()with a spinlock, as concurrent callers would otherwise select the same cpu. The per-cpu counters no longer need to be atomics.
Quoted 8 lines
Testing on a 64-core host with 64 IO-queues shows nvme-tcp connection time reduced from 61ms to 11ms.
Signed-off-by Surabhi Gogte <[email protected]>--- drivers/nvme/host/tcp.c | 124 +++++++++++++++++++++++++--------------- 1 file changed, 79 insertions(+), 45 deletions(-)
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich