Skip to content

nvme-tcp: parallelize I/O setup queue in connect

Surabhi Gogte linux-nvme lkml 4 versions – Applied

Status from this message: Thanks, applied to nvme-7.4.

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v4 Latest3 patches 11 repliesApplied
  2. v33 patches 7 repliesSuperseded
  3. v22 patches 13 repliesSuperseded
Earlier versions · 1
VersionPostedPatchesRepliesStatus
v11 patch0 repliesSuperseded

Changes in v4 from the cover letter

- Moved conditional from nvme_tcp_configure_io_queues() to

Patches in v4 Applied

#Subject and filesSizeRepliesApplied
1/3nvme-tcp: store and use the caller's network namespacedrivers/nvme/host/tcp.c+6 -10 replies
2/3nvme-tcp: refactor I/O queue setup pathdrivers/nvme/host/tcp.c+19 -60 replies
3/3nvme-tcp: parallelize I/O queue allocation and startupdrivers/nvme/host/tcp.c+77 -4210 replies

Reviews trailers per patch

All 3 patches have a Reviewed-by or Acked-by.

PatchHannes ReineckeSagi GrimbergSurabhi Gogte
1/3R-b R-b
2/3R-b R-b
3/3R-b T-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 3 reviewers.

Read the v4 thread

Cover letter [PATCH v4 0/3] nvme-tcp: parallelize I/O setup queue in connect

Similar to commit 2a8513091d2f ("nvme-rdma: parallelize I/O queue allocation and startup"), I/O queue allocation and start steps (including authentication) can be parallelized for nvme-tcp as well.

Patch 1 fixes a pre-existing bug: nvme_tcp_alloc_queue() used current->nsproxy->net_ns to create sockets during initial connect while it used kernel's init_net for reconnect and error recovery paths. The fix captures the caller's network namespace at controller allocation time.

Patch 2 is a preparatory refactor: it splits the queue-count negotiation and TLS PSK checks into helpers so that the individual steps can be called directly from nvme_tcp_configure_io_queues(). No functional change.

Patch 3 has the async implementation: it fans the per-queue alloc and start out over an async domain, serializes the per-cpu queue-count in nvme_tcp_set_queue_io_cpu() that can now run concurrently, and uses ctrl->net for all socket creation.

Testing on a 64-core host with 64 IO-queues shows nvme-tcp connection time reduced from 61ms to 11ms.

blktests - nvme suite, TCP transport (nvme_trtype=tcp)

Tested with:
- Ubuntu 22.04 LTS (Jammy Jellyfish), x86_64, KubeVirt VM
- nvme-cli v2.13 / libnvme v1.14 (built with libkeyutils)
- ktls-utils tag ktls-utils-1.4.0 (c63c04b) (tlshd) linked against gnutls 3.8.5

---
v4:
- Moved conditional from nvme_tcp_configure_io_queues() to

    nvme_tcp_setup_io_queues().

v3:
- Added patch 1 fixing a network namespace bug: reconnect and error recovery paths were creating sockets in init_net instead of the controller's namespace. Patch 3 now uses ctrl->net (from patch 1) instead of hardcoding init_net.
- Patches renumbered accordingly (v2 patch 1->2, patch 2->3).

v2:
- Resending the entire series to fix missing numbering in the original patch 2 subject line. No functional code changes.

v1: https://lore.kernel.org/[email protected]/
v2: https://lore.kernel.org/[email protected]/
v3: https://lore.kernel.org/[email protected]/
---

Surabhi Gogte (3):
nvme-tcp: store and use the caller's network namespace
nvme-tcp: refactor I/O queue setup path
nvme-tcp: parallelize I/O queue allocation and startup

 drivers/nvme/host/tcp.c | 139 +++++++++++++++++++++++++++-------------
 1 file changed, 96 insertions(+), 43 deletions(-)
Signature · 1 line
2.55.0