nvme-tcp: parallelize I/O setup queue in connect
Status from this message: Thanks, applied to nvme-7.4.
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v4 Latest3 patches 11 repliesApplied
- v33 patches 7 repliesSuperseded
- v22 patches 13 repliesSuperseded
Earlier versions · 1
| Version | Posted | Patches | Replies | Status |
|---|---|---|---|---|
| v1 | 1 patch | 0 replies | Superseded |
Changes in v4 from the cover letter
- Moved conditional from nvme_tcp_configure_io_queues() to
Patches in v4 Applied
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/3 | nvme-tcp: store and use the caller's network namespacedrivers/ | +6 | 0 replies | |
| 2/3 | nvme-tcp: refactor I/O queue setup pathdrivers/ | +19 | 0 replies | |
| 3/3 | nvme-tcp: parallelize I/O queue allocation and startupdrivers/ | +77 | 10 replies |
Reviews trailers per patch
All 3 patches have a Reviewed-by or Acked-by.
| Patch | Hannes Reinecke | Sagi Grimberg | Surabhi Gogte |
|---|---|---|---|
| 1/3 | R-b | R-b | |
| 2/3 | R-b | R-b | |
| 3/3 | R-b | T-b |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 3 reviewers.
Cover letter [PATCH v4 0/3] nvme-tcp: parallelize I/O setup queue in connect
Similar to commit 2a8513091d2f ("nvme-rdma: parallelize I/O queue allocation and startup"), I/O queue allocation and start steps (including authentication) can be parallelized for nvme-tcp as well.
Patch 1 fixes a pre-existing bug: nvme_tcp_alloc_queue() used current->nsproxy->net_ns to create sockets during initial connect while it used kernel's init_net for reconnect and error recovery paths. The fix captures the caller's network namespace at controller allocation time.
Patch 2 is a preparatory refactor: it splits the queue-count negotiation and TLS PSK checks into helpers so that the individual steps can be called directly from nvme_tcp_configure_io_queues(). No functional change.
Patch 3 has the async implementation: it fans the per-queue alloc and start out over an async domain, serializes the per-cpu queue-count in nvme_tcp_set_queue_io_cpu() that can now run concurrently, and uses ctrl->net for all socket creation.
Testing on a 64-core host with 64 IO-queues shows nvme-tcp connection time reduced from 61ms to 11ms.
blktests - nvme suite, TCP transport (nvme_trtype=tcp)
Tested with:
- Ubuntu 22.04 LTS (Jammy Jellyfish), x86_64, KubeVirt VM
- nvme-cli v2.13 / libnvme v1.14 (built with libkeyutils)
- ktls-utils tag ktls-utils-1.4.0 (c63c04b) (tlshd) linked against gnutls 3.8.5
---
v4:
- Moved conditional from nvme_tcp_configure_io_queues() to
nvme_tcp_setup_io_queues().
v3:
- Added patch 1 fixing a network namespace bug: reconnect and error recovery paths were creating sockets in init_net instead of the controller's namespace. Patch 3 now uses ctrl->net (from patch 1) instead of hardcoding init_net.
- Patches renumbered accordingly (v2 patch 1->2, patch 2->3).
v2:
- Resending the entire series to fix missing numbering in the original patch 2 subject line. No functional code changes.
v1: https://lore.kernel.org/[email protected]/
v2: https://lore.kernel.org/[email protected]/
v3: https://lore.kernel.org/[email protected]/
---
Surabhi Gogte (3):
nvme-tcp: store and use the caller's network namespace
nvme-tcp: refactor I/O queue setup path
nvme-tcp: parallelize I/O queue allocation and startup
drivers/nvme/host/tcp.c | 139 +++++++++++++++++++++++++++------------- 1 file changed, 96 insertions(+), 43 deletions(-)
Signature · 1 line
2.55.0