Skip to content

nvme-tcp: NIC topology aware I/O queue scaling and queue info export

Nilay Shroff linux-nvme 3 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v3 Latest2 patches 7 repliesIn review
  2. v24 patches 0 repliesSuperseded
  3. v13 patches 9 repliesSuperseded

Changes in v3 from the cover letter

- Dropped patch #1 and #2 from v2
- Added new debugfs attribute parameter "is_visible" which could be useful to hide the attribute for certain configuration or transport.
Link to v2: https://lore.kernel.org/all/[email protected]/

Patches in v3 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/2nvme: add debugfs helpers for NVMe driversdrivers/nvme/host/Makefile drivers/nvme/host/debugfs.c drivers/nvme/host/nvme.h+144 -10 replies
2/2nvme: expose queue information via debugfsdrivers/nvme/host/core.c drivers/nvme/host/debugfs.c drivers/nvme/host/nvme.h +1 more+134 -10 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v3 yet.

Read the v3 thread

Cover letter [PATCH v3 0/2] nvme-tcp: NIC topology aware I/O queue scaling and queue info export

This patch series has been revised based on feedback from the netdev community [1].

Following the feedback, the logic for determining the number of fabric I/O queues (nr-io-queues) has been moved to userspace in nvme-cli [2]. As a result, the first two patches from the previous series, which implemented this logic in the kernel driver, have been dropped. This series contains the remaining two patches, which export NVMe/TCP I/O queue information through debugfs to facilitate host-side network tuning.

The first patch introduces debugfs helper functions that can be used by drivers to export debugfs attributes.

The second patch exports NVMe/TCP I/O queue and TCP flow information through debugfs. For each I/O queue, it provides:
- queue id (qid)
- CPU affinity
- TCP flow (src/dst IP and ports)

This enables userspace tools to configure:
- IRQ affinity
- RPS/XPS
- ntuple steering
- or any other scaling as deemed feasible

Together with tunning, these changes allow better alignment of:
flow -> NIC queue -> IRQ -> CPU -> NVMe/TCP I/O worker

Improving this alignment can enhance flow-to-CPU locality, potentially improving performance and reducing I/O latency. Performance evaluation results comparing the baseline with the tuned configuration are available in [3].

Thanks!

[1]: https://lore.kernel.org/all/[email protected]/
[2]: https://github.com/linux-nvme/nvme-cli/pull/4140
[3]: https://lore.kernel.org/all/[email protected]/

Changes from v2
- Dropped patch #1 and #2 from v2
- Added new debugfs attribute parameter "is_visible" which could be useful to hide the attribute for certain configuration or transport.
Link to v2: https://lore.kernel.org/all/[email protected]/

Nilay Shroff (2):
nvme: add debugfs helpers for NVMe drivers
nvme: expose queue information via debugfs

 drivers/nvme/host/Makefile  |   2 +-
 drivers/nvme/host/core.c    |   3 +
 drivers/nvme/host/debugfs.c | 207 ++++++++++++++++++++++++++++++++++++
 drivers/nvme/host/nvme.h    |  12 +++
 drivers/nvme/host/tcp.c     |  54 ++++++++++
 5 files changed, 277 insertions(+), 1 deletion(-)
 create mode 100644 drivers/nvme/host/debugfs.c
Signature · 1 line
2.53.0