nvme-tcp: NIC topology aware I/O queue scaling and queue info export
Subsystems (3)
v2 cover nvme-tcp: NIC topology aware I/O queue scaling and queue info export
Hi,
This series has been updated based on the feedback received during LSFMM. The changelog is updated accordingly.
The NVMe/TCP host driver currently provisions I/O queues primarily based on CPU availability rather than the capabilities and topology of the underlying network interface.
On modern systems with many CPUs but fewer NIC hardware queues, this can lead to multiple NVMe/TCP I/O workers contending for the same TX/RX queue, resulting in increased lock contention, cacheline bouncing, and degraded throughput.
This RFC proposes a set of changes to better align NVMe/TCP I/O queues with NIC queue resources, and to expose queue/flow information to enable more effective system-level tuning.
Key ideas
---------
1. Scale NVMe/TCP I/O queues based on NIC queue count
Instead of relying solely on CPU count, limit the number of I/O workers to:
min(num_online_cpus, netdev->real_num_{tx,rx}_queues)2. Improve CPU locality
Align NVMe/TCP I/O workers with CPUs associated with NIC IRQ affinity to reduce cross-CPU traffic and improve cache locality.
3. Expose queue and flow information via debugfs
Export per-I/O queue information including:
- queue id (qid)
- CPU affinity
- TCP flow (src/dst IP and ports)
This enables userspace tools to configure:
- IRQ affinity
- RPS/XPS
- ntuple steering
- or any other scaling as deemed feasible
4. Provide infrastructure for extensible debugfs support in NVMe
Together, these changes allow better alignment of:
flow -> NIC queue -> IRQ -> CPU -> NVMe/TCP I/O worker
Performance Evaluation
----------------------
Tests were conducted using fio over NVMe/TCP with the following parameters:
ioengine=io_uring
direct=1
bs=4k
numjobs=<#nic-queues>
iodepth=64
System:
CPUs: 72
NIC: 100G mlx5Two configurations were evaluated.
Scenario 1: NIC queues < CPU count
----------------------------------
- CPUs: 72
- NIC queues: 32
Baseline Patched Patched + tuning
randread 3141 MB/s 3228 MB/s 7509 MB/s
(767k IOPS) (788k IOPS) (1833k IOPS)randwrite 4510 MB/s 6172 MB/s 7518 MB/s
(1101k IOPS) (1507k IOPS) (1836k IOPS)randrw (read) 2156 MB/s 2560 MB/s 3932 MB/s
(526k IOPS) (625k IOPS) (960k IOPS)randrw (write) 2155 MB/s 2560 MB/s 3932 MB/s
(526k IOPS) (625k IOPS) (960k IOPS)Observation:
When CPU count exceeds NIC queue count, the baseline configuration suffers from queue contention. The proposed changes provide modest improvements on their own, and when combined with queue-aware tuning (IRQ affinity, ntuple steering, and CPU alignment), enable up to ~1.5x–2.5x throughput improvement.
Scenario 2: NIC queues == CPU count
-----------------------------------
- CPUs: 72
- NIC queues: 72
Baseline Patched + tuning
randread 4310 MB/s 7987 MB/s
(1052k IOPS) (1950k IOPS)randwrite 7947 MB/s 7972 MB/s
(1940k IOPS) (1946k IOPS)
randrw (read) 3583 MB/s 4030 MB/s
(875k IOPS) (984k IOPS)
randrw (write) 3583 MB/s 4029 MB/s
(875k IOPS) (984k IOPS)
Observation:
When NIC queues are already aligned with CPU count, the baseline performs well. The proposed changes maintain write performance (no regression) and still improve read and mixed workloads due to better flow-to-CPU locality.
Notes on tuning
---------------
The "patched + tuning" configuration includes:
- aligning NVMe/TCP I/O workers with NIC queue count
- IRQ affinity configuration per RX queue
- ntuple-based flow steering
- CPU/queue affinity alignment
These tuning steps are enabled by the queue/flow information exposed through this patchset.
As usual, feedback/comment/suggestions are most welcome!
Changes from v1:
- remove the "match-hw-queues" fabric option; always limit the number of NVMe/TCP I/O queues to min(num_online_cpus, num_nic_queues)
- drop the diagnostic patch reporting NIC queue underutilization
- split the netdev helper into a separate patch
- move the netdev helper implementation to net/core/dev.c
Link to v1: https://lore.kernel.org/all/[email protected]/
Nilay Shroff (4):
net: add helper for device lookup by destination address
nvme-tcp: limit I/O queue count based on NIC queue count
nvme: add debugfs helpers for NVMe drivers
nvme: expose queue information via debugfs
drivers/nvme/host/Makefile | 2 +- drivers/nvme/host/core.c | 3 + drivers/nvme/host/debugfs.c | 190 ++++++++++++++++++++++++++++++++++++ drivers/nvme/host/nvme.h | 12 +++ drivers/nvme/host/tcp.c | 120 ++++++++++++++++++++++- include/linux/netdevice.h | 5 + net/core/dev.c | 83 ++++++++++++++++ 7 files changed, 413 insertions(+), 2 deletions(-) create mode 100644 drivers/nvme/host/debugfs.c
Signature · 1 line
2.53.0
PATCH v2 1/4 net: add helper for device lookup by destination address
Add netdev_get_by_addr(), a helper that looks up the routing table to retrieve the network device associated with a destination address. The helper also supports netdev reference tracking.
--- include/linux/netdevice.h | 5 +++ net/core/dev.c | 84 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 89 insertions(+)
include/linux/netdevice.h +5 -0
@@ -3487,6 +3487,11 @@ struct net_device *netdev_get_by_index(struct net *net, int ifindex,struct net_device *netdev_get_by_index_lock (struct net *net,int ifindex);struct net_device *netdev_get_by_name (struct net *net,const char *name,netdevice_tracker *tracker,gfp_t gfp); +struct net_device *netdev_get_by_addr (struct net *net, +struct sockaddr_storage *src, +struct sockaddr_storage *dest, +netdevice_tracker *tracker, +gfp_t gfp);struct net_device *netdev_get_by_flags_rcu (struct net *net,netdevice_tracker *tracker,unsigned short flags,unsigned short mask);struct net_device *dev_get_by_index_rcu (struct net *net,int ifindex);
net/core/dev.c +84 -0
96 lines not on this page · raw message
Signature · 1 line
2.53.0
Jakub Kicinski
maintainerRe: [PATCH v2 1/4] net: add helper for device lookup by destination address
Quoted 3 lines · On Mon, 27 Jul 2026 20:46:44 +0530 Nilay Shroff wrote:
Add
netdev_get_by_addr(), a helper that looks up the routing table to retrieve the network device associated with a destination address. The helper also supports netdev reference tracking.
This does not explain why you're doing this.
Please CC the whole series to netdev or write better commit messages.
Re: [PATCH v2 1/4] net: add helper for device lookup by destination address
Quoted 7 lines · Jakub Kicinski
Quoted 3 lines · On Mon, 27 Jul 2026 20:46:44 +0530 Nilay Shroff wrote:
Add
netdev_get_by_addr(), a helper that looks up the routing table to retrieve the network device associated with a destination address. The helper also supports netdev reference tracking.This does not explain why you're doing this.
Please CC the whole series to netdev or write better commit messages.
Yeah sure my bad, I would resend the series including all netdev folks in Cc.
Thanks,
--Nilay
PATCH v2 2/4 nvme-tcp: limit I/O queue count based on NIC queue count
In reply to Nilay ShroffNVMe-TCP currently provisions I/O queues based primarily on the number of online CPUs. On systems where the CPU count significantly exceeds the number of NIC hardware queues, multiple NVMe-TCP I/O queues end up sharing the same NIC TX/RX queues. This increases lock contention, cacheline bouncing, and inter-processor interrupts (IPIs), reducing I/O efficiency.
Limit the number of NVMe-TCP default I/O queues to the smaller of the number of online CPUs and the number of NIC hardware queues. Aligning the number of NVMe-TCP I/O queues with the NIC queue topology reduces queue sharing, improves locality, and can improve throughput while reducing tail latency.
The number of NVMe-TCP I/O queues is now limited to:
min(num_online_cpus, num_nic_queues)
--- drivers/nvme/host/tcp.c | 61 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 61 insertions(+)
drivers/nvme/host/tcp.c +61 -0
@@ -1774,6 +1774,50 @@ static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl,return ret; } +static struct net_device *nvme_tcp_get_netdev (struct nvme_ctrl *ctrl, +netdevice_tracker *tracker,gfp_t gfp) +{ +struct net_device *dev =NULL ; + +if (ctrl->opts->mask &NVMF_OPT_HOST_IFACE ) + dev =netdev_get_by_name (&init_net, ctrl->opts->host_iface, + tracker, gfp); +else { +struct nvme_tcp_ctrl *tctrl =to_tcp_ctrl (ctrl); +struct sockaddr_storage *src =NULL , *dest =NULL ; + +if (ctrl->opts->mask &NVMF_OPT_HOST_TRADDR ) + src = &tctrl->src_addr; + + dest = &tctrl->addr; + + dev =netdev_get_by_addr (&init_net, src, dest, tracker, gfp); + } +return dev; +} + +/* +* Returns number of active NIC queues (min of TX/RX), or 0 if device cannot +* be determined. +*/ +static int nvme_tcp_get_netdev_current_queue_count (struct nvme_ctrl *ctrl) +{ +struct net_device *dev; +int tx_queues, rx_queues; +netdevice_tracker tracker; + + dev =nvme_tcp_get_netdev (ctrl, &tracker,GFP_KERNEL ); +if (!dev) +return 0 ; + + tx_queues = dev->real_num_tx_queues; + rx_queues = dev->real_num_rx_queues; + +netdev_put (dev, &tracker); + +return min (tx_queues, rx_queues); +} +static int nvme_tcp_alloc_queue (struct nvme_ctrl *nctrl,int qid,key_serial_t pskid) {
@@ -2165,6 +2209,23 @@ static int nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl)unsigned int nr_io_queues;int ret; +if (!(ctrl->opts->mask &NVMF_OPT_NR_IO_QUEUES )) { +int nr_hw_queues; + + nr_hw_queues =nvme_tcp_get_netdev_current_queue_count (ctrl); +if (nr_hw_queues <=0 ) + goto init_queue; + + ctrl->opts->nr_io_queues =min (nr_hw_queues,num_online_cpus ()); + +if (ctrl->opts->nr_io_queues <num_online_cpus ()) +dev_info (ctrl->device, +"limiting I/O queues to %u (NIC queues %d, CPUs %u)\n" , + ctrl->opts->nr_io_queues, nr_hw_queues, +num_online_cpus ()); + } + +init_queue: nr_io_queues =nvmf_nr_io_queues (ctrl->opts); ret =nvme_set_queue_count (ctrl, &nr_io_queues);if (ret)
Signature · 1 line
2.53.0
kernel test robot
build botkernel test robot noticed "blktests.nvme/048.fail" on:
Hello,
kernel test robot noticed "blktests.nvme/048.fail" on:
commit: 657b4cd151e042afc228615869b2a6876b91b8fc ("[PATCH v2 2/4] nvme-tcp: limit I/O queue count based on NIC queue count")
url: https://github.com/intel-lab-lkp/linux/commits/Nilay-Shroff/net-add-helper-for-device-lookup-by-destination-address/20260731-154725
patch link: https://lore.kernel.org/all/[email protected]/
patch subject: [PATCH v2 2/4] nvme-tcp: limit I/O queue count based on NIC queue count
in testcase: blktests
version: blktests-x86_64-72a6349-1_20260804
with following parameters:
disk: 1SSD test: nvme-group-02 nvme_trtype: tcp
config: x86_64-rhel-9.4-func
compiler: gcc-14
test machine: 16 threads Intel(R) Core(TM) i7-13620H (Raptor Lake) with 32G memory
(please refer to attached dmesg/kmsg for entire log/backtrace)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]> | Closes: | https://lore.kernel.org/oe-lkp/[email protected]
nvme/048 (tr=tcp) (Test queue count changes on reconnect)
runtime 4.358s ...
nvme/048 (tr=tcp) (Test queue count changes on reconnect) [failed]
runtime 4.358s ... 7.340s
--- tests/nvme/048.out 2026-08-04 23:49:51.000000000 +0000
+++ /mnt/nvme-group-02/nodev_tr_tcp/nvme/048.out.bad 2026-08-06 06:04:55.281322408 +0000
@@ -1,2 +1,4 @@
Running nvme/048
+expected queue count 3 not set
+FAIL
Test completeThe kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20260811/[email protected]
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
Re:
Quoted 36 lines · kernel test robot
Hello,
kernel test robot noticed "blktests.nvme/048.fail" on:
commit:
657b4cd151e042afc228615869b2a6876b91b8fc("[PATCH v2 2/4] nvme-tcp: limit I/O queue count based on NIC queue count")
url: https://github.com/intel-lab-lkp/linux/commits/Nilay-Shroff/net-add-helper-for-device-lookup-by-destination-address/20260731-154725
patch link: https://lore.kernel.org/all/[email protected]/
patch subject: [PATCH v2 2/4] nvme-tcp: limit I/O queue count based on NIC queue countin testcase: blktests
version: blktests-x86_64-72a6349-1_20260804
with following parameters:disk: 1SSD test: nvme-group-02 nvme_trtype: tcpconfig: x86_64-rhel-9.4-func
compiler: gcc-14
test machine: 16 threads Intel(R) Core(TM) i7-13620H (Raptor Lake) with 32G memory(please refer to attached dmesg/kmsg for entire log/backtrace)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]> | Closes: | https://lore.kernel.org/oe-lkp/[email protected]nvme/048 (tr=tcp) (Test queue count changes on reconnect)
Quoted 11 lines
runtime 4.358s ...
nvme/048 (tr=tcp) (Test queue count changes on reconnect) [failed]
runtime 4.358s ... 7.340s
--- tests/nvme/048.out 2026-08-04 23:49:51.000000000 +0000
+++ /mnt/nvme-group-02/nodev_tr_tcp/nvme/048.out.bad 2026-08-06 06:04:55.281322408 +0000
@@ -1,2 +1,4 @@
Running nvme/048
+expected queue count 3 not set
+FAIL
Test completeThanks for the report!
This requires changes in blktests nvme/048. So once this change/commit is merged upstream, I'd send another patch to update the test.
On another note, you may want to check your automation script which sends auto-email. I see the email subject line is empty.
Thanks,
--Nilay
Re:
Quoted 48 lines · Nilay Shroff
Quoted 36 lines · kernel test robot
Hello,
kernel test robot noticed "blktests.nvme/048.fail" on:
commit:
657b4cd151e042afc228615869b2a6876b91b8fc("[PATCH v2 2/4] nvme-tcp: limit I/O queue count based on NIC queue count")
url: https://github.com/intel-lab-lkp/linux/commits/Nilay-Shroff/net-add-helper-for-device-lookup-by-destination-address/20260731-154725
patch link: https://lore.kernel.org/all/[email protected]/
patch subject: [PATCH v2 2/4] nvme-tcp: limit I/O queue count based on NIC queue countin testcase: blktests
version: blktests-x86_64-72a6349-1_20260804
with following parameters:disk: 1SSD test: nvme-group-02 nvme_trtype: tcpconfig: x86_64-rhel-9.4-func
compiler: gcc-14
test machine: 16 threads Intel(R) Core(TM) i7-13620H (Raptor Lake) with 32G memory(please refer to attached dmesg/kmsg for entire log/backtrace)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]> | Closes: | https://lore.kernel.org/oe-lkp/[email protected]nvme/048 (tr=tcp) (Test queue count changes on reconnect)
Quoted 11 lines
runtime 4.358s ... nvme/048 (tr=tcp) (Test queue count changes on reconnect) [failed] runtime 4.358s ... 7.340s --- tests/nvme/048.out 2026-08-04 23:49:51.000000000 +0000 +++ /mnt/nvme-group-02/nodev_tr_tcp/nvme/048.out.bad 2026-08-06 06:04:55.281322408 +0000 @@ -1,2 +1,4 @@ Running nvme/048 +expected queue count 3 not set +FAIL Test complete
Quoted 8 lines
Thanks for the report!
This requires changes in blktests nvme/048. So once this change/commit is merged upstream, I'd send another patch to update the test.
On another note, you may want to check your automation script which sends auto-email. I see the email subject line is empty.
Thanks for the kind reminder. Will adjust it.
Quoted 2 lines
Thanks,
--Nilay
PATCH v2 3/4 nvme: add debugfs helpers for NVMe drivers
In reply to Nilay ShroffIntroduce helper APIs that allow NVMe drivers to register and unregister debugfs entries, along with a reusable attribute structure for defining new debugfs files.
The implementation uses seq_file interfaces to safely expose per- namespace or per-path statistics, while supporting both simple show callbacks and full seq_operations.
This will be used by subsequent patches to expose NVMe-TCP queue and flow information for tuning NVMe TCP I/O workqueue and network stack components.
--- drivers/nvme/host/Makefile | 2 +- drivers/nvme/host/debugfs.c | 129 ++++++++++++++++++++++++++++++++++++ drivers/nvme/host/nvme.h | 10 +++ 3 files changed, 140 insertions(+), 1 deletion(-) create mode 100644 drivers/nvme/host/debugfs.c
drivers/nvme/host/Makefile +1 -1
@@ -10,7 +10,7 @@ obj-$(CONFIG_NVME_FC) += nvme-fc.o obj-$(CONFIG_NVME_TCP) += nvme-tcp.o obj-$(CONFIG_NVME_APPLE) += nvme-apple.o -nvme-core-y += core.o ioctl.o sysfs.o pr.o +nvme-core-y += core.o ioctl.o sysfs.o pr.o debugfs.o nvme-core-$(CONFIG_NVME_VERBOSE_ERRORS) += constants.o nvme-core-$(CONFIG_TRACING) += trace.o nvme-core-$(CONFIG_NVME_MULTIPATH) += multipath.o
drivers/nvme/host/debugfs.c new file +129 -0
129 lines not on this page · raw message
drivers/nvme/host/nvme.h +10 -0
@@ -474,6 +474,16 @@ struct nvme_ctrl {u16 awupf;/* 0's based value. */ }; +void nvme_debugfs_register (struct gendisk *disk); +static inline void nvme_debugfs_unregister (struct gendisk *disk) +{ +/* +* Nothing to do for now. When the request queue is unregistered, +* all files under q->debugfs_dir are recursively deleted. +* This is just a placeholder; the compiler will optimize it out. +*/ +} +static inline enum nvme_ctrl_state nvme_ctrl_state (struct nvme_ctrl *ctrl) {return READ_ONCE (ctrl->state);
Signature · 1 line
2.53.0
PATCH v2 4/4 nvme: expose queue information via debugfs
In reply to Nilay ShroffAdd a new debugfs attribute "io_queue_info" to expose per-queue information for NVMe controllers. For NVMe-TCP, this includes the CPU handling each I/O queue and the associated TCP flow (source and destination address/port).
This information can be useful for understanding and tuning the interaction between NVMe-TCP I/O queues and network stack components, such as IRQ affinity, RPS/RFS, XPS, or NIC flow steering (ntuple).
The data is exported using seq_file interfaces to allow iteration over all controller queues.
--- drivers/nvme/host/core.c | 3 ++ drivers/nvme/host/debugfs.c | 64 ++++++++++++++++++++++++++++++++++++- drivers/nvme/host/nvme.h | 2 ++ drivers/nvme/host/tcp.c | 54 +++++++++++++++++++++++++++++++ 4 files changed, 122 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +3 -0
@@ -4264,6 +4264,8 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)if (device_add_disk (ctrl->device, ns->disk, nvme_ns_attr_groups)) goto out_cleanup_ns_from_list; +nvme_debugfs_register (ns->disk); +if (!nvme_ns_head_multipath (ns->head))nvme_add_ns_cdev (ns);
@@ -4344,6 +4346,7 @@ static void nvme_ns_remove(struct nvme_ns *ns)nvme_mpath_remove_sysfs_link (ns); +nvme_debugfs_unregister (ns->disk);del_gendisk (ns->disk);mutex_lock (&ns->ctrl->namespaces_lock);
drivers/nvme/host/debugfs.c +63 -1
76 lines not on this page · raw message
drivers/nvme/host/nvme.h +2 -0
@@ -663,6 +663,8 @@ struct nvme_ctrl_ops {void (*print_device_info)(struct nvme_ctrl *ctrl);bool (*supports_pci_p2pdma)(struct nvme_ctrl *ctrl);unsigned long (*get_virt_boundary)(struct nvme_ctrl *ctrl,bool is_admin); +int (*print_io_queue_info)(struct seq_file *m,struct nvme_ctrl *ctrl, +int qid); };/*
drivers/nvme/host/tcp.c +54 -0
66 lines not on this page · raw message
Signature · 1 line
2.53.0