nvme-tcp: NIC topology aware I/O queue scaling and queue info export
cover nvme-tcp: NIC topology aware I/O queue scaling and queue info export
Hi,
The NVMe/TCP host driver currently provisions I/O queues primarily based on CPU availability rather than the capabilities and topology of the underlying network interface.
On modern systems with many CPUs but fewer NIC hardware queues, this can lead to multiple NVMe/TCP I/O workers contending for the same TX/RX queue, resulting in increased lock contention, cacheline bouncing, and degraded throughput.
This RFC proposes a set of changes to better align NVMe/TCP I/O queues with NIC queue resources, and to expose queue/flow information to enable more effective system-level tuning.
Key ideas
---------
1. Scale NVMe/TCP I/O queues based on NIC queue count
Instead of relying solely on CPU count, limit the number of I/O workers to:
min(num_online_cpus, netdev->real_num_{tx,rx}_queues)2. Improve CPU locality
Align NVMe/TCP I/O workers with CPUs associated with NIC IRQ affinity to reduce cross-CPU traffic and improve cache locality.
3. Expose queue and flow information via debugfs
Export per-I/O queue information including:
- queue id (qid)
- CPU affinity
- TCP flow (src/dst IP and ports)
This enables userspace tools to configure:
- IRQ affinity
- RPS/XPS
- ntuple steering
- or any other scaling as deemed feasible
4. Provide infrastructure for extensible debugfs support in NVMe
Together, these changes allow better alignment of:
flow -> NIC queue -> IRQ -> CPU -> NVMe/TCP I/O worker
Performance Evaluation
----------------------
Tests were conducted using fio over NVMe/TCP with the following parameters:
ioengine=io_uring
direct=1
bs=4k
numjobs=<#nic-queues>
iodepth=64
System:
CPUs: 72
NIC: 100G mlx5Two configurations were evaluated.
Scenario 1: NIC queues < CPU count
----------------------------------
- CPUs: 72
- NIC queues: 32
Baseline Patched Patched + tuning
randread 3141 MB/s 3228 MB/s 7509 MB/s
(767k IOPS) (788k IOPS) (1833k IOPS)randwrite 4510 MB/s 6172 MB/s 7518 MB/s
(1101k IOPS) (1507k IOPS) (1836k IOPS)randrw (read) 2156 MB/s 2560 MB/s 3932 MB/s
(526k IOPS) (625k IOPS) (960k IOPS)randrw (write) 2155 MB/s 2560 MB/s 3932 MB/s
(526k IOPS) (625k IOPS) (960k IOPS)Observation:
When CPU count exceeds NIC queue count, the baseline configuration suffers from queue contention. The proposed changes provide modest improvements on their own, and when combined with queue-aware tuning (IRQ affinity, ntuple steering, and CPU alignment), enable up to ~1.5x–2.5x throughput improvement.
Scenario 2: NIC queues == CPU count
-----------------------------------
- CPUs: 72
- NIC queues: 72
Baseline Patched + tuning
randread 4310 MB/s 7987 MB/s
(1052k IOPS) (1950k IOPS)randwrite 7947 MB/s 7972 MB/s
(1940k IOPS) (1946k IOPS)
randrw (read) 3583 MB/s 4030 MB/s
(875k IOPS) (984k IOPS)
randrw (write) 3583 MB/s 4029 MB/s
(875k IOPS) (984k IOPS)
Observation:
When NIC queues are already aligned with CPU count, the baseline performs well. The proposed changes maintain write performance (no regression) and still improve read and mixed workloads due to better flow-to-CPU locality.
Notes on tuning
---------------
The "patched + tuning" configuration includes:
- aligning NVMe/TCP I/O workers with NIC queue count
- IRQ affinity configuration per RX queue
- ntuple-based flow steering
- CPU/queue affinity alignment
These tuning steps are enabled by the queue/flow information exposed through this patchset.
Discussion
----------
This RFC aims to start discussion around:
- Whether NVMe/TCP queue scaling should consider NIC queue topology
- How best to expose queue/flow information to userspace
- The role of userspace vs kernel in steering decisions
As usual, feedback/comment/suggestions are most welcome!
Reference to LSF/MM/BPF abstarct: https://lore.kernel.org/all/[email protected]/
Nilay Shroff (4):
nvme-tcp: optionally limit I/O queue count based on NIC queues
nvme-tcp: add a diagnostic message when NIC queues are underutilized
nvme: add debugfs helpers for NVMe drivers
nvme: expose queue information via debugfs
drivers/nvme/host/Makefile | 2 +- drivers/nvme/host/core.c | 3 + drivers/nvme/host/debugfs.c | 162 +++++++++++++++++++++++++++ drivers/nvme/host/fabrics.c | 4 + drivers/nvme/host/fabrics.h | 3 + drivers/nvme/host/nvme.h | 12 ++ drivers/nvme/host/tcp.c | 211 +++++++++++++++++++++++++++++++++++- 7 files changed, 395 insertions(+), 2 deletions(-) create mode 100644 drivers/nvme/host/debugfs.c
Signature · 1 line
2.53.0
RFC PATCH 4/4 nvme: expose queue information via debugfs
Add a new debugfs attribute "io_queue_info" to expose per-queue information for NVMe controllers. For NVMe-TCP, this includes the CPU handling each I/O queue and the associated TCP flow (source and destination address/port).
This information can be useful for understanding and tuning the interaction between NVMe-TCP I/O queues and network stack components, such as IRQ affinity, RPS/RFS, XPS, or NIC flow steering (ntuple).
The data is exported using seq_file interfaces to allow iteration over all controller queues.
--- drivers/nvme/host/core.c | 3 +++ drivers/nvme/host/debugfs.c | 53 ++++++++++++++++++++++++++++++++++++- drivers/nvme/host/nvme.h | 2 ++ drivers/nvme/host/tcp.c | 52 ++++++++++++++++++++++++++++++++++++ 4 files changed, 109 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +3 -0
@@ -4207,6 +4207,8 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)if (device_add_disk (ctrl->device, ns->disk, nvme_ns_attr_groups)) goto out_cleanup_ns_from_list; +nvme_debugfs_register (ns->disk); +if (!nvme_ns_head_multipath (ns->head))nvme_add_ns_cdev (ns);
@@ -4285,6 +4287,7 @@ static void nvme_ns_remove(struct nvme_ns *ns)nvme_mpath_remove_sysfs_link (ns); +nvme_debugfs_unregister (ns->disk);del_gendisk (ns->disk);mutex_lock (&ns->ctrl->namespaces_lock);
drivers/nvme/host/debugfs.c +52 -1
65 lines not on this page · raw message
drivers/nvme/host/nvme.h +2 -0
@@ -650,6 +650,8 @@ struct nvme_ctrl_ops {void (*print_device_info)(struct nvme_ctrl *ctrl);bool (*supports_pci_p2pdma)(struct nvme_ctrl *ctrl);unsigned long (*get_virt_boundary)(struct nvme_ctrl *ctrl,bool is_admin); +int (*print_io_queue_info)(struct seq_file *m,struct nvme_ctrl *ctrl, +int qid); };/*
drivers/nvme/host/tcp.c +52 -0
64 lines not on this page · raw message
Signature · 1 line
2.53.0
RFC PATCH 1/4 nvme-tcp: optionally limit I/O queue count based on NIC queues
In reply to Nilay ShroffNVMe-TCP currently provisions I/O queues primarily based on CPU availability. On systems where the number of CPUs significantly exceeds the number of NIC hardware queues, this can lead to multiple I/O queues sharing the same NIC TX/RX queues, resulting in increased lock contention, cacheline bouncing, and inter-processor interrupts (IPIs).
In such configurations, limiting the number of NVMe-TCP I/O queues to the number of NIC hardware queues can improve performance by reducing contention and improving locality. Aligning NVMe-TCP worker threads with NIC queue topology may also help reduce tail latency.
Add a new transport option "match_hw_queues" to allow users to optionally limit the number of NVMe-TCP I/O queues to the number of NIC TX/RX queues. When enabled, the number of I/O queues is set to:
min(num_online_cpus, num_nic_queues)
This behavior is opt-in and does not change existing defaults.
--- drivers/nvme/host/fabrics.c | 4 ++ drivers/nvme/host/fabrics.h | 3 + drivers/nvme/host/tcp.c | 120 +++++++++++++++++++++++++++++++++++- 3 files changed, 126 insertions(+), 1 deletion(-)
drivers/nvme/host/fabrics.c +4 -0
@@ -709,6 +709,7 @@ static const match_table_t opt_tokens = { { NVMF_OPT_TLS, "tls" }, { NVMF_OPT_CONCAT, "concat" }, #endif + { NVMF_OPT_MATCH_HW_QUEUES, "match_hw_queues" }, { NVMF_OPT_ERR, NULL } };
@@ -1064,6 +1065,9 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts, } opts->concat = true;break ; +case NVMF_OPT_MATCH_HW_QUEUES : + opts->match_hw_queues = true; +break ;default :pr_warn ("unknown parameter or missing value '%s' in ctrl creation request\n" , p);
drivers/nvme/host/fabrics.h +3 -0
@@ -67,6 +67,7 @@ enum {NVMF_OPT_KEYRING =1 <<26 ,NVMF_OPT_TLS_KEY =1 <<27 ,NVMF_OPT_CONCAT =1 <<28 , + NVMF_OPT_MATCH_HW_QUEUES = 1 << 29, };/**
@@ -106,6 +107,7 @@ enum {* @disable_sqflow: disable controller sq flow control * @hdr_digest: generate/verify header digest (TCP) * @data_digest: generate/verify data digest (TCP) +* @match_hw_queues: limit controller IO queue count based on NIC queues (TCP) * @nr_write_queues: number of queues for write I/O * @nr_poll_queues: number of queues for polling I/O * @tos: type of service
@@ -136,6 +138,7 @@ struct nvmf_ctrl_options {bool disable_sqflow;bool hdr_digest;bool data_digest; +bool match_hw_queues;unsigned int nr_write_queues;unsigned int nr_poll_queues;int tos;
drivers/nvme/host/tcp.c +119 -1
144 lines not on this page · raw message
Signature · 1 line
2.53.0
RFC PATCH 2/4 nvme-tcp: add a diagnostic message when NIC queues are underutilized
In reply to Nilay ShroffSome systems may configure fewer NIC queues than supported by the hardware. When the number of NVMe-TCP I/O queues is limited by the number of active NIC queues, this can result in suboptimal performance.
Add a diagnostic message to warn when the configured NIC queue count is lower than the maximum supported queue count, as reported by the driver. This may help users identify configurations where increasing the NIC queue count could improve performance.
This change is informational only and does not modify NIC configuration.
--- drivers/nvme/host/tcp.c | 45 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 42 insertions(+), 3 deletions(-)
drivers/nvme/host/tcp.c +42 -3
@@ -11,6 +11,7 @@#include <linux/crc32.h> #include <linux/nvme-tcp.h> #include <linux/nvme-keyring.h> +#include <linux/ethtool.h> #include <net/sock.h> #include <net/tcp.h> #include <net/tls.h>
@@ -20,6 +21,7 @@#include <linux/in6.h> #include <linux/blk-mq.h> #include <net/busy_poll.h> +#include <net/netdev_lock.h> #include <trace/events/sock.h> #include "nvme.h"
@@ -1861,6 +1863,35 @@ static int nvme_tcp_get_netdev_current_queue_count(struct nvme_ctrl *ctrl)return min (tx_queues, rx_queues); } +static int nvme_tcp_get_netdev_max_queue_count (struct nvme_ctrl *ctrl) +{ +struct net_device *dev; +struct ethtool_channels channels = {0 }; +int max =0 ; + + dev =nvme_tcp_get_netdev (ctrl); +if (!dev) +return 0 ; + +rtnl_lock (); +if (!dev->ethtool_ops || !dev->ethtool_ops->get_channels) + goto out; + +netdev_lock_ops (dev); + + dev->ethtool_ops->get_channels (dev, &channels); +if (channels.max_combined) + max = channels.max_combined; +else + max =min (channels.max_rx, channels.max_tx); + +netdev_unlock_ops (dev); +out: +rtnl_unlock (); + +return max; +} +static int nvme_tcp_alloc_queue (struct nvme_ctrl *nctrl,int qid,key_serial_t pskid) {
@@ -2245,19 +2276,27 @@ static int nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl)if (!(ctrl->opts->mask &NVMF_OPT_NR_IO_QUEUES ) && (ctrl->opts->mask &NVMF_OPT_MATCH_HW_QUEUES )) { -int nr_hw_queues; +int nr_hw_queues, max_hw_queues; nr_hw_queues =nvme_tcp_get_netdev_current_queue_count (ctrl);if (nr_hw_queues <=0 ) goto init_queue; ctrl->opts->nr_io_queues =min (nr_hw_queues,num_online_cpus ()); - -if (ctrl->opts->nr_io_queues <num_online_cpus ()) +if (ctrl->opts->nr_io_queues <num_online_cpus ()) {dev_info (ctrl->device,"limiting I/O queues to %u (NIC queues %d, CPUs %u)\n" , ctrl->opts->nr_io_queues, nr_hw_queues,num_online_cpus ()); + + max_hw_queues = +nvme_tcp_get_netdev_max_queue_count (ctrl); +if (max_hw_queues > nr_hw_queues) +dev_info (ctrl->device, +"NIC supports %u queues but only %u are configured; " +"consider increasing queue count for better perfromance\n" , + max_hw_queues, nr_hw_queues); + } } init_queue:
Signature · 1 line
2.53.0
RFC PATCH 3/4 nvme: add debugfs helpers for NVMe drivers
In reply to Nilay ShroffIntroduce helper APIs that allow NVMe drivers to register and unregister debugfs entries, along with a reusable attribute structure for defining new debugfs files.
The implementation uses seq_file interfaces to safely expose per- namespace or per-path statistics, while supporting both simple show callbacks and full seq_operations.
This will be used by subsequent patches to expose NVMe-TCP queue and flow information for tuning NVMe TCP I/O workqueue and network stack components.
--- drivers/nvme/host/Makefile | 2 +- drivers/nvme/host/debugfs.c | 111 ++++++++++++++++++++++++++++++++++++ drivers/nvme/host/nvme.h | 10 ++++ 3 files changed, 122 insertions(+), 1 deletion(-) create mode 100644 drivers/nvme/host/debugfs.c
drivers/nvme/host/Makefile +1 -1
@@ -10,7 +10,7 @@ obj-$(CONFIG_NVME_FC) += nvme-fc.o obj-$(CONFIG_NVME_TCP) += nvme-tcp.o obj-$(CONFIG_NVME_APPLE) += nvme-apple.o -nvme-core-y += core.o ioctl.o sysfs.o pr.o +nvme-core-y += core.o ioctl.o sysfs.o pr.o debugfs.o nvme-core-$(CONFIG_NVME_VERBOSE_ERRORS) += constants.o nvme-core-$(CONFIG_TRACING) += trace.o nvme-core-$(CONFIG_NVME_MULTIPATH) += multipath.o
drivers/nvme/host/debugfs.c new file +111 -0
111 lines not on this page · raw message
drivers/nvme/host/nvme.h +10 -0
@@ -468,6 +468,16 @@ struct nvme_ctrl {u16 awupf;/* 0's based value. */ }; +void nvme_debugfs_register (struct gendisk *disk); +static inline void nvme_debugfs_unregister (struct gendisk *disk) +{ +/* +* Nothing to do for now. When the request queue is unregistered, +* all files under q->debugfs_dir are recursively deleted. +* This is just a placeholder; the compiler will optimize it out. +*/ +} +static inline enum nvme_ctrl_state nvme_ctrl_state (struct nvme_ctrl *ctrl) {return READ_ONCE (ctrl->state);
Signature · 1 line
2.53.0
Quoted 125 lines · Nilay Shroff
Hi,
The NVMe/TCP host driver currently provisions I/O queues primarily based on CPU availability rather than the capabilities and topology of the underlying network interface.
On modern systems with many CPUs but fewer NIC hardware queues, this can lead to multiple NVMe/TCP I/O workers contending for the same TX/RX queue, resulting in increased lock contention, cacheline bouncing, and degraded throughput.
This RFC proposes a set of changes to better align NVMe/TCP I/O queues with NIC queue resources, and to expose queue/flow information to enable more effective system-level tuning.
Key ideas
---------1. Scale NVMe/TCP I/O queues based on NIC queue count
Instead of relying solely on CPU count, limit the number of I/O workers to: min(num_online_cpus, netdev->real_num_{tx,rx}_queues)2. Improve CPU locality
Align NVMe/TCP I/O workers with CPUs associated with NIC IRQ affinity to reduce cross-CPU traffic and improve cache locality.3. Expose queue and flow information via debugfs
Export per-I/O queue information including:
- queue id (qid)
- CPU affinity
- TCP flow (src/dst IP and ports)This enables userspace tools to configure:
- IRQ affinity
- RPS/XPS
- ntuple steering
- or any other scaling as deemed feasible4. Provide infrastructure for extensible debugfs support in NVMe
Together, these changes allow better alignment of:
flow -> NIC queue -> IRQ -> CPU -> NVMe/TCP I/O workerPerformance Evaluation
----------------------
Tests were conducted using fio over NVMe/TCP with the following parameters:ioengine=io_uring direct=1 bs=4k numjobs=<#nic-queues> iodepth=64 System: CPUs: 72 NIC: 100G mlx5Two configurations were evaluated.
Scenario 1: NIC queues < CPU count
----------------------------------
- CPUs: 72
- NIC queues: 32Baseline Patched Patched + tuning
randread 3141 MB/s 3228 MB/s 7509 MB/s(767k IOPS) (788k IOPS) (1833k IOPS)randwrite 4510 MB/s 6172 MB/s 7518 MB/s
(1101k IOPS) (1507k IOPS) (1836k IOPS)randrw (read) 2156 MB/s 2560 MB/s 3932 MB/s
(526k IOPS) (625k IOPS) (960k IOPS)randrw (write) 2155 MB/s 2560 MB/s 3932 MB/s
(526k IOPS) (625k IOPS) (960k IOPS)Observation:
When CPU count exceeds NIC queue count, the baseline configuration suffers from queue contention. The proposed changes provide modest improvements on their own, and when combined with queue-aware tuning (IRQ affinity, ntuple steering, and CPU alignment), enable up to ~1.5x–2.5x throughput improvement.Scenario 2: NIC queues == CPU count
------------------------------------ CPUs: 72
- NIC queues: 72Baseline Patched + tuning
randread 4310 MB/s 7987 MB/s(1052k IOPS) (1950k IOPS)randwrite 7947 MB/s 7972 MB/s
(1940k IOPS) (1946k IOPS)randrw (read) 3583 MB/s 4030 MB/s
(875k IOPS) (984k IOPS)randrw (write) 3583 MB/s 4029 MB/s
(875k IOPS) (984k IOPS)Observation:
When NIC queues are already aligned with CPU count, the baseline performs well. The proposed changes maintain write performance (no regression) and still improve read and mixed workloads due to better flow-to-CPU locality.Notes on tuning
---------------
The "patched + tuning" configuration includes:
- aligning NVMe/TCP I/O workers with NIC queue count
- IRQ affinity configuration per RX queue
- ntuple-based flow steering
- CPU/queue affinity alignmentThese tuning steps are enabled by the queue/flow information exposed through this patchset.
Discussion
----------
This RFC aims to start discussion around:
- Whether NVMe/TCP queue scaling should consider NIC queue topology
- How best to expose queue/flow information to userspace
- The role of userspace vs kernel in steering decisions
Quoted 4 lines
As usual, feedback/comment/suggestions are most welcome!
Reference to LSF/MM/BPF abstarct: https://lore.kernel.org/all/[email protected]/
Weelll ... we have been debating this back and forth over recent years:
Should we check for hardware limitations for NVMe-over-Fabrics or not?
Initially it sounds appealing, and in fact I've worked on several attempts myself. But in the end there are far more things which need to be considered:
-> For networking, number of queues is not really telling us anything.
Most NICs have distinct RX and TX queues, and the number (of both!)
varies quite dramatically.
-> The number of queues does _not_ indicate that all queues are used
simultaneously. That is down to things like RSS and friends.
I gave a stab at configuring _that_ but it's patently horrible
trying to out-guess things for yourself.
-> It'll only work if you run directly on the NIC. As soon as there
is anything in between (qemu? Tunnelling?) you are out of luck.
So yeah, we should have a discussion here.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Christoph Hellwig
maintainerRe: [RFC PATCH 1/4] nvme-tcp: optionally limit I/O queue count based on NIC queues
In reply to Nilay ShroffQuoted 4 lines
In such configurations, limiting the number of NVMe-TCP I/O queues to the number of NIC hardware queues can improve performance by reducing contention and improving locality. Aligning NVMe-TCP worker threads with NIC queue topology may also help reduce tail latency.
Yes, this sounds useful.
Quoted 8 lines
Add a new transport option "
match_hw_queues" to allow users to optionally limit the number of NVMe-TCP I/O queues to the number of NIC TX/RX queues. When enabled, the number of I/O queues is set to:
min(num_online_cpus, num_nic_queues)This behavior is opt-in and does not change existing defaults.
Any good reason for that? For PCI and RDMA we try to do the right thing by default.
drivers/nvme/host/tcp.c static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl, · line 1772
Quoted 6 lines
drivers/nvme/host/tcp.c +6
-0+static struct net_device *nvme_tcp_get_netdev (struct nvme_ctrl *ctrl) +{ +struct net_device *dev =NULL ; + +if (ctrl->opts->mask &NVMF_OPT_HOST_IFACE ) + dev =dev_get_by_name (&init_net, ctrl->opts->host_iface);
Return early here instead of the giant indentation for the new options.
drivers/nvme/host/tcp.c static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl, · line 1776
Quoted 4 lines
drivers/nvme/host/tcp.c +4
-0+else { +struct nvme_tcp_ctrl *tctrl =to_tcp_ctrl (ctrl); + +if (tctrl->addr.ss_family ==AF_INET ) {
And then split each address family into a helper. And to me those look like something that should be in net/.
drivers/nvme/host/tcp.c static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl, · line 1847
Quoted 6 lines
drivers/nvme/host/tcp.c +6
-0+ +/* +* Returns number of active NIC queues (min of TX/RX), or 0 if device cannot +* be determined. +*/ +static int nvme_tcp_get_netdev_current_queue_count (struct nvme_ctrl *ctrl)
drop _current to make this a bit more readable?
drivers/nvme/host/tcp.c static int nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl) · line 2247
Quoted 6 lines
drivers/nvme/host/tcp.c +2
-0@@ -2144,6 +2243,24 @@ static int nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl)unsigned int nr_io_queues;int ret; +if (!(ctrl->opts->mask &NVMF_OPT_NR_IO_QUEUES ) && + (ctrl->opts->mask &NVMF_OPT_MATCH_HW_QUEUES )) {
The more readable formatting would be:
if (!(ctrl->opts->mask & NVMF_OPT_NR_IO_QUEUES) &&
(ctrl->opts->mask & NVMF_OPT_MATCH_HW_QUEUES)) {Quoted 4 lines
drivers/nvme/host/tcp.c +4
-0+int nr_hw_queues; + + nr_hw_queues =nvme_tcp_get_netdev_current_queue_count (ctrl); +if (nr_hw_queues <=0 )
drivers/nvme/host/tcp.c static int nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl) · line 2261
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+ goto init_queue; + + ctrl->opts->nr_io_queues =min (nr_hw_queues,num_online_cpus ()); + +if (ctrl->opts->nr_io_queues <num_online_cpus ()) +dev_info (ctrl->device, +"limiting I/O queues to %u (NIC queues %d, CPUs %u)\n" , + ctrl->opts->nr_io_queues, nr_hw_queues, +num_online_cpus ()); + }
And splitting this into a helper would help keeping the flow sane.
Sagi Grimberg
maintainerRe: [RFC PATCH 1/4] nvme-tcp: optionally limit I/O queue count based on NIC queues
In reply to Nilay ShroffQuoted 5 lines · Nilay Shroff
NVMe-TCP currently provisions I/O queues primarily based on CPU availability. On systems where the number of CPUs significantly exceeds the number of NIC hardware queues, this can lead to multiple I/O queues sharing the same NIC TX/RX queues, resulting in increased lock contention, cacheline bouncing, and inter-processor interrupts (IPIs).
Yes, I agree it is very inefficient to create something like 192 queues in practice. Nevermind that this is pretty much never the case because real controllers will limit the number of IO queues to something much lower than that, in the majority of cases probably a handful or more.
Please note that this is very much in common with RDMA, so the patch series should probably address both.
Quoted 5 lines
In such configurations, limiting the number of NVMe-TCP I/O queues to the number of NIC hardware queues can improve performance by reducing contention and improving locality. Aligning NVMe-TCP worker threads with NIC queue topology may also help reduce tail latency.
As mentioned, from what I know, when using real nvmf arrays, the number of queues will usually be much lower than both the cpu count as well as the NIC hw queues.
Quoted 8 lines
Add a new transport option "
match_hw_queues" to allow users to optionally limit the number of NVMe-TCP I/O queues to the number of NIC TX/RX queues. When enabled, the number of I/O queues is set to:
min(num_online_cpus, num_nic_queues)This behavior is opt-in and does not change existing defaults.
In my mind, there is no real reason for an opt-in. The opt-in should
probably be if the user actually wants to use num_online_cpus() worth of queues (e.g. user explicitly asked for nr_io_queues).
Quoted 35 lines
Signed-off-by Nilay Shroff <[email protected]>--- drivers/nvme/host/fabrics.c | 4 ++ drivers/nvme/host/fabrics.h | 3 + drivers/nvme/host/tcp.c | 120 +++++++++++++++++++++++++++++++++++- 3 files changed, 126 insertions(+), 1 deletion(-)drivers/nvme/host/fabrics.c +4
-0@@ -709,6 +709,7 @@ static const match_table_t opt_tokens = { { NVMF_OPT_TLS, "tls" }, { NVMF_OPT_CONCAT, "concat" }, #endif + { NVMF_OPT_MATCH_HW_QUEUES, "match_hw_queues" }, { NVMF_OPT_ERR, NULL } };@@ -1064,6 +1065,9 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts, } opts->concat = true;break ; +case NVMF_OPT_MATCH_HW_QUEUES : + opts->match_hw_queues = true; +break ;default :pr_warn ("unknown parameter or missing value '%s' in ctrl creation request\n" , p);drivers/nvme/host/fabrics.h +0
-0
drivers/nvme/host/fabrics.h enum { · line 71
Quoted 5 lines
drivers/nvme/host/fabrics.h +1
-0@@ -67,6 +67,7 @@ enum {NVMF_OPT_KEYRING =1 <<26 ,NVMF_OPT_TLS_KEY =1 <<27 ,NVMF_OPT_CONCAT =1 <<28 , + NVMF_OPT_MATCH_HW_QUEUES = 1 << 29, };
No need for the above in my mind.
Quoted 91 lines
drivers/nvme/host/fabrics.h +2
-0/** @@ -106,6 +107,7 @@ enum {* @disable_sqflow: disable controller sq flow control * @hdr_digest: generate/verify header digest (TCP) * @data_digest: generate/verify data digest (TCP) +* @match_hw_queues: limit controller IO queue count based on NIC queues (TCP) * @nr_write_queues: number of queues for write I/O * @nr_poll_queues: number of queues for polling I/O * @tos: type of service @@ -136,6 +138,7 @@ struct nvmf_ctrl_options {bool disable_sqflow;bool hdr_digest;bool data_digest; +bool match_hw_queues;unsigned int nr_write_queues;unsigned int nr_poll_queues;int tos;drivers/nvme/host/tcp.c +58
-0@@ -16,6 +16,8 @@#include <net/tls.h> #include <net/tls_prot.h> #include <net/handshake.h> +#include <net/ip6_route.h> +#include <linux/in6.h> #include <linux/blk-mq.h> #include <net/busy_poll.h> #include <trace/events/sock.h> @@ -1762,6 +1764,103 @@ static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl,return ret; } +static struct net_device *nvme_tcp_get_netdev (struct nvme_ctrl *ctrl) +{ +struct net_device *dev =NULL ; + +if (ctrl->opts->mask &NVMF_OPT_HOST_IFACE ) + dev =dev_get_by_name (&init_net, ctrl->opts->host_iface); +else { +struct nvme_tcp_ctrl *tctrl =to_tcp_ctrl (ctrl); + +if (tctrl->addr.ss_family ==AF_INET ) { +struct rtable *rt; +struct flowi4 fl4 = {}; +struct sockaddr_in *addr = + (struct sockaddr_in *)&tctrl->addr; + + fl4.daddr = addr->sin_addr.s_addr; +if (ctrl->opts->mask &NVMF_OPT_HOST_TRADDR ) { + addr = (struct sockaddr_in *)&tctrl->src_addr; + fl4.saddr = addr->sin_addr.s_addr; + } + fl4.flowi4_proto =IPPROTO_TCP ; + + rt =ip_route_output_key (&init_net, &fl4); +if (IS_ERR (rt)) +return NULL ; + + dev =dst_dev (&rt->dst); +/* +* Get reference to netdev as ip_rt_put() will +* release the netdev reference. +*/ +if (dev) +dev_hold (dev); + +ip_rt_put (rt); + + }else if (tctrl->addr.ss_family ==AF_INET6 ) { +struct dst_entry *dst; +struct flowi6 fl6 = {}; +struct sockaddr_in6 *addr6 = + (struct sockaddr_in6 *)&tctrl->addr; + + fl6.daddr = addr6->sin6_addr; +if (ctrl->opts->mask &NVMF_OPT_HOST_TRADDR ) { + addr6 = (struct sockaddr_in6 *)&tctrl->src_addr; + fl6.saddr = addr6->sin6_addr; + } + fl6.flowi6_proto =IPPROTO_TCP ; + + dst =ip6_route_output (&init_net,NULL , &fl6); +if (dst->error) { +dst_release (dst); +return NULL ; + } + + dev =dst_dev (dst);
drivers/nvme/host/tcp.c static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl, · line 1832
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+/* +* Get reference to netdev as dst_release() will +* release the netdev reference. +*/ +if (dev) +dev_hold (dev); + +dst_release (dst); + } + }
This looks like a helper that should be outside of nvme-tcp. Nothing specific to it here. Something like dev_get_by_dstaddr()
Quoted 15 lines
drivers/nvme/host/tcp.c +15
-0+ +return dev; +} + +static void nvme_tcp_put_netdev (struct net_device *dev) +{ +if (dev) +dev_put (dev); +} + +/* +* Returns number of active NIC queues (min of TX/RX), or 0 if device cannot +* be determined. +*/ +static int nvme_tcp_get_netdev_current_queue_count (struct nvme_ctrl *ctrl)
drivers/nvme/host/tcp.c static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl, · line 1857
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+{ +struct net_device *dev; +int tx_queues, rx_queues; + + dev =nvme_tcp_get_netdev (ctrl); +if (!dev) +return 0 ; + + tx_queues = dev->real_num_tx_queues; + rx_queues = dev->real_num_rx_queues;
I can see various ways how this can get wrong with the variety of stacked network devices. For example for bonding, this can easily diverge with the slave devices queues (in theory at least). Also vlan/vxlan devices will also not represent the real hw queues iirc.
This is a good example of how nvme-tcp is different than the other drivers. It sits on top of an abstraction layer, which prevents it from "not getting it wrong". It may get it right *some* of the time, but it can also get it wrong...
Maybe an explicit optin is warranted here...
I would not be against having this approach in case an explicit opt-in is passed by the user I suppose.
btw such an approach would be much more robust in nvme-rdma which does not see this set of abstractions.
One thing that I will comment in addition, is that nvme-tcp is likely to see *multiple* controllers (HA fundamentals for nvmf arrays), so I think that improving performance in this scenario would be much more impactful.
Sagi Grimberg
maintainerRe: [RFC PATCH 2/4] nvme-tcp: add a diagnostic message when NIC queues are underutilized
In reply to Nilay ShroffQuoted 10 lines · Nilay Shroff
Some systems may configure fewer NIC queues than supported by the hardware. When the number of NVMe-TCP I/O queues is limited by the number of active NIC queues, this can result in suboptimal performance.
Add a diagnostic message to warn when the configured NIC queue count is lower than the maximum supported queue count, as reported by the driver. This may help users identify configurations where increasing the NIC queue count could improve performance.
This change is informational only and does not modify NIC configuration.
I don't think that we want this at all. I don't think this is nvme-tcp place to print such a log message at all. Especially not every time it connects to a controller.
If you think you need to add this, create a userspace nvmf tool that tests/validates a host.
Sagi Grimberg
maintainerRe: [RFC PATCH 4/4] nvme: expose queue information via debugfs
In reply to Nilay ShroffQuoted 11 lines · Nilay Shroff
Add a new debugfs attribute "
io_queue_info" to expose per-queue information for NVMe controllers. For NVMe-TCP, this includes the CPU handling each I/O queue and the associated TCP flow (source and destination address/port).This information can be useful for understanding and tuning the interaction between NVMe-TCP I/O queues and network stack components, such as IRQ affinity, RPS/RFS, XPS, or NIC flow steering (ntuple).
The data is exported using
seq_fileinterfaces to allow iteration over all controller queues.
Don't really mind having this. Not sure who will actually go through the process of mangling RFS/RPS/XPS for based on this 5-tuple, but ok...
Sagi Grimberg
maintainerQuoted 146 lines · Hannes Reinecke
Quoted 125 lines · Nilay Shroff
Hi,
The NVMe/TCP host driver currently provisions I/O queues primarily based on CPU availability rather than the capabilities and topology of the underlying network interface.
On modern systems with many CPUs but fewer NIC hardware queues, this can lead to multiple NVMe/TCP I/O workers contending for the same TX/RX queue, resulting in increased lock contention, cacheline bouncing, and degraded throughput.
This RFC proposes a set of changes to better align NVMe/TCP I/O queues with NIC queue resources, and to expose queue/flow information to enable more effective system-level tuning.
Key ideas
---------1. Scale NVMe/TCP I/O queues based on NIC queue count
Instead of relying solely on CPU count, limit the number of I/O workers to:
min(num_online_cpus, netdev->real_num_{tx,rx}_queues)2. Improve CPU locality
Align NVMe/TCP I/O workers with CPUs associated with NIC IRQ affinity to reduce cross-CPU traffic and improve cache locality.3. Expose queue and flow information via debugfs
Export per-I/O queue information including:
- queue id (qid)
- CPU affinity
- TCP flow (src/dst IP and ports)This enables userspace tools to configure:
- IRQ affinity
- RPS/XPS
- ntuple steering
- or any other scaling as deemed feasible4. Provide infrastructure for extensible debugfs support in NVMe
Together, these changes allow better alignment of:
flow -> NIC queue -> IRQ -> CPU -> NVMe/TCP I/O workerPerformance Evaluation
----------------------
Tests were conducted using fio over NVMe/TCP with the following parameters:
ioengine=io_uring
direct=1
bs=4k
numjobs=<#nic-queues>
iodepth=64
System:
CPUs: 72
NIC: 100G mlx5Two configurations were evaluated.
Scenario 1: NIC queues < CPU count
----------------------------------
- CPUs: 72
- NIC queues: 32Baseline Patched Patched + tuning randread 3141 MB/s 3228 MB/s 7509 MB/s (767k IOPS) (788k IOPS) (1833k IOPS)
randwrite 4510 MB/s 6172 MB/s 7518 MB/s (1101k IOPS) (1507k IOPS) (1836k IOPS)
randrw (read) 2156 MB/s 2560 MB/s 3932 MB/s (526k IOPS) (625k IOPS) (960k IOPS)
randrw (write) 2155 MB/s 2560 MB/s 3932 MB/s (526k IOPS) (625k IOPS) (960k IOPS)
Observation:
When CPU count exceeds NIC queue count, the baseline configuration suffers from queue contention. The proposed changes provide modest improvements on their own, and when combined with queue-aware tuning (IRQ affinity, ntuple steering, and CPU alignment), enable up to ~1.5x–2.5x throughput improvement.Scenario 2: NIC queues == CPU count
------------------------------------ CPUs: 72
- NIC queues: 72Baseline Patched + tuning randread 4310 MB/s 7987 MB/s (1052k IOPS) (1950k IOPS)
randwrite 7947 MB/s 7972 MB/s (1940k IOPS) (1946k IOPS)
randrw (read) 3583 MB/s 4030 MB/s (875k IOPS) (984k IOPS)
randrw (write) 3583 MB/s 4029 MB/s (875k IOPS) (984k IOPS)
Observation:
When NIC queues are already aligned with CPU count, the baseline performs well. The proposed changes maintain write performance (no regression) and still improve read and mixed workloads due to better flow-to-CPU locality.Notes on tuning
---------------
The "patched + tuning" configuration includes:
- aligning NVMe/TCP I/O workers with NIC queue count
- IRQ affinity configuration per RX queue
- ntuple-based flow steering
- CPU/queue affinity alignmentThese tuning steps are enabled by the queue/flow information exposed through this patchset.
Discussion
----------
This RFC aims to start discussion around:
- Whether NVMe/TCP queue scaling should consider NIC queue topology
- How best to expose queue/flow information to userspace
- The role of userspace vs kernel in steering decisionsQuoted 4 lines
As usual, feedback/comment/suggestions are most welcome!
Reference to LSF/MM/BPF abstarct: https://lore.kernel.org/all/[email protected]/
Weelll ... we have been debating this back and forth over recent years:
Should we check for hardware limitations for NVMe-over-Fabrics or not?Initially it sounds appealing, and in fact I've worked on several attempts myself. But in the end there are far more things which need to be considered:
-> For networking, number of queues is not really telling us anything.
Most NICs have distinct RX and TX queues, and the number (of both!)
varies quite dramatically.
-> The number of queues does _not_ indicate that all queues are used
simultaneously. That is down to things like RSS and friends.
I gave a stab at configuring _that_ but it's patently horrible
trying to out-guess things for yourself.
-> It'll only work if you run directly on the NIC. As soon as there
is anything in between (qemu? Tunnelling?) you are out of luck.
Quoted 1 line
So yeah, we should have a discussion here.
TBH, I don't think that this is very useful. I mentioned some areas on why on patch #1
But the main reason is that I think that the majority the gains that you are showing is the tuning - which is somewhat unrelated to the driver, and TBH, I doubt anyone will actually do in reality.
Quoted 131 lines · Hannes Reinecke
Quoted 125 lines · Nilay Shroff
Hi,
The NVMe/TCP host driver currently provisions I/O queues primarily based on CPU availability rather than the capabilities and topology of the underlying network interface.
On modern systems with many CPUs but fewer NIC hardware queues, this can lead to multiple NVMe/TCP I/O workers contending for the same TX/RX queue, resulting in increased lock contention, cacheline bouncing, and degraded throughput.
This RFC proposes a set of changes to better align NVMe/TCP I/O queues with NIC queue resources, and to expose queue/flow information to enable more effective system-level tuning.
Key ideas
---------1. Scale NVMe/TCP I/O queues based on NIC queue count
Instead of relying solely on CPU count, limit the number of I/O workers to:
min(num_online_cpus, netdev->real_num_{tx,rx}_queues)2. Improve CPU locality
Align NVMe/TCP I/O workers with CPUs associated with NIC IRQ affinity to reduce cross-CPU traffic and improve cache locality.3. Expose queue and flow information via debugfs
Export per-I/O queue information including:
- queue id (qid)
- CPU affinity
- TCP flow (src/dst IP and ports)This enables userspace tools to configure:
- IRQ affinity
- RPS/XPS
- ntuple steering
- or any other scaling as deemed feasible4. Provide infrastructure for extensible debugfs support in NVMe
Together, these changes allow better alignment of:
flow -> NIC queue -> IRQ -> CPU -> NVMe/TCP I/O workerPerformance Evaluation
----------------------
Tests were conducted using fio over NVMe/TCP with the following parameters:
ioengine=io_uring
direct=1
bs=4k
numjobs=<#nic-queues>
iodepth=64
System:
CPUs: 72
NIC: 100G mlx5Two configurations were evaluated.
Scenario 1: NIC queues < CPU count
----------------------------------
- CPUs: 72
- NIC queues: 32Baseline Patched Patched + tuning randread 3141 MB/s 3228 MB/s 7509 MB/s (767k IOPS) (788k IOPS) (1833k IOPS)
randwrite 4510 MB/s 6172 MB/s 7518 MB/s (1101k IOPS) (1507k IOPS) (1836k IOPS)
randrw (read) 2156 MB/s 2560 MB/s 3932 MB/s (526k IOPS) (625k IOPS) (960k IOPS)
randrw (write) 2155 MB/s 2560 MB/s 3932 MB/s (526k IOPS) (625k IOPS) (960k IOPS)
Observation:
When CPU count exceeds NIC queue count, the baseline configuration suffers from queue contention. The proposed changes provide modest improvements on their own, and when combined with queue-aware tuning (IRQ affinity, ntuple steering, and CPU alignment), enable up to ~1.5x–2.5x throughput improvement.Scenario 2: NIC queues == CPU count
------------------------------------ CPUs: 72
- NIC queues: 72Baseline Patched + tuning randread 4310 MB/s 7987 MB/s (1052k IOPS) (1950k IOPS)
randwrite 7947 MB/s 7972 MB/s (1940k IOPS) (1946k IOPS)
randrw (read) 3583 MB/s 4030 MB/s (875k IOPS) (984k IOPS)
randrw (write) 3583 MB/s 4029 MB/s (875k IOPS) (984k IOPS)
Observation:
When NIC queues are already aligned with CPU count, the baseline performs well. The proposed changes maintain write performance (no regression) and still improve read and mixed workloads due to better flow-to-CPU locality.Notes on tuning
---------------
The "patched + tuning" configuration includes:
- aligning NVMe/TCP I/O workers with NIC queue count
- IRQ affinity configuration per RX queue
- ntuple-based flow steering
- CPU/queue affinity alignmentThese tuning steps are enabled by the queue/flow information exposed through this patchset.
Discussion
----------
This RFC aims to start discussion around:
- Whether NVMe/TCP queue scaling should consider NIC queue topology
- How best to expose queue/flow information to userspace
- The role of userspace vs kernel in steering decisionsQuoted 4 lines
As usual, feedback/comment/suggestions are most welcome!
Reference to LSF/MM/BPF abstarct: https://lore.kernel.org/all/[email protected]/
Quoted 8 lines
Weelll ... we have been debating this back and forth over recent years:
Should we check for hardware limitations for NVMe-over-Fabrics or not?Initially it sounds appealing, and in fact I've worked on several attempts myself. But in the end there are far more things which need to be considered:
-> For networking, number of queues is not really telling us anything.
Most NICs have distinct RX and TX queues, and the number (of both!)
varies quite dramatically.
The proposed I/O queue scaling follows a conservative approach based on currently configured NIC queues:
if (NIC exposes combined TX/RX queues):
nr_hw_queues = min(num_online_cpus, combined_tx_rx);
else:
real_hw_queues = min(real_num_tx_queues, real_num_rx_queues);
nr_hw_queues = min(num_online_cpus, real_hw_queues);The intent here is not to model full NIC behavior, but to avoid obvious over-subscription when the number of NVMe/TCP I/O workers significantly exceeds the available NIC queues.
Also, this is not enabled by default. It is gated behind the match-hw-queues fabric option, so existing setups are unaffected unless explicitly enabled.
Quoted 4 lines
-> The number of queues does _not_ indicate that all queues are used
simultaneously. That is down to things like RSS and friends. I gave a stab at configuring _that_ but it's patently horrible trying to out-guess things for yourself.
Agreed that queue count alone does not imply effective parallelism, as traffic distribution depends on RSS/RPS/XPS.
This patchset does not attempt to infer or control how queues are used. Instead, it treats the currently configured number of TX/RX queues as a conservative upper bound for I/O worker scaling.
In addition, the patchset exposes queue, CPU, and flow information via debugfs. This allows userspace to configure steering policies (IRQ affinity, ntuple, RPS/XPS) based on actual system behavior. If NIC supports configuring n-tuple filter then it's possible to distribute each I/O flow to a unique queue. In fact, exporting the nvmf-tcp I/O flow and its cpu information via debugfs should be very useful to configure n-tuple and thus distribute each flow to a unique queue and thus it shall help align tx, rx, IRQ and tcp worker on the same cpu.
Quoted 3 lines
-> It'll only work if you run directly on the NIC. As soon as there
is anything in between (qemu? Tunnelling?) you are out of luck.
Agreed that in environments where NIC topology does not reflect the effective data path (e.g., certain QEMU or tunneling configurations), this heuristic may not be meaningful. In such cases, users can simply avoid enabling "match-hw-queues" and retain the existing behavior.
That said, there are also configurations using QEMU (e.g., vhost-net with multiqueue, or VFIO passthrough) or SR-IOV where NIC queue topology is still relevant, and this approach can provide benefit.
Overall, the goal here is:
- to avoid clear over-provisioning of I/O workers, and
- to expose sufficient information for userspace driven tuning (using RPS/XPS/n-tuple etc.)
Quoted 2 lines
So yeah, we should have a discussion here.
Sure, looking forward for further discussion.
Thanks,
--Nilay
Re: [RFC PATCH 1/4] nvme-tcp: optionally limit I/O queue count based on NIC queues
In reply to Christoph HellwigQuoted 12 lines · Christoph Hellwig
Quoted 4 lines
In such configurations, limiting the number of NVMe-TCP I/O queues to the number of NIC hardware queues can improve performance by reducing contention and improving locality. Aligning NVMe-TCP worker threads with NIC queue topology may also help reduce tail latency.
Yes, this sounds useful.
Quoted 5 lines
Add a new transport option "
match_hw_queues" to allow users to optionally limit the number of NVMe-TCP I/O queues to the number of NIC TX/RX queues. When enabled, the number of I/O queues is set to:
Quoted 7 lines
Quoted 3 lines
min(num_online_cpus, num_nic_queues)This behavior is opt-in and does not change existing defaults.
Any good reason for that? For PCI and RDMA we try to do the right thing by default.
The only reason was that in certain complex typologies it may not be really possible (for instance, QEMU) to get the real num of tx/rx queues. In such situation, I thought we're better off using this feature and hence I added the opt-in. But yes I'd also love to remove this option and find a better way to detect such cases where we can't get the real num of tx/rx queues and thus aromatically fallback to creating as many I/O queues as num of online cpus. I'd explore this and see if that's possible.
Quoted 9 lines
drivers/nvme/host/tcp.c static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl, · line 1772
Quoted 6 lines
drivers/nvme/host/tcp.c +6
-0+static struct net_device *nvme_tcp_get_netdev (struct nvme_ctrl *ctrl) +{ +struct net_device *dev =NULL ; + +if (ctrl->opts->mask &NVMF_OPT_HOST_IFACE ) + dev =dev_get_by_name (&init_net, ctrl->opts->host_iface);Return early here instead of the giant indentation for the new options.
Yes okay, makes sense!
Quoted 8 lines
drivers/nvme/host/tcp.c static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl, · line 1776
Quoted 4 lines
drivers/nvme/host/tcp.c +4
-0+else { +struct nvme_tcp_ctrl *tctrl =to_tcp_ctrl (ctrl); + +if (tctrl->addr.ss_family ==AF_INET ) {And then split each address family into a helper. And to me those look like something that should be in
net/.
Hmm okay, I think if we want to add these helpers under net/ then it should be in include/net/route.h and include/net/ip6_route.h for IPv4 and IPv6 respectively.
Quoted 9 lines
drivers/nvme/host/tcp.c static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl, · line 1847
Quoted 6 lines
drivers/nvme/host/tcp.c +6
-0+ +/* +* Returns number of active NIC queues (min of TX/RX), or 0 if device cannot +* be determined. +*/ +static int nvme_tcp_get_netdev_current_queue_count (struct nvme_ctrl *ctrl)drop
_currentto make this a bit more readable?
Sure.
Quoted 4 lines
Quoted 4 lines
drivers/nvme/host/tcp.c +0
-0@@ -2144,6 +2243,24 @@ static int nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl)unsigned int nr_io_queues;int ret;
Quoted 8 lines
drivers/nvme/host/tcp.c static int nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl) · line 2247
Quoted 2 lines
drivers/nvme/host/tcp.c +2
-0+if (!(ctrl->opts->mask &NVMF_OPT_NR_IO_QUEUES ) && + (ctrl->opts->mask &NVMF_OPT_MATCH_HW_QUEUES )) {The more readable formatting would be:
if (!(ctrl->opts->mask & NVMF_OPT_NR_IO_QUEUES) && (ctrl->opts->mask & NVMF_OPT_MATCH_HW_QUEUES)) {
Yep, I will change this.
Quoted 8 lines
Quoted 8 lines
drivers/nvme/host/tcp.c +8
-0+int nr_hw_queues; + + nr_hw_queues =nvme_tcp_get_netdev_current_queue_count (ctrl); +if (nr_hw_queues <=0 ) + goto init_queue; + + ctrl->opts->nr_io_queues =min (nr_hw_queues,num_online_cpus ()); +
Quoted 9 lines
drivers/nvme/host/tcp.c static int nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl) · line 2261
Quoted 6 lines
drivers/nvme/host/tcp.c +6
-0+if (ctrl->opts->nr_io_queues <num_online_cpus ()) +dev_info (ctrl->device, +"limiting I/O queues to %u (NIC queues %d, CPUs %u)\n" , + ctrl->opts->nr_io_queues, nr_hw_queues, +num_online_cpus ()); + }And splitting this into a helper would help keeping the flow sane.
Alright, will make it into separate helper.
Thanks,
--Nilay
Re: [RFC PATCH 1/4] nvme-tcp: optionally limit I/O queue count based on NIC queues
In reply to Sagi GrimbergQuoted 9 lines · Sagi Grimberg
Quoted 5 lines · Nilay Shroff
NVMe-TCP currently provisions I/O queues primarily based on CPU availability. On systems where the number of CPUs significantly exceeds the number of NIC hardware queues, this can lead to multiple I/O queues sharing the same NIC TX/RX queues, resulting in increased lock contention, cacheline bouncing, and inter-processor interrupts (IPIs).
Quoted 5 lines
Yes, I agree it is very inefficient to create something like 192 queues in practice. Nevermind that this is pretty much never the case because real controllers will limit the number of IO queues to something much lower than that, in the majority of cases probably a handful or more.
Yes, it may not always be possible (or meaningful) to have a 1:1 mapping between NVMe/TCP I/O queues and the real controller’s internal queues.
The intent here is not to enforce such a mapping, but rather to improve host-side locality. Even when the controller limits the number of I/O queues, mismatch with NIC queue topology can still lead to multiple I/O workers contending on the same TX/RX queues. This change tries to avoid such over-subscription on the host side.
Quoted 3 lines
Please note that this is very much in common with RDMA, so the patch series should probably address both.
Yes agreed — the model in RDMA is more tightly coupled to hardware resources. It probably makes sense to consider that separately, as the problem space is simpler and the mapping is more direct.
Quoted 15 lines
Quoted 5 lines
In such configurations, limiting the number of NVMe-TCP I/O queues to the number of NIC hardware queues can improve performance by reducing contention and improving locality. Aligning NVMe-TCP worker threads with NIC queue topology may also help reduce tail latency.
As mentioned, from what I know, when using real nvmf arrays, the number of queues will usually be much lower than both the cpu count as well as the NIC hw queues.
Quoted 5 lines
Add a new transport option "
match_hw_queues" to allow users to optionally limit the number of NVMe-TCP I/O queues to the number of NIC TX/RX queues. When enabled, the number of I/O queues is set to:
Quoted 7 lines
Quoted 3 lines
min(num_online_cpus, num_nic_queues)This behavior is opt-in and does not change existing defaults.
In my mind, there is no real reason for an opt-in. The opt-in should
probably be if the user actually wants to usenum_online_cpus()worth of queues (e.g. user explicitly asked fornr_io_queues).Quoted 1 line
Yes that makes sense, but in certain complex typologies such as host running inside QEMU or stacked network devices, it may not be possible to find the real num of tx/rx queues exposed by real NIC cards. So for such cased we may not want to alter the existing behavior and hence I choose to support opt-in.
Quoted 135 lines
Quoted 33 lines
Signed-off-by Nilay Shroff <[email protected]>---
drivers/nvme/host/fabrics.c| 4 ++drivers/nvme/host/fabrics.h| 3 + drivers/nvme/host/tcp.c | 120 +++++++++++++++++++++++++++++++++++-3 files changed, 126 insertions(+), 1 deletion(-)drivers/nvme/host/fabrics.c +4
-0@@ -709,6 +709,7 @@ static const match_table_t opt_tokens = { { NVMF_OPT_TLS, "tls" }, { NVMF_OPT_CONCAT, "concat" }, #endif + { NVMF_OPT_MATCH_HW_QUEUES, "match_hw_queues" }, { NVMF_OPT_ERR, NULL } };@@ -1064,6 +1065,9 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts, } opts->concat = true; break; + case NVMF_OPT_MATCH_HW_QUEUES: + opts->match_hw_queues = true; + break;default:
pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n", p);drivers/nvme/host/fabrics.h +0
-0drivers/nvme/host/fabrics.h enum { · line 71
Quoted 5 lines
drivers/nvme/host/fabrics.h +1
-0@@ -67,6 +67,7 @@ enum { NVMF_OPT_KEYRING = 1 << 26, NVMF_OPT_TLS_KEY = 1 << 27, NVMF_OPT_CONCAT = 1 << 28, + NVMF_OPT_MATCH_HW_QUEUES = 1 << 29, };No need for the above in my mind.
Quoted 94 lines
drivers/nvme/host/fabrics.h +2
-0/**@@ -106,6 +107,7 @@ enum { * @disable_sqflow: disable controller sq flow control * @hdr_digest: generate/verify header digest (TCP) * @data_digest: generate/verify data digest (TCP) + * @match_hw_queues: limit controller IO queue count based on NIC queues (TCP) * @nr_write_queues: number of queues for write I/O * @nr_poll_queues: number of queues for polling I/O * @tos: type of service@@ -136,6 +138,7 @@ struct nvmf_ctrl_options { bool disable_sqflow; bool hdr_digest; bool data_digest; + bool match_hw_queues; unsigned int nr_write_queues; unsigned int nr_poll_queues; int tos;drivers/nvme/host/tcp.c +63
-0@@ -16,6 +16,8 @@ #include <net/tls.h> #include <net/tls_prot.h> #include <net/handshake.h> +#include <net/ip6_route.h> +#include <linux/in6.h> #include <linux/blk-mq.h> #include <net/busy_poll.h> #include <trace/events/sock.h>@@ -1762,6 +1764,103 @@ static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl, return ret; } +static struct net_device *nvme_tcp_get_netdev (struct nvme_ctrl *ctrl) +{ + struct net_device *dev = NULL; + + if (ctrl->opts->mask & NVMF_OPT_HOST_IFACE) + dev = dev_get_by_name(&init_net, ctrl->opts->host_iface); + else { + struct nvme_tcp_ctrl *tctrl = to_tcp_ctrl(ctrl); + + if (tctrl->addr.ss_family == AF_INET) { + struct rtable *rt; + struct flowi4 fl4 = {}; + struct sockaddr_in *addr = + (struct sockaddr_in *)&tctrl->addr; + + fl4.daddr = addr->sin_addr.s_addr; + if (ctrl->opts->mask & NVMF_OPT_HOST_TRADDR) { + addr = (struct sockaddr_in *)&tctrl->src_addr; + fl4.saddr = addr->sin_addr.s_addr; + } + fl4.flowi4_proto = IPPROTO_TCP; + + rt = ip_route_output_key(&init_net, &fl4); + if (IS_ERR(rt)) + return NULL; + + dev = dst_dev(&rt->dst); + /* + * Get reference to netdev as ip_rt_put() will + * release the netdev reference. + */ + if (dev) + dev_hold(dev); + + ip_rt_put(rt); + + } else if (tctrl->addr.ss_family == AF_INET6) { + struct dst_entry *dst; + struct flowi6 fl6 = {}; + struct sockaddr_in6 *addr6 = + (struct sockaddr_in6 *)&tctrl->addr; + + fl6.daddr = addr6->sin6_addr; + if (ctrl->opts->mask & NVMF_OPT_HOST_TRADDR) { + addr6 = (struct sockaddr_in6 *)&tctrl->src_addr; + fl6.saddr = addr6->sin6_addr; + } + fl6.flowi6_proto = IPPROTO_TCP; + + dst = ip6_route_output(&init_net, NULL, &fl6); + if (dst->error) { + dst_release(dst); + return NULL; + } + + dev = dst_dev(dst); + /* + * Get reference to netdev as dst_release() will + * release the netdev reference. + */ + if (dev)
Quoted 8 lines
drivers/nvme/host/tcp.c static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl, · line 1831
Quoted 5 lines
drivers/nvme/host/tcp.c +5
-0+ dev_hold(dev); + + dst_release(dst); + } + }This looks like a helper that should be outside of nvme-tcp. Nothing specific to it here. Something like
dev_get_by_dstaddr()
Yeah, in another thread Christoph also had similar opinion. So I'd add a helper under net/.
Quoted 31 lines
Quoted 15 lines
drivers/nvme/host/tcp.c +15
-0+ + return dev; +} + +static void nvme_tcp_put_netdev (struct net_device *dev) +{ + if (dev) + dev_put(dev); +} + +/* +* Returns number of active NIC queues (min of TX/RX), or 0 if device cannot +* be determined. +*/ +static int nvme_tcp_get_netdev_current_queue_count (struct nvme_ctrl *ctrl)drivers/nvme/host/tcp.c static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl, · line 1857
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+{ + struct net_device *dev; + int tx_queues, rx_queues; + + dev = nvme_tcp_get_netdev(ctrl); + if (!dev) + return 0; + + tx_queues = dev->real_num_tx_queues; + rx_queues = dev->real_num_rx_queues;I can see various ways how this can get wrong with the variety of stacked network devices. For example for bonding, this can easily diverge with the slave devices queues (in theory at least). Also vlan/vxlan devices will also not represent the real hw queues iirc.
Quoted 8 lines
This is a good example of how nvme-tcp is different than the other drivers. It sits on top of an abstraction layer, which prevents it from "not getting it wrong". It may get it right *some* of the time, but it can also get it wrong...
Maybe an explicit optin is warranted here...
I would not be against having this approach in case an explicit opt-in is passed by the user I suppose.
Yes so this is the _real_ reason why I proposed opt-in.
Quoted 6 lines
btw such an approach would be much more robust in nvme-rdma which does not see this set of abstractions.
One thing that I will comment in addition, is that nvme-tcp is likely to see *multiple* controllers (HA fundamentals for nvmf arrays), so I think that improving performance in this scenario would be much more impactful.
Hmm, yes really good point. This patch focuses on improving per-controller locality, but I agree that coordinating resource usage across controllers sharing the same NIC could provide additional benefits. This seems like a natural next step, thanks for highlighting this. I will address this in a separate patch, once current work is accepted.
Thanks,
--Nilay
Quoted 149 lines · Sagi Grimberg
Quoted 146 lines · Hannes Reinecke
Quoted 125 lines · Nilay Shroff
Hi,
The NVMe/TCP host driver currently provisions I/O queues primarily based on CPU availability rather than the capabilities and topology of the underlying network interface.
On modern systems with many CPUs but fewer NIC hardware queues, this can lead to multiple NVMe/TCP I/O workers contending for the same TX/RX queue, resulting in increased lock contention, cacheline bouncing, and degraded throughput.
This RFC proposes a set of changes to better align NVMe/TCP I/O queues with NIC queue resources, and to expose queue/flow information to enable more effective system-level tuning.
Key ideas
---------1. Scale NVMe/TCP I/O queues based on NIC queue count
Instead of relying solely on CPU count, limit the number of I/O workers to:
min(num_online_cpus, netdev->real_num_{tx,rx}_queues)2. Improve CPU locality
Align NVMe/TCP I/O workers with CPUs associated with NIC IRQ affinity to reduce cross-CPU traffic and improve cache locality.3. Expose queue and flow information via debugfs
Export per-I/O queue information including:
- queue id (qid)
- CPU affinity
- TCP flow (src/dst IP and ports)This enables userspace tools to configure:
- IRQ affinity
- RPS/XPS
- ntuple steering
- or any other scaling as deemed feasible4. Provide infrastructure for extensible debugfs support in NVMe
Together, these changes allow better alignment of:
flow -> NIC queue -> IRQ -> CPU -> NVMe/TCP I/O workerPerformance Evaluation
----------------------
Tests were conducted using fio over NVMe/TCP with the following parameters:
ioengine=io_uring
direct=1
bs=4k
numjobs=<#nic-queues>
iodepth=64
System:
CPUs: 72
NIC: 100G mlx5Two configurations were evaluated.
Scenario 1: NIC queues < CPU count
----------------------------------
- CPUs: 72
- NIC queues: 32Baseline Patched Patched + tuning randread 3141 MB/s 3228 MB/s 7509 MB/s (767k IOPS) (788k IOPS) (1833k IOPS)
randwrite 4510 MB/s 6172 MB/s 7518 MB/s (1101k IOPS) (1507k IOPS) (1836k IOPS)
randrw (read) 2156 MB/s 2560 MB/s 3932 MB/s (526k IOPS) (625k IOPS) (960k IOPS)
randrw (write) 2155 MB/s 2560 MB/s 3932 MB/s (526k IOPS) (625k IOPS) (960k IOPS)
Observation:
When CPU count exceeds NIC queue count, the baseline configuration suffers from queue contention. The proposed changes provide modest improvements on their own, and when combined with queue-aware tuning (IRQ affinity, ntuple steering, and CPU alignment), enable up to ~1.5x–2.5x throughput improvement.Scenario 2: NIC queues == CPU count
------------------------------------ CPUs: 72
- NIC queues: 72Baseline Patched + tuning randread 4310 MB/s 7987 MB/s (1052k IOPS) (1950k IOPS)
randwrite 7947 MB/s 7972 MB/s (1940k IOPS) (1946k IOPS)
randrw (read) 3583 MB/s 4030 MB/s (875k IOPS) (984k IOPS)
randrw (write) 3583 MB/s 4029 MB/s (875k IOPS) (984k IOPS)
Observation:
When NIC queues are already aligned with CPU count, the baseline performs well. The proposed changes maintain write performance (no regression) and still improve read and mixed workloads due to better flow-to-CPU locality.Notes on tuning
---------------
The "patched + tuning" configuration includes:
- aligning NVMe/TCP I/O workers with NIC queue count
- IRQ affinity configuration per RX queue
- ntuple-based flow steering
- CPU/queue affinity alignmentThese tuning steps are enabled by the queue/flow information exposed through this patchset.
Discussion
----------
This RFC aims to start discussion around:
- Whether NVMe/TCP queue scaling should consider NIC queue topology
- How best to expose queue/flow information to userspace
- The role of userspace vs kernel in steering decisionsQuoted 4 lines
As usual, feedback/comment/suggestions are most welcome!
Reference to LSF/MM/BPF abstarct: https://lore.kernel.org/all/[email protected]/
Weelll ... we have been debating this back and forth over recent years:
Should we check for hardware limitations for NVMe-over-Fabrics or not?Initially it sounds appealing, and in fact I've worked on several attempts myself. But in the end there are far more things which need to be considered:
-> For networking, number of queues is not really telling us anything.
Most NICs have distinct RX and TX queues, and the number (of both!)
varies quite dramatically.
-> The number of queues does _not_ indicate that all queues are used
simultaneously. That is down to things like RSS and friends.
I gave a stab at configuring _that_ but it's patently horrible
trying to out-guess things for yourself.
-> It'll only work if you run directly on the NIC. As soon as there
is anything in between (qemu? Tunnelling?) you are out of luck.
Quoted 7 lines
Quoted 1 line
So yeah, we should have a discussion here.
TBH, I don't think that this is very useful. I mentioned some areas on why on patch #1
But the main reason is that I think that the majority the gains that you are showing is the tuning - which is somewhat unrelated to the driver, and TBH, I doubt anyone will actually do in reality.
Even without additional tuning, aligning the NVMe/TCP I/O workers with CPU and NIC queue locality already provides measurable performance benefits (primarily visible in random write workloads, as shown in Scenario 1).
The additional gains come from system-level tuning (e.g., XPS/RPS/RSS), which further improves utilization of NIC queues and CPU locality. However, the patch enables this tuning by exposing queue/flow information and establishing better default alignment.
While such tuning may not be applied in all deployments, IMO, it should be commonly used in performance-sensitive environments where users aim to fully utilize available NIC and CPU resources.
Thanks,
--Nilay
Re: [RFC PATCH 4/4] nvme: expose queue information via debugfs
In reply to Sagi GrimbergQuoted 12 lines · Sagi Grimberg
Quoted 9 lines · Nilay Shroff
Add a new debugfs attribute "
io_queue_info" to expose per-queue information for NVMe controllers. For NVMe-TCP, this includes the CPU handling each I/O queue and the associated TCP flow (source and destination address/port).This information can be useful for understanding and tuning the interaction between NVMe-TCP I/O queues and network stack components, such as IRQ affinity, RPS/RFS, XPS, or NIC flow steering (ntuple).
Quoted 5 lines
Quoted 2 lines
The data is exported using
seq_fileinterfaces to allow iteration over all controller queues.Don't really mind having this. Not sure who will actually go through the process of mangling RFS/RPS/XPS for based on this 5-tuple, but ok...
Yeah it may not be used always but I think in performance sensitive workload, user would like to leverage this information for tuning I/O stack.
Thanks,
--Nilay
Re: [RFC PATCH 2/4] nvme-tcp: add a diagnostic message when NIC queues are underutilized
In reply to Sagi GrimbergQuoted 12 lines · Sagi Grimberg
Quoted 9 lines · Nilay Shroff
Some systems may configure fewer NIC queues than supported by the hardware. When the number of NVMe-TCP I/O queues is limited by the number of active NIC queues, this can result in suboptimal performance.
Add a diagnostic message to warn when the configured NIC queue count is lower than the maximum supported queue count, as reported by the driver. This may help users identify configurations where increasing the NIC queue count could improve performance.
Quoted 8 lines
Quoted 1 line
This change is informational only and does not modify NIC configuration.
I don't think that we want this at all. I don't think this is nvme-tcp place to print such a log message at all. Especially not every time it connects to a controller.
If you think you need to add this, create a userspace nvmf tool that tests/validates a host.
Okay we can drop this from kernel and may add it in userspace tool such as nvme connect...
Thanks,
--Nilay