nvme-tcp: NIC topology aware I/O queue scaling and queue info export
v3 cover nvme-tcp: NIC topology aware I/O queue scaling and queue info export
This patch series has been revised based on feedback from the netdev community [1].
Following the feedback, the logic for determining the number of fabric I/O queues (nr-io-queues) has been moved to userspace in nvme-cli [2]. As a result, the first two patches from the previous series, which implemented this logic in the kernel driver, have been dropped. This series contains the remaining two patches, which export NVMe/TCP I/O queue information through debugfs to facilitate host-side network tuning.
The first patch introduces debugfs helper functions that can be used by drivers to export debugfs attributes.
The second patch exports NVMe/TCP I/O queue and TCP flow information through debugfs. For each I/O queue, it provides:
- queue id (qid)
- CPU affinity
- TCP flow (src/dst IP and ports)
This enables userspace tools to configure:
- IRQ affinity
- RPS/XPS
- ntuple steering
- or any other scaling as deemed feasible
Together with tunning, these changes allow better alignment of:
flow -> NIC queue -> IRQ -> CPU -> NVMe/TCP I/O worker
Improving this alignment can enhance flow-to-CPU locality, potentially improving performance and reducing I/O latency. Performance evaluation results comparing the baseline with the tuned configuration are available in [3].
Thanks!
[1]: https://lore.kernel.org/all/[email protected]/
[2]: https://github.com/linux-nvme/nvme-cli/pull/4140
[3]: https://lore.kernel.org/all/[email protected]/
Changes from v2
- Dropped patch #1 and #2 from v2
- Added new debugfs attribute parameter "is_visible" which could be useful to hide the attribute for certain configuration or transport.
Link to v2: https://lore.kernel.org/all/[email protected]/
Nilay Shroff (2):
nvme: add debugfs helpers for NVMe drivers
nvme: expose queue information via debugfs
drivers/nvme/host/Makefile | 2 +- drivers/nvme/host/core.c | 3 + drivers/nvme/host/debugfs.c | 207 ++++++++++++++++++++++++++++++++++++ drivers/nvme/host/nvme.h | 12 +++ drivers/nvme/host/tcp.c | 54 ++++++++++ 5 files changed, 277 insertions(+), 1 deletion(-) create mode 100644 drivers/nvme/host/debugfs.c
Signature · 1 line
2.53.0
PATCH v3 1/2 nvme: add debugfs helpers for NVMe drivers
Introduce helper APIs that allow NVMe drivers to register and unregister debugfs entries, along with a reusable attribute structure for defining new debugfs files.
The implementation uses seq_file interfaces to safely expose per- namespace or per-path statistics, while supporting both simple show callbacks and full seq_operations.
This will be used by subsequent patches to expose NVMe-TCP queue and flow information for tuning NVMe TCP I/O workqueue and network stack components.
--- drivers/nvme/host/Makefile | 2 +- drivers/nvme/host/debugfs.c | 133 ++++++++++++++++++++++++++++++++++++ drivers/nvme/host/nvme.h | 10 +++ 3 files changed, 144 insertions(+), 1 deletion(-) create mode 100644 drivers/nvme/host/debugfs.c
drivers/nvme/host/Makefile +1 -1
@@ -11,7 +11,7 @@ obj-$(CONFIG_NVME_FC) += nvme-fc.o obj-$(CONFIG_NVME_TCP) += nvme-tcp.o obj-$(CONFIG_NVME_APPLE) += nvme-apple.o -nvme-core-y += core.o ioctl.o sysfs.o pr.o +nvme-core-y += core.o ioctl.o sysfs.o pr.o debugfs.o nvme-core-$(CONFIG_NVME_VERBOSE_ERRORS) += constants.o nvme-core-$(CONFIG_TRACING) += trace.o nvme-core-$(CONFIG_NVME_MULTIPATH) += multipath.o
drivers/nvme/host/debugfs.c new file +133 -0
133 lines not on this page · raw message
drivers/nvme/host/nvme.h +10 -0
@@ -483,6 +483,16 @@ struct nvme_ctrl {u16 awupf;/* 0's based value. */ }; +void nvme_debugfs_register (struct gendisk *disk); +static inline void nvme_debugfs_unregister (struct gendisk *disk) +{ +/* +* Nothing to do for now. When the request queue is unregistered, +* all files under q->debugfs_dir are recursively deleted. +* This is just a placeholder; the compiler will optimize it out. +*/ +} +static inline enum nvme_ctrl_state nvme_ctrl_state (struct nvme_ctrl *ctrl) {return READ_ONCE (ctrl->state);
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v3 1/2] nvme: add debugfs helpers for NVMe drivers
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] nvme: missing debugfs_mutex and NOIO scope in nvme_debugfs_create_files() --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] nvme: missing debugfs_mutex and NOIO scope in nvme_debugfs_create_files() --
commit cd2cc953bf8eddec233eaabc47f820bf78836863
Author: Nilay Shroff <[email protected]>
nvme: add debugfs helpers for NVMe drivers
This patch introduces helper APIs and an attribute structure for NVMe drivers to register and unregister seq_file-based debugfs entries on a request queue's debugfs directory.
Quoted 6 lines
drivers/nvme/host/debugfs.c new file +0
-0@@ -0,0 +1,133 @@
[ ... ]
drivers/nvme/host/debugfs.c line 127
Quoted 13 lines
drivers/nvme/host/debugfs.c new file +13
-0+static void nvme_debugfs_create_files (struct request_queue *q, +const struct nvme_debugfs_attr *attr,void *data) +{ +if (WARN_ON_ONCE (!q->debugfs_dir)) +return ; + +for (; attr->name; attr++) { +if (attr->is_visible && !attr->is_visible (data)) +continue ; +debugfs_create_file_aux (attr->name, attr->mode, q->debugfs_dir, + data, (void *)attr, &nvme_debugfs_fops); + } +}
[Severity: Medium]
Should nvme_debugfs_create_files() hold q->debugfs_mutex and run under a memalloc_noio_save() scope when accessing q->debugfs_dir and creating debugfs files?
In include/linux/blkdev.h, struct request_queue documents that debugfs_mutex serializes all debugfs metadata operations on debugfs_dir:
struct dentry *debugfs_dir; struct dentry *sched_debugfs_dir; struct dentry *rqos_debugfs_dir; /* * Serializes all debugfs metadata operations using the above dentries. */ struct mutex debugfs_mutex;
Furthermore, block/blk.h notes in blk_debugfs_lock_nomemsave() why NOIO context is needed alongside debugfs_mutex (as introduced in commit dfe48ea17973 ("blk-mq: use NOIO context to prevent deadlock during debugfs creation")):
/*
* debugfs directory and file creation can trigger fs reclaim, which can enter
* back into the block layer request_queue. This can cause deadlock if the
* queue is frozen. Use NOIO context together with debugfs_mutex to prevent fs
* reclaim from triggering block I/O.
*/
When nvme_alloc_ns() calls nvme_debugfs_register(ns->disk) after device_add_disk(), the namespace is already on ctrl->namespaces and the disk is live in sysfs, while blk_register_queue() has already dropped blk_debugfs_lock(q). If the queue is concurrently frozen (for example, by a controller reset calling nvme_start_freeze() or a sysfs write calling blk_mq_freeze_queue()), could GFP_KERNEL allocations inside debugfs_create_file_aux() trigger direct filesystem reclaim into the frozen queue and deadlock, or race with concurrent blktrace, scheduler, rqos, or queue teardown debugfs operations on q->debugfs_dir?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
PATCH v3 2/2 nvme: expose queue information via debugfs
In reply to Nilay ShroffAdd a new debugfs attribute "io_queue_info" to expose per-queue information for NVMe controllers. For NVMe-TCP, this includes the CPU handling each I/O queue and the associated TCP flow (source and destination address/port).
This information can be useful for understanding and tuning the interaction between NVMe-TCP I/O queues and network stack components, such as IRQ affinity, RPS/RFS, XPS, or NIC flow steering (ntuple).
The data is exported using seq_file interfaces to allow iteration over all controller queues.
--- drivers/nvme/host/core.c | 3 ++ drivers/nvme/host/debugfs.c | 76 ++++++++++++++++++++++++++++++++++++- drivers/nvme/host/nvme.h | 2 + drivers/nvme/host/tcp.c | 54 ++++++++++++++++++++++++++ 4 files changed, 134 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +3 -0
@@ -4316,6 +4316,8 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)if (device_add_disk (ctrl->device, ns->disk, nvme_ns_attr_groups)) goto out_cleanup_ns_from_list; +nvme_debugfs_register (ns->disk); +if (!nvme_ns_head_multipath (ns->head))nvme_add_ns_cdev (ns);
@@ -4399,6 +4401,7 @@ static void nvme_ns_remove(struct nvme_ns *ns)nvme_mpath_remove_sysfs_link (ns); +nvme_debugfs_unregister (ns->disk);del_gendisk (ns->disk);mutex_lock (&ns->ctrl->namespaces_lock);
drivers/nvme/host/debugfs.c +75 -1
94 lines not on this page · raw message
drivers/nvme/host/nvme.h +2 -0
@@ -679,6 +679,8 @@ struct nvme_ctrl_ops {void (*print_device_info)(struct nvme_ctrl *ctrl);bool (*supports_pci_p2pdma)(struct nvme_ctrl *ctrl);unsigned long (*get_virt_boundary)(struct nvme_ctrl *ctrl,bool is_admin); +int (*print_io_queue_info)(struct seq_file *m,struct nvme_ctrl *ctrl, +int qid); };/*
drivers/nvme/host/tcp.c +54 -0
66 lines not on this page · raw message
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v3 2/2] nvme: expose queue information via debugfs
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] nvme-tcp: race with queue teardown in nvme_tcp_print_io_queue_info() - [Medium] nvme: debugfs…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] nvme-tcp: race with queue teardown in nvme_tcp_print_io_queue_info()
- [Medium] nvme: debugfs: nvme_io_queue_info_start() skips queues on resumed reads
- [Low] nvme: debugfs: missing static qualifier on nvme_io_queue_info_seq_ops --
commit 0f7d7347ae3aacfc41bb3e9a63226f587e2b9ffd
Author: Nilay Shroff <[email protected]>
nvme: expose queue information via debugfs
This commit adds an io_queue_info debugfs attribute to expose per-queue information for NVMe controllers using seq_file iteration. For NVMe-TCP, it outputs each I/O queue's bound CPU and TCP socket source and destination address and port.
Quoted 9 lines
drivers/nvme/host/debugfs.c +1
-0@@ -25,6 +25,61 @@ struct nvme_debugfs_ctx {struct nvme_debugfs_attr *attr; }; +static void *nvme_io_queue_info_start (struct seq_file *m,loff_t *pos)
drivers/nvme/host/debugfs.c struct nvme_debugfs_ctx { · line 38
Quoted 10 lines
drivers/nvme/host/debugfs.c +10
-0+{ +struct nvme_debugfs_ctx *ctx = m->private; +struct nvme_ns *ns = ctx->data; +struct nvme_ctrl *ctrl = ns->ctrl; + +/* +* IO queues starts at offset 1. +*/ +return (++*pos < ctrl->queue_count) ? pos :NULL ; +}
[Severity: Medium]
Will unconditionally pre-incrementing *pos in nvme_io_queue_info_start() cause queues to be skipped when seq_read_iter() resumes reading on a subsequent read() call or after a seq_file buffer overflow?
In fs/seq_file.c:seq_read_iter(), the Fill loop advances m->index via m->op->next() before checking whether the user buffer is full or whether m->op->show() overflowed the seq_file buffer:
p = m->op->next(m, p, &m->index);
...
if (!p || IS_ERR(p)) // no next record for us
break;
if (m->count >= iov_iter_count(iter))
break;
err = m->op->show(m, p);
if (err > 0) { // ->show() says "skip it"
m->count = offs;
} else if (err || seq_has_overflowed(m)) {
m->count = offs;
break;
}When the loop stops because the seq_file buffer overflowed or the caller's buffer was filled, m->index has already been incremented by nvme_io_queue_info_next() to the next unshown queue ID. On the next read, seq_read_iter() calls nvme_io_queue_info_start(m, &m->index), which executes ++*pos a second time and skips that queue.
Should nvme_io_queue_info_start() only advance *pos when *pos is 0?
Quoted 23 lines
drivers/nvme/host/debugfs.c +23
-0+ +static void *nvme_io_queue_info_next (struct seq_file *m,void *v,loff_t *pos) +{ +struct nvme_debugfs_ctx *ctx = m->private; +struct nvme_ns *ns = ctx->data; +struct nvme_ctrl *ctrl = ns->ctrl; + +return (++*pos < ctrl->queue_count) ? pos :NULL ; +} + +static void nvme_io_queue_info_stop (struct seq_file *m,void *v) +{ +/* nothing to be done */ +} + +static int nvme_io_queue_info_show (struct seq_file *m,void *v) +{ +struct nvme_debugfs_ctx *ctx = m->private; +struct nvme_ns *ns = ctx->data; +struct nvme_ctrl *ctrl = ns->ctrl; + +if (ctrl->ops->print_io_queue_info) +return ctrl->ops->print_io_queue_info (m, ctrl, *(loff_t *)v);
drivers/nvme/host/debugfs.c struct nvme_debugfs_ctx { · line 71
Quoted 10 lines
drivers/nvme/host/debugfs.c +10
-0+ +return 0 ; +} + +const struct seq_operations nvme_io_queue_info_seq_ops = { + .start = nvme_io_queue_info_start, + .next = nvme_io_queue_info_next, + .stop = nvme_io_queue_info_stop, + .show = nvme_io_queue_info_show +};
[Severity: Low]
Should nvme_io_queue_info_seq_ops be marked static since it is only used within drivers/nvme/host/debugfs.c?
[ ... ]
Quoted 24 lines
drivers/nvme/host/tcp.c +16
-0@@ -2691,6 +2691,59 @@ static void nvme_tcp_free_ctrl(struct nvme_ctrl *nctrl)kfree (ctrl); } +static int nvme_tcp_print_io_queue_info (struct seq_file *m, +struct nvme_ctrl *ctrl,int qid) +{ +int cpu; +struct sockaddr_storage src, dst; +struct nvme_tcp_ctrl *tctrl =to_tcp_ctrl (ctrl); +struct nvme_tcp_queue *queue = &tctrl->queues[qid]; +int ret = -EINVAL ; + +if (!qid || qid >= ctrl->queue_count) +return -EINVAL ; + +if (!test_bit (NVME_TCP_Q_LIVE , &queue->flags)) +return 0 ; + +mutex_lock (&queue->queue_lock);
drivers/nvme/host/tcp.c static void nvme_tcp_free_ctrl(struct nvme_ctrl *nctrl) · line 2719
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+if (!queue->sock) + goto unlock; + + ret =kernel_getsockname (queue->sock, (struct sockaddr *)&src); +if (ret <=0 ) + goto unlock; + + ret =kernel_getpeername (queue->sock, (struct sockaddr *)&dst); +if (ret <=0 ) + goto unlock;
[Severity: High]
Can nvme_tcp_print_io_queue_info() race with queue teardown during controller reset or error recovery?
Holding a reference on struct nvme_ns keeps the namespace and controller allocated, but does not prevent nvme_reset_ctrl_work() or nvme_tcp_error_recovery_work() from running nvme_tcp_teardown_io_queues() while io_queue_info is being read.
In nvme_tcp_stop_queue_nowait(), NVME_TCP_Q_LIVE is cleared under queue->queue_lock while queue->sock remains non-NULL:
mutex_lock(&queue->queue_lock); if (test_and_clear_bit(NVME_TCP_Q_LIVE, &queue->flags)) __nvme_tcp_stop_queue(queue); /* Stopping the queue will disable TLS */ queue->tls_enabled = false; mutex_unlock(&queue->queue_lock);
Shortly after, nvme_tcp_free_io_queues() -> nvme_tcp_free_queue() runs without holding queue->queue_lock, frees queue->sock before clearing the pointer, and destroys queue->queue_lock:
__fput_sync(queue->sock->file); queue->sock = NULL; memalloc_noio_restore(noio_flag); kfree(queue->pdu); mutex_destroy(&queue->send_mutex); mutex_destroy(&queue->queue_lock); mutex_destroy(&queue->pf_cache_lock);
If nvme_tcp_print_io_queue_info() checks test_bit(NVME_TCP_Q_LIVE, &queue->flags) just before teardown, it can acquire queue->queue_lock after nvme_tcp_stop_queue_nowait() unlocks it, see queue->sock != NULL because NVME_TCP_Q_LIVE is not re-checked under the lock, and dereference queue->sock in kernel_getsockname() and kernel_getpeername() concurrently with __fput_sync(queue->sock->file), or touch queue->queue_lock while nvme_tcp_free_queue() destroys it (and reconnect re-initializes it in nvme_tcp_alloc_queue()).
In addition, if !queue->sock is taken under queue->queue_lock (where ret is still -EINVAL) or if kernel_getpeername() returns -ENOTCONN on a shut-down socket, returning a negative error code from the seq_file show callback aborts the entire seq_read_iter() read instead of skipping the stopped queue.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Quoted 9 lines · Nilay Shroff
Following the feedback, the logic for determining the number of fabric I/O queues (nr-io-queues) has been moved to userspace in nvme-cli [2]. As a result, the first two patches from the previous series, which implemented this logic in the kernel driver, have been dropped. This series contains the remaining two patches, which export NVMe/TCP I/O queue information through debugfs to facilitate host-side network tuning.
The first patch introduces debugfs helper functions that can be used by drivers to export debugfs attributes.
What is the expectation, what's the use case here? Should this information be available in a productive system? In this case debugfs might not be the right choice.
Quoted 8 lines · Daniel Wagner
Quoted 7 lines · Nilay Shroff
Following the feedback, the logic for determining the number of fabric I/O queues (nr-io-queues) has been moved to userspace in nvme-cli [2]. As a result, the first two patches from the previous series, which implemented this logic in the kernel driver, have been dropped. This series contains the remaining two patches, which export NVMe/TCP I/O queue information through debugfs to facilitate host-side network tuning.
Quoted 6 lines
Quoted 2 lines
The first patch introduces debugfs helper functions that can be used by drivers to export debugfs attributes.
What is the expectation, what's the use case here? Should this information be available in a productive system? In this case debugfs might not be the right choice.
Ideally, this information should be available on production systems to facilitate network stack tuning.
Would you recommend exposing it through sysfs rather than debugfs? During the last LSFMM, I received feedback suggesting debugfs, so I implemented the interface accordingly in this patch series. However, I'm open to moving it to sysfs if we agree that it is the more appropriate interface.
Thanks,
--Nilay
Quoted 16 lines · Nilay Shroff
Quoted 8 lines · Daniel Wagner
Quoted 7 lines · Nilay Shroff
Following the feedback, the logic for determining the number of fabric I/O queues (nr-io-queues) has been moved to userspace in nvme-cli [2]. As a result, the first two patches from the previous series, which implemented this logic in the kernel driver, have been dropped. This series contains the remaining two patches, which export NVMe/TCP I/O queue information through debugfs to facilitate host-side network tuning.
Quoted 6 lines
Quoted 2 lines
The first patch introduces debugfs helper functions that can be used by drivers to export debugfs attributes.
What is the expectation, what's the use case here? Should this information be available in a productive system? In this case debugfs might not be the right choice.
Quoted 8 lines
Ideally, this information should be available on production systems to facilitate network stack tuning.
Would you recommend exposing it through sysfs rather than debugfs? During the last LSFMM, I received feedback suggesting debugfs, so I implemented the interface accordingly in this patch series. However, I'm open to moving it to sysfs if we agree that it is the more appropriate interface.
Hmm. This matches what netdevs do with their tx/rx queue mapping, and that is in sysfs. Typically I would prefer debugfs for this, but here I really don't mind as it's somewhere in the middle between generally
useful and being a niche usecase.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Quoted 25 lines · Hannes Reinecke
Quoted 16 lines · Nilay Shroff
Quoted 8 lines · Daniel Wagner
Quoted 7 lines · Nilay Shroff
Following the feedback, the logic for determining the number of fabric I/O queues (nr-io-queues) has been moved to userspace in nvme-cli [2]. As a result, the first two patches from the previous series, which implemented this logic in the kernel driver, have been dropped. This series contains the remaining two patches, which export NVMe/TCP I/O queue information through debugfs to facilitate host-side network tuning.
Quoted 6 lines
Quoted 2 lines
The first patch introduces debugfs helper functions that can be used by drivers to export debugfs attributes.
What is the expectation, what's the use case here? Should this information be available in a productive system? In this case debugfs might not be the right choice.
Quoted 8 lines
Ideally, this information should be available on production systems to facilitate network stack tuning.
Would you recommend exposing it through sysfs rather than debugfs? During the last LSFMM, I received feedback suggesting debugfs, so I implemented the interface accordingly in this patch series. However, I'm open to moving it to sysfs if we agree that it is the more appropriate interface.
Quoted 4 lines
Hmm. This matches what netdevs do with their tx/rx queue mapping, and that is in sysfs. Typically I would prefer debugfs for this, but here I really don't mind as it's somewhere in the middle between generally
useful and being a niche usecase.
Alright, then I'll move this under sysfs and spin a new version.
Thanks,
--Nilay
Quoted 1 line · Nilay Shroff
Quoted 1 line
Alright, then I'll move this under sysfs and spin a new version.
FWIW, I can imagine that it might be useful to add some support in nvme-cli, so sysfs would be the choice for this.
Quoted 5 lines · Daniel Wagner
Quoted 1 line · Nilay Shroff
Quoted 1 line
Alright, then I'll move this under sysfs and spin a new version.
FWIW, I can imagine that it might be useful to add some support in nvme-cli, so sysfs would be the choice for this.
Yes agreed!
Thanks,
--Nilay
Quoted 9 lines · Nilay Shroff
Quoted 5 lines · Daniel Wagner
Quoted 1 line · Nilay Shroff
Quoted 1 line
Alright, then I'll move this under sysfs and spin a new version.
FWIW, I can imagine that it might be useful to add some support in nvme-cli, so sysfs would be the choice for this.
Yes agreed!
Yeah, that was my reasoning, too.
So we should store it in sysfs.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich