Skip to contentSkip to conversation

nvme-multipath: introduce latency I/O policy

PATCH Superseded Nilay Shroff 4 participants 14 messages May 20 – Jun 18

Subsystems (3)

v6 of 8 Superseded Latest is v8 →

lore
Share & tools
lore
#0 lore raw

v6 cover nvme-multipath: introduce latency I/O policy

Hi,

This series introduces a new latency I/O policy for NVMe native multipath. Existing policies such as numa, round-robin, and queue-depth are static and do not adapt to real-time transport performance. The numa selects the path closest to the NUMA node of the current CPU, optimizing memory and path locality, but ignores actual path performance. The round-robin distributes I/O evenly across all paths, providing fairness but not performance awareness. The queue-depth reacts to instantaneous queue occupancy, avoiding heavily loaded paths, but does not account for actual latency, throughput, or link speed.

The new latency policy addresses these gaps selecting paths dynamically based on measured I/O latency for both PCIe and fabrics. Latency is derived by passively sampling I/O completions. Each path is assigned a weight proportional to its latency score, and I/Os are then forwarded accordingly. As condition changes (e.g. latency spikes, bandwidth differences), path weights are updated, automatically steering traffic toward better-performing paths.

Early results show reduced tail latency under mixed workloads and improved throughput by exploiting higher-speed links more effectively. For example, with NVMf/TCP using two paths (one throttled with ~30 ms delay), fio results with random read/write/rw workloads (direct I/O) showed:

        numa         round-robin   queue-depth  adaptive
        -----------  -----------   -----------  ---------
READ:   50.0 MiB/s   105 MiB/s     230 MiB/s    350 MiB/s
WRITE:  65.9 MiB/s   125 MiB/s     385 MiB/s    446 MiB/s
RW:     R:30.6 MiB/s R:56.5 MiB/s  R:122 MiB/s  R:175 MiB/s
        W:30.7 MiB/s W:56.5 MiB/s  W:122 MiB/s  W:175 MiB/s

This pathcset includes totla 8 patches:
[PATCH 1/8] block: expose blk_stat_{enable,disable}_accounting()
- Make blk_stat APIs available to block drivers.
- Needed for per-path latency measurement.

[PATCH 2/8] nvme-multipath: pass I/O type to nvme_find_path()
- This is the prep patch which updates nvme_find_path() signature

[PATCH 3/8] nvme-multipath: add latency I/O policy
- Implement path scoring based on latency (EWMA).
- Distribute I/O proportionally to per-path weights.

[PATCH 4/8] nvme: add generic debugfs support
- Introduce generic debugfs support for NVMe module

[PATCH 5/8] nvme-multipath: add debugfs attribute latency_ewma_shift
- Adds a debugfs attribute to control ewma shift

[PATCH 6/8] nvme-multipath: add debugfs attribute latency_batch_timeout
- Adds a debugfs attribute to control latency batch window interval

[PATCH 7/8] nvme-multipath: add debugfs attribute latency_stat
- Add “latency_stat” under per-path and head debugfs directories to expose latency policy state and statistics.

[PATCH 8/8] nvme-multipath: add documentation for latency I/O policy
- Includes documentation for latency I/O multipath policy.

LSFMM discussion:
=================
During lsfmm 2026, it was decided to rename this I/O policy from "adaptive" to "latency". This series reflects that rename.

The discussion at lsfmm also focused extensively on the latency measurement model, including whether latency should be tracked per-CPU or per-NUMA, and whether separate I/O-size buckets should be maintained for different request sizes.

After detailed discussion and evaluation of throughput results, the consensus was to initially measure I/O completion latency on a per-CPU basis. The available performance data showed that the per-CPU implementation already provides sufficient averaging across CPUs while keeping the design relatively simple.

The use of additional I/O-size buckets did not demonstrate meaningful throughput improvement in the general case and would introduce extra complexity into the fast path and accounting logic. As a result, the consensus was to avoid I/O-size bucketing for now and keep the policy focused on per-CPU latency measurement.

If future real-world workloads demonstrate a clear benefit from I/O-size-aware latency accounting, the policy can be extended later to support it.

As ususal, feedback and suggestions are most welcome!

Thanks!

Changes from v5:
- Rename the policy from "adaptive" to "latency". The entire series updates policy names, function names, and variable names accordingly, without introducing any functional changes. (lsfmm discussion)
- The second patch is now splitted into two patches:
Patch #2: prep patch where we pass op_type to nvme_find_path() Patch #3: core patch which introduces latency I/O policy
(Sagi)
- Rename ewma_update() to calc_ewma_update() (Sagi)
Link to v5: https://lore.kernel.org/all/[email protected]/

Changes from v4:
- Added patch #7 which includes the documentation for adaptive I/O policy. (Guixin Liu)
Link to v4: https://lore.kernel.org/all/[email protected]/

Changes from v3:
- Update the adaptive APIs name (which actually enable/disable adaptive policy) to reflect the actual work it does. Also removed the misleading use of "current_path" from the adaptive policy code (Hannes Reinecke)
- Move adaptive_ewma_shift and adaptive_weight_timeout attributes from sysfs to debugfs (Hannes Reinecke)
Link to v3: https://lore.kernel.org/all/[email protected]/

Changes from v2:
- Addede a new patch to allow user to configure EWMA shift through sysfs (Hannes Reinecke)
- Added a new patch to allow user to configure path weight calculation timeout (Hannes Reinecke)
- Distinguish between read/write and other commands (e.g. admin comamnd) and calculate path weight for other commands which is separate from read/write weight. (Hannes Reinecke)
- Normalize per-path weight in the range from 0-128 instead

    of 0-100 (Hannes Reinecke)

- Restructure and optimize adaptive I/O forwarding code to use
one loop instead of two (Hannes Reinecke)
Link to v2: https://lore.kernel.org/all/[email protected]/

Changes from v1:
- Ensure that the completion of I/O occurs on the same CPU as the submitting I/O CPU (Hannes Reinecke)
- Remove adapter link speed from the path weight calculation
(Hannes Reinecke)
- Add adaptive I/O stat under debugfs instead of current sysfs (Hannes Reinecke)
- Move path weight calculation to a workqueue from IO completion code path
Link to v1: https://lore.kernel.org/all/[email protected]/

Nilay Shroff (8):
block: expose blk_stat_{enable,disable}_accounting() to drivers
nvme-multipath: pass I/O type to nvme_find_path()
nvme-multipath: add support for latency I/O policy
nvme: add generic debugfs support
nvme-multipath: add debugfs attribute latency_ewma_shift
nvme-multipath: add debugfs attribute latency_batch_timeout
nvme-multipath: add debugfs attribute latency_stat
nvme-multipath: add documentation for latency I/O policy

 Documentation/admin-guide/nvme-multipath.rst |  19 +
 block/blk-stat.h                             |   4 -
 drivers/nvme/host/Makefile                   |   2 +-
 drivers/nvme/host/core.c                     |  21 +-
 drivers/nvme/host/debugfs.c                  | 345 ++++++++++++++
 drivers/nvme/host/ioctl.c                    |  38 +-
 drivers/nvme/host/multipath.c                | 446 ++++++++++++++++++-
 drivers/nvme/host/nvme.h                     |  84 +++-
 drivers/nvme/host/pr.c                       |   6 +-
 drivers/nvme/host/sysfs.c                    |   2 +-
 include/linux/blk-mq.h                       |   4 +
 11 files changed, 941 insertions(+), 30 deletions(-)
 create mode 100644 drivers/nvme/host/debugfs.c
Signature · 1 line
2.53.0
#1 lore raw

PATCH v6 1/8 block: expose blk_stat_{enable,disable}_accounting() to drivers

The functions blk_stat_enable_accounting() and blk_stat_disable_accounting() are currently exported, but their prototypes are only defined in a private header. Move these prototypes into a common header so that block drivers can directly use these APIs.

---
 block/blk-stat.h       | 4 ----
 include/linux/blk-mq.h | 4 ++++
 2 files changed, 4 insertions(+), 4 deletions(-)
block/blk-stat.h +0 -4
index cc5b66e7ee60..b1614a1ef4ad 100644
@@ -70,10 +70,6 @@ void blk_free_queue_stats(struct blk_queue_stats *);
 
 void blk_stat_add(struct request *rq, u64 now);
 
-/* record time/size info in request but not add a callback */
-void blk_stat_enable_accounting(struct request_queue *q);
-void blk_stat_disable_accounting(struct request_queue *q);
-
 /**
  * blk_stat_alloc_callback() - Allocate a block statistics callback.
  * @timer_fn: Timer callback function.
include/linux/blk-mq.h +4 -0
index 18a2388ba581..88aabc2dd738 100644
@@ -753,6 +753,10 @@ int blk_rq_poll(struct request *rq, struct io_comp_batch *iob,
 
 bool blk_mq_queue_inflight(struct request_queue *q);
 
+/* record time/size info in request but not add a callback */
+void blk_stat_enable_accounting(struct request_queue *q);
+void blk_stat_disable_accounting(struct request_queue *q);
+
 enum {
 	/* return when out of requests */
 	BLK_MQ_REQ_NOWAIT	= (__force blk_mq_req_flags_t)(1 << 0),
Signature · 1 line
2.53.0
#5 lore raw

PATCH v6 2/8 nvme-multipath: pass I/O type to nvme_find_path()

In reply to Nilay Shroff

Currently, nvme_find_path() only accepts an nvme_ns_head argument. However, the upcoming latency-aware I/O policy also needs to know the I/O type (read/write/other) associated with the request in order to make path selection decisions.

Update nvme_find_path() to accept an additional argument describing the I/O type. This patch does not introduce any functional change and only prepares the interface for subsequent latency-policy changes.

---
 drivers/nvme/host/ioctl.c     | 38 ++++++++++++++++++++++++++++++++---
 drivers/nvme/host/multipath.c |  9 +++++----
 drivers/nvme/host/nvme.h      | 19 +++++++++++++++++-
 drivers/nvme/host/pr.c        |  6 ++++--
 drivers/nvme/host/sysfs.c     |  2 +-
 5 files changed, 63 insertions(+), 11 deletions(-)
drivers/nvme/host/ioctl.c +35 -3
index 08889b20e5d8..7a9043350227 100644
@@ -699,18 +699,29 @@ static int nvme_ns_head_ctrl_ioctl(struct nvme_ns *ns, unsigned int cmd,
 int nvme_ns_head_ioctl(struct block_device *bdev, blk_mode_t mode,
 		unsigned int cmd, unsigned long arg)
 {
+	u8 opcode;
 	struct nvme_ns_head *head = bdev->bd_disk->private_data;
 	bool open_for_write = mode & BLK_OPEN_WRITE;
 	void __user *argp = (void __user *)arg;
 	struct nvme_ns *ns;
 	int srcu_idx, ret = -EWOULDBLOCK;
 	unsigned int flags = 0;
+	unsigned int op_type = NVME_STAT_OTHER;
 
 	if (bdev_is_partition(bdev))
 		flags |= NVME_IOCTL_PARTITION;
 
+	if (cmd == NVME_IOCTL_SUBMIT_IO) {
+		if (get_user(opcode, (u8 *)argp))
+			return -EFAULT;
+		if (opcode == nvme_cmd_write)
+			op_type = NVME_STAT_WRITE;
+		else if (opcode == nvme_cmd_read)
+			op_type = NVME_STAT_READ;
+	}
+
 	srcu_idx = srcu_read_lock(&head->srcu);
-	ns = nvme_find_path(head);
+	ns = nvme_find_path(head, op_type);
 	if (!ns)
 		goto out_unlock;
 
@@ -732,6 +743,7 @@ int nvme_ns_head_ioctl(struct block_device *bdev, blk_mode_t mode,
 long nvme_ns_head_chr_ioctl(struct file *file, unsigned int cmd,
 		unsigned long arg)
 {
+	u8 opcode;
 	bool open_for_write = file->f_mode & FMODE_WRITE;
 	struct cdev *cdev = file_inode(file)->i_cdev;
 	struct nvme_ns_head *head =
@@ -739,9 +751,19 @@ long nvme_ns_head_chr_ioctl(struct file *file, unsigned int cmd,
 	void __user *argp = (void __user *)arg;
 	struct nvme_ns *ns;
 	int srcu_idx, ret = -EWOULDBLOCK;
+	unsigned int op_type = NVME_STAT_OTHER;
+
+	if (cmd == NVME_IOCTL_SUBMIT_IO) {
+		if (get_user(opcode, (u8 *)argp))
+			return -EFAULT;
+		if (opcode == nvme_cmd_write)
+			op_type = NVME_STAT_WRITE;
+		else if (opcode == nvme_cmd_read)
+			op_type = NVME_STAT_READ;
+	}
 
 	srcu_idx = srcu_read_lock(&head->srcu);
-	ns = nvme_find_path(head);
+	ns = nvme_find_path(head, op_type);
 	if (!ns)
 		goto out_unlock;
 
@@ -761,7 +783,17 @@ int nvme_ns_head_chr_uring_cmd(struct io_uring_cmd *ioucmd,
 	struct cdev *cdev = file_inode(ioucmd->file)->i_cdev;
 	struct nvme_ns_head *head = container_of(cdev, struct nvme_ns_head, cdev);
 	int srcu_idx = srcu_read_lock(&head->srcu);
-	struct nvme_ns *ns = nvme_find_path(head);
+	const struct nvme_uring_cmd *cmd = io_uring_sqe128_cmd(ioucmd->sqe,
+						struct nvme_uring_cmd);
+	__u8 opcode = READ_ONCE(cmd->opcode);
+	unsigned int op_type = NVME_STAT_OTHER;
+
+	if (opcode == nvme_cmd_write)
+		op_type = NVME_STAT_WRITE;
+	else if (opcode == nvme_cmd_read)
+		op_type = NVME_STAT_READ;
+
+	struct nvme_ns *ns = nvme_find_path(head, op_type);
 	int ret = -EINVAL;
 
 	if (ns)
drivers/nvme/host/multipath.c +5 -4
index 263161cb8ac0..90f449780e72 100644
@@ -446,7 +446,8 @@ static struct nvme_ns *nvme_numa_path(struct nvme_ns_head *head)
 	return ns;
 }
 
-inline struct nvme_ns *nvme_find_path(struct nvme_ns_head *head)
+inline struct nvme_ns *nvme_find_path(struct nvme_ns_head *head,
+		unsigned int op_type)
 {
 	switch (READ_ONCE(head->subsys->iopolicy)) {
 	case NVME_IOPOLICY_QD:
@@ -508,7 +509,7 @@ static void nvme_ns_head_submit_bio(struct bio *bio)
 		return;
 
 	srcu_idx = srcu_read_lock(&head->srcu);
-	ns = nvme_find_path(head);
+	ns = nvme_find_path(head, nvme_data_dir(bio_op(bio)));
 	if (likely(ns)) {
 		bio_set_dev(bio, ns->disk->part0);
 		bio->bi_opf |= REQ_NVME_MPATH;
@@ -550,7 +551,7 @@ static int nvme_ns_head_get_unique_id(struct gendisk *disk, u8 id[16],
 	int srcu_idx, ret = -EWOULDBLOCK;
 
 	srcu_idx = srcu_read_lock(&head->srcu);
-	ns = nvme_find_path(head);
+	ns = nvme_find_path(head, NVME_STAT_OTHER);
 	if (ns)
 		ret = nvme_ns_get_unique_id(ns, id, type);
 	srcu_read_unlock(&head->srcu, srcu_idx);
@@ -566,7 +567,7 @@ static int nvme_ns_head_report_zones(struct gendisk *disk, sector_t sector,
 	int srcu_idx, ret = -EWOULDBLOCK;
 
 	srcu_idx = srcu_read_lock(&head->srcu);
-	ns = nvme_find_path(head);
+	ns = nvme_find_path(head, NVME_STAT_OTHER);
 	if (ns)
 		ret = nvme_ns_report_zones(ns, sector, nr_zones, args);
 	srcu_read_unlock(&head->srcu, srcu_idx);
drivers/nvme/host/nvme.h +18 -1
index ccd5e05dac98..39e986e5f184 100644
@@ -514,6 +514,13 @@ struct nvme_ns_ids {
 	u8	csi;
 };
 
+enum nvme_stat_group {
+	NVME_STAT_READ,
+	NVME_STAT_WRITE,
+	NVME_STAT_OTHER,
+	NVME_NUM_STAT_GROUPS
+};
+
 /*
  * Anchor structure for namespaces.  There is one for each namespace in a
  * NVMe subsystem that any of our controllers can see, and the namespace
@@ -1017,7 +1024,17 @@ extern const struct attribute_group *nvme_dev_attr_groups[];
 extern const struct block_device_operations nvme_bdev_ops;
 
 void nvme_delete_ctrl_sync(struct nvme_ctrl *ctrl);
-struct nvme_ns *nvme_find_path(struct nvme_ns_head *head);
+struct nvme_ns *nvme_find_path(struct nvme_ns_head *head, unsigned int op_type);
+static inline int nvme_data_dir(const enum req_op op)
+{
+	if (op == REQ_OP_READ)
+		return NVME_STAT_READ;
+	else if (op_is_write(op))
+		return NVME_STAT_WRITE;
+	else
+		return NVME_STAT_OTHER;
+}
+
 #ifdef CONFIG_NVME_MULTIPATH
 static inline bool nvme_ctrl_use_ana(struct nvme_ctrl *ctrl)
 {
drivers/nvme/host/pr.c +4 -2
index fe7dbe264815..80e106f151d2 100644
@@ -53,10 +53,12 @@ static int nvme_send_ns_head_pr_command(struct block_device *bdev,
 		struct nvme_command *c, void *data, unsigned int data_len)
 {
 	struct nvme_ns_head *head = bdev->bd_disk->private_data;
-	int srcu_idx = srcu_read_lock(&head->srcu);
-	struct nvme_ns *ns = nvme_find_path(head);
+	int srcu_idx;
+	struct nvme_ns *ns;
 	int ret = -EWOULDBLOCK;
 
+	srcu_idx = srcu_read_lock(&head->srcu);
+	ns = nvme_find_path(head, NVME_STAT_OTHER);
 	if (ns) {
 		c->common.nsid = cpu_to_le32(ns->head->ns_id);
 		ret = nvme_submit_sync_cmd(ns->queue, c, data, data_len);
drivers/nvme/host/sysfs.c +1 -1
index e59758616f27..b6b718bd310d 100644
@@ -194,7 +194,7 @@ static int ns_head_update_nuse(struct nvme_ns_head *head)
 		return 0;
 
 	srcu_idx = srcu_read_lock(&head->srcu);
-	ns = nvme_find_path(head);
+	ns = nvme_find_path(head, NVME_STAT_OTHER);
 	if (!ns)
 		goto out_unlock;
 
Signature · 1 line
2.53.0
#7 lore raw

PATCH v6 3/8 nvme-multipath: add support for latency I/O policy

In reply to Nilay Shroff

This commit introduces a new I/O policy named "latency". Users can configure it by writing "latency" to "/sys/class/nvme-subsystem/nvme- subsystemX/iopolicy"

The "latency" policy dynamically distributes I/O based on measured I/O completion latency. The main idea is to calculate latency for each path, derive a weight, and then proportionally forward I/O according to those weights.

To ensure scalability, path latency is measured per-CPU. Each CPU maintains its own statistics, and I/O forwarding uses these per-CPU values. Every ~15 seconds, a simple average latency of per-CPU batched samples are computed and fed into an Exponentially Weighted Moving Average (EWMA):

avg_latency = div_u64(batch, batch_count);
new_ewma_latency = (prev_ewma_latency * (WEIGHT-1) + avg_latency)/WEIGHT

With WEIGHT = 8, this assigns 7/8 (~87.5%) weight to the previous latency value and 1/8 (~12.5%) to the most recent latency. This smoothing reduces jitter, adapts quickly to changing conditions, avoids storing historical samples, and works well for both low and high I/O rates. Path weights are then derived from the smoothed (EWMA) latency as follows (example with two paths A and B):

path_A_score = NSEC_PER_SEC / path_A_ewma_latency path_B_score = NSEC_PER_SEC / path_B_ewma_latency total_score = path_A_score + path_B_score

path_A_weight = (path_A_score * 64) / total_score path_B_weight = (path_B_score * 64) / total_score

where:
- path_X_ewma_latency is the smoothed latency of a path in nanoseconds
- NSEC_PER_SEC is used as a scaling factor since valid latencies are < 1 second
- weights are normalized to a 0–64 scale across all paths.

Path credits are refilled based on this weight, with one credit consumed per I/O. When all credits are consumed, the credits are refilled again based on the current weight. This ensures that I/O is distributed across paths proportionally to their calculated weight.

---
 drivers/nvme/host/core.c      |  18 +-
 drivers/nvme/host/multipath.c | 430 +++++++++++++++++++++++++++++++++-
 drivers/nvme/host/nvme.h      |  53 ++++-
 3 files changed, 487 insertions(+), 14 deletions(-)
drivers/nvme/host/core.c +15 -3
index c3032d6ad6b1..82db59927556 100644
@@ -677,6 +677,9 @@ static void nvme_free_ns_head(struct kref *ref)
 	cleanup_srcu_struct(&head->srcu);
 	nvme_put_subsystem(head->subsys);
 	kfree(head->plids);
+#ifdef CONFIG_NVME_MULTIPATH
+	free_percpu(head->latency_path);
+#endif
 	kfree(head);
 }
 
@@ -694,6 +697,7 @@ static void nvme_free_ns(struct kref *kref)
 {
 	struct nvme_ns *ns = container_of(kref, struct nvme_ns, kref);
 
+	nvme_free_ns_stat(ns);
 	put_disk(ns->disk);
 	nvme_put_ns_head(ns->head);
 	nvme_put_ctrl(ns->ctrl);
@@ -4174,6 +4178,9 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)
 	if (nvme_init_ns_head(ns, info))
 		goto out_cleanup_disk;
 
+	if (nvme_alloc_ns_stat(ns))
+		goto out_unlink_ns;
+
 	/*
 	 * If multipathing is enabled, the device name for all disks and not
 	 * just those that represent shared namespaces needs to be based on the
@@ -4198,7 +4205,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)
 	}
 
 	if (nvme_update_ns_info(ns, info))
-		goto out_unlink_ns;
+		goto out_free_ns_stat;
 
 	mutex_lock(&ctrl->namespaces_lock);
 	/*
@@ -4207,7 +4214,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)
 	 */
 	if (test_bit(NVME_CTRL_FROZEN, &ctrl->flags)) {
 		mutex_unlock(&ctrl->namespaces_lock);
-		goto out_unlink_ns;
+		goto out_free_ns_stat;
 	}
 	nvme_ns_add_to_ctrl_list(ns);
 	mutex_unlock(&ctrl->namespaces_lock);
@@ -4231,6 +4238,8 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)
 	list_del_rcu(&ns->list);
 	mutex_unlock(&ctrl->namespaces_lock);
 	synchronize_srcu(&ctrl->srcu);
+out_free_ns_stat:
+	nvme_free_ns_stat(ns);
  out_unlink_ns:
 	mutex_lock(&ctrl->subsys->lock);
 	list_del_rcu(&ns->siblings);
@@ -4270,10 +4279,13 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 
 	/*
 	 * Ensure that !NVME_NS_READY is seen by other threads to prevent
-	 * this ns going back into current_path.
+	 * this ns going back into current_path/latency_path.
 	 */
 	synchronize_srcu(&ns->head->srcu);
 
+	if (test_bit(NVME_NS_PATH_STAT, &ns->flags))
+		nvme_cancel_ns_latency_weight_work(ns);
+
 	/* wait for concurrent submissions */
 	if (nvme_mpath_clear_current_path(ns))
 		synchronize_srcu(&ns->head->srcu);
drivers/nvme/host/multipath.c +420 -10

530 lines not on this page · raw message

drivers/nvme/host/nvme.h +52 -1

107 lines not on this page · raw message

Signature · 1 line
2.53.0
#8 lore raw

PATCH v6 4/8 nvme: add generic debugfs support

In reply to Nilay Shroff

Add generic infrastructure for creating and managing debugfs files in the NVMe module. This introduces helper APIs that allow NVMe drivers to register and unregister debugfs entries, along with a reusable attribute structure for defining new debugfs files.

The implementation uses seq_file interfaces to safely expose per-NS and per-NS-head statistics, while supporting both simple show callbacks and full seq_operations.

---
 drivers/nvme/host/Makefile    |   2 +-
 drivers/nvme/host/core.c      |   3 +
 drivers/nvme/host/debugfs.c   | 138 ++++++++++++++++++++++++++++++++++
 drivers/nvme/host/multipath.c |   2 +
 drivers/nvme/host/nvme.h      |  10 +++
 5 files changed, 154 insertions(+), 1 deletion(-)
 create mode 100644 drivers/nvme/host/debugfs.c
drivers/nvme/host/Makefile +1 -1
index 6414ec968f99..7962dfc3b2ad 100644
@@ -10,7 +10,7 @@ obj-$(CONFIG_NVME_FC)			+= nvme-fc.o
 obj-$(CONFIG_NVME_TCP)			+= nvme-tcp.o
 obj-$(CONFIG_NVME_APPLE)		+= nvme-apple.o
 
-nvme-core-y				+= core.o ioctl.o sysfs.o pr.o
+nvme-core-y				+= core.o ioctl.o sysfs.o pr.o debugfs.o
 nvme-core-$(CONFIG_NVME_VERBOSE_ERRORS)	+= constants.o
 nvme-core-$(CONFIG_TRACING)		+= trace.o
 nvme-core-$(CONFIG_NVME_MULTIPATH)	+= multipath.o
drivers/nvme/host/core.c +3 -0
index 82db59927556..ca2be0472e3f 100644
@@ -4224,6 +4224,8 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)
 	if (device_add_disk(ctrl->device, ns->disk, nvme_ns_attr_groups))
 		goto out_cleanup_ns_from_list;
 
+	nvme_debugfs_register(ns->disk);
+
 	if (!nvme_ns_head_multipath(ns->head))
 		nvme_add_ns_cdev(ns);
 
@@ -4307,6 +4309,7 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 
 	nvme_mpath_remove_sysfs_link(ns);
 
+	nvme_debugfs_unregister(ns->disk);
 	del_gendisk(ns->disk);
 
 	mutex_lock(&ns->ctrl->namespaces_lock);
drivers/nvme/host/debugfs.c new file +138 -0

138 lines not on this page · raw message

drivers/nvme/host/multipath.c +2 -0
index e1089a75a6fe..541d12b73b74 100644
@@ -1078,6 +1078,7 @@ static void nvme_remove_head(struct nvme_ns_head *head)
 
 		nvme_cdev_del(&head->cdev, &head->cdev_device);
 		synchronize_srcu(&head->srcu);
+		nvme_debugfs_unregister(head->disk);
 		del_gendisk(head->disk);
 	}
 	nvme_put_ns_head(head);
@@ -1187,6 +1188,7 @@ static void nvme_mpath_set_live(struct nvme_ns *ns)
 		}
 		nvme_add_ns_head_cdev(head);
 		queue_work(nvme_wq, &head->partition_scan_work);
+		nvme_debugfs_register(head->disk);
 	}
 
 	nvme_mpath_add_sysfs_link(ns->head);
drivers/nvme/host/nvme.h +10 -0
index 22e54c74c1a6..7ee9689ce07e 100644
@@ -1067,6 +1067,16 @@ static inline int nvme_data_dir(const enum req_op op)
 		return NVME_STAT_OTHER;
 }
 
+void nvme_debugfs_register(struct gendisk *disk);
+static inline void nvme_debugfs_unregister(struct gendisk *disk)
+{
+	/*
+	 * Nothing to do for now. When the request queue is unregistered,
+	 * all files under q->debugfs_dir are recursively deleted.
+	 * This is just a placeholder; the compiler will optimize it out.
+	 */
+}
+
 #ifdef CONFIG_NVME_MULTIPATH
 static inline bool nvme_ctrl_use_ana(struct nvme_ctrl *ctrl)
 {
Signature · 1 line
2.53.0
#9 lore raw

PATCH v6 5/8 nvme-multipath: add debugfs attribute latency_ewma_shift

In reply to Nilay Shroff

By default, the EWMA (Exponentially Weighted Moving Average) shift value, used for storing latency samples for latency iopolicy, is set to 3. The EWMA is calculated using the following formula:

ewma = (old * ((1 << ewma_shift) - 1) + new) >> ewma_shift;

The default value of 3 assigns ~87.5% weight to the existing EWMA value and ~12.5% weight to the new latency sample. This provides a stable average that smooths out short-term variations.

However, different workloads may require faster or slower adaptation to changing conditions. This commit introduces a new debugfs attribute, latency_ewma_shift, allowing users to tune the weighting factor.

For example:
- latency_ewma_shift = 2 => 75% old, 25% new
- latency_ewma_shift = 1 => 50% old, 50% new
- latency_ewma_shift = 0 => 0% old, 100% new

---
 drivers/nvme/host/debugfs.c   | 46 +++++++++++++++++++++++++++++++++++
 drivers/nvme/host/multipath.c |  9 ++++---
 drivers/nvme/host/nvme.h      |  1 +
 3 files changed, 52 insertions(+), 4 deletions(-)
drivers/nvme/host/debugfs.c +46 -0
index 26a50566e4a1..4371d7aafae8 100644
@@ -105,8 +105,54 @@ static const struct file_operations nvme_debugfs_fops = {
 	.release = nvme_debugfs_release,
 };
 
+#ifdef CONFIG_NVME_MULTIPATH
+static int nvme_latency_ewma_shift_show(void *data, struct seq_file *m)
+{
+	struct nvme_ns_head *head = data;
+
+	seq_printf(m, "%u\n", READ_ONCE(head->latency_ewma_shift));
+	return 0;
+}
+
+static ssize_t nvme_latency_ewma_shift_store(void *data,
+		const char __user *ubuf, size_t count, loff_t *ppos)
+{
+	struct nvme_ns_head *head = data;
+	char kbuf[8];
+	u32 res;
+	int ret;
+	size_t len;
+	char *arg;
+
+	len = min(sizeof(kbuf) - 1, count);
+
+	if (copy_from_user(kbuf, ubuf, len))
+		return -EFAULT;
+
+	kbuf[len] = '\0';
+	arg = strstrip(kbuf);
+
+	ret = kstrtou32(arg, 0, &res);
+	if (ret)
+		return ret;
+
+	/*
+	 * Values greater than 8 are nonsensical, as they effectively assign
+	 * zero weight to new samples.
+	 */
+	if (res > 8)
+		return -EINVAL;
+
+	WRITE_ONCE(head->latency_ewma_shift, res);
+	return count;
+}
+#endif
 
 static const struct nvme_debugfs_attr nvme_mpath_debugfs_attrs[] = {
+#ifdef CONFIG_NVME_MULTIPATH
+	{"latency_ewma_shift", 0600, nvme_latency_ewma_shift_show,
+			nvme_latency_ewma_shift_store},
+#endif
 	{},
 };
 
drivers/nvme/host/multipath.c +5 -4
index 541d12b73b74..3e76e07a0376 100644
@@ -281,10 +281,9 @@ static void nvme_mpath_weight_work(struct work_struct *weight_work)
  * For instance, with EWMA_SHIFT = 3, this assigns 7/8 (~87.5 %) weight to
  * the existing/old ewma and 1/8 (~12.5%) weight to the new sample.
  */
-static inline u64 calc_ewma_update(u64 old, u64 new)
+static inline u64 calc_ewma_update(u64 old, u64 new, u32 ewma_shift)
 {
-	return (old * ((1 << NVME_DEFAULT_LATENCY_EWMA_SHIFT) - 1)
-			+ new) >> NVME_DEFAULT_LATENCY_EWMA_SHIFT;
+	return (old * ((1 << ewma_shift) - 1) + new) >> ewma_shift;
 }
 
 static void nvme_mpath_add_sample(struct request *rq, struct nvme_ns *ns)
@@ -375,7 +374,8 @@ static void nvme_mpath_add_sample(struct request *rq, struct nvme_ns *ns)
 		if (unlikely(!stat->slat_ns))
 			WRITE_ONCE(stat->slat_ns, avg_lat_ns);
 		else {
-			slat_ns = calc_ewma_update(stat->slat_ns, avg_lat_ns);
+			slat_ns = calc_ewma_update(stat->slat_ns, avg_lat_ns,
+					READ_ONCE(head->latency_ewma_shift));
 			WRITE_ONCE(stat->slat_ns, slat_ns);
 		}
 
@@ -1113,6 +1113,7 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)
 	INIT_WORK(&head->partition_scan_work, nvme_partition_scan_work);
 	INIT_DELAYED_WORK(&head->remove_work, nvme_remove_head_work);
 	head->delayed_removal_secs = 0;
+	head->latency_ewma_shift = NVME_DEFAULT_LATENCY_EWMA_SHIFT;
 
 	/*
 	 * If "multipath_always_on" is enabled, a multipath node is added
drivers/nvme/host/nvme.h +1 -0
index 7ee9689ce07e..40009c024ab8 100644
@@ -599,6 +599,7 @@ struct nvme_ns_head {
 	unsigned int		delayed_removal_secs;
 
 	struct nvme_ns * __percpu	*latency_path;
+	u32				latency_ewma_shift;
 
 #define NVME_NSHEAD_DISK_LIVE		0
 #define NVME_NSHEAD_QUEUE_IF_NO_PATH	1
Signature · 1 line
2.53.0
#10 lore raw

PATCH v6 6/8 nvme-multipath: add debugfs attribute latency_batch_timeout

In reply to Nilay Shroff

By default, the latency I/O policy accumulates latency samples over a 15-second window. When this window expires, the driver computes the average latency and updates the smoothed (EWMA) latency value. The path weight is then recalculated based on this data.

A 15-second window provides a good balance for most workloads, as it helps smooth out transient latency spikes and produces a more stable path weight profile. However, some workloads may benefit from faster or slower adaptation to changing latency conditions.

This commit introduces a new debugfs attribute, latency_batch_timeout, which allows users to configure the latency batch window and thus path weight calculation interval based on their workload requirements.

---
 drivers/nvme/host/debugfs.c   | 37 +++++++++++++++++++++++++++++++++++
 drivers/nvme/host/multipath.c |  8 ++++++--
 drivers/nvme/host/nvme.h      |  1 +
 3 files changed, 44 insertions(+), 2 deletions(-)
drivers/nvme/host/debugfs.c +37 -0
index 4371d7aafae8..63b0ad5d105b 100644
@@ -146,12 +146,49 @@ static ssize_t nvme_latency_ewma_shift_store(void *data,
 	WRITE_ONCE(head->latency_ewma_shift, res);
 	return count;
 }
+
+static int nvme_latency_batch_timeout_show(void *data, struct seq_file *m)
+{
+	struct nvme_ns_head *head = data;
+
+	seq_printf(m, "%llu\n",
+		div_u64(READ_ONCE(head->latency_batch_timeout), NSEC_PER_SEC));
+	return 0;
+}
+
+static ssize_t nvme_latency_batch_timeout_store(void *data,
+		const char __user *ubuf, size_t count, loff_t *ppos)
+{
+	struct nvme_ns_head *head = data;
+	char kbuf[8];
+	u32 res;
+	int ret;
+	size_t len;
+	char *arg;
+
+	len = min(sizeof(kbuf) - 1, count);
+
+	if (copy_from_user(kbuf, ubuf, len))
+		return -EFAULT;
+
+	kbuf[len] = '\0';
+	arg = strstrip(kbuf);
+
+	ret = kstrtou32(arg, 0, &res);
+	if (ret)
+		return ret;
+
+	WRITE_ONCE(head->latency_batch_timeout, res * NSEC_PER_SEC);
+	return count;
+}
 #endif
 
 static const struct nvme_debugfs_attr nvme_mpath_debugfs_attrs[] = {
 #ifdef CONFIG_NVME_MULTIPATH
 	{"latency_ewma_shift", 0600, nvme_latency_ewma_shift_show,
 			nvme_latency_ewma_shift_store},
+	{"latency_batch_timeout", 0600, nvme_latency_batch_timeout_show,
+			nvme_latency_batch_timeout_store},
 #endif
 	{},
 };
drivers/nvme/host/multipath.c +6 -2
index 3e76e07a0376..aa817bfa4b81 100644
@@ -349,8 +349,11 @@ static void nvme_mpath_add_sample(struct request *rq, struct nvme_ns *ns)
 	stat->batch_count++;
 	stat->nr_samples++;
 
-	if (now > stat->last_batch_ts && ((now - stat->last_batch_ts) >=
-			NVME_DEFAULT_LATENCY_BATCH_TIMEOUT)) {
+	if (now > stat->last_batch_ts) {
+		u64 timeout = READ_ONCE(head->latency_batch_timeout);
+
+		if ((now - stat->last_batch_ts) < timeout)
+			return;
 
 		/*
 		 * Find simple average latency for the last epoch (~15 sec
@@ -1114,6 +1117,7 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)
 	INIT_DELAYED_WORK(&head->remove_work, nvme_remove_head_work);
 	head->delayed_removal_secs = 0;
 	head->latency_ewma_shift = NVME_DEFAULT_LATENCY_EWMA_SHIFT;
+	head->latency_batch_timeout = NVME_DEFAULT_LATENCY_BATCH_TIMEOUT;
 
 	/*
 	 * If "multipath_always_on" is enabled, a multipath node is added
drivers/nvme/host/nvme.h +1 -0
index 40009c024ab8..a694dd091a16 100644
@@ -600,6 +600,7 @@ struct nvme_ns_head {
 
 	struct nvme_ns * __percpu	*latency_path;
 	u32				latency_ewma_shift;
+	u64				latency_batch_timeout;
 
 #define NVME_NSHEAD_DISK_LIVE		0
 #define NVME_NSHEAD_QUEUE_IF_NO_PATH	1
Signature · 1 line
2.53.0
#11 lore raw

PATCH v6 7/8 nvme-multipath: add debugfs attribute latency_stat

In reply to Nilay Shroff

This commit introduces a new debugfs attribute, "latency_stat", under both per-path and head debugfs directories (defined under /sys/kernel/ debug/block/). This attribute provides visibility into the internal state of the latency I/O policy to aid in debugging and performance analysis.

For per-path entries, "latency_stat" reports the corresponding path statistics such as I/O weight, selection count, processed samples, and ignored samples.

For head entries, it reports per-CPU statistics for each reachable path, including I/O weight, path score, smoothed (EWMA) latency, selection count, processed samples, and ignored samples.

These additions enhance observability of the I/O path selection behavior and help diagnose imbalance or instability in multipath performance.

---
 drivers/nvme/host/debugfs.c | 124 ++++++++++++++++++++++++++++++++++++
 1 file changed, 124 insertions(+)
drivers/nvme/host/debugfs.c +124 -0
index 63b0ad5d105b..b4bc7c710050 100644
@@ -181,6 +181,126 @@ static ssize_t nvme_latency_batch_timeout_store(void *data,
 	WRITE_ONCE(head->latency_batch_timeout, res * NSEC_PER_SEC);
 	return count;
 }
+
+static void *nvme_mpath_latency_stat_start(struct seq_file *m, loff_t *pos)
+{
+	struct nvme_ns *ns;
+	struct nvme_debugfs_ctx *ctx = m->private;
+	struct nvme_ns_head *head = ctx->data;
+
+	if (!head->disk)
+		return NULL;
+
+	/* Remember srcu index, so we can unlock later. */
+	ctx->srcu_idx = srcu_read_lock(&head->srcu);
+	ns = list_first_or_null_rcu(&head->list, struct nvme_ns, siblings);
+
+	while (*pos && ns) {
+		ns = list_next_or_null_rcu(&head->list, &ns->siblings,
+				struct nvme_ns, siblings);
+		(*pos)--;
+	}
+
+	return ns;
+}
+
+static void *nvme_mpath_latency_stat_next(struct seq_file *m, void *v,
+		loff_t *pos)
+{
+	struct nvme_ns *ns = v;
+	struct nvme_debugfs_ctx *ctx = m->private;
+	struct nvme_ns_head *head = ctx->data;
+
+	(*pos)++;
+
+	return list_next_or_null_rcu(&head->list, &ns->siblings,
+			struct nvme_ns, siblings);
+}
+
+static void nvme_mpath_latency_stat_stop(struct seq_file *m, void *v)
+{
+	struct nvme_debugfs_ctx *ctx = m->private;
+	struct nvme_ns_head *head = ctx->data;
+	int srcu_idx = ctx->srcu_idx;
+
+	if (!head->disk)
+		return;
+
+	srcu_read_unlock(&head->srcu, srcu_idx);
+}
+
+static int nvme_mpath_latency_stat_show(struct seq_file *m, void *v)
+{
+	int i, cpu;
+	struct nvme_path_lat_stat *stat;
+	struct nvme_ns *ns = v;
+
+	seq_printf(m, "%s:\n", ns->disk->disk_name);
+	for_each_online_cpu(cpu) {
+		seq_printf(m, "cpu %d : ", cpu);
+		for (i = 0; i < NVME_NUM_STAT_GROUPS; i++) {
+			stat = &per_cpu_ptr(ns->path_lat, cpu)[i].stat;
+			seq_printf(m, "%u %u %llu %llu %llu %llu %llu ",
+				stat->weight, stat->credit, stat->score,
+				stat->slat_ns, stat->sel,
+				stat->nr_samples, stat->nr_ignored);
+		}
+		seq_putc(m, '\n');
+	}
+	return 0;
+}
+
+static const struct seq_operations nvme_mpath_latency_stat_seq_ops = {
+	.start = nvme_mpath_latency_stat_start,
+	.next  = nvme_mpath_latency_stat_next,
+	.stop  = nvme_mpath_latency_stat_stop,
+	.show  = nvme_mpath_latency_stat_show
+};
+
+static void nvme_latency_stat_read_all(struct nvme_ns *ns,
+		struct nvme_path_lat_stat *batch)
+{
+	int i, cpu;
+	u32 ncpu[NVME_NUM_STAT_GROUPS] = {0};
+	struct nvme_path_lat_stat *stat;
+
+	for_each_online_cpu(cpu) {
+		for (i = 0; i < NVME_NUM_STAT_GROUPS; i++) {
+			stat = &per_cpu_ptr(ns->path_lat, cpu)[i].stat;
+			batch[i].sel += stat->sel;
+			batch[i].nr_samples += stat->nr_samples;
+			batch[i].nr_ignored += stat->nr_ignored;
+			batch[i].weight += stat->weight;
+			if (stat->weight)
+				ncpu[i]++;
+		}
+	}
+
+	for (i = 0; i < NVME_NUM_STAT_GROUPS; i++) {
+		if (!ncpu[i])
+			continue;
+		batch[i].weight = DIV_U64_ROUND_CLOSEST(batch[i].weight,
+				ncpu[i]);
+	}
+}
+
+static int nvme_ns_latency_stat_show(void *data, struct seq_file *m)
+{
+	int i;
+	struct nvme_path_lat_stat stat[NVME_NUM_STAT_GROUPS] = {0};
+	struct nvme_ns *ns = (struct nvme_ns *)data;
+
+	if (!ns->head->disk)
+		return 0;
+
+	nvme_latency_stat_read_all(ns, stat);
+	for (i = 0; i < NVME_NUM_STAT_GROUPS; i++) {
+		seq_printf(m, "%u %llu %llu %llu ",
+			stat[i].weight, stat[i].sel,
+			stat[i].nr_samples, stat[i].nr_ignored);
+	}
+	return 0;
+}
 #endif
 
 static const struct nvme_debugfs_attr nvme_mpath_debugfs_attrs[] = {
@@ -189,11 +309,15 @@ static const struct nvme_debugfs_attr nvme_mpath_debugfs_attrs[] = {
 			nvme_latency_ewma_shift_store},
 	{"latency_batch_timeout", 0600, nvme_latency_batch_timeout_show,
 			nvme_latency_batch_timeout_store},
+	{"latency_stat", 0400, .seq_ops = &nvme_mpath_latency_stat_seq_ops},
 #endif
 	{},
 };
 
 static const struct nvme_debugfs_attr nvme_ns_debugfs_attrs[] = {
+#ifdef CONFIG_NVME_MULTIPATH
+	{"latency_stat", 0400, nvme_ns_latency_stat_show},
+#endif
 	{},
 };
 
Signature · 1 line
2.53.0
#12 lore raw

PATCH v6 8/8 nvme-multipath: add documentation for latency I/O policy

In reply to Nilay Shroff

Update the nvme-multipath documentation to describe the latency I/O policy, its behavior, and when it is suitable for use.

---
 Documentation/admin-guide/nvme-multipath.rst | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
Documentation/admin-guide/nvme-multipath.rst +19 -0
index 97ca1ccef459..41a8054638ff 100644
@@ -70,3 +70,22 @@ When to use the queue-depth policy:
   1. High load with small I/Os: Effectively balances load across paths when
      the load is high, and I/O operations consist of small, relatively
      fixed-sized requests.
+
+Latency
+--------
+
+The latency policy manages I/O requests based on path latency. It periodically
+calculates a weight for each path and distributes I/O accordingly. Paths with
+higher latency receive lower weights, resulting in fewer I/O requests being sent
+to them, while paths with lower latency handle a proportionally larger share of
+the I/O load.
+
+When to use the latency policy
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. Homogeneous Path Performance: Utilizes all available paths efficiently when
+   their performance characteristics (e.g., latency, bandwidth) are similar.
+
+2. Heterogeneous Path Performance: Dynamically distributes I/O based on per-path
+   performance characteristics. Paths with lower latency receive a higher share
+   of I/O compared to those with higher latency.
Signature · 1 line
2.53.0
#2 lore raw

Re: [PATCHv6 1/8] block: expose blk_stat_{enable,disable}_accounting() to drivers

In reply to Nilay Shroff
Quoted 11 lines · Nilay Shroff

The functions blk_stat_enable_accounting() and blk_stat_disable_accounting() are currently exported, but their prototypes are only defined in a private header. Move these prototypes into a common header so that block drivers can directly use these APIs.

---
  block/blk-stat.h       | 4 ----
  include/linux/blk-mq.h | 4 ++++

struct request_queue comes from blkdev.h, so maybe that would be a more appropriate place to locate the prototype.

Thanks,
John

#3 lore raw

Re: [PATCHv6 1/8] block: expose blk_stat_{enable,disable}_accounting() to drivers

Quoted 6 lines · John Garry
Quoted 5 lines · Nilay Shroff

The functions blk_stat_enable_accounting() and blk_stat_disable_accounting() are currently exported, but their prototypes are only defined in a private header. Move these prototypes into a common header so that block drivers can directly use these APIs.

Quoted 9 lines
Quoted 6 lines

---
  block/blk-stat.h       | 4 ----
  include/linux/blk-mq.h | 4 ++++

struct request_queue comes from blkdev.h, so maybe that would be a more appropriate place to locate the prototype.

yes both blk-mq.h and blkdev.h should work well. We do also have few prototypes defined in linux/blk-mq.h which uses request_queue. Personally I don't have anything against using blkdev.h. So I will move it under blkdev.h.

Thanks,
--Nilay

#4 lore raw

Re: [PATCHv6 1/8] block: expose blk_stat_{enable,disable}_accounting() to drivers

blk_stat_enable_accounting is currently only useful for blk-mq based drivers, so please keep it there.

Reviewed-by on v6 2/8
#6 lore raw

Re: [PATCHv6 2/8] nvme-multipath: pass I/O type to nvme_find_path()

In reply to Nilay Shroff
Quoted 9 lines · Nilay Shroff

Currently, nvme_find_path() only accepts an nvme_ns_head argument. However, the upcoming latency-aware I/O policy also needs to know the I/O type (read/write/other) associated with the request in order to make path selection decisions.

Update nvme_find_path() to accept an additional argument describing the I/O type. This patch does not introduce any functional change and only prepares the interface for subsequent latency-policy changes.

Quoted 9 lines
---
  drivers/nvme/host/ioctl.c     | 38 ++++++++++++++++++++++++++++++++---
  drivers/nvme/host/multipath.c |  9 +++++----
  drivers/nvme/host/nvme.h      | 19 +++++++++++++++++-
  drivers/nvme/host/pr.c        |  6 ++++--
  drivers/nvme/host/sysfs.c     |  2 +-
  5 files changed, 63 insertions(+), 11 deletions(-)

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
#13 lore raw
In reply to Nilay Shroff

Hi Keith,

A gentle ping on this series.
Would you please consider pulling it for the current merge window? In the latest revision, I've addressed all review comments and updated the series based on the feedback and discussions during LSFMM. If there are any remaining concerns or if you'd like to see further changes, please let me know and I'll be happy to address them.

Thanks,
--Nilay

Quoted 165 lines · Nilay Shroff

Hi,

This series introduces a new latency I/O policy for NVMe native multipath. Existing policies such as numa, round-robin, and queue-depth are static and do not adapt to real-time transport performance. The numa selects the path closest to the NUMA node of the current CPU, optimizing memory and path locality, but ignores actual path performance. The round-robin distributes I/O evenly across all paths, providing fairness but not performance awareness. The queue-depth reacts to instantaneous queue occupancy, avoiding heavily loaded paths, but does not account for actual latency, throughput, or link speed.

The new latency policy addresses these gaps selecting paths dynamically based on measured I/O latency for both PCIe and fabrics. Latency is derived by passively sampling I/O completions. Each path is assigned a weight proportional to its latency score, and I/Os are then forwarded accordingly. As condition changes (e.g. latency spikes, bandwidth differences), path weights are updated, automatically steering traffic toward better-performing paths.

Early results show reduced tail latency under mixed workloads and improved throughput by exploiting higher-speed links more effectively. For example, with NVMf/TCP using two paths (one throttled with ~30 ms delay), fio results with random read/write/rw workloads (direct I/O) showed:

         numa         round-robin   queue-depth  adaptive
         -----------  -----------   -----------  ---------
READ:   50.0 MiB/s   105 MiB/s     230 MiB/s    350 MiB/s
WRITE:  65.9 MiB/s   125 MiB/s     385 MiB/s    446 MiB/s
RW:     R:30.6 MiB/s R:56.5 MiB/s  R:122 MiB/s  R:175 MiB/s
         W:30.7 MiB/s W:56.5 MiB/s  W:122 MiB/s  W:175 MiB/s

This pathcset includes totla 8 patches:
[PATCH 1/8] block: expose blk_stat_{enable,disable}_accounting()
- Make blk_stat APIs available to block drivers.
- Needed for per-path latency measurement.

[PATCH 2/8] nvme-multipath: pass I/O type to nvme_find_path()
- This is the prep patch which updates nvme_find_path() signature

[PATCH 3/8] nvme-multipath: add latency I/O policy
- Implement path scoring based on latency (EWMA).
- Distribute I/O proportionally to per-path weights.

[PATCH 4/8] nvme: add generic debugfs support
- Introduce generic debugfs support for NVMe module

[PATCH 5/8] nvme-multipath: add debugfs attribute latency_ewma_shift
- Adds a debugfs attribute to control ewma shift

[PATCH 6/8] nvme-multipath: add debugfs attribute latency_batch_timeout
- Adds a debugfs attribute to control latency batch window interval

[PATCH 7/8] nvme-multipath: add debugfs attribute latency_stat
- Add “latency_stat” under per-path and head debugfs directories to expose latency policy state and statistics.

[PATCH 8/8] nvme-multipath: add documentation for latency I/O policy
- Includes documentation for latency I/O multipath policy.

LSFMM discussion:
=================
During lsfmm 2026, it was decided to rename this I/O policy from "adaptive" to "latency". This series reflects that rename.

The discussion at lsfmm also focused extensively on the latency measurement model, including whether latency should be tracked per-CPU or per-NUMA, and whether separate I/O-size buckets should be maintained for different request sizes.

After detailed discussion and evaluation of throughput results, the consensus was to initially measure I/O completion latency on a per-CPU basis. The available performance data showed that the per-CPU implementation already provides sufficient averaging across CPUs while keeping the design relatively simple.

The use of additional I/O-size buckets did not demonstrate meaningful throughput improvement in the general case and would introduce extra complexity into the fast path and accounting logic. As a result, the consensus was to avoid I/O-size bucketing for now and keep the policy focused on per-CPU latency measurement.

If future real-world workloads demonstrate a clear benefit from I/O-size-aware latency accounting, the policy can be extended later to support it.

As ususal, feedback and suggestions are most welcome!

Thanks!

Changes from v5:
- Rename the policy from "adaptive" to "latency". The entire series updates policy names, function names, and variable names accordingly, without introducing any functional changes. (lsfmm discussion)
- The second patch is now splitted into two patches:
Patch #2: prep patch where we pass op_type to nvme_find_path() Patch #3: core patch which introduces latency I/O policy
(Sagi)
- Rename ewma_update() to calc_ewma_update() (Sagi)
Link to v5: https://lore.kernel.org/all/[email protected]/

Changes from v4:
- Added patch #7 which includes the documentation for adaptive I/O policy. (Guixin Liu)
Link to v4: https://lore.kernel.org/all/[email protected]/

Changes from v3:
- Update the adaptive APIs name (which actually enable/disable adaptive policy) to reflect the actual work it does. Also removed the misleading use of "current_path" from the adaptive policy code (Hannes Reinecke)
- Move adaptive_ewma_shift and adaptive_weight_timeout attributes from sysfs to debugfs (Hannes Reinecke)
Link to v3: https://lore.kernel.org/all/[email protected]/

Changes from v2:
- Addede a new patch to allow user to configure EWMA shift through sysfs (Hannes Reinecke)
- Added a new patch to allow user to configure path weight calculation timeout (Hannes Reinecke)
- Distinguish between read/write and other commands (e.g. admin comamnd) and calculate path weight for other commands which is separate from read/write weight. (Hannes Reinecke)
- Normalize per-path weight in the range from 0-128 instead

     of 0-100 (Hannes Reinecke)

- Restructure and optimize adaptive I/O forwarding code to use
one loop instead of two (Hannes Reinecke)
Link to v2: https://lore.kernel.org/all/[email protected]/

Changes from v1:
- Ensure that the completion of I/O occurs on the same CPU as the submitting I/O CPU (Hannes Reinecke)
- Remove adapter link speed from the path weight calculation (Hannes Reinecke)
- Add adaptive I/O stat under debugfs instead of current sysfs (Hannes Reinecke)
- Move path weight calculation to a workqueue from IO completion code path
Link to v1: https://lore.kernel.org/all/[email protected]/

Nilay Shroff (8):
block: expose blk_stat_{enable,disable}_accounting() to drivers
nvme-multipath: pass I/O type to nvme_find_path()
nvme-multipath: add support for latency I/O policy
nvme: add generic debugfs support
nvme-multipath: add debugfs attribute latency_ewma_shift
nvme-multipath: add debugfs attribute latency_batch_timeout
nvme-multipath: add debugfs attribute latency_stat
nvme-multipath: add documentation for latency I/O policy

  Documentation/admin-guide/nvme-multipath.rst |  19 +
  block/blk-stat.h                             |   4 -
  drivers/nvme/host/Makefile                   |   2 +-
  drivers/nvme/host/core.c                     |  21 +-
  drivers/nvme/host/debugfs.c                  | 345 ++++++++++++++
  drivers/nvme/host/ioctl.c                    |  38 +-
  drivers/nvme/host/multipath.c                | 446 ++++++++++++++++++-
  drivers/nvme/host/nvme.h                     |  84 +++-
  drivers/nvme/host/pr.c                       |   6 +-
  drivers/nvme/host/sysfs.c                    |   2 +-
  include/linux/blk-mq.h                       |   4 +
  11 files changed, 941 insertions(+), 30 deletions(-)

create mode 100644 drivers/nvme/host/debugfs.c