Skip to contentSkip to conversation

nvme-multipath: introduce latency I/O policy

PATCH In review Nilay Shroff 5 participants 25 messages Aug 15 – Sep 14

Subsystems (5)
Documentation SCSI Block layer NVMe +1 more subsystems
lore
Share & tools
lore
#0 lore raw

v8 cover nvme-multipath: introduce latency I/O policy

Hi,

This series introduces a new latency I/O policy for NVMe native multipath. Existing policies such as numa, round-robin, and queue-depth are static and do not adapt to real-time transport performance. The numa selects the path closest to the NUMA node of the current CPU, optimizing memory and path locality, but ignores actual path performance. The round-robin distributes I/O evenly across all paths, providing fairness but not performance awareness. The queue-depth reacts to instantaneous queue occupancy, avoiding heavily loaded paths, but does not account for actual latency, throughput, or link speed.

The new latency policy addresses these gaps selecting paths dynamically based on measured I/O latency for both PCIe and fabrics. Latency is derived by passively sampling I/O completions. Each path is assigned a weight proportional to its latency score, and I/Os are then forwarded accordingly. As condition changes (e.g. latency spikes, bandwidth differences), path weights are updated, automatically steering traffic toward better-performing paths.

Early results show reduced tail latency under mixed workloads and improved throughput by exploiting higher-speed links more effectively. For example, with NVMf/TCP using two paths (one throttled with ~30 ms delay), fio results with random read/write/rw workloads (direct I/O) showed:

        numa         round-robin   queue-depth  adaptive
        -----------  -----------   -----------  ---------
READ:   50.0 MiB/s   105 MiB/s     230 MiB/s    350 MiB/s
WRITE:  65.9 MiB/s   125 MiB/s     385 MiB/s    446 MiB/s
RW:     R:30.6 MiB/s R:56.5 MiB/s  R:122 MiB/s  R:175 MiB/s
        W:30.7 MiB/s W:56.5 MiB/s  W:122 MiB/s  W:175 MiB/s

This pathcset includes totla 8 patches:
[PATCH 1/10] block: expose blk_stat_{enable,disable}_accounting()
- Make blk_stat APIs available to block drivers.
- Needed for per-path latency measurement.

[PATCH 2/10] block: record I/O request start time for passthru request
- Record I/O start time for I/O passthru requests.
- This is prep patch which allows measuring I/O completion latency for passthru requests.

[PATCH 3/10] block: support nesting for blk-mq flag QUEUE_FLAG_SAME_FORCE
- Support nesting for QUEUE_FLAG_SAME_FORCE as multiple users could toggle QUEUE_FLAG_SAME_FORCE.

[PATCH 4/10] nvme-multipath: pass I/O type to nvme_find_path()
- This is the prep patch which updates nvme_find_path() signature

[PATCH 5/10] nvme-multipath: add latency I/O policy
- Implement path scoring based on latency (EWMA).
- Distribute I/O proportionally to per-path weights.

[PATCH 6/10] nvme: add generic debugfs support
- Introduce generic debugfs support for NVMe module

[PATCH 7/10] nvme-multipath: add debugfs attribute latency_ewma_shift
- Adds a debugfs attribute to control ewma shift

[PATCH 8/10] nvme-multipath: add debugfs attribute latency_batch_timeout
- Adds a debugfs attribute to control latency batch window interval

[PATCH 9/10] nvme-multipath: add debugfs attribute latency_stat
- Add “latency_stat” under per-path and head debugfs directories to expose latency policy state and statistics.

[PATCH 10/10] nvme-multipath: add documentation for latency I/O policy
- Includes documentation for latency I/O multipath policy.

LSFMM discussion:
=================
During lsfmm 2026, it was decided to rename this I/O policy from "adaptive" to "latency". This series reflects that rename.

The discussion at lsfmm also focused extensively on the latency measurement model, including whether latency should be tracked per-CPU or per-NUMA, and whether separate I/O-size buckets should be maintained for different request sizes.

After detailed discussion and evaluation of throughput results, the consensus was to initially measure I/O completion latency on a per-CPU basis. The available performance data showed that the per-CPU implementation already provides sufficient averaging across CPUs while keeping the design relatively simple.

The use of additional I/O-size buckets did not demonstrate meaningful throughput improvement in the general case and would introduce extra complexity into the fast path and accounting logic. As a result, the consensus was to avoid I/O-size bucketing for now and keep the policy focused on per-CPU latency measurement.

If future real-world workloads demonstrate a clear benefit from I/O-size-aware latency accounting, the policy can be extended later to support it.

As ususal, feedback and suggestions are most welcome!

Thanks!

Changes from v7:
- Rebased on nvme-7.3 (John Garry)
- Added Clang lock context annotations where appropriate (John Garry)
- Added support for nesting QUEUE_FLAG_SAME_FORCE in a new patch

    3/10 (John Garry)

- Free head->latency_path in nvme_mpath_put_disk() instead of nvme_free_ns_head() (John Garry)
- Replaced this_cpu_ptr() with per_cpu_ptr() (John Garry)
- Few miscellaneous and cosmetic changes such as adding comments updating commit description etc.

Link to v7: https://lore.kernel.org/all/[email protected]/

Changes from v6:
- Add patch 2/9, which records I/O start time for the passthru requests (Guixin Liu)
- Correctly record the passthru I/O data direction (Guixin Liu)
- Clear NVME_NS_PATH_STAT before cancelling the latency weight work. (Guixin Liu)

Link to v6: https://lore.kernel.org/all/[email protected]/

Changes from v5:
- Rename the policy from "adaptive" to "latency". The entire series updates policy names, function names, and variable names accordingly, without introducing any functional changes. (lsfmm discussion)
- The second patch is now splitted into two patches:
Patch #2: prep patch where we pass op_type to nvme_find_path() Patch #3: core patch which introduces latency I/O policy
(Sagi)
- Rename ewma_update() to calc_ewma_update() (Sagi)
Link to v5: https://lore.kernel.org/all/[email protected]/

Changes from v4:
- Added patch #7 which includes the documentation for adaptive I/O policy. (Guixin Liu)
Link to v4: https://lore.kernel.org/all/[email protected]/

Changes from v3:
- Update the adaptive APIs name (which actually enable/disable adaptive policy) to reflect the actual work it does. Also removed the misleading use of "current_path" from the adaptive policy code (Hannes Reinecke)
- Move adaptive_ewma_shift and adaptive_weight_timeout attributes from sysfs to debugfs (Hannes Reinecke)
Link to v3: https://lore.kernel.org/all/[email protected]/

Changes from v2:
- Addede a new patch to allow user to configure EWMA shift through sysfs (Hannes Reinecke)
- Added a new patch to allow user to configure path weight calculation timeout (Hannes Reinecke)
- Distinguish between read/write and other commands (e.g. admin comamnd) and calculate path weight for other commands which is separate from read/write weight. (Hannes Reinecke)
- Normalize per-path weight in the range from 0-128 instead

    of 0-100 (Hannes Reinecke)

- Restructure and optimize adaptive I/O forwarding code to use
one loop instead of two (Hannes Reinecke)
Link to v2: https://lore.kernel.org/all/[email protected]/

Changes from v1:
- Ensure that the completion of I/O occurs on the same CPU as the submitting I/O CPU (Hannes Reinecke)
- Remove adapter link speed from the path weight calculation
(Hannes Reinecke)
- Add adaptive I/O stat under debugfs instead of current sysfs (Hannes Reinecke)
- Move path weight calculation to a workqueue from IO completion code path
Link to v1: https://lore.kernel.org/all/[email protected]/

Nilay Shroff (10):
block: expose blk_stat_{enable,disable}_accounting() to drivers
block: record I/O request start time for passthru request
block: support nesting for blk-mq flag QUEUE_FLAG_SAME_FORCE
nvme-multipath: pass I/O type to nvme_find_path()
nvme-multipath: add support for latency I/O policy
nvme: add generic debugfs support
nvme-multipath: add debugfs attribute latency_ewma_shift
nvme-multipath: add debugfs attribute latency_batch_timeout
nvme-multipath: add debugfs attribute latency_stat
nvme-multipath: add documentation for latency I/O policy

 Documentation/admin-guide/nvme-multipath.rst |  19 +
 block/blk-mq.c                               |  53 +-
 block/blk-stat.h                             |   4 -
 block/blk-sysfs.c                            |   6 +-
 drivers/nvme/host/Makefile                   |   2 +-
 drivers/nvme/host/core.c                     |  15 +-
 drivers/nvme/host/debugfs.c                  | 347 ++++++++++++++
 drivers/nvme/host/ioctl.c                    |  44 +-
 drivers/nvme/host/multipath.c                | 478 ++++++++++++++++++-
 drivers/nvme/host/nvme.h                     | 110 ++++-
 drivers/nvme/host/pr.c                       |   6 +-
 drivers/nvme/host/sysfs.c                    |   2 +-
 drivers/ufs/host/ufs-mediatek.c              |   2 +-
 include/linux/blk-mq.h                       |   6 +
 include/linux/blkdev.h                       |   3 +
 15 files changed, 1057 insertions(+), 40 deletions(-)
 create mode 100644 drivers/nvme/host/debugfs.c
Signature · 1 line
2.53.0
#1 lore raw

PATCH v8 1/10 block: expose blk_stat_{enable,disable}_accounting() to drivers

The functions blk_stat_enable_accounting() and blk_stat_disable_accounting() are currently exported, but their prototypes are only defined in a private header. Move these prototypes into a common header so that block drivers can directly use these APIs.

---
 block/blk-stat.h       | 4 ----
 include/linux/blk-mq.h | 4 ++++
 2 files changed, 4 insertions(+), 4 deletions(-)
block/blk-stat.h +0 -4
index cc5b66e7ee60..b1614a1ef4ad 100644
@@ -70,10 +70,6 @@ void blk_free_queue_stats(struct blk_queue_stats *);
 
 void blk_stat_add(struct request *rq, u64 now);
 
-/* record time/size info in request but not add a callback */
-void blk_stat_enable_accounting(struct request_queue *q);
-void blk_stat_disable_accounting(struct request_queue *q);
-
 /**
  * blk_stat_alloc_callback() - Allocate a block statistics callback.
  * @timer_fn: Timer callback function.
include/linux/blk-mq.h +4 -0
index af878597afb8..3956909764bf 100644
@@ -753,6 +753,10 @@ int blk_rq_poll(struct request *rq, struct io_comp_batch *iob,
 
 bool blk_mq_queue_inflight(struct request_queue *q);
 
+/* record time/size info in request but not add a callback */
+void blk_stat_enable_accounting(struct request_queue *q);
+void blk_stat_disable_accounting(struct request_queue *q);
+
 enum {
 	/* return when out of requests */
 	BLK_MQ_REQ_NOWAIT	= (__force blk_mq_req_flags_t)(1 << 0),
Signature · 1 line
2.53.0
#2 lore raw

PATCH v8 2/10 block: record I/O request start time for passthru request

In reply to Nilay Shroff

While starting an I/O request, blk_mq_start_request() records the request start timestamp only for non-passthrough requests when QUEUE_FLAG_STATS is enabled.

However, the latency based multipath policy uses request completion latency to evaluate path performance, and I/O is issued as passthrough requests. Since passthru requests never initialize rq->io_start_time_ns, their latency cannot be computed.

Record io_start_time_ns for all requests whenever QUEUE_FLAG_STATS is enabled.

---
 block/blk-mq.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
block/blk-mq.c +7 -5
index 2c850330a32b..38922209a24f 100644
@@ -1340,12 +1340,14 @@ void blk_mq_start_request(struct request *rq)
 
 	trace_block_rq_issue(rq);
 
-	if (test_bit(QUEUE_FLAG_STATS, &q->queue_flags) &&
-	    !blk_rq_is_passthrough(rq)) {
+	if (test_bit(QUEUE_FLAG_STATS, &q->queue_flags)) {
 		rq->io_start_time_ns = blk_time_get_ns();
-		rq->stats_sectors = blk_rq_sectors(rq);
-		rq->rq_flags |= RQF_STATS;
-		rq_qos_issue(q, rq);
+
+		if (!blk_rq_is_passthrough(rq)) {
+			rq->stats_sectors = blk_rq_sectors(rq);
+			rq->rq_flags |= RQF_STATS;
+			rq_qos_issue(q, rq);
+		}
 	}
 
 	WARN_ON_ONCE(blk_mq_rq_state(rq) != MQ_RQ_IDLE);
Signature · 1 line
2.53.0
#3 lore raw

PATCH v8 3/10 block: support nesting for blk-mq flag QUEUE_FLAG_SAME_FORCE

In reply to Nilay Shroff

QUEUE_FLAG_SAME_FORCE is currently used when setting rq_affinity through sysfs as well as by UFS mediatek driver while configuring scsi parameters. A subsequent patch adding a latency-based I/O policy for NVMe multipath will also use this flag.

With multiple users able to set and clear QUEUE_FLAG_SAME_FORCE, the flag needs to support nesting so that one user clearing the flag does not inadvertently disable it for another user.

Add a nesting counter, q->same_force_depth, for QUEUE_FLAG_SAME_FORCE. The flag is set when the first user acquires it and the nesting counter is incremented for each subsequent user. Similarly, each user releases its reference by decrementing the counter. The flag is cleared only when the last user releases it and the counter reaches zero.

Preserve the existing sysfs rq_affinity semantics with a new q->same_force_sysfs flag. When userspace enables QUEUE_FLAG_SAME_FORCE by writing 2 to rq_affinity, mark q->same_force_sysfs as set and increment q->same_force_depth by one. Subsequent writes of 2 to rq_affinity while q->same_force_sysfs is already set are ignored, so repeated writes of 2 from userspace do not increase q->same_force_depth. Similarly, writing 0 or 1 decrements the q->same_force_depth and if nesting counter reached to 0 then clears the QUEUE_FLAG_SAME_FORCE. This ensures that multiple writes of 2 to rq_affinity do not require multiple writes of 0 or 1.

This change ensures that sysfs interface retains its existing set/clear semantics while also allowing other kernel users to hold or release QUEUE_FLAG_SAME_FORCE.

Added two new APIs blk_mq_same_force_set() and blk_mq_same_force_clear() to set and clear QUEUE_FLAG_SAME_FORCE respectively. Also, updated existing call paths using these new APIs which toggles QUEUE_FLAG_SAME_FORCE.

---
 block/blk-mq.c                  | 41 +++++++++++++++++++++++++++++++++
 block/blk-sysfs.c               |  6 ++---
 drivers/ufs/host/ufs-mediatek.c |  2 +-
 include/linux/blk-mq.h          |  2 ++
 include/linux/blkdev.h          |  3 +++
 5 files changed, 50 insertions(+), 4 deletions(-)
block/blk-mq.c +41 -0
index 38922209a24f..17e8befb02bc 100644
@@ -360,6 +360,47 @@ void blk_mq_unquiesce_tagset(struct blk_mq_tag_set *set)
 }
 EXPORT_SYMBOL_GPL(blk_mq_unquiesce_tagset);
 
+void blk_mq_same_force_set(struct request_queue *q, bool from_sysfs)
+{
+	unsigned long flags = 0;
+
+	spin_lock_irqsave(&q->queue_lock, flags);
+
+	if (from_sysfs) {
+		if (q->same_force_sysfs)
+			goto unlock;
+		q->same_force_sysfs = true;
+	}
+
+	if (!q->same_force_depth++)
+		blk_queue_flag_set(QUEUE_FLAG_SAME_FORCE, q);
+unlock:
+	spin_unlock_irqrestore(&q->queue_lock, flags);
+}
+EXPORT_SYMBOL_GPL(blk_mq_same_force_set);
+
+void blk_mq_same_force_clear(struct request_queue *q, bool from_sysfs)
+{
+	unsigned long flags = 0;
+
+	spin_lock_irqsave(&q->queue_lock, flags);
+
+	if (from_sysfs) {
+		if (!q->same_force_sysfs)
+			goto unlock;
+		q->same_force_sysfs = false;
+	}
+
+	if (WARN_ON_ONCE(q->same_force_depth <= 0))
+		goto unlock;
+
+	if (!--q->same_force_depth)
+		blk_queue_flag_clear(QUEUE_FLAG_SAME_FORCE, q);
+unlock:
+	spin_unlock_irqrestore(&q->queue_lock, flags);
+}
+EXPORT_SYMBOL_GPL(blk_mq_same_force_clear);
+
 void blk_mq_wake_waiters(struct request_queue *q)
 {
 	struct blk_mq_hw_ctx *hctx;
block/blk-sysfs.c +3 -3
index 520972676ab4..a3ec8ffab1ee 100644
@@ -497,13 +497,13 @@ queue_rq_affinity_store(struct gendisk *disk, const char *page, size_t count)
 	 */
 	if (val == 2) {
 		blk_queue_flag_set(QUEUE_FLAG_SAME_COMP, q);
-		blk_queue_flag_set(QUEUE_FLAG_SAME_FORCE, q);
+		blk_mq_same_force_set(q, true);
 	} else if (val == 1) {
 		blk_queue_flag_set(QUEUE_FLAG_SAME_COMP, q);
-		blk_queue_flag_clear(QUEUE_FLAG_SAME_FORCE, q);
+		blk_mq_same_force_clear(q, true);
 	} else if (val == 0) {
 		blk_queue_flag_clear(QUEUE_FLAG_SAME_COMP, q);
-		blk_queue_flag_clear(QUEUE_FLAG_SAME_FORCE, q);
+		blk_mq_same_force_clear(q, true);
 	}
 #endif
 	return ret;
drivers/ufs/host/ufs-mediatek.c +1 -1
index 3991a51263a6..8d80ac6fb5d2 100644
@@ -2311,7 +2311,7 @@ static void ufs_mtk_config_scsi_dev(struct scsi_device *sdev)
 
 	dev_dbg(hba->dev, "lu %llu scsi device configured", sdev->lun);
 	if (sdev->lun == 2)
-		blk_queue_flag_set(QUEUE_FLAG_SAME_FORCE, sdev->request_queue);
+		blk_mq_same_force_set(sdev->request_queue, false);
 }
 
 /*
include/linux/blk-mq.h +2 -0
index 3956909764bf..30be3eb5a37b 100644
@@ -943,6 +943,8 @@ void blk_mq_wait_quiesce_done(struct blk_mq_tag_set *set);
 void blk_mq_quiesce_tagset(struct blk_mq_tag_set *set);
 void blk_mq_unquiesce_tagset(struct blk_mq_tag_set *set);
 void blk_mq_unquiesce_queue(struct request_queue *q);
+void blk_mq_same_force_set(struct request_queue *q, bool from_sysfs);
+void blk_mq_same_force_clear(struct request_queue *q, bool from_sysfs);
 void blk_mq_delay_run_hw_queue(struct blk_mq_hw_ctx *hctx, unsigned long msecs);
 void blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async);
 void blk_mq_run_hw_queues(struct request_queue *q, bool async);
include/linux/blkdev.h +3 -0
index 9213a5716f95..32d0fb47d73c 100644
@@ -529,6 +529,9 @@ struct request_queue {
 
 	int			quiesce_depth;
 
+	int			same_force_depth;
+	bool			same_force_sysfs;
+
 	struct gendisk		*disk;
 
 	/*
Signature · 1 line
2.53.0
#4 lore raw

Re: [PATCH v8 03/10] block: support nesting for blk-mq flag QUEUE_FLAG_SAME_FORCE

Quoted 26 lines · Nilay Shroff

QUEUE_FLAG_SAME_FORCE is currently used when setting rq_affinity through sysfs as well as by UFS mediatek driver while configuring scsi parameters. A subsequent patch adding a latency-based I/O policy for NVMe multipath will also use this flag.

With multiple users able to set and clear QUEUE_FLAG_SAME_FORCE, the flag needs to support nesting so that one user clearing the flag does not inadvertently disable it for another user.

Add a nesting counter, q->same_force_depth, for QUEUE_FLAG_SAME_FORCE. The flag is set when the first user acquires it and the nesting counter is incremented for each subsequent user. Similarly, each user releases its reference by decrementing the counter. The flag is cleared only when the last user releases it and the counter reaches zero.

Preserve the existing sysfs rq_affinity semantics with a new q->same_force_sysfs flag. When userspace enables QUEUE_FLAG_SAME_FORCE by writing 2 to rq_affinity, mark q->same_force_sysfs as set and increment q->same_force_depth by one. Subsequent writes of 2 to rq_affinity while q->same_force_sysfs is already set are ignored, so repeated writes of 2 from userspace do not increase q->same_force_depth. Similarly, writing 0 or 1 decrements the q->same_force_depth and if nesting counter reached to 0 then clears the QUEUE_FLAG_SAME_FORCE. This ensures that multiple writes of 2 to rq_affinity do not require multiple writes of 0 or 1.

Quoted 9 lines

This change ensures that sysfs interface retains its existing set/clear semantics while also allowing other kernel users to hold or release QUEUE_FLAG_SAME_FORCE.

Added two new APIs blk_mq_same_force_set() and blk_mq_same_force_clear() to set and clear QUEUE_FLAG_SAME_FORCE respectively. Also, updated existing call paths using these new APIs which toggles QUEUE_FLAG_SAME_FORCE.

Yikes. Is this really necessary?
The flag can be set by different subsystems/drivers, sure, but from what I've seen each user is mutually exclusive.
I'd rather disallow modifying this flag if the driver sets it, to avoid upper layers or sysfs modifying it.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
#5 lore raw

Re: [PATCH v8 03/10] block: support nesting for blk-mq flag QUEUE_FLAG_SAME_FORCE

Quoted 36 lines · Hannes Reinecke
Quoted 26 lines · Nilay Shroff

QUEUE_FLAG_SAME_FORCE is currently used when setting rq_affinity through sysfs as well as by UFS mediatek driver while configuring scsi parameters. A subsequent patch adding a latency-based I/O policy for NVMe multipath will also use this flag.

With multiple users able to set and clear QUEUE_FLAG_SAME_FORCE, the flag needs to support nesting so that one user clearing the flag does not inadvertently disable it for another user.

Add a nesting counter, q->same_force_depth, for QUEUE_FLAG_SAME_FORCE. The flag is set when the first user acquires it and the nesting counter is incremented for each subsequent user. Similarly, each user releases its reference by decrementing the counter. The flag is cleared only when the last user releases it and the counter reaches zero.

Preserve the existing sysfs rq_affinity semantics with a new q->same_force_sysfs flag. When userspace enables QUEUE_FLAG_SAME_FORCE by writing 2 to rq_affinity, mark q->same_force_sysfs as set and increment q->same_force_depth by one. Subsequent writes of 2 to rq_affinity while q->same_force_sysfs is already set are ignored, so repeated writes of 2 from userspace do not increase q->same_force_depth. Similarly, writing 0 or 1 decrements the q->same_force_depth and if nesting counter reached to 0 then clears the QUEUE_FLAG_SAME_FORCE. This ensures that multiple writes of 2 to rq_affinity do not require multiple writes of 0 or 1.

Quoted 9 lines

This change ensures that sysfs interface retains its existing set/clear semantics while also allowing other kernel users to hold or release QUEUE_FLAG_SAME_FORCE.

Added two new APIs blk_mq_same_force_set() and blk_mq_same_force_clear() to set and clear QUEUE_FLAG_SAME_FORCE respectively. Also, updated existing call paths using these new APIs which toggles QUEUE_FLAG_SAME_FORCE.

Quoted 6 lines

Yikes. Is this really necessary?
The flag can be set by different subsystems/drivers, sure, but from what I've seen each user is mutually exclusive.
I'd rather disallow modifying this flag if the driver sets it, to avoid upper layers or sysfs modifying it.

If we want to disallow userspace from modifying QUEUE_FLAG_SAME_FORCE while a driver owns it, then a userspace write to rq_affinity would have to either fail (e.g. with -EINVAL) or be silently ignored. In either case, that changes the existing userspace/sysfs semantics and may break userspace that expects the write to succeed.

Or are you suggesting that a write to rq_affinity to enable QUEUE_FLAG_SAME_FORCE should succeed when the driver has already set it? If so, what should happen when the driver subsequently releases QUEUE_FLAG_SAME_FORCE? In that case, we would clear QUEUE_FLAG_SAME_FORCE when the driver releases it, even though userspace may have previously written 2 to rq_affinity. That would effectively reset the state requested by userspace without informing userspace.

This seems to require keeping track of the userspace request separately from the driver's ownership of the flag, which is one of the reasons I introduced same_force_sysfs along with the nesting counter.

Thanks,
--Nilay

#6 lore raw

Re: [PATCH v8 03/10] block: support nesting for blk-mq flag QUEUE_FLAG_SAME_FORCE

Quoted 44 lines · Nilay Shroff
Quoted 36 lines · Hannes Reinecke
Quoted 26 lines · Nilay Shroff

QUEUE_FLAG_SAME_FORCE is currently used when setting rq_affinity through sysfs as well as by UFS mediatek driver while configuring scsi parameters. A subsequent patch adding a latency-based I/O policy for NVMe multipath will also use this flag.

With multiple users able to set and clear QUEUE_FLAG_SAME_FORCE, the flag needs to support nesting so that one user clearing the flag does not inadvertently disable it for another user.

Add a nesting counter, q->same_force_depth, for QUEUE_FLAG_SAME_FORCE. The flag is set when the first user acquires it and the nesting counter is incremented for each subsequent user. Similarly, each user releases its reference by decrementing the counter. The flag is cleared only when the last user releases it and the counter reaches zero.

Preserve the existing sysfs rq_affinity semantics with a new q->same_force_sysfs flag. When userspace enables QUEUE_FLAG_SAME_FORCE by writing 2 to rq_affinity, mark q->same_force_sysfs as set and increment q->same_force_depth by one. Subsequent writes of 2 to rq_affinity while q->same_force_sysfs is already set are ignored, so repeated writes of 2 from userspace do not increase q->same_force_depth. Similarly, writing 0 or 1 decrements the q->same_force_depth and if nesting counter reached to 0 then clears the QUEUE_FLAG_SAME_FORCE. This ensures that multiple writes of 2 to rq_affinity do not require multiple writes of 0 or 1.

Quoted 9 lines

This change ensures that sysfs interface retains its existing set/clear semantics while also allowing other kernel users to hold or release QUEUE_FLAG_SAME_FORCE.

Added two new APIs blk_mq_same_force_set() and blk_mq_same_force_clear() to set and clear QUEUE_FLAG_SAME_FORCE respectively. Also, updated existing call paths using these new APIs which toggles QUEUE_FLAG_SAME_FORCE.

Quoted 6 lines

Yikes. Is this really necessary?
The flag can be set by different subsystems/drivers, sure, but from what I've seen each user is mutually exclusive.
I'd rather disallow modifying this flag if the driver sets it, to avoid upper layers or sysfs modifying it.

Quoted 6 lines

If we want to disallow userspace from modifying QUEUE_FLAG_SAME_FORCE while a driver owns it, then a userspace write to rq_affinity would have to either fail (e.g. with -EINVAL) or be silently ignored. In either case, that changes the existing userspace/sysfs semantics and may break userspace that expects the write to succeed.

But _allowing_ userspace to modify it for drivers which requires it to be set will break the drivers.
Similar for the upcoming latency-based I/O scheduler; disabling QUEUE_FLAG_SAME_FORCE there will cause misaccounting and the scheduler will not work properly.

So I would argue to have a flag for making QUEUE_FLAG_SAME_FORCE driver owned, and reject any attempts to change it via sysfs with EINVAL.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
#7 lore raw

Re: [PATCH v8 03/10] block: support nesting for blk-mq flag QUEUE_FLAG_SAME_FORCE

Quoted 51 lines · Hannes Reinecke
Quoted 44 lines · Nilay Shroff
Quoted 36 lines · Hannes Reinecke
Quoted 26 lines · Nilay Shroff

QUEUE_FLAG_SAME_FORCE is currently used when setting rq_affinity through sysfs as well as by UFS mediatek driver while configuring scsi parameters. A subsequent patch adding a latency-based I/O policy for NVMe multipath will also use this flag.

With multiple users able to set and clear QUEUE_FLAG_SAME_FORCE, the flag needs to support nesting so that one user clearing the flag does not inadvertently disable it for another user.

Add a nesting counter, q->same_force_depth, for QUEUE_FLAG_SAME_FORCE. The flag is set when the first user acquires it and the nesting counter is incremented for each subsequent user. Similarly, each user releases its reference by decrementing the counter. The flag is cleared only when the last user releases it and the counter reaches zero.

Preserve the existing sysfs rq_affinity semantics with a new q->same_force_sysfs flag. When userspace enables QUEUE_FLAG_SAME_FORCE by writing 2 to rq_affinity, mark q->same_force_sysfs as set and increment q->same_force_depth by one. Subsequent writes of 2 to rq_affinity while q->same_force_sysfs is already set are ignored, so repeated writes of 2 from userspace do not increase q->same_force_depth. Similarly, writing 0 or 1 decrements the q->same_force_depth and if nesting counter reached to 0 then clears the QUEUE_FLAG_SAME_FORCE. This ensures that multiple writes of 2 to rq_affinity do not require multiple writes of 0 or 1.

Quoted 9 lines

This change ensures that sysfs interface retains its existing set/clear semantics while also allowing other kernel users to hold or release QUEUE_FLAG_SAME_FORCE.

Added two new APIs blk_mq_same_force_set() and blk_mq_same_force_clear() to set and clear QUEUE_FLAG_SAME_FORCE respectively. Also, updated existing call paths using these new APIs which toggles QUEUE_FLAG_SAME_FORCE.

Quoted 6 lines

Yikes. Is this really necessary?
The flag can be set by different subsystems/drivers, sure, but from what I've seen each user is mutually exclusive.
I'd rather disallow modifying this flag if the driver sets it, to avoid upper layers or sysfs modifying it.

Quoted 6 lines

If we want to disallow userspace from modifying QUEUE_FLAG_SAME_FORCE while a driver owns it, then a userspace write to rq_affinity would have to either fail (e.g. with -EINVAL) or be silently ignored. In either case, that changes the existing userspace/sysfs semantics and may break userspace that expects the write to succeed.

Quoted 8 lines

But _allowing_ userspace to modify it for drivers which requires it to be set will break the drivers.
Similar for the upcoming latency-based I/O scheduler; disabling QUEUE_FLAG_SAME_FORCE there will cause misaccounting and the scheduler will not work properly.

So I would argue to have a flag for making QUEUE_FLAG_SAME_FORCE driver owned, and reject any attempts to change it via sysfs with EINVAL.

I understand your concern. However, with the nesting counter, a userspace clear cannot disable QUEUE_FLAG_SAME_FORCE while another user, such as a driver or the latency-based I/O policy, still holds a reference to it.

For example, starting with:
q->same_force_depth = 0;
q->same_force_sysfs = false;

The sequence would be:

1. Userspace enables QUEUE_FLAG_SAME_FORCE by writing 2 to rq_affinity:
q->same_force_depth = 1;
q->same_force_sysfs = true;
QUEUE_FLAG_SAME_FORCE = set;

2. The driver then enables QUEUE_FLAG_SAME_FORCE:
q->same_force_depth = 2;
QUEUE_FLAG_SAME_FORCE = set;

3. Userspace disables it by writing 0 or 1 to rq_affinity:
q->same_force_depth = 1;
q->same_force_sysfs = false;
QUEUE_FLAG_SAME_FORCE = still set; 4. The driver subsequently releases its reference:
q->same_force_depth = 0;
QUEUE_FLAG_SAME_FORCE = clear;

Thus, the userspace clear only releases the sysfs reference; it does not actually clear QUEUE_FLAG_SAME_FORCE while the driver still holds its reference.

The same applies to the upcoming latency-based I/O policy. As long as the policy holds a reference, a userspace write to rq_affinity cannot disable QUEUE_FLAG_SAME_FORCE underneath it.

This is the motivation for the nesting support: it allows multiple users to independently acquire and release the flag without one user inadvertently affecting another user's requirement for the flag.

Does the above explanation help address your concern, or do you have any further concern?

Thanks,
--Nilay

#8 lore raw

PATCH v8 4/10 nvme-multipath: pass I/O type to nvme_find_path()

In reply to Nilay Shroff

Currently, nvme_find_path() only accepts an nvme_ns_head argument. However, the upcoming latency-aware I/O policy also needs to know the I/O type (read/write/other) associated with the request in order to make path selection decisions.

Update nvme_find_path() to accept an additional argument describing the I/O type. Classify requests into three categories: READ, WRITE, and OTHER. Admin commands and I/O requests that are neither reads nor writes are classified as OTHER.

This patch does not introduce any functional change and only prepares the interface for subsequent latency-policy changes.

---
 drivers/nvme/host/ioctl.c     | 44 +++++++++++++++++++++++++++---
 drivers/nvme/host/multipath.c |  9 ++++---
 drivers/nvme/host/nvme.h      | 50 ++++++++++++++++++++++++++++++++++-
 drivers/nvme/host/pr.c        |  6 +++--
 drivers/nvme/host/sysfs.c     |  2 +-
 5 files changed, 100 insertions(+), 11 deletions(-)
drivers/nvme/host/ioctl.c +41 -3
index 6539d4750098..b0538ed2adbc 100644
@@ -751,12 +751,25 @@ int nvme_ns_head_ioctl(struct block_device *bdev, blk_mode_t mode,
 	struct nvme_ns *ns;
 	int srcu_idx, ret = -EWOULDBLOCK;
 	unsigned int flags = 0;
+	unsigned int op_type = NVME_STAT_GROUP_OTHER;
 
 	if (bdev_is_partition(bdev))
 		flags |= NVME_IOCTL_PARTITION;
 
+	if (cmd == NVME_IOCTL_SUBMIT_IO) {
+		u8 opcode;
+
+		if (get_user(opcode, (u8 *)argp))
+			return -EFAULT;
+
+		if (opcode == nvme_cmd_write)
+			op_type = NVME_STAT_GROUP_WRITE;
+		else if (opcode == nvme_cmd_read)
+			op_type = NVME_STAT_GROUP_READ;
+	}
+
 	srcu_idx = srcu_read_lock(&head->srcu);
-	ns = nvme_find_path(head);
+	ns = nvme_find_path(head, op_type);
 	if (!ns)
 		goto out_unlock;
 
@@ -785,9 +798,22 @@ long nvme_ns_head_chr_ioctl(struct file *file, unsigned int cmd,
 	void __user *argp = (void __user *)arg;
 	struct nvme_ns *ns;
 	int srcu_idx, ret = -EWOULDBLOCK;
+	unsigned int op_type = NVME_STAT_GROUP_OTHER;
+
+	if (cmd == NVME_IOCTL_SUBMIT_IO) {
+		u8 opcode;
+
+		if (get_user(opcode, (u8 *)argp))
+			return -EFAULT;
+
+		if (opcode == nvme_cmd_write)
+			op_type = NVME_STAT_GROUP_WRITE;
+		else if (opcode == nvme_cmd_read)
+			op_type = NVME_STAT_GROUP_READ;
+	}
 
 	srcu_idx = srcu_read_lock(&head->srcu);
-	ns = nvme_find_path(head);
+	ns = nvme_find_path(head, op_type);
 	if (!ns)
 		goto out_unlock;
 
@@ -804,12 +830,24 @@ long nvme_ns_head_chr_ioctl(struct file *file, unsigned int cmd,
 int nvme_ns_head_chr_uring_cmd(struct io_uring_cmd *ioucmd,
 		unsigned int issue_flags)
 {
+	struct nvme_ns *ns;
+	unsigned int op_type;
 	struct cdev *cdev = file_inode(ioucmd->file)->i_cdev;
 	struct nvme_ns_head *head = container_of(cdev, struct nvme_ns_head, cdev);
 	int srcu_idx = srcu_read_lock(&head->srcu);
-	struct nvme_ns *ns = nvme_find_path(head);
 	int ret = -EINVAL;
+	const struct nvme_uring_cmd *cmd = io_uring_sqe128_cmd(ioucmd->sqe,
+						struct nvme_uring_cmd);
+	__u8 opcode = READ_ONCE(cmd->opcode);
+
+	if (opcode == nvme_cmd_write)
+		op_type = NVME_STAT_GROUP_WRITE;
+	else if (opcode == nvme_cmd_read)
+		op_type = NVME_STAT_GROUP_READ;
+	else
+		op_type = NVME_STAT_GROUP_OTHER;
 
+	ns = nvme_find_path(head, op_type);
 	if (ns)
 		ret = nvme_ns_uring_cmd(ns, ioucmd, issue_flags);
 	srcu_read_unlock(&head->srcu, srcu_idx);
drivers/nvme/host/multipath.c +5 -4
index 75dbb58286a3..95768eaef843 100644
@@ -484,7 +484,8 @@ static struct nvme_ns *nvme_numa_path(struct nvme_ns_head *head)
 	return ns;
 }
 
-inline struct nvme_ns *nvme_find_path(struct nvme_ns_head *head)
+inline struct nvme_ns *nvme_find_path(struct nvme_ns_head *head,
+		enum nvme_stat_group op_type)
 {
 	switch (READ_ONCE(head->subsys->iopolicy)) {
 	case NVME_IOPOLICY_QD:
@@ -547,7 +548,7 @@ static void nvme_ns_head_submit_bio(struct bio *bio)
 		return;
 
 	srcu_idx = srcu_read_lock(&head->srcu);
-	ns = nvme_find_path(head);
+	ns = nvme_find_path(head, __nvme_get_stat_group(bio_op(bio)));
 	if (likely(ns)) {
 		bio_set_dev(bio, ns->disk->part0);
 		/*
@@ -597,7 +598,7 @@ static int nvme_ns_head_get_unique_id(struct gendisk *disk, u8 id[16],
 	int srcu_idx, ret = -EWOULDBLOCK;
 
 	srcu_idx = srcu_read_lock(&head->srcu);
-	ns = nvme_find_path(head);
+	ns = nvme_find_path(head, NVME_STAT_GROUP_OTHER);
 	if (ns)
 		ret = nvme_ns_get_unique_id(ns, id, type);
 	srcu_read_unlock(&head->srcu, srcu_idx);
@@ -613,7 +614,7 @@ static int nvme_ns_head_report_zones(struct gendisk *disk, sector_t sector,
 	int srcu_idx, ret = -EWOULDBLOCK;
 
 	srcu_idx = srcu_read_lock(&head->srcu);
-	ns = nvme_find_path(head);
+	ns = nvme_find_path(head, NVME_STAT_GROUP_OTHER);
 	if (ns)
 		ret = nvme_ns_report_zones(ns, sector, nr_zones, args);
 	srcu_read_unlock(&head->srcu, srcu_idx);
drivers/nvme/host/nvme.h +49 -1

63 lines not on this page · raw message

drivers/nvme/host/pr.c +4 -2
index fe7dbe264815..715e6c242bd1 100644
@@ -53,10 +53,12 @@ static int nvme_send_ns_head_pr_command(struct block_device *bdev,
 		struct nvme_command *c, void *data, unsigned int data_len)
 {
 	struct nvme_ns_head *head = bdev->bd_disk->private_data;
-	int srcu_idx = srcu_read_lock(&head->srcu);
-	struct nvme_ns *ns = nvme_find_path(head);
+	int srcu_idx;
+	struct nvme_ns *ns;
 	int ret = -EWOULDBLOCK;
 
+	srcu_idx = srcu_read_lock(&head->srcu);
+	ns = nvme_find_path(head, NVME_STAT_GROUP_OTHER);
 	if (ns) {
 		c->common.nsid = cpu_to_le32(ns->head->ns_id);
 		ret = nvme_submit_sync_cmd(ns->queue, c, data, data_len);
drivers/nvme/host/sysfs.c +1 -1
index abf8edaae371..e95543fecb2a 100644
@@ -195,7 +195,7 @@ static int ns_head_update_nuse(struct nvme_ns_head *head)
 		return 0;
 
 	srcu_idx = srcu_read_lock(&head->srcu);
-	ns = nvme_find_path(head);
+	ns = nvme_find_path(head, NVME_STAT_GROUP_OTHER);
 	if (!ns)
 		goto out_unlock;
 
Signature · 1 line
2.53.0
maintainer
#9 lore raw

Re: [PATCH v8 04/10] nvme-multipath: pass I/O type to nvme_find_path()

Quoted 4 lines · Nilay Shroff
drivers/nvme/host/ioctl.c +0 -0
@@ -804,12 +830,24 @@ long nvme_ns_head_chr_ioctl(struct file *file, unsigned int cmd,
 int nvme_ns_head_chr_uring_cmd(struct io_uring_cmd *ioucmd,
 		unsigned int issue_flags)
 {

drivers/nvme/host/ioctl.c long nvme_ns_head_chr_ioctl(struct file *file, unsigned int cmd, · line 841

Quoted 10 lines
drivers/nvme/host/ioctl.c +5 -1
+	struct nvme_ns *ns;
+	unsigned int op_type;
 	struct cdev *cdev = file_inode(ioucmd->file)->i_cdev;
 	struct nvme_ns_head *head = container_of(cdev, struct nvme_ns_head, cdev);
 	int srcu_idx = srcu_read_lock(&head->srcu);
-	struct nvme_ns *ns = nvme_find_path(head);
 	int ret = -EINVAL;
+	const struct nvme_uring_cmd *cmd = io_uring_sqe128_cmd(ioucmd->sqe,
+						struct nvme_uring_cmd);
+	__u8 opcode = READ_ONCE(cmd->opcode);

Just fyi, the READ_ONCE is used because this is shared memory with user space. We re-read the same field later when constructing the command, so if the user does something tricky like changing that opcode, you can have the wrong group for the command that actually gets dispatched.

#10 lore raw

Re: [PATCH v8 04/10] nvme-multipath: pass I/O type to nvme_find_path()

Quoted 12 lines · Keith Busch
Quoted 11 lines · Nilay Shroff
drivers/nvme/host/ioctl.c +2 -1
@@ -804,12 +830,24 @@ long nvme_ns_head_chr_ioctl(struct file *file, unsigned int cmd,
  int nvme_ns_head_chr_uring_cmd(struct io_uring_cmd *ioucmd,
  		unsigned int issue_flags)
  {
+	struct nvme_ns *ns;
+	unsigned int op_type;
  	struct cdev *cdev = file_inode(ioucmd->file)->i_cdev;
  	struct nvme_ns_head *head = container_of(cdev, struct nvme_ns_head, cdev);
  	int srcu_idx = srcu_read_lock(&head->srcu);
-	struct nvme_ns *ns = nvme_find_path(head);
  	int ret = -EINVAL;
Quoted 8 lines

drivers/nvme/host/ioctl.c long nvme_ns_head_chr_ioctl(struct file *file, unsigned int cmd, · line 841

Quoted 3 lines
drivers/nvme/host/ioctl.c +3 -0
+	const struct nvme_uring_cmd *cmd = io_uring_sqe128_cmd(ioucmd->sqe,
+						struct nvme_uring_cmd);
+	__u8 opcode = READ_ONCE(cmd->opcode);

Just fyi, the READ_ONCE is used because this is shared memory with user space. We re-read the same field later when constructing the command, so if the user does something tricky like changing that opcode, you can have the wrong group for the command that actually gets dispatched.

Yes, agreed. If userspace modifies cmd->opcode while the SQE is being consumed by the kernel, we could end up selecting a path based on one opcode value and dispatching a command with another. However, per the io_uring ABI/protocol, userspace must not modify or reuse an SQE after publishing it to the kernel until the kernel has consumed it (i.e. advanced the SQ ring head). Therefore, under the expected protocol, the SQE contents are stable while the kernel is consuming them.

Nevertheless, if userspace modifies an SQE before it has been consumed by the kernel, then that violates the io_uring SQE ownership protocol, and the resulting behavior is not something we need to account for here, IMO.

Thanks,
--Nilay

#11 lore raw

PATCH v8 5/10 nvme-multipath: add support for latency I/O policy

In reply to Nilay Shroff

This commit introduces a new I/O policy named "latency". Users can configure it by writing "latency" to "/sys/class/nvme-subsystem/nvme- subsystemX/iopolicy"

The "latency" policy dynamically distributes I/O based on measured I/O completion latency. The main idea is to calculate latency for each path, derive a weight, and then proportionally forward I/O according to those weights.

To ensure scalability, path latency is measured per-CPU. Each CPU maintains its own statistics, and I/O forwarding uses these per-CPU values. Every ~15 seconds, a simple average latency of per-CPU batched samples are computed and fed into an Exponentially Weighted Moving Average (EWMA):

avg_latency = div_u64(batch, batch_count);
new_ewma_latency = (prev_ewma_latency * (WEIGHT-1) + avg_latency)/WEIGHT

With WEIGHT = 8, this assigns 7/8 (~87.5%) weight to the previous latency value and 1/8 (~12.5%) to the most recent latency. This smoothing reduces jitter, adapts quickly to changing conditions, avoids storing historical samples, and works well for both low and high I/O rates. Path weights are then derived from the smoothed (EWMA) latency as follows (example with two paths A and B):

path_A_score = NSEC_PER_SEC / path_A_ewma_latency path_B_score = NSEC_PER_SEC / path_B_ewma_latency total_score = path_A_score + path_B_score

path_A_weight = (path_A_score * 64) / total_score path_B_weight = (path_B_score * 64) / total_score

where:
- path_X_ewma_latency is the smoothed latency of a path in nanoseconds
- NSEC_PER_SEC is used as a scaling factor since valid latencies are < 1 second
- weights are normalized to a 0–64 scale across all paths.

Path credits are refilled based on this weight, with one credit consumed per I/O. When all credits are consumed, the credits are refilled again based on the current weight. This ensures that I/O is distributed across paths proportionally to their calculated weight.

---
 drivers/nvme/host/core.c      |  12 +-
 drivers/nvme/host/multipath.c | 462 +++++++++++++++++++++++++++++++++-
 drivers/nvme/host/nvme.h      |  48 +++-
 3 files changed, 507 insertions(+), 15 deletions(-)
drivers/nvme/host/core.c +9 -3
index 1322c678f4eb..a480e33fd984 100644
@@ -719,6 +719,7 @@ static void nvme_free_ns(struct kref *kref)
 {
 	struct nvme_ns *ns = container_of(kref, struct nvme_ns, kref);
 
+	nvme_free_ns_stat(ns);
 	put_disk(ns->disk);
 	nvme_put_ns_head(ns->head);
 	nvme_put_ctrl(ns->ctrl);
@@ -4258,6 +4259,9 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)
 	if (nvme_init_ns_head(ns, info))
 		goto out_cleanup_disk;
 
+	if (nvme_mpath_alloc_ns_stat(ns))
+		goto out_unlink_ns;
+
 	/*
 	 * If multipathing is enabled, the device name for all disks and not
 	 * just those that represent shared namespaces needs to be based on the
@@ -4282,7 +4286,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)
 	}
 
 	if (nvme_update_ns_info(ns, info))
-		goto out_unlink_ns;
+		goto out_free_ns_stat;
 
 	mutex_lock(&ctrl->namespaces_lock);
 	/*
@@ -4291,7 +4295,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)
 	 */
 	if (test_bit(NVME_CTRL_FROZEN, &ctrl->flags)) {
 		mutex_unlock(&ctrl->namespaces_lock);
-		goto out_unlink_ns;
+		goto out_free_ns_stat;
 	}
 	blk_queue_rq_timeout(ns->queue, ctrl->io_timeout);
 	nvme_ns_add_to_ctrl_list(ns);
@@ -4316,6 +4320,8 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)
 	list_del_rcu(&ns->list);
 	mutex_unlock(&ctrl->namespaces_lock);
 	synchronize_srcu(&ctrl->srcu);
+out_free_ns_stat:
+	nvme_free_ns_stat(ns);
  out_unlink_ns:
 	mutex_lock(&ctrl->subsys->lock);
 	list_del_rcu(&ns->siblings);
@@ -4355,7 +4361,7 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 
 	/*
 	 * Ensure that !NVME_NS_READY is seen by other threads to prevent
-	 * this ns going back into current_path.
+	 * this ns going back into current_path/latency_path.
 	 */
 	synchronize_srcu(&ns->head->srcu);
 
drivers/nvme/host/multipath.c +451 -11

569 lines not on this page · raw message

drivers/nvme/host/nvme.h +47 -1

102 lines not on this page · raw message

Signature · 1 line
2.53.0
#12 lore raw

PATCH v8 6/10 nvme: add generic debugfs support

In reply to Nilay Shroff

Add generic infrastructure for creating and managing debugfs files in the NVMe module. This introduces helper APIs that allow NVMe drivers to register and unregister debugfs entries, along with a reusable attribute structure for defining new debugfs files.

The implementation uses seq_file interfaces to safely expose per-NS and per-NS-head statistics, while supporting both simple show callbacks and full seq_operations.

---
 drivers/nvme/host/Makefile    |   2 +-
 drivers/nvme/host/core.c      |   3 +
 drivers/nvme/host/debugfs.c   | 141 ++++++++++++++++++++++++++++++++++
 drivers/nvme/host/multipath.c |   2 +
 drivers/nvme/host/nvme.h      |  10 +++
 5 files changed, 157 insertions(+), 1 deletion(-)
 create mode 100644 drivers/nvme/host/debugfs.c
drivers/nvme/host/Makefile +1 -1
index 67563a69f7dc..314d3439406b 100644
@@ -11,7 +11,7 @@ obj-$(CONFIG_NVME_FC)			+= nvme-fc.o
 obj-$(CONFIG_NVME_TCP)			+= nvme-tcp.o
 obj-$(CONFIG_NVME_APPLE)		+= nvme-apple.o
 
-nvme-core-y				+= core.o ioctl.o sysfs.o pr.o
+nvme-core-y				+= core.o ioctl.o sysfs.o pr.o debugfs.o
 nvme-core-$(CONFIG_NVME_VERBOSE_ERRORS)	+= constants.o
 nvme-core-$(CONFIG_TRACING)		+= trace.o
 nvme-core-$(CONFIG_NVME_MULTIPATH)	+= multipath.o
drivers/nvme/host/core.c +3 -0
index a480e33fd984..0fde693dd9b8 100644
@@ -4306,6 +4306,8 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)
 	if (device_add_disk(ctrl->device, ns->disk, nvme_ns_attr_groups))
 		goto out_cleanup_ns_from_list;
 
+	nvme_debugfs_register(ns->disk);
+
 	if (!nvme_ns_head_multipath(ns->head))
 		nvme_add_ns_cdev(ns);
 
@@ -4388,6 +4390,7 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 
 	nvme_mpath_remove_sysfs_link(ns);
 
+	nvme_debugfs_unregister(ns->disk);
 	del_gendisk(ns->disk);
 
 	mutex_lock(&ns->ctrl->namespaces_lock);
drivers/nvme/host/debugfs.c new file +141 -0

141 lines not on this page · raw message

drivers/nvme/host/multipath.c +2 -0
index 5501c22bd662..8d5f1e42d10a 100644
@@ -1137,6 +1137,7 @@ static void nvme_remove_head(struct nvme_ns_head *head)
 
 		if (test_and_clear_bit(NVME_NSHEAD_CDEV_LIVE, &head->flags))
 			nvme_cdev_del(&head->cdev, &head->cdev_device);
+		nvme_debugfs_unregister(head->disk);
 		del_gendisk(head->disk);
 	}
 	nvme_put_ns_head(head);
@@ -1244,6 +1245,7 @@ static void nvme_mpath_set_live(struct nvme_ns *ns)
 		}
 		nvme_add_ns_head_cdev(head);
 		queue_work(nvme_wq, &head->partition_scan_work);
+		nvme_debugfs_register(head->disk);
 	}
 
 	nvme_mpath_add_sysfs_link(ns->head);
drivers/nvme/host/nvme.h +10 -0
index 850a5f6afca5..040307c6534a 100644
@@ -1127,6 +1127,16 @@ static inline enum nvme_stat_group nvme_get_stat_group(struct request *req)
 	return __nvme_get_stat_group(req_op(req));
 }
 
+void nvme_debugfs_register(struct gendisk *disk);
+static inline void nvme_debugfs_unregister(struct gendisk *disk)
+{
+	/*
+	 * Nothing to do for now. When the request queue is unregistered,
+	 * all files under q->debugfs_dir are recursively deleted.
+	 * This is just a placeholder; the compiler will optimize it out.
+	 */
+}
+
 #ifdef CONFIG_NVME_MULTIPATH
 static inline bool nvme_ctrl_use_ana(struct nvme_ctrl *ctrl)
 {
Signature · 1 line
2.53.0
#13 lore raw

PATCH v8 7/10 nvme-multipath: add debugfs attribute latency_ewma_shift

In reply to Nilay Shroff

By default, the EWMA (Exponentially Weighted Moving Average) shift value, used for storing latency samples for latency iopolicy, is set to 3. The EWMA is calculated using the following formula:

ewma = (old * ((1 << ewma_shift) - 1) + new) >> ewma_shift;

The default value of 3 assigns ~87.5% weight to the existing EWMA value and ~12.5% weight to the new latency sample. This provides a stable average that smooths out short-term variations.

However, different workloads may require faster or slower adaptation to changing conditions. This commit introduces a new debugfs attribute, latency_ewma_shift, allowing users to tune the weighting factor.

For example:
- latency_ewma_shift = 2 => 75% old, 25% new
- latency_ewma_shift = 1 => 50% old, 50% new
- latency_ewma_shift = 0 => 0% old, 100% new

---
 drivers/nvme/host/debugfs.c   | 46 +++++++++++++++++++++++++++++++++++
 drivers/nvme/host/multipath.c |  9 ++++---
 drivers/nvme/host/nvme.h      |  1 +
 3 files changed, 52 insertions(+), 4 deletions(-)
drivers/nvme/host/debugfs.c +46 -0
index 04f8250fe607..ca1a670bea78 100644
@@ -108,8 +108,54 @@ static const struct file_operations nvme_debugfs_fops = {
 	.release = nvme_debugfs_release,
 };
 
+#ifdef CONFIG_NVME_MULTIPATH
+static int nvme_latency_ewma_shift_show(void *data, struct seq_file *m)
+{
+	struct nvme_ns_head *head = data;
+
+	seq_printf(m, "%u\n", READ_ONCE(head->latency_ewma_shift));
+	return 0;
+}
+
+static ssize_t nvme_latency_ewma_shift_store(void *data,
+		const char __user *ubuf, size_t count, loff_t *ppos)
+{
+	struct nvme_ns_head *head = data;
+	char kbuf[8];
+	u32 res;
+	int ret;
+	size_t len;
+	char *arg;
+
+	len = min(sizeof(kbuf) - 1, count);
+
+	if (copy_from_user(kbuf, ubuf, len))
+		return -EFAULT;
+
+	kbuf[len] = '\0';
+	arg = strstrip(kbuf);
+
+	ret = kstrtou32(arg, 0, &res);
+	if (ret)
+		return ret;
+
+	/*
+	 * Values greater than 8 are nonsensical, as they effectively assign
+	 * zero weight to new samples.
+	 */
+	if (res > 8)
+		return -EINVAL;
+
+	WRITE_ONCE(head->latency_ewma_shift, res);
+	return count;
+}
+#endif
 
 static const struct nvme_debugfs_attr nvme_mpath_debugfs_attrs[] = {
+#ifdef CONFIG_NVME_MULTIPATH
+	{"latency_ewma_shift", 0600, nvme_latency_ewma_shift_show,
+			nvme_latency_ewma_shift_store},
+#endif
 	{},
 };
 
drivers/nvme/host/multipath.c +5 -4
index 8d5f1e42d10a..82a4f2326b79 100644
@@ -293,10 +293,9 @@ static void nvme_mpath_weight_work(struct work_struct *weight_work)
  * For instance, with EWMA_SHIFT = 3, this assigns 7/8 (~87.5 %) weight to
  * the existing/old ewma and 1/8 (~12.5%) weight to the new sample.
  */
-static inline u64 calc_ewma_update(u64 old, u64 new)
+static inline u64 calc_ewma_update(u64 old, u64 new, u32 ewma_shift)
 {
-	return (old * ((1 << NVME_DEFAULT_LATENCY_EWMA_SHIFT) - 1)
-			+ new) >> NVME_DEFAULT_LATENCY_EWMA_SHIFT;
+	return (old * ((1 << ewma_shift) - 1) + new) >> ewma_shift;
 }
 
 static void nvme_mpath_add_sample(struct request *rq, struct nvme_ns *ns)
@@ -388,7 +387,8 @@ static void nvme_mpath_add_sample(struct request *rq, struct nvme_ns *ns)
 		if (unlikely(!stat->slat_ns))
 			WRITE_ONCE(stat->slat_ns, avg_lat_ns);
 		else {
-			slat_ns = calc_ewma_update(stat->slat_ns, avg_lat_ns);
+			slat_ns = calc_ewma_update(stat->slat_ns, avg_lat_ns,
+					READ_ONCE(head->latency_ewma_shift));
 			WRITE_ONCE(stat->slat_ns, slat_ns);
 		}
 
@@ -1170,6 +1170,7 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)
 	INIT_WORK(&head->requeue_work, nvme_requeue_work);
 	INIT_WORK(&head->partition_scan_work, nvme_partition_scan_work);
 	INIT_DELAYED_WORK(&head->remove_work, nvme_remove_head_work);
+	head->latency_ewma_shift = NVME_DEFAULT_LATENCY_EWMA_SHIFT;
 
 	/*
 	 * If "multipath_always_on" is enabled, a multipath node is added
drivers/nvme/host/nvme.h +1 -0
index 040307c6534a..55e4d1344e5e 100644
@@ -626,6 +626,7 @@ struct nvme_ns_head {
 	atomic_long_t		io_requeue_no_usable_path_count;
 	atomic_long_t		io_fail_no_available_path_count;
 	struct nvme_ns * __percpu	*latency_path;
+	u32				latency_ewma_shift;
 
 #define NVME_NSHEAD_DISK_LIVE		0
 #define NVME_NSHEAD_QUEUE_IF_NO_PATH	1
Signature · 1 line
2.53.0
#14 lore raw

PATCH v8 8/10 nvme-multipath: add debugfs attribute latency_batch_timeout

In reply to Nilay Shroff

By default, the latency I/O policy accumulates latency samples over a 15-second window. When this window expires, the driver computes the average latency and updates the smoothed (EWMA) latency value. The path weight is then recalculated based on this data.

A 15-second window provides a good balance for most workloads, as it helps smooth out transient latency spikes and produces a more stable path weight profile. However, some workloads may benefit from faster or slower adaptation to changing latency conditions.

This commit introduces a new debugfs attribute, latency_batch_timeout, which allows users to configure the latency batch window and thus path weight calculation interval based on their workload requirements.

---
 drivers/nvme/host/debugfs.c   | 37 +++++++++++++++++++++++++++++++++++
 drivers/nvme/host/multipath.c |  8 ++++++--
 drivers/nvme/host/nvme.h      |  1 +
 3 files changed, 44 insertions(+), 2 deletions(-)
drivers/nvme/host/debugfs.c +37 -0
index ca1a670bea78..b8850d37edce 100644
@@ -149,12 +149,49 @@ static ssize_t nvme_latency_ewma_shift_store(void *data,
 	WRITE_ONCE(head->latency_ewma_shift, res);
 	return count;
 }
+
+static int nvme_latency_batch_timeout_show(void *data, struct seq_file *m)
+{
+	struct nvme_ns_head *head = data;
+
+	seq_printf(m, "%llu\n",
+		div_u64(READ_ONCE(head->latency_batch_timeout), NSEC_PER_SEC));
+	return 0;
+}
+
+static ssize_t nvme_latency_batch_timeout_store(void *data,
+		const char __user *ubuf, size_t count, loff_t *ppos)
+{
+	struct nvme_ns_head *head = data;
+	char kbuf[8];
+	u32 res;
+	int ret;
+	size_t len;
+	char *arg;
+
+	len = min(sizeof(kbuf) - 1, count);
+
+	if (copy_from_user(kbuf, ubuf, len))
+		return -EFAULT;
+
+	kbuf[len] = '\0';
+	arg = strstrip(kbuf);
+
+	ret = kstrtou32(arg, 0, &res);
+	if (ret)
+		return ret;
+
+	WRITE_ONCE(head->latency_batch_timeout, res * NSEC_PER_SEC);
+	return count;
+}
 #endif
 
 static const struct nvme_debugfs_attr nvme_mpath_debugfs_attrs[] = {
 #ifdef CONFIG_NVME_MULTIPATH
 	{"latency_ewma_shift", 0600, nvme_latency_ewma_shift_show,
 			nvme_latency_ewma_shift_store},
+	{"latency_batch_timeout", 0600, nvme_latency_batch_timeout_show,
+			nvme_latency_batch_timeout_store},
 #endif
 	{},
 };
drivers/nvme/host/multipath.c +6 -2
index 82a4f2326b79..a4cca27e8eb5 100644
@@ -362,8 +362,11 @@ static void nvme_mpath_add_sample(struct request *rq, struct nvme_ns *ns)
 	stat->batch_count++;
 	stat->nr_samples++;
 
-	if (now > stat->last_batch_ts && ((now - stat->last_batch_ts) >=
-			NVME_DEFAULT_LATENCY_BATCH_TIMEOUT)) {
+	if (now > stat->last_batch_ts) {
+		u64 timeout = READ_ONCE(head->latency_batch_timeout);
+
+		if ((now - stat->last_batch_ts) < timeout)
+			return;
 
 		/*
 		 * Find simple average latency for the last epoch (~15 sec
@@ -1171,6 +1174,7 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)
 	INIT_WORK(&head->partition_scan_work, nvme_partition_scan_work);
 	INIT_DELAYED_WORK(&head->remove_work, nvme_remove_head_work);
 	head->latency_ewma_shift = NVME_DEFAULT_LATENCY_EWMA_SHIFT;
+	head->latency_batch_timeout = NVME_DEFAULT_LATENCY_BATCH_TIMEOUT;
 
 	/*
 	 * If "multipath_always_on" is enabled, a multipath node is added
drivers/nvme/host/nvme.h +1 -0
index 55e4d1344e5e..133a22611597 100644
@@ -627,6 +627,7 @@ struct nvme_ns_head {
 	atomic_long_t		io_fail_no_available_path_count;
 	struct nvme_ns * __percpu	*latency_path;
 	u32				latency_ewma_shift;
+	u64				latency_batch_timeout;
 
 #define NVME_NSHEAD_DISK_LIVE		0
 #define NVME_NSHEAD_QUEUE_IF_NO_PATH	1
Signature · 1 line
2.53.0
#15 lore raw

PATCH v8 9/10 nvme-multipath: add debugfs attribute latency_stat

In reply to Nilay Shroff

This commit introduces a new debugfs attribute, "latency_stat", under both per-path and head debugfs directories (defined under /sys/kernel/ debug/block/). This attribute provides visibility into the internal state of the latency I/O policy to aid in debugging and performance analysis.

For per-path entries, "latency_stat" reports the corresponding path statistics such as I/O weight, selection count, processed samples, and ignored samples.

For head entries, it reports per-CPU statistics for each reachable path, including I/O weight, path score, smoothed (EWMA) latency, selection count, processed samples, and ignored samples.

These additions enhance observability of the I/O path selection behavior and help diagnose imbalance or instability in multipath performance.

---
 drivers/nvme/host/debugfs.c | 123 ++++++++++++++++++++++++++++++++++++
 1 file changed, 123 insertions(+)
drivers/nvme/host/debugfs.c +123 -0
index b8850d37edce..b3496951f0b1 100644
@@ -184,6 +184,125 @@ static ssize_t nvme_latency_batch_timeout_store(void *data,
 	WRITE_ONCE(head->latency_batch_timeout, res * NSEC_PER_SEC);
 	return count;
 }
+
+#define TO_CTX(m)       ((struct nvme_debugfs_ctx *)m->private)
+#define TO_NS_HEAD(m)   ((struct nvme_ns_head *)(TO_CTX(m)->data))
+
+static void *nvme_mpath_latency_stat_start(struct seq_file *m, loff_t *pos)
+	__acquires_shared(&TO_NS_HEAD(m)->srcu)
+{
+	struct nvme_ns *ns;
+	struct nvme_debugfs_ctx *ctx = m->private;
+	struct nvme_ns_head *head = ctx->data;
+	loff_t n = *pos;
+
+	/* Remember srcu index, so we can unlock later. */
+	ctx->srcu_idx = srcu_read_lock(&head->srcu);
+	ns = list_first_or_null_rcu(&head->list, struct nvme_ns, siblings);
+
+	while (n && ns) {
+		ns = list_next_or_null_rcu(&head->list, &ns->siblings,
+				struct nvme_ns, siblings);
+		n--;
+	}
+
+	return ns;
+}
+
+static void *nvme_mpath_latency_stat_next(struct seq_file *m, void *v,
+		loff_t *pos)
+{
+	struct nvme_ns *ns = v;
+	struct nvme_debugfs_ctx *ctx = m->private;
+	struct nvme_ns_head *head = ctx->data;
+
+	(*pos)++;
+
+	return list_next_or_null_rcu(&head->list, &ns->siblings,
+			struct nvme_ns, siblings);
+}
+
+static void nvme_mpath_latency_stat_stop(struct seq_file *m, void *v)
+	__releases_shared(&TO_NS_HEAD(m)->srcu)
+{
+	struct nvme_debugfs_ctx *ctx = m->private;
+	struct nvme_ns_head *head = ctx->data;
+
+	srcu_read_unlock(&head->srcu, ctx->srcu_idx);
+}
+
+static int nvme_mpath_latency_stat_show(struct seq_file *m, void *v)
+{
+	int i, cpu;
+	struct nvme_path_lat_stat *stat;
+	struct nvme_ns *ns = v;
+
+	seq_printf(m, "%s:\n", ns->disk->disk_name);
+	for_each_online_cpu(cpu) {
+		seq_printf(m, "cpu %d : ", cpu);
+		for (i = 0; i < NVME_NUM_STAT_GROUPS; i++) {
+			stat = &per_cpu_ptr(ns->path_lat, cpu)[i].stat;
+			seq_printf(m, "%u %u %llu %llu %llu %llu %llu ",
+				stat->weight, stat->credit, stat->score,
+				stat->slat_ns, stat->sel,
+				stat->nr_samples, stat->nr_ignored);
+		}
+		seq_putc(m, '\n');
+	}
+	return 0;
+}
+
+static const struct seq_operations nvme_mpath_latency_stat_seq_ops = {
+	.start = nvme_mpath_latency_stat_start,
+	.next  = nvme_mpath_latency_stat_next,
+	.stop  = nvme_mpath_latency_stat_stop,
+	.show  = nvme_mpath_latency_stat_show
+};
+
+static void nvme_latency_stat_read_all(struct nvme_ns *ns,
+		struct nvme_path_lat_stat *batch)
+{
+	int i, cpu;
+	u32 ncpu[NVME_NUM_STAT_GROUPS] = {0};
+	struct nvme_path_lat_stat *stat;
+
+	for_each_online_cpu(cpu) {
+		for (i = 0; i < NVME_NUM_STAT_GROUPS; i++) {
+			stat = &per_cpu_ptr(ns->path_lat, cpu)[i].stat;
+			batch[i].sel += stat->sel;
+			batch[i].nr_samples += stat->nr_samples;
+			batch[i].nr_ignored += stat->nr_ignored;
+			batch[i].weight += stat->weight;
+			if (stat->weight)
+				ncpu[i]++;
+		}
+	}
+
+	for (i = 0; i < NVME_NUM_STAT_GROUPS; i++) {
+		if (!ncpu[i])
+			continue;
+		batch[i].weight = DIV_U64_ROUND_CLOSEST(batch[i].weight,
+				ncpu[i]);
+	}
+}
+
+static int nvme_ns_latency_stat_show(void *data, struct seq_file *m)
+{
+	int i;
+	struct nvme_path_lat_stat stat[NVME_NUM_STAT_GROUPS] = {0};
+	struct nvme_ns *ns = (struct nvme_ns *)data;
+
+	if (!ns->head->disk)
+		return 0;
+
+	nvme_latency_stat_read_all(ns, stat);
+	for (i = 0; i < NVME_NUM_STAT_GROUPS; i++) {
+		seq_printf(m, "%u %llu %llu %llu ",
+			stat[i].weight, stat[i].sel,
+			stat[i].nr_samples, stat[i].nr_ignored);
+	}
+	return 0;
+}
 #endif
 
 static const struct nvme_debugfs_attr nvme_mpath_debugfs_attrs[] = {
@@ -192,11 +311,15 @@ static const struct nvme_debugfs_attr nvme_mpath_debugfs_attrs[] = {
 			nvme_latency_ewma_shift_store},
 	{"latency_batch_timeout", 0600, nvme_latency_batch_timeout_show,
 			nvme_latency_batch_timeout_store},
+	{"latency_stat", 0400, .seq_ops = &nvme_mpath_latency_stat_seq_ops},
 #endif
 	{},
 };
 
 static const struct nvme_debugfs_attr nvme_ns_debugfs_attrs[] = {
+#ifdef CONFIG_NVME_MULTIPATH
+	{"latency_stat", 0400, nvme_ns_latency_stat_show},
+#endif
 	{},
 };
 
Signature · 1 line
2.53.0
#16 lore raw

PATCH v8 10/10 nvme-multipath: add documentation for latency I/O policy

In reply to Nilay Shroff

Update the nvme-multipath documentation to describe the latency I/O policy, its behavior, and when it is suitable for use.

---
 Documentation/admin-guide/nvme-multipath.rst | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
Documentation/admin-guide/nvme-multipath.rst +19 -0
index 97ca1ccef459..41a8054638ff 100644
@@ -70,3 +70,22 @@ When to use the queue-depth policy:
   1. High load with small I/Os: Effectively balances load across paths when
      the load is high, and I/O operations consist of small, relatively
      fixed-sized requests.
+
+Latency
+--------
+
+The latency policy manages I/O requests based on path latency. It periodically
+calculates a weight for each path and distributes I/O accordingly. Paths with
+higher latency receive lower weights, resulting in fewer I/O requests being sent
+to them, while paths with lower latency handle a proportionally larger share of
+the I/O load.
+
+When to use the latency policy
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. Homogeneous Path Performance: Utilizes all available paths efficiently when
+   their performance characteristics (e.g., latency, bandwidth) are similar.
+
+2. Heterogeneous Path Performance: Dynamically distributes I/O based on per-path
+   performance characteristics. Paths with lower latency receive a higher share
+   of I/O compared to those with higher latency.
Signature · 1 line
2.53.0
maintainer
#17 lore raw
In reply to Nilay Shroff
Quoted 20 lines · Nilay Shroff

Hi,

This series introduces a new latency I/O policy for NVMe native multipath. Existing policies such as numa, round-robin, and queue-depth are static and do not adapt to real-time transport performance. The numa selects the path closest to the NUMA node of the current CPU, optimizing memory and path locality, but ignores actual path performance. The round-robin distributes I/O evenly across all paths, providing fairness but not performance awareness. The queue-depth reacts to instantaneous queue occupancy, avoiding heavily loaded paths, but does not account for actual latency, throughput, or link speed.

The new latency policy addresses these gaps selecting paths dynamically based on measured I/O latency for both PCIe and fabrics. Latency is derived by passively sampling I/O completions. Each path is assigned a weight proportional to its latency score, and I/Os are then forwarded accordingly. As condition changes (e.g. latency spikes, bandwidth differences), path weights are updated, automatically steering traffic toward better-performing paths.

Quoted 5 lines

Early results show reduced tail latency under mixed workloads and improved throughput by exploiting higher-speed links more effectively. For example, with NVMf/TCP using two paths (one throttled with ~30 ms delay), fio results with random read/write/rw workloads (direct I/O) showed:

TBH, I do not know if this measurement represent any real-life scenario. I do think that occasional packet drops are a real-life scenario, and it would be a worthy use-case to optimize for. Can you perhaps measure how the path selectors compare in this case?

Quoted 7 lines
         numa         round-robin   queue-depth  adaptive
         -----------  -----------   -----------  ---------
READ:   50.0 MiB/s   105 MiB/s     230 MiB/s    350 MiB/s
WRITE:  65.9 MiB/s   125 MiB/s     385 MiB/s    446 MiB/s
RW:     R:30.6 MiB/s R:56.5 MiB/s  R:122 MiB/s  R:175 MiB/s
         W:30.7 MiB/s W:56.5 MiB/s  W:122 MiB/s  W:175 MiB/s

And I'm assuming there are zero downsides for the normal case?

Quoted 60 lines

This pathcset includes totla 8 patches:
[PATCH 1/10] block: expose blk_stat_{enable,disable}_accounting()
- Make blk_stat APIs available to block drivers.
- Needed for per-path latency measurement.

[PATCH 2/10] block: record I/O request start time for passthru request
- Record I/O start time for I/O passthru requests.
- This is prep patch which allows measuring I/O completion latency for passthru requests.

[PATCH 3/10] block: support nesting for blk-mq flag QUEUE_FLAG_SAME_FORCE
- Support nesting for QUEUE_FLAG_SAME_FORCE as multiple users could toggle QUEUE_FLAG_SAME_FORCE.

[PATCH 4/10] nvme-multipath: pass I/O type to nvme_find_path()
- This is the prep patch which updates nvme_find_path() signature

[PATCH 5/10] nvme-multipath: add latency I/O policy
- Implement path scoring based on latency (EWMA).
- Distribute I/O proportionally to per-path weights.

[PATCH 6/10] nvme: add generic debugfs support
- Introduce generic debugfs support for NVMe module

[PATCH 7/10] nvme-multipath: add debugfs attribute latency_ewma_shift
- Adds a debugfs attribute to control ewma shift

[PATCH 8/10] nvme-multipath: add debugfs attribute latency_batch_timeout
- Adds a debugfs attribute to control latency batch window interval

[PATCH 9/10] nvme-multipath: add debugfs attribute latency_stat
- Add “latency_stat” under per-path and head debugfs directories to expose latency policy state and statistics.

[PATCH 10/10] nvme-multipath: add documentation for latency I/O policy
- Includes documentation for latency I/O multipath policy.

LSFMM discussion:
=================
During lsfmm 2026, it was decided to rename this I/O policy from "adaptive" to "latency". This series reflects that rename.

The discussion at lsfmm also focused extensively on the latency measurement model, including whether latency should be tracked per-CPU or per-NUMA, and whether separate I/O-size buckets should be maintained for different request sizes.

After detailed discussion and evaluation of throughput results, the consensus was to initially measure I/O completion latency on a per-CPU basis. The available performance data showed that the per-CPU implementation already provides sufficient averaging across CPUs while keeping the design relatively simple.

The use of additional I/O-size buckets did not demonstrate meaningful throughput improvement in the general case and would introduce extra complexity into the fast path and accounting logic. As a result, the consensus was to avoid I/O-size bucketing for now and keep the policy focused on per-CPU latency measurement.

Quoted 5 lines

If future real-world workloads demonstrate a clear benefit from I/O-size-aware latency accounting, the policy can be extended later to support it.

As ususal, feedback and suggestions are most welcome!

Nilay, do we have evidence that round-robin/queue-depth are better for any workload? As a user, I would be very confused with the amount of path selectors I have available and which should I choose.

#18 lore raw
Quoted 29 lines · Sagi Grimberg
Quoted 20 lines · Nilay Shroff

Hi,

This series introduces a new latency I/O policy for NVMe native multipath. Existing policies such as numa, round-robin, and queue-depth are static and do not adapt to real-time transport performance. The numa selects the path closest to the NUMA node of the current CPU, optimizing memory and path locality, but ignores actual path performance. The round-robin distributes I/O evenly across all paths, providing fairness but not performance awareness. The queue-depth reacts to instantaneous queue occupancy, avoiding heavily loaded paths, but does not account for actual latency, throughput, or link speed.

The new latency policy addresses these gaps selecting paths dynamically based on measured I/O latency for both PCIe and fabrics. Latency is derived by passively sampling I/O completions. Each path is assigned a weight proportional to its latency score, and I/Os are then forwarded accordingly. As condition changes (e.g. latency spikes, bandwidth differences), path weights are updated, automatically steering traffic toward better-performing paths.

Quoted 5 lines

Early results show reduced tail latency under mixed workloads and improved throughput by exploiting higher-speed links more effectively. For example, with NVMf/TCP using two paths (one throttled with ~30 ms delay), fio results with random read/write/rw workloads (direct I/O) showed:

Quoted 5 lines

TBH, I do not know if this measurement represent any real-life scenario. I do think that occasional packet drops are a real-life scenario, and it would be a worthy use-case to optimize for. Can you perhaps measure how the path selectors compare in this case?

Okay so I have now measured another workload where I simulated packet loss/drops which are occasional and compared different path selectors. In this measurement I have a shared NVMe namespace configured which is reachable over two tcp paths. Now to simulate the occasional packet loss/drop I have configured one of the paths to experience occasional packet loss/drop as shown below over the period of 480 seconds:

0 120 240 360 480

|-------------|-------------|-------------|-------------|
|<--5% drop-->|<--no drop-->|<--3% drop-->|<--no drop-->|

As shown above for the first 120 seconds the path experiences 5% packet drop, for the next 120 seconds path sees no packet drop and again for subsequent 120 seconds path experiences 3% packet drop and for the rest of the duration (during last 120 seconds) there's no packet drop observed by the path. With this simulation, I ran fio workload for 480 seconds leveraging direct I/O, bs=4k, iodepth=64, numjobs=32 and ioengine=io_uring. Shown below is bw observed running fio test using different I/O policies:

             numa     round-robin queue-depth latency
             (MiB/s)  (MiB/s)     (MiB/s)     (MiB/s)
             -------  ----------- ----------- ---------
randread:    1288     1202        1464        1642
randwrite:   1456     1493        1779        1944
randrw:      R:623    R:594       R:750       R:822
             W:623    W:594       W:750       W:822
Quoted 11 lines
Quoted 7 lines

         numa         round-robin   queue-depth  adaptive -----------  -----------   -----------  --------- READ:   50.0 MiB/s   105 MiB/s     230 MiB/s    350 MiB/s WRITE:  65.9 MiB/s   125 MiB/s     385 MiB/s    446 MiB/s RW:     R:30.6 MiB/s R:56.5 MiB/s  R:122 MiB/s  R:175 MiB/s W:30.7 MiB/s W:56.5 MiB/s  W:122 MiB/s  W:175 MiB/s

And I'm assuming there are zero downsides for the normal case?

For the normal case where all paths are symmetric I saw queue-depth, latency and round-robin policies yielding nearly same bandwidth. However for numa policy, it depends on CPU/numa locality.

Quoted 63 lines
Quoted 63 lines

This pathcset includes totla 8 patches:
[PATCH 1/10] block: expose blk_stat_{enable,disable}_accounting()
   - Make blk_stat APIs available to block drivers.
   - Needed for per-path latency measurement.

[PATCH 2/10] block: record I/O request start time for passthru request
   - Record I/O start time for I/O passthru requests.
   - This is prep patch which allows measuring I/O completion latency for passthru requests.

[PATCH 3/10] block: support nesting for blk-mq flag QUEUE_FLAG_SAME_FORCE
   - Support nesting for QUEUE_FLAG_SAME_FORCE as multiple users could toggle QUEUE_FLAG_SAME_FORCE.

[PATCH 4/10] nvme-multipath: pass I/O type to nvme_find_path()
   - This is the prep patch which updates nvme_find_path() signature [PATCH 5/10] nvme-multipath: add latency I/O policy
   - Implement path scoring based on latency (EWMA).
   - Distribute I/O proportionally to per-path weights.

[PATCH 6/10] nvme: add generic debugfs support
   - Introduce generic debugfs support for NVMe module

[PATCH 7/10] nvme-multipath: add debugfs attribute latency_ewma_shift
   - Adds a debugfs attribute to control ewma shift

[PATCH 8/10] nvme-multipath: add debugfs attribute latency_batch_timeout
   - Adds a debugfs attribute to control latency batch window interval

[PATCH 9/10] nvme-multipath: add debugfs attribute latency_stat
   - Add “latency_stat” under per-path and head debugfs directories to expose latency policy state and statistics.

[PATCH 10/10] nvme-multipath: add documentation for latency I/O policy
   - Includes documentation for latency I/O multipath policy.

LSFMM discussion:
=================
During lsfmm 2026, it was decided to rename this I/O policy from "adaptive" to "latency". This series reflects that rename.

The discussion at lsfmm also focused extensively on the latency measurement model, including whether latency should be tracked per-CPU or per-NUMA, and whether separate I/O-size buckets should be maintained for different request sizes.

After detailed discussion and evaluation of throughput results, the consensus was to initially measure I/O completion latency on a per-CPU basis. The available performance data showed that the per-CPU implementation already provides sufficient averaging across CPUs while keeping the design relatively simple.

The use of additional I/O-size buckets did not demonstrate meaningful throughput improvement in the general case and would introduce extra complexity into the fast path and accounting logic. As a result, the consensus was to avoid I/O-size bucketing for now and keep the policy focused on per-CPU latency measurement.

If future real-world workloads demonstrate a clear benefit from I/O-size-aware latency accounting, the policy can be extended later to support it.

Quoted 5 lines
Quoted 1 line

As ususal, feedback and suggestions are most welcome!

Nilay, do we have evidence that round-robin/queue-depth are better for any workload? As a user, I would be very confused with the amount of path selectors I have available and which should I choose.

From my experiments, when the paths are symmetric, both round-robin and queue-depth (and for that matter latency) exhibit similar behavior, with the workload being distributed roughly equally across the active paths.

When the paths are asymmetric, I found queue-depth to perform better than round-robin. Queue-depth tries to steer I/O toward the less-loaded path based on the number of in-flight I/Os on each path, whereas round-robin continues to distribute I/O evenly across all active paths.

However, queue-depth still has a limitation in this scenario. It uses the number of in-flight I/Os as an indirect indication of path performance, it does not have a direct signal of the actual I/O completion latency. For example, if one path starts experiencing packet loss, I/O completion on that path can become significantly slower. Queue-depth can react to this as the path accumulates more outstanding I/Os, but it can still continue sending I/O to the degraded path as long as its queue depth remains comparable to the healthy path. In other words, it can reduce the amount of I/O sent to the degraded path, but it cannot directly account for how much slower that path has become.

The latency policy uses I/O completion latency as the signal instead. When one path becomes degraded, its observed latency increases and its path score/weight decreases. Consequently, the policy shifts more I/O towards the healthy path. This allows the healthy path to sustain a higher queue depth while the degraded path receives substantially less I/O, rather than trying to maintain a similar queue depth across both paths.

This is also reflected in the packet-loss experiment above. Round-robin continues to distribute I/O across both paths, while queue-depth does a better job by reacting to the increased queue occupancy of the degraded path. The latency policy goes one step further by directly using the increased completion latency as a signal and therefore steers more I/O toward the healthy path, resulting in higher throughput.

So based on the results I have so far, I would characterize the existing policies as follows: round-robin is useful when paths are symmetric and equal distribution is desired. The queue-depth is preferable when paths are asymmetric and queue occupancy provides a useful indication of path load and the latency policy is intended for cases where path performance can vary dynamically and we want the policy to adapt based on actual observed latency.

Thanks,
--Nilay

maintainer
#19 lore raw
Quoted 182 lines · Nilay Shroff
Quoted 29 lines · Sagi Grimberg
Quoted 20 lines · Nilay Shroff

Hi,

This series introduces a new latency I/O policy for NVMe native multipath. Existing policies such as numa, round-robin, and queue-depth are static and do not adapt to real-time transport performance. The numa selects the path closest to the NUMA node of the current CPU, optimizing memory and path locality, but ignores actual path performance. The round-robin distributes I/O evenly across all paths, providing fairness but not performance awareness. The queue-depth reacts to instantaneous queue occupancy, avoiding heavily loaded paths, but does not account for actual latency, throughput, or link speed.

The new latency policy addresses these gaps selecting paths dynamically based on measured I/O latency for both PCIe and fabrics. Latency is derived by passively sampling I/O completions. Each path is assigned a weight proportional to its latency score, and I/Os are then forwarded accordingly. As condition changes (e.g. latency spikes, bandwidth differences), path weights are updated, automatically steering traffic toward better-performing paths.

Quoted 5 lines

Early results show reduced tail latency under mixed workloads and improved throughput by exploiting higher-speed links more effectively. For example, with NVMf/TCP using two paths (one throttled with ~30 ms delay), fio results with random read/write/rw workloads (direct I/O) showed:

Quoted 5 lines

TBH, I do not know if this measurement represent any real-life scenario. I do think that occasional packet drops are a real-life scenario, and it would be a worthy use-case to optimize for. Can you perhaps measure how the path selectors compare in this case?

Okay so I have now measured another workload where I simulated packet loss/drops which are occasional and compared different path selectors. In this measurement I have a shared NVMe namespace configured which is reachable over two tcp paths. Now to simulate the occasional packet loss/drop I have configured one of the paths to experience occasional packet loss/drop as shown below over the period of 480 seconds:

0            120           240           360           480

|-------------|-------------|-------------|-------------|
|<--5% drop-->|<--no drop-->|<--3% drop-->|<--no drop-->|

As shown above for the first 120 seconds the path experiences 5% packet drop, for the next 120 seconds path sees no packet drop and again for subsequent 120 seconds path experiences 3% packet drop and for the rest of the duration (during last 120 seconds) there's no packet drop observed by the path. With this simulation, I ran fio workload for 480 seconds leveraging direct I/O, bs=4k, iodepth=64, numjobs=32 and ioengine=io_uring. Shown below is bw observed running fio test using different I/O policies:

             numa     round-robin queue-depth latency (MiB/s)  (MiB/s)     (MiB/s)     (MiB/s) -------  ----------- ----------- --------- randread:    1288     1202        1464        1642 randwrite:   1456     1493        1779        1944 randrw:      R:623    R:594       R:750       R:822 W:623    W:594       W:750       W:822

Quoted 11 lines
Quoted 7 lines

         numa         round-robin   queue-depth  adaptive -----------  -----------   -----------  --------- READ:   50.0 MiB/s   105 MiB/s     230 MiB/s    350 MiB/s WRITE:  65.9 MiB/s   125 MiB/s     385 MiB/s    446 MiB/s RW:     R:30.6 MiB/s R:56.5 MiB/s  R:122 MiB/s  R:175 MiB/s W:30.7 MiB/s W:56.5 MiB/s  W:122 MiB/s  W:175 MiB/s

And I'm assuming there are zero downsides for the normal case?

For the normal case where all paths are symmetric I saw queue-depth, latency and round-robin policies yielding nearly same bandwidth. However for numa policy, it depends on CPU/numa locality.

Quoted 63 lines
Quoted 63 lines

This pathcset includes totla 8 patches:
[PATCH 1/10] block: expose blk_stat_{enable,disable}_accounting()
   - Make blk_stat APIs available to block drivers.
   - Needed for per-path latency measurement.

[PATCH 2/10] block: record I/O request start time for passthru request
   - Record I/O start time for I/O passthru requests.
   - This is prep patch which allows measuring I/O completion latency for passthru requests.

[PATCH 3/10] block: support nesting for blk-mq flag QUEUE_FLAG_SAME_FORCE
   - Support nesting for QUEUE_FLAG_SAME_FORCE as multiple users could toggle QUEUE_FLAG_SAME_FORCE.

[PATCH 4/10] nvme-multipath: pass I/O type to nvme_find_path()
   - This is the prep patch which updates nvme_find_path() signature [PATCH 5/10] nvme-multipath: add latency I/O policy
   - Implement path scoring based on latency (EWMA).
   - Distribute I/O proportionally to per-path weights.

[PATCH 6/10] nvme: add generic debugfs support
   - Introduce generic debugfs support for NVMe module

[PATCH 7/10] nvme-multipath: add debugfs attribute latency_ewma_shift
   - Adds a debugfs attribute to control ewma shift

[PATCH 8/10] nvme-multipath: add debugfs attribute latency_batch_timeout
   - Adds a debugfs attribute to control latency batch window interval

[PATCH 9/10] nvme-multipath: add debugfs attribute latency_stat
   - Add “latency_stat” under per-path and head debugfs directories to expose latency policy state and statistics.

[PATCH 10/10] nvme-multipath: add documentation for latency I/O policy
   - Includes documentation for latency I/O multipath policy.

LSFMM discussion:
=================
During lsfmm 2026, it was decided to rename this I/O policy from "adaptive" to "latency". This series reflects that rename.

The discussion at lsfmm also focused extensively on the latency measurement model, including whether latency should be tracked per-CPU or per-NUMA, and whether separate I/O-size buckets should be maintained for different request sizes.

After detailed discussion and evaluation of throughput results, the consensus was to initially measure I/O completion latency on a per-CPU basis. The available performance data showed that the per-CPU implementation already provides sufficient averaging across CPUs while keeping the design relatively simple.

The use of additional I/O-size buckets did not demonstrate meaningful throughput improvement in the general case and would introduce extra complexity into the fast path and accounting logic. As a result, the consensus was to avoid I/O-size bucketing for now and keep the policy focused on per-CPU latency measurement.

If future real-world workloads demonstrate a clear benefit from I/O-size-aware latency accounting, the policy can be extended later to support it.

Quoted 5 lines
Quoted 1 line

As ususal, feedback and suggestions are most welcome!

Nilay, do we have evidence that round-robin/queue-depth are better for any workload? As a user, I would be very confused with the amount of path selectors I have available and which should I choose.

From my experiments, when the paths are symmetric, both round-robin and queue-depth (and for that matter latency) exhibit similar behavior, with the workload being distributed roughly equally across the active paths.

When the paths are asymmetric, I found queue-depth to perform better than round-robin. Queue-depth tries to steer I/O toward the less-loaded path based on the number of in-flight I/Os on each path, whereas round-robin continues to distribute I/O evenly across all active paths.

However, queue-depth still has a limitation in this scenario. It uses the number of in-flight I/Os as an indirect indication of path performance, it does not have a direct signal of the actual I/O completion latency. For example, if one path starts experiencing packet loss, I/O completion on that path can become significantly slower. Queue-depth can react to this as the path accumulates more outstanding I/Os, but it can still continue sending I/O to the degraded path as long as its queue depth remains comparable to the healthy path. In other words, it can reduce the amount of I/O sent to the degraded path, but it cannot directly account for how much slower that path has become.

The latency policy uses I/O completion latency as the signal instead. When one path becomes degraded, its observed latency increases and its path score/weight decreases. Consequently, the policy shifts more I/O towards the healthy path. This allows the healthy path to sustain a higher queue depth while the degraded path receives substantially less I/O, rather than trying to maintain a similar queue depth across both paths.

This is also reflected in the packet-loss experiment above. Round-robin continues to distribute I/O across both paths, while queue-depth does a better job by reacting to the increased queue occupancy of the degraded path. The latency policy goes one step further by directly using the increased completion latency as a signal and therefore steers more I/O toward the healthy path, resulting in higher throughput.

Quoted 7 lines

So based on the results I have so far, I would characterize the existing policies as follows: round-robin is useful when paths are symmetric and equal distribution is desired. The queue-depth is preferable when paths are asymmetric and queue occupancy provides a useful indication of path load and the latency policy is intended for cases where path performance can vary dynamically and we want the policy to adapt based on actual observed latency.

If you have policies A, B, and C and you say:
- In certain conditions policy C > A, B
- In some conditions C = B > A
- In all other cases C = B = A

This means that C should always be used, and A, B should never be used.

Hence I ask, should we really have this as an option? or should we deprecate round-robin/queue-depth and have only numa|latency?

I would like to avoid introducing this as a config knob if it is always behaves better.

What do others think?

#20 lore raw
Quoted 200 lines · Sagi Grimberg
Quoted 182 lines · Nilay Shroff
Quoted 29 lines · Sagi Grimberg
Quoted 20 lines · Nilay Shroff

Hi,

This series introduces a new latency I/O policy for NVMe native multipath. Existing policies such as numa, round-robin, and queue-depth are static and do not adapt to real-time transport performance. The numa selects the path closest to the NUMA node of the current CPU, optimizing memory and path locality, but ignores actual path performance. The round-robin distributes I/O evenly across all paths, providing fairness but not performance awareness. The queue-depth reacts to instantaneous queue occupancy, avoiding heavily loaded paths, but does not account for actual latency, throughput, or link speed.

The new latency policy addresses these gaps selecting paths dynamically based on measured I/O latency for both PCIe and fabrics. Latency is derived by passively sampling I/O completions. Each path is assigned a weight proportional to its latency score, and I/Os are then forwarded accordingly. As condition changes (e.g. latency spikes, bandwidth differences), path weights are updated, automatically steering traffic toward better-performing paths.

Quoted 5 lines

Early results show reduced tail latency under mixed workloads and improved throughput by exploiting higher-speed links more effectively. For example, with NVMf/TCP using two paths (one throttled with ~30 ms delay), fio results with random read/write/rw workloads (direct I/O) showed:

Quoted 5 lines

TBH, I do not know if this measurement represent any real-life scenario. I do think that occasional packet drops are a real-life scenario, and it would be a worthy use-case to optimize for. Can you perhaps measure how the path selectors compare in this case?

Okay so I have now measured another workload where I simulated packet loss/drops which are occasional and compared different path selectors. In this measurement I have a shared NVMe namespace configured which is reachable over two tcp paths. Now to simulate the occasional packet loss/drop I have configured one of the paths to experience occasional packet loss/drop as shown below over the period of 480 seconds:

0            120           240           360           480

|-------------|-------------|-------------|-------------|
|<--5% drop-->|<--no drop-->|<--3% drop-->|<--no drop-->|

As shown above for the first 120 seconds the path experiences 5% packet drop, for the next 120 seconds path sees no packet drop and again for subsequent 120 seconds path experiences 3% packet drop and for the rest of the duration (during last 120 seconds) there's no packet drop observed by the path. With this simulation, I ran fio workload for 480 seconds leveraging direct I/O, bs=4k, iodepth=64, numjobs=32 and ioengine=io_uring. Shown below is bw observed running fio test using different I/O policies:

             numa     round-robin queue-depth latency (MiB/s)  (MiB/s)     (MiB/s)     (MiB/s) -------  ----------- ----------- --------- randread:    1288     1202        1464        1642 randwrite:   1456     1493        1779        1944 randrw:      R:623    R:594       R:750       R:822 W:623    W:594       W:750       W:822

Quoted 11 lines
Quoted 7 lines

         numa         round-robin   queue-depth  adaptive -----------  -----------   -----------  --------- READ:   50.0 MiB/s   105 MiB/s     230 MiB/s    350 MiB/s WRITE:  65.9 MiB/s   125 MiB/s     385 MiB/s    446 MiB/s RW:     R:30.6 MiB/s R:56.5 MiB/s  R:122 MiB/s  R:175 MiB/s W:30.7 MiB/s W:56.5 MiB/s  W:122 MiB/s  W:175 MiB/s

And I'm assuming there are zero downsides for the normal case?

For the normal case where all paths are symmetric I saw queue-depth, latency and round-robin policies yielding nearly same bandwidth. However for numa policy, it depends on CPU/numa locality.

Quoted 63 lines
Quoted 63 lines

This pathcset includes totla 8 patches:
[PATCH 1/10] block: expose blk_stat_{enable,disable}_accounting()
   - Make blk_stat APIs available to block drivers.
   - Needed for per-path latency measurement.

[PATCH 2/10] block: record I/O request start time for passthru request
   - Record I/O start time for I/O passthru requests.
   - This is prep patch which allows measuring I/O completion latency for passthru requests.

[PATCH 3/10] block: support nesting for blk-mq flag QUEUE_FLAG_SAME_FORCE
   - Support nesting for QUEUE_FLAG_SAME_FORCE as multiple users could toggle QUEUE_FLAG_SAME_FORCE.

[PATCH 4/10] nvme-multipath: pass I/O type to nvme_find_path()
   - This is the prep patch which updates nvme_find_path() signature [PATCH 5/10] nvme-multipath: add latency I/O policy
   - Implement path scoring based on latency (EWMA).
   - Distribute I/O proportionally to per-path weights.

[PATCH 6/10] nvme: add generic debugfs support
   - Introduce generic debugfs support for NVMe module

[PATCH 7/10] nvme-multipath: add debugfs attribute latency_ewma_shift
   - Adds a debugfs attribute to control ewma shift

[PATCH 8/10] nvme-multipath: add debugfs attribute latency_batch_timeout
   - Adds a debugfs attribute to control latency batch window interval

[PATCH 9/10] nvme-multipath: add debugfs attribute latency_stat
   - Add “latency_stat” under per-path and head debugfs directories to expose latency policy state and statistics.

[PATCH 10/10] nvme-multipath: add documentation for latency I/O policy
   - Includes documentation for latency I/O multipath policy.

LSFMM discussion:
=================
During lsfmm 2026, it was decided to rename this I/O policy from "adaptive" to "latency". This series reflects that rename.

The discussion at lsfmm also focused extensively on the latency measurement model, including whether latency should be tracked per-CPU or per-NUMA, and whether separate I/O-size buckets should be maintained for different request sizes.

After detailed discussion and evaluation of throughput results, the consensus was to initially measure I/O completion latency on a per-CPU basis. The available performance data showed that the per-CPU implementation already provides sufficient averaging across CPUs while keeping the design relatively simple.

The use of additional I/O-size buckets did not demonstrate meaningful throughput improvement in the general case and would introduce extra complexity into the fast path and accounting logic. As a result, the consensus was to avoid I/O-size bucketing for now and keep the policy focused on per-CPU latency measurement.

If future real-world workloads demonstrate a clear benefit from I/O-size-aware latency accounting, the policy can be extended later to support it.

Quoted 5 lines
Quoted 1 line

As ususal, feedback and suggestions are most welcome!

Nilay, do we have evidence that round-robin/queue-depth are better for any workload? As a user, I would be very confused with the amount of path selectors I have available and which should I choose.

From my experiments, when the paths are symmetric, both round-robin and queue-depth (and for that matter latency) exhibit similar behavior, with the workload being distributed roughly equally across the active paths.

When the paths are asymmetric, I found queue-depth to perform better than round-robin. Queue-depth tries to steer I/O toward the less-loaded path based on the number of in-flight I/Os on each path, whereas round-robin continues to distribute I/O evenly across all active paths.

However, queue-depth still has a limitation in this scenario. It uses the number of in-flight I/Os as an indirect indication of path performance, it does not have a direct signal of the actual I/O completion latency. For example, if one path starts experiencing packet loss, I/O completion on that path can become significantly slower. Queue-depth can react to this as the path accumulates more outstanding I/Os, but it can still continue sending I/O to the degraded path as long as its queue depth remains comparable to the healthy path. In other words, it can reduce the amount of I/O sent to the degraded path, but it cannot directly account for how much slower that path has become.

The latency policy uses I/O completion latency as the signal instead. When one path becomes degraded, its observed latency increases and its path score/weight decreases. Consequently, the policy shifts more I/O towards the healthy path. This allows the healthy path to sustain a higher queue depth while the degraded path receives substantially less I/O, rather than trying to maintain a similar queue depth across both paths.

This is also reflected in the packet-loss experiment above. Round-robin continues to distribute I/O across both paths, while queue-depth does a better job by reacting to the increased queue occupancy of the degraded path. The latency policy goes one step further by directly using the increased completion latency as a signal and therefore steers more I/O toward the healthy path, resulting in higher throughput.

Quoted 7 lines

So based on the results I have so far, I would characterize the existing policies as follows: round-robin is useful when paths are symmetric and equal distribution is desired. The queue-depth is preferable when paths are asymmetric and queue occupancy provides a useful indication of path load and the latency policy is intended for cases where path performance can vary dynamically and we want the policy to adapt based on actual observed latency.

If you have policies A, B, and C and you say:
- In certain conditions policy C > A, B
- In some conditions C = B > A
- In all other cases C = B = A

This means that C should always be used, and A, B should never be used.

Quoted 7 lines

Hence I ask, should we really have this as an option? or should we deprecate round-robin/queue-depth and have only numa|latency?

I would like to avoid introducing this as a config knob if it is always behaves better.

What do others think?

Good point, however my view is that we may not want to immediately deprecate or remove queue-depth/round-robin. Based on my testing so far, across a variety of workloads, including intermittent packet loss, the latency policy has performed better than queue-depth and round-robin. However, I would like to see it evaluated against a broader set of workloads and scenarios, including cases that I may not have considered/known.

Once we have broader evidence and establish that latency consistently performs at least as well as queue-depth/round-robin without any significant downside, then I think it would be reasonable to consider deprecating those policies and keeping only numa|latency.

Until then, I would prefer to keep queue-depth and round-robin as baseline policies against which we can compare the latency policy.

But let's wait and see what others think.

Thanks,
--Nilay

#21 lore raw
Quoted 3 lines · Nilay Shroff
Quoted 2 lines · Sagi Grimberg

[ .. ]>> I would like to avoid introducing this as a config knob if it is

Quoted 10 lines
Quoted 3 lines

always behaves better.

What do others think?

Good point, however my view is that we may not want to immediately deprecate or remove queue-depth/round-robin. Based on my testing so far, across a variety of workloads, including intermittent packet
loss, the latency policy has performed better than queue-depth and
round-robin. However, I would like to see it evaluated against a
broader set of workloads and scenarios, including cases that I > may not have considered/known.

Quoted 7 lines

Once we have broader evidence and establish that latency consistently performs at least as well as queue-depth/round-robin without any significant downside, then I think it would be reasonable to consider
deprecating those policies and keeping only numa|latency.>
Until then, I would prefer to keep queue-depth and round-robin as baseline policies against which we can compare the latency policy.

But let's wait and see what others think.

I think that we can deprecate round-robin now that we have queue-depth and latency.
But I also think that we need both, queue-depth _and_ latency as both latch on different properties, and there are use-cases where one might have workloads which prefer one or the other.

Having a config option to select the default (and possibly disable some I/O schedulers) would be completely fine with me, too.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
#22 lore raw

I think Nilay has done some great work with the latency scheduler, and we have it in our queue for evaluation. But we have not done so yet, so I would appreciate if queue-depth would continue to be available until the latency scheduler has been more fully established as the prime choice.

As to round-robin, I agree it would be great to see it disappear from all of our customers installations, but I have found it useful in evaluating multipath failover behavior because it is easier to reason about and construct the test cases. So, I would appreciate if it would remain for debugging only. I would certainly be open to a mechanism that made it a little bit hard to set round-robin as the queue policy, though.

Sincerely,
Randy Jennings

Quoted 44 lines · Hannes Reinecke
Quoted 3 lines · Nilay Shroff
Quoted 2 lines · Sagi Grimberg

[ .. ]>> I would like to avoid introducing this as a config knob if it is

Quoted 10 lines
Quoted 3 lines

always behaves better.

What do others think?

Good point, however my view is that we may not want to immediately deprecate or remove queue-depth/round-robin. Based on my testing so far, across a variety of workloads, including intermittent packet loss, the latency policy has performed better than queue-depth and round-robin. However, I would like to see it evaluated against a broader set of workloads and scenarios, including cases that I > may

not have considered/known.

Quoted 10 lines

Once we have broader evidence and establish that latency consistently performs at least as well as queue-depth/round-robin without any significant downside, then I think it would be reasonable to consider deprecating those policies and keeping only numa|latency.> Until then, I would prefer to keep queue-depth and round-robin as baseline policies against which we can compare the latency policy.

But let's wait and see what others think.

I think that we can deprecate round-robin now that we have queue-depth and latency.
But I also think that we need both, queue-depth _and_ latency as both latch on different properties, and there are use-cases where one might have workloads which prefer one or the other.

Having a config option to select the default (and possibly disable some I/O schedulers) would be completely fine with me, too.

Cheers,

Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
[email protected] +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich

maintainer
#23 lore raw
Quoted 7 lines

I think Nilay has done some great work with the latency scheduler, and we have it in our queue for evaluation. But we have not done so yet, so I would appreciate if queue-depth would continue to be available until the latency scheduler has been more fully established as the prime choice.

Quoted 7 lines

As to round-robin, I agree it would be great to see it disappear from all of our customers installations, but I have found it useful in evaluating multipath failover behavior because it is easier to reason about and construct the test cases. So, I would appreciate if it would remain for debugging only. I would certainly be open to a mechanism that made it a little bit hard to set round-robin as the queue policy, though.

Randy, would it be possible for you folks to evaluate it as well? I will ask some folks on our side as well. I'm concerned that Nilay is the only one who has evaluated it in his lab.

Overall, I am perfectly fine with the approach this path selector is taking, but I worry that we are growing path-selectors that are slightly different in subtle ways... TBH, I still don't understand what use-cases this path selector is more appropriate than queue-depth, or vice-versa.

#24 lore raw
Quoted 3 lines

Randy, would it be possible for you folks to evaluate it as well? I will ask some folks on our side as well. I'm concerned that Nilay is the only one who has evaluated it in his lab.

Yes, absolutely, but there is going to be a time delay due to wrapping up some current priorities. I'd say we'd get to it in more than 2 months but less than 6. That is the best promise I can give right now. (Yes, it is behind some work for stretched subsystems and CCR/CQT, which are higher priorities for me.)

Quoted 4 lines

Overall, I am perfectly fine with the approach this path selector is taking, but I worry
that we are growing path-selectors that are slightly different in subtle ways...

Agreed.

Quoted 2 lines

TBH, I still don't understand what use-cases this path selector is more appropriate than queue-depth, or vice-versa.

I had this question, too, and we talked about it in Zagreb. My understanding from that discussion is that latency can handle full queues better than queue-depth. When you have shallow queues, you still get information to make a path decision on.

Sincerely,
Randy Jennings