nvme-multipath: introduce latency I/O policy
Subsystems (5)
v8 cover nvme-multipath: introduce latency I/O policy
Hi,
This series introduces a new latency I/O policy for NVMe native multipath. Existing policies such as numa, round-robin, and queue-depth are static and do not adapt to real-time transport performance. The numa selects the path closest to the NUMA node of the current CPU, optimizing memory and path locality, but ignores actual path performance. The round-robin distributes I/O evenly across all paths, providing fairness but not performance awareness. The queue-depth reacts to instantaneous queue occupancy, avoiding heavily loaded paths, but does not account for actual latency, throughput, or link speed.
The new latency policy addresses these gaps selecting paths dynamically based on measured I/O latency for both PCIe and fabrics. Latency is derived by passively sampling I/O completions. Each path is assigned a weight proportional to its latency score, and I/Os are then forwarded accordingly. As condition changes (e.g. latency spikes, bandwidth differences), path weights are updated, automatically steering traffic toward better-performing paths.
Early results show reduced tail latency under mixed workloads and improved throughput by exploiting higher-speed links more effectively. For example, with NVMf/TCP using two paths (one throttled with ~30 ms delay), fio results with random read/write/rw workloads (direct I/O) showed:
numa round-robin queue-depth adaptive
----------- ----------- ----------- ---------
READ: 50.0 MiB/s 105 MiB/s 230 MiB/s 350 MiB/s
WRITE: 65.9 MiB/s 125 MiB/s 385 MiB/s 446 MiB/s
RW: R:30.6 MiB/s R:56.5 MiB/s R:122 MiB/s R:175 MiB/s
W:30.7 MiB/s W:56.5 MiB/s W:122 MiB/s W:175 MiB/sThis pathcset includes totla 8 patches:
[PATCH 1/10] block: expose blk_stat_{enable,disable}_accounting()
- Make blk_stat APIs available to block drivers.
- Needed for per-path latency measurement.
[PATCH 2/10] block: record I/O request start time for passthru request
- Record I/O start time for I/O passthru requests.
- This is prep patch which allows measuring I/O completion latency for passthru requests.
[PATCH 3/10] block: support nesting for blk-mq flag QUEUE_FLAG_SAME_FORCE
- Support nesting for QUEUE_FLAG_SAME_FORCE as multiple users could toggle QUEUE_FLAG_SAME_FORCE.
[PATCH 4/10] nvme-multipath: pass I/O type to nvme_find_path()
- This is the prep patch which updates nvme_find_path() signature
[PATCH 5/10] nvme-multipath: add latency I/O policy
- Implement path scoring based on latency (EWMA).
- Distribute I/O proportionally to per-path weights.
[PATCH 6/10] nvme: add generic debugfs support
- Introduce generic debugfs support for NVMe module
[PATCH 7/10] nvme-multipath: add debugfs attribute latency_ewma_shift
- Adds a debugfs attribute to control ewma shift
[PATCH 8/10] nvme-multipath: add debugfs attribute latency_batch_timeout
- Adds a debugfs attribute to control latency batch window interval
[PATCH 9/10] nvme-multipath: add debugfs attribute latency_stat
- Add “latency_stat” under per-path and head debugfs directories to expose latency policy state and statistics.
[PATCH 10/10] nvme-multipath: add documentation for latency I/O policy
- Includes documentation for latency I/O multipath policy.
LSFMM discussion:
=================
During lsfmm 2026, it was decided to rename this I/O policy from "adaptive" to "latency". This series reflects that rename.
The discussion at lsfmm also focused extensively on the latency measurement model, including whether latency should be tracked per-CPU or per-NUMA, and whether separate I/O-size buckets should be maintained for different request sizes.
After detailed discussion and evaluation of throughput results, the consensus was to initially measure I/O completion latency on a per-CPU basis. The available performance data showed that the per-CPU implementation already provides sufficient averaging across CPUs while keeping the design relatively simple.
The use of additional I/O-size buckets did not demonstrate meaningful throughput improvement in the general case and would introduce extra complexity into the fast path and accounting logic. As a result, the consensus was to avoid I/O-size bucketing for now and keep the policy focused on per-CPU latency measurement.
If future real-world workloads demonstrate a clear benefit from I/O-size-aware latency accounting, the policy can be extended later to support it.
As ususal, feedback and suggestions are most welcome!
Thanks!
Changes from v7:
- Rebased on nvme-7.3 (John Garry)
- Added Clang lock context annotations where appropriate (John Garry)
- Added support for nesting QUEUE_FLAG_SAME_FORCE in a new patch
3/10 (John Garry)
- Free head->latency_path in nvme_mpath_put_disk() instead of nvme_free_ns_head() (John Garry)
- Replaced this_cpu_ptr() with per_cpu_ptr() (John Garry)
- Few miscellaneous and cosmetic changes such as adding comments updating commit description etc.
Link to v7: https://lore.kernel.org/all/[email protected]/
Changes from v6:
- Add patch 2/9, which records I/O start time for the passthru requests (Guixin Liu)
- Correctly record the passthru I/O data direction (Guixin Liu)
- Clear NVME_NS_PATH_STAT before cancelling the latency weight work. (Guixin Liu)
Link to v6: https://lore.kernel.org/all/[email protected]/
Changes from v5:
- Rename the policy from "adaptive" to "latency". The entire series updates policy names, function names, and variable names accordingly, without introducing any functional changes. (lsfmm discussion)
- The second patch is now splitted into two patches:
Patch #2: prep patch where we pass op_type to nvme_find_path() Patch #3: core patch which introduces latency I/O policy
(Sagi)
- Rename ewma_update() to calc_ewma_update() (Sagi)
Link to v5: https://lore.kernel.org/all/[email protected]/
Changes from v4:
- Added patch #7 which includes the documentation for adaptive I/O policy. (Guixin Liu)
Link to v4: https://lore.kernel.org/all/[email protected]/
Changes from v3:
- Update the adaptive APIs name (which actually enable/disable adaptive policy) to reflect the actual work it does. Also removed the misleading use of "current_path" from the adaptive policy code (Hannes Reinecke)
- Move adaptive_ewma_shift and adaptive_weight_timeout attributes from sysfs to debugfs (Hannes Reinecke)
Link to v3: https://lore.kernel.org/all/[email protected]/
Changes from v2:
- Addede a new patch to allow user to configure EWMA shift through sysfs (Hannes Reinecke)
- Added a new patch to allow user to configure path weight calculation timeout (Hannes Reinecke)
- Distinguish between read/write and other commands (e.g. admin comamnd) and calculate path weight for other commands which is separate from read/write weight. (Hannes Reinecke)
- Normalize per-path weight in the range from 0-128 instead
of 0-100 (Hannes Reinecke)
- Restructure and optimize adaptive I/O forwarding code to use
one loop instead of two (Hannes Reinecke)
Link to v2: https://lore.kernel.org/all/[email protected]/
Changes from v1:
- Ensure that the completion of I/O occurs on the same CPU as the submitting I/O CPU (Hannes Reinecke)
- Remove adapter link speed from the path weight calculation
(Hannes Reinecke)
- Add adaptive I/O stat under debugfs instead of current sysfs (Hannes Reinecke)
- Move path weight calculation to a workqueue from IO completion code path
Link to v1: https://lore.kernel.org/all/[email protected]/
Nilay Shroff (10):
block: expose blk_stat_{enable,disable}_accounting() to drivers
block: record I/O request start time for passthru request
block: support nesting for blk-mq flag QUEUE_FLAG_SAME_FORCE
nvme-multipath: pass I/O type to nvme_find_path()
nvme-multipath: add support for latency I/O policy
nvme: add generic debugfs support
nvme-multipath: add debugfs attribute latency_ewma_shift
nvme-multipath: add debugfs attribute latency_batch_timeout
nvme-multipath: add debugfs attribute latency_stat
nvme-multipath: add documentation for latency I/O policy
Documentation/admin-guide/nvme-multipath.rst | 19 + block/blk-mq.c | 53 +- block/blk-stat.h | 4 - block/blk-sysfs.c | 6 +- drivers/nvme/host/Makefile | 2 +- drivers/nvme/host/core.c | 15 +- drivers/nvme/host/debugfs.c | 347 ++++++++++++++ drivers/nvme/host/ioctl.c | 44 +- drivers/nvme/host/multipath.c | 478 ++++++++++++++++++- drivers/nvme/host/nvme.h | 110 ++++- drivers/nvme/host/pr.c | 6 +- drivers/nvme/host/sysfs.c | 2 +- drivers/ufs/host/ufs-mediatek.c | 2 +- include/linux/blk-mq.h | 6 + include/linux/blkdev.h | 3 + 15 files changed, 1057 insertions(+), 40 deletions(-) create mode 100644 drivers/nvme/host/debugfs.c
Signature · 1 line
2.53.0
PATCH v8 1/10 block: expose blk_stat_{enable,disable}_accounting() to drivers
The functions blk_stat_enable_accounting() and blk_stat_disable_accounting() are currently exported, but their prototypes are only defined in a private header. Move these prototypes into a common header so that block drivers can directly use these APIs.
--- block/blk-stat.h | 4 ---- include/linux/blk-mq.h | 4 ++++ 2 files changed, 4 insertions(+), 4 deletions(-)
block/blk-stat.h +0 -4
@@ -70,10 +70,6 @@ void blk_free_queue_stats(struct blk_queue_stats *);void blk_stat_add (struct request *rq,u64 now); -/* record time/size info in request but not add a callback */ -void blk_stat_enable_accounting (struct request_queue *q); -void blk_stat_disable_accounting (struct request_queue *q); -/** * blk_stat_alloc_callback() - Allocate a block statistics callback. * @timer_fn: Timer callback function.
include/linux/blk-mq.h +4 -0
@@ -753,6 +753,10 @@ int blk_rq_poll(struct request *rq, struct io_comp_batch *iob,bool blk_mq_queue_inflight (struct request_queue *q); +/* record time/size info in request but not add a callback */ +void blk_stat_enable_accounting (struct request_queue *q); +void blk_stat_disable_accounting (struct request_queue *q); +enum {/* return when out of requests */ BLK_MQ_REQ_NOWAIT = (__force blk_mq_req_flags_t)(1 <<0 ),
Signature · 1 line
2.53.0
PATCH v8 2/10 block: record I/O request start time for passthru request
In reply to Nilay ShroffWhile starting an I/O request, blk_mq_start_request() records the request start timestamp only for non-passthrough requests when QUEUE_FLAG_STATS is enabled.
However, the latency based multipath policy uses request completion latency to evaluate path performance, and I/O is issued as passthrough requests. Since passthru requests never initialize rq->io_start_time_ns, their latency cannot be computed.
Record io_start_time_ns for all requests whenever QUEUE_FLAG_STATS is enabled.
--- block/blk-mq.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
block/blk-mq.c +7 -5
@@ -1340,12 +1340,14 @@ void blk_mq_start_request(struct request *rq)trace_block_rq_issue (rq); -if (test_bit (QUEUE_FLAG_STATS , &q->queue_flags) && - !blk_rq_is_passthrough (rq)) { +if (test_bit (QUEUE_FLAG_STATS , &q->queue_flags)) { rq->io_start_time_ns =blk_time_get_ns (); - rq->stats_sectors =blk_rq_sectors (rq); - rq->rq_flags |=RQF_STATS ; -rq_qos_issue (q, rq); + +if (!blk_rq_is_passthrough (rq)) { + rq->stats_sectors =blk_rq_sectors (rq); + rq->rq_flags |=RQF_STATS ; +rq_qos_issue (q, rq); + } }WARN_ON_ONCE (blk_mq_rq_state (rq) !=MQ_RQ_IDLE );
Signature · 1 line
2.53.0
PATCH v8 3/10 block: support nesting for blk-mq flag QUEUE_FLAG_SAME_FORCE
In reply to Nilay ShroffQUEUE_FLAG_SAME_FORCE is currently used when setting rq_affinity through sysfs as well as by UFS mediatek driver while configuring scsi parameters. A subsequent patch adding a latency-based I/O policy for NVMe multipath will also use this flag.
With multiple users able to set and clear QUEUE_FLAG_SAME_FORCE, the flag needs to support nesting so that one user clearing the flag does not inadvertently disable it for another user.
Add a nesting counter, q->same_force_depth, for QUEUE_FLAG_SAME_FORCE. The flag is set when the first user acquires it and the nesting counter is incremented for each subsequent user. Similarly, each user releases its reference by decrementing the counter. The flag is cleared only when the last user releases it and the counter reaches zero.
Preserve the existing sysfs rq_affinity semantics with a new q->same_force_sysfs flag. When userspace enables QUEUE_FLAG_SAME_FORCE by writing 2 to rq_affinity, mark q->same_force_sysfs as set and increment q->same_force_depth by one. Subsequent writes of 2 to rq_affinity while q->same_force_sysfs is already set are ignored, so repeated writes of 2 from userspace do not increase q->same_force_depth. Similarly, writing 0 or 1 decrements the q->same_force_depth and if nesting counter reached to 0 then clears the QUEUE_FLAG_SAME_FORCE. This ensures that multiple writes of 2 to rq_affinity do not require multiple writes of 0 or 1.
This change ensures that sysfs interface retains its existing set/clear semantics while also allowing other kernel users to hold or release QUEUE_FLAG_SAME_FORCE.
Added two new APIs blk_mq_same_force_set() and blk_mq_same_force_clear() to set and clear QUEUE_FLAG_SAME_FORCE respectively. Also, updated existing call paths using these new APIs which toggles QUEUE_FLAG_SAME_FORCE.
--- block/blk-mq.c | 41 +++++++++++++++++++++++++++++++++ block/blk-sysfs.c | 6 ++--- drivers/ufs/host/ufs-mediatek.c | 2 +- include/linux/blk-mq.h | 2 ++ include/linux/blkdev.h | 3 +++ 5 files changed, 50 insertions(+), 4 deletions(-)
block/blk-mq.c +41 -0
@@ -360,6 +360,47 @@ void blk_mq_unquiesce_tagset(struct blk_mq_tag_set *set) }EXPORT_SYMBOL_GPL (blk_mq_unquiesce_tagset); +void blk_mq_same_force_set (struct request_queue *q,bool from_sysfs) +{ +unsigned long flags =0 ; + +spin_lock_irqsave (&q->queue_lock, flags); + +if (from_sysfs) { +if (q->same_force_sysfs) + goto unlock; + q->same_force_sysfs = true; + } + +if (!q->same_force_depth++) +blk_queue_flag_set (QUEUE_FLAG_SAME_FORCE , q); +unlock: +spin_unlock_irqrestore (&q->queue_lock, flags); +} +EXPORT_SYMBOL_GPL (blk_mq_same_force_set); + +void blk_mq_same_force_clear (struct request_queue *q,bool from_sysfs) +{ +unsigned long flags =0 ; + +spin_lock_irqsave (&q->queue_lock, flags); + +if (from_sysfs) { +if (!q->same_force_sysfs) + goto unlock; + q->same_force_sysfs = false; + } + +if (WARN_ON_ONCE (q->same_force_depth <=0 )) + goto unlock; + +if (!--q->same_force_depth) +blk_queue_flag_clear (QUEUE_FLAG_SAME_FORCE , q); +unlock: +spin_unlock_irqrestore (&q->queue_lock, flags); +} +EXPORT_SYMBOL_GPL (blk_mq_same_force_clear); +void blk_mq_wake_waiters (struct request_queue *q) {struct blk_mq_hw_ctx *hctx;
block/blk-sysfs.c +3 -3
@@ -497,13 +497,13 @@ queue_rq_affinity_store(struct gendisk *disk, const char *page, size_t count)*/ if (val ==2 ) {blk_queue_flag_set (QUEUE_FLAG_SAME_COMP , q); -blk_queue_flag_set (QUEUE_FLAG_SAME_FORCE , q); +blk_mq_same_force_set (q, true); }else if (val ==1 ) {blk_queue_flag_set (QUEUE_FLAG_SAME_COMP , q); -blk_queue_flag_clear (QUEUE_FLAG_SAME_FORCE , q); +blk_mq_same_force_clear (q, true); }else if (val ==0 ) {blk_queue_flag_clear (QUEUE_FLAG_SAME_COMP , q); -blk_queue_flag_clear (QUEUE_FLAG_SAME_FORCE , q); +blk_mq_same_force_clear (q, true); }#endif return ret;
drivers/ufs/host/ufs-mediatek.c +1 -1
@@ -2311,7 +2311,7 @@ static void ufs_mtk_config_scsi_dev(struct scsi_device *sdev)dev_dbg (hba->dev,"lu %llu scsi device configured" , sdev->lun);if (sdev->lun ==2 ) -blk_queue_flag_set (QUEUE_FLAG_SAME_FORCE , sdev->request_queue); +blk_mq_same_force_set (sdev->request_queue, false); }/*
include/linux/blk-mq.h +2 -0
@@ -943,6 +943,8 @@ void blk_mq_wait_quiesce_done(struct blk_mq_tag_set *set);void blk_mq_quiesce_tagset (struct blk_mq_tag_set *set);void blk_mq_unquiesce_tagset (struct blk_mq_tag_set *set);void blk_mq_unquiesce_queue (struct request_queue *q); +void blk_mq_same_force_set (struct request_queue *q,bool from_sysfs); +void blk_mq_same_force_clear (struct request_queue *q,bool from_sysfs);void blk_mq_delay_run_hw_queue (struct blk_mq_hw_ctx *hctx,unsigned long msecs);void blk_mq_run_hw_queue (struct blk_mq_hw_ctx *hctx,bool async);void blk_mq_run_hw_queues (struct request_queue *q,bool async);
include/linux/blkdev.h +3 -0
@@ -529,6 +529,9 @@ struct request_queue {int quiesce_depth; +int same_force_depth; +bool same_force_sysfs; +struct gendisk *disk;/*
Signature · 1 line
2.53.0
Re: [PATCH v8 03/10] block: support nesting for blk-mq flag QUEUE_FLAG_SAME_FORCE
Quoted 26 lines · Nilay Shroff
QUEUE_FLAG_SAME_FORCEis currently used when settingrq_affinitythrough sysfs as well as by UFS mediatek driver while configuring scsi parameters. A subsequent patch adding a latency-based I/O policy for NVMe multipath will also use this flag.With multiple users able to set and clear
QUEUE_FLAG_SAME_FORCE, the flag needs to support nesting so that one user clearing the flag does not inadvertently disable it for another user.Add a nesting counter,
q->same_force_depth, forQUEUE_FLAG_SAME_FORCE. The flag is set when the first user acquires it and the nesting counter is incremented for each subsequent user. Similarly, each user releases its reference by decrementing the counter. The flag is cleared only when the last user releases it and the counter reaches zero.Preserve the existing sysfs
rq_affinitysemantics with a newq->same_force_sysfsflag. When userspace enablesQUEUE_FLAG_SAME_FORCEby writing 2 torq_affinity, markq->same_force_sysfsas set and incrementq->same_force_depthby one. Subsequent writes of 2 torq_affinitywhileq->same_force_sysfsis already set are ignored, so repeated writes of 2 from userspace do not increaseq->same_force_depth. Similarly, writing 0 or 1 decrements theq->same_force_depthand if nesting counter reached to 0 then clears theQUEUE_FLAG_SAME_FORCE. This ensures that multiple writes of 2 torq_affinitydo not require multiple writes of 0 or 1.
Quoted 9 lines
This change ensures that sysfs interface retains its existing set/clear semantics while also allowing other kernel users to hold or release
QUEUE_FLAG_SAME_FORCE.Added two new APIs
blk_mq_same_force_set()andblk_mq_same_force_clear()to set and clearQUEUE_FLAG_SAME_FORCErespectively. Also, updated existing call paths using these new APIs which togglesQUEUE_FLAG_SAME_FORCE.
Yikes. Is this really necessary?
The flag can be set by different subsystems/drivers, sure, but from what I've seen each user is mutually exclusive.
I'd rather disallow modifying this flag if the driver sets it, to avoid upper layers or sysfs modifying it.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Re: [PATCH v8 03/10] block: support nesting for blk-mq flag QUEUE_FLAG_SAME_FORCE
Quoted 36 lines · Hannes Reinecke
Quoted 26 lines · Nilay Shroff
QUEUE_FLAG_SAME_FORCEis currently used when settingrq_affinitythrough sysfs as well as by UFS mediatek driver while configuring scsi parameters. A subsequent patch adding a latency-based I/O policy for NVMe multipath will also use this flag.With multiple users able to set and clear
QUEUE_FLAG_SAME_FORCE, the flag needs to support nesting so that one user clearing the flag does not inadvertently disable it for another user.Add a nesting counter,
q->same_force_depth, forQUEUE_FLAG_SAME_FORCE. The flag is set when the first user acquires it and the nesting counter is incremented for each subsequent user. Similarly, each user releases its reference by decrementing the counter. The flag is cleared only when the last user releases it and the counter reaches zero.Preserve the existing sysfs
rq_affinitysemantics with a newq->same_force_sysfsflag. When userspace enablesQUEUE_FLAG_SAME_FORCEby writing 2 torq_affinity, markq->same_force_sysfsas set and incrementq->same_force_depthby one. Subsequent writes of 2 torq_affinitywhileq->same_force_sysfsis already set are ignored, so repeated writes of 2 from userspace do not increaseq->same_force_depth. Similarly, writing 0 or 1 decrements theq->same_force_depthand if nesting counter reached to 0 then clears theQUEUE_FLAG_SAME_FORCE. This ensures that multiple writes of 2 torq_affinitydo not require multiple writes of 0 or 1.Quoted 9 lines
This change ensures that sysfs interface retains its existing set/clear semantics while also allowing other kernel users to hold or release
QUEUE_FLAG_SAME_FORCE.Added two new APIs
blk_mq_same_force_set()andblk_mq_same_force_clear()to set and clearQUEUE_FLAG_SAME_FORCErespectively. Also, updated existing call paths using these new APIs which togglesQUEUE_FLAG_SAME_FORCE.
Quoted 6 lines
Yikes. Is this really necessary?
The flag can be set by different subsystems/drivers, sure, but from what I've seen each user is mutually exclusive.
I'd rather disallow modifying this flag if the driver sets it, to avoid upper layers or sysfs modifying it.
If we want to disallow userspace from modifying QUEUE_FLAG_SAME_FORCE while a driver owns it, then a userspace write to rq_affinity would have to either fail (e.g. with -EINVAL) or be silently ignored. In either case, that changes the existing userspace/sysfs semantics and may break userspace that expects the write to succeed.
Or are you suggesting that a write to rq_affinity to enable QUEUE_FLAG_SAME_FORCE should succeed when the driver has already set it? If so, what should happen when the driver subsequently releases QUEUE_FLAG_SAME_FORCE? In that case, we would clear QUEUE_FLAG_SAME_FORCE when the driver releases it, even though userspace may have previously written 2 to rq_affinity. That would effectively reset the state requested by userspace without informing userspace.
This seems to require keeping track of the userspace request separately from the driver's ownership of the flag, which is one of the reasons I introduced same_force_sysfs along with the nesting counter.
Thanks,
--Nilay
Re: [PATCH v8 03/10] block: support nesting for blk-mq flag QUEUE_FLAG_SAME_FORCE
Quoted 44 lines · Nilay Shroff
Quoted 36 lines · Hannes Reinecke
Quoted 26 lines · Nilay Shroff
QUEUE_FLAG_SAME_FORCEis currently used when settingrq_affinitythrough sysfs as well as by UFS mediatek driver while configuring scsi parameters. A subsequent patch adding a latency-based I/O policy for NVMe multipath will also use this flag.With multiple users able to set and clear
QUEUE_FLAG_SAME_FORCE, the flag needs to support nesting so that one user clearing the flag does not inadvertently disable it for another user.Add a nesting counter,
q->same_force_depth, forQUEUE_FLAG_SAME_FORCE. The flag is set when the first user acquires it and the nesting counter is incremented for each subsequent user. Similarly, each user releases its reference by decrementing the counter. The flag is cleared only when the last user releases it and the counter reaches zero.Preserve the existing sysfs
rq_affinitysemantics with a newq->same_force_sysfsflag. When userspace enablesQUEUE_FLAG_SAME_FORCEby writing 2 torq_affinity, markq->same_force_sysfsas set and incrementq->same_force_depthby one. Subsequent writes of 2 torq_affinitywhileq->same_force_sysfsis already set are ignored, so repeated writes of 2 from userspace do not increaseq->same_force_depth. Similarly, writing 0 or 1 decrements theq->same_force_depthand if nesting counter reached to 0 then clears theQUEUE_FLAG_SAME_FORCE. This ensures that multiple writes of 2 torq_affinitydo not require multiple writes of 0 or 1.Quoted 9 lines
This change ensures that sysfs interface retains its existing set/clear semantics while also allowing other kernel users to hold or release
QUEUE_FLAG_SAME_FORCE.Added two new APIs
blk_mq_same_force_set()andblk_mq_same_force_clear()to set and clearQUEUE_FLAG_SAME_FORCErespectively. Also, updated existing call paths using these new APIs which togglesQUEUE_FLAG_SAME_FORCE.Quoted 6 lines
Yikes. Is this really necessary?
The flag can be set by different subsystems/drivers, sure, but from what I've seen each user is mutually exclusive.
I'd rather disallow modifying this flag if the driver sets it, to avoid upper layers or sysfs modifying it.
Quoted 6 lines
If we want to disallow userspace from modifying
QUEUE_FLAG_SAME_FORCEwhile a driver owns it, then a userspace write torq_affinitywould have to either fail (e.g. with-EINVAL) or be silently ignored. In either case, that changes the existing userspace/sysfs semantics and may break userspace that expects the write to succeed.
But _allowing_ userspace to modify it for drivers which requires it to be set will break the drivers.
Similar for the upcoming latency-based I/O scheduler; disabling QUEUE_FLAG_SAME_FORCE there will cause misaccounting and the scheduler will not work properly.
So I would argue to have a flag for making QUEUE_FLAG_SAME_FORCE driver owned, and reject any attempts to change it via sysfs with EINVAL.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Re: [PATCH v8 03/10] block: support nesting for blk-mq flag QUEUE_FLAG_SAME_FORCE
Quoted 51 lines · Hannes Reinecke
Quoted 44 lines · Nilay Shroff
Quoted 36 lines · Hannes Reinecke
Quoted 26 lines · Nilay Shroff
QUEUE_FLAG_SAME_FORCEis currently used when settingrq_affinitythrough sysfs as well as by UFS mediatek driver while configuring scsi parameters. A subsequent patch adding a latency-based I/O policy for NVMe multipath will also use this flag.With multiple users able to set and clear
QUEUE_FLAG_SAME_FORCE, the flag needs to support nesting so that one user clearing the flag does not inadvertently disable it for another user.Add a nesting counter,
q->same_force_depth, forQUEUE_FLAG_SAME_FORCE. The flag is set when the first user acquires it and the nesting counter is incremented for each subsequent user. Similarly, each user releases its reference by decrementing the counter. The flag is cleared only when the last user releases it and the counter reaches zero.Preserve the existing sysfs
rq_affinitysemantics with a newq->same_force_sysfsflag. When userspace enablesQUEUE_FLAG_SAME_FORCEby writing 2 torq_affinity, markq->same_force_sysfsas set and incrementq->same_force_depthby one. Subsequent writes of 2 torq_affinitywhileq->same_force_sysfsis already set are ignored, so repeated writes of 2 from userspace do not increaseq->same_force_depth. Similarly, writing 0 or 1 decrements theq->same_force_depthand if nesting counter reached to 0 then clears theQUEUE_FLAG_SAME_FORCE. This ensures that multiple writes of 2 torq_affinitydo not require multiple writes of 0 or 1.Quoted 9 lines
This change ensures that sysfs interface retains its existing set/clear semantics while also allowing other kernel users to hold or release
QUEUE_FLAG_SAME_FORCE.Added two new APIs
blk_mq_same_force_set()andblk_mq_same_force_clear()to set and clearQUEUE_FLAG_SAME_FORCErespectively. Also, updated existing call paths using these new APIs which togglesQUEUE_FLAG_SAME_FORCE.Quoted 6 lines
Yikes. Is this really necessary?
The flag can be set by different subsystems/drivers, sure, but from what I've seen each user is mutually exclusive.
I'd rather disallow modifying this flag if the driver sets it, to avoid upper layers or sysfs modifying it.Quoted 6 lines
If we want to disallow userspace from modifying
QUEUE_FLAG_SAME_FORCEwhile a driver owns it, then a userspace write torq_affinitywould have to either fail (e.g. with-EINVAL) or be silently ignored. In either case, that changes the existing userspace/sysfs semantics and may break userspace that expects the write to succeed.
Quoted 8 lines
But _allowing_ userspace to modify it for drivers which requires it to be set will break the drivers.
Similar for the upcoming latency-based I/O scheduler; disablingQUEUE_FLAG_SAME_FORCEthere will cause misaccounting and the scheduler will not work properly.So I would argue to have a flag for making
QUEUE_FLAG_SAME_FORCEdriver owned, and reject any attempts to change it via sysfs with EINVAL.
I understand your concern. However, with the nesting counter, a userspace clear cannot disable QUEUE_FLAG_SAME_FORCE while another user, such as a driver or the latency-based I/O policy, still holds a reference to it.
For example, starting with:q->same_force_depth = 0;q->same_force_sysfs = false;
The sequence would be:
1. Userspace enables QUEUE_FLAG_SAME_FORCE by writing 2 to rq_affinity:
q->same_force_depth = 1;
q->same_force_sysfs = true;
QUEUE_FLAG_SAME_FORCE = set;
2. The driver then enables QUEUE_FLAG_SAME_FORCE:
q->same_force_depth = 2;
QUEUE_FLAG_SAME_FORCE = set;
3. Userspace disables it by writing 0 or 1 to rq_affinity:
q->same_force_depth = 1;
q->same_force_sysfs = false;
QUEUE_FLAG_SAME_FORCE = still set; 4. The driver subsequently releases its reference:
q->same_force_depth = 0;
QUEUE_FLAG_SAME_FORCE = clear;
Thus, the userspace clear only releases the sysfs reference; it does not actually clear QUEUE_FLAG_SAME_FORCE while the driver still holds its reference.
The same applies to the upcoming latency-based I/O policy. As long as the policy holds a reference, a userspace write to rq_affinity cannot disable QUEUE_FLAG_SAME_FORCE underneath it.
This is the motivation for the nesting support: it allows multiple users to independently acquire and release the flag without one user inadvertently affecting another user's requirement for the flag.
Does the above explanation help address your concern, or do you have any further concern?
Thanks,
--Nilay
PATCH v8 4/10 nvme-multipath: pass I/O type to nvme_find_path()
In reply to Nilay ShroffCurrently, nvme_find_path() only accepts an nvme_ns_head argument. However, the upcoming latency-aware I/O policy also needs to know the I/O type (read/write/other) associated with the request in order to make path selection decisions.
Update nvme_find_path() to accept an additional argument describing the I/O type. Classify requests into three categories: READ, WRITE, and OTHER. Admin commands and I/O requests that are neither reads nor writes are classified as OTHER.
This patch does not introduce any functional change and only prepares the interface for subsequent latency-policy changes.
--- drivers/nvme/host/ioctl.c | 44 +++++++++++++++++++++++++++--- drivers/nvme/host/multipath.c | 9 ++++--- drivers/nvme/host/nvme.h | 50 ++++++++++++++++++++++++++++++++++- drivers/nvme/host/pr.c | 6 +++-- drivers/nvme/host/sysfs.c | 2 +- 5 files changed, 100 insertions(+), 11 deletions(-)
drivers/nvme/host/ioctl.c +41 -3
@@ -751,12 +751,25 @@ int nvme_ns_head_ioctl(struct block_device *bdev, blk_mode_t mode,struct nvme_ns *ns;int srcu_idx, ret = -EWOULDBLOCK ;unsigned int flags =0 ; +unsigned int op_type =NVME_STAT_GROUP_OTHER ;if (bdev_is_partition (bdev)) flags |=NVME_IOCTL_PARTITION ; +if (cmd ==NVME_IOCTL_SUBMIT_IO ) { +u8 opcode; + +if (get_user (opcode, (u8 *)argp)) +return -EFAULT ; + +if (opcode == nvme_cmd_write) + op_type =NVME_STAT_GROUP_WRITE ; +else if (opcode == nvme_cmd_read) + op_type =NVME_STAT_GROUP_READ ; + } + srcu_idx =srcu_read_lock (&head->srcu); - ns =nvme_find_path (head); + ns =nvme_find_path (head, op_type);if (!ns) goto out_unlock;
@@ -785,9 +798,22 @@ long nvme_ns_head_chr_ioctl(struct file *file, unsigned int cmd,void __user *argp = (void __user *)arg;struct nvme_ns *ns;int srcu_idx, ret = -EWOULDBLOCK ; +unsigned int op_type =NVME_STAT_GROUP_OTHER ; + +if (cmd ==NVME_IOCTL_SUBMIT_IO ) { +u8 opcode; + +if (get_user (opcode, (u8 *)argp)) +return -EFAULT ; + +if (opcode == nvme_cmd_write) + op_type =NVME_STAT_GROUP_WRITE ; +else if (opcode == nvme_cmd_read) + op_type =NVME_STAT_GROUP_READ ; + } srcu_idx =srcu_read_lock (&head->srcu); - ns =nvme_find_path (head); + ns =nvme_find_path (head, op_type);if (!ns) goto out_unlock;
@@ -804,12 +830,24 @@ long nvme_ns_head_chr_ioctl(struct file *file, unsigned int cmd,int nvme_ns_head_chr_uring_cmd (struct io_uring_cmd *ioucmd,unsigned int issue_flags) { +struct nvme_ns *ns; +unsigned int op_type;struct cdev *cdev =file_inode (ioucmd->file)->i_cdev; struct nvme_ns_head *head = container_of(cdev, struct nvme_ns_head, cdev);int srcu_idx =srcu_read_lock (&head->srcu); -struct nvme_ns *ns =nvme_find_path (head);int ret = -EINVAL ; +const struct nvme_uring_cmd *cmd =io_uring_sqe128_cmd (ioucmd->sqe, + struct nvme_uring_cmd); +__u8 opcode =READ_ONCE (cmd->opcode); + +if (opcode == nvme_cmd_write) + op_type =NVME_STAT_GROUP_WRITE ; +else if (opcode == nvme_cmd_read) + op_type =NVME_STAT_GROUP_READ ; +else + op_type =NVME_STAT_GROUP_OTHER ; + ns =nvme_find_path (head, op_type);if (ns) ret =nvme_ns_uring_cmd (ns, ioucmd, issue_flags);srcu_read_unlock (&head->srcu, srcu_idx);
drivers/nvme/host/multipath.c +5 -4
@@ -484,7 +484,8 @@ static struct nvme_ns *nvme_numa_path(struct nvme_ns_head *head)return ns; } -inline struct nvme_ns *nvme_find_path (struct nvme_ns_head *head) +inline struct nvme_ns *nvme_find_path (struct nvme_ns_head *head, +enum nvme_stat_group op_type) {switch (READ_ONCE (head->subsys->iopolicy)) {case NVME_IOPOLICY_QD :
@@ -547,7 +548,7 @@ static void nvme_ns_head_submit_bio(struct bio *bio)return ; srcu_idx =srcu_read_lock (&head->srcu); - ns =nvme_find_path (head); + ns =nvme_find_path (head,__nvme_get_stat_group (bio_op (bio)));if (likely (ns)) {bio_set_dev (bio, ns->disk->part0);/*
@@ -597,7 +598,7 @@ static int nvme_ns_head_get_unique_id(struct gendisk *disk, u8 id[16],int srcu_idx, ret = -EWOULDBLOCK ; srcu_idx =srcu_read_lock (&head->srcu); - ns =nvme_find_path (head); + ns =nvme_find_path (head,NVME_STAT_GROUP_OTHER );if (ns) ret =nvme_ns_get_unique_id (ns, id, type);srcu_read_unlock (&head->srcu, srcu_idx);
@@ -613,7 +614,7 @@ static int nvme_ns_head_report_zones(struct gendisk *disk, sector_t sector,int srcu_idx, ret = -EWOULDBLOCK ; srcu_idx =srcu_read_lock (&head->srcu); - ns =nvme_find_path (head); + ns =nvme_find_path (head,NVME_STAT_GROUP_OTHER );if (ns) ret =nvme_ns_report_zones (ns, sector, nr_zones, args);srcu_read_unlock (&head->srcu, srcu_idx);
drivers/nvme/host/nvme.h +49 -1
63 lines not on this page · raw message
drivers/nvme/host/pr.c +4 -2
@@ -53,10 +53,12 @@ static int nvme_send_ns_head_pr_command(struct block_device *bdev, struct nvme_command *c, void *data, unsigned int data_len) { struct nvme_ns_head *head = bdev->bd_disk->private_data; -int srcu_idx =srcu_read_lock (&head->srcu); -struct nvme_ns *ns =nvme_find_path (head); +int srcu_idx; +struct nvme_ns *ns;int ret = -EWOULDBLOCK ; + srcu_idx =srcu_read_lock (&head->srcu); + ns =nvme_find_path (head,NVME_STAT_GROUP_OTHER );if (ns) { c->common.nsid =cpu_to_le32 (ns->head->ns_id); ret =nvme_submit_sync_cmd (ns->queue, c, data, data_len);
drivers/nvme/host/sysfs.c +1 -1
@@ -195,7 +195,7 @@ static int ns_head_update_nuse(struct nvme_ns_head *head)return 0 ; srcu_idx =srcu_read_lock (&head->srcu); - ns =nvme_find_path (head); + ns =nvme_find_path (head,NVME_STAT_GROUP_OTHER );if (!ns) goto out_unlock;
Signature · 1 line
2.53.0
Keith Busch
maintainerRe: [PATCH v8 04/10] nvme-multipath: pass I/O type to nvme_find_path()
Quoted 4 lines · Nilay Shroff
drivers/nvme/host/ioctl.c +0
-0@@ -804,12 +830,24 @@ long nvme_ns_head_chr_ioctl(struct file *file, unsigned int cmd,int nvme_ns_head_chr_uring_cmd (struct io_uring_cmd *ioucmd,unsigned int issue_flags) {
drivers/nvme/host/ioctl.c long nvme_ns_head_chr_ioctl(struct file *file, unsigned int cmd, · line 841
Quoted 10 lines
drivers/nvme/host/ioctl.c +5
-1+struct nvme_ns *ns; +unsigned int op_type;struct cdev *cdev =file_inode (ioucmd->file)->i_cdev; struct nvme_ns_head *head = container_of(cdev, struct nvme_ns_head, cdev);int srcu_idx =srcu_read_lock (&head->srcu); -struct nvme_ns *ns =nvme_find_path (head);int ret = -EINVAL ; +const struct nvme_uring_cmd *cmd =io_uring_sqe128_cmd (ioucmd->sqe, + struct nvme_uring_cmd); +__u8 opcode =READ_ONCE (cmd->opcode);
Just fyi, the READ_ONCE is used because this is shared memory with user space. We re-read the same field later when constructing the command, so if the user does something tricky like changing that opcode, you can have the wrong group for the command that actually gets dispatched.
Re: [PATCH v8 04/10] nvme-multipath: pass I/O type to nvme_find_path()
Quoted 12 lines · Keith Busch
Quoted 11 lines · Nilay Shroff
drivers/nvme/host/ioctl.c +2
-1@@ -804,12 +830,24 @@ long nvme_ns_head_chr_ioctl(struct file *file, unsigned int cmd,int nvme_ns_head_chr_uring_cmd (struct io_uring_cmd *ioucmd,unsigned int issue_flags) { +struct nvme_ns *ns; +unsigned int op_type;struct cdev *cdev =file_inode (ioucmd->file)->i_cdev; struct nvme_ns_head *head = container_of(cdev, struct nvme_ns_head, cdev);int srcu_idx =srcu_read_lock (&head->srcu); -struct nvme_ns *ns =nvme_find_path (head);int ret = -EINVAL ;
Quoted 8 lines
drivers/nvme/host/ioctl.c long nvme_ns_head_chr_ioctl(struct file *file, unsigned int cmd, · line 841
Quoted 3 lines
drivers/nvme/host/ioctl.c +3
-0+const struct nvme_uring_cmd *cmd =io_uring_sqe128_cmd (ioucmd->sqe, + struct nvme_uring_cmd); +__u8 opcode =READ_ONCE (cmd->opcode);Just fyi, the
READ_ONCEis used because this is shared memory with user space. We re-read the same field later when constructing the command, so if the user does something tricky like changing that opcode, you can have the wrong group for the command that actually gets dispatched.
Yes, agreed. If userspace modifies cmd->opcode while the SQE is being consumed by the kernel, we could end up selecting a path based on one opcode value and dispatching a command with another. However, per the io_uring ABI/protocol, userspace must not modify or reuse an SQE after publishing it to the kernel until the kernel has consumed it (i.e. advanced the SQ ring head). Therefore, under the expected protocol, the SQE contents are stable while the kernel is consuming them.
Nevertheless, if userspace modifies an SQE before it has been consumed by the kernel, then that violates the io_uring SQE ownership protocol, and the resulting behavior is not something we need to account for here, IMO.
Thanks,
--Nilay
PATCH v8 5/10 nvme-multipath: add support for latency I/O policy
In reply to Nilay ShroffThis commit introduces a new I/O policy named "latency". Users can configure it by writing "latency" to "/sys/class/nvme-subsystem/nvme- subsystemX/iopolicy"
The "latency" policy dynamically distributes I/O based on measured I/O completion latency. The main idea is to calculate latency for each path, derive a weight, and then proportionally forward I/O according to those weights.
To ensure scalability, path latency is measured per-CPU. Each CPU maintains its own statistics, and I/O forwarding uses these per-CPU values. Every ~15 seconds, a simple average latency of per-CPU batched samples are computed and fed into an Exponentially Weighted Moving Average (EWMA):
avg_latency = div_u64(batch, batch_count);new_ewma_latency = (prev_ewma_latency * (WEIGHT-1) + avg_latency)/WEIGHT
With WEIGHT = 8, this assigns 7/8 (~87.5%) weight to the previous latency value and 1/8 (~12.5%) to the most recent latency. This smoothing reduces jitter, adapts quickly to changing conditions, avoids storing historical samples, and works well for both low and high I/O rates. Path weights are then derived from the smoothed (EWMA) latency as follows (example with two paths A and B):
path_A_score = NSEC_PER_SEC / path_A_ewma_latency path_B_score = NSEC_PER_SEC / path_B_ewma_latency total_score = path_A_score + path_B_score
path_A_weight = (path_A_score * 64) / total_score path_B_weight = (path_B_score * 64) / total_score
where:
- path_X_ewma_latency is the smoothed latency of a path in nanoseconds
- NSEC_PER_SEC is used as a scaling factor since valid latencies are < 1 second
- weights are normalized to a 0–64 scale across all paths.
Path credits are refilled based on this weight, with one credit consumed per I/O. When all credits are consumed, the credits are refilled again based on the current weight. This ensures that I/O is distributed across paths proportionally to their calculated weight.
--- drivers/nvme/host/core.c | 12 +- drivers/nvme/host/multipath.c | 462 +++++++++++++++++++++++++++++++++- drivers/nvme/host/nvme.h | 48 +++- 3 files changed, 507 insertions(+), 15 deletions(-)
drivers/nvme/host/core.c +9 -3
@@ -719,6 +719,7 @@ static void nvme_free_ns(struct kref *kref) { struct nvme_ns *ns = container_of(kref, struct nvme_ns, kref); +nvme_free_ns_stat (ns);put_disk (ns->disk);nvme_put_ns_head (ns->head);nvme_put_ctrl (ns->ctrl);
@@ -4258,6 +4259,9 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)if (nvme_init_ns_head (ns, info)) goto out_cleanup_disk; +if (nvme_mpath_alloc_ns_stat (ns)) + goto out_unlink_ns; +/* * If multipathing is enabled, the device name for all disks and not * just those that represent shared namespaces needs to be based on the
@@ -4282,7 +4286,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info) }if (nvme_update_ns_info (ns, info)) - goto out_unlink_ns; + goto out_free_ns_stat;mutex_lock (&ctrl->namespaces_lock);/*
@@ -4291,7 +4295,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)*/ if (test_bit (NVME_CTRL_FROZEN , &ctrl->flags)) {mutex_unlock (&ctrl->namespaces_lock); - goto out_unlink_ns; + goto out_free_ns_stat; }blk_queue_rq_timeout (ns->queue, ctrl->io_timeout);nvme_ns_add_to_ctrl_list (ns);
@@ -4316,6 +4320,8 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)list_del_rcu (&ns->list);mutex_unlock (&ctrl->namespaces_lock);synchronize_srcu (&ctrl->srcu); +out_free_ns_stat: +nvme_free_ns_stat (ns); out_unlink_ns:mutex_lock (&ctrl->subsys->lock);list_del_rcu (&ns->siblings);
@@ -4355,7 +4361,7 @@ static void nvme_ns_remove(struct nvme_ns *ns)/* * Ensure that !NVME_NS_READY is seen by other threads to prevent -* this ns going back into current_path. +* this ns going back into current_path/latency_path. */ synchronize_srcu (&ns->head->srcu);
drivers/nvme/host/multipath.c +451 -11
569 lines not on this page · raw message
drivers/nvme/host/nvme.h +47 -1
102 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH v8 6/10 nvme: add generic debugfs support
In reply to Nilay ShroffAdd generic infrastructure for creating and managing debugfs files in the NVMe module. This introduces helper APIs that allow NVMe drivers to register and unregister debugfs entries, along with a reusable attribute structure for defining new debugfs files.
The implementation uses seq_file interfaces to safely expose per-NS and per-NS-head statistics, while supporting both simple show callbacks and full seq_operations.
--- drivers/nvme/host/Makefile | 2 +- drivers/nvme/host/core.c | 3 + drivers/nvme/host/debugfs.c | 141 ++++++++++++++++++++++++++++++++++ drivers/nvme/host/multipath.c | 2 + drivers/nvme/host/nvme.h | 10 +++ 5 files changed, 157 insertions(+), 1 deletion(-) create mode 100644 drivers/nvme/host/debugfs.c
drivers/nvme/host/Makefile +1 -1
@@ -11,7 +11,7 @@ obj-$(CONFIG_NVME_FC) += nvme-fc.o obj-$(CONFIG_NVME_TCP) += nvme-tcp.o obj-$(CONFIG_NVME_APPLE) += nvme-apple.o -nvme-core-y += core.o ioctl.o sysfs.o pr.o +nvme-core-y += core.o ioctl.o sysfs.o pr.o debugfs.o nvme-core-$(CONFIG_NVME_VERBOSE_ERRORS) += constants.o nvme-core-$(CONFIG_TRACING) += trace.o nvme-core-$(CONFIG_NVME_MULTIPATH) += multipath.o
drivers/nvme/host/core.c +3 -0
@@ -4306,6 +4306,8 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)if (device_add_disk (ctrl->device, ns->disk, nvme_ns_attr_groups)) goto out_cleanup_ns_from_list; +nvme_debugfs_register (ns->disk); +if (!nvme_ns_head_multipath (ns->head))nvme_add_ns_cdev (ns);
@@ -4388,6 +4390,7 @@ static void nvme_ns_remove(struct nvme_ns *ns)nvme_mpath_remove_sysfs_link (ns); +nvme_debugfs_unregister (ns->disk);del_gendisk (ns->disk);mutex_lock (&ns->ctrl->namespaces_lock);
drivers/nvme/host/debugfs.c new file +141 -0
141 lines not on this page · raw message
drivers/nvme/host/multipath.c +2 -0
@@ -1137,6 +1137,7 @@ static void nvme_remove_head(struct nvme_ns_head *head)if (test_and_clear_bit (NVME_NSHEAD_CDEV_LIVE , &head->flags))nvme_cdev_del (&head->cdev, &head->cdev_device); +nvme_debugfs_unregister (head->disk);del_gendisk (head->disk); }nvme_put_ns_head (head);
@@ -1244,6 +1245,7 @@ static void nvme_mpath_set_live(struct nvme_ns *ns) }nvme_add_ns_head_cdev (head);queue_work (nvme_wq, &head->partition_scan_work); +nvme_debugfs_register (head->disk); }nvme_mpath_add_sysfs_link (ns->head);
drivers/nvme/host/nvme.h +10 -0
@@ -1127,6 +1127,16 @@ static inline enum nvme_stat_group nvme_get_stat_group(struct request *req)return __nvme_get_stat_group (req_op (req)); } +void nvme_debugfs_register (struct gendisk *disk); +static inline void nvme_debugfs_unregister (struct gendisk *disk) +{ +/* +* Nothing to do for now. When the request queue is unregistered, +* all files under q->debugfs_dir are recursively deleted. +* This is just a placeholder; the compiler will optimize it out. +*/ +} +#ifdef CONFIG_NVME_MULTIPATH static inline bool nvme_ctrl_use_ana (struct nvme_ctrl *ctrl) {
Signature · 1 line
2.53.0
PATCH v8 7/10 nvme-multipath: add debugfs attribute latency_ewma_shift
In reply to Nilay ShroffBy default, the EWMA (Exponentially Weighted Moving Average) shift value, used for storing latency samples for latency iopolicy, is set to 3. The EWMA is calculated using the following formula:
ewma = (old * ((1 << ewma_shift) - 1) + new) >> ewma_shift;
The default value of 3 assigns ~87.5% weight to the existing EWMA value and ~12.5% weight to the new latency sample. This provides a stable average that smooths out short-term variations.
However, different workloads may require faster or slower adaptation to changing conditions. This commit introduces a new debugfs attribute, latency_ewma_shift, allowing users to tune the weighting factor.
For example:
- latency_ewma_shift = 2 => 75% old, 25% new
- latency_ewma_shift = 1 => 50% old, 50% new
- latency_ewma_shift = 0 => 0% old, 100% new
--- drivers/nvme/host/debugfs.c | 46 +++++++++++++++++++++++++++++++++++ drivers/nvme/host/multipath.c | 9 ++++--- drivers/nvme/host/nvme.h | 1 + 3 files changed, 52 insertions(+), 4 deletions(-)
drivers/nvme/host/debugfs.c +46 -0
@@ -108,8 +108,54 @@ static const struct file_operations nvme_debugfs_fops = { .release = nvme_debugfs_release, }; +#ifdef CONFIG_NVME_MULTIPATH +static int nvme_latency_ewma_shift_show (void *data,struct seq_file *m) +{ +struct nvme_ns_head *head = data; + +seq_printf (m,"%u\n" ,READ_ONCE (head->latency_ewma_shift)); +return 0 ; +} + +static ssize_t nvme_latency_ewma_shift_store (void *data, +const char __user *ubuf,size_t count,loff_t *ppos) +{ +struct nvme_ns_head *head = data; +char kbuf[8 ]; +u32 res; +int ret; +size_t len; +char *arg; + + len =min (sizeof (kbuf) -1 , count); + +if (copy_from_user (kbuf, ubuf, len)) +return -EFAULT ; + + kbuf[len] ='\0' ; + arg =strstrip (kbuf); + + ret =kstrtou32 (arg,0 , &res); +if (ret) +return ret; + +/* +* Values greater than 8 are nonsensical, as they effectively assign +* zero weight to new samples. +*/ +if (res >8 ) +return -EINVAL ; + +WRITE_ONCE (head->latency_ewma_shift, res); +return count; +} +#endif static const struct nvme_debugfs_attr nvme_mpath_debugfs_attrs[] = { +#ifdef CONFIG_NVME_MULTIPATH + {"latency_ewma_shift" ,0600 , nvme_latency_ewma_shift_show, + nvme_latency_ewma_shift_store}, +#endif {}, };
drivers/nvme/host/multipath.c +5 -4
@@ -293,10 +293,9 @@ static void nvme_mpath_weight_work(struct work_struct *weight_work)* For instance, with EWMA_SHIFT = 3, this assigns 7/8 (~87.5 %) weight to * the existing/old ewma and 1/8 (~12.5%) weight to the new sample. */ -static inline u64 calc_ewma_update (u64 old,u64 new) +static inline u64 calc_ewma_update (u64 old,u64 new,u32 ewma_shift) { -return (old * ((1 <<NVME_DEFAULT_LATENCY_EWMA_SHIFT ) -1 ) - + new) >>NVME_DEFAULT_LATENCY_EWMA_SHIFT ; +return (old * ((1 << ewma_shift) -1 ) + new) >> ewma_shift; }static void nvme_mpath_add_sample (struct request *rq,struct nvme_ns *ns)
@@ -388,7 +387,8 @@ static void nvme_mpath_add_sample(struct request *rq, struct nvme_ns *ns)if (unlikely (!stat->slat_ns))WRITE_ONCE (stat->slat_ns, avg_lat_ns);else { - slat_ns =calc_ewma_update (stat->slat_ns, avg_lat_ns); + slat_ns =calc_ewma_update (stat->slat_ns, avg_lat_ns, +READ_ONCE (head->latency_ewma_shift));WRITE_ONCE (stat->slat_ns, slat_ns); }
@@ -1170,6 +1170,7 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)INIT_WORK (&head->requeue_work, nvme_requeue_work);INIT_WORK (&head->partition_scan_work, nvme_partition_scan_work);INIT_DELAYED_WORK (&head->remove_work, nvme_remove_head_work); + head->latency_ewma_shift =NVME_DEFAULT_LATENCY_EWMA_SHIFT ;/* * If "multipath_always_on" is enabled, a multipath node is added
drivers/nvme/host/nvme.h +1 -0
@@ -626,6 +626,7 @@ struct nvme_ns_head {atomic_long_t io_requeue_no_usable_path_count;atomic_long_t io_fail_no_available_path_count;struct nvme_ns *__percpu *latency_path; +u32 latency_ewma_shift;#define NVME_NSHEAD_DISK_LIVE 0#define NVME_NSHEAD_QUEUE_IF_NO_PATH 1
Signature · 1 line
2.53.0
PATCH v8 8/10 nvme-multipath: add debugfs attribute latency_batch_timeout
In reply to Nilay ShroffBy default, the latency I/O policy accumulates latency samples over a 15-second window. When this window expires, the driver computes the average latency and updates the smoothed (EWMA) latency value. The path weight is then recalculated based on this data.
A 15-second window provides a good balance for most workloads, as it helps smooth out transient latency spikes and produces a more stable path weight profile. However, some workloads may benefit from faster or slower adaptation to changing latency conditions.
This commit introduces a new debugfs attribute, latency_batch_timeout, which allows users to configure the latency batch window and thus path weight calculation interval based on their workload requirements.
--- drivers/nvme/host/debugfs.c | 37 +++++++++++++++++++++++++++++++++++ drivers/nvme/host/multipath.c | 8 ++++++-- drivers/nvme/host/nvme.h | 1 + 3 files changed, 44 insertions(+), 2 deletions(-)
drivers/nvme/host/debugfs.c +37 -0
@@ -149,12 +149,49 @@ static ssize_t nvme_latency_ewma_shift_store(void *data,WRITE_ONCE (head->latency_ewma_shift, res);return count; } + +static int nvme_latency_batch_timeout_show (void *data,struct seq_file *m) +{ +struct nvme_ns_head *head = data; + +seq_printf (m,"%llu\n" , +div_u64 (READ_ONCE (head->latency_batch_timeout),NSEC_PER_SEC )); +return 0 ; +} + +static ssize_t nvme_latency_batch_timeout_store (void *data, +const char __user *ubuf,size_t count,loff_t *ppos) +{ +struct nvme_ns_head *head = data; +char kbuf[8 ]; +u32 res; +int ret; +size_t len; +char *arg; + + len =min (sizeof (kbuf) -1 , count); + +if (copy_from_user (kbuf, ubuf, len)) +return -EFAULT ; + + kbuf[len] ='\0' ; + arg =strstrip (kbuf); + + ret =kstrtou32 (arg,0 , &res); +if (ret) +return ret; + +WRITE_ONCE (head->latency_batch_timeout, res *NSEC_PER_SEC ); +return count; +}#endif static const struct nvme_debugfs_attr nvme_mpath_debugfs_attrs[] = { #ifdef CONFIG_NVME_MULTIPATH {"latency_ewma_shift" ,0600 , nvme_latency_ewma_shift_show, nvme_latency_ewma_shift_store}, + {"latency_batch_timeout" ,0600 , nvme_latency_batch_timeout_show, + nvme_latency_batch_timeout_store}, #endif {}, };
drivers/nvme/host/multipath.c +6 -2
@@ -362,8 +362,11 @@ static void nvme_mpath_add_sample(struct request *rq, struct nvme_ns *ns) stat->batch_count++; stat->nr_samples++; -if (now > stat->last_batch_ts && ((now - stat->last_batch_ts) >= -NVME_DEFAULT_LATENCY_BATCH_TIMEOUT )) { +if (now > stat->last_batch_ts) { +u64 timeout =READ_ONCE (head->latency_batch_timeout); + +if ((now - stat->last_batch_ts) < timeout) +return ;/* * Find simple average latency for the last epoch (~15 sec
@@ -1171,6 +1174,7 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)INIT_WORK (&head->partition_scan_work, nvme_partition_scan_work);INIT_DELAYED_WORK (&head->remove_work, nvme_remove_head_work); head->latency_ewma_shift =NVME_DEFAULT_LATENCY_EWMA_SHIFT ; + head->latency_batch_timeout =NVME_DEFAULT_LATENCY_BATCH_TIMEOUT ;/* * If "multipath_always_on" is enabled, a multipath node is added
drivers/nvme/host/nvme.h +1 -0
@@ -627,6 +627,7 @@ struct nvme_ns_head {atomic_long_t io_fail_no_available_path_count;struct nvme_ns *__percpu *latency_path;u32 latency_ewma_shift; +u64 latency_batch_timeout;#define NVME_NSHEAD_DISK_LIVE 0#define NVME_NSHEAD_QUEUE_IF_NO_PATH 1
Signature · 1 line
2.53.0
PATCH v8 9/10 nvme-multipath: add debugfs attribute latency_stat
In reply to Nilay ShroffThis commit introduces a new debugfs attribute, "latency_stat", under both per-path and head debugfs directories (defined under /sys/kernel/ debug/block/). This attribute provides visibility into the internal state of the latency I/O policy to aid in debugging and performance analysis.
For per-path entries, "latency_stat" reports the corresponding path statistics such as I/O weight, selection count, processed samples, and ignored samples.
For head entries, it reports per-CPU statistics for each reachable path, including I/O weight, path score, smoothed (EWMA) latency, selection count, processed samples, and ignored samples.
These additions enhance observability of the I/O path selection behavior and help diagnose imbalance or instability in multipath performance.
--- drivers/nvme/host/debugfs.c | 123 ++++++++++++++++++++++++++++++++++++ 1 file changed, 123 insertions(+)
drivers/nvme/host/debugfs.c +123 -0
@@ -184,6 +184,125 @@ static ssize_t nvme_latency_batch_timeout_store(void *data,WRITE_ONCE (head->latency_batch_timeout, res *NSEC_PER_SEC );return count; } + +#define TO_CTX (m) ((struct nvme_debugfs_ctx *)m->private) +#define TO_NS_HEAD (m) ((struct nvme_ns_head *)(TO_CTX(m)->data)) + +static void *nvme_mpath_latency_stat_start (struct seq_file *m,loff_t *pos) + __acquires_shared(&TO_NS_HEAD(m)->srcu) +{ +struct nvme_ns *ns; +struct nvme_debugfs_ctx *ctx = m->private; +struct nvme_ns_head *head = ctx->data; +loff_t n = *pos; + +/* Remember srcu index, so we can unlock later. */ + ctx->srcu_idx =srcu_read_lock (&head->srcu); + ns = list_first_or_null_rcu(&head->list, struct nvme_ns, siblings); + +while (n && ns) { + ns =list_next_or_null_rcu (&head->list, &ns->siblings, + struct nvme_ns, siblings); + n--; + } + +return ns; +} + +static void *nvme_mpath_latency_stat_next (struct seq_file *m,void *v, +loff_t *pos) +{ +struct nvme_ns *ns = v; +struct nvme_debugfs_ctx *ctx = m->private; +struct nvme_ns_head *head = ctx->data; + + (*pos)++; + +return list_next_or_null_rcu (&head->list, &ns->siblings, + struct nvme_ns, siblings); +} + +static void nvme_mpath_latency_stat_stop (struct seq_file *m,void *v) + __releases_shared(&TO_NS_HEAD(m)->srcu) +{ +struct nvme_debugfs_ctx *ctx = m->private; +struct nvme_ns_head *head = ctx->data; + +srcu_read_unlock (&head->srcu, ctx->srcu_idx); +} + +static int nvme_mpath_latency_stat_show (struct seq_file *m,void *v) +{ +int i, cpu; +struct nvme_path_lat_stat *stat; +struct nvme_ns *ns = v; + +seq_printf (m,"%s:\n" , ns->disk->disk_name); +for_each_online_cpu (cpu) { +seq_printf (m,"cpu %d : " , cpu); +for (i =0 ; i <NVME_NUM_STAT_GROUPS ; i++) { + stat = &per_cpu_ptr (ns->path_lat, cpu)[i].stat; +seq_printf (m,"%u %u %llu %llu %llu %llu %llu " , + stat->weight, stat->credit, stat->score, + stat->slat_ns, stat->sel, + stat->nr_samples, stat->nr_ignored); + } +seq_putc (m,'\n' ); + } +return 0 ; +} + +static const struct seq_operations nvme_mpath_latency_stat_seq_ops = { + .start = nvme_mpath_latency_stat_start, + .next = nvme_mpath_latency_stat_next, + .stop = nvme_mpath_latency_stat_stop, + .show = nvme_mpath_latency_stat_show +}; + +static void nvme_latency_stat_read_all (struct nvme_ns *ns, +struct nvme_path_lat_stat *batch) +{ +int i, cpu; +u32 ncpu[NVME_NUM_STAT_GROUPS ] = {0 }; +struct nvme_path_lat_stat *stat; + +for_each_online_cpu (cpu) { +for (i =0 ; i <NVME_NUM_STAT_GROUPS ; i++) { + stat = &per_cpu_ptr (ns->path_lat, cpu)[i].stat; + batch[i].sel += stat->sel; + batch[i].nr_samples += stat->nr_samples; + batch[i].nr_ignored += stat->nr_ignored; + batch[i].weight += stat->weight; +if (stat->weight) + ncpu[i]++; + } + } + +for (i =0 ; i <NVME_NUM_STAT_GROUPS ; i++) { +if (!ncpu[i]) +continue ; + batch[i].weight =DIV_U64_ROUND_CLOSEST (batch[i].weight, + ncpu[i]); + } +} + +static int nvme_ns_latency_stat_show (void *data,struct seq_file *m) +{ +int i; +struct nvme_path_lat_stat stat[NVME_NUM_STAT_GROUPS ] = {0 }; +struct nvme_ns *ns = (struct nvme_ns *)data; + +if (!ns->head->disk) +return 0 ; + +nvme_latency_stat_read_all (ns, stat); +for (i =0 ; i <NVME_NUM_STAT_GROUPS ; i++) { +seq_printf (m,"%u %llu %llu %llu " , + stat[i].weight, stat[i].sel, + stat[i].nr_samples, stat[i].nr_ignored); + } +return 0 ; +}#endif static const struct nvme_debugfs_attr nvme_mpath_debugfs_attrs[] = {
@@ -192,11 +311,15 @@ static const struct nvme_debugfs_attr nvme_mpath_debugfs_attrs[] = { nvme_latency_ewma_shift_store}, {"latency_batch_timeout", 0600, nvme_latency_batch_timeout_show, nvme_latency_batch_timeout_store}, + {"latency_stat", 0400, .seq_ops = &nvme_mpath_latency_stat_seq_ops}, #endif {}, };static const struct nvme_debugfs_attr nvme_ns_debugfs_attrs[] = { +#ifdef CONFIG_NVME_MULTIPATH + {"latency_stat" ,0400 , nvme_ns_latency_stat_show}, +#endif {}, };
Signature · 1 line
2.53.0
PATCH v8 10/10 nvme-multipath: add documentation for latency I/O policy
In reply to Nilay ShroffUpdate the nvme-multipath documentation to describe the latency I/O policy, its behavior, and when it is suitable for use.
--- Documentation/admin-guide/nvme-multipath.rst | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
Documentation/admin-guide/nvme-multipath.rst +19 -0
@@ -70,3 +70,22 @@ When to use the queue-depth policy: 1. High load with small I/Os: Effectively balances load across paths when the load is high, and I/O operations consist of small, relatively fixed-sized requests. + +Latency +-------- + +The latency policy manages I/O requests based on path latency. It periodically +calculates a weight for each path and distributes I/O accordingly. Paths with +higher latency receive lower weights, resulting in fewer I/O requests being sent +to them, while paths with lower latency handle a proportionally larger share of +the I/O load. + +When to use the latency policy +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +1. Homogeneous Path Performance: Utilizes all available paths efficiently when + their performance characteristics (e.g., latency, bandwidth) are similar. + +2. Heterogeneous Path Performance: Dynamically distributes I/O based on per-path + performance characteristics. Paths with lower latency receive a higher share + of I/O compared to those with higher latency.
Signature · 1 line
2.53.0
Sagi Grimberg
maintainerQuoted 20 lines · Nilay Shroff
Hi,
This series introduces a new latency I/O policy for NVMe native multipath. Existing policies such as numa, round-robin, and queue-depth are static and do not adapt to real-time transport performance. The numa selects the path closest to the NUMA node of the current CPU, optimizing memory and path locality, but ignores actual path performance. The round-robin distributes I/O evenly across all paths, providing fairness but not performance awareness. The queue-depth reacts to instantaneous queue occupancy, avoiding heavily loaded paths, but does not account for actual latency, throughput, or link speed.
The new latency policy addresses these gaps selecting paths dynamically based on measured I/O latency for both PCIe and fabrics. Latency is derived by passively sampling I/O completions. Each path is assigned a weight proportional to its latency score, and I/Os are then forwarded accordingly. As condition changes (e.g. latency spikes, bandwidth differences), path weights are updated, automatically steering traffic toward better-performing paths.
Quoted 5 lines
Early results show reduced tail latency under mixed workloads and improved throughput by exploiting higher-speed links more effectively. For example, with NVMf/TCP using two paths (one throttled with ~30 ms delay), fio results with random read/write/rw workloads (direct I/O) showed:
TBH, I do not know if this measurement represent any real-life scenario. I do think that occasional packet drops are a real-life scenario, and it would be a worthy use-case to optimize for. Can you perhaps measure how the path selectors compare in this case?
Quoted 7 lines
numa round-robin queue-depth adaptive
----------- ----------- ----------- ---------
READ: 50.0 MiB/s 105 MiB/s 230 MiB/s 350 MiB/s
WRITE: 65.9 MiB/s 125 MiB/s 385 MiB/s 446 MiB/s
RW: R:30.6 MiB/s R:56.5 MiB/s R:122 MiB/s R:175 MiB/s
W:30.7 MiB/s W:56.5 MiB/s W:122 MiB/s W:175 MiB/sAnd I'm assuming there are zero downsides for the normal case?
Quoted 60 lines
This pathcset includes totla 8 patches:
[PATCH 1/10] block: expose blk_stat_{enable,disable}_accounting()
- Makeblk_statAPIs available to block drivers.
- Needed for per-path latency measurement.[PATCH 2/10] block: record I/O request start time for passthru request
- Record I/O start time for I/O passthru requests.
- This is prep patch which allows measuring I/O completion latency for passthru requests.[PATCH 3/10] block: support nesting for blk-mq flag
QUEUE_FLAG_SAME_FORCE
- Support nesting forQUEUE_FLAG_SAME_FORCEas multiple users could toggleQUEUE_FLAG_SAME_FORCE.[PATCH 4/10] nvme-multipath: pass I/O type to
nvme_find_path()
- This is the prep patch which updatesnvme_find_path()signature[PATCH 5/10] nvme-multipath: add latency I/O policy
- Implement path scoring based on latency (EWMA).
- Distribute I/O proportionally to per-path weights.[PATCH 6/10] nvme: add generic debugfs support
- Introduce generic debugfs support for NVMe module[PATCH 7/10] nvme-multipath: add debugfs attribute
latency_ewma_shift
- Adds a debugfs attribute to control ewma shift[PATCH 8/10] nvme-multipath: add debugfs attribute
latency_batch_timeout
- Adds a debugfs attribute to control latency batch window interval[PATCH 9/10] nvme-multipath: add debugfs attribute
latency_stat
- Add “latency_stat” under per-path and head debugfs directories to expose latency policy state and statistics.[PATCH 10/10] nvme-multipath: add documentation for latency I/O policy
- Includes documentation for latency I/O multipath policy.LSFMM discussion:
=================
During lsfmm 2026, it was decided to rename this I/O policy from "adaptive" to "latency". This series reflects that rename.The discussion at lsfmm also focused extensively on the latency measurement model, including whether latency should be tracked per-CPU or per-NUMA, and whether separate I/O-size buckets should be maintained for different request sizes.
After detailed discussion and evaluation of throughput results, the consensus was to initially measure I/O completion latency on a per-CPU basis. The available performance data showed that the per-CPU implementation already provides sufficient averaging across CPUs while keeping the design relatively simple.
The use of additional I/O-size buckets did not demonstrate meaningful throughput improvement in the general case and would introduce extra complexity into the fast path and accounting logic. As a result, the consensus was to avoid I/O-size bucketing for now and keep the policy focused on per-CPU latency measurement.
Quoted 5 lines
If future real-world workloads demonstrate a clear benefit from I/O-size-aware latency accounting, the policy can be extended later to support it.
As ususal, feedback and suggestions are most welcome!
Nilay, do we have evidence that round-robin/queue-depth are better for any workload? As a user, I would be very confused with the amount of path selectors I have available and which should I choose.
Quoted 29 lines · Sagi Grimberg
Quoted 20 lines · Nilay Shroff
Hi,
This series introduces a new latency I/O policy for NVMe native multipath. Existing policies such as numa, round-robin, and queue-depth are static and do not adapt to real-time transport performance. The numa selects the path closest to the NUMA node of the current CPU, optimizing memory and path locality, but ignores actual path performance. The round-robin distributes I/O evenly across all paths, providing fairness but not performance awareness. The queue-depth reacts to instantaneous queue occupancy, avoiding heavily loaded paths, but does not account for actual latency, throughput, or link speed.
The new latency policy addresses these gaps selecting paths dynamically based on measured I/O latency for both PCIe and fabrics. Latency is derived by passively sampling I/O completions. Each path is assigned a weight proportional to its latency score, and I/Os are then forwarded accordingly. As condition changes (e.g. latency spikes, bandwidth differences), path weights are updated, automatically steering traffic toward better-performing paths.
Quoted 5 lines
Early results show reduced tail latency under mixed workloads and improved throughput by exploiting higher-speed links more effectively. For example, with NVMf/TCP using two paths (one throttled with ~30 ms delay), fio results with random read/write/rw workloads (direct I/O) showed:
Quoted 5 lines
TBH, I do not know if this measurement represent any real-life scenario. I do think that occasional packet drops are a real-life scenario, and it would be a worthy use-case to optimize for. Can you perhaps measure how the path selectors compare in this case?
Okay so I have now measured another workload where I simulated packet loss/drops which are occasional and compared different path selectors. In this measurement I have a shared NVMe namespace configured which is reachable over two tcp paths. Now to simulate the occasional packet loss/drop I have configured one of the paths to experience occasional packet loss/drop as shown below over the period of 480 seconds:
0 120 240 360 480
|-------------|-------------|-------------|-------------| |<--5% drop-->|<--no drop-->|<--3% drop-->|<--no drop-->|
As shown above for the first 120 seconds the path experiences 5% packet drop, for the next 120 seconds path sees no packet drop and again for subsequent 120 seconds path experiences 3% packet drop and for the rest of the duration (during last 120 seconds) there's no packet drop observed by the path. With this simulation, I ran fio workload for 480 seconds leveraging direct I/O, bs=4k, iodepth=64, numjobs=32 and ioengine=io_uring. Shown below is bw observed running fio test using different I/O policies:
numa round-robin queue-depth latency
(MiB/s) (MiB/s) (MiB/s) (MiB/s)
------- ----------- ----------- ---------
randread: 1288 1202 1464 1642
randwrite: 1456 1493 1779 1944
randrw: R:623 R:594 R:750 R:822
W:623 W:594 W:750 W:822Quoted 11 lines
Quoted 7 lines
numa round-robin queue-depth adaptive ----------- ----------- ----------- --------- READ: 50.0 MiB/s 105 MiB/s 230 MiB/s 350 MiB/s WRITE: 65.9 MiB/s 125 MiB/s 385 MiB/s 446 MiB/s RW: R:30.6 MiB/s R:56.5 MiB/s R:122 MiB/s R:175 MiB/s W:30.7 MiB/s W:56.5 MiB/s W:122 MiB/s W:175 MiB/s
And I'm assuming there are zero downsides for the normal case?
For the normal case where all paths are symmetric I saw queue-depth, latency and round-robin policies yielding nearly same bandwidth. However for numa policy, it depends on CPU/numa locality.
Quoted 63 lines
Quoted 63 lines
This pathcset includes totla 8 patches:
[PATCH 1/10] block: expose blk_stat_{enable,disable}_accounting()
- Makeblk_statAPIs available to block drivers.
- Needed for per-path latency measurement.[PATCH 2/10] block: record I/O request start time for passthru request
- Record I/O start time for I/O passthru requests.
- This is prep patch which allows measuring I/O completion latency for passthru requests.[PATCH 3/10] block: support nesting for blk-mq flag
QUEUE_FLAG_SAME_FORCE
- Support nesting forQUEUE_FLAG_SAME_FORCEas multiple users could toggleQUEUE_FLAG_SAME_FORCE.[PATCH 4/10] nvme-multipath: pass I/O type to
nvme_find_path()
- This is the prep patch which updatesnvme_find_path()signature [PATCH 5/10] nvme-multipath: add latency I/O policy
- Implement path scoring based on latency (EWMA).
- Distribute I/O proportionally to per-path weights.[PATCH 6/10] nvme: add generic debugfs support
- Introduce generic debugfs support for NVMe module[PATCH 7/10] nvme-multipath: add debugfs attribute
latency_ewma_shift
- Adds a debugfs attribute to control ewma shift[PATCH 8/10] nvme-multipath: add debugfs attribute
latency_batch_timeout
- Adds a debugfs attribute to control latency batch window interval[PATCH 9/10] nvme-multipath: add debugfs attribute
latency_stat
- Add “latency_stat” under per-path and head debugfs directories to expose latency policy state and statistics.[PATCH 10/10] nvme-multipath: add documentation for latency I/O policy
- Includes documentation for latency I/O multipath policy.LSFMM discussion:
=================
During lsfmm 2026, it was decided to rename this I/O policy from "adaptive" to "latency". This series reflects that rename.The discussion at lsfmm also focused extensively on the latency measurement model, including whether latency should be tracked per-CPU or per-NUMA, and whether separate I/O-size buckets should be maintained for different request sizes.
After detailed discussion and evaluation of throughput results, the consensus was to initially measure I/O completion latency on a per-CPU basis. The available performance data showed that the per-CPU implementation already provides sufficient averaging across CPUs while keeping the design relatively simple.
The use of additional I/O-size buckets did not demonstrate meaningful throughput improvement in the general case and would introduce extra complexity into the fast path and accounting logic. As a result, the consensus was to avoid I/O-size bucketing for now and keep the policy focused on per-CPU latency measurement.
If future real-world workloads demonstrate a clear benefit from I/O-size-aware latency accounting, the policy can be extended later to support it.
Quoted 5 lines
Quoted 1 line
As ususal, feedback and suggestions are most welcome!
Nilay, do we have evidence that round-robin/queue-depth are better for any workload? As a user, I would be very confused with the amount of path selectors I have available and which should I choose.
From my experiments, when the paths are symmetric, both round-robin and queue-depth (and for that matter latency) exhibit similar behavior, with the workload being distributed roughly equally across the active paths.
When the paths are asymmetric, I found queue-depth to perform better than round-robin. Queue-depth tries to steer I/O toward the less-loaded path based on the number of in-flight I/Os on each path, whereas round-robin continues to distribute I/O evenly across all active paths.
However, queue-depth still has a limitation in this scenario. It uses the number of in-flight I/Os as an indirect indication of path performance, it does not have a direct signal of the actual I/O completion latency. For example, if one path starts experiencing packet loss, I/O completion on that path can become significantly slower. Queue-depth can react to this as the path accumulates more outstanding I/Os, but it can still continue sending I/O to the degraded path as long as its queue depth remains comparable to the healthy path. In other words, it can reduce the amount of I/O sent to the degraded path, but it cannot directly account for how much slower that path has become.
The latency policy uses I/O completion latency as the signal instead. When one path becomes degraded, its observed latency increases and its path score/weight decreases. Consequently, the policy shifts more I/O towards the healthy path. This allows the healthy path to sustain a higher queue depth while the degraded path receives substantially less I/O, rather than trying to maintain a similar queue depth across both paths.
This is also reflected in the packet-loss experiment above. Round-robin continues to distribute I/O across both paths, while queue-depth does a better job by reacting to the increased queue occupancy of the degraded path. The latency policy goes one step further by directly using the increased completion latency as a signal and therefore steers more I/O toward the healthy path, resulting in higher throughput.
So based on the results I have so far, I would characterize the existing policies as follows: round-robin is useful when paths are symmetric and equal distribution is desired. The queue-depth is preferable when paths are asymmetric and queue occupancy provides a useful indication of path load and the latency policy is intended for cases where path performance can vary dynamically and we want the policy to adapt based on actual observed latency.
Thanks,
--Nilay
Sagi Grimberg
maintainerQuoted 182 lines · Nilay Shroff
Quoted 29 lines · Sagi Grimberg
Quoted 20 lines · Nilay Shroff
Hi,
This series introduces a new latency I/O policy for NVMe native multipath. Existing policies such as numa, round-robin, and queue-depth are static and do not adapt to real-time transport performance. The numa selects the path closest to the NUMA node of the current CPU, optimizing memory and path locality, but ignores actual path performance. The round-robin distributes I/O evenly across all paths, providing fairness but not performance awareness. The queue-depth reacts to instantaneous queue occupancy, avoiding heavily loaded paths, but does not account for actual latency, throughput, or link speed.
The new latency policy addresses these gaps selecting paths dynamically based on measured I/O latency for both PCIe and fabrics. Latency is derived by passively sampling I/O completions. Each path is assigned a weight proportional to its latency score, and I/Os are then forwarded accordingly. As condition changes (e.g. latency spikes, bandwidth differences), path weights are updated, automatically steering traffic toward better-performing paths.
Quoted 5 lines
Early results show reduced tail latency under mixed workloads and improved throughput by exploiting higher-speed links more effectively. For example, with NVMf/TCP using two paths (one throttled with ~30 ms delay), fio results with random read/write/rw workloads (direct I/O) showed:
Quoted 5 lines
TBH, I do not know if this measurement represent any real-life scenario. I do think that occasional packet drops are a real-life scenario, and it would be a worthy use-case to optimize for. Can you perhaps measure how the path selectors compare in this case?
Okay so I have now measured another workload where I simulated packet loss/drops which are occasional and compared different path selectors. In this measurement I have a shared NVMe namespace configured which is reachable over two tcp paths. Now to simulate the occasional packet loss/drop I have configured one of the paths to experience occasional packet loss/drop as shown below over the period of 480 seconds:
0 120 240 360 480
|-------------|-------------|-------------|-------------| |<--5% drop-->|<--no drop-->|<--3% drop-->|<--no drop-->|As shown above for the first 120 seconds the path experiences 5% packet drop, for the next 120 seconds path sees no packet drop and again for subsequent 120 seconds path experiences 3% packet drop and for the rest of the duration (during last 120 seconds) there's no packet drop observed by the path. With this simulation, I ran fio workload for 480 seconds leveraging direct I/O, bs=4k, iodepth=64, numjobs=32 and ioengine=io_uring. Shown below is bw observed running fio test using different I/O policies:
numa round-robin queue-depth latency (MiB/s) (MiB/s) (MiB/s) (MiB/s) ------- ----------- ----------- --------- randread: 1288 1202 1464 1642 randwrite: 1456 1493 1779 1944 randrw: R:623 R:594 R:750 R:822 W:623 W:594 W:750 W:822
Quoted 11 lines
Quoted 7 lines
numa round-robin queue-depth adaptive ----------- ----------- ----------- --------- READ: 50.0 MiB/s 105 MiB/s 230 MiB/s 350 MiB/s WRITE: 65.9 MiB/s 125 MiB/s 385 MiB/s 446 MiB/s RW: R:30.6 MiB/s R:56.5 MiB/s R:122 MiB/s R:175 MiB/s W:30.7 MiB/s W:56.5 MiB/s W:122 MiB/s W:175 MiB/s
And I'm assuming there are zero downsides for the normal case?
For the normal case where all paths are symmetric I saw queue-depth, latency and round-robin policies yielding nearly same bandwidth. However for numa policy, it depends on CPU/numa locality.
Quoted 63 lines
Quoted 63 lines
This pathcset includes totla 8 patches:
[PATCH 1/10] block: expose blk_stat_{enable,disable}_accounting()
- Makeblk_statAPIs available to block drivers.
- Needed for per-path latency measurement.[PATCH 2/10] block: record I/O request start time for passthru request
- Record I/O start time for I/O passthru requests.
- This is prep patch which allows measuring I/O completion latency for passthru requests.[PATCH 3/10] block: support nesting for blk-mq flag
QUEUE_FLAG_SAME_FORCE
- Support nesting forQUEUE_FLAG_SAME_FORCEas multiple users could toggleQUEUE_FLAG_SAME_FORCE.[PATCH 4/10] nvme-multipath: pass I/O type to
nvme_find_path()
- This is the prep patch which updatesnvme_find_path()signature [PATCH 5/10] nvme-multipath: add latency I/O policy
- Implement path scoring based on latency (EWMA).
- Distribute I/O proportionally to per-path weights.[PATCH 6/10] nvme: add generic debugfs support
- Introduce generic debugfs support for NVMe module[PATCH 7/10] nvme-multipath: add debugfs attribute
latency_ewma_shift
- Adds a debugfs attribute to control ewma shift[PATCH 8/10] nvme-multipath: add debugfs attribute
latency_batch_timeout
- Adds a debugfs attribute to control latency batch window interval[PATCH 9/10] nvme-multipath: add debugfs attribute
latency_stat
- Add “latency_stat” under per-path and head debugfs directories to expose latency policy state and statistics.[PATCH 10/10] nvme-multipath: add documentation for latency I/O policy
- Includes documentation for latency I/O multipath policy.LSFMM discussion:
=================
During lsfmm 2026, it was decided to rename this I/O policy from "adaptive" to "latency". This series reflects that rename.The discussion at lsfmm also focused extensively on the latency measurement model, including whether latency should be tracked per-CPU or per-NUMA, and whether separate I/O-size buckets should be maintained for different request sizes.
After detailed discussion and evaluation of throughput results, the consensus was to initially measure I/O completion latency on a per-CPU basis. The available performance data showed that the per-CPU implementation already provides sufficient averaging across CPUs while keeping the design relatively simple.
The use of additional I/O-size buckets did not demonstrate meaningful throughput improvement in the general case and would introduce extra complexity into the fast path and accounting logic. As a result, the consensus was to avoid I/O-size bucketing for now and keep the policy focused on per-CPU latency measurement.
If future real-world workloads demonstrate a clear benefit from I/O-size-aware latency accounting, the policy can be extended later to support it.
Quoted 5 lines
Quoted 1 line
As ususal, feedback and suggestions are most welcome!
Nilay, do we have evidence that round-robin/queue-depth are better for any workload? As a user, I would be very confused with the amount of path selectors I have available and which should I choose.
From my experiments, when the paths are symmetric, both round-robin and queue-depth (and for that matter latency) exhibit similar behavior, with the workload being distributed roughly equally across the active paths.
When the paths are asymmetric, I found queue-depth to perform better than round-robin. Queue-depth tries to steer I/O toward the less-loaded path based on the number of in-flight I/Os on each path, whereas round-robin continues to distribute I/O evenly across all active paths.
However, queue-depth still has a limitation in this scenario. It uses the number of in-flight I/Os as an indirect indication of path performance, it does not have a direct signal of the actual I/O completion latency. For example, if one path starts experiencing packet loss, I/O completion on that path can become significantly slower. Queue-depth can react to this as the path accumulates more outstanding I/Os, but it can still continue sending I/O to the degraded path as long as its queue depth remains comparable to the healthy path. In other words, it can reduce the amount of I/O sent to the degraded path, but it cannot directly account for how much slower that path has become.
The latency policy uses I/O completion latency as the signal instead. When one path becomes degraded, its observed latency increases and its path score/weight decreases. Consequently, the policy shifts more I/O towards the healthy path. This allows the healthy path to sustain a higher queue depth while the degraded path receives substantially less I/O, rather than trying to maintain a similar queue depth across both paths.
This is also reflected in the packet-loss experiment above. Round-robin continues to distribute I/O across both paths, while queue-depth does a better job by reacting to the increased queue occupancy of the degraded path. The latency policy goes one step further by directly using the increased completion latency as a signal and therefore steers more I/O toward the healthy path, resulting in higher throughput.
Quoted 7 lines
So based on the results I have so far, I would characterize the existing policies as follows: round-robin is useful when paths are symmetric and equal distribution is desired. The queue-depth is preferable when paths are asymmetric and queue occupancy provides a useful indication of path load and the latency policy is intended for cases where path performance can vary dynamically and we want the policy to adapt based on actual observed latency.
If you have policies A, B, and C and you say:
- In certain conditions policy C > A, B
- In some conditions C = B > A
- In all other cases C = B = A
This means that C should always be used, and A, B should never be used.
Hence I ask, should we really have this as an option? or should we deprecate round-robin/queue-depth and have only numa|latency?
I would like to avoid introducing this as a config knob if it is always behaves better.
What do others think?
Quoted 200 lines · Sagi Grimberg
Quoted 182 lines · Nilay Shroff
Quoted 29 lines · Sagi Grimberg
Quoted 20 lines · Nilay Shroff
Hi,
This series introduces a new latency I/O policy for NVMe native multipath. Existing policies such as numa, round-robin, and queue-depth are static and do not adapt to real-time transport performance. The numa selects the path closest to the NUMA node of the current CPU, optimizing memory and path locality, but ignores actual path performance. The round-robin distributes I/O evenly across all paths, providing fairness but not performance awareness. The queue-depth reacts to instantaneous queue occupancy, avoiding heavily loaded paths, but does not account for actual latency, throughput, or link speed.
The new latency policy addresses these gaps selecting paths dynamically based on measured I/O latency for both PCIe and fabrics. Latency is derived by passively sampling I/O completions. Each path is assigned a weight proportional to its latency score, and I/Os are then forwarded accordingly. As condition changes (e.g. latency spikes, bandwidth differences), path weights are updated, automatically steering traffic toward better-performing paths.
Quoted 5 lines
Early results show reduced tail latency under mixed workloads and improved throughput by exploiting higher-speed links more effectively. For example, with NVMf/TCP using two paths (one throttled with ~30 ms delay), fio results with random read/write/rw workloads (direct I/O) showed:
Quoted 5 lines
TBH, I do not know if this measurement represent any real-life scenario. I do think that occasional packet drops are a real-life scenario, and it would be a worthy use-case to optimize for. Can you perhaps measure how the path selectors compare in this case?
Okay so I have now measured another workload where I simulated packet loss/drops which are occasional and compared different path selectors. In this measurement I have a shared NVMe namespace configured which is reachable over two tcp paths. Now to simulate the occasional packet loss/drop I have configured one of the paths to experience occasional packet loss/drop as shown below over the period of 480 seconds:
0 120 240 360 480
|-------------|-------------|-------------|-------------| |<--5% drop-->|<--no drop-->|<--3% drop-->|<--no drop-->|As shown above for the first 120 seconds the path experiences 5% packet drop, for the next 120 seconds path sees no packet drop and again for subsequent 120 seconds path experiences 3% packet drop and for the rest of the duration (during last 120 seconds) there's no packet drop observed by the path. With this simulation, I ran fio workload for 480 seconds leveraging direct I/O, bs=4k, iodepth=64, numjobs=32 and ioengine=io_uring. Shown below is bw observed running fio test using different I/O policies:
numa round-robin queue-depth latency (MiB/s) (MiB/s) (MiB/s) (MiB/s) ------- ----------- ----------- --------- randread: 1288 1202 1464 1642 randwrite: 1456 1493 1779 1944 randrw: R:623 R:594 R:750 R:822 W:623 W:594 W:750 W:822
Quoted 11 lines
Quoted 7 lines
numa round-robin queue-depth adaptive ----------- ----------- ----------- --------- READ: 50.0 MiB/s 105 MiB/s 230 MiB/s 350 MiB/s WRITE: 65.9 MiB/s 125 MiB/s 385 MiB/s 446 MiB/s RW: R:30.6 MiB/s R:56.5 MiB/s R:122 MiB/s R:175 MiB/s W:30.7 MiB/s W:56.5 MiB/s W:122 MiB/s W:175 MiB/s
And I'm assuming there are zero downsides for the normal case?
For the normal case where all paths are symmetric I saw queue-depth, latency and round-robin policies yielding nearly same bandwidth. However for numa policy, it depends on CPU/numa locality.
Quoted 63 lines
Quoted 63 lines
This pathcset includes totla 8 patches:
[PATCH 1/10] block: expose blk_stat_{enable,disable}_accounting()
- Makeblk_statAPIs available to block drivers.
- Needed for per-path latency measurement.[PATCH 2/10] block: record I/O request start time for passthru request
- Record I/O start time for I/O passthru requests.
- This is prep patch which allows measuring I/O completion latency for passthru requests.[PATCH 3/10] block: support nesting for blk-mq flag
QUEUE_FLAG_SAME_FORCE
- Support nesting forQUEUE_FLAG_SAME_FORCEas multiple users could toggleQUEUE_FLAG_SAME_FORCE.[PATCH 4/10] nvme-multipath: pass I/O type to
nvme_find_path()
- This is the prep patch which updatesnvme_find_path()signature [PATCH 5/10] nvme-multipath: add latency I/O policy
- Implement path scoring based on latency (EWMA).
- Distribute I/O proportionally to per-path weights.[PATCH 6/10] nvme: add generic debugfs support
- Introduce generic debugfs support for NVMe module[PATCH 7/10] nvme-multipath: add debugfs attribute
latency_ewma_shift
- Adds a debugfs attribute to control ewma shift[PATCH 8/10] nvme-multipath: add debugfs attribute
latency_batch_timeout
- Adds a debugfs attribute to control latency batch window interval[PATCH 9/10] nvme-multipath: add debugfs attribute
latency_stat
- Add “latency_stat” under per-path and head debugfs directories to expose latency policy state and statistics.[PATCH 10/10] nvme-multipath: add documentation for latency I/O policy
- Includes documentation for latency I/O multipath policy.LSFMM discussion:
=================
During lsfmm 2026, it was decided to rename this I/O policy from "adaptive" to "latency". This series reflects that rename.The discussion at lsfmm also focused extensively on the latency measurement model, including whether latency should be tracked per-CPU or per-NUMA, and whether separate I/O-size buckets should be maintained for different request sizes.
After detailed discussion and evaluation of throughput results, the consensus was to initially measure I/O completion latency on a per-CPU basis. The available performance data showed that the per-CPU implementation already provides sufficient averaging across CPUs while keeping the design relatively simple.
The use of additional I/O-size buckets did not demonstrate meaningful throughput improvement in the general case and would introduce extra complexity into the fast path and accounting logic. As a result, the consensus was to avoid I/O-size bucketing for now and keep the policy focused on per-CPU latency measurement.
If future real-world workloads demonstrate a clear benefit from I/O-size-aware latency accounting, the policy can be extended later to support it.
Quoted 5 lines
Quoted 1 line
As ususal, feedback and suggestions are most welcome!
Nilay, do we have evidence that round-robin/queue-depth are better for any workload? As a user, I would be very confused with the amount of path selectors I have available and which should I choose.
From my experiments, when the paths are symmetric, both round-robin and queue-depth (and for that matter latency) exhibit similar behavior, with the workload being distributed roughly equally across the active paths.
When the paths are asymmetric, I found queue-depth to perform better than round-robin. Queue-depth tries to steer I/O toward the less-loaded path based on the number of in-flight I/Os on each path, whereas round-robin continues to distribute I/O evenly across all active paths.
However, queue-depth still has a limitation in this scenario. It uses the number of in-flight I/Os as an indirect indication of path performance, it does not have a direct signal of the actual I/O completion latency. For example, if one path starts experiencing packet loss, I/O completion on that path can become significantly slower. Queue-depth can react to this as the path accumulates more outstanding I/Os, but it can still continue sending I/O to the degraded path as long as its queue depth remains comparable to the healthy path. In other words, it can reduce the amount of I/O sent to the degraded path, but it cannot directly account for how much slower that path has become.
The latency policy uses I/O completion latency as the signal instead. When one path becomes degraded, its observed latency increases and its path score/weight decreases. Consequently, the policy shifts more I/O towards the healthy path. This allows the healthy path to sustain a higher queue depth while the degraded path receives substantially less I/O, rather than trying to maintain a similar queue depth across both paths.
This is also reflected in the packet-loss experiment above. Round-robin continues to distribute I/O across both paths, while queue-depth does a better job by reacting to the increased queue occupancy of the degraded path. The latency policy goes one step further by directly using the increased completion latency as a signal and therefore steers more I/O toward the healthy path, resulting in higher throughput.
Quoted 7 lines
So based on the results I have so far, I would characterize the existing policies as follows: round-robin is useful when paths are symmetric and equal distribution is desired. The queue-depth is preferable when paths are asymmetric and queue occupancy provides a useful indication of path load and the latency policy is intended for cases where path performance can vary dynamically and we want the policy to adapt based on actual observed latency.
If you have policies A, B, and C and you say:
- In certain conditions policy C > A, B
- In some conditions C = B > A
- In all other cases C = B = AThis means that C should always be used, and A, B should never be used.
Quoted 7 lines
Hence I ask, should we really have this as an option? or should we deprecate round-robin/queue-depth and have only numa|latency?
I would like to avoid introducing this as a config knob if it is always behaves better.
What do others think?
Good point, however my view is that we may not want to immediately deprecate or remove queue-depth/round-robin. Based on my testing so far, across a variety of workloads, including intermittent packet loss, the latency policy has performed better than queue-depth and round-robin. However, I would like to see it evaluated against a broader set of workloads and scenarios, including cases that I may not have considered/known.
Once we have broader evidence and establish that latency consistently performs at least as well as queue-depth/round-robin without any significant downside, then I think it would be reasonable to consider deprecating those policies and keeping only numa|latency.
Until then, I would prefer to keep queue-depth and round-robin as baseline policies against which we can compare the latency policy.
But let's wait and see what others think.
Thanks,
--Nilay
Quoted 3 lines · Nilay Shroff
Quoted 2 lines · Sagi Grimberg
[ .. ]>> I would like to avoid introducing this as a config knob if it is
Quoted 10 lines
Quoted 3 lines
always behaves better.
What do others think?
Good point, however my view is that we may not want to immediately deprecate or remove queue-depth/round-robin. Based on my testing so far, across a variety of workloads, including intermittent packet
loss, the latency policy has performed better than queue-depth and
round-robin. However, I would like to see it evaluated against a
broader set of workloads and scenarios, including cases that I > may not have considered/known.
Quoted 7 lines
Once we have broader evidence and establish that latency consistently performs at least as well as queue-depth/round-robin without any significant downside, then I think it would be reasonable to consider
deprecating those policies and keeping only numa|latency.>
Until then, I would prefer to keep queue-depth and round-robin as baseline policies against which we can compare the latency policy.But let's wait and see what others think.
I think that we can deprecate round-robin now that we have queue-depth and latency.
But I also think that we need both, queue-depth _and_ latency as both latch on different properties, and there are use-cases where one might have workloads which prefer one or the other.
Having a config option to select the default (and possibly disable some I/O schedulers) would be completely fine with me, too.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
I think Nilay has done some great work with the latency scheduler, and we have it in our queue for evaluation. But we have not done so yet, so I would appreciate if queue-depth would continue to be available until the latency scheduler has been more fully established as the prime choice.
As to round-robin, I agree it would be great to see it disappear from all of our customers installations, but I have found it useful in evaluating multipath failover behavior because it is easier to reason about and construct the test cases. So, I would appreciate if it would remain for debugging only. I would certainly be open to a mechanism that made it a little bit hard to set round-robin as the queue policy, though.
Sincerely,
Randy Jennings
Quoted 44 lines · Hannes Reinecke
Quoted 3 lines · Nilay Shroff
Quoted 2 lines · Sagi Grimberg
[ .. ]>> I would like to avoid introducing this as a config knob if it is
Quoted 10 lines
Quoted 3 lines
always behaves better.
What do others think?
Good point, however my view is that we may not want to immediately deprecate or remove queue-depth/round-robin. Based on my testing so far, across a variety of workloads, including intermittent packet loss, the latency policy has performed better than queue-depth and round-robin. However, I would like to see it evaluated against a broader set of workloads and scenarios, including cases that I > may
not have considered/known.
Quoted 10 lines
Once we have broader evidence and establish that latency consistently performs at least as well as queue-depth/round-robin without any significant downside, then I think it would be reasonable to consider deprecating those policies and keeping only numa|latency.> Until then, I would prefer to keep queue-depth and round-robin as baseline policies against which we can compare the latency policy.
But let's wait and see what others think.
I think that we can deprecate round-robin now that we have queue-depth and latency.
But I also think that we need both, queue-depth _and_ latency as both latch on different properties, and there are use-cases where one might have workloads which prefer one or the other.Having a config option to select the default (and possibly disable some I/O schedulers) would be completely fine with me, too.
Cheers,
Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
[email protected] +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Sagi Grimberg
maintainerQuoted 7 lines
I think Nilay has done some great work with the latency scheduler, and we have it in our queue for evaluation. But we have not done so yet, so I would appreciate if queue-depth would continue to be available until the latency scheduler has been more fully established as the prime choice.
Quoted 7 lines
As to round-robin, I agree it would be great to see it disappear from all of our customers installations, but I have found it useful in evaluating multipath failover behavior because it is easier to reason about and construct the test cases. So, I would appreciate if it would remain for debugging only. I would certainly be open to a mechanism that made it a little bit hard to set round-robin as the queue policy, though.
Randy, would it be possible for you folks to evaluate it as well? I will ask some folks on our side as well. I'm concerned that Nilay is the only one who has evaluated it in his lab.
Overall, I am perfectly fine with the approach this path selector is taking, but I worry that we are growing path-selectors that are slightly different in subtle ways... TBH, I still don't understand what use-cases this path selector is more appropriate than queue-depth, or vice-versa.
Quoted 3 lines
Randy, would it be possible for you folks to evaluate it as well? I will ask some folks on our side as well. I'm concerned that Nilay is the only one who has evaluated it in his lab.
Yes, absolutely, but there is going to be a time delay due to wrapping up some current priorities. I'd say we'd get to it in more than 2 months but less than 6. That is the best promise I can give right now. (Yes, it is behind some work for stretched subsystems and CCR/CQT, which are higher priorities for me.)
Quoted 4 lines
Overall, I am perfectly fine with the approach this path selector is taking, but I worry
that we are growing path-selectors that are slightly different in subtle ways...
Agreed.
Quoted 2 lines
TBH, I still don't understand what use-cases this path selector is more appropriate than queue-depth, or vice-versa.
I had this question, too, and we talked about it in Zagreb. My understanding from that discussion is that latency can handle full queues better than queue-depth. When you have shallow queues, you still get information to make a path decision on.
Sincerely,
Randy Jennings