nvme-multipath: introduce adaptive I/O policy
Subsystems (2)
cover nvme-multipath: introduce adaptive I/O policy
Hi,
This series introduces a new adaptive I/O policy for NVMe native multipath. Existing policies—numa, round-robin, and queue-depth—are static and do not adapt to real-time transport performance. The numa selects the path closest to the NUMA node of the current CPU, optimizing memory and path locality, but ignores actual path performance. The round-robin distributes I/O evenly across all paths, providing fairness but not performance awareness. The queue-depth reacts to instantaneous queue occupancy, avoiding heavily loaded paths, but does not account for actual latency, throughput, or link speed.
The new adaptive policy addresses these gaps selecting paths dynamically based on measured I/O latency and, for fabrics, the negotiated link speed. Latency is derived by passively sampling I/O completions. Link speed is queried from the adapter and factored into path scoring. Each path is assigned a weight proportional to its score, and I/Os are then forwarded accordingly. As conditions change (e.g. latency spikes, bandwidth differences), path weights are updated, automatically steering traffic toward better-performing paths.
Early results show reduced tail latency under mixed workloads and improved throughput by exploiting higher-speed links more effectively. For example, with NVMf/TCP using two paths (one throttled with ~30 ms delay), fio results with random read/write/rw workloads (direct I/O) showed:
numa round-robin queue-depth adaptive
----------- ----------- ----------- ---------
READ: 50.0 MiB/s 105 MiB/s 230 MiB/s 350 MiB/s
WRITE: 65.9 MiB/s 125 MiB/s 385 MiB/s 446 MiB/s
RW: R:30.6 MiB/s R:56.5 MiB/s R:122 MiB/s R:175 MiB/s
W:30.7 MiB/s W:56.5 MiB/s W:122 MiB/s W:175 MiB/sThis pathcset includes totla 5 patches:
[PATCH 1/5] block: expose blk_stat_{enable,disable}_accounting()
- Make blk_stat APIs available to block drivers.
- Needed for per-path latency measurement in adaptive policy.
[PATCH 2/5] nvme-multipath: add adaptive I/O policy
- Implement path scoring based on latency (EWMA).
- Distribute I/O proportionally to per-path weights.
[PATCH 3/5] nvme-multipath: add sysfs attribute for adaptive policy
- Introduce "adp_stat" under nvme path block device.
- Provide observability of latency, weight, and selection stats.
[PATCH 4/5] nvme-tcp: export NIC link speed
- Retrieve negotiated link speed (Mbps) from the adapter.
- Expose via sysfs for visibility/debugging.
[PATCH 5/5] nvme-multipath: factor link speed into path scoring
- Adjust adaptive path weights using link speed as a multiplier.
- Favor higher bandwidth links while still considering latency.
Currently, link speed reporting is implemented only for TCP NICs. Support for Fibre Channel adapters will follow in a future patch.
As ususal, feedback and suggestions are most welcome!
Thanks!
Nilay Shroff (5):
block: expose blk_stat_{enable,disable}_accounting() to drivers
nvme-multipath: add support for adaptive I/O policy
nvme-multipath: add sysfs attribute for adaptive I/O policy
nvmf-tcp: add support for retrieving adapter link speed
nvme-multipath: factor fabric link speed into path score
block/blk-stat.h | 4 - drivers/nvme/host/core.c | 10 +- drivers/nvme/host/ioctl.c | 7 +- drivers/nvme/host/multipath.c | 441 +++++++++++++++++++++++++++++++++- drivers/nvme/host/nvme.h | 38 ++- drivers/nvme/host/pr.c | 6 +- drivers/nvme/host/sysfs.c | 12 +- drivers/nvme/host/tcp.c | 66 +++++ include/linux/blk-mq.h | 4 + 9 files changed, 562 insertions(+), 26 deletions(-)
Signature · 1 line
2.51.0
RFC PATCH 1/5 block: expose blk_stat_{enable,disable}_accounting() to drivers
The functions blk_stat_enable_accounting() and blk_stat_disable_accounting() are currently exported, but their prototypes are only defined in a private header. Move these prototypes into a common header so that block drivers can directly use these APIs.
--- block/blk-stat.h | 4 ---- include/linux/blk-mq.h | 4 ++++ 2 files changed, 4 insertions(+), 4 deletions(-)
block/blk-stat.h +0 -4
@@ -67,10 +67,6 @@ void blk_free_queue_stats(struct blk_queue_stats *);void blk_stat_add (struct request *rq,u64 now); -/* record time/size info in request but not add a callback */ -void blk_stat_enable_accounting (struct request_queue *q); -void blk_stat_disable_accounting (struct request_queue *q); -/** * blk_stat_alloc_callback() - Allocate a block statistics callback. * @timer_fn: Timer callback function.
include/linux/blk-mq.h +4 -0
@@ -732,6 +732,10 @@ int blk_rq_poll(struct request *rq, struct io_comp_batch *iob,bool blk_mq_queue_inflight (struct request_queue *q); +/* record time/size info in request but not add a callback */ +void blk_stat_enable_accounting (struct request_queue *q); +void blk_stat_disable_accounting (struct request_queue *q); +enum {/* return when out of requests */ BLK_MQ_REQ_NOWAIT = (__force blk_mq_req_flags_t)(1 <<0 ),
Signature · 1 line
2.51.0
RFC PATCH 2/5 nvme-multipath: add support for adaptive I/O policy
In reply to Nilay ShroffThis commit introduces a new I/O policy named "adaptive". Users can configure it by writing "adaptive" to "/sys/class/nvme-subsystem/nvme- subsystemX/iopolicy"
The adaptive policy dynamically distributes I/O based on measured completion latency. The main idea is to calculate latency for each path, derive a weight, and then proportionally forward I/O according to those weights.
To ensure scalability, path latency is measured per-CPU. Each CPU maintains its own statistics, and I/O forwarding uses these per-CPU values. Every ~15 seconds, a simple average latency of per-CPU batched samples are computed and fed into an Exponentially Weighted Moving Average (EWMA):
avg_latency = div_u64(batch, batch_count);new_ewma_latency = (prev_ewma_latency * (WEIGHT-1) + avg_latency)/WEIGHT
With WEIGHT = 8, this assigns 7/8 (~87.5%) weight to the previous latency value and 1/8 (~12.5%) to the most recent latency. This smoothing reduces jitter, adapts quickly to changing conditions, avoids storing historical samples, and works well for both low and high I/O rates. Path weights are then derived from the smoothed (EWMA) latency as follows (example with two paths A and B):
path_A_score = NSEC_PER_SEC / path_A_ewma_latency path_B_score = NSEC_PER_SEC / path_B_ewma_latency total_score = path_A_score + path_B_score
path_A_weight = (path_A_score * 100) / total_score path_B_weight = (path_B_score * 100) / total_score
where:
- path_X_ewma_latency is the smoothed latency of a path in ns
- NSEC_PER_SEC is used as a scaling factor since valid latencies are < 1 second
- weights are normalized to a 0–100 scale across all paths.
Path credits are refilled based on this weight, with one credit consumed per I/O. When all credits are consumed, the credits are refilled again based on the current weight. This ensures that I/O is distributed across paths proportionally to their calculated weight.
--- drivers/nvme/host/core.c | 10 +- drivers/nvme/host/ioctl.c | 7 +- drivers/nvme/host/multipath.c | 353 ++++++++++++++++++++++++++++++++-- drivers/nvme/host/nvme.h | 33 +++- drivers/nvme/host/pr.c | 6 +- drivers/nvme/host/sysfs.c | 2 +- 6 files changed, 389 insertions(+), 22 deletions(-)
drivers/nvme/host/core.c +8 -2
@@ -689,6 +689,7 @@ static void nvme_free_ns(struct kref *kref) { struct nvme_ns *ns = container_of(kref, struct nvme_ns, kref); +nvme_mpath_free_stat (ns);put_disk (ns->disk);nvme_put_ns_head (ns->head);nvme_put_ctrl (ns->ctrl);
@@ -4132,6 +4133,9 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)if (nvme_init_ns_head (ns, info)) goto out_cleanup_disk; +if (nvme_mpath_alloc_stat (ns)) + goto out_unlink_ns; +/* * If multipathing is enabled, the device name for all disks and not * just those that represent shared namespaces needs to be based on the
@@ -4156,7 +4160,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info) }if (nvme_update_ns_info (ns, info)) - goto out_unlink_ns; + goto out_free_ns_stat;mutex_lock (&ctrl->namespaces_lock);/*
@@ -4165,7 +4169,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)*/ if (test_bit (NVME_CTRL_FROZEN , &ctrl->flags)) {mutex_unlock (&ctrl->namespaces_lock); - goto out_unlink_ns; + goto out_free_ns_stat; }nvme_ns_add_to_ctrl_list (ns);mutex_unlock (&ctrl->namespaces_lock);
@@ -4196,6 +4200,8 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)list_del_rcu (&ns->list);mutex_unlock (&ctrl->namespaces_lock);synchronize_srcu (&ctrl->srcu); +out_free_ns_stat: +nvme_mpath_free_stat (ns); out_unlink_ns:mutex_lock (&ctrl->subsys->lock);list_del_rcu (&ns->siblings);
drivers/nvme/host/ioctl.c +4 -3
@@ -716,7 +716,7 @@ int nvme_ns_head_ioctl(struct block_device *bdev, blk_mode_t mode, flags |=NVME_IOCTL_PARTITION ; srcu_idx =srcu_read_lock (&head->srcu); - ns =nvme_find_path (head); + ns =nvme_find_path (head, open_for_write ?WRITE :READ );if (!ns) goto out_unlock;
@@ -747,7 +747,7 @@ long nvme_ns_head_chr_ioctl(struct file *file, unsigned int cmd,int srcu_idx, ret = -EWOULDBLOCK ; srcu_idx =srcu_read_lock (&head->srcu); - ns =nvme_find_path (head); + ns =nvme_find_path (head, open_for_write ?WRITE :READ );if (!ns) goto out_unlock;
@@ -767,7 +767,8 @@ int nvme_ns_head_chr_uring_cmd(struct io_uring_cmd *ioucmd,struct cdev *cdev =file_inode (ioucmd->file)->i_cdev; struct nvme_ns_head *head = container_of(cdev, struct nvme_ns_head, cdev);int srcu_idx =srcu_read_lock (&head->srcu); -struct nvme_ns *ns =nvme_find_path (head); +struct nvme_ns *ns =nvme_find_path (head, + ioucmd->file->f_mode &FMODE_WRITE ?WRITE :READ );int ret = -EINVAL ;if (ns)
drivers/nvme/host/multipath.c +340 -13
460 lines not on this page · raw message
drivers/nvme/host/nvme.h +32 -1
@@ -421,6 +421,7 @@ enum nvme_iopolicy {NVME_IOPOLICY_NUMA ,NVME_IOPOLICY_RR ,NVME_IOPOLICY_QD , + NVME_IOPOLICY_ADAPTIVE, }; struct nvme_subsystem {
@@ -459,6 +460,19 @@ struct nvme_ns_ids {u8 csi; }; +struct nvme_path_stat { +u64 nr_samples;/* total num of samples processed */ +u64 nr_ignored;/* num. of samples ignored */ +u64 slat_ns;/* smoothed (ewma) latency in nanoseconds */ +u64 score;/* score used for weight calculation */ +u64 last_weight_ts;/* timestamp of the last time calculation */ +u64 sel;/* num of times this path is selcted for I/O */ +u64 batch;/* accumulated latency sum for current window */ +u32 batch_count;/* num of samples accumulated in current window */ +u32 weight;/* path weight */ +u32 credit;/* path credit for I/O forwarding */ +}; +/* * Anchor structure for namespaces. There is one for each namespace in a * NVMe subsystem that any of our controllers can see, and the namespace
@@ -534,6 +548,7 @@ struct nvme_ns {#ifdef CONFIG_NVME_MULTIPATH enum nvme_ana_state ana_state;u32 ana_grpid; +struct nvme_path_stat __percpu *cpu_stat;#endif struct list_head siblings;struct kref kref;
@@ -545,6 +560,7 @@ struct nvme_ns {#define NVME_NS_FORCE_RO 3#define NVME_NS_READY 4#define NVME_NS_SYSFS_ATTR_LINK 5 +#define NVME_NS_PATH_STAT 6struct cdev cdev;struct device cdev_device;
@@ -949,7 +965,7 @@ extern const struct attribute_group *nvme_dev_attr_groups[];extern const struct block_device_operations nvme_bdev_ops;void nvme_delete_ctrl_sync (struct nvme_ctrl *ctrl); -struct nvme_ns *nvme_find_path (struct nvme_ns_head *head); +struct nvme_ns *nvme_find_path (struct nvme_ns_head *head,unsigned int rw);#ifdef CONFIG_NVME_MULTIPATH static inline bool nvme_ctrl_use_ana (struct nvme_ctrl *ctrl) {
@@ -977,6 +993,7 @@ void nvme_mpath_revalidate_paths(struct nvme_ns *ns);void nvme_mpath_clear_ctrl_paths (struct nvme_ctrl *ctrl);void nvme_mpath_remove_disk (struct nvme_ns_head *head);void nvme_mpath_start_request (struct request *rq); +int nvme_mpath_alloc_stat (struct nvme_ns *ns);void nvme_mpath_end_request (struct request *rq);static inline void nvme_trace_bio_complete (struct request *req)
@@ -1005,6 +1022,13 @@ static inline bool nvme_mpath_queue_if_no_path(struct nvme_ns_head *head)return true; return false; } +static inline void nvme_mpath_free_stat (struct nvme_ns *ns) +{ +if (!ns->head->disk) +return ; + +free_percpu (ns->cpu_stat); +}#else #define multipath falsestatic inline bool nvme_ctrl_use_ana (struct nvme_ctrl *ctrl)
@@ -1096,6 +1120,13 @@ static inline bool nvme_mpath_queue_if_no_path(struct nvme_ns_head *head) {return false; } +static inline int nvme_mpath_alloc_stat (struct nvme_ns *ns) +{ +return 0 ; +} +static inline void nvme_mpath_free_stat (struct nvme_ns *ns) +{ +}#endif /* CONFIG_NVME_MULTIPATH */ int nvme_ns_get_unique_id(struct nvme_ns *ns, u8 id[16],
drivers/nvme/host/pr.c +4 -2
@@ -53,10 +53,12 @@ static int nvme_send_ns_head_pr_command(struct block_device *bdev, struct nvme_command *c, void *data, unsigned int data_len) { struct nvme_ns_head *head = bdev->bd_disk->private_data; -int srcu_idx =srcu_read_lock (&head->srcu); -struct nvme_ns *ns =nvme_find_path (head); +int srcu_idx; +struct nvme_ns *ns;int ret = -EWOULDBLOCK ; + srcu_idx =srcu_read_lock (&head->srcu); + ns =nvme_find_path (head,nvme_is_write (c) ?WRITE :READ );if (ns) { c->common.nsid =cpu_to_le32 (ns->head->ns_id); ret =nvme_submit_sync_cmd (ns->queue, c, data, data_len);
drivers/nvme/host/sysfs.c +1 -1
@@ -194,7 +194,7 @@ static int ns_head_update_nuse(struct nvme_ns_head *head)return 0 ; srcu_idx =srcu_read_lock (&head->srcu); - ns =nvme_find_path (head); + ns =nvme_find_path (head,READ );if (!ns) goto out_unlock;
Signature · 1 line
2.51.0
Re: [RFC PATCH 2/5] nvme-multipath: add support for adaptive I/O policy
Quoted 224 lines · Nilay Shroff
This commit introduces a new I/O policy named "adaptive". Users can configure it by writing "adaptive" to "
/sys/class/nvme-subsystem/nvme-subsystemX/iopolicy"The adaptive policy dynamically distributes I/O based on measured completion latency. The main idea is to calculate latency for each path, derive a weight, and then proportionally forward I/O according to those weights.
To ensure scalability, path latency is measured per-CPU. Each CPU maintains its own statistics, and I/O forwarding uses these per-CPU values. Every ~15 seconds, a simple average latency of per-CPU batched samples are computed and fed into an Exponentially Weighted Moving Average (EWMA):
avg_latency=div_u64(batch, batch_count);new_ewma_latency= (prev_ewma_latency* (WEIGHT-1) + avg_latency)/WEIGHTWith WEIGHT = 8, this assigns 7/8 (~87.5%) weight to the previous latency value and 1/8 (~12.5%) to the most recent latency. This smoothing reduces jitter, adapts quickly to changing conditions, avoids storing historical samples, and works well for both low and high I/O rates. Path weights are then derived from the smoothed (EWMA) latency as follows (example with two paths A and B):
path_A_score=NSEC_PER_SEC/path_A_ewma_latencypath_B_score=NSEC_PER_SEC/path_B_ewma_latencytotal_score=path_A_score+path_B_score
path_A_weight= (path_A_score* 100) /total_scorepath_B_weight= (path_B_score* 100) /total_scorewhere:
-path_X_ewma_latencyis the smoothed latency of a path in ns
-NSEC_PER_SECis used as a scaling factor since valid latencies are < 1 second
- weights are normalized to a 0–100 scale across all paths.Path credits are refilled based on this weight, with one credit consumed per I/O. When all credits are consumed, the credits are refilled again based on the current weight. This ensures that I/O is distributed across paths proportionally to their calculated weight.
Signed-off-by Nilay Shroff <[email protected]>--- drivers/nvme/host/core.c | 10 +- drivers/nvme/host/ioctl.c | 7 +- drivers/nvme/host/multipath.c | 353 ++++++++++++++++++++++++++++++++-- drivers/nvme/host/nvme.h | 33 +++- drivers/nvme/host/pr.c | 6 +- drivers/nvme/host/sysfs.c | 2 +- 6 files changed, 389 insertions(+), 22 deletions(-)drivers/nvme/host/core.c +8
-2@@ -689,6 +689,7 @@ static void nvme_free_ns(struct kref *kref) { struct nvme_ns *ns = container_of(kref, struct nvme_ns, kref); +nvme_mpath_free_stat (ns);put_disk (ns->disk);nvme_put_ns_head (ns->head);nvme_put_ctrl (ns->ctrl);@@ -4132,6 +4133,9 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)if (nvme_init_ns_head (ns, info)) goto out_cleanup_disk; +if (nvme_mpath_alloc_stat (ns)) + goto out_unlink_ns; +/* * If multipathing is enabled, the device name for all disks and not * just those that represent shared namespaces needs to be based on the @@ -4156,7 +4160,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info) }if (nvme_update_ns_info (ns, info)) - goto out_unlink_ns; + goto out_free_ns_stat;mutex_lock (&ctrl->namespaces_lock);/* @@ -4165,7 +4169,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)*/ if (test_bit (NVME_CTRL_FROZEN , &ctrl->flags)) {mutex_unlock (&ctrl->namespaces_lock); - goto out_unlink_ns; + goto out_free_ns_stat; }nvme_ns_add_to_ctrl_list (ns);mutex_unlock (&ctrl->namespaces_lock);@@ -4196,6 +4200,8 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)list_del_rcu (&ns->list);mutex_unlock (&ctrl->namespaces_lock);synchronize_srcu (&ctrl->srcu); +out_free_ns_stat: +nvme_mpath_free_stat (ns); out_unlink_ns:mutex_lock (&ctrl->subsys->lock);list_del_rcu (&ns->siblings);drivers/nvme/host/ioctl.c +4
-3@@ -716,7 +716,7 @@ int nvme_ns_head_ioctl(struct block_device *bdev, blk_mode_t mode, flags |=NVME_IOCTL_PARTITION ; srcu_idx =srcu_read_lock (&head->srcu); - ns =nvme_find_path (head); + ns =nvme_find_path (head, open_for_write ?WRITE :READ );if (!ns) goto out_unlock;@@ -747,7 +747,7 @@ long nvme_ns_head_chr_ioctl(struct file *file, unsigned int cmd,int srcu_idx, ret = -EWOULDBLOCK ; srcu_idx =srcu_read_lock (&head->srcu); - ns =nvme_find_path (head); + ns =nvme_find_path (head, open_for_write ?WRITE :READ );if (!ns) goto out_unlock;@@ -767,7 +767,8 @@ int nvme_ns_head_chr_uring_cmd(struct io_uring_cmd *ioucmd,struct cdev *cdev =file_inode (ioucmd->file)->i_cdev; struct nvme_ns_head *head = container_of(cdev, struct nvme_ns_head, cdev);int srcu_idx =srcu_read_lock (&head->srcu); -struct nvme_ns *ns =nvme_find_path (head); +struct nvme_ns *ns =nvme_find_path (head, + ioucmd->file->f_mode &FMODE_WRITE ?WRITE :READ );int ret = -EINVAL ;if (ns)drivers/nvme/host/multipath.c +58
-3@@ -6,6 +6,8 @@#include <linux/backing-dev.h> #include <linux/moduleparam.h> #include <linux/vmalloc.h> +#include <linux/blk-mq.h> +#include <linux/math64.h> #include <trace/events/block.h> #include "nvme.h" @@ -66,9 +68,10 @@ MODULE_PARM_DESC(multipath_always_on, "create multipath node always except for private namespace with non-unique nsid; note that this also implicitly enables native multipath support");static const char *nvme_iopolicy_names[] = { - [NVME_IOPOLICY_NUMA ] ="numa" , - [NVME_IOPOLICY_RR ] ="round-robin" , - [NVME_IOPOLICY_QD ] ="queue-depth" , + [NVME_IOPOLICY_NUMA ] ="numa" , + [NVME_IOPOLICY_RR ] ="round-robin" , + [NVME_IOPOLICY_QD ] ="queue-depth" , + [NVME_IOPOLICY_ADAPTIVE ] ="adaptive" , };static int iopolicy =NVME_IOPOLICY_NUMA ;@@ -83,6 +86,8 @@ static int nvme_set_iopolicy(const char *val, const struct kernel_param *kp) iopolicy =NVME_IOPOLICY_RR ; else if (!strncmp(val, "queue-depth", 11)) iopolicy =NVME_IOPOLICY_QD ; + else if (!strncmp(val, "adaptive", 8)) + iopolicy =NVME_IOPOLICY_ADAPTIVE ; else return -EINVAL;@@ -196,6 +201,190 @@ void nvme_mpath_start_request(struct request *rq) }EXPORT_SYMBOL_GPL (nvme_mpath_start_request); +int nvme_mpath_alloc_stat (struct nvme_ns *ns) +{ +gfp_t gfp =GFP_KERNEL | __GFP_ZERO; + +if (!ns->head->disk) +return 0 ; + + ns->cpu_stat =__alloc_percpu_gfp ( +2 *sizeof (struct nvme_path_stat ), + __alignof__(struct nvme_path_stat ), + gfp); +if (!ns->cpu_stat) +return -ENOMEM ; + +return 0 ; +} + +#define NVME_EWMA_SHIFT 3 +static inline u64 ewma_update (u64 old,u64 new) +{ +return (old * ((1 <<NVME_EWMA_SHIFT ) -1 ) + new) >>NVME_EWMA_SHIFT ; +} + +static void nvme_mpath_add_sample (struct request *rq,struct nvme_ns *ns) +{ +int cpu, srcu_idx; +unsigned int rw; +struct nvme_path_stat *stat; +struct nvme_ns *cur_ns; +u32 weight; +u64 now, latency, avg_lat_ns; +u64 total_score =0 ; +struct nvme_ns_head *head = ns->head; + +if (list_is_singular (&head->list)) +return ; + + now =ktime_get_ns (); + latency = now >= rq->io_start_time_ns ? now - rq->io_start_time_ns :0 ; +if (!latency) +return ; + +/* +* As completion code path is serialized(i.e. no same completion queue +* update code could run simultaneously on multiple cpu) we can safely +* access per cpu nvme path stat here from another cpu (in case the +* completion cpu is different from submission cpu). +* The only field which could be accessed simultaneously here is the +* path ->weight which may be accessed by this function as well as I/O +* submission path during path selection logic and we protect ->weight
drivers/nvme/host/multipath.c void nvme_mpath_start_request(struct request *rq) · line 263
Quoted 10 lines
drivers/nvme/host/multipath.c +10
-0+* using READ_ONCE/WRITE_ONCE. Yes this may not be 100% accurate but +* we also don't need to be so accurate here as the path credit would +* be anyways refilled, based on path weight, once path consumes all +* its credits. And we limit path weight/credit max up to 100. Please +* also refer nvme_adaptive_path(). +*/ + cpu =blk_mq_rq_cpu (rq); + rw =rq_data_dir (rq); + stat = &per_cpu_ptr (ns->cpu_stat, cpu)[rw]; +
This is tad awkward for setups where #CPUs > #paths.
Quoted 48 lines
drivers/nvme/host/multipath.c +48
-0+/* +* If latency > ~1s then ignore this sample to prevent EWMA from being +* skewed by pathological outliers (multi-second waits, controller +* timeouts etc.). This keeps path scores representative of normal +* performance and avoids instability from rare spikes. If such high +* latency is real, ANA state reporting or keep-alive error counters +* will mark the path unhealthy and remove it from the head node list, +* so we safely skip such sample here. +*/ +if (unlikely (latency >NSEC_PER_SEC )) { + stat->nr_ignored++; +return ; + } + +/* +* Accumulate latency samples and increment the batch count for each +* ~15 second interval. When the interval expires, compute the simple +* average latency over that window, then update the smoothed (EWMA) +* latency. The path weight is recalculated based on this smoothed +* latency. +*/ + stat->batch += latency; + stat->batch_count++; + stat->nr_samples++; + +if (now > stat->last_weight_ts && + (now - stat->last_weight_ts) >=15 *NSEC_PER_SEC ) { + + stat->last_weight_ts = now; + +/* +* Find simple average latency for the last epoch (~15 sec +* interval). +*/ + avg_lat_ns =div_u64 (stat->batch, stat->batch_count); + +/* +* Calculate smooth/EWMA (Exponentially Weighted Moving Average) +* latency. EWMA is preferred over simple average latency +* because it smooths naturally, reduces jitter from sudden +* spikes, and adapts faster to changing conditions. It also +* avoids storing historical samples, and works well for both +* slow and fast I/O rates. +* Formula: +* slat_ns = (prev_slat_ns * (WEIGHT - 1) + (latency)) / WEIGHT +* With WEIGHT = 8, this assigns 7/8 (~87.5 %) weight to the +* existing latency and 1/8 (~12.5%) weight to the new latency. +*/
drivers/nvme/host/multipath.c void nvme_mpath_start_request(struct request *rq) · line 321
Quoted 10 lines
drivers/nvme/host/multipath.c +10
-0+if (unlikely (!stat->slat_ns)) + stat->slat_ns = avg_lat_ns; +else + stat->slat_ns =ewma_update (stat->slat_ns, avg_lat_ns); + + stat->batch = stat->batch_count =0 ; + + srcu_idx =srcu_read_lock (&head->srcu); +list_for_each_entry_srcu (cur_ns, &head->list, siblings, +srcu_read_lock_held (&head->srcu)) {
And this is even more awkward as we need to iterate over all paths (during completion!).
Do we really need to do this?
What would happen if we just measure the latency on the local CPU and do away with this loop?
We would have less samples, true, but we would even be able to not only differentiate between distinct path latency but also between different CPU latencies; I would think this being a bonus for multi-socket machines.
_And_ we wouldn't need to worry about path failures, which is bound to expose some race conditions if we need to iterate paths at the same time than path failures are being handled.
But nevertheless: great job!
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Re: [RFC PATCH 2/5] nvme-multipath: add support for adaptive I/O policy
Quoted 276 lines · Hannes Reinecke
Quoted 210 lines · Nilay Shroff
This commit introduces a new I/O policy named "adaptive". Users can configure it by writing "adaptive" to "
/sys/class/nvme-subsystem/nvme-subsystemX/iopolicy"The adaptive policy dynamically distributes I/O based on measured completion latency. The main idea is to calculate latency for each path, derive a weight, and then proportionally forward I/O according to those weights.
To ensure scalability, path latency is measured per-CPU. Each CPU maintains its own statistics, and I/O forwarding uses these per-CPU values. Every ~15 seconds, a simple average latency of per-CPU batched samples are computed and fed into an Exponentially Weighted Moving Average (EWMA):
avg_latency=div_u64(batch, batch_count);new_ewma_latency= (prev_ewma_latency* (WEIGHT-1) + avg_latency)/WEIGHTWith WEIGHT = 8, this assigns 7/8 (~87.5%) weight to the previous latency value and 1/8 (~12.5%) to the most recent latency. This smoothing reduces jitter, adapts quickly to changing conditions, avoids storing historical samples, and works well for both low and high I/O rates. Path weights are then derived from the smoothed (EWMA) latency as follows (example with two paths A and B):
path_A_score=NSEC_PER_SEC/path_A_ewma_latencypath_B_score=NSEC_PER_SEC/path_B_ewma_latencytotal_score =path_A_score+path_B_score
path_A_weight= (path_A_score* 100) /total_scorepath_B_weight= (path_B_score* 100) /total_scorewhere:
-path_X_ewma_latencyis the smoothed latency of a path in ns
-NSEC_PER_SECis used as a scaling factor since valid latencies are < 1 second
- weights are normalized to a 0–100 scale across all paths.Path credits are refilled based on this weight, with one credit consumed per I/O. When all credits are consumed, the credits are refilled again based on the current weight. This ensures that I/O is distributed across paths proportionally to their calculated weight.
Signed-off-by Nilay Shroff <[email protected]>---
drivers/nvme/host/core.c | 10 +- drivers/nvme/host/ioctl.c | 7 +-drivers/nvme/host/multipath.c| 353 ++++++++++++++++++++++++++++++++-- drivers/nvme/host/nvme.h | 33 +++- drivers/nvme/host/pr.c | 6 +- drivers/nvme/host/sysfs.c | 2 +-6 files changed, 389 insertions(+), 22 deletions(-)drivers/nvme/host/core.c +6
-2@@ -689,6 +689,7 @@ static void nvme_free_ns(struct kref *kref) { struct nvme_ns *ns = container_of(kref, struct nvme_ns, kref); + nvme_mpath_free_stat(ns); put_disk(ns->disk); nvme_put_ns_head(ns->head); nvme_put_ctrl(ns->ctrl);@@ -4132,6 +4133,9 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info) if (nvme_init_ns_head(ns, info)) goto out_cleanup_disk; + if (nvme_mpath_alloc_stat(ns)) + goto out_unlink_ns; + /** If multipathing is enabled, the device name for all disks and not * just those that represent shared namespaces needs to be based on the @@ -4156,7 +4160,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info) } if (nvme_update_ns_info(ns, info)) - goto out_unlink_ns; + goto out_free_ns_stat; mutex_lock(&ctrl->namespaces_lock); /*@@ -4165,7 +4169,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)*/ if (test_bit(NVME_CTRL_FROZEN, &ctrl->flags)) { mutex_unlock(&ctrl->namespaces_lock); - goto out_unlink_ns; + goto out_free_ns_stat; } nvme_ns_add_to_ctrl_list(ns); mutex_unlock(&ctrl->namespaces_lock);@@ -4196,6 +4200,8 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info) list_del_rcu(&ns->list); mutex_unlock(&ctrl->namespaces_lock); synchronize_srcu(&ctrl->srcu); +out_free_ns_stat: + nvme_mpath_free_stat(ns);
out_unlink_ns:
mutex_lock(&ctrl->subsys->lock);
list_del_rcu(&ns->siblings);drivers/nvme/host/ioctl.c +1
-1@@ -716,7 +716,7 @@ int nvme_ns_head_ioctl(struct block_device *bdev, blk_mode_t mode, flags |= NVME_IOCTL_PARTITION; srcu_idx = srcu_read_lock(&head->srcu); - ns = nvme_find_path(head); + ns = nvme_find_path(head, open_for_write ? WRITE : READ); if (!ns) goto out_unlock; @@ -747,7 +747,7 @@ long nvme_ns_head_chr_ioctl(struct file *file, unsigned int cmd, int srcu_idx, ret = -EWOULDBLOCK;
srcu_idx=srcu_read_lock(&head->srcu);
- ns =nvme_find_path(head);
+ ns =nvme_find_path(head, open_for_write ? WRITE : READ);
if (!ns)
gotoout_unlock;
@@ -767,7 +767,8 @@ int nvme_ns_head_chr_uring_cmd(structio_uring_cmd*ioucmd, struct cdev *cdev = file_inode(ioucmd->file)->i_cdev;
structnvme_ns_head*head =container_of(cdev, struct nvme_ns_head, cdev); intsrcu_idx=srcu_read_lock(&head->srcu);
- structnvme_ns*ns =nvme_find_path(head);
+ structnvme_ns*ns = nvme_find_path(head,
+ioucmd->file->f_mode&FMODE_WRITE? WRITE : READ);
int ret =-EINVAL;
if (ns)drivers/nvme/host/multipath.c +2
-0@@ -6,6 +6,8 @@ #include <linux/backing-dev.h> #include <linux/moduleparam.h> #include <linux/vmalloc.h> +#include <linux/blk-mq.h> +#include <linux/math64.h> #include <trace/events/block.h> #include "nvme.h" @@ -66,9 +68,10 @@ MODULE_PARM_DESC(multipath_always_on,"create multipath node always except for private namespace with non-unique nsid; note that this also implicitly enables native multipath support"); static const char *nvme_iopolicy_names[] = {
- [NVME_IOPOLICY_NUMA] = "numa",
- [NVME_IOPOLICY_RR] = "round-robin",
- [NVME_IOPOLICY_QD] = "queue-depth",
+ [NVME_IOPOLICY_NUMA] = "numa",
+ [NVME_IOPOLICY_RR] = "round-robin",
+ [NVME_IOPOLICY_QD] = "queue-depth",
+ [NVME_IOPOLICY_ADAPTIVE] = "adaptive",
};
static int iopolicy =NVME_IOPOLICY_NUMA;drivers/nvme/host/multipath.c +2
-0@@ -83,6 +86,8 @@ static int nvme_set_iopolicy(const char *val, const struct kernel_param *kp) iopolicy = NVME_IOPOLICY_RR; else if (!strncmp(val, "queue-depth", 11)) iopolicy = NVME_IOPOLICY_QD; + else if (!strncmp(val, "adaptive", 8)) + iopolicy = NVME_IOPOLICY_ADAPTIVE;else
return-EINVAL;
@@ -196,6 +201,190 @@ voidnvme_mpath_start_request(struct request *rq)}
EXPORT_SYMBOL_GPL(nvme_mpath_start_request);
+intnvme_mpath_alloc_stat(struct nvme_ns *ns)drivers/nvme/host/multipath.c +49
-0+{ + gfp_t gfp = GFP_KERNEL | __GFP_ZERO; + + if (!ns->head->disk) + return 0; + + ns->cpu_stat = __alloc_percpu_gfp( + 2 * sizeof(struct nvme_path_stat), + __alignof__(struct nvme_path_stat), + gfp); + if (!ns->cpu_stat) + return -ENOMEM; + + return 0; +} + +#define NVME_EWMA_SHIFT 3 +static inline u64 ewma_update (u64 old,u64 new) +{ + return (old * ((1 << NVME_EWMA_SHIFT) - 1) + new) >> NVME_EWMA_SHIFT; +} + +static void nvme_mpath_add_sample (struct request *rq,struct nvme_ns *ns) +{ + int cpu, srcu_idx; + unsigned int rw; + struct nvme_path_stat *stat; + struct nvme_ns *cur_ns; + u32 weight; + u64 now, latency, avg_lat_ns; + u64 total_score = 0; + struct nvme_ns_head *head = ns->head; + + if (list_is_singular(&head->list)) + return; + + now = ktime_get_ns(); + latency = now >= rq->io_start_time_ns ? now - rq->io_start_time_ns : 0; + if (!latency) + return; + + /* +* As completion code path is serialized(i.e. no same completion queue +* update code could run simultaneously on multiple cpu) we can safely +* access per cpu nvme path stat here from another cpu (in case the +* completion cpu is different from submission cpu). +* The only field which could be accessed simultaneously here is the +* path ->weight which may be accessed by this function as well as I/O +* submission path during path selection logic and we protect ->weight drivers/nvme/host/multipath.c void nvme_mpath_start_request(struct request *rq) · line 263
Quoted 10 lines
drivers/nvme/host/multipath.c +10
-0+* using READ_ONCE/WRITE_ONCE. Yes this may not be 100% accurate but +* we also don't need to be so accurate here as the path credit would +* be anyways refilled, based on path weight, once path consumes all +* its credits. And we limit path weight/credit max up to 100. Please +* also refer nvme_adaptive_path(). +*/ + cpu = blk_mq_rq_cpu(rq); + rw = rq_data_dir(rq); + stat = &per_cpu_ptr(ns->cpu_stat, cpu)[rw]; +This is tad awkward for setups where #CPUs > #paths.
Quoted 53 lines
+ /*
+ * If latency > ~1s then ignore this sample to prevent EWMA from being + * skewed by pathological outliers (multi-second waits, controller + * timeouts etc.). This keeps path scores representative of normal + * performance and avoids instability from rare spikes. If such high + * latency is real, ANA state reporting or keep-alive error counters + * will mark the path unhealthy and remove it from the head node list, + * so we safely skip such sample here.
+ */
+ if (unlikely(latency > NSEC_PER_SEC)) {
+ stat->nr_ignored++;
+ return;
+ }drivers/nvme/host/multipath.c +40
-0+ + /* + * Accumulate latency samples and increment the batch count for each + * ~15 second interval. When the interval expires, compute the simple + * average latency over that window, then update the smoothed (EWMA) + * latency. The path weight is recalculated based on this smoothed + * latency. + */ + stat->batch += latency; + stat->batch_count++; + stat->nr_samples++; + + if (now > stat->last_weight_ts && + (now - stat->last_weight_ts) >= 15 * NSEC_PER_SEC) { + + stat->last_weight_ts = now; + + /* + * Find simple average latency for the last epoch (~15 sec + * interval). + */ + avg_lat_ns = div_u64(stat->batch, stat->batch_count); + + /* + * Calculate smooth/EWMA (Exponentially Weighted Moving Average) + * latency. EWMA is preferred over simple average latency + * because it smooths naturally, reduces jitter from sudden + * spikes, and adapts faster to changing conditions. It also + * avoids storing historical samples, and works well for both + * slow and fast I/O rates. + * Formula: + * slat_ns = (prev_slat_ns * (WEIGHT - 1) + (latency)) / WEIGHT + * With WEIGHT = 8, this assigns 7/8 (~87.5 %) weight to the + * existing latency and 1/8 (~12.5%) weight to the new latency. + */ + if (unlikely(!stat->slat_ns)) + stat->slat_ns = avg_lat_ns; + else + stat->slat_ns = ewma_update(stat->slat_ns, avg_lat_ns); +
Quoted 9 lines
Quoted 5 lines
+
stat->batch=stat->batch_count= 0;
+
+srcu_idx=srcu_read_lock(&head->srcu);
+list_for_each_entry_srcu(cur_ns, &head->list, siblings, + srcu_read_lock_held(&head->srcu)){And this is even more awkward as we need to iterate over all paths (during completion!).
Hmm yes, but we only iterate once every ~15 seconds per CPU, so the overhead is minimal. Typically we don’t have a large number of paths to deal with: enterprise SSDs usually expose at most two controllers, and even in fabrics setups the path count is usually limited to around 4–6. So the loop should run quite fast.
Also, looping in itself isn’t unusual — for example, the queue-depth I/O policy already iterates over all paths in the submission path to check queue depth before dispatching each I/O. That said, if looping in the completion path is still a concern, we could consider moving this into a dedicated worker thread instead. What do you think?
Quoted 8 lines
Do we really need to do this?
What would happen if we just measure the latency on the local CPU and do away with this loop?
We would have less samples, true, but we would even be able to not only differentiate between distinct path latency but also between different CPU latencies; I would think this being a bonus for multi-socket machines.
The idea is to keep per-cpu view consistent for each path. As we know, in NVMe/fabrics multipath, submission and completion CPUs don’t necessarily match (depends on the host’s irq/core mapping). And so if we were to measure the latency/EWMA locally per-cpu then the per-CPU accumulator might be biased towards the completion CPU, not the submission CPU. For instance, if submission is on CPU A but completion lands on CPU B, then CPU A’s weights never reflect it's I/O experience — they’ll be skewed by how interrupts get steered.
So on a multi socket/NUMA systems, depending on topology, calculating local per-cpu ewma/latency may or may not line up. For example:
- If we have #cpu <= #vectors supported by NVMe disk then typically
we have 1:1 mapping between submission and completion queues and hence all completions for a queue are steered to the same CPU that also submits, then per-CPU stats are accurate.
- But when #CPUs > #vectors, completions may be centralized or spread differently. In that case, the per-CPU latency view can be distorted — e.g., CPU A may submit, but CPU B takes completions, so CPU A’s weights never reflect its own I/O behavior.
Quoted 4 lines
_And_ we wouldn't need to worry about path failures, which is bound to expose some race conditions if we need to iterate paths at the same time than path failures are being handled.
Yes agreed we may have some race here and so the path score/weight may be skewed when that happens but then that'd be auto-corrected in the next epoc (after ~15 sec) when we re-calculate the path weight/score again, isn't it?
Quoted 1 line
But nevertheless: great job!
Thank you :)
--Nilay
Re: [RFC PATCH 2/5] nvme-multipath: add support for adaptive I/O policy
Quoted 4 lines · Nilay Shroff
Quoted 1 line · Hannes Reinecke
On 9/21/25 13:12, Nilay Shroff wrote:
[ .. ]>>> + srcu_idx = srcu_read_lock(&head->srcu);
Quoted 10 lines
Quoted 6 lines
Quoted 2 lines
+
list_for_each_entry_srcu(cur_ns, &head->list, siblings, + srcu_read_lock_held(&head->srcu)){And this is even more awkward as we need to iterate over all paths (during completion!).
Hmm yes, but we only iterate once every ~15 seconds per CPU, so the overhead is minimal. Typically we don’t have a large number of paths to deal with: enterprise SSDs usually expose at most two controllers, and even in fabrics setups the path count is usually limited to around 4–6. So the loop should run quite fast.
Hmm. Not from my experience. There is at least one implementation from a rather substantial array vendor exposing up to low hundreds of queues.
Quoted 5 lines
Also, looping in itself isn’t unusual — for example, the queue-depth I/O policy already iterates over all paths in the submission path to check queue depth before dispatching each I/O. That said, if looping in the completion path is still a concern, we could consider moving this into a dedicated worker thread instead. What do you think?
Not sure if that's a good idea; either the worker thread runs asynchronous to the completion and then we have to deal with reliably adding up numbers, or we're running synchronous and lose performance. Still think that _not_ iterating and just adding up single-cpu latencies might be worthwhile.
Quoted 8 lines
Quoted 8 lines
Do we really need to do this?
What would happen if we just measure the latency on the local CPU and do away with this loop?
We would have less samples, true, but we would even be able to not only differentiate between distinct path latency but also between different CPU latencies; I would think this being a bonus for multi-socket machines.
Quoted 8 lines
The idea is to keep per-cpu view consistent for each path. As we know, in NVMe/fabrics multipath, submission and completion CPUs don’t necessarily match (depends on the host’s irq/core mapping). And so if we were to measure the latency/EWMA locally per-cpu then the per-CPU accumulator might be biased towards the completion CPU, not the submission CPU. For instance, if submission is on CPU A but completion lands on CPU B, then CPU A’s weights never reflect it's I/O experience — they’ll be skewed by how interrupts get steered.
True. Problem is that for the #CPUs > #queues we're setting up a cpu affinity group, and interrupts are directed to one of the CPU in that group. I had hoped that the blk-mq code would raise a softirq in that case and call .end_request on the cpu registered in the request itself. Probably need to be evaluated.
Quoted 11 lines
So on a multi socket/NUMA systems, depending on topology, calculating local per-cpu ewma/latency may or may not line up. For example:
- If we have #cpu <= #vectors supported by NVMe disk then typically
we have 1:1 mapping between submission and completion queues and hence all completions for a queue are steered to the same CPU that also submits, then per-CPU stats are accurate.- But when #CPUs > #vectors, completions may be centralized or spread differently. In that case, the per-CPU latency view can be distorted — e.g., CPU A may submit, but CPU B takes completions, so CPU A’s weights never reflect its own I/O behavior.
See above. We might check if blk-mq doesn't cover for this case already. Thing is, I actually _do_ want to measure per-CPU latency. On a multi-socket system it really does matter whether an I/O is run from a CPU on the socket attached to the PCI device, or from an off-socket CPU. If we are calculating just the per-path latency we completely miss that (as blk-mq will spread out across _all_ cpus), but if we are measuring a per-cpu latency we will end up with a differential matrix where cpus with the lowest latency will be preferred.
So if we have a system with two sockets and two PCI HBAs, each connected to a different socket, using per-path latency will be spreading out I/Os across all cpus. Using per-cpu latency will direct I/Os to the cpus with the lowest latency, preferring the local cpus.
Quoted 8 lines
Quoted 4 lines
_And_ we wouldn't need to worry about path failures, which is bound to expose some race conditions if we need to iterate paths at the same time than path failures are being handled.
Yes agreed we may have some race here and so the path score/weight may be skewed when that happens but then that'd be auto-corrected in the next epoc (after ~15 sec) when we re-calculate the path weight/score again, isn't it?
Let's see. I still would want to check if we can't do per-cpu statistics, as that would automatically avoid any races :-)
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Re: [RFC PATCH 2/5] nvme-multipath: add support for adaptive I/O policy
Quoted 12 lines · Hannes Reinecke
Quoted 4 lines · Nilay Shroff
Quoted 1 line · Hannes Reinecke
On 9/21/25 13:12, Nilay Shroff wrote:
[ .. ]>>> +
srcu_idx=srcu_read_lock(&head->srcu);Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
+
list_for_each_entry_srcu(cur_ns, &head->list, siblings, + srcu_read_lock_held(&head->srcu)){And this is even more awkward as we need to iterate over all paths (during completion!).
Quoted 8 lines
Quoted 4 lines
Hmm yes, but we only iterate once every ~15 seconds per CPU, so the overhead is minimal. Typically we don’t have a large number of paths to deal with: enterprise SSDs usually expose at most two controllers, and even in fabrics setups the path count is usually limited to around 4–6. So the loop should run quite fast.
Hmm. Not from my experience. There is at least one implementation from a rather substantial array vendor exposing up to low hundreds of queues.
Sorry but I think we're discussing two different thing here? I'm referring to the nvme paths (i.e. nvmeXcYnz) and probably you're referring to the DMA queues or hw I/O queues. If that's true then yes some enterprise SSD may expose low hundreds of queues, for instance, on my SSD I see 128 queues being supported:
# nvme get-feature -f 0x7 /dev/nvme0 -H get-feature:0x07 (Number of Queues), Current value:0x007f007f Number of IO Completion Queues Allocated (NCQA): 128 Number of IO Submission Queues Allocated (NSQA): 128
But when it comes to the nvme paths, that should be limited (mostly single digit). So my point was iterating those paths (and that's what the above code implements) shouldn't be expensive. But please correct me if my assumption was wrong.
Quoted 6 lines
Quoted 5 lines
Also, looping in itself isn’t unusual — for example, the queue-depth I/O policy already iterates over all paths in the submission path to check queue depth before dispatching each I/O. That said, if looping in the completion path is still a concern, we could consider moving this into a dedicated worker thread instead. What do you think?
Quoted 6 lines
Not sure if that's a good idea; either the worker thread runs asynchronous to the completion and then we have to deal with reliably adding up numbers, or we're running synchronous and lose performance. Still think that _not_ iterating and just adding up single-cpu latencies might be worthwhile.
Yes worker thread should run asynchronously with completion and we can deal with it in one of the two ways:
- If accuracy is critical, we could use atomic counters (e.g. atomic_cmpxchg) to update batch counts and calculate path weights asynchronously in the worker.
- If a small skew is acceptable, snapshot-and-reset is sufficient. Since EWMA smoothing absorbs micro-errors, missing 1–2 samples in a 15s window at real I/O rates is negligible. Many kernel subsystems (block stats, net flow stats, etc.) already rely on snapshot-and-reset without atomic, exactly for this reason.
IMO, for the adaptive policy, we don’t need atomic_xchg() unless we’re chasing perfect stats. Snapshot-and-reset per-CPU accumulators is good enough, because EWMA smoothing and weight recalculation will easily absorb micro-errors. So what do you prefer if in case we choose worker thread?
Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Do we really need to do this?
What would happen if we just measure the latency on the local CPU and do away with this loop?
We would have less samples, true, but we would even be able to not only differentiate between distinct path latency but also between different CPU latencies; I would think this being a bonus for multi-socket machines.Quoted 8 lines
The idea is to keep per-cpu view consistent for each path. As we know, in NVMe/fabrics multipath, submission and completion CPUs don’t necessarily match (depends on the host’s irq/core mapping). And so if we were to measure the latency/EWMA locally per-cpu then the per-CPU accumulator might be biased towards the completion CPU, not the submission CPU. For instance, if submission is on CPU A but completion lands on CPU B, then CPU A’s weights never reflect it's I/O experience — they’ll be skewed by how interrupts get steered.
Quoted 6 lines
True. Problem is that for the #CPUs > #queues we're setting up a cpu affinity group, and interrupts are directed to one of the CPU in that group. I had hoped that the blk-mq code would raise a softirq in that case and call .end_request on the cpu registered in the request itself. Probably need to be evaluated.
I already evaluated this. If completion ends up on a cpu which is different from the submission cpu then yes block layer code may raise softirq to steer the completion on the same cpu as submission. But this is not guaranteed always. For instance if poll queues are used for submission then submission and completion on the same cpu is not guaranteed. Similarly, if threaded interrupt is configured then this is not guaranteed. In fact blk-mq also supports a flag named QUEUE_FLAG_SAME_FORCE which forces completion on the same CPU as the submission CPU. But again this flags also doesn't work always if we have poll-queue or thread interrupt is configured. Please refer blk_mq_complete_request_remote().
Quoted 11 lines
Quoted 11 lines
So on a multi socket/NUMA systems, depending on topology, calculating local per-cpu ewma/latency may or may not line up. For example:
- If we have #cpu <= #vectors supported by NVMe disk then typically
we have 1:1 mapping between submission and completion queues and hence all completions for a queue are steered to the same CPU that also submits, then per-CPU stats are accurate.- But when #CPUs > #vectors, completions may be centralized or spread differently. In that case, the per-CPU latency view can be distorted — e.g., CPU A may submit, but CPU B takes completions, so CPU A’s weights never reflect its own I/O behavior.
Quoted 15 lines
See above. We might check if blk-mq doesn't cover for this case already. Thing is, I actually _do_ want to measure per-CPU latency. On a multi-socket system it really does matter whether an I/O is run from a CPU on the socket attached to the PCI device, or from an off-socket CPU. If we are calculating just the per-path latency we completely miss that (as blk-mq will spread out across _all_ cpus), but if we are measuring a per-cpu latency we will end up with a differential matrix where cpus with the lowest latency will be preferred.
So if we have a system with two sockets and two PCI HBAs, each connected to a different socket, using per-path latency will be spreading out I/Os across all cpus. Using per-cpu latency will direct I/Os to the cpus with the lowest latency, preferring the local cpus.
Yes, agreed and so in this proposed patch we measure per-cpu latency and _NOT_ per-path latency. Measuring the per-cpu latency and using it for forwarding I/O is ideal for multi socket/NUMA systems.
Quoted 11 lines
Quoted 8 lines
Quoted 4 lines
_And_ we wouldn't need to worry about path failures, which is bound to expose some race conditions if we need to iterate paths at the same time than path failures are being handled.
Yes agreed we may have some race here and so the path score/weight may be skewed when that happens but then that'd be auto-corrected in the next epoc (after ~15 sec) when we re-calculate the path weight/score again, isn't it?
Let's see. I still would want to check if we can't do per-cpu statistics, as that would automatically avoid any races :-)
As I mentioned above, there are certain use cases (like poll-queues, threaded interrupts etc.) where submission and completion could happen on different CPUs, so we have to account for such cases and cross over CPU while accumulating the batch latency.
Thanks,
--Nilay
RFC PATCH 3/5 nvme-multipath: add sysfs attribute for adaptive I/O policy
In reply to Nilay ShroffThis commit introduces a new sysfs attribute, "adp_stat", under the nvme path block device. This attribute provides visibility into the state of the adaptive I/O policy and is intended to aid debugging and observability. We now also calculate the per-path aggregated smoothed (EWMA) latency for reporting it under this new attribute.
The attribute reports per-path aggregated statistics, including I/O weight, smoothed (EWMA) latency, selection count, processed samples, and ignored samples.
--- drivers/nvme/host/multipath.c | 77 ++++++++++++++++++++++++++++++++++- drivers/nvme/host/nvme.h | 2 + drivers/nvme/host/sysfs.c | 5 +++ 3 files changed, 82 insertions(+), 2 deletions(-)
drivers/nvme/host/multipath.c +75 -2
@@ -224,6 +224,22 @@ static inline u64 ewma_update(u64 old, u64 new)return (old * ((1 <<NVME_EWMA_SHIFT ) -1 ) + new) >>NVME_EWMA_SHIFT ; } +static inline void path_ewma_update (atomic64_t *ptr,u64 new) +{ +u64 old, slat; + +/* +* Since multiple CPUs may update the per-path smoothed (EWMA) +* latency concurrently, we use an atomic compare-and-exchange +* loop to safely apply the update without losing intermediate +* changes. +*/ +do { + old =atomic64_read (ptr); + slat =ewma_update (old, new); + }while (atomic64_cmpxchg (ptr, old, slat) != old); +} +static void nvme_mpath_add_sample (struct request *rq,struct nvme_ns *ns) {int cpu, srcu_idx;
@@ -308,11 +324,18 @@ static void nvme_mpath_add_sample(struct request *rq, struct nvme_ns *ns)* slat_ns = (prev_slat_ns * (WEIGHT - 1) + (latency)) / WEIGHT * With WEIGHT = 8, this assigns 7/8 (~87.5 %) weight to the * existing latency and 1/8 (~12.5%) weight to the new latency. +* +* Please note that we also calculate here the smooth/EWMA +* latency per path which is then used for exposing aggregated +* per-path latency using sysfs for observability/debugging. */ -if (unlikely (!stat->slat_ns)) +if (unlikely (!stat->slat_ns)) { stat->slat_ns = avg_lat_ns; -else +atomic64_set (&ns->slat_ns[rw], avg_lat_ns); + }else { stat->slat_ns =ewma_update (stat->slat_ns, avg_lat_ns); +path_ewma_update (&ns->slat_ns[rw], avg_lat_ns); + } stat->batch = stat->batch_count =0 ;
@@ -437,6 +460,7 @@ static void nvme_mpath_reset_current_stat(struct nvme_ns *ns) stat =per_cpu_ptr (ns->cpu_stat, cpu);memset (stat,0 ,2 *sizeof (struct nvme_path_stat )); } +memset (ns->slat_ns,0 ,sizeof (ns->slat_ns)); }static bool nvme_mpath_set_current_adaptive_path (struct nvme_ns *ns)
@@ -1450,6 +1474,55 @@ static ssize_t numa_nodes_show(struct device *dev, struct device_attribute *attr }DEVICE_ATTR_RO (numa_nodes); +static void adp_stat_read_all (struct nvme_ns *ns,struct nvme_path_stat *batch) +{ +int i, cpu; +int ncpu[2 ] = {0 }; +struct nvme_path_stat *stat; + +for_each_online_cpu (cpu) { + stat =per_cpu_ptr (ns->cpu_stat, cpu); + +for (i =0 ; i <2 ; i++) { +if (stat[i].weight) { + batch[i].weight += stat[i].weight; + batch[i].sel += stat[i].sel; + batch[i].nr_samples += stat[i].nr_samples; + batch[i].nr_ignored += stat[i].nr_ignored; + ncpu[i]++; + } + } + } + +for (i =0 ; i <2 ; i++) { +if (!ncpu[i]) +continue ; + batch[i].weight =DIV_U64_ROUND_CLOSEST (batch[i].weight, ncpu[i]); + } +} + +static ssize_t adp_stat_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +struct nvme_path_stat stat[2 ] = {0 }; +struct nvme_ns *ns =nvme_get_ns_from_dev (dev); + +adp_stat_read_all (ns, stat); +return sysfs_emit (buf,"%u %llu %llu %llu %llu %u %llu %llu %llu %llu\n" , + stat[READ ].weight, +atomic64_read (&ns->slat_ns[READ ]), + stat[READ ].sel, + stat[READ ].nr_samples, + stat[READ ].nr_ignored, + stat[WRITE ].weight, +atomic64_read (&ns->slat_ns[WRITE ]), + stat[WRITE ].sel, + stat[WRITE ].nr_samples, + stat[WRITE ].nr_ignored); + +} +DEVICE_ATTR_RO (adp_stat); +static ssize_t delayed_removal_secs_show (struct device *dev,struct device_attribute *attr,char *buf) {
drivers/nvme/host/nvme.h +2 -0
@@ -548,6 +548,7 @@ struct nvme_ns {#ifdef CONFIG_NVME_MULTIPATH enum nvme_ana_state ana_state;u32 ana_grpid; +atomic64_t slat_ns[2 ];/* path smoothed (EWMA) latency in nanosconds */ struct nvme_path_stat __percpu *cpu_stat;#endif struct list_head siblings;
@@ -1009,6 +1010,7 @@ extern struct device_attribute dev_attr_ana_grpid;extern struct device_attribute dev_attr_ana_state;extern struct device_attribute dev_attr_queue_depth;extern struct device_attribute dev_attr_numa_nodes; +extern struct device_attribute dev_attr_adp_stat;extern struct device_attribute dev_attr_delayed_removal_secs;extern struct device_attribute subsys_attr_iopolicy;
drivers/nvme/host/sysfs.c +5 -0
@@ -260,6 +260,7 @@ static struct attribute *nvme_ns_attrs[] = { &dev_attr_ana_state.attr, &dev_attr_queue_depth.attr, &dev_attr_numa_nodes.attr, + &dev_attr_adp_stat.attr, &dev_attr_delayed_removal_secs.attr, #endif &dev_attr_io_passthru_err_log_enabled.attr,
@@ -303,6 +304,10 @@ static umode_t nvme_ns_attrs_are_visible(struct kobject *kobj,if (!nvme_disk_is_ns_head (disk)) return 0; } +if (a == &dev_attr_adp_stat.attr) { +if (nvme_disk_is_ns_head (dev_to_disk (dev))) +return 0 ; + }#endif return a->mode; }
Signature · 1 line
2.51.0
Re: [RFC PATCH 3/5] nvme-multipath: add sysfs attribute for adaptive I/O policy
Quoted 10 lines · Nilay Shroff
This commit introduces a new sysfs attribute, "
adp_stat", under the nvme path block device. This attribute provides visibility into the state of the adaptive I/O policy and is intended to aid debugging and observability. We now also calculate the per-path aggregated smoothed (EWMA) latency for reporting it under this new attribute.The attribute reports per-path aggregated statistics, including I/O weight, smoothed (EWMA) latency, selection count, processed samples, and ignored samples.
Quoted 7 lines
Signed-off-by Nilay Shroff <[email protected]>--- drivers/nvme/host/multipath.c | 77 ++++++++++++++++++++++++++++++++++- drivers/nvme/host/nvme.h | 2 + drivers/nvme/host/sysfs.c | 5 +++ 3 files changed, 82 insertions(+), 2 deletions(-)
Wouldn't this be better off if situated in the debugfs directly? Exposing the stats is not really crucial to operations, and mainly for debugging purposes only.
Exposing the weight from the EWMA algorithm, OTOH, really does influence the performance, and might be an idea to expose.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Re: [RFC PATCH 3/5] nvme-multipath: add sysfs attribute for adaptive I/O policy
Quoted 18 lines · Hannes Reinecke
Quoted 10 lines · Nilay Shroff
This commit introduces a new sysfs attribute, "
adp_stat", under the nvme path block device. This attribute provides visibility into the state of the adaptive I/O policy and is intended to aid debugging and observability. We now also calculate the per-path aggregated smoothed (EWMA) latency for reporting it under this new attribute.The attribute reports per-path aggregated statistics, including I/O weight, smoothed (EWMA) latency, selection count, processed samples, and ignored samples.
Quoted 7 lines
Signed-off-by Nilay Shroff <[email protected]>---
drivers/nvme/host/multipath.c| 77 ++++++++++++++++++++++++++++++++++- drivers/nvme/host/nvme.h | 2 + drivers/nvme/host/sysfs.c | 5 +++3 files changed, 82 insertions(+), 2 deletions(-)
Quoted 7 lines
Wouldn't this be better off if situated in the debugfs directly? Exposing the stats is not really crucial to operations, and mainly for debugging purposes only.
Exposing the weight from the EWMA algorithm, OTOH, really does influence the performance, and might be an idea to expose.
Yes I think exposing this under debugfs is good idea. Maybe we could also expose per-cpu stat under debugfs.
Thanks,
--Nilay
RFC PATCH 4/5 nvmf-tcp: add support for retrieving adapter link speed
In reply to Nilay ShroffAdd support for retrieving the negotiated NIC link speed (in Mbps). This value can be factored into path scoring for the adaptive I/O policy. For visibility and debugging, a new sysfs attribute "speed" is also added under the NVMe path block device.
--- drivers/nvme/host/multipath.c | 11 ++++++ drivers/nvme/host/nvme.h | 3 ++ drivers/nvme/host/sysfs.c | 5 +++ drivers/nvme/host/tcp.c | 66 +++++++++++++++++++++++++++++++++++ 4 files changed, 85 insertions(+)
drivers/nvme/host/multipath.c +11 -0
@@ -548,6 +548,8 @@ void nvme_mpath_revalidate_paths(struct nvme_ns *ns)clear_bit (NVME_NS_READY , &ns->flags);nvme_mpath_reset_current_stat (ns); +if (ns->ctrl->ops->get_link_speed) + ns->speed = ns->ctrl->ops->get_link_speed (ns->ctrl); }srcu_read_unlock (&head->srcu, srcu_idx);
@@ -1566,6 +1568,15 @@ static ssize_t delayed_removal_secs_store(struct device *dev,DEVICE_ATTR_RW (delayed_removal_secs); +static ssize_t speed_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct nvme_ns *ns =nvme_get_ns_from_dev (dev); + +return sysfs_emit (buf,"%u\n" , ns->speed); +} +DEVICE_ATTR_RO (speed); +static int nvme_lookup_ana_group_desc (struct nvme_ctrl *ctrl,struct nvme_ana_group_desc *desc,void *data) {
drivers/nvme/host/nvme.h +3 -0
@@ -548,6 +548,7 @@ struct nvme_ns {#ifdef CONFIG_NVME_MULTIPATH enum nvme_ana_state ana_state;u32 ana_grpid; +u32 speed;/* path link speed (in Mbps) for fabrics */ atomic64_t slat_ns[2 ];/* path smoothed (EWMA) latency in nanosconds */ struct nvme_path_stat __percpu *cpu_stat;#endif
@@ -593,6 +594,7 @@ struct nvme_ctrl_ops {void (*delete_ctrl)(struct nvme_ctrl *ctrl);void (*stop_ctrl)(struct nvme_ctrl *ctrl);int (*get_address)(struct nvme_ctrl *ctrl,char *buf,int size); +u32 (*get_link_speed)(struct nvme_ctrl *ctrl);void (*print_device_info)(struct nvme_ctrl *ctrl);bool (*supports_pci_p2pdma)(struct nvme_ctrl *ctrl); };
@@ -1012,6 +1014,7 @@ extern struct device_attribute dev_attr_queue_depth;extern struct device_attribute dev_attr_numa_nodes;extern struct device_attribute dev_attr_adp_stat;extern struct device_attribute dev_attr_delayed_removal_secs; +extern struct device_attribute dev_attr_speed;extern struct device_attribute subsys_attr_iopolicy;static inline bool nvme_disk_is_ns_head (struct gendisk *disk)
drivers/nvme/host/sysfs.c +5 -0
@@ -262,6 +262,7 @@ static struct attribute *nvme_ns_attrs[] = { &dev_attr_numa_nodes.attr, &dev_attr_adp_stat.attr, &dev_attr_delayed_removal_secs.attr, + &dev_attr_speed.attr, #endif &dev_attr_io_passthru_err_log_enabled.attr, NULL,
@@ -308,6 +309,10 @@ static umode_t nvme_ns_attrs_are_visible(struct kobject *kobj,if (nvme_disk_is_ns_head (dev_to_disk (dev))) return 0; } +if (a == &dev_attr_speed.attr) { +if (nvme_disk_is_ns_head (dev_to_disk (dev))) +return 0 ; + }#endif return a->mode; }
drivers/nvme/host/tcp.c +66 -0
84 lines not on this page · raw message
Signature · 1 line
2.51.0
Re: [RFC PATCH 4/5] nvmf-tcp: add support for retrieving adapter link speed
Quoted 5 lines · Nilay Shroff
Add support for retrieving the negotiated NIC link speed (in Mbps). This value can be factored into path scoring for the adaptive I/O policy. For visibility and debugging, a new sysfs attribute "speed" is also added under the NVMe path block device.
Quoted 8 lines
Signed-off-by Nilay Shroff <[email protected]>--- drivers/nvme/host/multipath.c | 11 ++++++ drivers/nvme/host/nvme.h | 3 ++ drivers/nvme/host/sysfs.c | 5 +++ drivers/nvme/host/tcp.c | 66 +++++++++++++++++++++++++++++++++++ 4 files changed, 85 insertions(+)
Why not for FC? We can easily extract the link speed from there, too ...
But why do we need to do that? We already calculated the weighted average, so we _know_ the latency of each path. And then it's pretty much immaterial if a path runs with a given speed; if the latency is lower, that path is being preferred. Irrespective of the speed, which might be deceptive anyway as you'll only ever be able to retrieve the speed of the local link, not of the entire path.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Re: [RFC PATCH 4/5] nvmf-tcp: add support for retrieving adapter link speed
Quoted 14 lines · Hannes Reinecke
Quoted 5 lines · Nilay Shroff
Add support for retrieving the negotiated NIC link speed (in Mbps). This value can be factored into path scoring for the adaptive I/O policy. For visibility and debugging, a new sysfs attribute "speed" is also added under the NVMe path block device.
Quoted 8 lines
Signed-off-by Nilay Shroff <[email protected]>---
drivers/nvme/host/multipath.c| 11 ++++++ drivers/nvme/host/nvme.h | 3 ++ drivers/nvme/host/sysfs.c | 5 +++ drivers/nvme/host/tcp.c | 66 +++++++++++++++++++++++++++++++++++4 files changed, 85 insertions(+)
Quoted 2 lines
Why not for FC? We can easily extract the link speed from there, too ...
Yes it's easy to get the speed for FC. I just wanted to get feedback from the community about this idea and so didn't include it. But I will do that in the future patchset.
Quoted 8 lines
But why do we need to do that? We already calculated the weighted average, so we _know_ the latency of each path. And then it's pretty much immaterial if a path runs with a given speed; if the latency is lower, that path is being preferred. Irrespective of the speed, which might be deceptive anyway as you'll only ever be able to retrieve the speed of the local link, not of the entire path.
Consider a scenario with two paths: one over a high-capacity link (e.g. 1000 Mbps) and another over a much smaller link (e.g. 10 Mbps). If both paths report the same latency, the current formula would assign them identical weights. But in reality, the higher-capacity path can sustain a larger number of I/Os compared to the lower- capacity one.
In such cases, factoring in link speed allows us to assign proportionally higher weight to the higher-capacity path. At the same time, if that same path exhibits higher latency, it will be penalized accordingly, ensuring the final score balances both latency and bandwidth.
So, including link speed in the weight calculation helps capture both dimensions—latency sensitivity and throughput capacity—leading to a more accurate and proportional I/O distribution.
Thanks,
--Nilay
Re: [RFC PATCH 4/5] nvmf-tcp: add support for retrieving adapter link speed
Quoted 31 lines · Nilay Shroff
Quoted 14 lines · Hannes Reinecke
Quoted 5 lines · Nilay Shroff
Add support for retrieving the negotiated NIC link speed (in Mbps). This value can be factored into path scoring for the adaptive I/O policy. For visibility and debugging, a new sysfs attribute "speed" is also added under the NVMe path block device.
Quoted 8 lines
Signed-off-by Nilay Shroff <[email protected]>--- drivers/nvme/host/multipath.c | 11 ++++++ drivers/nvme/host/nvme.h | 3 ++ drivers/nvme/host/sysfs.c | 5 +++ drivers/nvme/host/tcp.c | 66 +++++++++++++++++++++++++++++++++++ 4 files changed, 85 insertions(+)Quoted 2 lines
Why not for FC? We can easily extract the link speed from there, too ...
Yes it's easy to get the speed for FC. I just wanted to get feedback from the community about this idea and so didn't include it. But I will do that in the future patchset.
Quoted 8 lines
But why do we need to do that? We already calculated the weighted average, so we _know_ the latency of each path. And then it's pretty much immaterial if a path runs with a given speed; if the latency is lower, that path is being preferred. Irrespective of the speed, which might be deceptive anyway as you'll only ever be able to retrieve the speed of the local link, not of the entire path.
Quoted 7 lines
Consider a scenario with two paths: one over a high-capacity link (e.g. 1000 Mbps) and another over a much smaller link (e.g. 10 Mbps). If both paths report the same latency, the current formula would assign them identical weights. But in reality, the higher-capacity path can sustain a larger number of I/Os compared to the lower- capacity one.
That would be correct if the transfer speed is assumed to be negligible. But I would assume that we do transfer mainly in units of PAGE_SIZE, so with 4k PAGE_SIZE we'll spend 400 ms on a 10Mbps link, but 4ms on a 1000Mbps link. That actually is one of the issues we're facing with measuring latency: we only have access to the combined latency (submission/data transfer/completion), so it's really hard to separate them out.
Quoted 5 lines
In such cases, factoring in link speed allows us to assign proportionally higher weight to the higher-capacity path. At the same time, if that same path exhibits higher latency, it will be penalized accordingly, ensuring the final score balances both latency and bandwidth.
See above. If we could measure them separately, yes. But we can't.
Quoted 4 lines
So, including link speed in the weight calculation helps capture both dimensions—latency sensitivity and throughput capacity—leading to a more accurate and proportional I/O distribution.
Would be true if we could measure it properly. But we can only get thespeed on the local link; everything behind that is anyone's guess, and it would skew measurements even more if we assume the same link speed
for the entire path.
Cheers,Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Re: [RFC PATCH 4/5] nvmf-tcp: add support for retrieving adapter link speed
Quoted 54 lines · Hannes Reinecke
Quoted 31 lines · Nilay Shroff
Quoted 14 lines · Hannes Reinecke
Quoted 5 lines · Nilay Shroff
Add support for retrieving the negotiated NIC link speed (in Mbps). This value can be factored into path scoring for the adaptive I/O policy. For visibility and debugging, a new sysfs attribute "speed" is also added under the NVMe path block device.
Quoted 8 lines
Signed-off-by Nilay Shroff <[email protected]>---
drivers/nvme/host/multipath.c| 11 ++++++ drivers/nvme/host/nvme.h | 3 ++ drivers/nvme/host/sysfs.c | 5 +++ drivers/nvme/host/tcp.c | 66 +++++++++++++++++++++++++++++++++++4 files changed, 85 insertions(+)Quoted 2 lines
Why not for FC? We can easily extract the link speed from there, too ...
Yes it's easy to get the speed for FC. I just wanted to get feedback from the community about this idea and so didn't include it. But I will do that in the future patchset.
Quoted 8 lines
But why do we need to do that? We already calculated the weighted average, so we _know_ the latency of each path. And then it's pretty much immaterial if a path runs with a given speed; if the latency is lower, that path is being preferred. Irrespective of the speed, which might be deceptive anyway as you'll only ever be able to retrieve the speed of the local link, not of the entire path.
Quoted 7 lines
Consider a scenario with two paths: one over a high-capacity link (e.g. 1000 Mbps) and another over a much smaller link (e.g. 10 Mbps). If both paths report the same latency, the current formula would assign them identical weights. But in reality, the higher-capacity path can sustain a larger number of I/Os compared to the lower- capacity one.
That would be correct if the transfer speed is assumed to be negligible. But I would assume that we do transfer mainly in units of
PAGE_SIZE, so with 4kPAGE_SIZEwe'll spend 400 ms on a 10Mbps link, but 4ms on a 1000Mbps link. That actually is one of the issues we're facing with measuring latency: we only have access to the combined latency (submission/data transfer/completion), so it's really hard to separate them out.Quoted 5 lines
In such cases, factoring in link speed allows us to assign proportionally higher weight to the higher-capacity path. At the same time, if that same path exhibits higher latency, it will be penalized accordingly, ensuring the final score balances both latency and bandwidth.
See above. If we could measure them separately, yes. But we can't.
Quoted 8 lines
Quoted 4 lines
So, including link speed in the weight calculation helps capture both dimensions—latency sensitivity and throughput capacity—leading to a more accurate and proportional I/O distribution.
Would be true if we could measure it properly. But we can only get thespeed on the local link; everything behind that is anyone's guess, and it would skew measurements even more if we assume the same link speed
for the entire path.
Yes you brought a very good point and I agree that we can’t reliably determine the end-to-end path capacity. Assuming the same link speed beyond the first hop may not always be correct and could easily skew the measurement.
Given that limitation, I agree it would be better to exclude link speed from the path scoring formula. Without a way to accurately capture the full path capacity, incorporating only the local link speed risks making the scoring misleading rather than more accurate.
Thanks,
--Nilay
RFC PATCH 5/5 nvme-multipath: factor fabric link speed into path score
In reply to Nilay ShroffIf the fabric adapter link speed is known, include it when calculating the path score for the adaptive I/O policy. Paths with higher link speed receive proportionally higher scores, while paths with lower link speed receive lower scores.
For example, in a multipath topology with two paths—one with higher link speed but higher latency, and another with lower link speed but lower latency—the scoring formula balances these factors. The result ensures that path selection does not blindly favor high link speed, but adjusts scores based on both link speed and latency to achieve proportional distribution.
The updated path scoring formula is:
path_X_score = link_speed_X * (NSEC_PER_SEC / path_X_ewma_latency)
where:
- link_speed_X is the negotiated link speed of the fabric adapter
(in Mbps),
- path_X_ewma_latency is the smoothed latency (ns) derived from I/O completions,
- NSEC_PER_SEC is used as a scaling factor.
Weights are then normalized across all paths:
path_X_weight = (path_X_score * 100) / total_score
This ensures that both lower latency and higher link speed contribute positively to path selection, while still distributing I/O proportionally when conditions differ across paths.
--- drivers/nvme/host/multipath.c | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-)
drivers/nvme/host/multipath.c +12 -8
@@ -246,7 +246,7 @@ static void nvme_mpath_add_sample(struct request *rq, struct nvme_ns *ns)unsigned int rw;struct nvme_path_stat *stat;struct nvme_ns *cur_ns; -u32 weight; +u32 weight, speed;u64 now, latency, avg_lat_ns;u64 total_score =0 ;struct nvme_ns_head *head = ns->head;
@@ -347,14 +347,18 @@ static void nvme_mpath_add_sample(struct request *rq, struct nvme_ns *ns)continue ;/* -* Compute the path score (inverse of smoothed latency), -* scaled by NSEC_PER_SEC. Floating point math is not -* available in the kernel, so fixed-point scaling is -* used instead. NSEC_PER_SEC is chosen as the scale -* because valid latencies are always < 1 second; and -* we ignore longer latencies. +* Compute the path score as the inverse of smoothed +* latency, scaled by NSEC_PER_SEC. If the device speed +* is known, it is factored in: higher speed increases +* the score, lower speed decreases it. Floating point +* math is unavailable in the kernel, so fixed-point +* scaling is used instead. NSEC_PER_SEC is chosen +* because valid latencies are always < 1 second; longer +* latencies are ignored. */ - stat->score =div_u64 (NSEC_PER_SEC , stat->slat_ns); + speed = cur_ns->speed ? :1 ; + stat->score = speed *div_u64 (NSEC_PER_SEC , + stat->slat_ns);/* Compute total score. */ total_score += stat->score;
Signature · 1 line
2.51.0