nvme: introduce the service-time iopolicy
Subsystems (2)
v4 cover nvme: introduce the service-time iopolicy
The queue-depth iopolicy counts requests, so a 128K I/O weighs the same as a 4K one, and it cannot express that one path is simply faster than another. This series adds a service-time iopolicy for NVMe native multipath, adapted from dm-ps-service-time: it picks the path with the lowest estimated service time, in_flight_bytes / relative_throughput, so balancing follows actual bytes in flight and per-path throughput weights.
Weights live in a per-path relative_throughput sysfs attribute (0-100, default 1, 0 makes the path a standby). Patch 3 documents the new attributes in the stable sysfs ABI; patch 1 carries the benchmark data in its commit log.
This version is rebased onto the nvme-7.4 branch ofgit://git.infradead.org/nvme.git.
Changes since v3:
- Rebased onto nvme-7.4, which adds the iopolicies and path_state attributes and the discovery-subsystem "none" handling.
- The iopolicies ABI entry now lists service-time too. nvme-7.4 already documents the attribute (with a hardcoded list of the three older policies); patch 3 extends that list along with the iopolicy accepted values.
Changes since v2:
- in_flight_bytes no longer leaks on failover: the accounted bytes are now released in nvme_failover_req() before blk_steal_bios() clears the request data length. Before, every failover (ANA transition, path error, queue_rq unwind) permanently left the request's bytes in the counter, skewing path selection until a controller reset. Reproduced with an ANA transition during load on the NVMe-oF/TCP test setup: the counter kept 512K of phantom bytes after all I/O completed; with the fix it returns to zero.
- The release clamps at zero instead of a plain subtraction. A request requeued across a controller reset is accounted against the pre-reset counter but completes after nvme_mpath_init_identify() re-zeroed it, and the plain subtraction would wrap negative, making the comparator treat the path as pathologically overloaded. This mirrors atomic_dec_if_positive() in the queue-depth accounting.
- The comparison operates on u64 instead of size_t, so the byte counts and the shift guard keep their magnitude on 32-bit builds, where size_t would truncate the atomic64 reads.
- Added "service-time" to the accepted values of the subsystem iopolicy ABI entry and to the policies listed by the iopolicies ABI entry.
No public v3 was posted; this version folds the review feedback from the v2 discussion and a rebase onto nvme-7.4.
On the adaptive I/O policy dependency Hannes raised during v1: that series has become the latency I/O policy and is still under revision. The two are complementary: the latency policy infers path speed from observed completions, service-time takes configured weights and byte counts.
Guixin Liu (3):
nvme-multipath: implement service-time iopolicy
nvme-multipath: document service-time iopolicy
nvme: document the service-time iopolicy sysfs attributes
Documentation/ABI/stable/sysfs-nvme | 35 +++- Documentation/admin-guide/nvme-multipath.rst | 31 +++- drivers/nvme/host/multipath.c | 189 ++++++++++++++++++- drivers/nvme/host/nvme.h | 6 + drivers/nvme/host/sysfs.c | 7 +- 5 files changed, 256 insertions(+), 12 deletions(-)
v2 of this series:
https://lore.kernel.org/r/[email protected]
v1 of this series:
https://lore.kernel.org/r/[email protected]
latency I/O policy v7:
https://lore.kernel.org/r/[email protected]
Signature · 1 line
2.43.7
PATCH v4 1/3 nvme-multipath: implement service-time iopolicy
The queue-depth iopolicy balances paths by the number of in-flight requests. This misjudges the load in two common cases: a 128K request counts the same as a 4K one, and there is no way to tell the selector that one path is simply faster than another. With mixed I/O sizes or paths of different bandwidth, large I/Os keep piling onto the slow path, driving up tail latencies.
Add a "service-time" iopolicy adapted from the dm-ps-service-time path selector. It selects the path with the lowest estimated service time, computed as in_flight_bytes / relative_throughput, so it tracks actual bytes in flight rather than request counts, and users can weight paths according to their throughput.
Each path keeps the total in-flight I/O size and a relative throughput weight as controller state, exposed through two new sysfs attributes on the path namespace devices: in_flight_bytes (read-only) reports the tracked load, and relative_throughput (read-write, 0-100, default 1) sets the weight. Setting relative_throughput to 0 makes the path a standby that is only selected when no path with a positive value is available, matching DM service-time semantics.
As in dm-ps-service-time, the path comparison cross-multiplies to avoid division, and shifts the load counters down when they grow too large to keep the products in range.
Failovers release the accounted bytes in nvme_failover_req() before blk_steal_bios() clears the request data length. The release clamps at zero: a request requeued across a controller reset completes against a counter that nvme_mpath_init_identify() has re-zeroed, and a plain subtraction would underflow, the case that atomic_dec_if_positive() guards against in the queue-depth accounting.
Usage:
echo service-time > /sys/module/nvme_core/parameters/iopolicy echo 4 > /sys/block/nvmeXcYnZ/relative_throughput
Test environment:
Two QEMU/KVM VMs connected via NVMe-oF TCP, 32 vCPU / 32G RAM each. The target exports a 1G RAM disk via nvmet-tcp on two ports, and the initiator connects both ports to form a native NVMe multipath device.
Asymmetric bandwidth is created by shaping the second path with tc-tbf (rate 500mbit, burst 128kb, latency 5ms). Measured single path bandwidth: ~237 MiB/s unshaped vs ~56 MiB/s shaped, a ratio of approximately 4:1.
Test 1 - Uniform 4k random read (iodepth=64, numjobs=4, 15s):
iopolicy IOPS BW(MiB/s) avg_lat(us) lat_stddev(us) -------------------- ----- --------- ----------- -------------- round-robin 59.1k 231 4316 3109 queue-depth 59.0k 230 4323 3046 service-time 1:1 59.3k 232 4302 3057 service-time 4:1 61.0k 238 4179 1294
With uniform I/O size, service-time 1:1 performs similarly to queue-depth since per-byte and per-request tracking are equivalent for fixed-size I/Os. service-time 4:1 with correct throughput weights improves IOPS by 3.4% and reduces latency stddev by 57%.
Test 2 - Mixed random read 4k/128k (bssplit=4k/50:128k/50,
iodepth=64, numjobs=4, 15s):
iopolicy IOPS BW(MiB/s) avg_lat(us) lat_stddev(us) -------------------- ---- --------- ----------- -------------- round-robin 7689 276 33209 46817 queue-depth 7642 276 33429 45868 service-time 1:1 7456 269 34279 47433 service-time 4:1 7747 277 32998 27102
With mixed I/O sizes on asymmetric paths, queue-depth treats 128K and 4K requests identically when counting in-flight depth, leading to suboptimal load distribution. service-time tracks actual bytes in flight, producing significantly more consistent tail latencies. service-time 4:1 vs queue-depth: latency stddev -41%.
Standby path (relative_throughput=0):
Setting a path's relative_throughput to 0 correctly excludes it from selection. With the slow path set to 0, all I/O is directed to the fast path only (IO split 118052:0), verified via diskstats.
--- drivers/nvme/host/multipath.c | 189 +++++++++++++++++++++++++++++++++- drivers/nvme/host/nvme.h | 6 ++ drivers/nvme/host/sysfs.c | 7 +- 3 files changed, 198 insertions(+), 4 deletions(-)
drivers/nvme/host/multipath.c +187 -2
@@ -69,6 +69,7 @@ static const char *nvme_iopolicy_names[] = { [NVME_IOPOLICY_NUMA] = "numa", [NVME_IOPOLICY_RR] = "round-robin", [NVME_IOPOLICY_QD] = "queue-depth", + [NVME_IOPOLICY_ST] = "service-time", };static int iopolicy =NVME_IOPOLICY_NUMA ;
@@ -107,7 +108,7 @@ static int nvme_get_iopolicy(char *buf, const struct kernel_param *kp)module_param_call (iopolicy, nvme_set_iopolicy, nvme_get_iopolicy, &iopolicy,0644 );MODULE_PARM_DESC (iopolicy, -"Default multipath I/O policy; 'numa' (default), 'round-robin' or 'queue-depth'" ); +"Default multipath I/O policy; 'numa' (default), 'round-robin', 'queue-depth' or 'service-time'" );void nvme_mpath_default_iopolicy (struct nvme_subsystem *subsys) {
@@ -144,6 +145,24 @@ void nvme_mpath_start_freeze(struct nvme_subsystem *subsys)blk_freeze_queue_start (h->disk->queue); } +/* +* Release in-flight bytes without letting the counter go negative: a +* request that sat in a requeue list across a controller reset is +* accounted against the pre-reset counter, but completes after +* nvme_mpath_init_identify() has zeroed it, in which case the plain +* subtraction would underflow. This mirrors atomic_dec_if_positive() +* in the queue-depth accounting. +*/ +static void nvme_st_release (struct nvme_ctrl *ctrl,unsigned int bytes) +{ +s64 old, new; + +do { + old =atomic64_read (&ctrl->in_flight_size); + new =max_t (s64,0 , old - bytes); + }while (!atomic64_try_cmpxchg (&ctrl->in_flight_size, &old, new)); +} +void nvme_failover_req (struct request *req) {struct nvme_ns *ns = req->q->queuedata;
@@ -164,6 +183,15 @@ void nvme_failover_req(struct request *req)queue_work (nvme_wq, &ns->ctrl->ana_work); } +/* +* blk_steal_bios() clears req->__data_len, so release the bytes +* accounted by the service-time policy before stealing the bios. +*/ +if (nvme_req (req)->flags &NVME_MPATH_CNT_ACTIVE_BYTES ) { +nvme_st_release (ns->ctrl,blk_rq_bytes (req)); +nvme_req (req)->flags &= ~NVME_MPATH_CNT_ACTIVE_BYTES ; + } +spin_lock_irqsave (&ns->head->requeue_lock, flags);for (bio = req->bio; bio; bio = bio->bi_next)bio_set_dev (bio, ns->head->disk->part0);
@@ -179,11 +207,16 @@ void nvme_mpath_start_request(struct request *rq) {struct nvme_ns *ns = rq->q->queuedata;struct gendisk *disk = ns->head->disk; +int iopolicy =READ_ONCE (ns->head->subsys->iopolicy); -if ((READ_ONCE (ns->head->subsys->iopolicy) ==NVME_IOPOLICY_QD ) && +if (iopolicy ==NVME_IOPOLICY_QD && !(nvme_req (rq)->flags &NVME_MPATH_CNT_ACTIVE )) {atomic_inc (&ns->ctrl->nr_active);nvme_req (rq)->flags |=NVME_MPATH_CNT_ACTIVE ; + }else if (iopolicy ==NVME_IOPOLICY_ST && + !(nvme_req (rq)->flags &NVME_MPATH_CNT_ACTIVE_BYTES )) { +atomic64_add (blk_rq_bytes (rq), &ns->ctrl->in_flight_size); +nvme_req (rq)->flags |=NVME_MPATH_CNT_ACTIVE_BYTES ; } if (!blk_queue_io_stat(disk->queue) ||
@@ -205,6 +238,10 @@ void nvme_mpath_end_request(struct request *rq)if (nvme_req (rq)->flags &NVME_MPATH_CNT_ACTIVE )atomic_dec_if_positive (&ns->ctrl->nr_active); +if (nvme_req (rq)->flags &NVME_MPATH_CNT_ACTIVE_BYTES ) { +nvme_st_release (ns->ctrl,blk_rq_bytes (rq)); +nvme_req (rq)->flags &= ~NVME_MPATH_CNT_ACTIVE_BYTES ; + }if (!(nvme_req (rq)->flags &NVME_MPATH_IO_STATS ))return ;
@@ -477,6 +514,109 @@ static struct nvme_ns *nvme_queue_depth_path(struct nvme_ns_head *head)return best_opt ? best_opt : best_nonopt; } +#define ST_MAX_RELATIVE_THROUGHPUT 100 +#define ST_MAX_RELATIVE_THROUGHPUT_SHIFT 7 +#define ST_MAX_INFLIGHT_SIZE \ + (U64_MAX >> ST_MAX_RELATIVE_THROUGHPUT_SHIFT) + +/* +* Compare estimated service time of two paths. +* Returns negative if ns1 is better, positive if ns2 is better, 0 if equal. +* +* Service time = in_flight_size / relative_throughput +* Cross-multiply to avoid division. +*/ +static int nvme_st_compare_load (struct nvme_ns *ns1,struct nvme_ns *ns2) +{ +u64 sz1, sz2, st1, st2; +unsigned int tp1 = ns1->ctrl->relative_throughput; +unsigned int tp2 = ns2->ctrl->relative_throughput; + + sz1 =atomic64_read (&ns1->ctrl->in_flight_size); + sz2 =atomic64_read (&ns2->ctrl->in_flight_size); + +/* Case 1: same throughput — compare load directly */ +if (tp1 == tp2) +return (sz1 > sz2) - (sz1 < sz2); + +/* +* Case 2a: same load — prefer higher throughput. +* Case 2b: one path has zero throughput — prefer the other. +*/ +if (sz1 == sz2 || !tp1 || !tp2) +return tp2 - tp1; + +/* +* Case 3: general comparison via cross-multiplication. +* st1 = sz1 / tp1 +* st2 = sz2 / tp2 +* Equivalent (since tp > 0): +* sz1 * tp2 <=> sz2 * tp1 +* +* Use comparisons instead of subtractions when returning: the +* intermediate values do not fit into the int return type. +*/ +if (unlikely (sz1 >=ST_MAX_INFLIGHT_SIZE || + sz2 >=ST_MAX_INFLIGHT_SIZE )) { + sz1 >>=ST_MAX_RELATIVE_THROUGHPUT_SHIFT ; + sz2 >>=ST_MAX_RELATIVE_THROUGHPUT_SHIFT ; + } + st1 = sz1 * tp2; + st2 = sz2 * tp1; +if (st1 != st2) +return (st1 > st2) - (st1 < st2); + +/* Case 4: equal service time — prefer higher throughput */ +return tp2 - tp1; +} + +static struct nvme_ns *nvme_service_time_path (struct nvme_ns_head *head) +{ +struct nvme_ns *best_opt =NULL , *best_nonopt =NULL , *ns; +struct nvme_ns *fallback_opt =NULL , *fallback_nonopt =NULL ; + +list_for_each_entry_srcu (ns, &head->list, siblings, +srcu_read_lock_held (&head->srcu)) { +if (nvme_path_is_disabled (ns)) +continue ; + +/* +* Paths with relative_throughput == 0 are only used when no +* path with a positive value is available (matching DM +* service-time semantics). +*/ +if (!ns->ctrl->relative_throughput) { +if (ns->ana_state ==NVME_ANA_OPTIMIZED && !fallback_opt) + fallback_opt = ns; +else if (ns->ana_state ==NVME_ANA_NONOPTIMIZED && + !fallback_nonopt) + fallback_nonopt = ns; +continue ; + } + +switch (ns->ana_state) { +case NVME_ANA_OPTIMIZED : +if (!best_opt || +nvme_st_compare_load (ns, best_opt) <0 ) + best_opt = ns; +break ; +case NVME_ANA_NONOPTIMIZED : +if (!best_nonopt || +nvme_st_compare_load (ns, best_nonopt) <0 ) + best_nonopt = ns; +break ; +default : +break ; + } + } + +if (best_opt) +return best_opt; +if (best_nonopt) +return best_nonopt; +return fallback_opt ? fallback_opt : fallback_nonopt; +} +static inline bool nvme_path_is_optimized (struct nvme_ns *ns) { return nvme_ctrl_state(ns->ctrl) == NVME_CTRL_LIVE &&
@@ -504,6 +644,8 @@ inline struct nvme_ns *nvme_find_path(struct nvme_ns_head *head)return nvme_queue_depth_path (head);case NVME_IOPOLICY_RR :return nvme_round_robin_path (head); +case NVME_IOPOLICY_ST : +return nvme_service_time_path (head);default :return nvme_numa_path (head); }
@@ -1192,6 +1334,47 @@ static ssize_t path_state_show(struct device *dev, }DEVICE_ATTR_RO (path_state); +static ssize_t in_flight_bytes_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct nvme_ns *ns =nvme_get_ns_from_dev (dev); + +if (ns->head->subsys->iopolicy !=NVME_IOPOLICY_ST ) +return 0 ; + +return sysfs_emit (buf,"%lld\n" ,atomic64_read (&ns->ctrl->in_flight_size)); +} +DEVICE_ATTR_RO (in_flight_bytes); + +static ssize_t relative_throughput_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct nvme_ns *ns =nvme_get_ns_from_dev (dev); + +if (ns->head->subsys->iopolicy !=NVME_IOPOLICY_ST ) +return 0 ; + +return sysfs_emit (buf,"%u\n" , ns->ctrl->relative_throughput); +} + +static ssize_t relative_throughput_store (struct device *dev, +struct device_attribute *attr,const char *buf,size_t count) +{ +struct nvme_ns *ns =nvme_get_ns_from_dev (dev); +unsigned int val; +int ret; + + ret =kstrtouint (buf,0 , &val); +if (ret <0 ) +return ret; +if (val >ST_MAX_RELATIVE_THROUGHPUT ) +return -EINVAL ; + + ns->ctrl->relative_throughput = val; +return count; +} +DEVICE_ATTR_RW (relative_throughput); +static ssize_t numa_nodes_show (struct device *dev,struct device_attribute *attr,char *buf) {
@@ -1555,6 +1738,7 @@ void nvme_mpath_init_ctrl(struct nvme_ctrl *ctrl)mutex_init (&ctrl->ana_lock);timer_setup (&ctrl->anatt_timer, nvme_anatt_timeout,0 );INIT_WORK (&ctrl->ana_work, nvme_ana_work); + ctrl->relative_throughput =1 ; }int nvme_mpath_init_identify (struct nvme_ctrl *ctrl,struct nvme_id_ctrl *id)
@@ -1570,6 +1754,7 @@ int nvme_mpath_init_identify(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)/* initialize this in the identify path to cover controller resets */ atomic_set (&ctrl->nr_active,0 ); +atomic64_set (&ctrl->in_flight_size,0 );if (!ctrl->max_namespaces || ctrl->max_namespaces >le32_to_cpu (id->nn)) {
drivers/nvme/host/nvme.h +6 -0
@@ -268,6 +268,7 @@ enum {NVME_REQ_USERCMD = (1 <<1 ),NVME_MPATH_IO_STATS = (1 <<2 ),NVME_MPATH_CNT_ACTIVE = (1 <<3 ), + NVME_MPATH_CNT_ACTIVE_BYTES = (1 << 4), };static inline struct nvme_request *nvme_req (struct request *req)
@@ -439,6 +440,8 @@ struct nvme_ctrl {struct timer_list anatt_timer;struct work_struct ana_work;atomic_t nr_active; +atomic64_t in_flight_size; +u8 relative_throughput;#endif #ifdef CONFIG_NVME_HOST_AUTH
@@ -492,6 +495,7 @@ enum nvme_iopolicy {NVME_IOPOLICY_NUMA ,NVME_IOPOLICY_RR ,NVME_IOPOLICY_QD , + NVME_IOPOLICY_ST, }; struct nvme_subsystem {
@@ -1095,6 +1099,8 @@ extern bool multipath;extern struct device_attribute dev_attr_ana_grpid;extern struct device_attribute dev_attr_ana_state;extern struct device_attribute dev_attr_queue_depth; +extern struct device_attribute dev_attr_in_flight_bytes; +extern struct device_attribute dev_attr_relative_throughput;extern struct device_attribute dev_attr_numa_nodes;extern struct device_attribute dev_attr_path_state;extern struct device_attribute dev_attr_delayed_removal_secs;
drivers/nvme/host/sysfs.c +5 -2
@@ -262,6 +262,8 @@ static struct attribute *nvme_ns_attrs[] = { &dev_attr_ana_grpid.attr, &dev_attr_ana_state.attr, &dev_attr_queue_depth.attr, + &dev_attr_in_flight_bytes.attr, + &dev_attr_relative_throughput.attr, &dev_attr_numa_nodes.attr, &dev_attr_path_state.attr, &dev_attr_delayed_removal_secs.attr,
@@ -298,8 +300,9 @@ static umode_t nvme_ns_attrs_are_visible(struct kobject *kobj,if (!nvme_ctrl_use_ana (nvme_get_ns_from_dev (dev)->ctrl))return 0 ; } -if (a == &dev_attr_queue_depth.attr || a == &dev_attr_numa_nodes.attr || - a == &dev_attr_path_state.attr) { +if (a == &dev_attr_queue_depth.attr || a == &dev_attr_path_state.attr || + a == &dev_attr_in_flight_bytes.attr || + a == &dev_attr_relative_throughput.attr || a == &dev_attr_numa_nodes.attr) {if (nvme_disk_is_ns_head (dev_to_disk (dev)))return 0 ; }
Signature · 1 line
2.43.7
PATCH v4 2/3 nvme-multipath: document service-time iopolicy
In reply to Guixin LiuAdd documentation for the service-time path selection policy: its algorithm, the in_flight_bytes and relative_throughput sysfs attributes, and guidance on when to prefer it over queue-depth.
Document that setting relative_throughput to 0 makes the path a standby that is only selected when no path with a positive value is available.
--- Documentation/admin-guide/nvme-multipath.rst | 31 ++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-)
Documentation/admin-guide/nvme-multipath.rst +29 -2
@@ -24,8 +24,8 @@ Policies All policies follow the ANA (Asymmetric Namespace Access) mechanism, meaning that when an optimized path is available, it will be chosen over a non-optimized -one. Current the NVMe multipath policies include numa(default), round-robin and -queue-depth. +one. Current the NVMe multipath policies include numa(default), round-robin, +queue-depth and service-time. To set the desired policy (e.g., round-robin), use one of the following methods: 1. echo -n "round-robin" > /sys/module/nvme_core/parameters/iopolicy
@@ -70,3 +70,30 @@ When to use the queue-depth policy: 1. High load with small I/Os: Effectively balances load across paths when the load is high, and I/O operations consist of small, relatively fixed-sized requests. + + +Service-Time +------------ + +The service-time policy selects the path with the lowest estimated service time. +It calculates service time as ``in_flight_bytes / relative_throughput`` for each +path, preferring the path that would complete I/O fastest. Unlike queue-depth +which counts requests regardless of size, service-time tracks actual bytes in +flight, making it aware of I/O sizes. + +Each path exposes two sysfs attributes under +``/sys/class/nvme/nvmeX/nvmeXcYnZ/``: + + - ``in_flight_bytes`` (read-only): Current bytes in flight on this path. + - ``relative_throughput`` (read-write): Relative throughput weight for this + path, default 1. The valid range is 0-100. Set higher values for faster + paths. If set to 0, the path is not selected while other paths with + positive values are available. + +When to use the service-time policy: + 1. Asymmetric Link Speeds: When paths have different bandwidths, set + ``relative_throughput`` proportionally (e.g., 2 for a link twice as fast) + to steer more traffic to faster paths. + 2. Mixed I/O Sizes: When workloads mix small and large I/Os (e.g., 4K and + 128K), service-time distributes load more accurately than queue-depth + because it accounts for actual bytes rather than request count.
Signature · 1 line
2.43.7
PATCH v4 3/3 nvme: document the service-time iopolicy sysfs attributes
In reply to Guixin LiuThe service-time iopolicy adds two sysfs attributes to the NVMe path namespace devices: in_flight_bytes, reporting the bytes currently in flight on the path's controller, and relative_throughput, the per-path throughput weight used by the service-time estimation.
Document both in the stable sysfs ABI, including the 0-100 value range, the default of 1, and the standby semantics of a relative_throughput of 0. Also add "service-time" to the accepted values of the subsystem iopolicy attribute and to the policies listed by the iopolicies attribute.
--- Documentation/ABI/stable/sysfs-nvme | 35 ++++++++++++++++++++++++----- 1 file changed, 29 insertions(+), 6 deletions(-)
Documentation/ABI/stable/sysfs-nvme +29 -6
@@ -347,6 +347,28 @@ Description: Requires CONFIG_NVME_MULTIPATH. +What: /sys/block/nvmeXcYnZ/in_flight_bytes +What: /sys/block/nvmeXcYnZ/relative_throughput +Date: October 2026 +KernelVersion: 7.5 +Contact: Guixin Liu <[email protected]> +Description: + in_flight_bytes: Shows the number of bytes currently in + flight on this path's controller. Returns empty if + iopolicy is not "service-time". + Requires CONFIG_NVME_MULTIPATH. + + relative_throughput: Shows or sets the relative + throughput weight of this path's controller, which the + "service-time" iopolicy uses to estimate the path service + time as in_flight_bytes / relative_throughput. The value is + an integer in the range 0-100 and defaults to 1; assign + higher values to faster paths. A path with a value of 0 is + only selected when no path with a positive value is + available. Reading returns empty if iopolicy is not + "service-time"; writing is accepted regardless of the + active policy. Requires CONFIG_NVME_MULTIPATH. + What: /sys/block/nvmeXnY/delayed_removal_secs Date: May 2025 KernelVersion: 6.16
@@ -477,9 +499,9 @@ Contact: Hannes Reinecke <[email protected]> Description: Shows or sets the multipath I/O path selection policy for this subsystem. Accepted values: "numa", "round-robin", - "queue-depth". Changing the policy clears all current path - selections. Only available when CONFIG_NVME_MULTIPATH is - enabled. + "queue-depth", "service-time". Changing the policy clears + all current path selections. Only available when + CONFIG_NVME_MULTIPATH is enabled. A discovery subsystem has no namespaces and hence no I/O paths to select from: reading returns "none" and writing
@@ -492,9 +514,10 @@ Contact: Laurence Oberman <[email protected]> Description: Shows the I/O path selection policies that may be written to the iopolicy attribute of this subsystem, separated by - spaces: "numa round-robin queue-depth". A discovery - subsystem accepts no policy at all and reports "none". - Only available when CONFIG_NVME_MULTIPATH is enabled. + spaces: "numa round-robin queue-depth service-time". A + discovery subsystem accepts no policy at all and reports + "none". Only available when CONFIG_NVME_MULTIPATH is + enabled. What: /sys/class/nvme-subsystem/nvme-subsysX/subsystype Date: September 2021
Signature · 1 line
2.43.7
Hi guys,
All the issues Sashiko raised are addressed in this v4, and it looks like Sashiko has no further comments on the current version now. Please review this series, all comments are welcome. Or if can tell me this approach is reasonable, I'll keep iterating on the implementation details.
Best Regards,
Guixin Liu
Quoted 81 lines · Guixin Liu
The queue-depth iopolicy counts requests, so a 128K I/O weighs the same as a 4K one, and it cannot express that one path is simply faster than another. This series adds a service-time iopolicy for NVMe native multipath, adapted from dm-ps-service-time: it picks the path with the lowest estimated service time,
in_flight_bytes/relative_throughput, so balancing follows actual bytes in flight and per-path throughput weights.Weights live in a per-path
relative_throughputsysfs attribute (0-100, default 1, 0 makes the path a standby). Patch 3 documents the new attributes in the stable sysfs ABI; patch 1 carries the benchmark data in its commit log.This version is rebased onto the nvme-7.4 branch of
git://git.infradead.org/nvme.git.Changes since v3:
- Rebased onto nvme-7.4, which adds the iopolicies and
path_stateattributes and the discovery-subsystem "none" handling.- The iopolicies ABI entry now lists service-time too. nvme-7.4 already documents the attribute (with a hardcoded list of the three older policies); patch 3 extends that list along with the iopolicy accepted values.
Changes since v2:
-
in_flight_bytesno longer leaks on failover: the accounted bytes are now released innvme_failover_req()beforeblk_steal_bios()clears the request data length. Before, every failover (ANA transition, path error,queue_rqunwind) permanently left the request's bytes in the counter, skewing path selection until a controller reset. Reproduced with an ANA transition during load on the NVMe-oF/TCP test setup: the counter kept 512K of phantom bytes after all I/O completed; with the fix it returns to zero.- The release clamps at zero instead of a plain subtraction. A request requeued across a controller reset is accounted against the pre-reset counter but completes after
nvme_mpath_init_identify()re-zeroed it, and the plain subtraction would wrap negative, making the comparator treat the path as pathologically overloaded. This mirrorsatomic_dec_if_positive()in the queue-depth accounting.- The comparison operates on u64 instead of
size_t, so the byte counts and the shift guard keep their magnitude on 32-bit builds, wheresize_twould truncate the atomic64 reads.- Added "service-time" to the accepted values of the subsystem iopolicy ABI entry and to the policies listed by the iopolicies ABI entry.
No public v3 was posted; this version folds the review feedback from the v2 discussion and a rebase onto nvme-7.4.
On the adaptive I/O policy dependency Hannes raised during v1: that series has become the latency I/O policy and is still under revision. The two are complementary: the latency policy infers path speed from observed completions, service-time takes configured weights and byte counts.
Guixin Liu (3):
nvme-multipath: implement service-time iopolicy
nvme-multipath: document service-time iopolicy
nvme: document the service-time iopolicy sysfs attributesDocumentation/ABI/stable/sysfs-nvme | 35 +++- Documentation/admin-guide/nvme-multipath.rst | 31 +++- drivers/nvme/host/multipath.c | 189 ++++++++++++++++++- drivers/nvme/host/nvme.h | 6 + drivers/nvme/host/sysfs.c | 7 +- 5 files changed, 256 insertions(+), 12 deletions(-)v2 of this series:
https://lore.kernel.org/r/[email protected]v1 of this series:
https://lore.kernel.org/r/[email protected]latency I/O policy v7:
https://lore.kernel.org/r/[email protected]
Hi,
A gentle ping on this series, all comments are welcome. If you have any concerns about this feature, please let me know. Best Regards, Guixin Liu
Quoted 81 lines · Guixin Liu
The queue-depth iopolicy counts requests, so a 128K I/O weighs the same as a 4K one, and it cannot express that one path is simply faster than another. This series adds a service-time iopolicy for NVMe native multipath, adapted from dm-ps-service-time: it picks the path with the lowest estimated service time,
in_flight_bytes/relative_throughput, so balancing follows actual bytes in flight and per-path throughput weights.Weights live in a per-path
relative_throughputsysfs attribute (0-100, default 1, 0 makes the path a standby). Patch 3 documents the new attributes in the stable sysfs ABI; patch 1 carries the benchmark data in its commit log.This version is rebased onto the nvme-7.4 branch of
git://git.infradead.org/nvme.git.Changes since v3:
- Rebased onto nvme-7.4, which adds the iopolicies and
path_stateattributes and the discovery-subsystem "none" handling.- The iopolicies ABI entry now lists service-time too. nvme-7.4 already documents the attribute (with a hardcoded list of the three older policies); patch 3 extends that list along with the iopolicy accepted values.
Changes since v2:
-
in_flight_bytesno longer leaks on failover: the accounted bytes are now released innvme_failover_req()beforeblk_steal_bios()clears the request data length. Before, every failover (ANA transition, path error,queue_rqunwind) permanently left the request's bytes in the counter, skewing path selection until a controller reset. Reproduced with an ANA transition during load on the NVMe-oF/TCP test setup: the counter kept 512K of phantom bytes after all I/O completed; with the fix it returns to zero.- The release clamps at zero instead of a plain subtraction. A request requeued across a controller reset is accounted against the pre-reset counter but completes after
nvme_mpath_init_identify()re-zeroed it, and the plain subtraction would wrap negative, making the comparator treat the path as pathologically overloaded. This mirrorsatomic_dec_if_positive()in the queue-depth accounting.- The comparison operates on u64 instead of
size_t, so the byte counts and the shift guard keep their magnitude on 32-bit builds, wheresize_twould truncate the atomic64 reads.- Added "service-time" to the accepted values of the subsystem iopolicy ABI entry and to the policies listed by the iopolicies ABI entry.
No public v3 was posted; this version folds the review feedback from the v2 discussion and a rebase onto nvme-7.4.
On the adaptive I/O policy dependency Hannes raised during v1: that series has become the latency I/O policy and is still under revision. The two are complementary: the latency policy infers path speed from observed completions, service-time takes configured weights and byte counts.
Guixin Liu (3):
nvme-multipath: implement service-time iopolicy
nvme-multipath: document service-time iopolicy
nvme: document the service-time iopolicy sysfs attributesDocumentation/ABI/stable/sysfs-nvme | 35 +++- Documentation/admin-guide/nvme-multipath.rst | 31 +++- drivers/nvme/host/multipath.c | 189 ++++++++++++++++++- drivers/nvme/host/nvme.h | 6 + drivers/nvme/host/sysfs.c | 7 +- 5 files changed, 256 insertions(+), 12 deletions(-)v2 of this series:
https://lore.kernel.org/r/[email protected]v1 of this series:
https://lore.kernel.org/r/[email protected]latency I/O policy v7:
https://lore.kernel.org/r/[email protected]