Skip to contentSkip to conversation

nvme: introduce the service-time iopolicy

PATCH In review Guixin Liu 1 participant 6 messages Oct 26 – Nov 16

Subsystems (2)
lore
Share & tools
lore

Guixin Liu

author
#0 lore raw

v4 cover nvme: introduce the service-time iopolicy

The queue-depth iopolicy counts requests, so a 128K I/O weighs the same as a 4K one, and it cannot express that one path is simply faster than another. This series adds a service-time iopolicy for NVMe native multipath, adapted from dm-ps-service-time: it picks the path with the lowest estimated service time, in_flight_bytes / relative_throughput, so balancing follows actual bytes in flight and per-path throughput weights.

Weights live in a per-path relative_throughput sysfs attribute (0-100, default 1, 0 makes the path a standby). Patch 3 documents the new attributes in the stable sysfs ABI; patch 1 carries the benchmark data in its commit log.

This version is rebased onto the nvme-7.4 branch of
git://git.infradead.org/nvme.git.

Changes since v3:

- Rebased onto nvme-7.4, which adds the iopolicies and path_state attributes and the discovery-subsystem "none" handling.

- The iopolicies ABI entry now lists service-time too. nvme-7.4 already documents the attribute (with a hardcoded list of the three older policies); patch 3 extends that list along with the iopolicy accepted values.

Changes since v2:

- in_flight_bytes no longer leaks on failover: the accounted bytes are now released in nvme_failover_req() before blk_steal_bios() clears the request data length. Before, every failover (ANA transition, path error, queue_rq unwind) permanently left the request's bytes in the counter, skewing path selection until a controller reset. Reproduced with an ANA transition during load on the NVMe-oF/TCP test setup: the counter kept 512K of phantom bytes after all I/O completed; with the fix it returns to zero.

- The release clamps at zero instead of a plain subtraction. A request requeued across a controller reset is accounted against the pre-reset counter but completes after nvme_mpath_init_identify() re-zeroed it, and the plain subtraction would wrap negative, making the comparator treat the path as pathologically overloaded. This mirrors atomic_dec_if_positive() in the queue-depth accounting.

- The comparison operates on u64 instead of size_t, so the byte counts and the shift guard keep their magnitude on 32-bit builds, where size_t would truncate the atomic64 reads.

- Added "service-time" to the accepted values of the subsystem iopolicy ABI entry and to the policies listed by the iopolicies ABI entry.

No public v3 was posted; this version folds the review feedback from the v2 discussion and a rebase onto nvme-7.4.

On the adaptive I/O policy dependency Hannes raised during v1: that series has become the latency I/O policy and is still under revision. The two are complementary: the latency policy infers path speed from observed completions, service-time takes configured weights and byte counts.

Guixin Liu (3):
nvme-multipath: implement service-time iopolicy
nvme-multipath: document service-time iopolicy
nvme: document the service-time iopolicy sysfs attributes

 Documentation/ABI/stable/sysfs-nvme          |  35 +++-
 Documentation/admin-guide/nvme-multipath.rst |  31 +++-
 drivers/nvme/host/multipath.c                | 189 ++++++++++++++++++-
 drivers/nvme/host/nvme.h                     |   6 +
 drivers/nvme/host/sysfs.c                    |   7 +-
 5 files changed, 256 insertions(+), 12 deletions(-)

v2 of this series:
https://lore.kernel.org/r/[email protected]

v1 of this series:
https://lore.kernel.org/r/[email protected]

latency I/O policy v7:
https://lore.kernel.org/r/[email protected]

Signature · 1 line
2.43.7
author
#1 lore raw

PATCH v4 1/3 nvme-multipath: implement service-time iopolicy

The queue-depth iopolicy balances paths by the number of in-flight requests. This misjudges the load in two common cases: a 128K request counts the same as a 4K one, and there is no way to tell the selector that one path is simply faster than another. With mixed I/O sizes or paths of different bandwidth, large I/Os keep piling onto the slow path, driving up tail latencies.

Add a "service-time" iopolicy adapted from the dm-ps-service-time path selector. It selects the path with the lowest estimated service time, computed as in_flight_bytes / relative_throughput, so it tracks actual bytes in flight rather than request counts, and users can weight paths according to their throughput.

Each path keeps the total in-flight I/O size and a relative throughput weight as controller state, exposed through two new sysfs attributes on the path namespace devices: in_flight_bytes (read-only) reports the tracked load, and relative_throughput (read-write, 0-100, default 1) sets the weight. Setting relative_throughput to 0 makes the path a standby that is only selected when no path with a positive value is available, matching DM service-time semantics.

As in dm-ps-service-time, the path comparison cross-multiplies to avoid division, and shifts the load counters down when they grow too large to keep the products in range.

Failovers release the accounted bytes in nvme_failover_req() before blk_steal_bios() clears the request data length. The release clamps at zero: a request requeued across a controller reset completes against a counter that nvme_mpath_init_identify() has re-zeroed, and a plain subtraction would underflow, the case that atomic_dec_if_positive() guards against in the queue-depth accounting.

Usage:
echo service-time > /sys/module/nvme_core/parameters/iopolicy echo 4 > /sys/block/nvmeXcYnZ/relative_throughput

Test environment:
Two QEMU/KVM VMs connected via NVMe-oF TCP, 32 vCPU / 32G RAM each. The target exports a 1G RAM disk via nvmet-tcp on two ports, and the initiator connects both ports to form a native NVMe multipath device.

Asymmetric bandwidth is created by shaping the second path with tc-tbf (rate 500mbit, burst 128kb, latency 5ms). Measured single path bandwidth: ~237 MiB/s unshaped vs ~56 MiB/s shaped, a ratio of approximately 4:1.

Test 1 - Uniform 4k random read (iodepth=64, numjobs=4, 15s):

  iopolicy              IOPS   BW(MiB/s)  avg_lat(us)  lat_stddev(us)
  --------------------  -----  ---------  -----------  --------------
  round-robin           59.1k  231        4316         3109
  queue-depth           59.0k  230        4323         3046
  service-time 1:1      59.3k  232        4302         3057
  service-time 4:1      61.0k  238        4179         1294

With uniform I/O size, service-time 1:1 performs similarly to queue-depth since per-byte and per-request tracking are equivalent for fixed-size I/Os. service-time 4:1 with correct throughput weights improves IOPS by 3.4% and reduces latency stddev by 57%.

Test 2 - Mixed random read 4k/128k (bssplit=4k/50:128k/50,

          iodepth=64, numjobs=4, 15s):
  iopolicy              IOPS  BW(MiB/s)  avg_lat(us)  lat_stddev(us)
  --------------------  ----  ---------  -----------  --------------
  round-robin           7689  276        33209        46817
  queue-depth           7642  276        33429        45868
  service-time 1:1      7456  269        34279        47433
  service-time 4:1      7747  277        32998        27102

With mixed I/O sizes on asymmetric paths, queue-depth treats 128K and 4K requests identically when counting in-flight depth, leading to suboptimal load distribution. service-time tracks actual bytes in flight, producing significantly more consistent tail latencies. service-time 4:1 vs queue-depth: latency stddev -41%.

Standby path (relative_throughput=0):
Setting a path's relative_throughput to 0 correctly excludes it from selection. With the slow path set to 0, all I/O is directed to the fast path only (IO split 118052:0), verified via diskstats.

---
 drivers/nvme/host/multipath.c | 189 +++++++++++++++++++++++++++++++++-
 drivers/nvme/host/nvme.h      |   6 ++
 drivers/nvme/host/sysfs.c     |   7 +-
 3 files changed, 198 insertions(+), 4 deletions(-)
drivers/nvme/host/multipath.c +187 -2
index 7010150221992..59e0b14f6d9da 100644
@@ -69,6 +69,7 @@ static const char *nvme_iopolicy_names[] = {
 	[NVME_IOPOLICY_NUMA]	= "numa",
 	[NVME_IOPOLICY_RR]	= "round-robin",
 	[NVME_IOPOLICY_QD]      = "queue-depth",
+	[NVME_IOPOLICY_ST]	= "service-time",
 };
 
 static int iopolicy = NVME_IOPOLICY_NUMA;
@@ -107,7 +108,7 @@ static int nvme_get_iopolicy(char *buf, const struct kernel_param *kp)
 module_param_call(iopolicy, nvme_set_iopolicy, nvme_get_iopolicy,
 	&iopolicy, 0644);
 MODULE_PARM_DESC(iopolicy,
-	"Default multipath I/O policy; 'numa' (default), 'round-robin' or 'queue-depth'");
+	"Default multipath I/O policy; 'numa' (default), 'round-robin', 'queue-depth' or 'service-time'");
 
 void nvme_mpath_default_iopolicy(struct nvme_subsystem *subsys)
 {
@@ -144,6 +145,24 @@ void nvme_mpath_start_freeze(struct nvme_subsystem *subsys)
 			blk_freeze_queue_start(h->disk->queue);
 }
 
+/*
+ * Release in-flight bytes without letting the counter go negative: a
+ * request that sat in a requeue list across a controller reset is
+ * accounted against the pre-reset counter, but completes after
+ * nvme_mpath_init_identify() has zeroed it, in which case the plain
+ * subtraction would underflow.  This mirrors atomic_dec_if_positive()
+ * in the queue-depth accounting.
+ */
+static void nvme_st_release(struct nvme_ctrl *ctrl, unsigned int bytes)
+{
+	s64 old, new;
+
+	do {
+		old = atomic64_read(&ctrl->in_flight_size);
+		new = max_t(s64, 0, old - bytes);
+	} while (!atomic64_try_cmpxchg(&ctrl->in_flight_size, &old, new));
+}
+
 void nvme_failover_req(struct request *req)
 {
 	struct nvme_ns *ns = req->q->queuedata;
@@ -164,6 +183,15 @@ void nvme_failover_req(struct request *req)
 		queue_work(nvme_wq, &ns->ctrl->ana_work);
 	}
 
+	/*
+	 * blk_steal_bios() clears req->__data_len, so release the bytes
+	 * accounted by the service-time policy before stealing the bios.
+	 */
+	if (nvme_req(req)->flags & NVME_MPATH_CNT_ACTIVE_BYTES) {
+		nvme_st_release(ns->ctrl, blk_rq_bytes(req));
+		nvme_req(req)->flags &= ~NVME_MPATH_CNT_ACTIVE_BYTES;
+	}
+
 	spin_lock_irqsave(&ns->head->requeue_lock, flags);
 	for (bio = req->bio; bio; bio = bio->bi_next)
 		bio_set_dev(bio, ns->head->disk->part0);
@@ -179,11 +207,16 @@ void nvme_mpath_start_request(struct request *rq)
 {
 	struct nvme_ns *ns = rq->q->queuedata;
 	struct gendisk *disk = ns->head->disk;
+	int iopolicy = READ_ONCE(ns->head->subsys->iopolicy);
 
-	if ((READ_ONCE(ns->head->subsys->iopolicy) == NVME_IOPOLICY_QD) &&
+	if (iopolicy == NVME_IOPOLICY_QD &&
 	    !(nvme_req(rq)->flags & NVME_MPATH_CNT_ACTIVE)) {
 		atomic_inc(&ns->ctrl->nr_active);
 		nvme_req(rq)->flags |= NVME_MPATH_CNT_ACTIVE;
+	} else if (iopolicy == NVME_IOPOLICY_ST &&
+		   !(nvme_req(rq)->flags & NVME_MPATH_CNT_ACTIVE_BYTES)) {
+		atomic64_add(blk_rq_bytes(rq), &ns->ctrl->in_flight_size);
+		nvme_req(rq)->flags |= NVME_MPATH_CNT_ACTIVE_BYTES;
 	}
 
 	if (!blk_queue_io_stat(disk->queue) ||
@@ -205,6 +238,10 @@ void nvme_mpath_end_request(struct request *rq)
 
 	if (nvme_req(rq)->flags & NVME_MPATH_CNT_ACTIVE)
 		atomic_dec_if_positive(&ns->ctrl->nr_active);
+	if (nvme_req(rq)->flags & NVME_MPATH_CNT_ACTIVE_BYTES) {
+		nvme_st_release(ns->ctrl, blk_rq_bytes(rq));
+		nvme_req(rq)->flags &= ~NVME_MPATH_CNT_ACTIVE_BYTES;
+	}
 
 	if (!(nvme_req(rq)->flags & NVME_MPATH_IO_STATS))
 		return;
@@ -477,6 +514,109 @@ static struct nvme_ns *nvme_queue_depth_path(struct nvme_ns_head *head)
 	return best_opt ? best_opt : best_nonopt;
 }
 
+#define ST_MAX_RELATIVE_THROUGHPUT	100
+#define ST_MAX_RELATIVE_THROUGHPUT_SHIFT	7
+#define ST_MAX_INFLIGHT_SIZE \
+	(U64_MAX >> ST_MAX_RELATIVE_THROUGHPUT_SHIFT)
+
+/*
+ * Compare estimated service time of two paths.
+ * Returns negative if ns1 is better, positive if ns2 is better, 0 if equal.
+ *
+ * Service time = in_flight_size / relative_throughput
+ * Cross-multiply to avoid division.
+ */
+static int nvme_st_compare_load(struct nvme_ns *ns1, struct nvme_ns *ns2)
+{
+	u64 sz1, sz2, st1, st2;
+	unsigned int tp1 = ns1->ctrl->relative_throughput;
+	unsigned int tp2 = ns2->ctrl->relative_throughput;
+
+	sz1 = atomic64_read(&ns1->ctrl->in_flight_size);
+	sz2 = atomic64_read(&ns2->ctrl->in_flight_size);
+
+	/* Case 1: same throughput — compare load directly */
+	if (tp1 == tp2)
+		return (sz1 > sz2) - (sz1 < sz2);
+
+	/*
+	 * Case 2a: same load — prefer higher throughput.
+	 * Case 2b: one path has zero throughput — prefer the other.
+	 */
+	if (sz1 == sz2 || !tp1 || !tp2)
+		return tp2 - tp1;
+
+	/*
+	 * Case 3: general comparison via cross-multiplication.
+	 *   st1 = sz1 / tp1
+	 *   st2 = sz2 / tp2
+	 * Equivalent (since tp > 0):
+	 *   sz1 * tp2 <=> sz2 * tp1
+	 *
+	 * Use comparisons instead of subtractions when returning: the
+	 * intermediate values do not fit into the int return type.
+	 */
+	if (unlikely(sz1 >= ST_MAX_INFLIGHT_SIZE ||
+		     sz2 >= ST_MAX_INFLIGHT_SIZE)) {
+		sz1 >>= ST_MAX_RELATIVE_THROUGHPUT_SHIFT;
+		sz2 >>= ST_MAX_RELATIVE_THROUGHPUT_SHIFT;
+	}
+	st1 = sz1 * tp2;
+	st2 = sz2 * tp1;
+	if (st1 != st2)
+		return (st1 > st2) - (st1 < st2);
+
+	/* Case 4: equal service time — prefer higher throughput */
+	return tp2 - tp1;
+}
+
+static struct nvme_ns *nvme_service_time_path(struct nvme_ns_head *head)
+{
+	struct nvme_ns *best_opt = NULL, *best_nonopt = NULL, *ns;
+	struct nvme_ns *fallback_opt = NULL, *fallback_nonopt = NULL;
+
+	list_for_each_entry_srcu(ns, &head->list, siblings,
+				 srcu_read_lock_held(&head->srcu)) {
+		if (nvme_path_is_disabled(ns))
+			continue;
+
+		/*
+		 * Paths with relative_throughput == 0 are only used when no
+		 * path with a positive value is available (matching DM
+		 * service-time semantics).
+		 */
+		if (!ns->ctrl->relative_throughput) {
+			if (ns->ana_state == NVME_ANA_OPTIMIZED && !fallback_opt)
+				fallback_opt = ns;
+			else if (ns->ana_state == NVME_ANA_NONOPTIMIZED &&
+				 !fallback_nonopt)
+				fallback_nonopt = ns;
+			continue;
+		}
+
+		switch (ns->ana_state) {
+		case NVME_ANA_OPTIMIZED:
+			if (!best_opt ||
+			    nvme_st_compare_load(ns, best_opt) < 0)
+				best_opt = ns;
+			break;
+		case NVME_ANA_NONOPTIMIZED:
+			if (!best_nonopt ||
+			    nvme_st_compare_load(ns, best_nonopt) < 0)
+				best_nonopt = ns;
+			break;
+		default:
+			break;
+		}
+	}
+
+	if (best_opt)
+		return best_opt;
+	if (best_nonopt)
+		return best_nonopt;
+	return fallback_opt ? fallback_opt : fallback_nonopt;
+}
+
 static inline bool nvme_path_is_optimized(struct nvme_ns *ns)
 {
 	return nvme_ctrl_state(ns->ctrl) == NVME_CTRL_LIVE &&
@@ -504,6 +644,8 @@ inline struct nvme_ns *nvme_find_path(struct nvme_ns_head *head)
 		return nvme_queue_depth_path(head);
 	case NVME_IOPOLICY_RR:
 		return nvme_round_robin_path(head);
+	case NVME_IOPOLICY_ST:
+		return nvme_service_time_path(head);
 	default:
 		return nvme_numa_path(head);
 	}
@@ -1192,6 +1334,47 @@ static ssize_t path_state_show(struct device *dev,
 }
 DEVICE_ATTR_RO(path_state);
 
+static ssize_t in_flight_bytes_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
+
+	if (ns->head->subsys->iopolicy != NVME_IOPOLICY_ST)
+		return 0;
+
+	return sysfs_emit(buf, "%lld\n", atomic64_read(&ns->ctrl->in_flight_size));
+}
+DEVICE_ATTR_RO(in_flight_bytes);
+
+static ssize_t relative_throughput_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
+
+	if (ns->head->subsys->iopolicy != NVME_IOPOLICY_ST)
+		return 0;
+
+	return sysfs_emit(buf, "%u\n", ns->ctrl->relative_throughput);
+}
+
+static ssize_t relative_throughput_store(struct device *dev,
+		struct device_attribute *attr, const char *buf, size_t count)
+{
+	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
+	unsigned int val;
+	int ret;
+
+	ret = kstrtouint(buf, 0, &val);
+	if (ret < 0)
+		return ret;
+	if (val > ST_MAX_RELATIVE_THROUGHPUT)
+		return -EINVAL;
+
+	ns->ctrl->relative_throughput = val;
+	return count;
+}
+DEVICE_ATTR_RW(relative_throughput);
+
 static ssize_t numa_nodes_show(struct device *dev, struct device_attribute *attr,
 		char *buf)
 {
@@ -1555,6 +1738,7 @@ void nvme_mpath_init_ctrl(struct nvme_ctrl *ctrl)
 	mutex_init(&ctrl->ana_lock);
 	timer_setup(&ctrl->anatt_timer, nvme_anatt_timeout, 0);
 	INIT_WORK(&ctrl->ana_work, nvme_ana_work);
+	ctrl->relative_throughput = 1;
 }
 
 int nvme_mpath_init_identify(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
@@ -1570,6 +1754,7 @@ int nvme_mpath_init_identify(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
 
 	/* initialize this in the identify path to cover controller resets */
 	atomic_set(&ctrl->nr_active, 0);
+	atomic64_set(&ctrl->in_flight_size, 0);
 
 	if (!ctrl->max_namespaces ||
 	    ctrl->max_namespaces > le32_to_cpu(id->nn)) {
drivers/nvme/host/nvme.h +6 -0
index a8ef62b8e34b8..7ea491d6fee82 100644
@@ -268,6 +268,7 @@ enum {
 	NVME_REQ_USERCMD		= (1 << 1),
 	NVME_MPATH_IO_STATS		= (1 << 2),
 	NVME_MPATH_CNT_ACTIVE		= (1 << 3),
+	NVME_MPATH_CNT_ACTIVE_BYTES	= (1 << 4),
 };
 
 static inline struct nvme_request *nvme_req(struct request *req)
@@ -439,6 +440,8 @@ struct nvme_ctrl {
 	struct timer_list anatt_timer;
 	struct work_struct ana_work;
 	atomic_t nr_active;
+	atomic64_t in_flight_size;
+	u8 relative_throughput;
 #endif
 
 #ifdef CONFIG_NVME_HOST_AUTH
@@ -492,6 +495,7 @@ enum nvme_iopolicy {
 	NVME_IOPOLICY_NUMA,
 	NVME_IOPOLICY_RR,
 	NVME_IOPOLICY_QD,
+	NVME_IOPOLICY_ST,
 };
 
 struct nvme_subsystem {
@@ -1095,6 +1099,8 @@ extern bool multipath;
 extern struct device_attribute dev_attr_ana_grpid;
 extern struct device_attribute dev_attr_ana_state;
 extern struct device_attribute dev_attr_queue_depth;
+extern struct device_attribute dev_attr_in_flight_bytes;
+extern struct device_attribute dev_attr_relative_throughput;
 extern struct device_attribute dev_attr_numa_nodes;
 extern struct device_attribute dev_attr_path_state;
 extern struct device_attribute dev_attr_delayed_removal_secs;
drivers/nvme/host/sysfs.c +5 -2
index 5224597a32f1d..968de4e7d4e66 100644
@@ -262,6 +262,8 @@ static struct attribute *nvme_ns_attrs[] = {
 	&dev_attr_ana_grpid.attr,
 	&dev_attr_ana_state.attr,
 	&dev_attr_queue_depth.attr,
+	&dev_attr_in_flight_bytes.attr,
+	&dev_attr_relative_throughput.attr,
 	&dev_attr_numa_nodes.attr,
 	&dev_attr_path_state.attr,
 	&dev_attr_delayed_removal_secs.attr,
@@ -298,8 +300,9 @@ static umode_t nvme_ns_attrs_are_visible(struct kobject *kobj,
 		if (!nvme_ctrl_use_ana(nvme_get_ns_from_dev(dev)->ctrl))
 			return 0;
 	}
-	if (a == &dev_attr_queue_depth.attr || a == &dev_attr_numa_nodes.attr ||
-	    a == &dev_attr_path_state.attr) {
+	if (a == &dev_attr_queue_depth.attr || a == &dev_attr_path_state.attr ||
+	    a == &dev_attr_in_flight_bytes.attr ||
+	    a == &dev_attr_relative_throughput.attr || a == &dev_attr_numa_nodes.attr) {
 		if (nvme_disk_is_ns_head(dev_to_disk(dev)))
 			return 0;
 	}
Signature · 1 line
2.43.7
author
#2 lore raw

PATCH v4 2/3 nvme-multipath: document service-time iopolicy

In reply to Guixin Liu

Add documentation for the service-time path selection policy: its algorithm, the in_flight_bytes and relative_throughput sysfs attributes, and guidance on when to prefer it over queue-depth.

Document that setting relative_throughput to 0 makes the path a standby that is only selected when no path with a positive value is available.

---
 Documentation/admin-guide/nvme-multipath.rst | 31 ++++++++++++++++++--
 1 file changed, 29 insertions(+), 2 deletions(-)
Documentation/admin-guide/nvme-multipath.rst +29 -2
index 97ca1ccef459b..2acfceaf3d652 100644
@@ -24,8 +24,8 @@ Policies
 
 All policies follow the ANA (Asymmetric Namespace Access) mechanism, meaning
 that when an optimized path is available, it will be chosen over a non-optimized
-one. Current the NVMe multipath policies include numa(default), round-robin and
-queue-depth.
+one. Current the NVMe multipath policies include numa(default), round-robin,
+queue-depth and service-time.
 
 To set the desired policy (e.g., round-robin), use one of the following methods:
    1. echo -n "round-robin" > /sys/module/nvme_core/parameters/iopolicy
@@ -70,3 +70,30 @@ When to use the queue-depth policy:
   1. High load with small I/Os: Effectively balances load across paths when
      the load is high, and I/O operations consist of small, relatively
      fixed-sized requests.
+
+
+Service-Time
+------------
+
+The service-time policy selects the path with the lowest estimated service time.
+It calculates service time as ``in_flight_bytes / relative_throughput`` for each
+path, preferring the path that would complete I/O fastest. Unlike queue-depth
+which counts requests regardless of size, service-time tracks actual bytes in
+flight, making it aware of I/O sizes.
+
+Each path exposes two sysfs attributes under
+``/sys/class/nvme/nvmeX/nvmeXcYnZ/``:
+
+  - ``in_flight_bytes`` (read-only): Current bytes in flight on this path.
+  - ``relative_throughput`` (read-write): Relative throughput weight for this
+    path, default 1. The valid range is 0-100. Set higher values for faster
+    paths. If set to 0, the path is not selected while other paths with
+    positive values are available.
+
+When to use the service-time policy:
+  1. Asymmetric Link Speeds: When paths have different bandwidths, set
+     ``relative_throughput`` proportionally (e.g., 2 for a link twice as fast)
+     to steer more traffic to faster paths.
+  2. Mixed I/O Sizes: When workloads mix small and large I/Os (e.g., 4K and
+     128K), service-time distributes load more accurately than queue-depth
+     because it accounts for actual bytes rather than request count.
Signature · 1 line
2.43.7
author
#3 lore raw

PATCH v4 3/3 nvme: document the service-time iopolicy sysfs attributes

In reply to Guixin Liu

The service-time iopolicy adds two sysfs attributes to the NVMe path namespace devices: in_flight_bytes, reporting the bytes currently in flight on the path's controller, and relative_throughput, the per-path throughput weight used by the service-time estimation.

Document both in the stable sysfs ABI, including the 0-100 value range, the default of 1, and the standby semantics of a relative_throughput of 0. Also add "service-time" to the accepted values of the subsystem iopolicy attribute and to the policies listed by the iopolicies attribute.

---
 Documentation/ABI/stable/sysfs-nvme | 35 ++++++++++++++++++++++++-----
 1 file changed, 29 insertions(+), 6 deletions(-)
Documentation/ABI/stable/sysfs-nvme +29 -6
index 9291595f8102f..9a682d66a3ec2 100644
@@ -347,6 +347,28 @@ Description:
 
 		Requires CONFIG_NVME_MULTIPATH.
 
+What:		/sys/block/nvmeXcYnZ/in_flight_bytes
+What:		/sys/block/nvmeXcYnZ/relative_throughput
+Date:		October 2026
+KernelVersion:	7.5
+Contact:	Guixin Liu <[email protected]>
+Description:
+		in_flight_bytes: Shows the number of bytes currently in
+		flight on this path's controller. Returns empty if
+		iopolicy is not "service-time".
+		Requires CONFIG_NVME_MULTIPATH.
+
+		relative_throughput: Shows or sets the relative
+		throughput weight of this path's controller, which the
+		"service-time" iopolicy uses to estimate the path service
+		time as in_flight_bytes / relative_throughput. The value is
+		an integer in the range 0-100 and defaults to 1; assign
+		higher values to faster paths. A path with a value of 0 is
+		only selected when no path with a positive value is
+		available. Reading returns empty if iopolicy is not
+		"service-time"; writing is accepted regardless of the
+		active policy. Requires CONFIG_NVME_MULTIPATH.
+
 What:		/sys/block/nvmeXnY/delayed_removal_secs
 Date:		May 2025
 KernelVersion:	6.16
@@ -477,9 +499,9 @@ Contact:	Hannes Reinecke <[email protected]>
 Description:
 		Shows or sets the multipath I/O path selection policy for
 		this subsystem. Accepted values: "numa", "round-robin",
-		"queue-depth". Changing the policy clears all current path
-		selections. Only available when CONFIG_NVME_MULTIPATH is
-		enabled.
+		"queue-depth", "service-time". Changing the policy clears
+		all current path selections. Only available when
+		CONFIG_NVME_MULTIPATH is enabled.
 
 		A discovery subsystem has no namespaces and hence no I/O
 		paths to select from: reading returns "none" and writing
@@ -492,9 +514,10 @@ Contact:	Laurence Oberman <[email protected]>
 Description:
 		Shows the I/O path selection policies that may be written
 		to the iopolicy attribute of this subsystem, separated by
-		spaces: "numa round-robin queue-depth". A discovery
-		subsystem accepts no policy at all and reports "none".
-		Only available when CONFIG_NVME_MULTIPATH is enabled.
+		spaces: "numa round-robin queue-depth service-time". A
+		discovery subsystem accepts no policy at all and reports
+		"none". Only available when CONFIG_NVME_MULTIPATH is
+		enabled.
 
 What:		/sys/class/nvme-subsystem/nvme-subsysX/subsystype
 Date:		September 2021
Signature · 1 line
2.43.7
author
#4 lore raw
In reply to Guixin Liu

Hi guys,
  All the issues Sashiko raised are addressed in this v4, and it looks like Sashiko has no further comments on the current version now. Please review this series, all comments are welcome. Or if can tell me this approach is reasonable, I'll keep iterating on the implementation details.

Best Regards,
Guixin Liu

Quoted 81 lines · Guixin Liu

The queue-depth iopolicy counts requests, so a 128K I/O weighs the same as a 4K one, and it cannot express that one path is simply faster than another. This series adds a service-time iopolicy for NVMe native multipath, adapted from dm-ps-service-time: it picks the path with the lowest estimated service time, in_flight_bytes / relative_throughput, so balancing follows actual bytes in flight and per-path throughput weights.

Weights live in a per-path relative_throughput sysfs attribute (0-100, default 1, 0 makes the path a standby). Patch 3 documents the new attributes in the stable sysfs ABI; patch 1 carries the benchmark data in its commit log.

This version is rebased onto the nvme-7.4 branch of
git://git.infradead.org/nvme.git.

Changes since v3:

- Rebased onto nvme-7.4, which adds the iopolicies and path_state attributes and the discovery-subsystem "none" handling.

- The iopolicies ABI entry now lists service-time too. nvme-7.4 already documents the attribute (with a hardcoded list of the three older policies); patch 3 extends that list along with the iopolicy accepted values.

Changes since v2:

- in_flight_bytes no longer leaks on failover: the accounted bytes are now released in nvme_failover_req() before blk_steal_bios() clears the request data length. Before, every failover (ANA transition, path error, queue_rq unwind) permanently left the request's bytes in the counter, skewing path selection until a controller reset. Reproduced with an ANA transition during load on the NVMe-oF/TCP test setup: the counter kept 512K of phantom bytes after all I/O completed; with the fix it returns to zero.

- The release clamps at zero instead of a plain subtraction. A request requeued across a controller reset is accounted against the pre-reset counter but completes after nvme_mpath_init_identify() re-zeroed it, and the plain subtraction would wrap negative, making the comparator treat the path as pathologically overloaded. This mirrors atomic_dec_if_positive() in the queue-depth accounting.

- The comparison operates on u64 instead of size_t, so the byte counts and the shift guard keep their magnitude on 32-bit builds, where size_t would truncate the atomic64 reads.

- Added "service-time" to the accepted values of the subsystem iopolicy ABI entry and to the policies listed by the iopolicies ABI entry.

No public v3 was posted; this version folds the review feedback from the v2 discussion and a rebase onto nvme-7.4.

On the adaptive I/O policy dependency Hannes raised during v1: that series has become the latency I/O policy and is still under revision. The two are complementary: the latency policy infers path speed from observed completions, service-time takes configured weights and byte counts.

Guixin Liu (3):
nvme-multipath: implement service-time iopolicy
nvme-multipath: document service-time iopolicy
nvme: document the service-time iopolicy sysfs attributes

 Documentation/ABI/stable/sysfs-nvme          |  35 +++-
 Documentation/admin-guide/nvme-multipath.rst |  31 +++-
 drivers/nvme/host/multipath.c                | 189 ++++++++++++++++++-
 drivers/nvme/host/nvme.h                     |   6 +
 drivers/nvme/host/sysfs.c                    |   7 +-
 5 files changed, 256 insertions(+), 12 deletions(-)

v2 of this series:
https://lore.kernel.org/r/[email protected]

v1 of this series:
https://lore.kernel.org/r/[email protected]

latency I/O policy v7:
https://lore.kernel.org/r/[email protected]

author
#5 lore raw
In reply to Guixin Liu

Hi,
  A gentle ping on this series, all comments are welcome. If you have any concerns about this feature, please let me know. Best Regards, Guixin Liu

Quoted 81 lines · Guixin Liu

The queue-depth iopolicy counts requests, so a 128K I/O weighs the same as a 4K one, and it cannot express that one path is simply faster than another. This series adds a service-time iopolicy for NVMe native multipath, adapted from dm-ps-service-time: it picks the path with the lowest estimated service time, in_flight_bytes / relative_throughput, so balancing follows actual bytes in flight and per-path throughput weights.

Weights live in a per-path relative_throughput sysfs attribute (0-100, default 1, 0 makes the path a standby). Patch 3 documents the new attributes in the stable sysfs ABI; patch 1 carries the benchmark data in its commit log.

This version is rebased onto the nvme-7.4 branch of
git://git.infradead.org/nvme.git.

Changes since v3:

- Rebased onto nvme-7.4, which adds the iopolicies and path_state attributes and the discovery-subsystem "none" handling.

- The iopolicies ABI entry now lists service-time too. nvme-7.4 already documents the attribute (with a hardcoded list of the three older policies); patch 3 extends that list along with the iopolicy accepted values.

Changes since v2:

- in_flight_bytes no longer leaks on failover: the accounted bytes are now released in nvme_failover_req() before blk_steal_bios() clears the request data length. Before, every failover (ANA transition, path error, queue_rq unwind) permanently left the request's bytes in the counter, skewing path selection until a controller reset. Reproduced with an ANA transition during load on the NVMe-oF/TCP test setup: the counter kept 512K of phantom bytes after all I/O completed; with the fix it returns to zero.

- The release clamps at zero instead of a plain subtraction. A request requeued across a controller reset is accounted against the pre-reset counter but completes after nvme_mpath_init_identify() re-zeroed it, and the plain subtraction would wrap negative, making the comparator treat the path as pathologically overloaded. This mirrors atomic_dec_if_positive() in the queue-depth accounting.

- The comparison operates on u64 instead of size_t, so the byte counts and the shift guard keep their magnitude on 32-bit builds, where size_t would truncate the atomic64 reads.

- Added "service-time" to the accepted values of the subsystem iopolicy ABI entry and to the policies listed by the iopolicies ABI entry.

No public v3 was posted; this version folds the review feedback from the v2 discussion and a rebase onto nvme-7.4.

On the adaptive I/O policy dependency Hannes raised during v1: that series has become the latency I/O policy and is still under revision. The two are complementary: the latency policy infers path speed from observed completions, service-time takes configured weights and byte counts.

Guixin Liu (3):
nvme-multipath: implement service-time iopolicy
nvme-multipath: document service-time iopolicy
nvme: document the service-time iopolicy sysfs attributes

 Documentation/ABI/stable/sysfs-nvme          |  35 +++-
 Documentation/admin-guide/nvme-multipath.rst |  31 +++-
 drivers/nvme/host/multipath.c                | 189 ++++++++++++++++++-
 drivers/nvme/host/nvme.h                     |   6 +
 drivers/nvme/host/sysfs.c                    |   7 +-
 5 files changed, 256 insertions(+), 12 deletions(-)

v2 of this series:
https://lore.kernel.org/r/[email protected]

v1 of this series:
https://lore.kernel.org/r/[email protected]

latency I/O policy v7:
https://lore.kernel.org/r/[email protected]