Skip to content

nvme: introduce the service-time iopolicy

Guixin Liu linux-nvme 4 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v4 Latest3 patches 2 repliesIn review
  2. v33 patches 2 repliesSuperseded
  3. v23 patches 6 repliesSuperseded
Earlier versions · 1
VersionPostedPatchesRepliesStatus
v12 patches3 repliesSuperseded

Changes in v4 from the cover letter

- Rebased onto nvme-7.4, which adds the iopolicies and path_state attributes and the discovery-subsystem "none" handling.

- The iopolicies ABI entry now lists service-time too. nvme-7.4 already documents the attribute (with a hardcoded list of the three older policies); patch 3 extends that list along with the iopolicy accepted values.

Patches in v4 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/3nvme-multipath: implement service-time iopolicydrivers/nvme/host/multipath.c drivers/nvme/host/nvme.h drivers/nvme/host/sysfs.c+198 -40 replies
2/3nvme-multipath: document service-time iopolicyDocumentation/admin-guide/nvme-multipath.rst+29 -20 replies
3/3nvme: document the service-time iopolicy sysfs attributesDocumentation/ABI/stable/sysfs-nvme+29 -60 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v4 yet.

Read the v4 thread

Cover letter [PATCH v4 0/3] nvme: introduce the service-time iopolicy

The queue-depth iopolicy counts requests, so a 128K I/O weighs the same as a 4K one, and it cannot express that one path is simply faster than another. This series adds a service-time iopolicy for NVMe native multipath, adapted from dm-ps-service-time: it picks the path with the lowest estimated service time, in_flight_bytes / relative_throughput, so balancing follows actual bytes in flight and per-path throughput weights.

Weights live in a per-path relative_throughput sysfs attribute (0-100, default 1, 0 makes the path a standby). Patch 3 documents the new attributes in the stable sysfs ABI; patch 1 carries the benchmark data in its commit log.

This version is rebased onto the nvme-7.4 branch of
git://git.infradead.org/nvme.git.

Changes since v3:

- Rebased onto nvme-7.4, which adds the iopolicies and path_state attributes and the discovery-subsystem "none" handling.

- The iopolicies ABI entry now lists service-time too. nvme-7.4 already documents the attribute (with a hardcoded list of the three older policies); patch 3 extends that list along with the iopolicy accepted values.

Changes since v2:

- in_flight_bytes no longer leaks on failover: the accounted bytes are now released in nvme_failover_req() before blk_steal_bios() clears the request data length. Before, every failover (ANA transition, path error, queue_rq unwind) permanently left the request's bytes in the counter, skewing path selection until a controller reset. Reproduced with an ANA transition during load on the NVMe-oF/TCP test setup: the counter kept 512K of phantom bytes after all I/O completed; with the fix it returns to zero.

- The release clamps at zero instead of a plain subtraction. A request requeued across a controller reset is accounted against the pre-reset counter but completes after nvme_mpath_init_identify() re-zeroed it, and the plain subtraction would wrap negative, making the comparator treat the path as pathologically overloaded. This mirrors atomic_dec_if_positive() in the queue-depth accounting.

- The comparison operates on u64 instead of size_t, so the byte counts and the shift guard keep their magnitude on 32-bit builds, where size_t would truncate the atomic64 reads.

- Added "service-time" to the accepted values of the subsystem iopolicy ABI entry and to the policies listed by the iopolicies ABI entry.

No public v3 was posted; this version folds the review feedback from the v2 discussion and a rebase onto nvme-7.4.

On the adaptive I/O policy dependency Hannes raised during v1: that series has become the latency I/O policy and is still under revision. The two are complementary: the latency policy infers path speed from observed completions, service-time takes configured weights and byte counts.

Guixin Liu (3):
nvme-multipath: implement service-time iopolicy
nvme-multipath: document service-time iopolicy
nvme: document the service-time iopolicy sysfs attributes

 Documentation/ABI/stable/sysfs-nvme          |  35 +++-
 Documentation/admin-guide/nvme-multipath.rst |  31 +++-
 drivers/nvme/host/multipath.c                | 189 ++++++++++++++++++-
 drivers/nvme/host/nvme.h                     |   6 +
 drivers/nvme/host/sysfs.c                    |   7 +-
 5 files changed, 256 insertions(+), 12 deletions(-)

v2 of this series:
https://lore.kernel.org/r/[email protected]

v1 of this series:
https://lore.kernel.org/r/[email protected]

latency I/O policy v7:
https://lore.kernel.org/r/[email protected]

Signature · 1 line
2.43.7