Skip to content

nvme: Refactor and expose per-controller timeout configuration

Maurizio Lombardi linux-nvme 5 versions – Applied

Status from this message: Thanks, applied to newly created branch, nvme-7.2.

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v5 Latest7 patches 13 repliesApplied
  2. v49 patches 37 repliesSuperseded
  3. v38 patches 29 repliesSuperseded
Earlier versions · 2
VersionPostedPatchesRepliesStatus
v25 patches0 repliesSuperseded
v1 RFC5 patches19 repliesSuperseded

Patches in v5 Applied

#Subject and filesSizeRepliesApplied
1/7nvme: Let the blocklayer set timeouts for requestsdrivers/nvme/host/core.c+0 -21 reply
2/7nvme: remove redundant timeout argument from nvme_wait_freeze_timeoutdrivers/nvme/host/apple.c drivers/nvme/host/core.c drivers/nvme/host/nvme.h +3 more+7 -62 replies
3/7nvme: add sysfs attribute to change admin timeout per nvme controllerdrivers/nvme/host/core.c drivers/nvme/host/nvme.h drivers/nvme/host/pci.c +1 more+44 -14 replies
4/7nvme: add sysfs attribute to change IO timeout per controllerdrivers/nvme/host/core.c drivers/nvme/host/nvme.h drivers/nvme/host/sysfs.c+51 -12 replies
5/7nvme-core: align fabrics_q teardown with admin_q in nvme_free_ctrldrivers/nvme/host/core.c+3 -30 replies
6/7nvmet-loop: do not alloc admin tag set during resetdrivers/nvme/target/loop.c+16 -150 replies
7/7nvme-core: warn on allocating admin tag set with existing queuedrivers/nvme/host/core.c+1 -63 replies

Reviews trailers per patch

All 7 patches have a Reviewed-by or Acked-by.

PatchMohamed KhalfellaChristoph HellwigDaniel WagnerSagi GrimbergHannes Reinecke
1/7R-b R-b R-b R-b
2/7R-b R-b R-b R-b
3/7R-b R-b R-b R-b R-b
4/7R-b R-b R-b R-b
5/7R-b R-b R-b R-b
6/7R-b R-b R-b R-b
7/7R-b R-b R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 5 reviewers.

Read the v5 thread

Cover letter [PATCH V5 0/7] nvme: Refactor and expose per-controller timeout configuration

This patchset tries to address some limitations in how the NVMe driver handles command timeouts.
Currently, the driver relies heavily on global module parameters (NVME_IO_TIMEOUT and NVME_ADMIN_TIMEOUT), making it difficult for users to tune timeouts for specific controllers that may have very different characteristics. Also, in some cases, manual changes to sysfs timeout values are ignored by the driver logic.

For example this patchset removes the unconditional timeout assignment in nvme_init_request. This allows the block layer to correctly apply the request queue's timeout settings, ensuring that user-initiated changes via sysfs are actually respected for all requests.

It introduces new sysfs attributes (admin_timeout and io_timeout) to the NVMe controller. This allows users to configure distinct timeout requirements for different controllers rather than relying on global module parameters.

Some examples:

Changes to the controller's io_timeout gets propagated to all the associated namespaces' queues:

# find /sys -name 'io_timeout' 
/sys/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0c0n1/queue/io_timeout
/sys/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0c0n2/queue/io_timeout
/sys/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0c0n3/queue/io_timeout
/sys/devices/virtual/nvme-fabrics/ctl/nvme0/io_timeout
# echo 27000 > /sys/devices/virtual/nvme-fabrics/ctl/nvme0/io_timeout
# cat /sys/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0c0n1/queue/io_timeout
27000
# cat /sys/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0c0n2/queue/io_timeout
27000
# cat /sys/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0c0n3/queue/io_timeout
27000

When adding a namespace target-side, the io_timeout is inherited from the controller's preferred timeout:

* target side *

# nvmetcli
/> cd subsystems/test-nqn/namespaces/4 
/subsystems/t.../namespaces/4> enable
The Namespace has been enabled.

************

* Host-side *
nvme nvme0: rescanning namespaces.

# find /sys -name 'io_timeout' 
/sys/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0c0n1/queue/io_timeout
/sys/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0c0n2/queue/io_timeout
/sys/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0c0n3/queue/io_timeout
/sys/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0c0n4/queue/io_timeout <-- new namespace
/sys/devices/virtual/nvme-fabrics/ctl/nvme0/io_timeout
# cat /sys/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0c0n4/queue/io_timeout
27000
***********

io_timeout and admin_timeout module parameters are used as default values for new controllers:

# nvme connect -t tcp -a 10.37.153.138 -s 8000 -n test-nqn2
connecting to device: nvme1
# cat /sys/devices/virtual/nvme-fabrics/ctl/nvme1/nvme1c1n1/queue/io_timeout
30000
# cat /sys/devices/virtual/nvme-fabrics/ctl/nvme1/admin_timeout
60000

V5: - A few cosmethical changes (80-char lines etc.)
- Move the ex patch 6 ("nvme: use per controller timeout waits") earlier in the serie, change it to just dropping the timeout parameter.
- ex patch 4 ("nvme: pci: use admin queue timeout")
eliminated, its nvme-pci changes are now part of patch 3
- ex patch 3 ("nvme: fix race condition between connected uevent")
has been already merged in linux-nvme tree.
- patch 7: change it to use "if (ctrl->fabrics_q)" instead of
"if (ctrl->ops->flags & NVME_F_FABRICS)" where it makes sense.
- In the commit messages of patch 3 and patch 4, I clarify that changing the timeouts of fabrics_q and connect_q queues is intended.

V4: - Refactor the nvmet-loop patch as suggested by Daniel Wagner
- Add a patch to fix a potential race condition between connected uevent and STARTED_ONCE controller's flag.

V3: - Rebase on top of nvme 7.1 branch
- add an admin_timeout variable to nvme_ctrl structure
- Wait until the controller has reached the LIVE state
for the first time before allowing the user to modify
the timeouts, this prevents the dereferencing of the admin_q, fabrics_q and connect_q queues before their initialization.
- move blk_put_queue(fabrics_q) to nvme_free_ctrl() to align it to admin_q teardown
- modify nvmet-loop to avoid deleting and re-creating the admin_q queue when the controller enters the resetting state.
- add a warning if nvme_alloc_admin_tag_set() is called twice against the same controller

V2: - Drop the RFC tag
- apply the timeout settings to fabrics_q and connect_q too
- Code style fixes
- remove unnecessary check for null admin_q in __nvme_delete_io_queues()
- Use DEVICE_ATTR() macro

Maurizio Lombardi (6):
nvme: remove redundant timeout argument from nvme_wait_freeze_timeout
nvme: add sysfs attribute to change admin timeout per nvme controller
nvme: add sysfs attribute to change IO timeout per controller
nvme-core: align fabrics_q teardown with admin_q in nvme_free_ctrl
nvmet-loop: do not alloc admin tag set during reset
nvme-core: warn on allocating admin tag set with existing queue

Maximilian Heyne (1):
nvme: Let the blocklayer set timeouts for requests

 drivers/nvme/host/apple.c  |  2 +-
 drivers/nvme/host/core.c   | 21 ++++-----
 drivers/nvme/host/nvme.h   |  4 +-
 drivers/nvme/host/pci.c    |  4 +-
 drivers/nvme/host/rdma.c   |  2 +-
 drivers/nvme/host/sysfs.c  | 88 ++++++++++++++++++++++++++++++++++++++
 drivers/nvme/host/tcp.c    |  2 +-
 drivers/nvme/target/loop.c | 31 +++++++-------
 8 files changed, 121 insertions(+), 33 deletions(-)
Signature · 1 line
2.54.0