nvme: Refactor and expose per-controller timeout configuration
Subsystems (2)
cover nvme: Refactor and expose per-controller timeout configuration
This patchset tries to address some limitations in how the NVMe driver handles command timeouts.
Currently, the driver relies heavily on global module parameters (NVME_IO_TIMEOUT and NVME_ADMIN_TIMEOUT), making it difficult for users to tune timeouts for specific controllers that may have very different characteristics. Also, in some cases, manual changes to sysfs timeout values are ignored by the driver logic.
For example this patchset removes the unconditional timeout assignment in nvme_init_request. This allows the block layer to correctly apply the request queue's timeout settings, ensuring that user-initiated changes via sysfs are actually respected for all requests.
It introduces new sysfs attributes (admin_timeout and io_timeout) to the NVMe controller. This allows users to configure distinct timeout requirements for different controllers rather than relying on global module parameters.
Some examples:
Changes to the controller's io_timeout gets propagated to all the associated namespaces' queues:
# find /sys -name 'io_timeout' /sys/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0c0n1/queue/io_timeout /sys/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0c0n2/queue/io_timeout /sys/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0c0n3/queue/io_timeout /sys/devices/virtual/nvme-fabrics/ctl/nvme0/io_timeout
# echo 27000 > /sys/devices/virtual/nvme-fabrics/ctl/nvme0/io_timeout # cat /sys/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0c0n1/queue/io_timeout 27000 # cat /sys/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0c0n2/queue/io_timeout 27000 # cat /sys/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0c0n3/queue/io_timeout 27000
When adding a namespace target-side, the io_timeout is inherited from the controller's preferred timeout:
* target side *
# nvmetcli /> cd subsystems/test-nqn/namespaces/4 /subsystems/t.../namespaces/4> enable The Namespace has been enabled.
************
* Host-side *
nvme nvme0: rescanning namespaces.
# find /sys -name 'io_timeout' /sys/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0c0n1/queue/io_timeout /sys/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0c0n2/queue/io_timeout /sys/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0c0n3/queue/io_timeout /sys/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0c0n4/queue/io_timeout <-- new namespace /sys/devices/virtual/nvme-fabrics/ctl/nvme0/io_timeout
# cat /sys/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0c0n4/queue/io_timeout 27000 ***********
io_timeout and admin_timeout module parameters are used as default values for new controllers:
# nvme connect -t tcp -a 10.37.153.138 -s 8000 -n test-nqn2 connecting to device: nvme1
# cat /sys/devices/virtual/nvme-fabrics/ctl/nvme1/nvme1c1n1/queue/io_timeout 30000 # cat /sys/devices/virtual/nvme-fabrics/ctl/nvme1/admin_timeout 60000
David Jeffery (1):
nvme: pci: use admin queue timeout over NVME_ADMIN_TIMEOUT
Heyne, Maximilian (1):
nvme: Let the blocklayer set timeouts for requests
Maurizio Lombardi (3):
nvme: add sysfs attribute to change admin timeout per nvme controller
nvme: add sysfs attribute to change IO timeout per nvme controller
nvme: use per controller timeout waits over depending on global
default
drivers/nvme/host/apple.c | 4 +-- drivers/nvme/host/core.c | 11 ++++--- drivers/nvme/host/nvme.h | 3 +- drivers/nvme/host/pci.c | 5 +-- drivers/nvme/host/rdma.c | 2 +- drivers/nvme/host/sysfs.c | 69 +++++++++++++++++++++++++++++++++++++++ drivers/nvme/host/tcp.c | 2 +- 7 files changed, 84 insertions(+), 12 deletions(-)
Signature · 1 line
2.53.0
RFC PATCH 1/5 nvme: Let the blocklayer set timeouts for requests
From: "Heyne, Maximilian" <[email protected]>
When initializing an nvme request which is about to be send to the block layer, we do not need to initialize its timeout. If it's left uninitialized at 0 the block layer will use the request queue's timeout in blk_add_timer (via nvme_start_request which is called from nvme_*_queue_rq). These timeouts are setup to either NVME_IO_TIMEOUT or NVME_ADMIN_TIMEOUT when the request queues were created.
Because the io_timeout of the IO queues can be modified via sysfs, the following situation can occur:
1) NVME_IO_TIMEOUT = 30 (default module parameter)
2) nvme1n1 is probed. IO queues default timeout is 30 s
3) manually change the IO timeout to 90 s
echo 90000 > /sys/class/nvme/nvme1/nvme1n1/queue/io_timeout
4) Any call of __submit_sync_cmd on nvme1n1 to an IO queue will issue commands with the 30 s timeout instead of the wanted 90 s which might be more suitable for this device.
Commit 470e900c8036 ("nvme: refactor nvme_alloc_request") silently changed the behavior for ioctl's already because it unconditionally overrides the request's timeout that was set in nvme_init_request. If it was unset by the user of the ioctl if will be overridden with 0 meaning the block layer will pick the request queue's IO timeout.
Following up on that, this patch further improves the consistency of IO timeout usage. However, there are still uses of NVME_IO_TIMEOUT which could be inconsistent with what is set in the device's request_queue by the user.
--- drivers/nvme/host/core.c | 2 -- 1 file changed, 2 deletions(-)
drivers/nvme/host/core.c +0 -2
@@ -724,10 +724,8 @@ void nvme_init_request(struct request *req, struct nvme_command *cmd)struct nvme_ns *ns = req->q->disk->private_data; logging_enabled = ns->head->passthru_err_log_enabled; - req->timeout =NVME_IO_TIMEOUT ; }else {/* no queuedata implies admin queue */ logging_enabled = nr->ctrl->passthru_err_log_enabled; - req->timeout =NVME_ADMIN_TIMEOUT ; }if (!logging_enabled)
Signature · 1 line
2.53.0
Re: [PATCH RFC 1/5] nvme: Let the blocklayer set timeouts for requests
Quoted 55 lines · Maurizio Lombardi
From: "Heyne, Maximilian" <[email protected]>
When initializing an nvme request which is about to be send to the block layer, we do not need to initialize its timeout. If it's left uninitialized at 0 the block layer will use the request queue's timeout in
blk_add_timer(vianvme_start_requestwhich is called from nvme_*_queue_rq). These timeouts are setup to eitherNVME_IO_TIMEOUTorNVME_ADMIN_TIMEOUTwhen the request queues were created.Because the
io_timeoutof the IO queues can be modified via sysfs, the following situation can occur:1)
NVME_IO_TIMEOUT= 30 (default module parameter)
2) nvme1n1 is probed. IO queues default timeout is 30 s
3) manually change the IO timeout to 90 s
echo 90000 >/sys/class/nvme/nvme1/nvme1n1/queue/io_timeout
4) Any call of__submit_sync_cmdon nvme1n1 to an IO queue will issue commands with the 30 s timeout instead of the wanted 90 s which might be more suitable for this device.Commit
470e900c8036("nvme: refactornvme_alloc_request") silently changed the behavior for ioctl's already because it unconditionally overrides the request's timeout that was set innvme_init_request. If it was unset by the user of the ioctl if will be overridden with 0 meaning the block layer will pick the request queue's IO timeout.Following up on that, this patch further improves the consistency of IO timeout usage. However, there are still uses of
NVME_IO_TIMEOUTwhich could be inconsistent with what is set in the device'srequest_queueby the user.Signed-off-by Maximilian Heyne <[email protected]>--- drivers/nvme/host/core.c | 2 -- 1 file changed, 2 deletions(-)drivers/nvme/host/core.c +0
-2@@ -724,10 +724,8 @@ void nvme_init_request(struct request *req, struct nvme_command *cmd)struct nvme_ns *ns = req->q->disk->private_data; logging_enabled = ns->head->passthru_err_log_enabled; - req->timeout =NVME_IO_TIMEOUT ; }else {/* no queuedata implies admin queue */ logging_enabled = nr->ctrl->passthru_err_log_enabled; - req->timeout =NVME_ADMIN_TIMEOUT ; }if (!logging_enabled)Signature · 1 line
2.53.0
I wonder what was the impact of these lines given that req->timeout is set to 0 by blk_mq_rq_ctx_init() when the request is allocated?
Re: [PATCH RFC 1/5] nvme: Let the blocklayer set timeouts for requests
Quoted 51 lines · Mohamed Khalfella
Quoted 50 lines · Maurizio Lombardi
From: "Heyne, Maximilian" <[email protected]>
When initializing an nvme request which is about to be send to the block layer, we do not need to initialize its timeout. If it's left uninitialized at 0 the block layer will use the request queue's timeout in
blk_add_timer(vianvme_start_requestwhich is called from nvme_*_queue_rq). These timeouts are setup to eitherNVME_IO_TIMEOUTorNVME_ADMIN_TIMEOUTwhen the request queues were created.Because the
io_timeoutof the IO queues can be modified via sysfs, the following situation can occur:1)
NVME_IO_TIMEOUT= 30 (default module parameter)
2) nvme1n1 is probed. IO queues default timeout is 30 s
3) manually change the IO timeout to 90 s
echo 90000 >/sys/class/nvme/nvme1/nvme1n1/queue/io_timeout
4) Any call of__submit_sync_cmdon nvme1n1 to an IO queue will issue commands with the 30 s timeout instead of the wanted 90 s which might be more suitable for this device.Commit
470e900c8036("nvme: refactornvme_alloc_request") silently changed the behavior for ioctl's already because it unconditionally overrides the request's timeout that was set innvme_init_request. If it was unset by the user of the ioctl if will be overridden with 0 meaning the block layer will pick the request queue's IO timeout.Following up on that, this patch further improves the consistency of IO timeout usage. However, there are still uses of
NVME_IO_TIMEOUTwhich could be inconsistent with what is set in the device'srequest_queueby the user.Signed-off-by Maximilian Heyne <[email protected]>--- drivers/nvme/host/core.c | 2 -- 1 file changed, 2 deletions(-)drivers/nvme/host/core.c +0
-2@@ -724,10 +724,8 @@ void nvme_init_request(struct request *req, struct nvme_command *cmd)struct nvme_ns *ns = req->q->disk->private_data; logging_enabled = ns->head->passthru_err_log_enabled; - req->timeout =NVME_IO_TIMEOUT ; }else {/* no queuedata implies admin queue */ logging_enabled = nr->ctrl->passthru_err_log_enabled; - req->timeout =NVME_ADMIN_TIMEOUT ; }
Quoted 9 lines
Quoted 5 lines
drivers/nvme/host/core.c +0
-0if (!logging_enabled)Signature · 1 line
2.53.0I wonder what was the impact of these lines given that
req->timeoutis set to 0 byblk_mq_rq_ctx_init()when the request is allocated?
Looking through the nvme code. blk_mq_rq_ctx_init is always called (indirectly, e.g. via blk_mq_alloc_request) before nvme_init_request and therefore the timeouts are set to NVME_IO_TIMEOUT/NVME_ADMIN_TIMEOUT. Now with these lines removed the timeout is 0 which means the request_queue's rq_timeout will be used, see blk_add_timer.
Amazon Web Services Development Center Germany GmbH Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christof Hellmis, Andreas Stieger Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597
RFC PATCH 2/5 nvme: add sysfs attribute to change admin timeout per nvme controller
In reply to Maurizio LombardiCurrently, there is no method to adjust the timeout values on a per controller basis with nvme admin queues. Add an admin_timeout attribute to nvme so that different nvme controllers which may have different timeout requirements can have custom admin timeouts set.
--- drivers/nvme/host/sysfs.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
drivers/nvme/host/sysfs.c +31 -0
@@ -601,6 +601,36 @@ static ssize_t dctype_show(struct device *dev, }static DEVICE_ATTR_RO (dctype); +static ssize_t nvme_admin_timeout_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); + +return sysfs_emit (buf,"%u\n" , +jiffies_to_msecs (ctrl->admin_q->rq_timeout)); +} + +static ssize_t nvme_admin_timeout_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); +u32 timeout; +int err; + + err =kstrtou32 (buf,10 , &timeout); +if (err || !timeout) +return -EINVAL ; + +blk_queue_rq_timeout (ctrl->admin_q,msecs_to_jiffies (timeout)); + +return count; +} + +static struct device_attribute dev_attr_admin_timeout = \ +__ATTR (admin_timeout,S_IRUGO |S_IWUSR , \ + nvme_admin_timeout_show, nvme_admin_timeout_store); +#ifdef CONFIG_NVME_HOST_AUTH static ssize_t nvme_ctrl_dhchap_secret_show (struct device *dev,struct device_attribute *attr,char *buf)
@@ -742,6 +772,7 @@ static struct attribute *nvme_dev_attrs[] = { &dev_attr_kato.attr, &dev_attr_cntrltype.attr, &dev_attr_dctype.attr, + &dev_attr_admin_timeout.attr, #ifdef CONFIG_NVME_HOST_AUTH &dev_attr_dhchap_secret.attr, &dev_attr_dhchap_ctrl_secret.attr,
Signature · 1 line
2.53.0
Re: [PATCH RFC 2/5] nvme: add sysfs attribute to change admin timeout per nvme controller
Quoted 7 lines · Maurizio Lombardi
Currently, there is no method to adjust the timeout values on a per controller basis with nvme admin queues. Add an
admin_timeoutattribute to nvme so that different nvme controllers which may have different timeout requirements can have custom admin timeouts set.Signed-off-by Maurizio Lombardi <[email protected]>
Quoted 57 lines
--- drivers/nvme/host/sysfs.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)drivers/nvme/host/sysfs.c +31
-0@@ -601,6 +601,36 @@ static ssize_t dctype_show(struct device *dev, }static DEVICE_ATTR_RO (dctype); +static ssize_t nvme_admin_timeout_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); + +return sysfs_emit (buf,"%u\n" , +jiffies_to_msecs (ctrl->admin_q->rq_timeout)); +} + +static ssize_t nvme_admin_timeout_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); +u32 timeout; +int err; + + err =kstrtou32 (buf,10 , &timeout); +if (err || !timeout) +return -EINVAL ; + +blk_queue_rq_timeout (ctrl->admin_q,msecs_to_jiffies (timeout)); + +return count; +} + +static struct device_attribute dev_attr_admin_timeout = \ +__ATTR (admin_timeout,S_IRUGO |S_IWUSR , \ + nvme_admin_timeout_show, nvme_admin_timeout_store); +#ifdef CONFIG_NVME_HOST_AUTH static ssize_t nvme_ctrl_dhchap_secret_show (struct device *dev,struct device_attribute *attr,char *buf)@@ -742,6 +772,7 @@ static struct attribute *nvme_dev_attrs[] = { &dev_attr_kato.attr, &dev_attr_cntrltype.attr, &dev_attr_dctype.attr, + &dev_attr_admin_timeout.attr, #ifdef CONFIG_NVME_HOST_AUTH &dev_attr_dhchap_secret.attr, &dev_attr_dhchap_ctrl_secret.attr,Signature · 1 line
2.53.0
Re: [PATCH RFC 2/5] nvme: add sysfs attribute to change admin timeout per nvme controller
In reply to Maurizio LombardiQuoted 7 lines · Maurizio Lombardi
Currently, there is no method to adjust the timeout values on a per controller basis with nvme admin queues. Add an
admin_timeoutattribute to nvme so that different nvme controllers which may have different timeout requirements can have custom admin timeouts set.Signed-off-by Maurizio Lombardi <[email protected]>
What would it take to have a "queue" directory representing the admin_queue for an nvme controller like we have "queue" for the io queues of the namespaces? This way we could change/inspect more than just the admin_timeout for the admin queue. Or does this not make any sense?
Quoted 31 lines
--- drivers/nvme/host/sysfs.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)drivers/nvme/host/sysfs.c +19
-0@@ -601,6 +601,36 @@ static ssize_t dctype_show(struct device *dev, }static DEVICE_ATTR_RO (dctype); +static ssize_t nvme_admin_timeout_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); + +return sysfs_emit (buf,"%u\n" , +jiffies_to_msecs (ctrl->admin_q->rq_timeout)); +} + +static ssize_t nvme_admin_timeout_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); +u32 timeout; +int err; + + err =kstrtou32 (buf,10 , &timeout); +if (err || !timeout)
drivers/nvme/host/sysfs.c static ssize_t dctype_show(struct device *dev, · line 632
Quoted 10 lines
drivers/nvme/host/sysfs.c +10
-0+return -EINVAL ; + +blk_queue_rq_timeout (ctrl->admin_q,msecs_to_jiffies (timeout)); + +return count; +} + +static struct device_attribute dev_attr_admin_timeout = \ +__ATTR (admin_timeout,S_IRUGO |S_IWUSR , \ + nvme_admin_timeout_show, nvme_admin_timeout_store);
Why not
DEVICE_ATTR(admin_timeout, S_IRUGO | S_IWUSR, nvme_admin_timeout_show nvme_admin_timeout_store);
?
Quoted 16 lines
drivers/nvme/host/sysfs.c +2
-0+#ifdef CONFIG_NVME_HOST_AUTH static ssize_t nvme_ctrl_dhchap_secret_show (struct device *dev,struct device_attribute *attr,char *buf)@@ -742,6 +772,7 @@ static struct attribute *nvme_dev_attrs[] = { &dev_attr_kato.attr, &dev_attr_cntrltype.attr, &dev_attr_dctype.attr, + &dev_attr_admin_timeout.attr, #ifdef CONFIG_NVME_HOST_AUTH &dev_attr_dhchap_secret.attr, &dev_attr_dhchap_ctrl_secret.attr,Signature · 1 line
2.53.0
Amazon Web Services Development Center Germany GmbH Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christof Hellmis, Andreas Stieger Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597
Re: [PATCH RFC 2/5] nvme: add sysfs attribute to change admin timeout per nvme controller
Quoted 11 lines · Maximilian Heyne
drivers/nvme/host/sysfs.c static ssize_t dctype_show(struct device *dev, · line 632
Quoted 4 lines
drivers/nvme/host/sysfs.c +4
-0+ +static struct device_attribute dev_attr_admin_timeout = \ +__ATTR (admin_timeout,S_IRUGO |S_IWUSR , \ + nvme_admin_timeout_show, nvme_admin_timeout_store);Why not
DEVICE_ATTR(admin_timeout, S_IRUGO | S_IWUSR, nvme_admin_timeout_show nvme_admin_timeout_store);?
No valid reason, I must have copy-pasted it from the old patchset sent by David Jeffery.
I will fix it, thanks.
Maurizio
Re: [PATCH RFC 2/5] nvme: add sysfs attribute to change admin timeout per nvme controller
In reply to Maurizio LombardiQuoted 39 lines · Maurizio Lombardi
Currently, there is no method to adjust the timeout values on a per controller basis with nvme admin queues. Add an
admin_timeoutattribute to nvme so that different nvme controllers which may have different timeout requirements can have custom admin timeouts set.Signed-off-by Maurizio Lombardi <[email protected]>--- drivers/nvme/host/sysfs.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)drivers/nvme/host/sysfs.c +20
-0@@ -601,6 +601,36 @@ static ssize_t dctype_show(struct device *dev, }static DEVICE_ATTR_RO (dctype); +static ssize_t nvme_admin_timeout_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); + +return sysfs_emit (buf,"%u\n" , +jiffies_to_msecs (ctrl->admin_q->rq_timeout)); +} + +static ssize_t nvme_admin_timeout_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); +u32 timeout; +int err; + + err =kstrtou32 (buf,10 , &timeout); +if (err || !timeout) +return -EINVAL ;
drivers/nvme/host/sysfs.c static ssize_t dctype_show(struct device *dev, · line 633
Quoted 10 lines
drivers/nvme/host/sysfs.c +10
-0+ +blk_queue_rq_timeout (ctrl->admin_q,msecs_to_jiffies (timeout)); + +return count; +} + +static struct device_attribute dev_attr_admin_timeout = \ +__ATTR (admin_timeout,S_IRUGO |S_IWUSR , \ + nvme_admin_timeout_show, nvme_admin_timeout_store); +
Do we want similar attribute to ctrl->fabrics_q?
Quoted 16 lines
drivers/nvme/host/sysfs.c +1
-0#ifdef CONFIG_NVME_HOST_AUTH static ssize_t nvme_ctrl_dhchap_secret_show (struct device *dev,struct device_attribute *attr,char *buf)@@ -742,6 +772,7 @@ static struct attribute *nvme_dev_attrs[] = { &dev_attr_kato.attr, &dev_attr_cntrltype.attr, &dev_attr_dctype.attr, + &dev_attr_admin_timeout.attr, #ifdef CONFIG_NVME_HOST_AUTH &dev_attr_dhchap_secret.attr, &dev_attr_dhchap_ctrl_secret.attr,Signature · 1 line
2.53.0
Re: [PATCH RFC 2/5] nvme: add sysfs attribute to change admin timeout per nvme controller
Quoted 15 lines · Mohamed Khalfella
Quoted 15 lines
drivers/nvme/host/sysfs.c +15
-0+static ssize_t nvme_admin_timeout_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); +u32 timeout; +int err; + + err =kstrtou32 (buf,10 , &timeout); +if (err || !timeout) +return -EINVAL ; + +blk_queue_rq_timeout (ctrl->admin_q,msecs_to_jiffies (timeout)); + +return count;
Quoted 8 lines
drivers/nvme/host/sysfs.c static ssize_t dctype_show(struct device *dev, · line 633
Quoted 6 lines
drivers/nvme/host/sysfs.c +6
-0+} + +static struct device_attribute dev_attr_admin_timeout = \ +__ATTR (admin_timeout,S_IRUGO |S_IWUSR , \ + nvme_admin_timeout_show, nvme_admin_timeout_store); +Do we want similar attribute to
ctrl->fabrics_q?
Makes sense, I think.
We don't need other attributes, it should be sufficient to modify nvme_admin_timeout_store() to do the following:
static ssize_t nvme_admin_timeout_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t count)
{
struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
u32 timeout;
int err;
err = kstrtou32(buf, 10, &timeout);
if (err || !timeout)
return -EINVAL;
blk_queue_rq_timeout(ctrl->admin_q, msecs_to_jiffies(timeout));
if (ctrl->ops->flags & NVME_F_FABRICS)
blk_queue_rq_timeout(ctrl->fabrics_q, msecs_to_jiffies(timeout));
return count;
}Maurizio
RFC PATCH 3/5 nvme: pci: use admin queue timeout over NVME_ADMIN_TIMEOUT
In reply to Maurizio LombardiFrom: David Jeffery <[email protected]>
While tearing down its queues, nvme-pci uses NVME_ADMIN_TIMEOUT as its timeout target. Instead, use the configured admin queue's timeout value when available to match the device's existing timeout setting.
--- drivers/nvme/host/pci.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/host/pci.c +2 -1
@@ -2911,9 +2911,10 @@ static bool __nvme_delete_io_queues(struct nvme_dev *dev, u8 opcode) {int nr_queues = dev->online_queues -1 , sent =0 ;unsigned long timeout; +struct request_queue *q = dev->ctrl.admin_q; retry: - timeout =NVME_ADMIN_TIMEOUT ; + timeout = q ? q->rq_timeout :NVME_ADMIN_TIMEOUT ;while (nr_queues >0 ) {if (nvme_delete_queue (&dev->queues[nr_queues], opcode))break ;
Signature · 1 line
2.53.0
Re: [PATCH RFC 3/5] nvme: pci: use admin queue timeout over NVME_ADMIN_TIMEOUT
Quoted 16 lines · Maurizio Lombardi
From: David Jeffery <[email protected]>
While tearing down its queues, nvme-pci uses
NVME_ADMIN_TIMEOUTas its timeout target. Instead, use the configured admin queue's timeout value when available to match the device's existing timeout setting.Signed-off-by David Jeffery <[email protected]>--- drivers/nvme/host/pci.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/nvme/host/pci.c +0
-0
drivers/nvme/host/pci.c static bool __nvme_delete_io_queues(struct nvme_dev *dev, u8 opcode) · line 2914
Quoted 4 lines
drivers/nvme/host/pci.c +1
-0@@ -2911,9 +2911,10 @@ static bool __nvme_delete_io_queues(struct nvme_dev *dev, u8 opcode) {int nr_queues = dev->online_queues -1 , sent =0 ;unsigned long timeout; +struct request_queue *q = dev->ctrl.admin_q;
Maybe place q between nr_queues and timeout?
drivers/nvme/host/pci.c static bool __nvme_delete_io_queues(struct nvme_dev *dev, u8 opcode) · line 2917
Quoted 4 lines
drivers/nvme/host/pci.c +1
-1retry: - timeout =NVME_ADMIN_TIMEOUT ; + timeout = q ? q->rq_timeout :NVME_ADMIN_TIMEOUT ;
Under what condition we expect admin_q to be NULL?
Quoted 7 lines
drivers/nvme/host/pci.c +0
-0while (nr_queues >0 ) {if (nvme_delete_queue (&dev->queues[nr_queues], opcode))break ;Signature · 1 line
2.53.0
Re: [PATCH RFC 3/5] nvme: pci: use admin queue timeout over NVME_ADMIN_TIMEOUT
In reply to Maurizio LombardiQuoted 16 lines · Maurizio Lombardi
From: David Jeffery <[email protected]>
While tearing down its queues, nvme-pci uses
NVME_ADMIN_TIMEOUTas its timeout target. Instead, use the configured admin queue's timeout value when available to match the device's existing timeout setting.Signed-off-by David Jeffery <[email protected]>--- drivers/nvme/host/pci.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/nvme/host/pci.c +0
-0
drivers/nvme/host/pci.c static bool __nvme_delete_io_queues(struct nvme_dev *dev, u8 opcode) · line 2917
Quoted 8 lines
drivers/nvme/host/pci.c +2
-1@@ -2911,9 +2911,10 @@ static bool __nvme_delete_io_queues(struct nvme_dev *dev, u8 opcode) {int nr_queues = dev->online_queues -1 , sent =0 ;unsigned long timeout; +struct request_queue *q = dev->ctrl.admin_q; retry: - timeout =NVME_ADMIN_TIMEOUT ; + timeout = q ? q->rq_timeout :NVME_ADMIN_TIMEOUT ;
Skimming over the code I think the check is unnecessary since ctrl.admin_q is only changed in nvme_alloc_admin_tag_set. Only when this succeeds, we'll setup io queues. When they get deleted we will still have the ctrl.admin_q being valid.
Quoted 7 lines
drivers/nvme/host/pci.c +0
-0while (nr_queues >0 ) {if (nvme_delete_queue (&dev->queues[nr_queues], opcode))break ;Signature · 1 line
2.53.0
Amazon Web Services Development Center Germany GmbH Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christof Hellmis, Andreas Stieger Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597
Re: [PATCH RFC 3/5] nvme: pci: use admin queue timeout over NVME_ADMIN_TIMEOUT
Quoted 22 lines · Maximilian Heyne
Quoted 21 lines · Maurizio Lombardi
From: David Jeffery <[email protected]>
While tearing down its queues, nvme-pci uses
NVME_ADMIN_TIMEOUTas its timeout target. Instead, use the configured admin queue's timeout value when available to match the device's existing timeout setting.Signed-off-by David Jeffery <[email protected]>--- drivers/nvme/host/pci.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/nvme/host/pci.c +1
-0@@ -2911,9 +2911,10 @@ static bool __nvme_delete_io_queues(struct nvme_dev *dev, u8 opcode) {int nr_queues = dev->online_queues -1 , sent =0 ;unsigned long timeout; +struct request_queue *q = dev->ctrl.admin_q;
Quoted 8 lines
drivers/nvme/host/pci.c static bool __nvme_delete_io_queues(struct nvme_dev *dev, u8 opcode) · line 2917
Quoted 3 lines
drivers/nvme/host/pci.c +1
-1retry: - timeout =NVME_ADMIN_TIMEOUT ; + timeout = q ? q->rq_timeout :NVME_ADMIN_TIMEOUT ;Skimming over the code I think the check is unnecessary since ctrl.admin_q is only changed in
nvme_alloc_admin_tag_set. Only when this succeeds, we'll setup io queues. When they get deleted we will still have the ctrl.admin_q being valid.
Yes, afaict you are right and this check is unnecessary, I will remove it.
Maurizio
RFC PATCH 4/5 nvme: add sysfs attribute to change IO timeout per nvme controller
In reply to Maurizio LombardiCurrently, there is no method to adjust the timeout values on a per controller basis with nvme I/O queues. Add an io_timeout attribute to nvme so that different nvme controllers which may have different timeout requirements can have custom I/O timeouts set.
--- drivers/nvme/host/apple.c | 2 +- drivers/nvme/host/core.c | 4 +++- drivers/nvme/host/nvme.h | 1 + drivers/nvme/host/sysfs.c | 38 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 43 insertions(+), 2 deletions(-)
drivers/nvme/host/apple.c +1 -1
@@ -1331,7 +1331,7 @@ static int apple_nvme_alloc_tagsets(struct apple_nvme *anv)if (anv->hw->has_lsq_nvmmu) anv->tagset.reserved_tags =APPLE_NVME_AQ_DEPTH ; anv->tagset.queue_depth = anv->hw->max_queue_depth -1 ; - anv->tagset.timeout =NVME_IO_TIMEOUT ; + anv->tagset.timeout = anv->ctrl.io_timeout; anv->tagset.numa_node =NUMA_NO_NODE ; anv->tagset.cmd_size =sizeof (struct apple_nvme_iod ); anv->tagset.driver_data = &anv->ioq;
drivers/nvme/host/core.c +3 -1
@@ -4169,6 +4169,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)mutex_unlock (&ctrl->namespaces_lock); goto out_unlink_ns; } +blk_queue_rq_timeout (ns->queue, ctrl->io_timeout);nvme_ns_add_to_ctrl_list (ns);mutex_unlock (&ctrl->namespaces_lock);synchronize_srcu (&ctrl->srcu);
@@ -4930,7 +4931,7 @@ int nvme_alloc_io_tag_set(struct nvme_ctrl *ctrl, struct blk_mq_tag_set *set, set->cmd_size = cmd_size; set->driver_data = ctrl; set->nr_hw_queues = ctrl->queue_count -1 ; - set->timeout =NVME_IO_TIMEOUT ; + set->timeout = ctrl->io_timeout; set->nr_maps = nr_maps; ret =blk_mq_alloc_tag_set (set);if (ret)
@@ -5107,6 +5108,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,memset (&ctrl->ka_cmd,0 ,sizeof (ctrl->ka_cmd)); ctrl->ka_cmd.common.opcode = nvme_admin_keep_alive; ctrl->ka_last_check_time = jiffies; + ctrl->io_timeout =NVME_IO_TIMEOUT ;BUILD_BUG_ON (NVME_DSM_MAX_RANGES *sizeof (struct nvme_dsm_range ) >PAGE_SIZE );
drivers/nvme/host/nvme.h +1 -0
@@ -316,6 +316,7 @@ struct nvme_ctrl {u16 mtfa;u32 ctrl_config;u32 queue_count; +u32 io_timeout;u64 cap;u32 max_hw_sectors;
drivers/nvme/host/sysfs.c +38 -0
@@ -631,6 +631,43 @@ static struct device_attribute dev_attr_admin_timeout = \__ATTR (admin_timeout,S_IRUGO |S_IWUSR , \ nvme_admin_timeout_show, nvme_admin_timeout_store); +static ssize_t nvme_io_timeout_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); + +return sysfs_emit (buf,"%u\n" ,jiffies_to_msecs (ctrl->io_timeout)); +} + +static ssize_t nvme_io_timeout_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); +struct nvme_ns *ns; +u32 timeout; +int err; + + err =kstrtou32 (buf,10 , &timeout); +if (err || !timeout) +return -EINVAL ; + +/* Take the namespaces_lock to avoid racing against nvme_alloc_ns() */ +mutex_lock (&ctrl->namespaces_lock); + + ctrl->io_timeout =msecs_to_jiffies (timeout); +list_for_each_entry (ns, &ctrl->namespaces, list) +blk_queue_rq_timeout (ns->queue, ctrl->io_timeout); + +mutex_unlock (&ctrl->namespaces_lock); + +return count; +} + +static struct device_attribute dev_attr_io_timeout = \ +__ATTR (io_timeout,S_IRUGO |S_IWUSR , \ + nvme_io_timeout_show, nvme_io_timeout_store); +#ifdef CONFIG_NVME_HOST_AUTH static ssize_t nvme_ctrl_dhchap_secret_show (struct device *dev,struct device_attribute *attr,char *buf)
@@ -773,6 +810,7 @@ static struct attribute *nvme_dev_attrs[] = { &dev_attr_cntrltype.attr, &dev_attr_dctype.attr, &dev_attr_admin_timeout.attr, + &dev_attr_io_timeout.attr, #ifdef CONFIG_NVME_HOST_AUTH &dev_attr_dhchap_secret.attr, &dev_attr_dhchap_ctrl_secret.attr,
Signature · 1 line
2.53.0
Re: [PATCH RFC 4/5] nvme: add sysfs attribute to change IO timeout per nvme controller
Quoted 32 lines · Maurizio Lombardi
Currently, there is no method to adjust the timeout values on a per controller basis with nvme I/O queues. Add an
io_timeoutattribute to nvme so that different nvme controllers which may have different timeout requirements can have custom I/O timeouts set.Signed-off-by Maurizio Lombardi <[email protected]>--- drivers/nvme/host/apple.c | 2 +- drivers/nvme/host/core.c | 4 +++- drivers/nvme/host/nvme.h | 1 + drivers/nvme/host/sysfs.c | 38 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 43 insertions(+), 2 deletions(-)drivers/nvme/host/apple.c +1
-1@@ -1331,7 +1331,7 @@ static int apple_nvme_alloc_tagsets(struct apple_nvme *anv)if (anv->hw->has_lsq_nvmmu) anv->tagset.reserved_tags =APPLE_NVME_AQ_DEPTH ; anv->tagset.queue_depth = anv->hw->max_queue_depth -1 ; - anv->tagset.timeout =NVME_IO_TIMEOUT ; + anv->tagset.timeout = anv->ctrl.io_timeout; anv->tagset.numa_node =NUMA_NO_NODE ; anv->tagset.cmd_size =sizeof (struct apple_nvme_iod ); anv->tagset.driver_data = &anv->ioq;drivers/nvme/host/core.c +0
-0
drivers/nvme/host/core.c static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info) · line 4172
Quoted 4 lines
drivers/nvme/host/core.c +1
-0@@ -4169,6 +4169,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)mutex_unlock (&ctrl->namespaces_lock); goto out_unlink_ns; } +blk_queue_rq_timeout (ns->queue, ctrl->io_timeout);
Given that io tagset->timeout has been initialized to ctrl->io_timeout Do we still need the line above?
Quoted 92 lines
drivers/nvme/host/core.c +2
-1nvme_ns_add_to_ctrl_list (ns);mutex_unlock (&ctrl->namespaces_lock);synchronize_srcu (&ctrl->srcu);@@ -4930,7 +4931,7 @@ int nvme_alloc_io_tag_set(struct nvme_ctrl *ctrl, struct blk_mq_tag_set *set, set->cmd_size = cmd_size; set->driver_data = ctrl; set->nr_hw_queues = ctrl->queue_count -1 ; - set->timeout =NVME_IO_TIMEOUT ; + set->timeout = ctrl->io_timeout; set->nr_maps = nr_maps; ret =blk_mq_alloc_tag_set (set);if (ret)@@ -5107,6 +5108,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,memset (&ctrl->ka_cmd,0 ,sizeof (ctrl->ka_cmd)); ctrl->ka_cmd.common.opcode = nvme_admin_keep_alive; ctrl->ka_last_check_time = jiffies; + ctrl->io_timeout =NVME_IO_TIMEOUT ;BUILD_BUG_ON (NVME_DSM_MAX_RANGES *sizeof (struct nvme_dsm_range ) >PAGE_SIZE );drivers/nvme/host/nvme.h +1
-0@@ -316,6 +316,7 @@ struct nvme_ctrl {u16 mtfa;u32 ctrl_config;u32 queue_count; +u32 io_timeout;u64 cap;u32 max_hw_sectors;drivers/nvme/host/sysfs.c +38
-0@@ -631,6 +631,43 @@ static struct device_attribute dev_attr_admin_timeout = \__ATTR (admin_timeout,S_IRUGO |S_IWUSR , \ nvme_admin_timeout_show, nvme_admin_timeout_store); +static ssize_t nvme_io_timeout_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); + +return sysfs_emit (buf,"%u\n" ,jiffies_to_msecs (ctrl->io_timeout)); +} + +static ssize_t nvme_io_timeout_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); +struct nvme_ns *ns; +u32 timeout; +int err; + + err =kstrtou32 (buf,10 , &timeout); +if (err || !timeout) +return -EINVAL ; + +/* Take the namespaces_lock to avoid racing against nvme_alloc_ns() */ +mutex_lock (&ctrl->namespaces_lock); + + ctrl->io_timeout =msecs_to_jiffies (timeout); +list_for_each_entry (ns, &ctrl->namespaces, list) +blk_queue_rq_timeout (ns->queue, ctrl->io_timeout); + +mutex_unlock (&ctrl->namespaces_lock); + +return count; +} + +static struct device_attribute dev_attr_io_timeout = \ +__ATTR (io_timeout,S_IRUGO |S_IWUSR , \ + nvme_io_timeout_show, nvme_io_timeout_store); +#ifdef CONFIG_NVME_HOST_AUTH static ssize_t nvme_ctrl_dhchap_secret_show (struct device *dev,struct device_attribute *attr,char *buf)@@ -773,6 +810,7 @@ static struct attribute *nvme_dev_attrs[] = { &dev_attr_cntrltype.attr, &dev_attr_dctype.attr, &dev_attr_admin_timeout.attr, + &dev_attr_io_timeout.attr, #ifdef CONFIG_NVME_HOST_AUTH &dev_attr_dhchap_secret.attr, &dev_attr_dhchap_ctrl_secret.attr,Signature · 1 line
2.53.0
Re: [PATCH RFC 4/5] nvme: add sysfs attribute to change IO timeout per nvme controller
Quoted 8 lines · Mohamed Khalfella
drivers/nvme/host/core.c static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info) · line 4172
Quoted 5 lines
drivers/nvme/host/core.c +1
-0@@ -4169,6 +4169,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)mutex_unlock (&ctrl->namespaces_lock); goto out_unlink_ns; } +blk_queue_rq_timeout (ns->queue, ctrl->io_timeout);Given that io
tagset->timeouthas been initialized toctrl->io_timeoutDo we still need the line above?
Yes, the line is necessary. While the tagset is initialized with ctrl->io_timeout, that happens only once during tagset allocation.
If the user later changes io_timeout via sysfs, ctrl->io_timeout is updated, but the tagset->timeout remains at the old default value. I deliberately do not update tagset->timeout in the sysfs handler because the tagset lifetime is tied to the controller state (e.g., it may be freed during a controller reset). Attempting to update the tagset from the sysfs path introduces a race condition where we might dereference a pointer to a tagset that is being destroyed.
Therefore, when a new namespace is allocated (nvme_alloc_ns), the block layer initializes the queue using the stale tagset->timeout. We must explicitly override that with the current ctrl->io_timeout to ensure the new namespace respects the runtime configuration.
Maurizio
Re: [PATCH RFC 4/5] nvme: add sysfs attribute to change IO timeout per nvme controller
Quoted 22 lines · Maurizio Lombardi
Quoted 8 lines · Mohamed Khalfella
drivers/nvme/host/core.c static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info) · line 4172
Quoted 5 lines
drivers/nvme/host/core.c +1
-0@@ -4169,6 +4169,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)mutex_unlock (&ctrl->namespaces_lock); goto out_unlink_ns; } +blk_queue_rq_timeout (ns->queue, ctrl->io_timeout);Given that io
tagset->timeouthas been initialized toctrl->io_timeoutDo we still need the line above?Yes, the line is necessary. While the tagset is initialized with
ctrl->io_timeout, that happens only once during tagset allocation.If the user later changes
io_timeoutvia sysfs,ctrl->io_timeoutis updated, but thetagset->timeoutremains at the old default value. I deliberately do not updatetagset->timeoutin the sysfs handler because the tagset lifetime is tied to the controller state (e.g., it may be freed during a controller reset). Attempting to update the tagset from the sysfs path introduces a race condition where we might dereference a pointer to a tagset that is being destroyed.
Quoted 4 lines
Therefore, when a new namespace is allocated (
nvme_alloc_ns), the block layer initializes the queue using the staletagset->timeout. We must explicitly override that with the currentctrl->io_timeoutto ensure the new namespace respects the runtime configuration.
Yeah. I got it now. Thanks for the explanation.
It would great if there is way to update admin and io tagsets timeout.
Re: [PATCH RFC 4/5] nvme: add sysfs attribute to change IO timeout per nvme controller
Quoted 3 lines · Mohamed Khalfella
Yeah. I got it now. Thanks for the explanation.
It would great if there is way to update admin and io tagsets timeout.
Yes, it would be much better.
I think it can be done, the only driver that in some cases sets the tagset pointer to NULL during reset is nvme-pci, but only after removing all the namespaces.
So changing the timeout field in the tagset should be doable, the only problem is avoid racing against nvme_alloc_ns().
I will try to come up with something.
Maurizio
Re: [PATCH RFC 4/5] nvme: add sysfs attribute to change IO timeout per nvme controller
Quoted 6 lines · Maurizio Lombardi
On Wed Feb 18, 2026 at 6:54 PM CET, Mohamed Khalfella wrote:
So changing the timeout field in the tagset should be doable, the only problem is avoid racing against
nvme_alloc_ns().I will try to come up with something.
I decided to keep the current design, calling blk_queue_rq_timeout() with the namespaces_lock mutex locked is the easiest solution
I am sending a V2 in a few moments.
Maurizio
Re: [PATCH RFC 4/5] nvme: add sysfs attribute to change IO timeout per nvme controller
Quoted 6 lines · Maurizio Lombardi
Quoted 5 lines · Maurizio Lombardi
On Wed Feb 18, 2026 at 6:54 PM CET, Mohamed Khalfella wrote:
So changing the timeout field in the tagset should be doable, the only problem is avoid racing against
nvme_alloc_ns().
Quoted 7 lines
Quoted 1 line
I will try to come up with something.
I decided to keep the current design, calling
blk_queue_rq_timeout()with thenamespaces_lockmutex locked is the easiest solutionI am sending a V2 in a few moments.
How about restricting changing io timeout to LIVE controllers only. This will make the problem easier to solve. Maybe something like the below?
enum nvme_ctrl_state state;
unsigned long flags;
spin_lock_irqsave(&ctrl->lock, flags);
state = nvme_ctrl_state(ctrl);
if (state != NVME_CTRL_LIVE) {
spin_unlock_irqrestore(&ctrl->lock, flags);
return -EBUSY
}
if (ctrl->queue_count > 1)
WRITE_ONCE(ctrl->tagset->timeout, timeout);
spin_unlock_irqrestore(&ctrl->lock, flags);
/* Take the namespaces_lock to avoid racing against nvme_alloc_ns() */
mutex_lock(&ctrl->namespaces_lock);
ctrl->io_timeout = msecs_to_jiffies(timeout);
list_for_each_entry(ns, &ctrl->namespaces, list)
blk_queue_rq_timeout(ns->queue, ctrl->io_timeout);
mutex_unlock(&ctrl->namespaces_lock);Quoted 2 lines
Maurizio
Re: [PATCH RFC 4/5] nvme: add sysfs attribute to change IO timeout per nvme controller
Quoted 18 lines · Mohamed Khalfella
Quoted 6 lines · Maurizio Lombardi
Quoted 5 lines · Maurizio Lombardi
On Wed Feb 18, 2026 at 6:54 PM CET, Mohamed Khalfella wrote:
So changing the timeout field in the tagset should be doable, the only problem is avoid racing against
nvme_alloc_ns().Quoted 7 lines
Quoted 1 line
I will try to come up with something.
I decided to keep the current design, calling
blk_queue_rq_timeout()with thenamespaces_lockmutex locked is the easiest solutionI am sending a V2 in a few moments.
How about restricting changing io timeout to LIVE controllers only. This will make the problem easier to solve. Maybe something like the below?
Quoted 9 lines
enum nvme_ctrl_state state;
unsigned long flags;
spin_lock_irqsave(&ctrl->lock, flags);
state = nvme_ctrl_state(ctrl);
if (state != NVME_CTRL_LIVE) {
spin_unlock_irqrestore(&ctrl->lock, flags);
return -EBUSY
}I previously considered this solution, but I discarded it for two reasons:
1) If the user sets the io_timeout too low, the controller ends up resetting non-stop. With this patch, it becomes hard to catch the controller in a LIVE state and fix the problem, because the controller is almost always RESETTING. You essentially have to wait until the driver gives up and removes the controller.
2) This doesn't prevent nvme_alloc_ns() from racing against the sysfs path.
Suppose that thread A is executing nvme_alloc_ns() and has just called disk = blk_mq_alloc_disk(ctrl->tagset, &lim, ns);
immediately after, the user changes io_timeout. Thread B takes the namespaces_lock, scans the namespaces list and calls blk_queue_rq_timeout() on each namespaces' queues.
thread A now can take the namespaces_lock and adds the new namespace to the list, but the problem is that the new namespace is using the stale timeout setting it inherited from ctrl->tagset.
Maurizio
Quoted 17 lines
if (ctrl->queue_count > 1) WRITE_ONCE(ctrl->tagset->timeout, timeout); spin_unlock_irqrestore(&ctrl->lock, flags); /* Take the namespaces_lock to avoid racing against nvme_alloc_ns() */ mutex_lock(&ctrl->namespaces_lock); ctrl->io_timeout = msecs_to_jiffies(timeout); list_for_each_entry(ns, &ctrl->namespaces, list) blk_queue_rq_timeout(ns->queue, ctrl->io_timeout); mutex_unlock(&ctrl->namespaces_lock);Quoted 2 lines
Maurizio
Re: [PATCH RFC 4/5] nvme: add sysfs attribute to change IO timeout per nvme controller
In reply to Maurizio LombardiQuoted 95 lines · Maurizio Lombardi
Currently, there is no method to adjust the timeout values on a per controller basis with nvme I/O queues. Add an
io_timeoutattribute to nvme so that different nvme controllers which may have different timeout requirements can have custom I/O timeouts set.Signed-off-by Maurizio Lombardi <[email protected]>--- drivers/nvme/host/apple.c | 2 +- drivers/nvme/host/core.c | 4 +++- drivers/nvme/host/nvme.h | 1 + drivers/nvme/host/sysfs.c | 38 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 43 insertions(+), 2 deletions(-)drivers/nvme/host/apple.c +1
-1@@ -1331,7 +1331,7 @@ static int apple_nvme_alloc_tagsets(struct apple_nvme *anv)if (anv->hw->has_lsq_nvmmu) anv->tagset.reserved_tags =APPLE_NVME_AQ_DEPTH ; anv->tagset.queue_depth = anv->hw->max_queue_depth -1 ; - anv->tagset.timeout =NVME_IO_TIMEOUT ; + anv->tagset.timeout = anv->ctrl.io_timeout; anv->tagset.numa_node =NUMA_NO_NODE ; anv->tagset.cmd_size =sizeof (struct apple_nvme_iod ); anv->tagset.driver_data = &anv->ioq;drivers/nvme/host/core.c +3
-1@@ -4169,6 +4169,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)mutex_unlock (&ctrl->namespaces_lock); goto out_unlink_ns; } +blk_queue_rq_timeout (ns->queue, ctrl->io_timeout);nvme_ns_add_to_ctrl_list (ns);mutex_unlock (&ctrl->namespaces_lock);synchronize_srcu (&ctrl->srcu);@@ -4930,7 +4931,7 @@ int nvme_alloc_io_tag_set(struct nvme_ctrl *ctrl, struct blk_mq_tag_set *set, set->cmd_size = cmd_size; set->driver_data = ctrl; set->nr_hw_queues = ctrl->queue_count -1 ; - set->timeout =NVME_IO_TIMEOUT ; + set->timeout = ctrl->io_timeout; set->nr_maps = nr_maps; ret =blk_mq_alloc_tag_set (set);if (ret)@@ -5107,6 +5108,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,memset (&ctrl->ka_cmd,0 ,sizeof (ctrl->ka_cmd)); ctrl->ka_cmd.common.opcode = nvme_admin_keep_alive; ctrl->ka_last_check_time = jiffies; + ctrl->io_timeout =NVME_IO_TIMEOUT ;BUILD_BUG_ON (NVME_DSM_MAX_RANGES *sizeof (struct nvme_dsm_range ) >PAGE_SIZE );drivers/nvme/host/nvme.h +1
-0@@ -316,6 +316,7 @@ struct nvme_ctrl {u16 mtfa;u32 ctrl_config;u32 queue_count; +u32 io_timeout;u64 cap;u32 max_hw_sectors;drivers/nvme/host/sysfs.c +19
-0@@ -631,6 +631,43 @@ static struct device_attribute dev_attr_admin_timeout = \__ATTR (admin_timeout,S_IRUGO |S_IWUSR , \ nvme_admin_timeout_show, nvme_admin_timeout_store); +static ssize_t nvme_io_timeout_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); + +return sysfs_emit (buf,"%u\n" ,jiffies_to_msecs (ctrl->io_timeout)); +} + +static ssize_t nvme_io_timeout_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); +struct nvme_ns *ns; +u32 timeout; +int err; + + err =kstrtou32 (buf,10 , &timeout); +if (err || !timeout)
drivers/nvme/host/sysfs.c static struct device_attribute dev_attr_admin_timeout = \ · line 662
Quoted 10 lines
drivers/nvme/host/sysfs.c +10
-0+return -EINVAL ; + +/* Take the namespaces_lock to avoid racing against nvme_alloc_ns() */ +mutex_lock (&ctrl->namespaces_lock); + + ctrl->io_timeout =msecs_to_jiffies (timeout); +list_for_each_entry (ns, &ctrl->namespaces, list) +blk_queue_rq_timeout (ns->queue, ctrl->io_timeout); + +mutex_unlock (&ctrl->namespaces_lock);
What about ctrl->connect_q? It uses io tagset also. Do we want to update its timeout also?
Quoted 24 lines
drivers/nvme/host/sysfs.c +9
-0+ +return count; +} + +static struct device_attribute dev_attr_io_timeout = \ +__ATTR (io_timeout,S_IRUGO |S_IWUSR , \ + nvme_io_timeout_show, nvme_io_timeout_store); +#ifdef CONFIG_NVME_HOST_AUTH static ssize_t nvme_ctrl_dhchap_secret_show (struct device *dev,struct device_attribute *attr,char *buf)@@ -773,6 +810,7 @@ static struct attribute *nvme_dev_attrs[] = { &dev_attr_cntrltype.attr, &dev_attr_dctype.attr, &dev_attr_admin_timeout.attr, + &dev_attr_io_timeout.attr, #ifdef CONFIG_NVME_HOST_AUTH &dev_attr_dhchap_secret.attr, &dev_attr_dhchap_ctrl_secret.attr,Signature · 1 line
2.53.0
RFC PATCH 5/5 nvme: use per controller timeout waits over depending on global default
In reply to Maurizio LombardiInstead of passing NVME_IO_TIMEOUT as a parameter with every call to nvme_wait_freeze_timeout, use the controller's preferred timeout.
--- drivers/nvme/host/apple.c | 2 +- drivers/nvme/host/core.c | 5 +++-- drivers/nvme/host/nvme.h | 2 +- drivers/nvme/host/pci.c | 2 +- drivers/nvme/host/rdma.c | 2 +- drivers/nvme/host/tcp.c | 2 +- 6 files changed, 8 insertions(+), 7 deletions(-)
drivers/nvme/host/apple.c +1 -1
@@ -858,7 +858,7 @@ static void apple_nvme_disable(struct apple_nvme *anv, bool shutdown)* doing a safe shutdown. */ if (!dead && shutdown && freeze) -nvme_wait_freeze_timeout (&anv->ctrl,NVME_IO_TIMEOUT ); +nvme_wait_freeze_timeout (&anv->ctrl);nvme_quiesce_io_queues (&anv->ctrl);
drivers/nvme/host/core.c +3 -2
@@ -5215,16 +5215,17 @@ void nvme_unfreeze(struct nvme_ctrl *ctrl) }EXPORT_SYMBOL_GPL (nvme_unfreeze); -int nvme_wait_freeze_timeout (struct nvme_ctrl *ctrl,long timeout) +int nvme_wait_freeze_timeout (struct nvme_ctrl *ctrl) {struct nvme_ns *ns;int srcu_idx; +unsigned long timeout = ctrl->io_timeout; srcu_idx =srcu_read_lock (&ctrl->srcu);list_for_each_entry_srcu (ns, &ctrl->namespaces, list,srcu_read_lock_held (&ctrl->srcu)) { timeout =blk_mq_freeze_queue_wait_timeout (ns->queue, timeout); -if (timeout <=0 ) +if (!timeout)break ; }srcu_read_unlock (&ctrl->srcu, srcu_idx);
drivers/nvme/host/nvme.h +1 -1
@@ -840,7 +840,7 @@ void nvme_sync_queues(struct nvme_ctrl *ctrl);void nvme_sync_io_queues (struct nvme_ctrl *ctrl);void nvme_unfreeze (struct nvme_ctrl *ctrl);void nvme_wait_freeze (struct nvme_ctrl *ctrl); -int nvme_wait_freeze_timeout (struct nvme_ctrl *ctrl,long timeout); +int nvme_wait_freeze_timeout (struct nvme_ctrl *ctrl);void nvme_start_freeze (struct nvme_ctrl *ctrl);static inline enum req_op nvme_req_op (struct nvme_command *cmd)
drivers/nvme/host/pci.c +1 -1
@@ -3096,7 +3096,7 @@ static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown)* if doing a safe shutdown. */ if (!dead && shutdown) -nvme_wait_freeze_timeout (&dev->ctrl,NVME_IO_TIMEOUT ); +nvme_wait_freeze_timeout (&dev->ctrl); }nvme_quiesce_io_queues (&dev->ctrl);
drivers/nvme/host/rdma.c +1 -1
@@ -888,7 +888,7 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)if (!new) {nvme_start_freeze (&ctrl->ctrl);nvme_unquiesce_io_queues (&ctrl->ctrl); -if (!nvme_wait_freeze_timeout (&ctrl->ctrl,NVME_IO_TIMEOUT )) { +if (!nvme_wait_freeze_timeout (&ctrl->ctrl)) {/* * If we timed out waiting for freeze we are likely to * be stuck. Fail the controller initialization just
drivers/nvme/host/tcp.c +1 -1
@@ -2193,7 +2193,7 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)if (!new) {nvme_start_freeze (ctrl);nvme_unquiesce_io_queues (ctrl); -if (!nvme_wait_freeze_timeout (ctrl,NVME_IO_TIMEOUT )) { +if (!nvme_wait_freeze_timeout (ctrl)) {/* * If we timed out waiting for freeze we are likely to * be stuck. Fail the controller initialization just
Signature · 1 line
2.53.0
Re: [PATCH RFC 5/5] nvme: use per controller timeout waits over depending on global default
Quoted 4 lines · Maurizio Lombardi
Instead of passing
NVME_IO_TIMEOUTas a parameter with every call tonvme_wait_freeze_timeout, use the controller's preferred timeout.Signed-off-by Maurizio Lombardi <[email protected]>
Quoted 27 lines
--- drivers/nvme/host/apple.c | 2 +- drivers/nvme/host/core.c | 5 +++-- drivers/nvme/host/nvme.h | 2 +- drivers/nvme/host/pci.c | 2 +- drivers/nvme/host/rdma.c | 2 +- drivers/nvme/host/tcp.c | 2 +- 6 files changed, 8 insertions(+), 7 deletions(-)drivers/nvme/host/apple.c +1
-1@@ -858,7 +858,7 @@ static void apple_nvme_disable(struct apple_nvme *anv, bool shutdown)* doing a safe shutdown. */ if (!dead && shutdown && freeze) -nvme_wait_freeze_timeout (&anv->ctrl,NVME_IO_TIMEOUT ); +nvme_wait_freeze_timeout (&anv->ctrl);nvme_quiesce_io_queues (&anv->ctrl);drivers/nvme/host/core.c +0
-0
drivers/nvme/host/core.c void nvme_unfreeze(struct nvme_ctrl *ctrl) · line 5222
Quoted 9 lines
drivers/nvme/host/core.c +2
-1@@ -5215,16 +5215,17 @@ void nvme_unfreeze(struct nvme_ctrl *ctrl) }EXPORT_SYMBOL_GPL (nvme_unfreeze); -int nvme_wait_freeze_timeout (struct nvme_ctrl *ctrl,long timeout) +int nvme_wait_freeze_timeout (struct nvme_ctrl *ctrl) {struct nvme_ns *ns;int srcu_idx; +unsigned long timeout = ctrl->io_timeout;
nit: Maybe move this to the top before ns?
Quoted 66 lines
drivers/nvme/host/core.c +1
-1srcu_idx =srcu_read_lock (&ctrl->srcu);list_for_each_entry_srcu (ns, &ctrl->namespaces, list,srcu_read_lock_held (&ctrl->srcu)) { timeout =blk_mq_freeze_queue_wait_timeout (ns->queue, timeout); -if (timeout <=0 ) +if (!timeout)break ; }srcu_read_unlock (&ctrl->srcu, srcu_idx);drivers/nvme/host/nvme.h +1
-1@@ -840,7 +840,7 @@ void nvme_sync_queues(struct nvme_ctrl *ctrl);void nvme_sync_io_queues (struct nvme_ctrl *ctrl);void nvme_unfreeze (struct nvme_ctrl *ctrl);void nvme_wait_freeze (struct nvme_ctrl *ctrl); -int nvme_wait_freeze_timeout (struct nvme_ctrl *ctrl,long timeout); +int nvme_wait_freeze_timeout (struct nvme_ctrl *ctrl);void nvme_start_freeze (struct nvme_ctrl *ctrl);static inline enum req_op nvme_req_op (struct nvme_command *cmd)drivers/nvme/host/pci.c +1
-1@@ -3096,7 +3096,7 @@ static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown)* if doing a safe shutdown. */ if (!dead && shutdown) -nvme_wait_freeze_timeout (&dev->ctrl,NVME_IO_TIMEOUT ); +nvme_wait_freeze_timeout (&dev->ctrl); }nvme_quiesce_io_queues (&dev->ctrl);drivers/nvme/host/rdma.c +1
-1@@ -888,7 +888,7 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)if (!new) {nvme_start_freeze (&ctrl->ctrl);nvme_unquiesce_io_queues (&ctrl->ctrl); -if (!nvme_wait_freeze_timeout (&ctrl->ctrl,NVME_IO_TIMEOUT )) { +if (!nvme_wait_freeze_timeout (&ctrl->ctrl)) {/* * If we timed out waiting for freeze we are likely to * be stuck. Fail the controller initialization just drivers/nvme/host/tcp.c +1
-1@@ -2193,7 +2193,7 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)if (!new) {nvme_start_freeze (ctrl);nvme_unquiesce_io_queues (ctrl); -if (!nvme_wait_freeze_timeout (ctrl,NVME_IO_TIMEOUT )) { +if (!nvme_wait_freeze_timeout (ctrl)) {/* * If we timed out waiting for freeze we are likely to * be stuck. Fail the controller initialization just Signature · 1 line
2.53.0