nvme: make firmware activation poll interval configurable
PATCH 1/1 nvme: make firmware activation poll interval configurable
From: Guzebing <[email protected]>
nvme_fw_act_work() polls the controller processing-paused status every 100 ms while firmware activation is pending. Some devices can complete online activation in only a few hundred milliseconds, so the fixed interval can add noticeable latency before the driver observes completion.
Add an nvme_core.fw_act_poll_interval_ms module parameter to make the poll interval tunable. Keep the default at 100 ms to preserve existing behavior, and accept values from 10 ms to 100 ms so systems that need faster completion detection can opt in to a shorter interval.
---
We recently observed this issue while performing online firmware updates for Gen5 NVMe SSDs in a production environment.
During firmware activation, the kernel quiesces I/O. Detecting the end of firmware activation earlier lets the driver unquiesce I/O earlier, which is important for the long-tail I/O latency of production workloads.
Operation steps:
nvme fw-download /dev/nvme13n1 \
-f ./Inventec_G238B_NVME_LDD5902Q_MZWL61T9HFLT-00B07.bin nvme fw-commit /dev/nvme13n1 -s 0 -a 3
nvme fw-log /dev/nvme13n1
Operation device:
Samsung PM9D3a Gen5 NVMe SSD, model MZWL67T6HBLC-00B07 Firmware update: LDD5702Q -> LDD5902Q
Operation results:
100 ms polling interval: 3 polls, 311 ms
10 ms polling interval: 24 polls, 266 ms
drivers/nvme/host/core.c | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +23 -1
@@ -58,6 +58,21 @@ module_param_named(io_timeout, nvme_io_timeout, uint, 0644);MODULE_PARM_DESC (io_timeout,"timeout in seconds for I/O" );EXPORT_SYMBOL_GPL (nvme_io_timeout); +#define NVME_FW_ACT_POLL_INTERVAL_MIN_MS 10 +#define NVME_FW_ACT_POLL_INTERVAL_MAX_MS 100 +static int fw_act_poll_interval_ms_set (const char *val, +const struct kernel_param *kp); +static const struct kernel_param_ops fw_act_poll_interval_ms_ops = { + .set = fw_act_poll_interval_ms_set, + .get = param_get_uint, +}; + +static unsigned int fw_act_poll_interval_ms =100 ; +module_param_cb (fw_act_poll_interval_ms, &fw_act_poll_interval_ms_ops, + &fw_act_poll_interval_ms,0644 ); +MODULE_PARM_DESC (fw_act_poll_interval_ms, +"firmware activation polling interval in ms (10-100)" ); +static unsigned char shutdown_timeout =5 ;module_param (shutdown_timeout, byte,0644 );MODULE_PARM_DESC (shutdown_timeout,"timeout in seconds for controller shutdown" );
@@ -106,6 +121,13 @@ module_param(disable_pi_offsets, bool, 0444);MODULE_PARM_DESC (disable_pi_offsets,"disable protection information if it has an offset" ); +static int fw_act_poll_interval_ms_set (const char *val, +const struct kernel_param *kp) +{ +return param_set_uint_minmax (val, kp,NVME_FW_ACT_POLL_INTERVAL_MIN_MS , +NVME_FW_ACT_POLL_INTERVAL_MAX_MS ); +} +/* * nvme_wq - hosts nvme related works that are not reset or delete * nvme_reset_wq - hosts nvme reset works
@@ -4798,7 +4820,7 @@ static void nvme_fw_act_work(struct work_struct *work)nvme_try_sched_reset (ctrl);return ; } -msleep (100 ); +msleep (READ_ONCE (fw_act_poll_interval_ms)); } if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_CONNECTING) ||
Signature · 1 line
2.20.1
Hi Keith, Jens, Christoph, Sagi,
Gentle ping.
The motivation is to reduce the extra time I/O queues remain quiesced after the controller has finished online firmware activation. In our Gen5 NVMe SSD update test, using a 10 ms poll interval instead of the current 100 ms interval reduced the observed completion detection time from 311 ms to 266 ms.
Could you advise whether this is a latency issue worth addressing upstream now? If so, would you prefer a fixed shorter polling interval instead of a module parameter?
If the current benefit is not compelling enough, would it be better to defer this until firmware activation times become shorter on more devices, where the fixed 100 ms polling interval would account for a larger part of the I/O quiesce time?
Thanks,
Guzebing
Quoted 89 lines · guzebing
From: Guzebing <[email protected]>
nvme_fw_act_work()polls the controller processing-paused status every 100 ms while firmware activation is pending. Some devices can complete online activation in only a few hundred milliseconds, so the fixed interval can add noticeable latency before the driver observes completion.Add an nvme_core.fw_act_poll_interval_ms module parameter to make the poll interval tunable. Keep the default at 100 ms to preserve existing behavior, and accept values from 10 ms to 100 ms so systems that need faster completion detection can opt in to a shorter interval.
Signed-off-by Guzebing <[email protected]>---
We recently observed this issue while performing online firmware updates for Gen5 NVMe SSDs in a production environment.During firmware activation, the kernel quiesces I/O. Detecting the end of firmware activation earlier lets the driver unquiesce I/O earlier, which is important for the long-tail I/O latency of production workloads.
Operation steps:
nvme fw-download /dev/nvme13n1 \ -f ./Inventec_G238B_NVME_LDD5902Q_MZWL61T9HFLT-00B07.binnvme fw-commit
/dev/nvme13n1-s 0 -a 3
nvme fw-log/dev/nvme13n1Operation device:
Samsung PM9D3a Gen5 NVMe SSD, model MZWL67T6HBLC-00B07 Firmware update: LDD5702Q -> LDD5902QOperation results:
100 ms polling interval: 3 polls, 311 ms
10 ms polling interval: 24 polls, 266 msdrivers/nvme/host/core.c | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-)drivers/nvme/host/core.c +23
-1@@ -58,6 +58,21 @@ module_param_named(io_timeout, nvme_io_timeout, uint, 0644);MODULE_PARM_DESC (io_timeout,"timeout in seconds for I/O" );EXPORT_SYMBOL_GPL (nvme_io_timeout); +#define NVME_FW_ACT_POLL_INTERVAL_MIN_MS 10 +#define NVME_FW_ACT_POLL_INTERVAL_MAX_MS 100 +static int fw_act_poll_interval_ms_set (const char *val, +const struct kernel_param *kp); +static const struct kernel_param_ops fw_act_poll_interval_ms_ops = { + .set = fw_act_poll_interval_ms_set, + .get = param_get_uint, +}; + +static unsigned int fw_act_poll_interval_ms =100 ; +module_param_cb (fw_act_poll_interval_ms, &fw_act_poll_interval_ms_ops, + &fw_act_poll_interval_ms,0644 ); +MODULE_PARM_DESC (fw_act_poll_interval_ms, +"firmware activation polling interval in ms (10-100)" ); +static unsigned char shutdown_timeout =5 ;module_param (shutdown_timeout, byte,0644 );MODULE_PARM_DESC (shutdown_timeout,"timeout in seconds for controller shutdown" );@@ -106,6 +121,13 @@ module_param(disable_pi_offsets, bool, 0444);MODULE_PARM_DESC (disable_pi_offsets,"disable protection information if it has an offset" ); +static int fw_act_poll_interval_ms_set (const char *val, +const struct kernel_param *kp) +{ +return param_set_uint_minmax (val, kp,NVME_FW_ACT_POLL_INTERVAL_MIN_MS , +NVME_FW_ACT_POLL_INTERVAL_MAX_MS ); +} +/* * nvme_wq - hosts nvme related works that are not reset or delete * nvme_reset_wq - hosts nvme reset works @@ -4798,7 +4820,7 @@ static void nvme_fw_act_work(struct work_struct *work)nvme_try_sched_reset (ctrl);return ; } -msleep (100 ); +msleep (READ_ONCE (fw_act_poll_interval_ms)); } if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_CONNECTING) ||
Christoph Hellwig
maintainerQuoted 14 lines · guzebing
From: Guzebing <[email protected]>
nvme_fw_act_work()polls the controller processing-paused status every 100 ms while firmware activation is pending. Some devices can complete online activation in only a few hundred milliseconds, so the fixed interval can add noticeable latency before the driver observes completion.Add an nvme_core.fw_act_poll_interval_ms module parameter to make the poll interval tunable. Keep the default at 100 ms to preserve existing behavior, and accept values from 10 ms to 100 ms so systems that need faster completion detection can opt in to a shorter interval.
Signed-off-by Guzebing <[email protected]>
Quoted 8 lines
---
We recently observed this issue while performing online firmware updates for Gen5 NVMe SSDs in a production environment.During firmware activation, the kernel quiesces I/O. Detecting the end of firmware activation earlier lets the driver unquiesce I/O earlier, which is important for the long-tail I/O latency of production workloads.
What value does this device report in the Maximum Time for Firmware Activation (MTFA) field? It might make sense to scale the polling time as a fraction of that instead of requiring a manual override.
Quoted 19 lines · Christoph Hellwig
Quoted 18 lines · guzebing
From: Guzebing <[email protected]>
nvme_fw_act_work()polls the controller processing-paused status every 100 ms while firmware activation is pending. Some devices can complete online activation in only a few hundred milliseconds, so the fixed interval can add noticeable latency before the driver observes completion.Add an nvme_core.fw_act_poll_interval_ms module parameter to make the poll interval tunable. Keep the default at 100 ms to preserve existing behavior, and accept values from 10 ms to 100 ms so systems that need faster completion detection can opt in to a shorter interval.
Signed-off-by Guzebing <[email protected]>---
We recently observed this issue while performing online firmware updates for Gen5 NVMe SSDs in a production environment.
Quoted 9 lines
Quoted 4 lines
During firmware activation, the kernel quiesces I/O. Detecting the end of firmware activation earlier lets the driver unquiesce I/O earlier, which is important for the long-tail I/O latency of production workloads.
What value does this device report in the Maximum Time for Firmware Activation (MTFA) field? It might make sense to scale the polling time as a fraction of that instead of requiring a manual override.
The Samsung PM9D3a Gen5 SSD reports MTFA = 10, i.e. 1000 ms.
I also checked another device, an Intel/Solidigm P5520 Gen4 drive. It reports MTFA = 100, i.e. 10000 ms, while the observed online activation time is about 800 ms.
I agree that deriving the polling interval from MTFA would be better than adding a module parameter. Given that MTFA is a conservative upper bound rather than a good estimate of the common activation time, would using a small fraction of it, for example MTFA / 100 clamped to 10..100 ms, be a reasonable policy for v2?
That would give 10 ms for the PM9D3a device above, while keeping the current 100 ms interval for the P5520 case and for large-MTFA devices.
Keith Busch
maintainerQuoted 6 lines · guzebing
The Samsung PM9D3a Gen5 SSD reports MTFA = 10, i.e. 1000 ms.
I also checked another device, an Intel/Solidigm P5520 Gen4 drive. It reports MTFA = 100, i.e. 10000 ms, while the observed online activation time is about 800 ms.
Quoted 8 lines
I agree that deriving the polling interval from MTFA would be better than adding a module parameter. Given that MTFA is a conservative upper bound rather than a good estimate of the common activation time, would using a small fraction of it, for example MTFA / 100 clamped to 10..100 ms, be a reasonable policy for v2?
That would give 10 ms for the PM9D3a device above, while keeping the current 100 ms interval for the P5520 case and for large-MTFA devices.
nvme_wait_ready() has a similar polling loop on the csts register, but it does a udelay_range for 1-2 msecs no matter what the ready timeout is. Maybe just do the same for consistency?
Quoted 13 lines · Keith Busch
Quoted 12 lines · guzebing
The Samsung PM9D3a Gen5 SSD reports MTFA = 10, i.e. 1000 ms.
I also checked another device, an Intel/Solidigm P5520 Gen4 drive. It reports MTFA = 100, i.e. 10000 ms, while the observed online activation time is about 800 ms.
I agree that deriving the polling interval from MTFA would be better than adding a module parameter. Given that MTFA is a conservative upper bound rather than a good estimate of the common activation time, would using a small fraction of it, for example MTFA / 100 clamped to 10..100 ms, be a reasonable policy for v2?
Quoted 6 lines
Quoted 2 lines
That would give 10 ms for the PM9D3a device above, while keeping the current 100 ms interval for the P5520 case and for large-MTFA devices.
nvme_wait_ready()has a similar polling loop on the csts register, but it does audelay_rangefor 1-2 msecs no matter what the ready timeout is. Maybe just do the same for consistency?
Keith, using the same 1-2 ms sleep range as nvme_wait_ready() looks simpler to me as well.
I measured two long-running nvme_wait_ready() calls in the normal nvme-pci NVMe controller-reset path for a live Intel P5520 (SSDPF2KX076T1), not the PCIe FLR fallback:
nvme_disable_ctrl() cleared NVME_CC_ENABLE in ctrl->ctrl_config, wrote the cached value to CC, and then waited for CSTS.RDY == 0:
1353 ms
nvme_enable_ctrl() set NVME_CC_ENABLE in ctrl->ctrl_config, wrote the cached value to CC, and then waited for CSTS.RDY == 1: 1070 ms
Both long-running waits used an effective timeout argument of 30 seconds. The same drive reports MTFA = 100 (10 seconds), while its observed online firmware activation time was about 800 ms. Thus, on this device, the ready waits already use the 1-2 ms sleep range for longer than the observed firmware activation.
Thanks both for the suggestions. Christoph, would this approach work for you as well? If so, I can respin v2 to drop the module parameter and replace msleep(100) with usleep_range(1000, 2000) in nvme_fw_act_work().
Christoph Hellwig
maintainerQuoted 6 lines · guzebing
Quoted 3 lines
nvme_wait_ready()has a similar polling loop on the csts register, but it does audelay_rangefor 1-2 msecs no matter what the ready timeout is. Maybe just do the same for consistency?Keith, using the same 1-2 ms sleep range as
nvme_wait_ready()looks simpler to me as well.
Let's go for that then.