Reset timeout for paused hardware
Subsystems (2)
cover Reset timeout for paused hardware
Hardware may temporarily stop processing commands that have been dispatched to it while activating new firmware. Some target implementation's paused state time exceeds the default request expiry, so any request dispatched before the driver could quiesce for the hardware's paused state will time out, and handling this may interrupt the firmware activation.
This two-part series provides a way for drivers to reset dispatched requests' timeout deadline, then uses this new mechanism from the nvme driver's fw activation work.
Keith Busch (2):
blk-mq: provide way to reset rq timeouts
nvme: reset request timeouts during fw activation
block/blk-mq.c | 30 ++++++++++++++++++++++++++++++ drivers/nvme/host/core.c | 20 ++++++++++++++++++++ include/linux/blk-mq.h | 1 + 3 files changed, 51 insertions(+)
Signature · 1 line
2.14.4
PATCH 1/2 blk-mq: provide way to reset rq deadline
If hardware has temporarily paused processing requests that its driver has dispatched, the driver may need to halt timeout detection during that temporary stoppage. When the hardware has un-paused request processing, the driver needs a way to rebase all dispatched request dealines using current jiffies so that time accrued during the paused state doesn't count against that request.
--- block/blk-mq.c | 30 ++++++++++++++++++++++++++++++ include/linux/blk-mq.h | 1 + 2 files changed, 31 insertions(+)
block/blk-mq.c +30 -0
@@ -960,6 +960,36 @@ static void blk_mq_timeout_work(struct work_struct *work)blk_queue_exit (q); } +static bool blk_mq_reset_rq (struct blk_mq_hw_ctx *hctx,struct request *rq, +void *priv,bool reserved) +{ +if (blk_mq_rq_state (rq) ==MQ_RQ_IN_FLIGHT ) +blk_add_timer (rq); +return true; +} + +/** +* blk_mq_reset_rqs - reset the timers on all inflight requests +* +* @q - request queue to iterate +* +* This is intended for use when a driver detects its hardware has temporarily +* paused processing commands. When the condition is initially detected, the +* driver should call blk_mq_quiesce_queue() to block new requests from +* dispatching, then blk_sync_queue() to halt any timeout work. When the +* hardware becomes operational again, the driver should call this function to +* reset previously dispatched request timers who's processing had been paused +* by the hardware, then call blk_mq_unquiesce_queue() to unblock future +* dispatch. +*/ +void blk_mq_reset_rqs (struct request_queue *q) +{ +if (WARN_ON (!blk_queue_quiesced (q))) +return ; +blk_mq_queue_tag_busy_iter (q, blk_mq_reset_rq,NULL ); +} +EXPORT_SYMBOL_GPL (blk_mq_reset_rqs); +struct flush_busy_ctx_data {struct blk_mq_hw_ctx *hctx;struct list_head *list;
include/linux/blk-mq.h +1 -0
@@ -327,6 +327,7 @@ void blk_freeze_queue_start(struct request_queue *q);void blk_mq_freeze_queue_wait (struct request_queue *q);int blk_mq_freeze_queue_wait_timeout (struct request_queue *q,unsigned long timeout); +void blk_mq_reset_rqs (struct request_queue *q);int blk_mq_map_queues (struct blk_mq_queue_map *qmap);void blk_mq_update_nr_hw_queues (struct blk_mq_tag_set *set,int nr_hw_queues);
Signature · 1 line
2.14.4
PATCH 2/2 nvme: reset request timeouts during fw activation
In reply to Keith BuschThe nvme controller may pause command processing during firmware activation. The driver will quiesce queues during this time, but commands dispatched prior to the notification will not be processed until the hardware completes this activation.
We do not want those requests to time out while the hardware is in this paused state as we don't expect those commands to complete during this time, and that handling will interfere with the firmware activation process.
In addition to quiescing the queues, halt timeout detection during the paused state and reset the dispatched request deadlines when the hardware exists that state. This action applies to IO and admin queues.
--- drivers/nvme/host/core.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
drivers/nvme/host/core.c +20 -0
@@ -89,6 +89,7 @@ static dev_t nvme_chr_devt;static struct class *nvme_class;static struct class *nvme_subsys_class; +static void nvme_reset_queues (struct nvme_ctrl *ctrl);static int nvme_revalidate_disk (struct gendisk *disk);static void nvme_put_subsystem (struct nvme_subsystem *subsys); static void nvme_remove_invalid_namespaces(struct nvme_ctrl *ctrl,
@@ -3605,6 +3606,11 @@ static void nvme_fw_act_work(struct work_struct *work)msecs_to_jiffies (admin_timeout *1000 );nvme_stop_queues (ctrl); +nvme_sync_queues (ctrl); + +blk_mq_quiesce_queue (ctrl->admin_q); +blk_sync_queue (ctrl->admin_q); +while (nvme_ctrl_pp_status (ctrl)) {if (time_after (jiffies, fw_act_timeout)) { dev_warn(ctrl->device,
@@ -3618,7 +3624,12 @@ static void nvme_fw_act_work(struct work_struct *work)if (ctrl->state !=NVME_CTRL_LIVE )return ; +blk_mq_reset_rqs (ctrl->admin_q); +blk_mq_unquiesce_queue (ctrl->admin_q); + +nvme_reset_queues (ctrl);nvme_start_queues (ctrl); +/* read FW slot information to clear the AER */ nvme_get_fw_slot_info (ctrl); }
@@ -3901,6 +3912,15 @@ void nvme_start_queues(struct nvme_ctrl *ctrl) }EXPORT_SYMBOL_GPL (nvme_start_queues); +static void nvme_reset_queues (struct nvme_ctrl *ctrl) +{ +struct nvme_ns *ns; + +down_read (&ctrl->namespaces_rwsem); +list_for_each_entry (ns, &ctrl->namespaces, list) +blk_mq_reset_rqs (ns->queue); +up_read (&ctrl->namespaces_rwsem); +}void nvme_sync_queues (struct nvme_ctrl *ctrl) {
Signature · 1 line
2.14.4
Re: [PATCH 2/2] nvme: reset request timeouts during fw activation
Quoted 32 lines · Keith Busch
The nvme controller may pause command processing during firmware activation. The driver will quiesce queues during this time, but commands dispatched prior to the notification will not be processed until the hardware completes this activation.
We do not want those requests to time out while the hardware is in this paused state as we don't expect those commands to complete during this time, and that handling will interfere with the firmware activation process.
In addition to quiescing the queues, halt timeout detection during the paused state and reset the dispatched request deadlines when the hardware exists that state. This action applies to IO and admin queues.
Signed-off-by Keith Busch <keith.busch at intel.com>--- drivers/nvme/host/core.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)drivers/nvme/host/core.c +1
-0@@ -89,6 +89,7 @@ static dev_t nvme_chr_devt;static struct class *nvme_class;static struct class *nvme_subsys_class; +static void nvme_reset_queues (struct nvme_ctrl *ctrl);static int nvme_revalidate_disk (struct gendisk *disk);static void nvme_put_subsystem (struct nvme_subsystem *subsys); static void nvme_remove_invalid_namespaces(struct nvme_ctrl *ctrl,
drivers/nvme/host/core.c static void nvme_fw_act_work(struct work_struct *work) · line 3612
Quoted 7 lines
drivers/nvme/host/core.c +4
-0@@ -3605,6 +3606,11 @@ static void nvme_fw_act_work(struct work_struct *work)msecs_to_jiffies (admin_timeout *1000 );nvme_stop_queues (ctrl); +nvme_sync_queues (ctrl); + +blk_mq_quiesce_queue (ctrl->admin_q); +blk_sync_queue (ctrl->admin_q);
blk_sync_queue() may not halt timeout detection completely since the timeout work may reset timer again.
Also reset still may come during activating FW, is that a problem?
Thanks,
Ming
Keith Busch
maintainerRe: [PATCH 2/2] nvme: reset request timeouts during fw activation
Quoted 4 lines · Ming Lei
Quoted 3 lines · Keith Busch
drivers/nvme/host/core.c +0
-0@@ -3605,6 +3606,11 @@ static void nvme_fw_act_work(struct work_struct *work)msecs_to_jiffies (admin_timeout *1000 );
Quoted 8 lines
drivers/nvme/host/core.c static void nvme_fw_act_work(struct work_struct *work) · line 3612
Quoted 5 lines
drivers/nvme/host/core.c +4
-0nvme_stop_queues (ctrl); +nvme_sync_queues (ctrl); + +blk_mq_quiesce_queue (ctrl->admin_q); +blk_sync_queue (ctrl->admin_q);
blk_sync_queue()may not halt timeout detection completely since the timeout work may reset timer again.
Doh! Didn't hit that in testing, but point taken.
Quoted 1 line
Also reset still may come during activating FW, is that a problem?
IO timeout and user initiated resets should be avoided. A state machine addition may be useful here.
Christoph Hellwig
maintainerRe: [PATCH 2/2] nvme: reset request timeouts during fw activation
Quoted 6 lines · Keith Busch
Doh! Didn't hit that in testing, but point taken.
Quoted 1 line
Also reset still may come during activating FW, is that a problem?
IO timeout and user initiated resets should be avoided. A state machine addition may be useful here.
Yep. It almost sounds like we'd want a PAUSED state where resets just keep returning RESET_TIMER without any other action.
Quoted 10 lines · Keith Busch
Hardware may temporarily stop processing commands that have been dispatched to it while activating new firmware. Some target implementation's paused state time exceeds the default request expiry, so any request dispatched before the driver could quiesce for the hardware's paused state will time out, and handling this may interrupt the firmware activation.
This two-part series provides a way for drivers to reset dispatched requests' timeout deadline, then uses this new mechanism from the nvme driver's fw activation work.
Hi Keith,
Is it essential to modify the block layer to implement this behavior change? Would it be possible to implement this behavior change by modifying the NVMe driver only, e.g. by modifying the nvme_timeout() function and by making that function return BLK_EH_RESET_TIMER while new firmware is being activated?
Thanks,
Bart.
Keith Busch
maintainerQuoted 12 lines · Bart Van Assche
Quoted 10 lines · Keith Busch
Hardware may temporarily stop processing commands that have been dispatched to it while activating new firmware. Some target implementation's paused state time exceeds the default request expiry, so any request dispatched before the driver could quiesce for the hardware's paused state will time out, and handling this may interrupt the firmware activation.
This two-part series provides a way for drivers to reset dispatched requests' timeout deadline, then uses this new mechanism from the nvme driver's fw activation work.
Quoted 7 lines
Hi Keith,
Is it essential to modify the block layer to implement this behavior change? Would it be possible to implement this behavior change by modifying the NVMe driver only, e.g. by modifying the
nvme_timeout()function and by making that function returnBLK_EH_RESET_TIMERwhile new firmware is being activated?
Good question.
We can't just do this from nvme_timeout(), though. That introduces races between timeout_work and fw_act_work if that fw work clears the condition that timeout needs to observe to return RESET_TIMER.
Even if we avoid that race, the rq->deadline needs to be adjusted to the current time after the h/w unpause because the time accumulated while h/w halted itself should not be counted against the request.
Quoted 23 lines · Keith Busch
Quoted 12 lines · Bart Van Assche
Quoted 10 lines · Keith Busch
Hardware may temporarily stop processing commands that have been dispatched to it while activating new firmware. Some target implementation's paused state time exceeds the default request expiry, so any request dispatched before the driver could quiesce for the hardware's paused state will time out, and handling this may interrupt the firmware activation.
This two-part series provides a way for drivers to reset dispatched requests' timeout deadline, then uses this new mechanism from the nvme driver's fw activation work.
Quoted 7 lines
Hi Keith,
Is it essential to modify the block layer to implement this behavior change? Would it be possible to implement this behavior change by modifying the NVMe driver only, e.g. by modifying the
nvme_timeout()function and by making that function returnBLK_EH_RESET_TIMERwhile new firmware is being activated?Good question.
Quoted 7 lines
We can't just do this from
nvme_timeout(), though. That introduces races betweentimeout_workandfw_act_workif that fw work clears the condition that timeout needs to observe to returnRESET_TIMER.Even if we avoid that race, the
rq->deadlineneeds to be adjusted to the current time after the h/w unpause because the time accumulated while h/w halted itself should not be counted against the request.
Hi Keith,
How about recording the time at which the firmware upgrade finished and making nvme_timeout() return RESET_TIMER if either a firmware upgrade is in progress or if it has finished less than (request timeout) seconds ago? The reason I'm asking this is because I'm concerned that the patches you posted would need a careful review to see whether or not these trigger new race conditions.
Thanks,
Bart.
Quoted 10 lines · Keith Busch
Hardware may temporarily stop processing commands that have been dispatched to it while activating new firmware. Some target implementation's paused state time exceeds the default request expiry, so any request dispatched before the driver could quiesce for the hardware's paused state will time out, and handling this may interrupt the firmware activation.
This two-part series provides a way for drivers to reset dispatched requests' timeout deadline, then uses this new mechanism from the nvme driver's fw activation work.
Just wondering why not freeze IO queues before updating FW?
Thanks,
Ming
Quoted 9 lines · Ming Lei
Quoted 7 lines · Keith Busch
Hardware may temporarily stop processing commands that have been dispatched to it while activating new firmware. Some target implementation's paused state time exceeds the default request expiry, so any request dispatched before the driver could quiesce for the hardware's paused state will time out, and handling this may interrupt the firmware activation.
Quoted 5 lines
Quoted 3 lines
This two-part series provides a way for drivers to reset dispatched requests' timeout deadline, then uses this new mechanism from the nvme driver's fw activation work.
Just wondering why not freeze IO queues before updating FW?
Yeah, that's a good question. A FW update may have been initiated out of band or from another host entirely. The driver can't count on preparing for hardware pausing command processing before it's happened, but we'll always find out asynchronously after it's too late to freeze.
Christoph Hellwig
maintainerQuoted 5 lines · Keith Busch
Yeah, that's a good question. A FW update may have been initiated out of band or from another host entirely. The driver can't count on preparing for hardware pausing command processing before it's happened, but we'll always find out asynchronously after it's too late to freeze.
I don't think that is the case at least for spec compliant devices.
Quoted 1 line
From NVMe 1.3:
Figure 49: Asynchronous Event Information - Notice
1h Firmware Activation Starting: The controller is starting
a firmware activation process during which command processing is paused. Host software may use CSTS.PP to determine when command processing has resumed. To clear this event, host software reads the Firmware Slot Information log page.
So we are supposed to get an AEN before the device stops processing commands.
Keith Busch
maintainerQuoted 20 lines · Christoph Hellwig
Quoted 5 lines · Keith Busch
Yeah, that's a good question. A FW update may have been initiated out of band or from another host entirely. The driver can't count on preparing for hardware pausing command processing before it's happened, but we'll always find out asynchronously after it's too late to freeze.
I don't think that is the case at least for spec compliant devices.
From NVMe 1.3:
Figure 49: Asynchronous Event Information - Notice
1h Firmware Activation Starting: The controller is starting
a firmware activation process during which command processing is paused. Host software may use CSTS.PP to determine when command processing has resumed. To clear this event, host software reads the Firmware Slot Information log page.
Quoted 2 lines
So we are supposed to get an AEN before the device stops processing commands.
Hm, I read the same section, but conclude differently (and at least some vendors did too). A spec compliant controller activating new firmware without reset would stop processing commands and set CSTS.PP first, then send the AEN. When the host is aware to poll Processing Paused, the controller hasn't been processing new commands for some time.
Could you give some more detail on your interpretation?
Christoph Hellwig
maintainerQuoted 12 lines · Keith Busch
Quoted 11 lines
Figure 49: Asynchronous Event Information - Notice
1h Firmware Activation Starting: The controller is starting
a firmware activation process during which command processing is paused. Host software may use CSTS.PP to determine when command processing has resumed. To clear this event, host software reads the Firmware Slot Information log page.So we are supposed to get an AEN before the device stops processing commands.
Quoted 7 lines
Hm, I read the same section, but conclude differently (and at least some vendors did too). A spec compliant controller activating new firmware without reset would stop processing commands and set CSTS.PP first, then send the AEN. When the host is aware to poll Processing Paused, the controller hasn't been processing new commands for some time.
Could you give some more detail on your interpretation?
What would be the point of the AEN if it wasn't sent at exactly the point when the controller stops acceppting command?
The wording is of course NVMe-like, but "The controller is starting a.." pretty clear implies this is sent at the beginning of the paused state, not the end.
Keith Busch
maintainerQuoted 19 lines · Christoph Hellwig
Quoted 18 lines · Keith Busch
Quoted 11 lines
Figure 49: Asynchronous Event Information - Notice
1h Firmware Activation Starting: The controller is starting
a firmware activation process during which command processing is paused. Host software may use CSTS.PP to determine when command processing has resumed. To clear this event, host software reads the Firmware Slot Information log page.So we are supposed to get an AEN before the device stops processing commands.
Hm, I read the same section, but conclude differently (and at least some vendors did too). A spec compliant controller activating new firmware without reset would stop processing commands and set CSTS.PP first, then send the AEN. When the host is aware to poll Processing Paused, the controller hasn't been processing new commands for some time.
Quoted 8 lines
Quoted 1 line
Could you give some more detail on your interpretation?
What would be the point of the AEN if it wasn't sent at exactly the point when the controller stops acceppting command?
The wording is of course NVMe-like, but "The controller is starting a.." pretty clear implies this is sent at the beginning of the paused state, not the end.
Right, the controller starts the pause at time T1, and sends the AEN at the same time. No new commands will be processed after T1 until the firmware activate completes.
The host sees the AEN a short time later, time T2.
The time between T1 - T2, we may have sent many commands that are not going to be processed, and we couldn't have known that when they were submitted. The controller still owns those commands, and we just need to adjust their deadlines once processing resumes.