Skip to contentSkip to conversation

Reset timeout for paused hardware

PATCH In review Keith Busch 6 participants 15 messages May 22, 2019 – May 23, 2019

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover Reset timeout for paused hardware

Hardware may temporarily stop processing commands that have been dispatched to it while activating new firmware. Some target implementation's paused state time exceeds the default request expiry, so any request dispatched before the driver could quiesce for the hardware's paused state will time out, and handling this may interrupt the firmware activation.

This two-part series provides a way for drivers to reset dispatched requests' timeout deadline, then uses this new mechanism from the nvme driver's fw activation work.

Keith Busch (2):
blk-mq: provide way to reset rq timeouts
nvme: reset request timeouts during fw activation

 block/blk-mq.c           | 30 ++++++++++++++++++++++++++++++
 drivers/nvme/host/core.c | 20 ++++++++++++++++++++
 include/linux/blk-mq.h   |  1 +
 3 files changed, 51 insertions(+)
Signature · 1 line
2.14.4
#1 lore raw

PATCH 1/2 blk-mq: provide way to reset rq deadline

If hardware has temporarily paused processing requests that its driver has dispatched, the driver may need to halt timeout detection during that temporary stoppage. When the hardware has un-paused request processing, the driver needs a way to rebase all dispatched request dealines using current jiffies so that time accrued during the paused state doesn't count against that request.

Signed-off-by Keith Busch <keith.busch at intel.com>
---
 block/blk-mq.c         | 30 ++++++++++++++++++++++++++++++
 include/linux/blk-mq.h |  1 +
 2 files changed, 31 insertions(+)
block/blk-mq.c +30 -0
index 08a6248d8536..9d85903d4e0c 100644
@@ -960,6 +960,36 @@ static void blk_mq_timeout_work(struct work_struct *work)
 	blk_queue_exit(q);
 }
 
+static bool blk_mq_reset_rq(struct blk_mq_hw_ctx *hctx, struct request *rq,
+			    void *priv, bool reserved)
+{
+	if (blk_mq_rq_state(rq) == MQ_RQ_IN_FLIGHT)
+		blk_add_timer(rq);
+	return true;
+}
+
+/**
+ * blk_mq_reset_rqs - reset the timers on all inflight requests
+ *
+ * @q - request queue to iterate
+ *
+ * This is intended for use when a driver detects its hardware has temporarily
+ * paused processing commands. When the condition is initially detected, the
+ * driver should call blk_mq_quiesce_queue() to block new requests from
+ * dispatching, then blk_sync_queue() to halt any timeout work. When the
+ * hardware becomes operational again, the driver should call this function to
+ * reset previously dispatched request timers who's processing had been paused
+ * by the hardware, then call blk_mq_unquiesce_queue() to unblock future
+ * dispatch.
+ */
+void blk_mq_reset_rqs(struct request_queue *q)
+{
+	if (WARN_ON(!blk_queue_quiesced(q)))
+		return;
+	blk_mq_queue_tag_busy_iter(q, blk_mq_reset_rq, NULL);
+}
+EXPORT_SYMBOL_GPL(blk_mq_reset_rqs);
+
 struct flush_busy_ctx_data {
 	struct blk_mq_hw_ctx *hctx;
 	struct list_head *list;
include/linux/blk-mq.h +1 -0
index 15d1aa53d96c..28c421ba5094 100644
@@ -327,6 +327,7 @@ void blk_freeze_queue_start(struct request_queue *q);
 void blk_mq_freeze_queue_wait(struct request_queue *q);
 int blk_mq_freeze_queue_wait_timeout(struct request_queue *q,
 				     unsigned long timeout);
+void blk_mq_reset_rqs(struct request_queue *q);
 
 int blk_mq_map_queues(struct blk_mq_queue_map *qmap);
 void blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, int nr_hw_queues);
Signature · 1 line
2.14.4
#2 lore raw

PATCH 2/2 nvme: reset request timeouts during fw activation

In reply to Keith Busch

The nvme controller may pause command processing during firmware activation. The driver will quiesce queues during this time, but commands dispatched prior to the notification will not be processed until the hardware completes this activation.

We do not want those requests to time out while the hardware is in this paused state as we don't expect those commands to complete during this time, and that handling will interfere with the firmware activation process.

In addition to quiescing the queues, halt timeout detection during the paused state and reset the dispatched request deadlines when the hardware exists that state. This action applies to IO and admin queues.

Signed-off-by Keith Busch <keith.busch at intel.com>
---
 drivers/nvme/host/core.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
drivers/nvme/host/core.c +20 -0
index 1b7c2afd84cb..37a9a66ada22 100644
@@ -89,6 +89,7 @@ static dev_t nvme_chr_devt;
 static struct class *nvme_class;
 static struct class *nvme_subsys_class;
 
+static void nvme_reset_queues(struct nvme_ctrl *ctrl);
 static int nvme_revalidate_disk(struct gendisk *disk);
 static void nvme_put_subsystem(struct nvme_subsystem *subsys);
 static void nvme_remove_invalid_namespaces(struct nvme_ctrl *ctrl,
@@ -3605,6 +3606,11 @@ static void nvme_fw_act_work(struct work_struct *work)
 				msecs_to_jiffies(admin_timeout * 1000);
 
 	nvme_stop_queues(ctrl);
+	nvme_sync_queues(ctrl);
+
+	blk_mq_quiesce_queue(ctrl->admin_q);
+	blk_sync_queue(ctrl->admin_q);
+
 	while (nvme_ctrl_pp_status(ctrl)) {
 		if (time_after(jiffies, fw_act_timeout)) {
 			dev_warn(ctrl->device,
@@ -3618,7 +3624,12 @@ static void nvme_fw_act_work(struct work_struct *work)
 	if (ctrl->state != NVME_CTRL_LIVE)
 		return;
 
+	blk_mq_reset_rqs(ctrl->admin_q);
+	blk_mq_unquiesce_queue(ctrl->admin_q);
+
+	nvme_reset_queues(ctrl);
 	nvme_start_queues(ctrl);
+
 	/* read FW slot information to clear the AER */
 	nvme_get_fw_slot_info(ctrl);
 }
@@ -3901,6 +3912,15 @@ void nvme_start_queues(struct nvme_ctrl *ctrl)
 }
 EXPORT_SYMBOL_GPL(nvme_start_queues);
 
+static void nvme_reset_queues(struct nvme_ctrl *ctrl)
+{
+	struct nvme_ns *ns;
+
+	down_read(&ctrl->namespaces_rwsem);
+	list_for_each_entry(ns, &ctrl->namespaces, list)
+		blk_mq_reset_rqs(ns->queue);
+	up_read(&ctrl->namespaces_rwsem);
+}
 
 void nvme_sync_queues(struct nvme_ctrl *ctrl)
 {
Signature · 1 line
2.14.4
#3 lore raw

Re: [PATCH 2/2] nvme: reset request timeouts during fw activation

Quoted 32 lines · Keith Busch

The nvme controller may pause command processing during firmware activation. The driver will quiesce queues during this time, but commands dispatched prior to the notification will not be processed until the hardware completes this activation.

We do not want those requests to time out while the hardware is in this paused state as we don't expect those commands to complete during this time, and that handling will interfere with the firmware activation process.

In addition to quiescing the queues, halt timeout detection during the paused state and reset the dispatched request deadlines when the hardware exists that state. This action applies to IO and admin queues.

Signed-off-by Keith Busch <keith.busch at intel.com>
---
 drivers/nvme/host/core.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
drivers/nvme/host/core.c +1 -0
index 1b7c2afd84cb..37a9a66ada22 100644
@@ -89,6 +89,7 @@ static dev_t nvme_chr_devt;
 static struct class *nvme_class;
 static struct class *nvme_subsys_class;
 
+static void nvme_reset_queues(struct nvme_ctrl *ctrl);
 static int nvme_revalidate_disk(struct gendisk *disk);
 static void nvme_put_subsystem(struct nvme_subsystem *subsys);
 static void nvme_remove_invalid_namespaces(struct nvme_ctrl *ctrl,

drivers/nvme/host/core.c static void nvme_fw_act_work(struct work_struct *work) · line 3612

Quoted 7 lines
drivers/nvme/host/core.c +4 -0
@@ -3605,6 +3606,11 @@ static void nvme_fw_act_work(struct work_struct *work)
 				msecs_to_jiffies(admin_timeout * 1000);
 
 	nvme_stop_queues(ctrl);
+	nvme_sync_queues(ctrl);
+
+	blk_mq_quiesce_queue(ctrl->admin_q);
+	blk_sync_queue(ctrl->admin_q);

blk_sync_queue() may not halt timeout detection completely since the timeout work may reset timer again.

Also reset still may come during activating FW, is that a problem?

Thanks,
Ming

maintainer
#4 lore raw

Re: [PATCH 2/2] nvme: reset request timeouts during fw activation

Quoted 4 lines · Ming Lei
Quoted 3 lines · Keith Busch
drivers/nvme/host/core.c +0 -0
@@ -3605,6 +3606,11 @@ static void nvme_fw_act_work(struct work_struct *work)
 				msecs_to_jiffies(admin_timeout * 1000);
 
Quoted 8 lines

drivers/nvme/host/core.c static void nvme_fw_act_work(struct work_struct *work) · line 3612

Quoted 5 lines
drivers/nvme/host/core.c +4 -0
 	nvme_stop_queues(ctrl);
+	nvme_sync_queues(ctrl);
+
+	blk_mq_quiesce_queue(ctrl->admin_q);
+	blk_sync_queue(ctrl->admin_q);

blk_sync_queue() may not halt timeout detection completely since the timeout work may reset timer again.

Doh! Didn't hit that in testing, but point taken.

Quoted 1 line

Also reset still may come during activating FW, is that a problem?

IO timeout and user initiated resets should be avoided. A state machine addition may be useful here.

#5 lore raw

Re: [PATCH 2/2] nvme: reset request timeouts during fw activation

Quoted 6 lines · Keith Busch

Doh! Didn't hit that in testing, but point taken.

Quoted 1 line

Also reset still may come during activating FW, is that a problem?

IO timeout and user initiated resets should be avoided. A state machine addition may be useful here.

Yep. It almost sounds like we'd want a PAUSED state where resets just keep returning RESET_TIMER without any other action.

#6 lore raw
In reply to Keith Busch
Quoted 10 lines · Keith Busch

Hardware may temporarily stop processing commands that have been dispatched to it while activating new firmware. Some target implementation's paused state time exceeds the default request expiry, so any request dispatched before the driver could quiesce for the hardware's paused state will time out, and handling this may interrupt the firmware activation.

This two-part series provides a way for drivers to reset dispatched requests' timeout deadline, then uses this new mechanism from the nvme driver's fw activation work.

Hi Keith,

Is it essential to modify the block layer to implement this behavior change? Would it be possible to implement this behavior change by modifying the NVMe driver only, e.g. by modifying the nvme_timeout() function and by making that function return BLK_EH_RESET_TIMER while new firmware is being activated?

Thanks,

Bart.

maintainer
#7 lore raw
Quoted 12 lines · Bart Van Assche
Quoted 10 lines · Keith Busch

Hardware may temporarily stop processing commands that have been dispatched to it while activating new firmware. Some target implementation's paused state time exceeds the default request expiry, so any request dispatched before the driver could quiesce for the hardware's paused state will time out, and handling this may interrupt the firmware activation.

This two-part series provides a way for drivers to reset dispatched requests' timeout deadline, then uses this new mechanism from the nvme driver's fw activation work.

Quoted 7 lines

Hi Keith,

Is it essential to modify the block layer to implement this behavior change? Would it be possible to implement this behavior change by modifying the NVMe driver only, e.g. by modifying the nvme_timeout() function and by making that function return BLK_EH_RESET_TIMER while new firmware is being activated?

Good question.

We can't just do this from nvme_timeout(), though. That introduces races between timeout_work and fw_act_work if that fw work clears the condition that timeout needs to observe to return RESET_TIMER.

Even if we avoid that race, the rq->deadline needs to be adjusted to the current time after the h/w unpause because the time accumulated while h/w halted itself should not be counted against the request.

#8 lore raw
Quoted 23 lines · Keith Busch
Quoted 12 lines · Bart Van Assche
Quoted 10 lines · Keith Busch

Hardware may temporarily stop processing commands that have been dispatched to it while activating new firmware. Some target implementation's paused state time exceeds the default request expiry, so any request dispatched before the driver could quiesce for the hardware's paused state will time out, and handling this may interrupt the firmware activation.

This two-part series provides a way for drivers to reset dispatched requests' timeout deadline, then uses this new mechanism from the nvme driver's fw activation work.

Quoted 7 lines

Hi Keith,

Is it essential to modify the block layer to implement this behavior change? Would it be possible to implement this behavior change by modifying the NVMe driver only, e.g. by modifying the nvme_timeout() function and by making that function return BLK_EH_RESET_TIMER while new firmware is being activated?

Good question.

Quoted 7 lines

We can't just do this from nvme_timeout(), though. That introduces races between timeout_work and fw_act_work if that fw work clears the condition that timeout needs to observe to return RESET_TIMER.

Even if we avoid that race, the rq->deadline needs to be adjusted to the current time after the h/w unpause because the time accumulated while h/w halted itself should not be counted against the request.

Hi Keith,

How about recording the time at which the firmware upgrade finished and making nvme_timeout() return RESET_TIMER if either a firmware upgrade is in progress or if it has finished less than (request timeout) seconds ago? The reason I'm asking this is because I'm concerned that the patches you posted would need a careful review to see whether or not these trigger new race conditions.

Thanks,

Bart.

#9 lore raw
In reply to Keith Busch
Quoted 10 lines · Keith Busch

Hardware may temporarily stop processing commands that have been dispatched to it while activating new firmware. Some target implementation's paused state time exceeds the default request expiry, so any request dispatched before the driver could quiesce for the hardware's paused state will time out, and handling this may interrupt the firmware activation.

This two-part series provides a way for drivers to reset dispatched requests' timeout deadline, then uses this new mechanism from the nvme driver's fw activation work.

Just wondering why not freeze IO queues before updating FW?

Thanks,
Ming

#10 lore raw
Quoted 9 lines · Ming Lei
Quoted 7 lines · Keith Busch

Hardware may temporarily stop processing commands that have been dispatched to it while activating new firmware. Some target implementation's paused state time exceeds the default request expiry, so any request dispatched before the driver could quiesce for the hardware's paused state will time out, and handling this may interrupt the firmware activation.

Quoted 5 lines
Quoted 3 lines

This two-part series provides a way for drivers to reset dispatched requests' timeout deadline, then uses this new mechanism from the nvme driver's fw activation work.

Just wondering why not freeze IO queues before updating FW?

Yeah, that's a good question. A FW update may have been initiated out of band or from another host entirely. The driver can't count on preparing for hardware pausing command processing before it's happened, but we'll always find out asynchronously after it's too late to freeze.

#11 lore raw
Quoted 5 lines · Keith Busch

Yeah, that's a good question. A FW update may have been initiated out of band or from another host entirely. The driver can't count on preparing for hardware pausing command processing before it's happened, but we'll always find out asynchronously after it's too late to freeze.

I don't think that is the case at least for spec compliant devices.

Quoted 1 line

From NVMe 1.3:

Figure 49: Asynchronous Event Information - Notice

1h Firmware Activation Starting: The controller is starting
a firmware activation process during which command processing is paused. Host software may use CSTS.PP to determine when command processing has resumed. To clear this event, host software reads the Firmware Slot Information log page.

So we are supposed to get an AEN before the device stops processing commands.

maintainer
#12 lore raw
Quoted 20 lines · Christoph Hellwig
Quoted 5 lines · Keith Busch

Yeah, that's a good question. A FW update may have been initiated out of band or from another host entirely. The driver can't count on preparing for hardware pausing command processing before it's happened, but we'll always find out asynchronously after it's too late to freeze.

I don't think that is the case at least for spec compliant devices.

From NVMe 1.3:

Figure 49: Asynchronous Event Information - Notice

1h Firmware Activation Starting: The controller is starting
a firmware activation process during which command processing is paused. Host software may use CSTS.PP to determine when command processing has resumed. To clear this event, host software reads the Firmware Slot Information log page.

Quoted 2 lines

So we are supposed to get an AEN before the device stops processing commands.

Hm, I read the same section, but conclude differently (and at least some vendors did too). A spec compliant controller activating new firmware without reset would stop processing commands and set CSTS.PP first, then send the AEN. When the host is aware to poll Processing Paused, the controller hasn't been processing new commands for some time.

Could you give some more detail on your interpretation?

#13 lore raw
Quoted 12 lines · Keith Busch
Quoted 11 lines

Figure 49: Asynchronous Event Information - Notice

1h Firmware Activation Starting: The controller is starting
a firmware activation process during which command processing is paused. Host software may use CSTS.PP to determine when command processing has resumed. To clear this event, host software reads the Firmware Slot Information log page.

So we are supposed to get an AEN before the device stops processing commands.

Quoted 7 lines

Hm, I read the same section, but conclude differently (and at least some vendors did too). A spec compliant controller activating new firmware without reset would stop processing commands and set CSTS.PP first, then send the AEN. When the host is aware to poll Processing Paused, the controller hasn't been processing new commands for some time.

Could you give some more detail on your interpretation?

What would be the point of the AEN if it wasn't sent at exactly the point when the controller stops acceppting command?

The wording is of course NVMe-like, but "The controller is starting a.." pretty clear implies this is sent at the beginning of the paused state, not the end.

maintainer
#14 lore raw
Quoted 19 lines · Christoph Hellwig
Quoted 18 lines · Keith Busch
Quoted 11 lines

Figure 49: Asynchronous Event Information - Notice

1h Firmware Activation Starting: The controller is starting
a firmware activation process during which command processing is paused. Host software may use CSTS.PP to determine when command processing has resumed. To clear this event, host software reads the Firmware Slot Information log page.

So we are supposed to get an AEN before the device stops processing commands.

Hm, I read the same section, but conclude differently (and at least some vendors did too). A spec compliant controller activating new firmware without reset would stop processing commands and set CSTS.PP first, then send the AEN. When the host is aware to poll Processing Paused, the controller hasn't been processing new commands for some time.

Quoted 8 lines
Quoted 1 line

Could you give some more detail on your interpretation?

What would be the point of the AEN if it wasn't sent at exactly the point when the controller stops acceppting command?

The wording is of course NVMe-like, but "The controller is starting a.." pretty clear implies this is sent at the beginning of the paused state, not the end.

Right, the controller starts the pause at time T1, and sends the AEN at the same time. No new commands will be processed after T1 until the firmware activate completes.

The host sees the AEN a short time later, time T2.

The time between T1 - T2, we may have sent many commands that are not going to be processed, and we couldn't have known that when they were submitted. The controller still owns those commands, and we just need to adjust their deadlines once processing resumes.