Skip to contentSkip to conversation

nvme: Improve processing paused support

PATCH New Keith Busch 5 participants 12 messages May 24, 2019 – Jun 1, 2019 NVMe

lore
Share & tools
lore
#0 lore raw

cover nvme: Improve processing paused support

From: Keith Busch <[email protected]>

Improve firmware activation over what we're currently doing. The main thing is that we do not want to esacalate recovery while the current status has CSTS.PP set as the recovery attempt may interfere with a critical controller operation.

There still exists a race where the controller may clear CSTS Processing Paused at the same time a stalled request times out. We would normally want to reset the timer for that, but this is at least a state that can handle error recovery without interrupting background operations, so the only harm caused from encountering this unlikely event is a longer latency back to ready.

My previous patch to address this used help from the block layer, but had some issues. This series only involves the nvme driver and just addresses the main concern without addressing the corner cases.

Keith Busch (3):
nvme-pci: reset timeout when processing is paused
nvme: rearm fw notification in admin only state
nvme: quiesce admin queue for fw activation

 drivers/nvme/host/core.c | 9 ++++++++-
 drivers/nvme/host/pci.c  | 2 +-
 2 files changed, 9 insertions(+), 2 deletions(-)
Signature · 1 line
2.14.4
#1 lore raw

PATCH 1/3 nvme-pci: reset timeout when processing is paused

From: Keith Busch <[email protected]>

Do not escalate request timeout handling if the controller has temporary paused command processing. We do not expect requests to complete during this transient state, so just reset the timer.

Signed-off-by Keith Busch <kbusch at kernel.org>
---
 drivers/nvme/host/pci.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/pci.c +1 -1
index f562154551ce..101e20522374 100644
@@ -1263,7 +1263,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)
 	 * the recovery mechanism will surely fail.
 	 */
 	mb();
-	if (pci_channel_offline(to_pci_dev(dev->dev)))
+	if (pci_channel_offline(to_pci_dev(dev->dev)) || (csts & NVME_CSTS_PP))
 		return BLK_EH_RESET_TIMER;
 
 	/*
Signature · 1 line
2.14.4
#2 lore raw

PATCH 1/3 nvme-pci: reset timeout when processing is paused

Quoted 16 lines · Keith Busch

From: Keith Busch <kbusch at kernel.org>

Do not escalate request timeout handling if the controller has temporary paused command processing. We do not expect requests to complete during this transient state, so just reset the timer.

Signed-off-by Keith Busch <kbusch at kernel.org>
---
 drivers/nvme/host/pci.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/pci.c +0 -0
index f562154551ce..101e20522374 100644

drivers/nvme/host/pci.c static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved) · line 1267

Quoted 6 lines
drivers/nvme/host/pci.c +1 -1
@@ -1263,7 +1263,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)
 	 * the recovery mechanism will surely fail.
 	 */
 	mb();
-	if (pci_channel_offline(to_pci_dev(dev->dev)))
+	if (pci_channel_offline(to_pci_dev(dev->dev)) || (csts & NVME_CSTS_PP))
 		return BLK_EH_RESET_TIMER;

Keith,

Is it okay to check CSTS.PP status without checking CC.EN ? Can't we just do the PP check like:

	if (pci_channel_offline(to_pci_dev(dev->dev)) ||
			nvme_ctrl_pp_status(&dev->ctrl))
		return BLK_EH_RESET_TIMER;

What do you think?

#3 lore raw

PATCH 1/3 nvme-pci: reset timeout when processing is paused

In reply to Keith Busch

drivers/nvme/host/pci.c static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved) · line 1266

Quoted 10 lines · Keith Busch
drivers/nvme/host/pci.c +1 -1
index f562154551ce..101e20522374 100644
@@ -1263,7 +1263,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)
 	 * the recovery mechanism will surely fail.
 	 */
 	mb();
-	if (pci_channel_offline(to_pci_dev(dev->dev)))
+	if (pci_channel_offline(to_pci_dev(dev->dev)) || (csts & NVME_CSTS_PP))

I think we at least need a ratelimited printk when this happens so people know they don't get timeouts because of CSTS.PP.

And maybe we need a timeout how long we allow the timeouts to extended due to CSTS.PP, otherwise a buggy device that never clears it would hold I/O hostage forever.

#4 lore raw

PATCH 2/3 nvme: rearm fw notification in admin only state

In reply to Keith Busch

From: Keith Busch <[email protected]>

Getting the firmware log is an admin command, so admin-only or live states are valid to rearm the activation notice.

Signed-off-by Keith Busch <kbusch at kernel.org>
---
 drivers/nvme/host/core.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +2 -1
index 1b7c2afd84cb..96dac2292897 100644
@@ -3615,7 +3615,8 @@ static void nvme_fw_act_work(struct work_struct *work)
 		msleep(100);
 	}
 
-	if (ctrl->state != NVME_CTRL_LIVE)
+	if (ctrl->state != NVME_CTRL_LIVE &&
+	    ctrl->state != NVME_CTRL_ADMIN_ONLY)
 		return;
 
 	nvme_start_queues(ctrl);
Signature · 1 line
2.14.4
#5 lore raw

PATCH 2/3 nvme: rearm fw notification in admin only state

drivers/nvme/host/core.c static void nvme_fw_act_work(struct work_struct *work) · line 3619

Quoted 2 lines · Keith Busch
drivers/nvme/host/core.c +2 -0
+	if (ctrl->state != NVME_CTRL_LIVE &&
+	    ctrl->state != NVME_CTRL_ADMIN_ONLY)

I think we really

a) need a nvme_ctrl_is_live() helper encapsulating the above
b) audit all checks for NVME_CTRL_LIVE if they really need to use this helper instead.

#6 lore raw

PATCH 3/3 nvme: quiesce admin queue for fw activation

In reply to Keith Busch

From: Keith Busch <[email protected]>

The controller's admin queue processing is paused during firmware activation, just like IO queues. Quiesce these, and inform the syslog this uncommon event is happening.

Signed-off-by Keith Busch <kbusch at kernel.org>
---
 drivers/nvme/host/core.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/nvme/host/core.c +6 -0
index 96dac2292897..5a6d27823f7f 100644
@@ -3604,7 +3604,10 @@ static void nvme_fw_act_work(struct work_struct *work)
 		fw_act_timeout = jiffies +
 				msecs_to_jiffies(admin_timeout * 1000);
 
+	dev_info(ctrl->device, "processing paused for fw activation\n");
 	nvme_stop_queues(ctrl);
+	blk_mq_quiesce_queue(ctrl->admin_q);
+
 	while (nvme_ctrl_pp_status(ctrl)) {
 		if (time_after(jiffies, fw_act_timeout)) {
 			dev_warn(ctrl->device,
@@ -3619,7 +3622,10 @@ static void nvme_fw_act_work(struct work_struct *work)
 	    ctrl->state != NVME_CTRL_ADMIN_ONLY)
 		return;
 
+	dev_info(ctrl->device, "processing resumed\n");
+	blk_mq_unquiesce_queue(ctrl->admin_q);
 	nvme_start_queues(ctrl);
+
 	/* read FW slot information to clear the AER */
 	nvme_get_fw_slot_info(ctrl);
 }
Signature · 1 line
2.14.4
#7 lore raw

PATCH 3/3 nvme: quiesce admin queue for fw activation

drivers/nvme/host/core.c static void nvme_fw_act_work(struct work_struct *work) · line 3609

Quoted 11 lines
drivers/nvme/host/core.c +2 -0
index 96dac2292897..5a6d27823f7f 100644
@@ -3604,7 +3604,10 @@ static void nvme_fw_act_work(struct work_struct *work)
 		fw_act_timeout = jiffies +
 				msecs_to_jiffies(admin_timeout * 1000);
 
+	dev_info(ctrl->device, "processing paused for fw activation\n");
 	nvme_stop_queues(ctrl);
+	blk_mq_quiesce_queue(ctrl->admin_q);

Keith,

Can we have an warning here to indicate if device firmware has not set the CSTS.PP yet. In that case, the information message that you have introduced here may be invalid. It would be great if we check the CSTS.PP first, and then print it out.

If it's not necessary to have it, please feel free to let me know. Of course, device has to prepared the processing paused status, though :)

drivers/nvme/host/core.c static void nvme_fw_act_work(struct work_struct *work) · line 3625

Quoted 9 lines
drivers/nvme/host/core.c +2 -0
+
 	while (nvme_ctrl_pp_status(ctrl)) {
 		if (time_after(jiffies, fw_act_timeout)) {
 			dev_warn(ctrl->device,
@@ -3619,7 +3622,10 @@ static void nvme_fw_act_work(struct work_struct *work)
 	    ctrl->state != NVME_CTRL_ADMIN_ONLY)
 		return;
 
+	dev_info(ctrl->device, "processing resumed\n");

Can we print it out after admin_q, io_q are unquiesced ?

drivers/nvme/host/core.c static void nvme_fw_act_work(struct work_struct *work) · line 3631

Quoted 6 lines
drivers/nvme/host/core.c +2 -0
+	blk_mq_unquiesce_queue(ctrl->admin_q);
 	nvme_start_queues(ctrl);
+
 	/* read FW slot information to clear the AER */
 	nvme_get_fw_slot_info(ctrl);
 }

I am very okay to quiesce the admin_q here.

Thanks,

#8 lore raw

PATCH 3/3 nvme: quiesce admin queue for fw activation

Quoted 14 lines · Minwoo Im

drivers/nvme/host/core.c static void nvme_fw_act_work(struct work_struct *work) · line 3609

Quoted 11 lines
drivers/nvme/host/core.c +2 -0
index 96dac2292897..5a6d27823f7f 100644
@@ -3604,7 +3604,10 @@ static void nvme_fw_act_work(struct work_struct *work)
              fw_act_timeout = jiffies +
                              msecs_to_jiffies(admin_timeout * 1000);
 
+     dev_info(ctrl->device, "processing paused for fw activation\n");
      nvme_stop_queues(ctrl);
+     blk_mq_quiesce_queue(ctrl->admin_q);

Keith,

Quoted 8 lines

Can we have an warning here to indicate if device firmware has not set the CSTS.PP yet. In that case, the information message that you have introduced here may be invalid. It would be great if we check the CSTS.PP first, and then print it out.

If it's not necessary to have it, please feel free to let me know. Of course, device has to prepared the processing paused status, though :)

Well, between the fw notice AEN and when the work can finally check CSTS.PP, we really have no way of knowing if the controller paused and resumed before we observed that controller status, so a warning would be a bit much. We're still quiescing, so the print is valid.

In case the controller did pause processing, and we hit the unusual timeout race that does not see CSTS.PP, the new prints make it clear that's what happened.

#9 lore raw

PATCH 3/3 nvme: quiesce admin queue for fw activation

Quoted 8 lines
Quoted 8 lines

Keith,

Can we have an warning here to indicate if device firmware has not set the CSTS.PP yet. In that case, the information message that you have introduced here may be invalid. It would be great if we check the CSTS.PP first, and then print it out.

Quoted 8 lines
Quoted 3 lines

If it's not necessary to have it, please feel free to let me know. Of course, device has to prepared the processing paused status, though :)

Well, between the fw notice AEN and when the work can finally check CSTS.PP, we really have no way of knowing if the controller paused and resumed before we observed that controller status, so a warning would be a bit much. We're still quiescing, so the print is valid.

That makes sense. The work scheduled from the AEN request cannot guarantee whether firmware has paused/resumed or not.

How about this, we prints out that the queues have been quiesced instead of "processing paused" which might make confusing between firmware just sets the CSTS.PS and the work scheduled has just quiesced the admin and io queues both.

Thanks for your kindly response, by the way.

Quoted 3 lines

In case the controller did pause processing, and we hit the unusual timeout race that does not see CSTS.PP, the new prints make it clear that's what happened.

maintainer
#10 lore raw

cover nvme: Improve processing paused support

In reply to Keith Busch

This looks sensible to me,

Minwoo, do you have any concerns or feedback? If not can I get your review tag?

#11 lore raw

cover nvme: Improve processing paused support

Quoted 4 lines · Sagi Grimberg

This looks sensible to me,

Minwoo, do you have any concerns or feedback? If not can I get your review tag?

Actually this series are looking good to me. But, I was just concerning that if there is better print statement or safe access method for CSTS.PP bit in the code.

But, the print message is not that critical so that I can make my review on this. I need to ask Keith about accessing CSTS.PP without seeing CC.EN can be valid in some cases, by the way :).

For this series:

Reviewed-by Minwoo Im <minwoo.im.dev at gmail.com>