Skip to contentSkip to conversation

blk-mq/nvme: improve nvme-pci reset handler

PATCH In review Ming Lei 1 participant 5 messages May 30, 2020 – Jun 11, 2020

Subsystems (2)
lore
Share & tools
lore

Ming Lei

author
#0 lore raw

v2 cover blk-mq/nvme: improve nvme-pci reset handler

Hi,

For nvme-pci, after controller is recovered, in-flight IOs are waited before updating nr hw queues. If new controller error happens during this period, nvme-pci driver deletes the controller and fails in-flight IO. This way is too violent, and not friendly from user viewpoint.

Add APIs for checking if queue is frozen, and replace nvme_wait_freeze in nvme-pci reset handler with checking if all ns queues are frozen & controller disabled. Then a fresh new reset can be scheduled for handling new controller error during waiting for in-flight IO completion.

So deleting controller & failing IOs can be avoided in this situation.

Without this patches, when fail io timeout injection is run, the controller can be removed very quickly. With this patch, no controller removing can be observed, and controller can recover to normal state after stopping to inject io timeout failure.

V2:
- give up after retrying enough times
- add comment on breaking because of shutdown

Ming Lei (3):
blk-mq: add API of blk_mq_queue_frozen
nvme: add nvme_frozen
nvme-pci: make nvme reset more reliable

 block/blk-mq.c           |  6 +++++
 drivers/nvme/host/core.c | 17 +++++++++++++-
 drivers/nvme/host/nvme.h |  3 +++
 drivers/nvme/host/pci.c  | 50 +++++++++++++++++++++++++++++++++-------
 include/linux/blk-mq.h   |  1 +
 5 files changed, 68 insertions(+), 9 deletions(-)
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Keith Busch <[email protected]>
Cc Max Gurtovoy <[email protected]>
Signature · 1 line
2.25.2
author
#1 lore raw

PATCH v2 1/3 blk-mq: add API of blk_mq_queue_frozen

blk_mq_freeze_queue_wait() isn't very flexible for some case, such as error recovery: when blk_mq_freeze_queue_wait is called in error recovery handler, new problem may be triggered on this controller, so in-flight IO may not complete when blk_mq_freeze_queue_wait() is called.

And error recovery is often run in single context, so dead lock may be triggered, because error recover handler can't move on.

Add one new API of blk_mq_queue_frozen(), error recovery handler may use this helper to query if the queue has been frozen completely. Meantime, the error recovery handler can check if there is hardware failure happened. If yes, error recovery handler can break from current handling, and run a fresh new recovery, so deadlock can be avoided.

This API will be used to improve error handling of nvme-pci's timeout handler.

Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Keith Busch <[email protected]>
Cc Max Gurtovoy <[email protected]>
---
 block/blk-mq.c         | 6 ++++++
 include/linux/blk-mq.h | 1 +
 2 files changed, 7 insertions(+)
block/blk-mq.c +6 -0
index a5ff8140e52c..dc4e73d7ad02 100644
@@ -148,6 +148,12 @@ void blk_mq_freeze_queue_wait(struct request_queue *q)
 }
 EXPORT_SYMBOL_GPL(blk_mq_freeze_queue_wait);
 
+bool blk_mq_queue_frozen(struct request_queue *q)
+{
+	return percpu_ref_is_zero(&q->q_usage_counter);
+}
+EXPORT_SYMBOL_GPL(blk_mq_queue_frozen);
+
 int blk_mq_freeze_queue_wait_timeout(struct request_queue *q,
 				     unsigned long timeout)
 {
include/linux/blk-mq.h +1 -0
index 43e38d21ca4a..1f867ad02c5c 100644
@@ -518,6 +518,7 @@ void blk_freeze_queue_start(struct request_queue *q);
 void blk_mq_freeze_queue_wait(struct request_queue *q);
 int blk_mq_freeze_queue_wait_timeout(struct request_queue *q,
 				     unsigned long timeout);
+bool blk_mq_queue_frozen(struct request_queue *q);
 
 int blk_mq_map_queues(struct blk_mq_queue_map *qmap);
 void blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, int nr_hw_queues);
Signature · 1 line
2.25.2
author
#2 lore raw

PATCH v2 2/3 nvme: add nvme_frozen

In reply to Ming Lei

Add one new API of nvme_frozen(), reset handler may use this helper to query if all ns queues have been frozen completely. Meantime, the reset handler can check if there is new hardware failure happened. If yes, reset handler can break from current handling, and schedule a fresh new recovery, so deadlock or deleting controller & fail all IOs can be avoided.

Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Keith Busch <[email protected]>
Cc Max Gurtovoy <[email protected]>
---
 drivers/nvme/host/core.c | 14 ++++++++++++++
 drivers/nvme/host/nvme.h |  1 +
 2 files changed, 15 insertions(+)
drivers/nvme/host/core.c +14 -0
index f3c037f5a9ba..469010607383 100644
@@ -4243,6 +4243,20 @@ void nvme_wait_freeze(struct nvme_ctrl *ctrl)
 }
 EXPORT_SYMBOL_GPL(nvme_wait_freeze);
 
+bool nvme_frozen(struct nvme_ctrl *ctrl)
+{
+	struct nvme_ns *ns;
+	int ret = 0;
+
+	down_read(&ctrl->namespaces_rwsem);
+	list_for_each_entry(ns, &ctrl->namespaces, list)
+		ret += !blk_mq_queue_frozen(ns->queue);
+	up_read(&ctrl->namespaces_rwsem);
+
+	return ret == 0;
+}
+EXPORT_SYMBOL_GPL(nvme_frozen);
+
 void nvme_start_freeze(struct nvme_ctrl *ctrl)
 {
 	struct nvme_ns *ns;
drivers/nvme/host/nvme.h +1 -0
index 2e04a36296d9..459e5952ff5f 100644
@@ -508,6 +508,7 @@ void nvme_unfreeze(struct nvme_ctrl *ctrl);
 void nvme_wait_freeze(struct nvme_ctrl *ctrl);
 void nvme_wait_freeze_timeout(struct nvme_ctrl *ctrl, long timeout);
 void nvme_start_freeze(struct nvme_ctrl *ctrl);
+bool nvme_frozen(struct nvme_ctrl *ctrl);
 
 #define NVME_QID_ANY -1
 struct request *nvme_alloc_request(struct request_queue *q,
Signature · 1 line
2.25.2
author
#3 lore raw

PATCH v2 3/3 nvme-pci: make nvme reset more reliable

In reply to Ming Lei

During waiting for in-flight IO completion in reset handler, timeout or controller failure still may happen, then the controller is deleted and all inflight IOs are failed. This way is too violent: 1) the timeout may be triggered exactly during nvme_wait_freeze() after controller is recovered by one occasional event; 2) the claimed retry times isn't respected.

Improve the reset handling by replacing nvme_wait_freeze with query & check controller. If all ns queues are frozen, the controller is reset successfully, otherwise check and see if the controller has been disabled. If yes, break from the current recovery and schedule a fresh new reset.

This way avoids to failing IO & removing controller unnecessarily.

Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Keith Busch <[email protected]>
Cc Max Gurtovoy <[email protected]>
---
 drivers/nvme/host/core.c |  3 ++-
 drivers/nvme/host/nvme.h |  2 ++
 drivers/nvme/host/pci.c  | 50 +++++++++++++++++++++++++++++++++-------
 3 files changed, 46 insertions(+), 9 deletions(-)
drivers/nvme/host/core.c +2 -1
index 469010607383..f4a00a85a47e 100644
@@ -45,9 +45,10 @@ static unsigned char shutdown_timeout = 5;
 module_param(shutdown_timeout, byte, 0644);
 MODULE_PARM_DESC(shutdown_timeout, "timeout in seconds for controller shutdown");
 
-static u8 nvme_max_retries = 5;
+u8 nvme_max_retries = 5;
 module_param_named(max_retries, nvme_max_retries, byte, 0644);
 MODULE_PARM_DESC(max_retries, "max number of retries a command may have");
+EXPORT_SYMBOL_GPL(nvme_max_retries);
 
 static unsigned long default_ps_max_latency_us = 100000;
 module_param(default_ps_max_latency_us, ulong, 0644);
drivers/nvme/host/nvme.h +2 -0
index 459e5952ff5f..ba4c84f16a0b 100644
@@ -25,6 +25,8 @@ extern unsigned int nvme_io_timeout;
 extern unsigned int admin_timeout;
 #define ADMIN_TIMEOUT	(admin_timeout * HZ)
 
+extern u8 nvme_max_retries;
+
 #define NVME_DEFAULT_KATO	5
 #define NVME_KATO_GRACE		10
 
drivers/nvme/host/pci.c +42 -8

109 lines not on this page · raw message

Signature · 1 line
2.25.2
author
#4 lore raw
In reply to Ming Lei
Quoted 35 lines · Ming Lei

Hi,

For nvme-pci, after controller is recovered, in-flight IOs are waited before updating nr hw queues. If new controller error happens during this period, nvme-pci driver deletes the controller and fails in-flight IO. This way is too violent, and not friendly from user viewpoint.

Add APIs for checking if queue is frozen, and replace nvme_wait_freeze in nvme-pci reset handler with checking if all ns queues are frozen & controller disabled. Then a fresh new reset can be scheduled for handling new controller error during waiting for in-flight IO completion.

So deleting controller & failing IOs can be avoided in this situation.

Without this patches, when fail io timeout injection is run, the controller can be removed very quickly. With this patch, no controller removing can be observed, and controller can recover to normal state after stopping to inject io timeout failure.

V2:
- give up after retrying enough times
- add comment on breaking because of shutdown

Ming Lei (3):
blk-mq: add API of blk_mq_queue_frozen
nvme: add nvme_frozen
nvme-pci: make nvme reset more reliable

 block/blk-mq.c           |  6 +++++
 drivers/nvme/host/core.c | 17 +++++++++++++-
 drivers/nvme/host/nvme.h |  3 +++
 drivers/nvme/host/pci.c  | 50 +++++++++++++++++++++++++++++++++-------
 include/linux/blk-mq.h   |  1 +
 5 files changed, 68 insertions(+), 9 deletions(-)
Quoted 4 lines
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Keith Busch <[email protected]>
Cc Max Gurtovoy <[email protected]>

Hello Guys,

Ping...

Thanks,
Ming