Skip to contentSkip to conversation

nvme-multipath: do not reset on unknown status

PATCH In review Keith Busch 5 participants 6 messages Feb 20, 2020 – Mar 10, 2020 NVMe

lore
Share & tools
lore

Keith Busch

author maintainer
#0 lore raw

PATCH 1/1 nvme-multipath: do not reset on unknown status

From: John Meneghini <[email protected]>

The nvme multipath error handling defaults to controller reset if the error is unknown. There are, however, no existing nvme status codes that indicate a reset should be used, and resetting causes unnecessary disruption to the rest of IO.

Change nvme's error handling to first check if failover should happen. If not, let the normal error handling take over rather than reset the controller.

---
 drivers/nvme/host/core.c      |  4 +---
 drivers/nvme/host/multipath.c | 21 +++++++++------------
 drivers/nvme/host/nvme.h      |  5 +++--
 3 files changed, 13 insertions(+), 17 deletions(-)
drivers/nvme/host/core.c +1 -3
index 84914223c537..0eef1ef8c659 100644
@@ -292,10 +292,8 @@ void nvme_complete_rq(struct request *req)
 
 	if (unlikely(status != BLK_STS_OK && nvme_req_needs_retry(req))) {
 		if ((req->cmd_flags & REQ_NVME_MPATH) &&
-		    blk_path_error(status)) {
-			nvme_failover_req(req);
+		    nvme_failover_req(req))
 			return;
-		}
 
 		if (!blk_queue_dying(req->q)) {
 			nvme_retry_req(req);
drivers/nvme/host/multipath.c +9 -12
index 797c18337d96..16df0baaeb40 100644
@@ -64,17 +64,12 @@ void nvme_set_disk_name(char *disk_name, struct nvme_ns *ns,
 	}
 }
 
-void nvme_failover_req(struct request *req)
+bool nvme_failover_req(struct request *req)
 {
 	struct nvme_ns *ns = req->q->queuedata;
 	u16 status = nvme_req(req)->status;
 	unsigned long flags;
 
-	spin_lock_irqsave(&ns->head->requeue_lock, flags);
-	blk_steal_bios(&ns->head->requeue_list, req);
-	spin_unlock_irqrestore(&ns->head->requeue_lock, flags);
-	blk_mq_end_request(req, 0);
-
 	switch (status & 0x7ff) {
 	case NVME_SC_ANA_TRANSITION:
 	case NVME_SC_ANA_INACCESSIBLE:
@@ -103,15 +98,17 @@ void nvme_failover_req(struct request *req)
 		nvme_mpath_clear_current_path(ns);
 		break;
 	default:
-		/*
-		 * Reset the controller for any non-ANA error as we don't know
-		 * what caused the error.
-		 */
-		nvme_reset_ctrl(ns->ctrl);
-		break;
+		/* This was a non-ANA error so follow the normal error path. */
+		return false;
 	}
 
+	spin_lock_irqsave(&ns->head->requeue_lock, flags);
+	blk_steal_bios(&ns->head->requeue_list, req);
+	spin_unlock_irqrestore(&ns->head->requeue_lock, flags);
+	blk_mq_end_request(req, 0);
+
 	kblockd_schedule_work(&ns->head->requeue_work);
+	return true;
 }
 
 void nvme_kick_requeue_lists(struct nvme_ctrl *ctrl)
drivers/nvme/host/nvme.h +3 -2
index 1024fec7914c..d800b9a51c2c 100644
@@ -550,7 +550,7 @@ void nvme_mpath_wait_freeze(struct nvme_subsystem *subsys);
 void nvme_mpath_start_freeze(struct nvme_subsystem *subsys);
 void nvme_set_disk_name(char *disk_name, struct nvme_ns *ns,
 			struct nvme_ctrl *ctrl, int *flags);
-void nvme_failover_req(struct request *req);
+bool nvme_failover_req(struct request *req);
 void nvme_kick_requeue_lists(struct nvme_ctrl *ctrl);
 int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl,struct nvme_ns_head *head);
 void nvme_mpath_add_disk(struct nvme_ns *ns, struct nvme_id_ns *id);
@@ -599,8 +599,9 @@ static inline void nvme_set_disk_name(char *disk_name, struct nvme_ns *ns,
 	sprintf(disk_name, "nvme%dn%d", ctrl->instance, ns->head->instance);
 }
 
-static inline void nvme_failover_req(struct request *req)
+static inline bool nvme_failover_req(struct request *req)
 {
+	return false;
 }
 static inline void nvme_kick_requeue_lists(struct nvme_ctrl *ctrl)
 {
Signature · 7 lines
2.21.0


_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#1 lore raw
Quoted 7 lines · Keith Busch

From: John Meneghini <[email protected]>

The nvme multipath error handling defaults to controller reset if the error is unknown. There are, however, no existing nvme status codes that indicate a reset should be used, and resetting causes unnecessary disruption to the rest of IO.

Quoted 7 lines

Change nvme's error handling to first check if failover should happen. If not, let the normal error handling take over rather than reset the controller.

This pretty much looks like my edited version of Hanne's patch with John's changelog and your edits..

drivers/nvme/host/core.c void nvme_complete_rq(struct request *req) · line 295

Quoted 5 lines
drivers/nvme/host/core.c +2 -2
 		if ((req->cmd_flags & REQ_NVME_MPATH) &&
+		    nvme_failover_req(req))
-		    blk_path_error(status)) {
-			nvme_failover_req(req);
+		    nvme_failover_req(req))

This conditional could now fit onto a single line.

Otherwise this looks fine to me.

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#2 lore raw

On 2/20/20, 11:47 AM, "Christoph Hellwig" <[email protected]> wrote:

This pretty much looks like my edited version of Hanne's patch with John's changelog and your edits..

That's because it is!

Review

Quoted 5 lines
              if ((req->cmd_flags & REQ_NVME_MPATH) &&
(quoted diff) +2 -2
+                 nvme_failover_req(req))
-                 blk_path_error(status)) {
-                     nvme_failover_req(req);
+                 nvme_failover_req(req))

This conditional could now fit onto a single line.

Otherwise this looks fine to me.

Here's another copy of the version I sent to Kieth, with the new commit message.

From: John Meneghini <[email protected]>

The nvme multipath error handling defaults to controller reset if the error is unknown. There are, however, no existing nvme status codes that indicate a reset should be used, and resetting causes unnecessary disruption to the rest of IO.

Change nvme's error handling to first check if failover should happen. If not, let the normal error handling take over rather than reset the controller.

---
 drivers/nvme/host/core.c      |  4 +---
 drivers/nvme/host/multipath.c | 22 ++++++++++------------
 drivers/nvme/host/nvme.h      |  5 +++--
 3 files changed, 14 insertions(+), 17 deletions(-)
drivers/nvme/host/core.c +1 -3
index 84914223c537..e1696fcf0f8b 100644
@@ -291,9 +291,7 @@ void nvme_complete_rq(struct request *req)
                nvme_req(req)->ctrl->comp_seen = true;
 
        if (unlikely(status != BLK_STS_OK && nvme_req_needs_retry(req))) {
-               if ((req->cmd_flags & REQ_NVME_MPATH) &&
-                   blk_path_error(status)) {
-                       nvme_failover_req(req);
+               if ((req->cmd_flags & REQ_NVME_MPATH) && nvme_failover_req(req)) {
                        return;
                }
 
drivers/nvme/host/multipath.c +4 -12
index 797c18337d96..8a1697f7986b 100644
@@ -64,18 +64,12 @@ void nvme_set_disk_name(char *disk_name, struct nvme_ns *ns,
        }
 }
 
-void nvme_failover_req(struct request *req)
+bool nvme_failover_req(struct request *req)
 {
        struct nvme_ns *ns = req->q->queuedata;
-       u16 status = nvme_req(req)->status;
        unsigned long flags;
 
-       spin_lock_irqsave(&ns->head->requeue_lock, flags);
-       blk_steal_bios(&ns->head->requeue_list, req);
-       spin_unlock_irqrestore(&ns->head->requeue_lock, flags);
-       blk_mq_end_request(req, 0);
-
-       switch (status & 0x7ff) {
+       switch (nvme_req(req)->status & 0x7ff) {
        case NVME_SC_ANA_TRANSITION:
        case NVME_SC_ANA_INACCESSIBLE:
        case NVME_SC_ANA_PERSISTENT_LOSS:
@@ -104,14 +98,18 @@ void nvme_failover_req(struct request *req)
                break;
        default:
                /*
-                * Reset the controller for any non-ANA error as we don't know
-                * what caused the error.
+                * This was a non-ANA error so follow the normal error path.
                 */
-               nvme_reset_ctrl(ns->ctrl);
-               break;
+               return false;
        }
 

void nvme_kick_requeue_lists(struct nvme_ctrl *ctrl)

drivers/nvme/host/nvme.h +3 -2
index 1024fec7914c..d800b9a51c2c 100644
@@ -550,7 +550,7 @@ void nvme_mpath_wait_freeze(struct nvme_subsystem *subsys);
 void nvme_mpath_start_freeze(struct nvme_subsystem *subsys);
 void nvme_set_disk_name(char *disk_name, struct nvme_ns *ns,
                        struct nvme_ctrl *ctrl, int *flags);
-void nvme_failover_req(struct request *req);
+bool nvme_failover_req(struct request *req);
 void nvme_kick_requeue_lists(struct nvme_ctrl *ctrl);
 int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl,struct nvme_ns_head *head);
 void nvme_mpath_add_disk(struct nvme_ns *ns, struct nvme_id_ns *id);
@@ -599,8 +599,9 @@ static inline void nvme_set_disk_name(char *disk_name, struct nvme_ns *ns,
        sprintf(disk_name, "nvme%dn%d", ctrl->instance, ns->head->instance);
 }
 
-static inline void nvme_failover_req(struct request *req)
+static inline bool nvme_failover_req(struct request *req)
 {
+       return false;
 }
 static inline void nvme_kick_requeue_lists(struct nvme_ctrl *ctrl)
 {
Signature · 6 lines
2.21.0

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Reviewed-by on v1 1/1
#3 lore raw
In reply to Keith Busch
Quoted 98 lines · Keith Busch

From: John Meneghini <[email protected]>

The nvme multipath error handling defaults to controller reset if the error is unknown. There are, however, no existing nvme status codes that indicate a reset should be used, and resetting causes unnecessary disruption to the rest of IO.

Change nvme's error handling to first check if failover should happen. If not, let the normal error handling take over rather than reset the controller.

---
  drivers/nvme/host/core.c      |  4 +---
  drivers/nvme/host/multipath.c | 21 +++++++++------------
  drivers/nvme/host/nvme.h      |  5 +++--
  3 files changed, 13 insertions(+), 17 deletions(-)
drivers/nvme/host/core.c +1 -3
index 84914223c537..0eef1ef8c659 100644
@@ -292,10 +292,8 @@ void nvme_complete_rq(struct request *req)
 
  	if (unlikely(status != BLK_STS_OK && nvme_req_needs_retry(req))) {
  		if ((req->cmd_flags & REQ_NVME_MPATH) &&
-		    blk_path_error(status)) {
-			nvme_failover_req(req);
+		    nvme_failover_req(req))
  			return;
-		}
 
  		if (!blk_queue_dying(req->q)) {
  			nvme_retry_req(req);
drivers/nvme/host/multipath.c +9 -12
index 797c18337d96..16df0baaeb40 100644
@@ -64,17 +64,12 @@ void nvme_set_disk_name(char *disk_name, struct nvme_ns *ns,
  	}
  }
 
-void nvme_failover_req(struct request *req)
+bool nvme_failover_req(struct request *req)
  {
  	struct nvme_ns *ns = req->q->queuedata;
  	u16 status = nvme_req(req)->status;
  	unsigned long flags;
 
-	spin_lock_irqsave(&ns->head->requeue_lock, flags);
-	blk_steal_bios(&ns->head->requeue_list, req);
-	spin_unlock_irqrestore(&ns->head->requeue_lock, flags);
-	blk_mq_end_request(req, 0);
-
  	switch (status & 0x7ff) {
  	case NVME_SC_ANA_TRANSITION:
  	case NVME_SC_ANA_INACCESSIBLE:
@@ -103,15 +98,17 @@ void nvme_failover_req(struct request *req)
  		nvme_mpath_clear_current_path(ns);
  		break;
  	default:
-		/*
-		 * Reset the controller for any non-ANA error as we don't know
-		 * what caused the error.
-		 */
-		nvme_reset_ctrl(ns->ctrl);
-		break;
+		/* This was a non-ANA error so follow the normal error path. */
+		return false;
  	}
 
+	spin_lock_irqsave(&ns->head->requeue_lock, flags);
+	blk_steal_bios(&ns->head->requeue_list, req);
+	spin_unlock_irqrestore(&ns->head->requeue_lock, flags);
+	blk_mq_end_request(req, 0);
+
  	kblockd_schedule_work(&ns->head->requeue_work);
+	return true;
  }
 
  void nvme_kick_requeue_lists(struct nvme_ctrl *ctrl)
drivers/nvme/host/nvme.h +1 -1
index 1024fec7914c..d800b9a51c2c 100644
@@ -550,7 +550,7 @@ void nvme_mpath_wait_freeze(struct nvme_subsystem *subsys);
  void nvme_mpath_start_freeze(struct nvme_subsystem *subsys);
  void nvme_set_disk_name(char *disk_name, struct nvme_ns *ns,
  			struct nvme_ctrl *ctrl, int *flags);
-void nvme_failover_req(struct request *req);
+bool nvme_failover_req(struct request *req);
  void nvme_kick_requeue_lists(struct nvme_ctrl *ctrl);
  int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl,struct nvme_ns_head *head);
  void nvme_mpath_add_disk(struct nvme_ns *ns, struct nvme_id_ns *id);

drivers/nvme/host/nvme.h static inline void nvme_set_disk_name(char *disk_name, struct nvme_ns *ns, · line 607

Quoted 10 lines
drivers/nvme/host/nvme.h +2 -1
@@ -599,8 +599,9 @@ static inline void nvme_set_disk_name(char *disk_name, struct nvme_ns *ns,
  	sprintf(disk_name, "nvme%dn%d", ctrl->instance, ns->head->instance);
  }
 
-static inline void nvme_failover_req(struct request *req)
+static inline bool nvme_failover_req(struct request *req)
  {
+	return false;
  }
  static inline void nvme_kick_requeue_lists(struct nvme_ctrl *ctrl)
  {

Cheers,

Hannes

Signature · 9 lines
Dr. Hannes Reinecke            Teamlead Storage & Networking
[email protected]                               +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#4 lore raw
In reply to Keith Busch
Quoted 11 lines · Keith Busch

From: John Meneghini <[email protected]>

The nvme multipath error handling defaults to controller reset if the error is unknown. There are, however, no existing nvme status codes that indicate a reset should be used, and resetting causes unnecessary disruption to the rest of IO.

Change nvme's error handling to first check if failover should happen. If not, let the normal error handling take over rather than reset the controller.

I've applied this to 5.7. Many developers had input to this patch. Please let me know if attribution should be adjusted, and I will fix ASAP.

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#5 lore raw

I think the answer to this question needs to come from Hannes and Christoph.

This patch is a version of a change Hannes and I worked on privately for some time.

I reported this problem on this list months ago, when I proposed an ACRE patch, and we've had a BZ open with SUSE for this problem. After some time working on this BZ Hannes took one of my suggestions and pushed a proposed patch out.

Christoph then significantly modified the patch, and I took both Christoph's changes and Hannes' changes and pushed up the final patch.

/John

P.S. that’s for working on this Keith, I realize my patch did not apply cleanly.

On 3/10/20, 5:06 PM, "Keith Busch" <[email protected]> wrote:

On Thu, Feb 20, 2020 at 7:53 AM Keith Busch <[email protected]> wrote:
>
> From: John Meneghini <[email protected]>
>
> The nvme multipath error handling defaults to controller reset if the > error is unknown. There are, however, no existing nvme status codes that > indicate a reset should be used, and resetting causes unnecessary > disruption to the rest of IO.
>
> Change nvme's error handling to first check if failover should happen. > If not, let the normal error handling take over rather than reset the > controller.

I've applied this to 5.7. Many developers had input to this patch. Please let me know if attribution should be adjusted, and I will fix ASAP.

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme