Skip to contentSkip to conversation

nvme-multipath: fixes for non-optimized paths

PATCH Applied Hannes Reinecke 3 participants 12 messages Jul 27, 2020 – Jul 28, 2020 NVMe

lore
Share & tools
lore
#0 lore raw

cover nvme-multipath: fixes for non-optimized paths

Hi all,

one of our partners informed us that the round-robin path selector didn't do the right thing when only one optimized path (or, indeed, none at all) are present.
These patches will fixup nvme_round_robin() to handle the case when less than two optimized (but several non-optimized) paths are present.

These pathes are a rework of the earlier patches send by Martin Wilck.

As usual, comments and reviews are welcome.

Hannes Reinecke (1):
nvme-multipath: do not fall back to __nvme_find_path() for
non-optimized paths

Martin Wilck (1):
nvme-multipath: fix logic for non-optimized paths

 drivers/nvme/host/multipath.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
Signature · 7 lines
2.16.4


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#1 lore raw

PATCH 1/2 nvme-multipath: fix logic for non-optimized paths

From: Martin Wilck <[email protected]>

Handle the special case where we have exactly one optimized path, which we should keep using in this case.

Signed off-by: Martin Wilck <[email protected]>

---
 drivers/nvme/host/multipath.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/nvme/host/multipath.c +6 -0
index 66509472fe06..fe8f7f123fac 100644
@@ -246,6 +246,12 @@ static struct nvme_ns *nvme_round_robin_path(struct nvme_ns_head *head,
 			fallback = ns;
 	}
 
+	/* No optimized path found, re-check the current path */
+	if (!nvme_path_is_disabled(old) &&
+	    old->ana_state == NVME_ANA_OPTIMIZED) {
+		found = old;
+		goto out;
+	}
 	if (!fallback)
 		return NULL;
 	found = fallback;
Signature · 7 lines
2.16.4


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#3 lore raw

PATCH 2/2 nvme-multipath: do not fall back to __nvme_find_path() for non-optimized paths

In reply to Hannes Reinecke

When nvme_round_robin_path() finds a valid namespace we should be using it; falling back to __nvme_find_path() for non-optimized paths will cause the result from nvme_round_robin_path() to be ignored for non-optimized paths.

---
 drivers/nvme/host/multipath.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/nvme/host/multipath.c +4 -1
index fe8f7f123fac..97d3c19b5684 100644
@@ -272,8 +272,11 @@ inline struct nvme_ns *nvme_find_path(struct nvme_ns_head *head)
 	struct nvme_ns *ns;
 
 	ns = srcu_dereference(head->current_path[node], &head->srcu);
-	if (READ_ONCE(head->subsys->iopolicy) == NVME_IOPOLICY_RR && ns)
+	if (READ_ONCE(head->subsys->iopolicy) == NVME_IOPOLICY_RR && ns) {
 		ns = nvme_round_robin_path(head, node, ns);
+		if (ns)
+			return ns;
+	}
 	if (unlikely(!ns || !nvme_path_is_optimized(ns)))
 		ns = __nvme_find_path(head, node);
 	return ns;
Signature · 7 lines
2.16.4


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
maintainer
#4 lore raw

Re: [PATCH 2/2] nvme-multipath: do not fall back to __nvme_find_path() for non-optimized paths

Quoted 16 lines · Hannes Reinecke

When nvme_round_robin_path() finds a valid namespace we should be using it; falling back to __nvme_find_path() for non-optimized paths will cause the result from nvme_round_robin_path() to be ignored for non-optimized paths.

---
  drivers/nvme/host/multipath.c | 5 ++++-
  1 file changed, 4 insertions(+), 1 deletion(-)
drivers/nvme/host/multipath.c +0 -0
index fe8f7f123fac..97d3c19b5684 100644
@@ -272,8 +272,11 @@ inline struct nvme_ns *nvme_find_path(struct nvme_ns_head *head)
  	struct nvme_ns *ns;

drivers/nvme/host/multipath.c inline struct nvme_ns *nvme_find_path(struct nvme_ns_head *head) · line 281

Quoted 10 lines
drivers/nvme/host/multipath.c +4 -1
 
  	ns = srcu_dereference(head->current_path[node], &head->srcu);
-	if (READ_ONCE(head->subsys->iopolicy) == NVME_IOPOLICY_RR && ns)
+	if (READ_ONCE(head->subsys->iopolicy) == NVME_IOPOLICY_RR && ns) {
  		ns = nvme_round_robin_path(head, node, ns);
+		if (ns)
+			return ns;
+	}
  	if (unlikely(!ns || !nvme_path_is_optimized(ns)))
  		ns = __nvme_find_path(head, node);

Maybe instead of the early return put the following condition on else?

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#5 lore raw

Re: [PATCH 2/2] nvme-multipath: do not fall back to __nvme_find_path() for non-optimized paths

Quoted 23 lines · Sagi Grimberg
Quoted 20 lines · Hannes Reinecke

When nvme_round_robin_path() finds a valid namespace we should be using it; falling back to __nvme_find_path() for non-optimized paths will cause the result from nvme_round_robin_path() to be ignored for non-optimized paths.

---
  drivers/nvme/host/multipath.c | 5 ++++-

  1 file changed, 4 insertions(+), 1 deletion(-)
drivers/nvme/host/multipath.c +1 -1
index fe8f7f123fac..97d3c19b5684 100644
@@ -272,8 +272,11 @@ inline struct nvme_ns *nvme_find_path(struct 
 nvme_ns_head *head)
       struct nvme_ns *ns;
       ns = srcu_dereference(head->current_path[node], &head->srcu);
-    if (READ_ONCE(head->subsys->iopolicy) == NVME_IOPOLICY_RR && ns)
+    if (READ_ONCE(head->subsys->iopolicy) == NVME_IOPOLICY_RR && ns) {
Quoted 8 lines

drivers/nvme/host/multipath.c inline struct nvme_ns *nvme_find_path(struct nvme_ns_head *head) · line 281

Quoted 6 lines
drivers/nvme/host/multipath.c +3 -0
           ns = nvme_round_robin_path(head, node, ns);
+        if (ns)
+            return ns;
+    }
       if (unlikely(!ns || !nvme_path_is_optimized(ns)))
           ns = __nvme_find_path(head, node);

Maybe instead of the early return put the following condition on else?

That depends on whether we want to have a fallback to __nvme_find_path() or not. With your suggestion we would lose that; I'd rather keep it.

Cheers,

Hannes

Signature · 9 lines
Dr. Hannes Reinecke            Teamlead Storage & Networking
[email protected]                               +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
maintainer Reviewed-by
#6 lore raw

Re: [PATCH 2/2] nvme-multipath: do not fall back to __nvme_find_path() for non-optimized paths

Quoted 30 lines
Quoted 21 lines
Quoted 21 lines

When nvme_round_robin_path() finds a valid namespace we should be using it; falling back to __nvme_find_path() for non-optimized paths will cause the result from nvme_round_robin_path() to be ignored for non-optimized paths.

---
  drivers/nvme/host/multipath.c | 5 ++++-

  1 file changed, 4 insertions(+), 1 deletion(-)
drivers/nvme/host/multipath.c +1 -1
index fe8f7f123fac..97d3c19b5684 100644
@@ -272,8 +272,11 @@ inline struct nvme_ns *nvme_find_path(struct 
 nvme_ns_head *head)
       struct nvme_ns *ns;
       ns = srcu_dereference(head->current_path[node], &head->srcu);
-    if (READ_ONCE(head->subsys->iopolicy) == NVME_IOPOLICY_RR && ns)
+    if (READ_ONCE(head->subsys->iopolicy) == NVME_IOPOLICY_RR && ns) {
Quoted 8 lines

drivers/nvme/host/multipath.c inline struct nvme_ns *nvme_find_path(struct nvme_ns_head *head) · line 281

Quoted 6 lines
drivers/nvme/host/multipath.c +3 -0
           ns = nvme_round_robin_path(head, node, ns);
+        if (ns)
+            return ns;
+    }
       if (unlikely(!ns || !nvme_path_is_optimized(ns)))
           ns = __nvme_find_path(head, node);

Maybe instead of the early return put the following condition on else?

Quoted 1 line

That depends on whether we want to have a fallback to __nvme_find_path() or not. With your suggestion we would lose that; I'd rather keep it.

OK...

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer Reviewed-by Sagi Grimberg
#7 lore raw

Re: [PATCH 2/2] nvme-multipath: do not fall back to __nvme_find_path() for non-optimized paths

In reply to Hannes Reinecke
Quoted 4 lines · Hannes Reinecke
Quoted 1 line

Maybe instead of the early return put the following condition on else?

That depends on whether we want to have a fallback to __nvme_find_path() or not. With your suggestion we would lose that; I'd rather keep it.

Why would we want to fall back? nvme_round_robin_path only returns NULL in case there is a single disabled path, in which case __nvme_find_path won't find anything either. I'd go for something like this:

---
From 4f578364a3d15898c7d715315a3371b2f71db416 Mon Sep 17 00:00:00 2001
From: Hannes Reinecke <[email protected]>
Date: Mon, 27 Jul 2020 18:08:03 +0200
Subject: nvme-multipath: do not fall back to __nvme_find_path() for
non-optimized paths

When nvme_round_robin_path() finds a valid namespace we should be using it; falling back to __nvme_find_path() for non-optimized paths will cause the result from nvme_round_robin_path() to be ignored for non-optimized paths.

---
 drivers/nvme/host/multipath.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
drivers/nvme/host/multipath.c +7 -4
index 93c70e1591de8f..3ded54d2c9c6ad 100644
@@ -281,10 +281,13 @@ inline struct nvme_ns *nvme_find_path(struct nvme_ns_head *head)
 	struct nvme_ns *ns;
 
 	ns = srcu_dereference(head->current_path[node], &head->srcu);
-	if (READ_ONCE(head->subsys->iopolicy) == NVME_IOPOLICY_RR && ns)
-		ns = nvme_round_robin_path(head, node, ns);
-	if (unlikely(!ns || !nvme_path_is_optimized(ns)))
-		ns = __nvme_find_path(head, node);
+	if (unlikely(!ns))
+		return __nvme_find_path(head, node);
+
+	if (READ_ONCE(head->subsys->iopolicy) == NVME_IOPOLICY_RR)
+		return nvme_round_robin_path(head, node, ns);
+	if (unlikely(!nvme_path_is_optimized(ns)))
+		return __nvme_find_path(head, node);
 	return ns;
 }
 
Signature · 7 lines
2.27.0


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#8 lore raw

Re: [PATCH 2/2] nvme-multipath: do not fall back to __nvme_find_path() for non-optimized paths

Quoted 43 lines · Christoph Hellwig
Quoted 4 lines · Hannes Reinecke
Quoted 1 line

Maybe instead of the early return put the following condition on else?

That depends on whether we want to have a fallback to __nvme_find_path() or not. With your suggestion we would lose that; I'd rather keep it.

Why would we want to fall back? nvme_round_robin_path only returns NULL in case there is a single disabled path, in which case __nvme_find_path won't find anything either. I'd go for something like this:

---
From 4f578364a3d15898c7d715315a3371b2f71db416 Mon Sep 17 00:00:00 2001
From: Hannes Reinecke <[email protected]>
Date: Mon, 27 Jul 2020 18:08:03 +0200
Subject: nvme-multipath: do not fall back to __nvme_find_path() for
non-optimized paths

When nvme_round_robin_path() finds a valid namespace we should be using it; falling back to __nvme_find_path() for non-optimized paths will cause the result from nvme_round_robin_path() to be ignored for non-optimized paths.

---
 drivers/nvme/host/multipath.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
drivers/nvme/host/multipath.c +0 -4
index 93c70e1591de8f..3ded54d2c9c6ad 100644
@@ -281,10 +281,13 @@ inline struct nvme_ns *nvme_find_path(struct nvme_ns_head *head)
 	struct nvme_ns *ns;
 
 	ns = srcu_dereference(head->current_path[node], &head->srcu);
-	if (READ_ONCE(head->subsys->iopolicy) == NVME_IOPOLICY_RR && ns)
-		ns = nvme_round_robin_path(head, node, ns);
-	if (unlikely(!ns || !nvme_path_is_optimized(ns)))
-		ns = __nvme_find_path(head, node);

drivers/nvme/host/multipath.c inline struct nvme_ns *nvme_find_path(struct nvme_ns_head *head) · line 293

Quoted 10 lines
drivers/nvme/host/multipath.c +7 -0
+	if (unlikely(!ns))
+		return __nvme_find_path(head, node);
+
+	if (READ_ONCE(head->subsys->iopolicy) == NVME_IOPOLICY_RR)
+		return nvme_round_robin_path(head, node, ns);
+	if (unlikely(!nvme_path_is_optimized(ns)))
+		return __nvme_find_path(head, node);
 	return ns;
 }
 

Yes, that would work, too.

Should I resend?

Cheers,

Hannes

Signature · 9 lines
Dr. Hannes Reinecke		           Kernel Storage Architect
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#9 lore raw

Re: [PATCH 2/2] nvme-multipath: do not fall back to __nvme_find_path() for non-optimized paths

Quoted 57 lines · Hannes Reinecke
Quoted 43 lines · Christoph Hellwig
Quoted 4 lines · Hannes Reinecke
Quoted 1 line

Maybe instead of the early return put the following condition on else?

That depends on whether we want to have a fallback to __nvme_find_path() or not. With your suggestion we would lose that; I'd rather keep it.

Why would we want to fall back? nvme_round_robin_path only returns NULL in case there is a single disabled path, in which case __nvme_find_path won't find anything either. I'd go for something like this:

---
From 4f578364a3d15898c7d715315a3371b2f71db416 Mon Sep 17 00:00:00 2001
From: Hannes Reinecke <[email protected]>
Date: Mon, 27 Jul 2020 18:08:03 +0200
Subject: nvme-multipath: do not fall back to __nvme_find_path() for
non-optimized paths

When nvme_round_robin_path() finds a valid namespace we should be using it; falling back to __nvme_find_path() for non-optimized paths will cause the result from nvme_round_robin_path() to be ignored for non-optimized paths.

---
 drivers/nvme/host/multipath.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
drivers/nvme/host/multipath.c +0 -4
index 93c70e1591de8f..3ded54d2c9c6ad 100644
@@ -281,10 +281,13 @@ inline struct nvme_ns *nvme_find_path(struct nvme_ns_head *head)
 	struct nvme_ns *ns;
 
 	ns = srcu_dereference(head->current_path[node], &head->srcu);
-	if (READ_ONCE(head->subsys->iopolicy) == NVME_IOPOLICY_RR && ns)
-		ns = nvme_round_robin_path(head, node, ns);
-	if (unlikely(!ns || !nvme_path_is_optimized(ns)))
-		ns = __nvme_find_path(head, node);

drivers/nvme/host/multipath.c inline struct nvme_ns *nvme_find_path(struct nvme_ns_head *head) · line 293

Quoted 10 lines
drivers/nvme/host/multipath.c +7 -0
+	if (unlikely(!ns))
+		return __nvme_find_path(head, node);
+
+	if (READ_ONCE(head->subsys->iopolicy) == NVME_IOPOLICY_RR)
+		return nvme_round_robin_path(head, node, ns);
+	if (unlikely(!nvme_path_is_optimized(ns)))
+		return __nvme_find_path(head, node);
 	return ns;
 }
 

Yes, that would work, too.

Quoted 1 line

Should I resend?

No need, I can apply it with the modification. But if you have a
RR test setup it would be great to run it through that again, just in case.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#10 lore raw

Re: [PATCH 2/2] nvme-multipath: do not fall back to __nvme_find_path() for non-optimized paths

Quoted 58 lines · Christoph Hellwig
Quoted 57 lines · Hannes Reinecke
Quoted 43 lines · Christoph Hellwig
Quoted 4 lines · Hannes Reinecke
Quoted 1 line

Maybe instead of the early return put the following condition on else?

That depends on whether we want to have a fallback to __nvme_find_path() or not. With your suggestion we would lose that; I'd rather keep it.

Why would we want to fall back? nvme_round_robin_path only returns NULL in case there is a single disabled path, in which case __nvme_find_path won't find anything either. I'd go for something like this:

---
From 4f578364a3d15898c7d715315a3371b2f71db416 Mon Sep 17 00:00:00 2001
From: Hannes Reinecke <[email protected]>
Date: Mon, 27 Jul 2020 18:08:03 +0200
Subject: nvme-multipath: do not fall back to __nvme_find_path() for
non-optimized paths

When nvme_round_robin_path() finds a valid namespace we should be using it; falling back to __nvme_find_path() for non-optimized paths will cause the result from nvme_round_robin_path() to be ignored for non-optimized paths.

---
 drivers/nvme/host/multipath.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
drivers/nvme/host/multipath.c +0 -4
index 93c70e1591de8f..3ded54d2c9c6ad 100644
@@ -281,10 +281,13 @@ inline struct nvme_ns *nvme_find_path(struct nvme_ns_head *head)
 	struct nvme_ns *ns;
 
 	ns = srcu_dereference(head->current_path[node], &head->srcu);
-	if (READ_ONCE(head->subsys->iopolicy) == NVME_IOPOLICY_RR && ns)
-		ns = nvme_round_robin_path(head, node, ns);
-	if (unlikely(!ns || !nvme_path_is_optimized(ns)))
-		ns = __nvme_find_path(head, node);

drivers/nvme/host/multipath.c inline struct nvme_ns *nvme_find_path(struct nvme_ns_head *head) · line 293

Quoted 10 lines
drivers/nvme/host/multipath.c +7 -0
+	if (unlikely(!ns))
+		return __nvme_find_path(head, node);
+
+	if (READ_ONCE(head->subsys->iopolicy) == NVME_IOPOLICY_RR)
+		return nvme_round_robin_path(head, node, ns);
+	if (unlikely(!nvme_path_is_optimized(ns)))
+		return __nvme_find_path(head, node);
 	return ns;
 }
 

Yes, that would work, too.

Quoted 6 lines
Quoted 1 line

Should I resend?

No need, I can apply it with the modification. But if you have a
RR test setup it would be great to run it through that again, just in case.

I'm at it.

Cheers,

Hannes

Signature · 9 lines
Dr. Hannes Reinecke		           Kernel Storage Architect
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme