Skip to contentSkip to conversation

nvme: allow to re-attach namespaces after all paths are down

PATCH Applied Hannes Reinecke 4 participants 10 messages May 17, 2021 – May 25, 2021 NVMe

v4 of 6 Applied Latest is v6 →

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 nvme: allow to re-attach namespaces after all paths are down

We should only remove the ns head from the list of heads per subsystem if the reference count drops to zero. Removing it at the start of nvme_ns_remove() will prevent us from reattaching the namespace to the correct ns head once a path becomes available again.

Changes to v3:
- Simplify if() clause to detect duplicate namespaces Changes to v2:
- Drop memcpy() statement
Changes to v1:
- Always check NSIDs after reattach

---
 drivers/nvme/host/core.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
drivers/nvme/host/core.c +8 -3
index 522c9b229f80..a60144a4054a 100644
@@ -546,6 +546,9 @@ static void nvme_free_ns_head(struct kref *ref)
 	struct nvme_ns_head *head =
 		container_of(ref, struct nvme_ns_head, ref);
 
+	mutex_lock(&head->subsys->lock);
+	list_del_init(&head->entry);
+	mutex_unlock(&head->subsys->lock);
 	nvme_mpath_remove_disk(head);
 	ida_simple_remove(&head->subsys->ns_ida, head->instance);
 	cleanup_srcu_struct(&head->srcu);
@@ -3605,7 +3608,11 @@ static int nvme_init_ns_head(struct nvme_ns *ns, unsigned nsid,
 		head->shared = is_shared;
 	} else {
 		ret = -EINVAL;
-		if (!is_shared || !head->shared) {
+		/*
+		 * If multipath is enabled we might hit an ns head with no
+		 * paths, but that doesn't indicate it's a shared namespace.
+		 */
+		if (!list_empty(&head->list) && (!is_shared || !head->shared)) {
 			dev_err(ctrl->device,
 				"Duplicate unshared namespace %d\n", nsid);
 			goto out_put_ns_head;
@@ -3764,8 +3771,6 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 
 	mutex_lock(&ns->ctrl->subsys->lock);
 	list_del_rcu(&ns->siblings);
-	if (list_empty(&ns->head->list))
-		list_del_init(&ns->head->entry);
 	mutex_unlock(&ns->ctrl->subsys->lock);
 
 	synchronize_rcu(); /* guarantee not available in head->list */
Signature · 7 lines
2.29.2


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
maintainer Reviewed-by on v4 1/1
#1 lore raw
Quoted 5 lines · Hannes Reinecke

We should only remove the ns head from the list of heads per subsystem if the reference count drops to zero. Removing it at the start of nvme_ns_remove() will prevent us from reattaching the namespace to the correct ns head once a path becomes available again.

This looks fine to me

There are potential corner cases associated with namespace management that can cause problems here, but it would certainly be a user error that needs manual intervention anyway. I suspect those scenarios are very uncommon too, so I think we can deal with that later if it becomes a problem.

Quoted 34 lines
---
 drivers/nvme/host/core.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
drivers/nvme/host/core.c +8 -1
index 522c9b229f80..a60144a4054a 100644
@@ -546,6 +546,9 @@ static void nvme_free_ns_head(struct kref *ref)
 	struct nvme_ns_head *head =
 		container_of(ref, struct nvme_ns_head, ref);
 
+	mutex_lock(&head->subsys->lock);
+	list_del_init(&head->entry);
+	mutex_unlock(&head->subsys->lock);
 	nvme_mpath_remove_disk(head);
 	ida_simple_remove(&head->subsys->ns_ida, head->instance);
 	cleanup_srcu_struct(&head->srcu);
@@ -3605,7 +3608,11 @@ static int nvme_init_ns_head(struct nvme_ns *ns, unsigned nsid,
 		head->shared = is_shared;
 	} else {
 		ret = -EINVAL;
-		if (!is_shared || !head->shared) {
+		/*
+		 * If multipath is enabled we might hit an ns head with no
+		 * paths, but that doesn't indicate it's a shared namespace.
+		 */
+		if (!list_empty(&head->list) && (!is_shared || !head->shared)) {
 			dev_err(ctrl->device,
 				"Duplicate unshared namespace %d\n", nsid);
 			goto out_put_ns_head;

drivers/nvme/host/core.c static void nvme_ns_remove(struct nvme_ns *ns) · line 3776

Quoted 8 lines
drivers/nvme/host/core.c +0 -2
@@ -3764,8 +3771,6 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 
 	mutex_lock(&ns->ctrl->subsys->lock);
 	list_del_rcu(&ns->siblings);
-	if (list_empty(&ns->head->list))
-		list_del_init(&ns->head->entry);
 	mutex_unlock(&ns->ctrl->subsys->lock);
 
 	synchronize_rcu(); /* guarantee not available in head->list */

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#2 lore raw
Quoted 9 lines · Keith Busch
Quoted 5 lines · Hannes Reinecke

We should only remove the ns head from the list of heads per subsystem if the reference count drops to zero. Removing it at the start of nvme_ns_remove() will prevent us from reattaching the namespace to the correct ns head once a path becomes available again.

This looks fine to me

Quoted 8 lines

There are potential corner cases associated with namespace management that can cause problems here, but it would certainly be a user error that needs manual intervention anyway. I suspect those scenarios are very uncommon too, so I think we can deal with that later if it becomes a problem.

How so?
We're keeping a copy of the NS IDs around, so after reconnect the namespace would have to provide same set of NS IDs.

If the namespace is different yet providing the same NS IDs we'll have a problem even with the current code, as we're using the ND ID to identify multipathed namespaces.

Or do you have a different scenario in mind?

Cheers,

Hannes

Signature · 9 lines
Dr. Hannes Reinecke		        Kernel Storage Architect
[email protected]			               +49 911 74053 688
SUSE Software Solutions Germany GmbH, 90409 Nürnberg
GF: F. Imendörffer, HRB 36809 (AG Nürnberg)

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
maintainer
#3 lore raw
Quoted 22 lines · Hannes Reinecke
Quoted 9 lines · Keith Busch
Quoted 5 lines · Hannes Reinecke

We should only remove the ns head from the list of heads per subsystem if the reference count drops to zero. Removing it at the start of nvme_ns_remove() will prevent us from reattaching the namespace to the correct ns head once a path becomes available again.

This looks fine to me

Quoted 8 lines

There are potential corner cases associated with namespace management that can cause problems here, but it would certainly be a user error that needs manual intervention anyway. I suspect those scenarios are very uncommon too, so I think we can deal with that later if it becomes a problem.

How so?
We're keeping a copy of the NS IDs around, so after reconnect the namespace would have to provide same set of NS IDs.

Quoted 5 lines

If the namespace is different yet providing the same NS IDs we'll have a problem even with the current code, as we're using the ND ID to identify multipathed namespaces.

Or do you have a different scenario in mind?

Sorry, I didn't mean to imply this creates a new problem. It's already a problem. :)

This just provides a chance for a recycled ID for a newly created namespace to attach to an in-use head associated with a previously deleted namespace. I am not really concerned about this scenario, though, since it wouldn't work today anyway.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#4 lore raw
Quoted 28 lines
Quoted 22 lines
Quoted 10 lines · Hannes Reinecke
Quoted 5 lines

We should only remove the ns head from the list of heads per subsystem if the reference count drops to zero. Removing it at the start of nvme_ns_remove() will prevent us from reattaching the namespace to the correct ns head once a path becomes available again.

This looks fine to me

Quoted 8 lines

There are potential corner cases associated with namespace management that can cause problems here, but it would certainly be a user error that needs manual intervention anyway. I suspect those scenarios are very uncommon too, so I think we can deal with that later if it becomes a problem.

How so?
We're keeping a copy of the NS IDs around, so after reconnect the namespace would have to provide same set of NS IDs.

Quoted 5 lines

If the namespace is different yet providing the same NS IDs we'll have a problem even with the current code, as we're using the ND ID to identify multipathed namespaces.

Or do you have a different scenario in mind?

Quoted 7 lines

Sorry, I didn't mean to imply this creates a new problem. It's already a problem. :)

This just provides a chance for a recycled ID for a newly created namespace to attach to an in-use head associated with a previously deleted namespace. I am not really concerned about this scenario, though, since it wouldn't work today anyway.

That is because we keep the device node around even though it doesn't have any bottom devices. Which is what I think Hannes tried to fix in an earlier patch.

Also, if an NVM subsystem happens to recycle nsids, it will trigger pretty surely. So I'm not sure this is an obscure case...

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#5 lore raw
Quoted 36 lines · Sagi Grimberg
Quoted 28 lines
Quoted 22 lines
Quoted 10 lines · Hannes Reinecke
Quoted 5 lines

We should only remove the ns head from the list of heads per subsystem if the reference count drops to zero. Removing it at the start of nvme_ns_remove() will prevent us from reattaching the namespace to the correct ns head once a path becomes available again.

This looks fine to me

Quoted 8 lines

There are potential corner cases associated with namespace management that can cause problems here, but it would certainly be a user error that needs manual intervention anyway. I suspect those scenarios are very uncommon too, so I think we can deal with that later if it becomes a problem.

How so?
We're keeping a copy of the NS IDs around, so after reconnect the namespace would have to provide same set of NS IDs.

Quoted 5 lines

If the namespace is different yet providing the same NS IDs we'll have a problem even with the current code, as we're using the ND ID to identify multipathed namespaces.

Or do you have a different scenario in mind?

Quoted 7 lines

Sorry, I didn't mean to imply this creates a new problem. It's already a problem. :)

This just provides a chance for a recycled ID for a newly created namespace to attach to an in-use head associated with a previously deleted namespace. I am not really concerned about this scenario, though, since it wouldn't work today anyway.

Quoted 6 lines

That is because we keep the device node around even though it doesn't have any bottom devices. Which is what I think Hannes tried to fix in an earlier patch.

Also, if an NVM subsystem happens to recycle nsids, it will trigger pretty surely. So I'm not sure this is an obscure case...

I'm not very concerned because (1) deleting an in-use namespace is surely a user error, and (2) it currently doesn't work either, and no one's complained yet AFAIK.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#6 lore raw
Quoted 11 lines
Quoted 7 lines

That is because we keep the device node around even though it doesn't have any bottom devices. Which is what I think Hannes tried to fix in an earlier patch.

Also, if an NVM subsystem happens to recycle nsids, it will trigger pretty surely. So I'm not sure this is an obscure case...

I'm not very concerned because (1) deleting an in-use namespace is surely a user error, and (2) it currently doesn't work either, and no one's complained yet AFAIK.

You're not wrong. It's just that one error (on ns X) can manifest in other potentially completely different namespaces created. I guess this kind of thing can happen when ns administration is done by adminX (or scriptX) and the ns mounting is done by userY where what each is doing is unknown to each other.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#7 lore raw
Quoted 12 lines · Sagi Grimberg
Quoted 11 lines
Quoted 7 lines

That is because we keep the device node around even though it doesn't have any bottom devices. Which is what I think Hannes tried to fix in an earlier patch.

Also, if an NVM subsystem happens to recycle nsids, it will trigger pretty surely. So I'm not sure this is an obscure case...

I'm not very concerned because (1) deleting an in-use namespace is surely a user error, and (2) it currently doesn't work either, and no one's complained yet AFAIK.

Quoted 5 lines

You're not wrong. It's just that one error (on ns X) can manifest in other potentially completely different namespaces created. I guess this kind of thing can happen when ns administration is done by adminX (or scriptX) and the ns mounting is done by userY where what each is doing is unknown to each other.

We also check for the matching identifiers in nvme_init_ns_head, so we'll error out there.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#8 lore raw
In reply to Sagi Grimberg
Quoted 12 lines · Sagi Grimberg
Quoted 11 lines
Quoted 7 lines

That is because we keep the device node around even though it doesn't have any bottom devices. Which is what I think Hannes tried to fix in an earlier patch.

Also, if an NVM subsystem happens to recycle nsids, it will trigger pretty surely. So I'm not sure this is an obscure case...

I'm not very concerned because (1) deleting an in-use namespace is surely a user error, and (2) it currently doesn't work either, and no one's complained yet AFAIK.

Quoted 5 lines

You're not wrong. It's just that one error (on ns X) can manifest in other potentially completely different namespaces created. I guess this kind of thing can happen when ns administration is done by adminX (or scriptX) and the ns mounting is done by userY where what each is doing is unknown to each other.

This can only happen if NSFEAT bit 3 is not set and the namespace does not provide a UUID. So if we are concerned about that we can restrict nvme multipathing to exclude these scenarios. But in all other cases CNS 03 _is_ identifying the namespace, and cannot be re-used for other namespaces. And we are keeping a copy of CNS 03 in nshead, so we'll be safe there.

Cheers,

Hannes

Signature · 9 lines
Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme