nvme: delete disk after last connection
cover nvme: delete disk after last connection
Hi all,
as there now is a consensus that we should destroy 'struct nshead' when the last reference is dropped (and not once the last opener is released) here now is the updated patchset to make it happen.
The first is the original patch from Keith, for correctly dropping the disk reference.
The second one is for not removing struct nshead from the subsystem lists once the last path is gone, but rather defer it until the struct itself is freed.
As usual, comments and reviews are welcome.
Hannes Reinecke (1):
nvme: do not detach nshead when a namespace is removed
Keith Busch (1):
nvme-mpath: delete disk after last connection
drivers/nvme/host/core.c | 13 ++++++++----- drivers/nvme/host/multipath.c | 1 - drivers/nvme/host/nvme.h | 2 +- 3 files changed, 9 insertions(+), 7 deletions(-)
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 1/2 nvme-mpath: delete disk after last connection
From: Keith Busch <[email protected]>
The multipath code currently deletes the disk only after all references to it are dropped rather than when the last path to that disk is lost. This has been reported to cause problems with some usage, like MD RAID.
Delete the disk when the last path is gone. This is the same behavior we currently have with non-multipathed nvme devices.
The following is just a simple example that demonstrates what is currently observed using a simple nvme loop back (loop setup file not shown):
# nvmetcli restore loop.json [ 31.156452] nvmet: adding nsid 1 to subsystem testnqn1 [ 31.159140] nvmet: adding nsid 1 to subsystem testnqn2
# nvme connect -t loop -n testnqn1 -q hostnqn [ 36.866302] nvmet: creating controller 1 for subsystem testnqn1 for NQN hostnqn. [ 36.872926] nvme nvme3: new ctrl: "testnqn1"
# nvme connect -t loop -n testnqn1 -q hostnqn [ 38.227186] nvmet: creating controller 2 for subsystem testnqn1 for NQN hostnqn. [ 38.234450] nvme nvme4: new ctrl: "testnqn1"
# nvme connect -t loop -n testnqn2 -q hostnqn [ 43.902761] nvmet: creating controller 3 for subsystem testnqn2 for NQN hostnqn. [ 43.907401] nvme nvme5: new ctrl: "testnqn2"
# nvme connect -t loop -n testnqn2 -q hostnqn [ 44.627689] nvmet: creating controller 4 for subsystem testnqn2 for NQN hostnqn. [ 44.641773] nvme nvme6: new ctrl: "testnqn2"
# mdadm --create /dev/md0 --level=mirror --raid-devices=2 /dev/nvme3n1 /dev/nvme5n1 [ 53.497038] md/raid1:md0: active with 2 out of 2 mirrors [ 53.501717] md0: detected capacity change from 0 to 66060288
# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 nvme5n1[1] nvme3n1[0]
64512 blocks super 1.2 [2/2] [UU]Now delete all paths to one of the namespaces:
# echo 1 > /sys/class/nvme/nvme3/delete_controller # echo 1 > /sys/class/nvme/nvme4/delete_controller
We have no path, but mdstat says:
# cat /proc/mdstat
Personalities : [raid1]
md0 : active (auto-read-only) raid1 nvme5n1[1]
64512 blocks super 1.2 [2/1] [_U]And this is reported to cause a problem.
With the proposed patch, the following messages appear:
[ 227.516807] md/raid1:md0: Disk failure on nvme3n1, disabling device. [ 227.516807] md/raid1:md0: Operation continuing on 1 devices.
And mdstat shows only the viable members:
# cat /proc/mdstat
Personalities : [raid1]
md0 : active (auto-read-only) raid1 nvme5n1[1]
64512 blocks super 1.2 [2/1] [_U]--- drivers/nvme/host/core.c | 5 ++++- drivers/nvme/host/multipath.c | 1 - drivers/nvme/host/nvme.h | 2 +- 3 files changed, 5 insertions(+), 3 deletions(-)
drivers/nvme/host/core.c +4 -1
@@ -542,7 +542,10 @@ static void nvme_free_ns_head(struct kref *ref)struct nvme_ns_head *head = container_of(ref, struct nvme_ns_head, ref); -nvme_mpath_remove_disk (head); +#ifdef CONFIG_NVME_MULTIPATH +if (head->disk) +put_disk (head->disk); +#endif ida_simple_remove (&head->subsys->ns_ida, head->instance);cleanup_srcu_struct (&head->srcu);nvme_put_subsystem (head->subsys);
drivers/nvme/host/multipath.c +0 -1
@@ -702,7 +702,6 @@ void nvme_mpath_remove_disk(struct nvme_ns_head *head)*/ head->disk->queue =NULL ; } -put_disk (head->disk); }int nvme_mpath_init (struct nvme_ctrl *ctrl,struct nvme_id_ctrl *id)
drivers/nvme/host/nvme.h +1 -1
@@ -680,7 +680,7 @@ static inline void nvme_mpath_check_last_path(struct nvme_ns *ns)struct nvme_ns_head *head = ns->head;if (head->disk &&list_empty (&head->list)) -kblockd_schedule_work (&head->requeue_work); +nvme_mpath_remove_disk (head); }static inline void nvme_trace_bio_complete (struct request *req)
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 1/2] nvme-mpath: delete disk after last connection
Quoted 81 lines · Hannes Reinecke
From: Keith Busch <[email protected]>
The multipath code currently deletes the disk only after all references to it are dropped rather than when the last path to that disk is lost. This has been reported to cause problems with some usage, like MD RAID.
Delete the disk when the last path is gone. This is the same behavior we currently have with non-multipathed nvme devices.
The following is just a simple example that demonstrates what is currently observed using a simple nvme loop back (loop setup file not shown):
# nvmetcli restore loop.json [ 31.156452] nvmet: adding nsid 1 to subsystem testnqn1 [ 31.159140] nvmet: adding nsid 1 to subsystem testnqn2# nvme connect -t loop -n testnqn1 -q hostnqn [ 36.866302] nvmet: creating controller 1 for subsystem testnqn1 for NQN hostnqn. [ 36.872926] nvme nvme3: new ctrl: "testnqn1"# nvme connect -t loop -n testnqn1 -q hostnqn [ 38.227186] nvmet: creating controller 2 for subsystem testnqn1 for NQN hostnqn. [ 38.234450] nvme nvme4: new ctrl: "testnqn1"# nvme connect -t loop -n testnqn2 -q hostnqn [ 43.902761] nvmet: creating controller 3 for subsystem testnqn2 for NQN hostnqn. [ 43.907401] nvme nvme5: new ctrl: "testnqn2"# nvme connect -t loop -n testnqn2 -q hostnqn [ 44.627689] nvmet: creating controller 4 for subsystem testnqn2 for NQN hostnqn. [ 44.641773] nvme nvme6: new ctrl: "testnqn2"# mdadm --create /dev/md0 --level=mirror --raid-devices=2 /dev/nvme3n1 /dev/nvme5n1 [ 53.497038] md/raid1:md0: active with 2 out of 2 mirrors [ 53.501717] md0: detected capacity change from 0 to 66060288# cat /proc/mdstat Personalities : [raid1] md0 : active raid1 nvme5n1[1] nvme3n1[0] 64512 blocks super 1.2 [2/2] [UU]Now delete all paths to one of the namespaces:
# echo 1 > /sys/class/nvme/nvme3/delete_controller # echo 1 > /sys/class/nvme/nvme4/delete_controllerWe have no path, but mdstat says:
# cat /proc/mdstat Personalities : [raid1] md0 : active (auto-read-only) raid1 nvme5n1[1] 64512 blocks super 1.2 [2/1] [_U]And this is reported to cause a problem.
With the proposed patch, the following messages appear:
[ 227.516807] md/raid1:md0: Disk failure on nvme3n1, disabling device. [ 227.516807] md/raid1:md0: Operation continuing on 1 devices.And mdstat shows only the viable members:
# cat /proc/mdstat Personalities : [raid1] md0 : active (auto-read-only) raid1 nvme5n1[1] 64512 blocks super 1.2 [2/1] [_U]Reported-by Hannes Reinecke <[email protected]>Signed-off-by Keith Busch <[email protected]>Signed-off-by Hannes Reinecke <[email protected]>--- drivers/nvme/host/core.c | 5 ++++- drivers/nvme/host/multipath.c | 1 - drivers/nvme/host/nvme.h | 2 +- 3 files changed, 5 insertions(+), 3 deletions(-)drivers/nvme/host/core.c +0
-0
drivers/nvme/host/core.c static void nvme_free_ns_head(struct kref *ref) · line 548
Quoted 8 lines
drivers/nvme/host/core.c +4
-1@@ -542,7 +542,10 @@ static void nvme_free_ns_head(struct kref *ref)struct nvme_ns_head *head = container_of(ref, struct nvme_ns_head, ref); -nvme_mpath_remove_disk (head); +#ifdef CONFIG_NVME_MULTIPATH +if (head->disk) +put_disk (head->disk); +#endif
Hannes, maybe make that nvme_mpath_put_disk(head) and place an empty stab in nvme.h like we do for other mpath variants?
Other than that, patch looks good, so you can add to the respin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 1/2] nvme-mpath: delete disk after last connection
In reply to Hannes ReineckeQuoted 3 lines · Hannes Reinecke
Reported-by Hannes Reinecke <[email protected]>Signed-off-by Keith Busch <[email protected]>Signed-off-by Hannes Reinecke <[email protected]>
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 2/2 nvme: do not detach nshead when a namespace is removed
In reply to Hannes Reinecke'struct nshead' and 'struct ns' have different lifetimes; the nshead can (and occasionally will) stay around even if no namespaces are connected, as it will only be finally removed once the last reference is gone.
But while it is in this state we should still be able to access it during lookup; a nshead with no controllers is perfectly valid (at least for fabrics), and the namespace has no business with removing the nshead from the subsystem lists.
This patch fixes the situation by only removing the nshead from the subsystem list once it's being finally removed.
--- drivers/nvme/host/core.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
drivers/nvme/host/core.c +4 -4
@@ -542,6 +542,10 @@ static void nvme_free_ns_head(struct kref *ref)struct nvme_ns_head *head = container_of(ref, struct nvme_ns_head, ref); +mutex_lock (&head->subsys->lock); +list_del_init (&head->entry); +mutex_unlock (&head->subsys->lock); +#ifdef CONFIG_NVME_MULTIPATH if (head->disk)put_disk (head->disk);
@@ -3993,8 +3997,6 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid, out_unlink_ns:mutex_lock (&ctrl->subsys->lock);list_del_rcu (&ns->siblings); -if (list_empty (&ns->head->list)) -list_del_init (&ns->head->entry);mutex_unlock (&ctrl->subsys->lock);nvme_put_ns_head (ns->head); out_free_queue:
@@ -4015,8 +4017,6 @@ static void nvme_ns_remove(struct nvme_ns *ns)mutex_lock (&ns->ctrl->subsys->lock);list_del_rcu (&ns->siblings); -if (list_empty (&ns->head->list)) -list_del_init (&ns->head->entry);mutex_unlock (&ns->ctrl->subsys->lock);synchronize_rcu ();/* guarantee not available in head->list */
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCH 2/2] nvme: do not detach nshead when a namespace is removed
Quoted 10 lines
'struct nshead' and 'struct ns' have different lifetimes; the nshead can (and occasionally will) stay around even if no namespaces are connected, as it will only be finally removed once the last reference is gone.
But while it is in this state we should still be able to access it during lookup; a nshead with no controllers is perfectly valid (at least for fabrics), and the namespace has no business with removing the nshead from the subsystem lists.
Quoted 2 lines
This patch fixes the situation by only removing the nshead from the subsystem list once it's being finally removed.
Hannes, please explain what is the bug you are solving here.
I actually think that the nshead should be removed from the list as soon as the last the head->list is empty.
What if for example a new ns is allocated on the subsystem and the nsid (which was already deleted) is reused, it will now be rejected when we match against existing nsheads which will contain this stale nshead that happens to have an open reference.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme