nvme-multipath: fix deadlock in device_add_disk()
cover nvme-multipath: fix deadlock in device_add_disk()
Hi all,
I'm having a testcase which repeatedly disables namespaces on the target assigning new UUID (to simulate namespace remapping) and enable that namespace again.
To throw in more fun these namespaces have their ANA group ID changes to simulate namespace moving around in a cluster, where only the paths local to the cluster node are active, and all other paths are inaccessible.
Essentially it's doing something like:
echo 0 > ${ns}/enable
<random delay>
echo "<dev>" > ${ns}/device_path
echo "<grpid>" > ${ns}/ana_grpid
uuidgen > ${ns}/device_uuid
echo 1 > ${ns}/enable
ie a similar testcase than the previous patchset, only this time I'm just doing an 'enable/disable' bit without removing the namespace from the target.
This is causing lockups in device_add_disk(), as the partition scan is constantly retrying I/O and never completes.
Funnily enough the very same issue should have been fixed with ecca390e8056 ("nvme: fix deadlock in disconnect during scan_work and/or ana_work"), but that fix seem to be imperfect.
As usual, comments and reviews are welcome.
Hannes Reinecke (3):
nvme-multipath: simplify loop in nvme_update_ana_state()
nvme-multipath: cannot disconnect controller on stuck partition scan
nvme-multipath: skip failed paths during partition scan
drivers/nvme/host/multipath.c | 51 ++++++++++++++++++++++++++--------- drivers/nvme/host/nvme.h | 1 + 2 files changed, 40 insertions(+), 12 deletions(-)
Signature · 1 line
2.35.3
PATCH 1/3 nvme-multipath: simplify loop in nvme_update_ana_state()
Currently nvme_update_ana_state() iterates over all namespaces with the srcu lock held, and calls the 'cb' function if a match on the list of namespaces from the ANA log is found. This assumes that the iteration itself is relatively quick, and the namespace state itself doesn't change during iteration. For more complex functions (eg if the 'cb' function triggers mpath_set_live() which then kicks off a partition scan) the namespace state might change while iterating, and the rcu-protected area becomes really long.
So change the loop to iterated over the entries in the ANA log, and call nvme_find_get_ns() on each entry.
With that the 'cb' function is executed outside of the RCU protected area, and normal reference counting rules apply.
--- drivers/nvme/host/multipath.c | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-)
drivers/nvme/host/multipath.c +6 -12
@@ -745,7 +745,6 @@ static int nvme_update_ana_state(struct nvme_ctrl *ctrl,u32 nr_nsids =le32_to_cpu (desc->nnsids), n =0 ;unsigned *nr_change_groups = data;struct nvme_ns *ns; -int srcu_idx; dev_dbg(ctrl->device, "ANA group %d: %s.\n", le32_to_cpu(desc->grpid),
@@ -757,21 +756,16 @@ static int nvme_update_ana_state(struct nvme_ctrl *ctrl,if (!nr_nsids)return 0 ; - srcu_idx =srcu_read_lock (&ctrl->srcu); -list_for_each_entry_rcu (ns, &ctrl->namespaces, list) { +for (n =0 ; n < nr_nsids; n++) {unsigned nsid; -again: + nsid =le32_to_cpu (desc->nsids[n]); -if (ns->head->ns_id < nsid) -continue ; -if (ns->head->ns_id == nsid) + ns =nvme_find_get_ns (ctrl, nsid); +if (ns) {nvme_update_ns_ana_state (desc, ns); -if (++n == nr_nsids) -break ; -if (ns->head->ns_id > nsid) - goto again; +nvme_put_ns (ns); + } } -srcu_read_unlock (&ctrl->srcu, srcu_idx);return 0 ; }
Signature · 1 line
2.35.3
Keith Busch
maintainerRe: [PATCH 1/3] nvme-multipath: simplify loop in nvme_update_ana_state()
Quoted 5 lines · Hannes Reinecke
Currently
nvme_update_ana_state()iterates over all namespaces with the srcu lock held, and calls the 'cb' function if a match on the list of namespaces from the ANA log is found. This assumes that the iteration itself is relatively quick, and the namespace state itself doesn't change during iteration.
Quoted 8 lines
For more complex functions (eg if the 'cb' function triggers
mpath_set_live()which then kicks off a partition scan) the namespace state might change while iterating, and the rcu-protected area becomes really long.
So change the loop to iterated over the entries in the ANA log, and callnvme_find_get_ns()on each entry.
With that the 'cb' function is executed outside of the RCU protected area, and normal reference counting rules apply.
This patch looks good to me.
Your commit log is word wrapping a bit short, though. We standardize on 72 characters.
Christoph Hellwig
maintainerRe: [PATCH 1/3] nvme-multipath: simplify loop in nvme_update_ana_state()
Quoted 4 lines · Keith Busch
This patch looks good to me.
Your commit log is word wrapping a bit short, though. We standardize on 72 characters.
The for loop is also missing a space before the opening brace while we're nitpicking.
The reason the code was written this way is that when there are a lot of namespace in an update we now need to a full list scan for each one, which can add a lot of overhead. So I guess we'll need to see why it is important to change it to counter that tradeoff.
Re: [PATCH 1/3] nvme-multipath: simplify loop in nvme_update_ana_state()
In reply to Hannes ReineckeQuoted 46 lines · Hannes Reinecke
Currently
nvme_update_ana_state()iterates over all namespaces with the srcu lock held, and calls the 'cb' function if a match on the list of namespaces from the ANA log is found. This assumes that the iteration itself is relatively quick, and the namespace state itself doesn't change during iteration. For more complex functions (eg if the 'cb' function triggersmpath_set_live()which then kicks off a partition scan) the namespace state might change while iterating, and the rcu-protected area becomes really long.
So change the loop to iterated over the entries in the ANA log, and callnvme_find_get_ns()on each entry.
With that the 'cb' function is executed outside of the RCU protected area, and normal reference counting rules apply.Signed-off-by Hannes Reinecke <[email protected]>--- drivers/nvme/host/multipath.c | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-)drivers/nvme/host/multipath.c +3
-7@@ -745,7 +745,6 @@ static int nvme_update_ana_state(struct nvme_ctrl *ctrl,u32 nr_nsids =le32_to_cpu (desc->nnsids), n =0 ;unsigned *nr_change_groups = data;struct nvme_ns *ns; -int srcu_idx; dev_dbg(ctrl->device, "ANA group %d: %s.\n", le32_to_cpu(desc->grpid),@@ -757,21 +756,16 @@ static int nvme_update_ana_state(struct nvme_ctrl *ctrl,if (!nr_nsids)return 0 ; - srcu_idx =srcu_read_lock (&ctrl->srcu); -list_for_each_entry_rcu (ns, &ctrl->namespaces, list) { +for (n =0 ; n < nr_nsids; n++) {unsigned nsid; -again: + nsid =le32_to_cpu (desc->nsids[n]); -if (ns->head->ns_id < nsid) -continue ; -if (ns->head->ns_id == nsid) + ns =nvme_find_get_ns (ctrl, nsid);
drivers/nvme/host/multipath.c static int nvme_update_ana_state(struct nvme_ctrl *ctrl, · line 774
Quoted 10 lines
drivers/nvme/host/multipath.c +3
-5+if (ns) {nvme_update_ns_ana_state (desc, ns); -if (++n == nr_nsids) -break ; -if (ns->head->ns_id > nsid) - goto again; +nvme_put_ns (ns); + } } -srcu_read_unlock (&ctrl->srcu, srcu_idx);
You also made the loop now n^2... which *may* be an issue for an environment with a lot of namespaces. It is simpler though...
PATCH 2/3 nvme-multipath: cannot disconnect controller on stuck partition scan
In reply to Hannes ReineckeWhen a namespace state is changed during partition scan triggered via nvme_scan_ns->nvme_mpath_set_live()->device_add_disk()
I/O might be returned with a path error, causing it to be retried on other paths. But if this happens to be the last path the process will be stuck.
Trying to disconnect this controller will call
nvme_unquiesce_io_queues()flush_work(&ctrl_scan_work)
where the first should abort/retry all I/O pending during scan such that the following 'flush_work' can succeeed. However, we explicitly do _not_ ignore paths from deleted controllers in nvme_mpath_is_disabled(), so that I/O on these devices will be _retried_, not aborted, and the scanning process continues to be stuck. So the process to disconnect the controller will be stuck in flush_work(), and that controller and all namespaces become unusable until the system is rebooted.
--- drivers/nvme/host/multipath.c | 7 +++++++ 1 file changed, 7 insertions(+)
drivers/nvme/host/multipath.c +7 -0
@@ -239,6 +239,13 @@ static bool nvme_path_is_disabled(struct nvme_ns *ns) {enum nvme_ctrl_state state =nvme_ctrl_state (ns->ctrl); +/* +* Skip deleted controllers for I/O from partition scan +*/ +if (state ==NVME_CTRL_DELETING && +mutex_is_locked (&ns->ctrl->scan_lock)) +return true; +/* * We don't treat NVME_CTRL_DELETING as a disabled path as I/O should * still be able to complete assuming that the controller is connected.
Signature · 1 line
2.35.3
Keith Busch
maintainerRe: [PATCH 2/3] nvme-multipath: cannot disconnect controller on stuck partition scan
Review static bool nvme_path_is_disabled(struct nvme_ns *ns) · line 247
Quoted 10 lines · Hannes Reinecke
(quoted diff) +6
-0@@ -239,6 +239,13 @@ static bool nvme_path_is_disabled(struct nvme_ns *ns) { enum nvme_ctrl_state state = nvme_ctrl_state(ns->ctrl); + /* + * Skip deleted controllers for I/O from partition scan + */ + if (state == NVME_CTRL_DELETING && + mutex_is_locked(&ns->ctrl->scan_lock)) + return true;
This feels off to me, using these seemingly unrelated dependencies to make these kinds of decisions.
We talked a couple weeks ago about suppressing the parition scanning during nvme scan_work. I know you said there was some reason it wouldn't work, but could you check the below? It seems okay to me.
---
drivers/nvme/host/multipath.c +7 -0
@@ -586,6 +586,12 @@ static void nvme_requeue_work(struct work_struct *work) container_of(work, struct nvme_ns_head, requeue_work);struct bio *bio, *next; +if (test_and_clear_bit (GD_SUPPRESS_PART_SCAN , &head->disk->state)) { +mutex_lock (&head->disk->open_mutex); +bdev_disk_changed (head->disk, false); +mutex_unlock (&head->disk->open_mutex); + } +spin_lock_irq (&head->requeue_lock); next =bio_list_get (&head->requeue_list);spin_unlock_irq (&head->requeue_lock);
@@ -629,6 +635,7 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)return PTR_ERR (head->disk); head->disk->fops = &nvme_ns_head_ops; head->disk->private_data = head; +set_bit (GD_SUPPRESS_PART_SCAN , &head->disk->state);sprintf (head->disk->disk_name,"nvme%dn%d" , ctrl->subsys->instance, head->instance);return 0 ;
--
Christoph Hellwig
maintainerRe: [PATCH 2/3] nvme-multipath: cannot disconnect controller on stuck partition scan
Quoted 25 lines · Keith Busch
Review static bool nvme_path_is_disabled(struct nvme_ns *ns) · line 247
Quoted 10 lines · Hannes Reinecke
(quoted diff) +6
-0@@ -239,6 +239,13 @@ static bool nvme_path_is_disabled(struct nvme_ns *ns) { enum nvme_ctrl_state state = nvme_ctrl_state(ns->ctrl); + /* + * Skip deleted controllers for I/O from partition scan + */ + if (state == NVME_CTRL_DELETING && + mutex_is_locked(&ns->ctrl->scan_lock)) + return true;This feels off to me, using these seemingly unrelated dependencies to make these kinds of decisions.
We talked a couple weeks ago about suppressing the parition scanning during nvme
scan_work. I know you said there was some reason it wouldn't work, but could you check the below? It seems okay to me.---
drivers/nvme/host/multipath.c +0
-0
drivers/nvme/host/multipath.c static void nvme_requeue_work(struct work_struct *work) · line 594
Quoted 9 lines
drivers/nvme/host/multipath.c +6
-0@@ -586,6 +586,12 @@ static void nvme_requeue_work(struct work_struct *work) container_of(work, struct nvme_ns_head, requeue_work);struct bio *bio, *next; +if (test_and_clear_bit (GD_SUPPRESS_PART_SCAN , &head->disk->state)) { +mutex_lock (&head->disk->open_mutex); +bdev_disk_changed (head->disk, false); +mutex_unlock (&head->disk->open_mutex); + } +
Rescan_work feels a little counter intuitive for the partition scanning. I guess this should work because requeue_work is scheduled from the end of nvme_mpath_set_live and gives us a context outside of scan_lock.
It'll need really good comments to explain this.
Re: [PATCH 2/3] nvme-multipath: cannot disconnect controller on stuck partition scan
In reply to Keith BuschQuoted 38 lines · Keith Busch
Review static bool nvme_path_is_disabled(struct nvme_ns *ns) · line 247
Quoted 10 lines · Hannes Reinecke
(quoted diff) +6
-0@@ -239,6 +239,13 @@ static bool nvme_path_is_disabled(struct nvme_ns *ns) { enum nvme_ctrl_state state = nvme_ctrl_state(ns->ctrl); + /* + * Skip deleted controllers for I/O from partition scan + */ + if (state == NVME_CTRL_DELETING && + mutex_is_locked(&ns->ctrl->scan_lock)) + return true;This feels off to me, using these seemingly unrelated dependencies to make these kinds of decisions.
We talked a couple weeks ago about suppressing the parition scanning during nvme
scan_work. I know you said there was some reason it wouldn't work, but could you check the below? It seems okay to me.---
drivers/nvme/host/multipath.c +6
-0@@ -586,6 +586,12 @@ static void nvme_requeue_work(struct work_struct *work) container_of(work, struct nvme_ns_head, requeue_work);struct bio *bio, *next; +if (test_and_clear_bit (GD_SUPPRESS_PART_SCAN , &head->disk->state)) { +mutex_lock (&head->disk->open_mutex); +bdev_disk_changed (head->disk, false); +mutex_unlock (&head->disk->open_mutex); + } +spin_lock_irq (&head->requeue_lock); next =bio_list_get (&head->requeue_list);spin_unlock_irq (&head->requeue_lock);
drivers/nvme/host/multipath.c int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)
Quoted 8 lines
drivers/nvme/host/multipath.c +1
-1@@ -629,6 +635,7 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)return PTR_ERR (head->disk); head->disk->fops = &nvme_ns_head_ops; head->disk->private_data = head; +set_bit (GD_SUPPRESS_PART_SCAN , &head->disk->state);sprintf (head->disk->disk_name,"nvme%dn%d" , ctrl->subsys->instance, head->instance);return 0 ; --
Hmm. Sadly, not really. Still stuck in partition scanning.
Guess we'll need to check if we have paths available before triggering partition scan; let me check ...
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Keith Busch
maintainerRe: [PATCH 2/3] nvme-multipath: cannot disconnect controller on stuck partition scan
Quoted 4 lines · Hannes Reinecke
Hmm. Sadly, not really. Still stuck in partition scanning.
Guess we'll need to check if we have paths available before triggering partition scan; let me check ...
I think you must have two paths, and the 2nd path cleared the suppress bit before the 1st one finished its called to device_add_disk(). How about this one?
---
drivers/nvme/host/multipath.c +8 -0
@@ -586,6 +586,13 @@ static void nvme_requeue_work(struct work_struct *work) container_of(work, struct nvme_ns_head, requeue_work);struct bio *bio, *next; +if (disk_live (head->disk) && +test_and_clear_bit (GD_SUPPRESS_PART_SCAN , &head->disk->state)) { +mutex_lock (&head->disk->open_mutex); +bdev_disk_changed (head->disk, false); +mutex_unlock (&head->disk->open_mutex); + } +spin_lock_irq (&head->requeue_lock); next =bio_list_get (&head->requeue_list);spin_unlock_irq (&head->requeue_lock);
@@ -629,6 +636,7 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)return PTR_ERR (head->disk); head->disk->fops = &nvme_ns_head_ops; head->disk->private_data = head; +set_bit (GD_SUPPRESS_PART_SCAN , &head->disk->state);sprintf (head->disk->disk_name,"nvme%dn%d" , ctrl->subsys->instance, head->instance);return 0 ;
--
Re: [PATCH 2/3] nvme-multipath: cannot disconnect controller on stuck partition scan
Quoted 10 lines · Keith Busch
Quoted 4 lines · Hannes Reinecke
Hmm. Sadly, not really. Still stuck in partition scanning.
Guess we'll need to check if we have paths available before triggering partition scan; let me check ...
I think you must have two paths, and the 2nd path cleared the suppress bit before the 1st one finished its called to
device_add_disk(). How about this one?
Nope. Still stuck, this time in bdev_disc_changed().
With my testcase _all_ paths return NS_NOT_READY during partition scan, so I/O is constantly bounced between paths, and partition scan never returns. Doesn't matter where you call it, it's stuck.
The only chance we have is to modify I/O handling during scanning (cf my new patchset).
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Keith Busch
maintainerRe: [PATCH 2/3] nvme-multipath: cannot disconnect controller on stuck partition scan
Quoted 7 lines · Hannes Reinecke
Quoted 6 lines · Keith Busch
Quoted 4 lines · Hannes Reinecke
Hmm. Sadly, not really. Still stuck in partition scanning.
Guess we'll need to check if we have paths available before triggering partition scan; let me check ...
Quoted 8 lines
Quoted 4 lines
I think you must have two paths, and the 2nd path cleared the suppress bit before the 1st one finished its called to
device_add_disk(). How about this one?Nope. Still stuck, this time in
bdev_disc_changed().
With my testcase _all_ paths returnNS_NOT_READYduring partition scan, so I/O is constantly bounced between paths, and partition scan never returns. Doesn't matter where you call it, it's stuck.
Oh right... We need the requeue_work to end the bio's when the disk is dead, but the work can't proceed because it's waiting on its own bio's. Darn, I guess this scheme would need yet another work_queue to do it.
Quoted 2 lines
The only chance we have is to modify I/O handling during scanning (cf my new patchset).
I hoped we could avoid special error handling paths, but I'll take another look at that approach.
Keith Busch
maintainerRe: [PATCH 2/3] nvme-multipath: cannot disconnect controller on stuck partition scan
In reply to Hannes ReineckeQuoted 3 lines · Hannes Reinecke
With my testcase _all_ paths return
NS_NOT_READYduring partition scan, so I/O is constantly bounced between paths, and partition scan never returns. Doesn't matter where you call it, it's stuck.
I'm trying to recreate this so just want to clarify the return status. NS_NOT_READY isn't a path error, so doesn't use failover. It eventually exhausts its retries, so I don't see a problem with that status.
If I synthesize a pathing error though, then yeah, I can recreate a problem.
Keith Busch
maintainerRe: [PATCH 2/3] nvme-multipath: cannot disconnect controller on stuck partition scan
In reply to Hannes ReineckeQuoted 3 lines · Hannes Reinecke
With my testcase _all_ paths return
NS_NOT_READYduring partition scan, so I/O is constantly bounced between paths, and partition scan never returns. Doesn't matter where you call it, it's stuck.
Assuming you mean a differet status, like NVME_SC_INTERNAL_PATH_ERROR, I can recreate a stuck scan. Let me get one more shot at fixing this by suppressing the scan to another context. This one is successful in my tests:
---
drivers/nvme/host/multipath.c +24 -6
@@ -580,6 +580,20 @@ static int nvme_add_ns_head_cdev(struct nvme_ns_head *head)return ret; } +static void nvme_scan_work (struct work_struct *work) +{ +struct nvme_ns_head *head = + container_of(work, struct nvme_ns_head, scan_work); + +if (WARN_ON_ONCE (!test_and_clear_bit (GD_SUPPRESS_PART_SCAN , + &head->disk->state))) +return ; + +mutex_lock (&head->disk->open_mutex); +bdev_disk_changed (head->disk, false); +mutex_unlock (&head->disk->open_mutex); +} +static void nvme_requeue_work (struct work_struct *work) { struct nvme_ns_head *head =
@@ -606,6 +620,7 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)bio_list_init (&head->requeue_list);spin_lock_init (&head->requeue_lock);INIT_WORK (&head->requeue_work, nvme_requeue_work); +INIT_WORK (&head->scan_work, nvme_scan_work);/* * Add a multipath node if the subsystems supports multiple controllers.
@@ -629,6 +644,7 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)return PTR_ERR (head->disk); head->disk->fops = &nvme_ns_head_ops; head->disk->private_data = head; +set_bit (GD_SUPPRESS_PART_SCAN , &head->disk->state);sprintf (head->disk->disk_name,"nvme%dn%d" , ctrl->subsys->instance, head->instance);return 0 ;
@@ -655,6 +671,7 @@ static void nvme_mpath_set_live(struct nvme_ns *ns)return ; }nvme_add_ns_head_cdev (head); +kblockd_schedule_work (&head->scan_work); }mutex_lock (&head->lock);
@@ -974,14 +991,14 @@ void nvme_mpath_shutdown_disk(struct nvme_ns_head *head)return ;if (test_and_clear_bit (NVME_NSHEAD_DISK_LIVE , &head->flags)) {nvme_cdev_del (&head->cdev, &head->cdev_device); +/* +* requeue I/O after NVME_NSHEAD_DISK_LIVE has been cleared +* to allow multipath to fail all I/O. +*/ +synchronize_srcu (&head->srcu); +kblockd_schedule_work (&head->requeue_work);del_gendisk (head->disk); } -/* -* requeue I/O after NVME_NSHEAD_DISK_LIVE has been cleared -* to allow multipath to fail all I/O. -*/ -synchronize_srcu (&head->srcu); -kblockd_schedule_work (&head->requeue_work); }void nvme_mpath_remove_disk (struct nvme_ns_head *head)
@@ -991,6 +1008,7 @@ void nvme_mpath_remove_disk(struct nvme_ns_head *head)/* make sure all pending bios are cleaned up */ kblockd_schedule_work (&head->requeue_work);flush_work (&head->requeue_work); +flush_work (&head->scan_work);put_disk (head->disk); }
drivers/nvme/host/nvme.h +1 -0
@@ -494,6 +494,7 @@ struct nvme_ns_head {struct bio_list requeue_list;spinlock_t requeue_lock;struct work_struct requeue_work; +struct work_struct scan_work;struct mutex lock;unsigned long flags;#define NVME_NSHEAD_DISK_LIVE 0
--
Re: [PATCH 2/3] nvme-multipath: cannot disconnect controller on stuck partition scan
Quoted 16 lines · Keith Busch
Quoted 3 lines · Hannes Reinecke
With my testcase _all_ paths return
NS_NOT_READYduring partition scan, so I/O is constantly bounced between paths, and partition scan never returns. Doesn't matter where you call it, it's stuck.Assuming you mean a differet status, like
NVME_SC_INTERNAL_PATH_ERROR, I can recreate a stuck scan. Let me get one more shot at fixing this by suppressing the scan to another context. This one is successful in my tests:---
drivers/nvme/host/multipath.c +0
-0
drivers/nvme/host/multipath.c static int nvme_add_ns_head_cdev(struct nvme_ns_head *head) · line 580
Quoted 1 line
drivers/nvme/host/multipath.c +0
-0@@ -580,6 +580,20 @@ static int nvme_add_ns_head_cdev(struct nvme_ns_head *head)return ret;
[ .. ]
Piling workqueues on workqueues ... but anyway, I'll give it a spin.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Re: [PATCH 2/3] nvme-multipath: cannot disconnect controller on stuck partition scan
Quoted 21 lines · Hannes Reinecke
Quoted 18 lines · Keith Busch
Quoted 3 lines · Hannes Reinecke
With my testcase _all_ paths return
NS_NOT_READYduring partition scan, so I/O is constantly bounced between paths, and partition scan never returns. Doesn't matter where you call it, it's stuck.Assuming you mean a differet status, like
NVME_SC_INTERNAL_PATH_ERROR, I can recreate a stuck scan. Let me get one more shot at fixing this by suppressing the scan to another context. This one is successful in my tests:---
drivers/nvme/host/multipath.c +0
-0@@ -580,6 +580,20 @@ static int nvme_add_ns_head_cdev(struct nvme_ns_head *head) return ret;[ .. ]
Quoted 2 lines
Piling workqueues on workqueues ... but anyway, I'll give it a spin.
And it even works. Care to send a proper patch?
You can add my 'Tested-by:' to it, too.
Thanks for doing this.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Christoph Hellwig
maintainerRe: [PATCH 2/3] nvme-multipath: cannot disconnect controller on stuck partition scan
In reply to Keith BuschQuoted 16 lines · Keith Busch
Quoted 3 lines · Hannes Reinecke
With my testcase _all_ paths return
NS_NOT_READYduring partition scan, so I/O is constantly bounced between paths, and partition scan never returns. Doesn't matter where you call it, it's stuck.Assuming you mean a differet status, like
NVME_SC_INTERNAL_PATH_ERROR, I can recreate a stuck scan. Let me get one more shot at fixing this by suppressing the scan to another context. This one is successful in my tests:---
drivers/nvme/host/multipath.c +0
-0
drivers/nvme/host/multipath.c static int nvme_add_ns_head_cdev(struct nvme_ns_head *head) · line 583
Quoted 4 lines
drivers/nvme/host/multipath.c +1
-0@@ -580,6 +580,20 @@ static int nvme_add_ns_head_cdev(struct nvme_ns_head *head)return ret; } +static void nvme_scan_work (struct work_struct *work)
scan_work feels a little too generic. Maybe part_scan_work?
Otherwise not pretty, but by far the best idea so far, so I guess should go with it. Can you also add a big fat comment explaining the rationale for the actual submission?
Re: [PATCH 2/3] nvme-multipath: cannot disconnect controller on stuck partition scan
In reply to Keith BuschQuoted 62 lines · Keith Busch
Quoted 3 lines · Hannes Reinecke
With my testcase _all_ paths return
NS_NOT_READYduring partition scan, so I/O is constantly bounced between paths, and partition scan never returns. Doesn't matter where you call it, it's stuck.Assuming you mean a differet status, like
NVME_SC_INTERNAL_PATH_ERROR, I can recreate a stuck scan. Let me get one more shot at fixing this by suppressing the scan to another context. This one is successful in my tests:---
drivers/nvme/host/multipath.c +17
-0@@ -580,6 +580,20 @@ static int nvme_add_ns_head_cdev(struct nvme_ns_head *head)return ret; } +static void nvme_scan_work (struct work_struct *work) +{ +struct nvme_ns_head *head = + container_of(work, struct nvme_ns_head, scan_work); + +if (WARN_ON_ONCE (!test_and_clear_bit (GD_SUPPRESS_PART_SCAN , + &head->disk->state))) +return ; + +mutex_lock (&head->disk->open_mutex); +bdev_disk_changed (head->disk, false); +mutex_unlock (&head->disk->open_mutex); +} +static void nvme_requeue_work (struct work_struct *work) { struct nvme_ns_head *head =@@ -606,6 +620,7 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)bio_list_init (&head->requeue_list);spin_lock_init (&head->requeue_lock);INIT_WORK (&head->requeue_work, nvme_requeue_work); +INIT_WORK (&head->scan_work, nvme_scan_work);/* * Add a multipath node if the subsystems supports multiple controllers. @@ -629,6 +644,7 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)return PTR_ERR (head->disk); head->disk->fops = &nvme_ns_head_ops; head->disk->private_data = head; +set_bit (GD_SUPPRESS_PART_SCAN , &head->disk->state);sprintf (head->disk->disk_name,"nvme%dn%d" , ctrl->subsys->instance, head->instance);return 0 ;@@ -655,6 +671,7 @@ static void nvme_mpath_set_live(struct nvme_ns *ns)return ; }nvme_add_ns_head_cdev (head); +kblockd_schedule_work (&head->scan_work); }mutex_lock (&head->lock);@@ -974,14 +991,14 @@ void nvme_mpath_shutdown_disk(struct nvme_ns_head *head)return ;
drivers/nvme/host/multipath.c void nvme_mpath_shutdown_disk(struct nvme_ns_head *head) · line 1001
Quoted 10 lines
drivers/nvme/host/multipath.c +6
-0if (test_and_clear_bit (NVME_NSHEAD_DISK_LIVE , &head->flags)) {nvme_cdev_del (&head->cdev, &head->cdev_device); +/* +* requeue I/O after NVME_NSHEAD_DISK_LIVE has been cleared +* to allow multipath to fail all I/O. +*/ +synchronize_srcu (&head->srcu); +kblockd_schedule_work (&head->requeue_work);del_gendisk (head->disk); }
I guess we need to split 'test_and_clear_bit()' into a 'test_bit()' when testing for the condition and a 'clear_bit()' after del_gendisk().
Otherwise we're having a race condition with nvme_mpath_set_live:
if (!test_and_set_bit(NVME_NSHEAD_DISK_LIVE, &head->flags)) {
rc = device_add_disk(&head->subsys->dev, head->disk,
nvme_ns_attr_groups);which could sneak in from another controller just after we cleared the NVME_NSHEAD_DISK_LIVE flag, causing device_add_disk() to fail as the same name is already registered.
Or nvme_cdev_del() to display nice kernel warnings as the cdev was not registered.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Keith Busch
maintainerRe: [PATCH 2/3] nvme-multipath: cannot disconnect controller on stuck partition scan
Quoted 23 lines · Hannes Reinecke
drivers/nvme/host/multipath.c void nvme_mpath_shutdown_disk(struct nvme_ns_head *head) · line 1001
Quoted 12 lines · Keith Busch
drivers/nvme/host/multipath.c +6
-0@@ -974,14 +991,14 @@ void nvme_mpath_shutdown_disk(struct nvme_ns_head *head)return ;if (test_and_clear_bit (NVME_NSHEAD_DISK_LIVE , &head->flags)) {nvme_cdev_del (&head->cdev, &head->cdev_device); +/* +* requeue I/O after NVME_NSHEAD_DISK_LIVE has been cleared +* to allow multipath to fail all I/O. +*/ +synchronize_srcu (&head->srcu); +kblockd_schedule_work (&head->requeue_work);del_gendisk (head->disk); }I guess we need to split '
test_and_clear_bit()' into a 'test_bit()' when testing for the condition and a 'clear_bit()' afterdel_gendisk().Otherwise we're having a race condition with
nvme_mpath_set_live:if (!test_and_set_bit(NVME_NSHEAD_DISK_LIVE, &head->flags)) { rc = device_add_disk(&head->subsys->dev, head->disk, nvme_ns_attr_groups);
Quoted 5 lines
which could sneak in from another controller just after we cleared the
NVME_NSHEAD_DISK_LIVEflag, causingdevice_add_disk()to fail as the same name is already registered.
Ornvme_cdev_del()to display nice kernel warnings as the cdev was not registered.
Is that actually happening for you? I don't think it's supposed to happen because mpath_shutdwon_disk only happens if the head is detached from the subsystem, so no other thread should be able to possibly attempt to access that head's disk for a different controller path. That new controller should have to allocate an entirely new head.
Re: [PATCH 2/3] nvme-multipath: cannot disconnect controller on stuck partition scan
Quoted 30 lines · Keith Busch
Quoted 23 lines · Hannes Reinecke
drivers/nvme/host/multipath.c void nvme_mpath_shutdown_disk(struct nvme_ns_head *head) · line 1001
Quoted 12 lines · Keith Busch
drivers/nvme/host/multipath.c +6
-0@@ -974,14 +991,14 @@ void nvme_mpath_shutdown_disk(struct nvme_ns_head *head)return ;if (test_and_clear_bit (NVME_NSHEAD_DISK_LIVE , &head->flags)) {nvme_cdev_del (&head->cdev, &head->cdev_device); +/* +* requeue I/O after NVME_NSHEAD_DISK_LIVE has been cleared +* to allow multipath to fail all I/O. +*/ +synchronize_srcu (&head->srcu); +kblockd_schedule_work (&head->requeue_work);del_gendisk (head->disk); }I guess we need to split '
test_and_clear_bit()' into a 'test_bit()' when testing for the condition and a 'clear_bit()' afterdel_gendisk().Otherwise we're having a race condition with
nvme_mpath_set_live:if (!test_and_set_bit(NVME_NSHEAD_DISK_LIVE, &head->flags)) { rc = device_add_disk(&head->subsys->dev, head->disk, nvme_ns_attr_groups);Quoted 5 lines
which could sneak in from another controller just after we cleared the
NVME_NSHEAD_DISK_LIVEflag, causingdevice_add_disk()to fail as the same name is already registered.
Ornvme_cdev_del()to display nice kernel warnings as the cdev was not registered.
Quoted 5 lines
Is that actually happening for you? I don't think it's supposed to happen because
mpath_shutdwon_diskonly happens if the head is detached from the subsystem, so no other thread should be able to possibly attempt to access that head's disk for a different controller path. That new controller should have to allocate an entirely new head.
Na, forget it. Not only doesn't it happen, but we need the 'test_and_set_bit()' at the start otherwise we can't abort I/O. So all's fine with your patch, and it even survives my testcase.
(The first to do so even after several runs, I might add :-)
Care to send an official patch?
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Keith Busch
maintainerRe: [PATCH 2/3] nvme-multipath: cannot disconnect controller on stuck partition scan
Quoted 7 lines · Hannes Reinecke
Quoted 5 lines · Keith Busch
Is that actually happening for you? I don't think it's supposed to happen because
mpath_shutdwon_diskonly happens if the head is detached from the subsystem, so no other thread should be able to possibly attempt to access that head's disk for a different controller path. That new controller should have to allocate an entirely new head.
Quoted 7 lines
Na, forget it. Not only doesn't it happen, but we need the '
test_and_set_bit()' at the start otherwise we can't abort I/O. So all's fine with your patch, and it even survives my testcase.(The first to do so even after several runs, I might add :-)
Care to send an official patch?
Should be in your mailbox. Here's the link:
Sagi Grimberg
maintainerRe: [PATCH 2/3] nvme-multipath: cannot disconnect controller on stuck partition scan
Quoted 11 lines · Keith Busch
Quoted 10 lines · Hannes Reinecke
Quoted 5 lines · Keith Busch
Is that actually happening for you? I don't think it's supposed to happen because
mpath_shutdwon_diskonly happens if the head is detached from the subsystem, so no other thread should be able to possibly attempt to access that head's disk for a different controller path. That new controller should have to allocate an entirely new head.Na, forget it. Not only doesn't it happen, but we need the '
test_and_set_bit()' at the start otherwise we can't abort I/O. So all's fine with your patch, and it even survives my testcase.
Quoted 6 lines
Quoted 3 lines
(The first to do so even after several runs, I might add :-)
Care to send an official patch?
Should be in your mailbox. Here's the link:
https://lore.kernel.org/linux-nvme/[email protected]/T/#u
I'm glad we didn't take the original version of this patch :)
PATCH 3/3 nvme-multipath: skip failed paths during partition scan
In reply to Hannes ReineckeFrom: Hannes Reinecke <[email protected]>
When an I/O error is encountered during scanning (ie when the scan_lock is held) we should avoid using this path until scanning is finished to avoid deadlocks with device_add_disk().
So set a new flag NVME_NS_SCAN_FAILED if a failover happened during scanning, and skip this path in nvme_available_paths().
Then we can check if that bit is set after device_add_disk() returned, and remove the disk again if no available paths are found. That allows the device to be recreated via the 'rescan' sysfs attribute once no I/O errors occur anymore.
--- drivers/nvme/host/multipath.c | 26 ++++++++++++++++++++++++++ drivers/nvme/host/nvme.h | 1 + 2 files changed, 27 insertions(+)
drivers/nvme/host/multipath.c +26 -0
@@ -102,6 +102,13 @@ void nvme_failover_req(struct request *req)queue_work (nvme_wq, &ns->ctrl->ana_work); } +/* +* Do not use this path during scanning +* to avoid deadlocks in device_add_disk() +*/ +if (mutex_is_locked (&ns->ctrl->scan_lock)) +set_bit (NVME_NS_SCAN_FAILED , &ns->flags); +spin_lock_irqsave (&ns->head->requeue_lock, flags);for (bio = req->bio; bio; bio = bio->bi_next) {bio_set_dev (bio, ns->head->disk->part0);
@@ -434,6 +441,10 @@ static bool nvme_available_path(struct nvme_ns_head *head)list_for_each_entry_rcu (ns, &head->list, siblings) {if (test_bit (NVME_CTRL_FAILFAST_EXPIRED , &ns->ctrl->flags))continue ; +if (test_bit (NVME_NS_SCAN_FAILED , &ns->flags) && +mutex_is_locked (&ns->ctrl->scan_lock)) +continue ; +switch (nvme_ctrl_state (ns->ctrl)) {case NVME_CTRL_LIVE :case NVME_CTRL_RESETTING :
@@ -659,6 +670,20 @@ static void nvme_mpath_set_live(struct nvme_ns *ns)clear_bit (NVME_NSHEAD_DISK_LIVE , &head->flags);return ; } +/* +* If there is no available path and NVME_NS_SCAN_FAILED is +* set an error occurred during partition scan triggered +* by device_add_disk(), and the disk is most certainly +* not live. +*/ +if (!nvme_available_path (head) && +test_and_clear_bit (NVME_NS_SCAN_FAILED , &ns->flags)) { +dev_dbg (ns->ctrl->device,"delete gendisk for nsid %d\n" , + head->ns_id); +clear_bit (NVME_NSHEAD_DISK_LIVE , &head->flags); +del_gendisk (head->disk); +return ; + }nvme_add_ns_head_cdev (head); }
@@ -732,6 +757,7 @@ static void nvme_update_ns_ana_state(struct nvme_ana_group_desc *desc, ns->ana_grpid =le32_to_cpu (desc->grpid); ns->ana_state = desc->state;clear_bit (NVME_NS_ANA_PENDING , &ns->flags); +clear_bit (NVME_NS_SCAN_FAILED , &ns->flags);/* * nvme_mpath_set_live() will trigger I/O to the multipath path device * and in turn to this path device. However we cannot accept this I/O
drivers/nvme/host/nvme.h +1 -0
@@ -527,6 +527,7 @@ struct nvme_ns {#define NVME_NS_ANA_PENDING 2#define NVME_NS_FORCE_RO 3#define NVME_NS_READY 4 +#define NVME_NS_SCAN_FAILED 5struct cdev cdev;struct device cdev_device;
Signature · 1 line
2.35.3
Christoph Hellwig
maintainerRe: [PATCH 3/3] nvme-multipath: skip failed paths during partition scan
Quoted 23 lines · Hannes Reinecke
From: Hannes Reinecke <[email protected]>
When an I/O error is encountered during scanning (ie when the
scan_lockis held) we should avoid using this path until scanning is finished to avoid deadlocks withdevice_add_disk().
So set a new flagNVME_NS_SCAN_FAILEDif a failover happened during scanning, and skip this path innvme_available_paths().
Then we can check if that bit is set afterdevice_add_disk()returned, and remove the disk again if no available paths are found. That allows the device to be recreated via the 'rescan' sysfs attribute once no I/O errors occur anymore.Signed-off-by Hannes Reinecke <[email protected]>--- drivers/nvme/host/multipath.c | 26 ++++++++++++++++++++++++++ drivers/nvme/host/nvme.h | 1 + 2 files changed, 27 insertions(+)drivers/nvme/host/multipath.c +0
-0
drivers/nvme/host/multipath.c void nvme_failover_req(struct request *req) · line 110
Quoted 9 lines
drivers/nvme/host/multipath.c +6
-0@@ -102,6 +102,13 @@ void nvme_failover_req(struct request *req)queue_work (nvme_wq, &ns->ctrl->ana_work); } +/* +* Do not use this path during scanning +* to avoid deadlocks in device_add_disk() +*/ +if (mutex_is_locked (&ns->ctrl->scan_lock)) +set_bit (NVME_NS_SCAN_FAILED , &ns->flags);
Err, no. mutex_is_locked is never a valid way to detect the calling context - obviously someone else could also be holding it.
Sagi Grimberg
maintainerRe: [PATCH 3/3] nvme-multipath: skip failed paths during partition scan
Quoted 9 lines
Quoted 9 lines
drivers/nvme/host/multipath.c +0
-0@@ -102,6 +102,13 @@ void nvme_failover_req(struct request *req)queue_work (nvme_wq, &ns->ctrl->ana_work); }
Quoted 9 lines
drivers/nvme/host/multipath.c void nvme_failover_req(struct request *req) · line 110
Quoted 6 lines
drivers/nvme/host/multipath.c +6
-0+/* +* Do not use this path during scanning +* to avoid deadlocks in device_add_disk() +*/ +if (mutex_is_locked (&ns->ctrl->scan_lock)) +set_bit (NVME_NS_SCAN_FAILED , &ns->flags);Err, no.
mutex_is_lockedis never a valid way to detect the calling context - obviously someone else could also be holding it.
Strongly agree here.