nvme: fixup race with namespace removal
cover nvme: fixup race with namespace removal
Hi all,
nvme namespace scanning might race with controller reset, causing the system to crash in myriad ways if nvme_scan_ns_list() decides to remove a namespace while resetting is active on another thread.
The main problem here is that namespace removal is not atomic. The namespace mutex needs to be dropped before calling nvme_ns_remove(), so if nvme_ns_remove() is called while traversing the list of namespaces there always will be a race condition.
This patchset tries to fix this problem by shortening the race window and check for valid namespaces in nvme_start_queue()/nvme_stop_queue().
As usual, comments and reviews are welcome.
Hannes Reinecke (3):
nvme: do not quiesce or unquiesce invalid namespaces
nvme: shorten race window in nvme_ns_remove()
nvme-multipath: test namespace state when selecting a new path
drivers/nvme/host/core.c | 18 ++++++++++++------ drivers/nvme/host/multipath.c | 6 ++++-- 2 files changed, 16 insertions(+), 8 deletions(-)
Signature · 1 line
2.16.4
PATCH 1/3 nvme: do not quiesce or unquiesce invalid namespaces
nvme_(start,stop)_queues() might race with namespace scanning. As namespaces might be removed during scanning, but the removal from the list is not atomic, those functions might trip over namespaces which are partially deleted, causing really interesting kernel crashes.
So validate the namespaces in nvme_(start,stop)_queues().
--- drivers/nvme/host/core.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
drivers/nvme/host/core.c +8 -2
@@ -3857,8 +3857,11 @@ void nvme_stop_queues(struct nvme_ctrl *ctrl)struct nvme_ns *ns;down_read (&ctrl->namespaces_rwsem); -list_for_each_entry (ns, &ctrl->namespaces, list) +list_for_each_entry (ns, &ctrl->namespaces, list) { +if (test_bit (NVME_NS_REMOVING , &ns->flags)) +continue ;blk_mq_quiesce_queue (ns->queue); + }up_read (&ctrl->namespaces_rwsem); }EXPORT_SYMBOL_GPL (nvme_stop_queues);
@@ -3868,8 +3871,11 @@ void nvme_start_queues(struct nvme_ctrl *ctrl)struct nvme_ns *ns;down_read (&ctrl->namespaces_rwsem); -list_for_each_entry (ns, &ctrl->namespaces, list) +list_for_each_entry (ns, &ctrl->namespaces, list) { +if (test_bit (NVME_NS_REMOVING , &ns->flags)) +continue ;blk_mq_unquiesce_queue (ns->queue); + }up_read (&ctrl->namespaces_rwsem); }EXPORT_SYMBOL_GPL (nvme_start_queues);
Signature · 1 line
2.16.4
PATCH 2/3 nvme: shorten race window in nvme_ns_remove()
In reply to Hannes Reineckenvme_ns_remove() sets the 'NVME_NS_REMOVING' bit at first, but only removes the namespace from the list at the very end. This opens a rether large race window during which other processes traversing the namespaces via list_for_each() will access partially deleted namespaces, causing unpredictable results.
This patch shortens the race window by removing the namespace from the list as early as possible.
--- drivers/nvme/host/core.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
drivers/nvme/host/core.c +4 -4
@@ -3316,6 +3316,10 @@ static void nvme_ns_remove(struct nvme_ns *ns)if (test_and_set_bit (NVME_NS_REMOVING , &ns->flags))return ; +down_write (&ns->ctrl->namespaces_rwsem); +list_del_init (&ns->list); +up_write (&ns->ctrl->namespaces_rwsem); +nvme_fault_inject_fini (ns);if (ns->disk && ns->disk->flags &GENHD_FL_UP ) {del_gendisk (ns->disk);
@@ -3329,10 +3333,6 @@ static void nvme_ns_remove(struct nvme_ns *ns)nvme_mpath_clear_current_path (ns);mutex_unlock (&ns->ctrl->subsys->lock); -down_write (&ns->ctrl->namespaces_rwsem); -list_del_init (&ns->list); -up_write (&ns->ctrl->namespaces_rwsem); -synchronize_srcu (&ns->head->srcu);nvme_mpath_check_last_path (ns);nvme_put_ns (ns);
Signature · 1 line
2.16.4
PATCH 3/3 nvme-multipath: test namespace state when selecting a new path
In reply to Hannes ReineckeWhen selecting a new path we need to check if the namespace is not about to be removed.
--- drivers/nvme/host/multipath.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/nvme/host/multipath.c +4 -2
@@ -130,7 +130,8 @@ static struct nvme_ns *__nvme_find_path(struct nvme_ns_head *head, int node)list_for_each_entry_rcu (ns, &head->list, siblings) {if (ns->ctrl->state !=NVME_CTRL_LIVE || -test_bit (NVME_NS_ANA_PENDING , &ns->flags)) +test_bit (NVME_NS_ANA_PENDING , &ns->flags) || +test_bit (NVME_NS_REMOVING , &ns->flags))continue ;if (READ_ONCE (head->subsys->iopolicy) ==NVME_IOPOLICY_NUMA )
@@ -185,7 +186,8 @@ static struct nvme_ns *nvme_round_robin_path(struct nvme_ns_head *head, ns != old; ns = nvme_next_ns(head, ns)) {if (ns->ctrl->state !=NVME_CTRL_LIVE || -test_bit (NVME_NS_ANA_PENDING , &ns->flags)) +test_bit (NVME_NS_ANA_PENDING , &ns->flags) || +test_bit (NVME_NS_REMOVING , &ns->flags))continue ;if (ns->ana_state ==NVME_ANA_OPTIMIZED ) {
Signature · 1 line
2.16.4
Christoph Hellwig
maintainerPATCH 2/3 nvme: shorten race window in nvme_ns_remove()
In reply to Hannes ReineckeQuoted 9 lines · Hannes Reinecke
nvme_ns_remove()sets the 'NVME_NS_REMOVING' bit at first, but only removes the namespace from the list at the very end. This opens a rether large race window during which other processes traversing the namespaces vialist_for_each()will access partially deleted namespaces, causing unpredictable results.
This patch shortens the race window by removing the namespace from the list as early as possible.Signed-off-by Hannes Reinecke <hare at suse.com>
With this patch I don't see any reason for your patch 1, given that the window between setting the REMOVING bit and removing from the list is so tiny.
That being said I think we really should also remove the namespace from ->siblings and clear the current path ASAP as well, which might remove the need for patch 3 as well.
Keith Busch
maintainerPATCH 2/3 nvme: shorten race window in nvme_ns_remove()
In reply to Hannes Reineckedrivers/nvme/host/core.c static void nvme_ns_remove(struct nvme_ns *ns) · line 3325
Quoted 11 lines · Hannes Reinecke
drivers/nvme/host/core.c +4
-0@@ -3316,6 +3316,10 @@ static void nvme_ns_remove(struct nvme_ns *ns)if (test_and_set_bit (NVME_NS_REMOVING , &ns->flags))return ; +down_write (&ns->ctrl->namespaces_rwsem); +list_del_init (&ns->list); +up_write (&ns->ctrl->namespaces_rwsem); +nvme_fault_inject_fini (ns);if (ns->disk && ns->disk->flags &GENHD_FL_UP ) {del_gendisk (ns->disk);
I think I understand the problem you've described, but I'm a little concerned with the early removal. This will call blk_cleanup_queue after the namespace is removed from our lists, and that does a blocking queue_freeze. If commands happen to be dispatched to that namespace, and the controller for some reason stops responding, we may not be able to recover during reset when the namespace queue is removed from the controller list. That's at least problematic if we have to go with the per-queue tag iterators that Jianchao proposed.
I may be wrong, or there might be another way to address this. It's a bit late where I'm at the moment, I'll revisit again with fresh eyes tomorrow.
PATCH 2/3 nvme: shorten race window in nvme_ns_remove()
In reply to Christoph HellwigQuoted 9 lines · Christoph Hellwig
Quoted 8 lines · Hannes Reinecke
nvme_ns_remove()sets the 'NVME_NS_REMOVING' bit at first, but only removes the namespace from the list at the very end. This opens a rether large race window during which other processes traversing the namespaces vialist_for_each()will access partially deleted namespaces, causing unpredictable results.
This patch shortens the race window by removing the namespace from the list as early as possible.
Quoted 6 lines
Quoted 1 line
Signed-off-by Hannes Reinecke <hare at suse.com>With this patch I don't see any reason for your patch 1, given that the window between setting the REMOVING bit and removing from the list is so tiny.
Tiny doesn't equal non-existent.
I'd rather make it explicit to make life easier for others having to debug issues here.
Quoted 4 lines
That being said I think we really should also remove the namespace from ->siblings and clear the current path ASAP as well, which might remove the need for patch 3 as well.
Okay, can do that.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke zSeries & Storage hare at suse.com +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: F. Imend?rffer, J. Smithard, D. Upmanyu, G. Norton HRB 21284 (AG N?rnberg)
PATCH 2/3 nvme: shorten race window in nvme_ns_remove()
In reply to Keith BuschQuoted 5 lines · Keith Busch
Quoted 4 lines · Hannes Reinecke
drivers/nvme/host/core.c +0
-0@@ -3316,6 +3316,10 @@ static void nvme_ns_remove(struct nvme_ns *ns)if (test_and_set_bit (NVME_NS_REMOVING , &ns->flags))return ;
Quoted 17 lines
drivers/nvme/host/core.c static void nvme_ns_remove(struct nvme_ns *ns) · line 3325
Quoted 7 lines
drivers/nvme/host/core.c +4
-0+down_write (&ns->ctrl->namespaces_rwsem); +list_del_init (&ns->list); +up_write (&ns->ctrl->namespaces_rwsem); +nvme_fault_inject_fini (ns);if (ns->disk && ns->disk->flags &GENHD_FL_UP ) {del_gendisk (ns->disk);I think I understand the problem you've described, but I'm a little concerned with the early removal. This will call
blk_cleanup_queueafter the namespace is removed from our lists, and that does a blockingqueue_freeze. If commands happen to be dispatched to that namespace, and the controller for some reason stops responding, we may not be able to recover during reset when the namespace queue is removed from the controller list. That's at least problematic if we have to go with the per-queue tag iterators that Jianchao proposed.
Hmm. I sort of see where you coming from.
But I was slightly worried about all the other callers where we iterate the namespace list; one would need to to a _really_ careful audit which are safe to traverse even with namespace removal running; some have to, others must not, and some simply don't care.
Hence this approach to make things easier.
Quoted 3 lines
I may be wrong, or there might be another way to address this. It's a bit late where I'm at the moment, I'll revisit again with fresh eyes tomorrow.
Thanks.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke zSeries & Storage hare at suse.com +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: F. Imend?rffer, J. Smithard, D. Upmanyu, G. Norton HRB 21284 (AG N?rnberg)
PATCH 2/3 nvme: shorten race window in nvme_ns_remove()
Quoted 22 lines · Hannes Reinecke
Quoted 12 lines · Keith Busch
Review static void nvme_ns_remove(struct nvme_ns *ns) · line 3324
Quoted 10 lines · Hannes Reinecke
(quoted diff) +3
-0@@ -3316,6 +3316,10 @@ static void nvme_ns_remove(struct nvme_ns *ns) ????? if (test_and_set_bit(NVME_NS_REMOVING, &ns->flags)) ????????? return; ? +??? down_write(&ns->ctrl->namespaces_rwsem); +??? list_del_init(&ns->list); +??? up_write(&ns->ctrl->namespaces_rwsem); + ????? nvme_fault_inject_fini(ns); ????? if (ns->disk && ns->disk->flags & GENHD_FL_UP) { ????????? del_gendisk(ns->disk);Quoted 9 lines
I think I understand the problem you've described, but I'm a little concerned with the early removal. This will call
blk_cleanup_queueafter the namespace is removed from our lists, and that does a blockingqueue_freeze. If commands happen to be dispatched to that namespace, and the controller for some reason stops responding, we may not be able to recover during reset when the namespace queue is removed from the controller list. That's at least problematic if we have to go with the per-queue tag iterators that Jianchao proposed.
Quoted 6 lines
Hmm. I sort of see where you coming from.
But I was slightly worried about all the other callers where we iterate the namespace list; one would need to to a _really_ careful audit which are safe to traverse even with namespace removal running; some have to, others must not, and some simply don't care.
Hence this approach to make things easier.
I like and agree with Hannes's later points.
But, Keiths statements make me nervous. With Hannes's patches, I could easily see the case where the ns->queue was quiesced when ns_remove is called, which I think causes issues with blk_freeze_queue. We would need to have ns_remove, after unlinking, to release the queue. is this right ?
-- james
PATCH 2/3 nvme: shorten race window in nvme_ns_remove()
Quoted 30 lines · James Smart
Quoted 22 lines · Hannes Reinecke
Quoted 12 lines · Keith Busch
Review static void nvme_ns_remove(struct nvme_ns *ns) · line 3324
Quoted 10 lines · Hannes Reinecke
(quoted diff) +3
-0@@ -3316,6 +3316,10 @@ static void nvme_ns_remove(struct nvme_ns *ns) ????? if (test_and_set_bit(NVME_NS_REMOVING, &ns->flags)) ????????? return; ? +??? down_write(&ns->ctrl->namespaces_rwsem); +??? list_del_init(&ns->list); +??? up_write(&ns->ctrl->namespaces_rwsem); + ????? nvme_fault_inject_fini(ns); ????? if (ns->disk && ns->disk->flags & GENHD_FL_UP) { ????????? del_gendisk(ns->disk);Quoted 9 lines
I think I understand the problem you've described, but I'm a little concerned with the early removal. This will call
blk_cleanup_queueafter the namespace is removed from our lists, and that does a blockingqueue_freeze. If commands happen to be dispatched to that namespace, and the controller for some reason stops responding, we may not be able to recover during reset when the namespace queue is removed from the controller list. That's at least problematic if we have to go with the per-queue tag iterators that Jianchao proposed.Quoted 6 lines
Hmm. I sort of see where you coming from.
But I was slightly worried about all the other callers where we iterate the namespace list; one would need to to a _really_ careful audit which are safe to traverse even with namespace removal running; some have to, others must not, and some simply don't care.
Hence this approach to make things easier.
Quoted 7 lines
I like and agree with Hannes's later points.
But, Keiths statements make me nervous. With Hannes's patches, I could easily see the case where the
ns->queuewas quiesced whenns_removeis called, which I think causes issues withblk_freeze_queue. We would need to havens_remove, after unlinking, to release the queue. is this right ?
Hmm. But maybe it's as simple as this:
drivers/nvme/host/core.c +1 -0
@@ -131,6 +131,7 @@ int nvme_reset_ctrl(struct nvme_ctrl *ctrl) {if (!nvme_change_ctrl_state (ctrl,NVME_CTRL_RESETTING ))return -EBUSY ; +flush_work (&ctrl->scan_work);if (!queue_work (nvme_reset_wq, &ctrl->reset_work))return -EBUSY ;
James? Keith?
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking hare at suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah HRB 21284 (AG N?rnberg)
PATCH 2/3 nvme: shorten race window in nvme_ns_remove()
Quoted 46 lines · Hannes Reinecke
Quoted 30 lines · James Smart
Quoted 22 lines · Hannes Reinecke
Quoted 12 lines · Keith Busch
Review static void nvme_ns_remove(struct nvme_ns *ns) · line 3324
Quoted 10 lines · Hannes Reinecke
(quoted diff) +3
-0@@ -3316,6 +3316,10 @@ static void nvme_ns_remove(struct nvme_ns *ns) if (test_and_set_bit(NVME_NS_REMOVING, &ns->flags)) return; + down_write(&ns->ctrl->namespaces_rwsem); + list_del_init(&ns->list); + up_write(&ns->ctrl->namespaces_rwsem); + nvme_fault_inject_fini(ns); if (ns->disk && ns->disk->flags & GENHD_FL_UP) { del_gendisk(ns->disk);Quoted 9 lines
I think I understand the problem you've described, but I'm a little concerned with the early removal. This will call
blk_cleanup_queueafter the namespace is removed from our lists, and that does a blockingqueue_freeze. If commands happen to be dispatched to that namespace, and the controller for some reason stops responding, we may not be able to recover during reset when the namespace queue is removed from the controller list. That's at least problematic if we have to go with the per-queue tag iterators that Jianchao proposed.Quoted 6 lines
Hmm. I sort of see where you coming from.
But I was slightly worried about all the other callers where we iterate the namespace list; one would need to to a _really_ careful audit which are safe to traverse even with namespace removal running; some have to, others must not, and some simply don't care.
Hence this approach to make things easier.Quoted 7 lines
I like and agree with Hannes's later points.
But, Keiths statements make me nervous. With Hannes's patches, I could easily see the case where the
ns->queuewas quiesced whenns_removeis called, which I think causes issues withblk_freeze_queue. We would need to havens_remove, after unlinking, to release the queue. is this right ?Hmm. But maybe it's as simple as this:
drivers/nvme/host/core.c +0
-0
drivers/nvme/host/core.c int nvme_reset_ctrl(struct nvme_ctrl *ctrl) · line 137
Quoted 7 lines
drivers/nvme/host/core.c +1
-0@@ -131,6 +131,7 @@ int nvme_reset_ctrl(struct nvme_ctrl *ctrl) {if (!nvme_change_ctrl_state (ctrl,NVME_CTRL_RESETTING ))return -EBUSY ; +flush_work (&ctrl->scan_work);if (!queue_work (nvme_reset_wq, &ctrl->reset_work))return -EBUSY ;
This way might cause deadlock because there is sync IO in scan work context.
thanks,
Ming
PATCH 2/3 nvme: shorten race window in nvme_ns_remove()
Quoted 42 lines · Ming Lei
Quoted 41 lines · Hannes Reinecke
Quoted 30 lines · James Smart
Quoted 22 lines · Hannes Reinecke
Quoted 12 lines · Keith Busch
Review static void nvme_ns_remove(struct nvme_ns *ns) · line 3324
Quoted 10 lines · Hannes Reinecke
(quoted diff) +3
-0@@ -3316,6 +3316,10 @@ static void nvme_ns_remove(struct nvme_ns *ns) if (test_and_set_bit(NVME_NS_REMOVING, &ns->flags)) return; + down_write(&ns->ctrl->namespaces_rwsem); + list_del_init(&ns->list); + up_write(&ns->ctrl->namespaces_rwsem); + nvme_fault_inject_fini(ns); if (ns->disk && ns->disk->flags & GENHD_FL_UP) { del_gendisk(ns->disk);Quoted 9 lines
I think I understand the problem you've described, but I'm a little concerned with the early removal. This will call
blk_cleanup_queueafter the namespace is removed from our lists, and that does a blockingqueue_freeze. If commands happen to be dispatched to that namespace, and the controller for some reason stops responding, we may not be able to recover during reset when the namespace queue is removed from the controller list. That's at least problematic if we have to go with the per-queue tag iterators that Jianchao proposed.Quoted 6 lines
Hmm. I sort of see where you coming from.
But I was slightly worried about all the other callers where we iterate the namespace list; one would need to to a _really_ careful audit which are safe to traverse even with namespace removal running; some have to, others must not, and some simply don't care.
Hence this approach to make things easier.Quoted 7 lines
I like and agree with Hannes's later points.
But, Keiths statements make me nervous. With Hannes's patches, I could easily see the case where the
ns->queuewas quiesced whenns_removeis called, which I think causes issues withblk_freeze_queue. We would need to havens_remove, after unlinking, to release the queue. is this right ?Hmm. But maybe it's as simple as this:
Quoted 16 lines
drivers/nvme/host/core.c int nvme_reset_ctrl(struct nvme_ctrl *ctrl) · line 137
Quoted 12 lines
drivers/nvme/host/core.c +1
-0@@ -131,6 +131,7 @@ int nvme_reset_ctrl(struct nvme_ctrl *ctrl) {if (!nvme_change_ctrl_state (ctrl,NVME_CTRL_RESETTING ))return -EBUSY ; +flush_work (&ctrl->scan_work);if (!queue_work (nvme_reset_wq, &ctrl->reset_work))return -EBUSY ;This way might cause deadlock because there is sync IO in scan work context.
But I sort of hoped that I/O would be aborted when the controller is resetting, which should cause the sync I/O to complete, right?
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke zSeries & Storage hare at suse.com +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: F. Imend?rffer, J. Smithard, D. Upmanyu, G. Norton HRB 21284 (AG N?rnberg)
PATCH 2/3 nvme: shorten race window in nvme_ns_remove()
Quoted 4 lines · Hannes Reinecke
Quoted 3 lines · Ming Lei
Quoted 2 lines · Hannes Reinecke
On 3/27/19 7:50 PM, James Smart wrote:
[ .. ]
Quoted 25 lines
Quoted 9 lines
Quoted 9 lines
Quoted 7 lines
But, Keiths statements make me nervous. With Hannes's patches, I could easily see the case where the
ns->queuewas quiesced whenns_removeis called, which I think causes issues withblk_freeze_queue. We would need
to havens_remove, after unlinking, to release the queue. is this right ?Hmm. But maybe it's as simple as this:
Quoted 16 lines
drivers/nvme/host/core.c int nvme_reset_ctrl(struct nvme_ctrl *ctrl) · line 137
Quoted 12 lines
drivers/nvme/host/core.c +1
-0@@ -131,6 +131,7 @@ int nvme_reset_ctrl(struct nvme_ctrl *ctrl) ?? { ????????? if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING)) ????????????????? return -EBUSY; +?????? flush_work(&ctrl->scan_work); ????????? if (!queue_work(nvme_reset_wq, &ctrl->reset_work)) ????????????????? return -EBUSY;This way might cause deadlock because there is sync IO in scan work context.
Quoted 3 lines
But I sort of hoped that I/O would be aborted when the controller is resetting, which should cause the sync I/O to complete, right?
But indeed, you're right (of course).
This should be better:
drivers/nvme/host/core.c +1 -0
@@ -3452,6 +3452,7 @@ void nvme_stop_ctrl(struct nvme_ctrl *ctrl) {nvme_mpath_stop (ctrl);nvme_stop_keep_alive (ctrl); +flush_work (&ctrl->scan_work);flush_work (&ctrl->async_event_work);cancel_work_sync (&ctrl->fw_act_work); }
drivers/nvme/host/rdma.c +0 -0
@@ -1761,8 +1761,8 @@ static void nvme_rdma_reset_ctrl_work(struct
work_struct *work)
int ret;
bool changed;- nvme_stop_ctrl(&ctrl->ctrl);
nvme_rdma_shutdown_ctrl(ctrl, false);+ nvme_stop_ctrl(&ctrl->ctrl);
if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_CONNECTING)) {
/* state change failure should never happen */(We need to twiddle the order on RDMA, as we'd need to abort the I/O first before we can flush the scan_work element).
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking hare at suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah HRB 21284 (AG N?rnberg)
PATCH 2/3 nvme: shorten race window in nvme_ns_remove()
In reply to Hannes ReineckeQuoted 55 lines · Hannes Reinecke
Quoted 32 lines · James Smart
Quoted 24 lines · Hannes Reinecke
Quoted 14 lines · Keith Busch
Review static void nvme_ns_remove(struct nvme_ns *ns) · line 3324
Quoted 10 lines · Hannes Reinecke
(quoted diff) +3
-0@@ -3316,6 +3316,10 @@ static void nvme_ns_remove(struct nvme_ns *ns) ????? if (test_and_set_bit(NVME_NS_REMOVING, &ns->flags)) ????????? return; ? +??? down_write(&ns->ctrl->namespaces_rwsem); +??? list_del_init(&ns->list); +??? up_write(&ns->ctrl->namespaces_rwsem); + ????? nvme_fault_inject_fini(ns); ????? if (ns->disk && ns->disk->flags & GENHD_FL_UP) { ????????? del_gendisk(ns->disk);I think I understand the problem you've described, but I'm a little concerned with the early removal. This will call
blk_cleanup_queueQuoted 9 lines
after the namespace is removed from our lists, and that does a blocking
queue_freeze. If commands happen to be dispatched to that namespace, and the controller for some reason stops responding, we may not be able
to recover during reset when the namespace queue is removed from the controller list. That's at least problematic if we have to go with the per-queue tag iterators that Jianchao proposed.Quoted 6 lines
Hmm. I sort of see where you coming from.
But I was slightly worried about all the other callers where we iterate the namespace list; one would need to to a _really_ careful audit which are safe to traverse even with namespace removal running; some have to, others must not, and some simply don't care.
Hence this approach to make things easier.Quoted 8 lines
I like and agree with Hannes's later points.
But, Keiths statements make me nervous. With Hannes's patches, I could easily see the case where the
ns->queuewas quiesced whenns_removeis called, which I think causes issues withblk_freeze_queue. We would need to havens_remove, after unlinking, to release the queue. is this right ?Hmm. But maybe it's as simple as this:
drivers/nvme/host/core.c +1
-0@@ -131,6 +131,7 @@ int nvme_reset_ctrl(struct nvme_ctrl *ctrl) ?{ ??????? if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING)) ??????????????? return -EBUSY; +?????? flush_work(&ctrl->scan_work); ??????? if (!queue_work(nvme_reset_wq, &ctrl->reset_work)) ??????????????? return -EBUSY;
Quoted 2 lines
James? Keith?
No...
As mentioned in the other thread. If the reset processing is held off, then nothing will kill the io that the scan thread has issued and is pending on the link. In the best case, you must wait for the command timeout to occur, which will want to re-invoke the reset path, which you've noop'd here, so I'm not fully sure it will clean up the io.? And, as reset hasn't run, the io queues on this controller have yet to be quiesced so you're going to start seeing filesystem errors.? The other headache - even if that 1st io is aborted and returned, just because the 1 scan io errored, it doesn't stop the scan thread. The thread is coded to keep looping on new namespaces so it'll attempt to send another/many commands.? Which means the new io's will fall into the pass through the check_ready routines as they will fall into the path that is only prevented by the connection state being nil, which would not have been set because the reset path was held off.?? Minimally a new change has to be put there.?? This is not something that can be solved by scheduling work elements at the transport level. The transport has no idea what ns's are using its io request queue what states they are in. It's a ns state issue.
-- james
PATCH 2/3 nvme: shorten race window in nvme_ns_remove()
In reply to Hannes ReineckeQuoted 55 lines · Hannes Reinecke
Quoted 4 lines · Hannes Reinecke
Quoted 3 lines · Ming Lei
Quoted 2 lines · Hannes Reinecke
On 3/27/19 7:50 PM, James Smart wrote:
[ .. ]
Quoted 25 lines
Quoted 9 lines
Quoted 9 lines
Quoted 7 lines
But, Keiths statements make me nervous. With Hannes's patches, I could easily see the case where the
ns->queuewas quiesced whenns_removeis called, which I think causes issues withblk_freeze_queue. We would need
to havens_remove, after unlinking, to release the queue. is this right ?Hmm. But maybe it's as simple as this:
Quoted 16 lines
drivers/nvme/host/core.c int nvme_reset_ctrl(struct nvme_ctrl *ctrl) · line 137
Quoted 12 lines
drivers/nvme/host/core.c +1
-0@@ -131,6 +131,7 @@ int nvme_reset_ctrl(struct nvme_ctrl *ctrl) {if (!nvme_change_ctrl_state (ctrl,NVME_CTRL_RESETTING ))return -EBUSY ; +flush_work (&ctrl->scan_work);if (!queue_work (nvme_reset_wq, &ctrl->reset_work))return -EBUSY ;This way might cause deadlock because there is sync IO in scan work context.
Quoted 3 lines
But I sort of hoped that I/O would be aborted when the controller is resetting, which should cause the sync I/O to complete, right?
But indeed, you're right (of course).
This should be better:
drivers/nvme/host/core.c +1
-0@@ -3452,6 +3452,7 @@ void nvme_stop_ctrl(struct nvme_ctrl *ctrl) {nvme_mpath_stop (ctrl);nvme_stop_keep_alive (ctrl); +flush_work (&ctrl->scan_work);flush_work (&ctrl->async_event_work);cancel_work_sync (&ctrl->fw_act_work); }drivers/nvme/host/rdma.c +0
-0
drivers/nvme/host/rdma.c static void nvme_rdma_reset_ctrl_work(struct · line 1766
Quoted 7 lines
drivers/nvme/host/rdma.c +1
-1@@ -1761,8 +1761,8 @@ static void nvme_rdma_reset_ctrl_work(struct work_struct *work)int ret;bool changed; -nvme_stop_ctrl (&ctrl->ctrl);nvme_rdma_shutdown_ctrl (ctrl, false); +nvme_stop_ctrl (&ctrl->ctrl);
After the controller is shutdown, no any IO can be done at all, then flush_work(&ctrl->scan_work) will wait for ever.
Thanks,
Ming Lei
PATCH 2/3 nvme: shorten race window in nvme_ns_remove()
In reply to Hannes ReineckeQuoted 67 lines · Hannes Reinecke
Quoted 4 lines · Hannes Reinecke
Quoted 3 lines · Ming Lei
Quoted 2 lines · Hannes Reinecke
On 3/27/19 7:50 PM, James Smart wrote:
[ .. ]
Quoted 27 lines
Quoted 11 lines
Quoted 11 lines
Quoted 9 lines
But, Keiths statements make me nervous. With Hannes's patches, I could
easily see the case where thens->queuewas quiesced whenns_removeis
called, which I think causes issues withblk_freeze_queue. We would need
to havens_remove, after unlinking, to release the queue. is this right ?Hmm. But maybe it's as simple as this:
Quoted 16 lines
drivers/nvme/host/core.c int nvme_reset_ctrl(struct nvme_ctrl *ctrl) · line 137
Quoted 12 lines
drivers/nvme/host/core.c +1
-0@@ -131,6 +131,7 @@ int nvme_reset_ctrl(struct nvme_ctrl *ctrl) ?? { ????????? if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING)) ????????????????? return -EBUSY; +?????? flush_work(&ctrl->scan_work); ????????? if (!queue_work(nvme_reset_wq, &ctrl->reset_work)) ????????????????? return -EBUSY;This way might cause deadlock because there is sync IO in scan work context.
Quoted 3 lines
But I sort of hoped that I/O would be aborted when the controller is resetting, which should cause the sync I/O to complete, right?
But indeed, you're right (of course).
This should be better:
drivers/nvme/host/core.c +1
-0@@ -3452,6 +3452,7 @@ void nvme_stop_ctrl(struct nvme_ctrl *ctrl) ?{ ??????? nvme_mpath_stop(ctrl); ??????? nvme_stop_keep_alive(ctrl); +?????? flush_work(&ctrl->scan_work); ??????? flush_work(&ctrl->async_event_work); ??????? cancel_work_sync(&ctrl->fw_act_work); ?}drivers/nvme/host/rdma.c +1
-1@@ -1761,8 +1761,8 @@ static void nvme_rdma_reset_ctrl_work(struct work_struct *work) ??????? int ret; ??????? bool changed; -?????? nvme_stop_ctrl(&ctrl->ctrl); ??????? nvme_rdma_shutdown_ctrl(ctrl, false); +?????? nvme_stop_ctrl(&ctrl->ctrl); ??????? if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_CONNECTING)) {??????????????? /* state change failure should never happen */
Quoted 5 lines
(We need to twiddle the order on RDMA, as we'd need to abort the I/O first before we can flush the
scan_workelement).
Cheers,Hannes
Looking at the flows - it looks like it should be ok. You're proving me wrong. Like changes need to be done in the other transports and they have to be audited for the same flows or the check_ready checks won't work right to continue to bounce ios so the flush can exit. I'm not sure having some of the async flows continue to run when doing the aborts as it may have side effects. But, I expect the issues aren't new. Simple change, but any change in this area can be dramatic.
It still bothers me how "happy path" the ns_scan path is. That the "validate" tears down the namespace on any error, even on a short connectivity disruption, shouldn't be happening. Deleting the namespace can have harsh consequences.
-- james
PATCH 2/3 nvme: shorten race window in nvme_ns_remove()
In reply to James SmartQuoted 72 lines · James Smart
Quoted 55 lines · Hannes Reinecke
Quoted 32 lines · James Smart
Quoted 24 lines · Hannes Reinecke
Quoted 14 lines · Keith Busch
Review static void nvme_ns_remove(struct nvme_ns *ns) · line 3324
Quoted 10 lines · Hannes Reinecke
(quoted diff) +3
-0@@ -3316,6 +3316,10 @@ static void nvme_ns_remove(struct nvme_ns *ns) ????? if (test_and_set_bit(NVME_NS_REMOVING, &ns->flags)) ????????? return; ? +??? down_write(&ns->ctrl->namespaces_rwsem); +??? list_del_init(&ns->list); +??? up_write(&ns->ctrl->namespaces_rwsem); + ????? nvme_fault_inject_fini(ns); ????? if (ns->disk && ns->disk->flags & GENHD_FL_UP) { ????????? del_gendisk(ns->disk);I think I understand the problem you've described, but I'm a little concerned with the early removal. This will call
blk_cleanup_queueQuoted 9 lines
after the namespace is removed from our lists, and that does a blocking
queue_freeze. If commands happen to be dispatched to that namespace, and the controller for some reason stops responding, we may not be able
to recover during reset when the namespace queue is removed from the controller list. That's at least problematic if we have to go with the per-queue tag iterators that Jianchao proposed.Quoted 6 lines
Hmm. I sort of see where you coming from.
But I was slightly worried about all the other callers where we iterate the namespace list; one would need to to a _really_ careful audit which are safe to traverse even with namespace removal running; some have to, others must not, and some simply don't care.
Hence this approach to make things easier.Quoted 8 lines
I like and agree with Hannes's later points.
But, Keiths statements make me nervous. With Hannes's patches, I could easily see the case where the
ns->queuewas quiesced whenns_removeis called, which I think causes issues withblk_freeze_queue. We would need to havens_remove, after unlinking, to release the queue. is this right ?Hmm. But maybe it's as simple as this:
drivers/nvme/host/core.c +1
-0@@ -131,6 +131,7 @@ int nvme_reset_ctrl(struct nvme_ctrl *ctrl) ?{ ??????? if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING)) ??????????????? return -EBUSY; +?????? flush_work(&ctrl->scan_work); ??????? if (!queue_work(nvme_reset_wq, &ctrl->reset_work)) ??????????????? return -EBUSY;Quoted 2 lines
James? Keith?
No...
As mentioned in the other thread. If the reset processing is held off, then nothing will kill the io that the scan thread has issued and is pending on the link. In the best case, you must wait for the command timeout to occur, which will want to re-invoke the reset path, which you've noop'd here, so I'm not fully sure it will clean up the io.? And, as reset hasn't run, the io queues on this controller have yet to be quiesced so you're going to start seeing filesystem errors.? The other headache - even if that 1st io is aborted and returned, just because the 1 scan io errored, it doesn't stop the scan thread. The thread is coded to keep looping on new namespaces so it'll attempt to send another/many
Quoted 9 lines
commands.? Which means the new io's will fall into the pass through the
check_readyroutines as they will fall into the path that is only prevented by the connection state being nil, which would not have been set because the reset path was held off.?? Minimally a new change has to be put there.?? This is not something that can be solved by scheduling work elements at the transport level. The transport has no idea what ns's are using its io request queue what states they are in. It's a ns state issue.
Please do check with my reply to Ming Lei.
We should be able to flush scan_work from nvme_stop_ctrl(); FC already stops all I/O before nvme_stop_ctrl() (so any sync I/O will be completed), and for RDMA we should be able to reshuffle nvme_stop_ctrl() and nvme_rdma_stop_io() to get to the same behaviour.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke zSeries & Storage hare at suse.com +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: F. Imend?rffer, J. Smithard, D. Upmanyu, G. Norton HRB 21284 (AG N?rnberg)
PATCH 2/3 nvme: shorten race window in nvme_ns_remove()
In reply to Ming LeiQuoted 57 lines · Ming Lei
Quoted 56 lines · Hannes Reinecke
Quoted 3 lines · Hannes Reinecke
Quoted 2 lines · Ming Lei
Quoted 1 line · Hannes Reinecke
On 3/27/19 7:50 PM, James Smart wrote:
[ .. ]
Quoted 24 lines
Quoted 9 lines
Quoted 9 lines
Quoted 7 lines
But, Keiths statements make me nervous. With Hannes's patches, I could easily see the case where the
ns->queuewas quiesced whenns_removeis called, which I think causes issues withblk_freeze_queue. We would need
to havens_remove, after unlinking, to release the queue. is this right ?Hmm. But maybe it's as simple as this:
Quoted 15 lines
drivers/nvme/host/core.c int nvme_reset_ctrl(struct nvme_ctrl *ctrl) · line 137
Quoted 12 lines
drivers/nvme/host/core.c +1
-0@@ -131,6 +131,7 @@ int nvme_reset_ctrl(struct nvme_ctrl *ctrl) {if (!nvme_change_ctrl_state (ctrl,NVME_CTRL_RESETTING ))return -EBUSY ; +flush_work (&ctrl->scan_work);if (!queue_work (nvme_reset_wq, &ctrl->reset_work))return -EBUSY ;This way might cause deadlock because there is sync IO in scan work context.
Quoted 3 lines
But I sort of hoped that I/O would be aborted when the controller is resetting, which should cause the sync I/O to complete, right?
But indeed, you're right (of course).
This should be better:
drivers/nvme/host/core.c +1
-0@@ -3452,6 +3452,7 @@ void nvme_stop_ctrl(struct nvme_ctrl *ctrl) {nvme_mpath_stop (ctrl);nvme_stop_keep_alive (ctrl); +flush_work (&ctrl->scan_work);flush_work (&ctrl->async_event_work);cancel_work_sync (&ctrl->fw_act_work); }drivers/nvme/host/rdma.c +0
-0@@ -1761,8 +1761,8 @@ static void nvme_rdma_reset_ctrl_work(struct work_struct *work)int ret;bool changed;
Quoted 8 lines
drivers/nvme/host/rdma.c static void nvme_rdma_reset_ctrl_work(struct · line 1766
Quoted 3 lines
drivers/nvme/host/rdma.c +1
-1-nvme_stop_ctrl (&ctrl->ctrl);nvme_rdma_shutdown_ctrl (ctrl, false); +nvme_stop_ctrl (&ctrl->ctrl);After the controller is shutdown, no any IO can be done at all, then
flush_work(&ctrl->scan_work)will wait for ever.Thanks,
Ming Lei
There's no guarantee io could have been done even prior to the shutdown
- it would likely be accepted and be waiting for completion. Key is the shutdown has to abort/complete anything outstanding and set the state/flags to bounce anything that arrives after. How its bounced depends on the type of io - it may be a resource busy or an outright error. With scan io (admin cmds), it should be a hard error.? Thus scan_work thread will see a stream of io errors from that point on.
-- james
PATCH 2/3 nvme: shorten race window in nvme_ns_remove()
In reply to James SmartQuoted 84 lines · James Smart
Quoted 67 lines · Hannes Reinecke
Quoted 4 lines · Hannes Reinecke
Quoted 3 lines · Ming Lei
Quoted 2 lines · Hannes Reinecke
On 3/27/19 7:50 PM, James Smart wrote:
[ .. ]
Quoted 27 lines
Quoted 11 lines
Quoted 11 lines
Quoted 9 lines
But, Keiths statements make me nervous. With Hannes's patches, I could
easily see the case where thens->queuewas quiesced whenns_removeis
called, which I think causes issues withblk_freeze_queue. We would need
to havens_remove, after unlinking, to release the queue. is this right ?Hmm. But maybe it's as simple as this:
Quoted 16 lines
drivers/nvme/host/core.c int nvme_reset_ctrl(struct nvme_ctrl *ctrl) · line 137
Quoted 12 lines
drivers/nvme/host/core.c +1
-0@@ -131,6 +131,7 @@ int nvme_reset_ctrl(struct nvme_ctrl *ctrl) ?? { ????????? if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING)) ????????????????? return -EBUSY; +?????? flush_work(&ctrl->scan_work); ????????? if (!queue_work(nvme_reset_wq, &ctrl->reset_work)) ????????????????? return -EBUSY;This way might cause deadlock because there is sync IO in scan work context.
Quoted 3 lines
But I sort of hoped that I/O would be aborted when the controller is resetting, which should cause the sync I/O to complete, right?
But indeed, you're right (of course).
This should be better:
drivers/nvme/host/core.c +1
-0@@ -3452,6 +3452,7 @@ void nvme_stop_ctrl(struct nvme_ctrl *ctrl) ?{ ??????? nvme_mpath_stop(ctrl); ??????? nvme_stop_keep_alive(ctrl); +?????? flush_work(&ctrl->scan_work); ??????? flush_work(&ctrl->async_event_work); ??????? cancel_work_sync(&ctrl->fw_act_work); ?}drivers/nvme/host/rdma.c +1
-1@@ -1761,8 +1761,8 @@ static void nvme_rdma_reset_ctrl_work(struct work_struct *work) ??????? int ret; ??????? bool changed; -?????? nvme_stop_ctrl(&ctrl->ctrl); ??????? nvme_rdma_shutdown_ctrl(ctrl, false); +?????? nvme_stop_ctrl(&ctrl->ctrl); ??????? if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_CONNECTING)) {??????????????? /* state change failure should never happen */
Quoted 5 lines
(We need to twiddle the order on RDMA, as we'd need to abort the I/O first before we can flush the
scan_workelement).
Cheers,Hannes
Looking at the flows - it looks like it should be ok. You're proving me wrong. Like changes need to be done in the other transports and they have to be audited for the same flows or the
check_readychecks won't work right to continue to bounce ios so the flush can exit. I'm not sure having some of the async flows continue to run when doing the aborts as it may have side effects. But, I expect the issues aren't new. Simple change, but any change in this area can be dramatic.
Quoted 5 lines
It still bothers me how "happy path" the
ns_scanpath is. That the "validate" tears down the namespace on any error, even on a short connectivity disruption, shouldn't be happening. Deleting the namespace can have harsh consequences.
Oh, I'm fully with you there.
It only took me some months to properly debug this issue. Personally, I'd mark the namespace as 'faulty', and requeue I/O for these cases (much like we do already when an error is returned). And then we can tear down namespaces after controller reset as we then have a stable state.
But that'll be quite a change to the flow of control, and I guess the powers that be have some opinion here.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking hare at suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah HRB 21284 (AG N?rnberg)
PATCH 2/3 nvme: shorten race window in nvme_ns_remove()
Quoted 98 lines · Hannes Reinecke
Quoted 85 lines · James Smart
Quoted 68 lines · Hannes Reinecke
Quoted 4 lines · Hannes Reinecke
Quoted 3 lines · Ming Lei
Quoted 2 lines · Hannes Reinecke
On 3/27/19 7:50 PM, James Smart wrote:
[ .. ]
Quoted 27 lines
Quoted 11 lines
Quoted 11 lines
Quoted 9 lines
But, Keiths statements make me nervous. With Hannes's patches, I could
easily see the case where thens->queuewas quiesced whenns_removeis
called, which I think causes issues withblk_freeze_queue. We would need
to havens_remove, after unlinking, to release the queue. is this right ?Hmm. But maybe it's as simple as this:
Quoted 16 lines
drivers/nvme/host/core.c int nvme_reset_ctrl(struct nvme_ctrl *ctrl) · line 137
Quoted 12 lines
drivers/nvme/host/core.c +1
-0@@ -131,6 +131,7 @@ int nvme_reset_ctrl(struct nvme_ctrl *ctrl) ?? { ????????? if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING)) ????????????????? return -EBUSY; +?????? flush_work(&ctrl->scan_work); ????????? if (!queue_work(nvme_reset_wq, &ctrl->reset_work)) ????????????????? return -EBUSY;This way might cause deadlock because there is sync IO in scan work context.
Quoted 3 lines
But I sort of hoped that I/O would be aborted when the controller is resetting, which should cause the sync I/O to complete, right?
But indeed, you're right (of course).
This should be better:
drivers/nvme/host/core.c +1
-0@@ -3452,6 +3452,7 @@ void nvme_stop_ctrl(struct nvme_ctrl *ctrl) ?{ ??????? nvme_mpath_stop(ctrl); ??????? nvme_stop_keep_alive(ctrl); +?????? flush_work(&ctrl->scan_work); ??????? flush_work(&ctrl->async_event_work); ??????? cancel_work_sync(&ctrl->fw_act_work); ?}drivers/nvme/host/rdma.c +1
-1@@ -1761,8 +1761,8 @@ static void nvme_rdma_reset_ctrl_work(struct work_struct *work) ??????? int ret; ??????? bool changed; -?????? nvme_stop_ctrl(&ctrl->ctrl); ??????? nvme_rdma_shutdown_ctrl(ctrl, false); +?????? nvme_stop_ctrl(&ctrl->ctrl); ??????? if (!nvme_change_ctrl_state(&ctrl->ctrl,
NVME_CTRL_CONNECTING)) {
??????????????? /* state change failure should never happen */Quoted 5 lines
(We need to twiddle the order on RDMA, as we'd need to abort the I/O first before we can flush the
scan_workelement).
Cheers,Hannes
Looking at the flows - it looks like it should be ok. You're proving me wrong. Like changes need to be done in the other transports and they have to be audited for the same flows or the
check_readychecks won't work right to continue to bounce ios so the flush can exit. I'm not sure having some of the async flows continue to run when doing the aborts as it may have side effects. But, I expect the issues aren't new. Simple change, but any change in this area can be dramatic.Quoted 5 lines
It still bothers me how "happy path" the
ns_scanpath is. That the "validate" tears down the namespace on any error, even on a short connectivity disruption, shouldn't be happening. Deleting the namespace can have harsh consequences.Oh, I'm fully with you there.
It only took me some months to properly debug this issue. Personally, I'd mark the namespace as 'faulty', and requeue I/O for these cases (much like we do already when an error is returned). And then we can tear down namespaces after controller reset as we then have a stable state.
Quoted 6 lines
But that'll be quite a change to the flow of control, and I guess the powers that be have some opinion here.
Cheers,
Hannes
Actually, we didn't solve anything.... got caught up with - can scan_work make progress.
The error scenario is:
transport enters reset: it must quiesce queues, mark connections invalid, terminate io, unquiesce queues.
This part has to run as the terminate is required to return the scan io so that scan can make progress. Issue is, scan can make progress by terminating the ns, which means the ns may be tripping on the bitmap when the transport calls the unquiesce queues.
your patch didn't change this sequence as the sequence is in the "shutdown" path. It doesn't matter that scan is waited for later.
I'm back to the original fix: changing quiesce/unquiesce and adding an unquiesce to ns_remove.
-- james
PATCH 2/3 nvme: shorten race window in nvme_ns_remove()
In reply to Hannes ReineckeQuoted 83 lines · Hannes Reinecke
Quoted 72 lines · James Smart
Quoted 55 lines · Hannes Reinecke
Quoted 32 lines · James Smart
Quoted 24 lines · Hannes Reinecke
Quoted 14 lines · Keith Busch
Review static void nvme_ns_remove(struct nvme_ns *ns) · line 3324
Quoted 10 lines · Hannes Reinecke
(quoted diff) +3
-0@@ -3316,6 +3316,10 @@ static void nvme_ns_remove(struct nvme_ns *ns) if (test_and_set_bit(NVME_NS_REMOVING, &ns->flags)) return; + down_write(&ns->ctrl->namespaces_rwsem); + list_del_init(&ns->list); + up_write(&ns->ctrl->namespaces_rwsem); + nvme_fault_inject_fini(ns); if (ns->disk && ns->disk->flags & GENHD_FL_UP) { del_gendisk(ns->disk);I think I understand the problem you've described, but I'm a little concerned with the early removal. This will call
blk_cleanup_queueQuoted 9 lines
after the namespace is removed from our lists, and that does a blocking
queue_freeze. If commands happen to be dispatched to that namespace, and the controller for some reason stops responding, we may not be able
to recover during reset when the namespace queue is removed from the controller list. That's at least problematic if we have to go with the per-queue tag iterators that Jianchao proposed.Quoted 6 lines
Hmm. I sort of see where you coming from.
But I was slightly worried about all the other callers where we iterate the namespace list; one would need to to a _really_ careful audit which are safe to traverse even with namespace removal running; some have to, others must not, and some simply don't care.
Hence this approach to make things easier.Quoted 8 lines
I like and agree with Hannes's later points.
But, Keiths statements make me nervous. With Hannes's patches, I could easily see the case where the
ns->queuewas quiesced whenns_removeis called, which I think causes issues withblk_freeze_queue. We would need to havens_remove, after unlinking, to release the queue. is this right ?Hmm. But maybe it's as simple as this:
drivers/nvme/host/core.c +1
-0@@ -131,6 +131,7 @@ int nvme_reset_ctrl(struct nvme_ctrl *ctrl) {if (!nvme_change_ctrl_state (ctrl,NVME_CTRL_RESETTING ))return -EBUSY ; +flush_work (&ctrl->scan_work);if (!queue_work (nvme_reset_wq, &ctrl->reset_work))return -EBUSY ;Quoted 2 lines
James? Keith?
No...
As mentioned in the other thread. If the reset processing is held off, then nothing will kill the io that the scan thread has issued and is pending on the link. In the best case, you must wait for the command timeout to occur, which will want to re-invoke the reset path, which you've noop'd here, so I'm not fully sure it will clean up the io. And, as reset hasn't run, the io queues on this controller have yet to be quiesced so you're going to start seeing filesystem errors. The other headache - even if that 1st io is aborted and returned, just because the 1 scan io errored, it doesn't stop the scan thread. The thread is coded to keep looping on new namespaces so it'll attempt to send another/many
Quoted 9 lines
commands. Which means the new io's will fall into the pass through the
check_readyroutines as they will fall into the path that is only prevented by the connection state being nil, which would not have been
set because the reset path was held off. Minimally a new change has to
be put there. This is not something that can be solved by scheduling
work elements at the transport level. The transport has no idea what ns's are using its io request queue what states they are in. It's a ns state issue.
Quoted 4 lines
Please do check with my reply to Ming Lei.
We should be able to flushscan_workfromnvme_stop_ctrl(); FC already stops all I/O beforenvme_stop_ctrl()(so any sync I/O will be completed),
Not at all, these in-flight IOs are canceled from device queue, and they are actually retried and requeued to blk-mq, and will be dispatched again after the reset is done.
Quoted 2 lines
and for RDMA we should be able to reshuffle
nvme_stop_ctrl()andnvme_rdma_stop_io()to get to the same behaviour.
Thanks,
Ming Lei