Skip to contentSkip to conversation

nvme controller reset and namespace scan work race conditions

RFC PATCH Superseded Sagi Grimberg 5 participants 42 messages Jul 29, 2019 – Aug 1, 2019 NVMe

v1 of 10 Superseded Latest is v10 →

lore
Share & tools
lore

Sagi Grimberg

author maintainer
#0 lore raw

cover nvme controller reset and namespace scan work race conditions

Hey Hannes,

Here is two patches that to my understanding of the issues you describe in your patchset: "nvme: flush rescan worker before resetting" and your report of "spurious I/O errors during failover".

Patch #1 avoids removing a namespace if the revalidation I/O failed because of a racing controller reset (or removal). This should fix any spurious I/O failures during path failover.

Patch #2 avoids a use-after-free condition in the case where the scan work ends up actually removing a namespace but is racing with a controller reset (that is scheduled after the removal) and is accessing the request queue (nvme_stop_queues) after it was destroyed.

The one trace that you have mentioned in our discussions that is still not clear to me is:
--

[67088.344034] WARNING: CPU: 4 PID: 25020 at
../lib/percpu-refcount.c:334 percpu_ref_kill_and_confirm+0x7a/0xa0
[...]
[67088.344106] Call Trace:
[67088.344112]  blk_freeze_queue_start+0x2a/0x40
[67088.344114]  blk_freeze_queue+0xe/0x40
[67088.344118]  nvme_update_disk_info+0x36/0x260 [nvme_core]
[67088.344122]  __nvme_revalidate_disk+0xca/0xf0 [nvme_core]
[67088.344125]  nvme_revalidate_disk+0xa6/0x120 [nvme_core]
[67088.344127]  ? blk_mq_get_tag+0xa3/0x220
[67088.344130]  revalidate_disk+0x23/0xc0
[67088.344133]  nvme_validate_ns+0x43/0x830 [nvme_core]
[67088.344137]  ? wake_up_q+0x70/0x70
[67088.344139]  ? blk_mq_free_request+0x12a/0x160
[67088.344142]  ? __nvme_submit_sync_cmd+0x73/0xe0 [nvme_core]
[67088.344145]  nvme_scan_work+0x2b3/0x350 [nvme_core]
[67088.344149]  process_one_work+0x1da/0x400
[67088.344150]  worker_thread+0x2b/0x3f0
[67088.344152]  ? process_one_work+0x400/0x400

--

Which indicates that we are revalidating a namespace that was already removed. Given that the only namespace removal that is outside of the scan_work is in nvme_remove_namespaces() which flushes the scan_work before it actually removes the namespaces. I'm still lost how this can happen.

Can you please apply the following two patches and report if they address the issues you are seeing? And if not, can you please report a call trace of the hanged threads?

And, given that your are in a multipath environment, can you apply these on top of: "nvme: fix controller removal race with scan work"?

Thanks.

Sagi Grimberg (2):
nvme: don't remove namespace if revalidate failed because of
controller reset
nvme: fix possible use-after-free condition when controller reset is
racing namespace scanning

 drivers/nvme/host/core.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)
Signature · 1 line
2.17.1
author maintainer
#1 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

If a controller reset is racing with a namespace revalidation, the revalidation I/O will surely fail, but we should not remove the namespace as we will execute the I/O when the controller is LIVE again. Specifically check if the controller is LIVE because as RESETTING/CONNECTING are transient and DELETING/DEAD will eventually remove the namespace in the removal code path.

This fixes sporious I/O errors in path failover coditions where the controller reset is racing with the namespace scan work.

Reported-by Hannes Reinecke <hare at suse.de>
Signed-off-by Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/core.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +2 -1
index fa31da0762b9..5f6970e7ba73 100644
@@ -3428,7 +3428,8 @@ static void nvme_validate_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 
 	ns = nvme_find_get_ns(ctrl, nsid);
 	if (ns) {
-		if (ns->disk && revalidate_disk(ns->disk))
+		if (ns->disk && revalidate_disk(ns->disk) &&
+		    ctrl->state != NVME_CTRL_LIVE)
 			nvme_ns_remove(ns);
 		nvme_put_ns(ns);
 	} else
Signature · 1 line
2.17.1
#2 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 21 lines · Sagi Grimberg

If a controller reset is racing with a namespace revalidation, the revalidation I/O will surely fail, but we should not remove the namespace as we will execute the I/O when the controller is LIVE again. Specifically check if the controller is LIVE because as RESETTING/CONNECTING are transient and DELETING/DEAD will eventually remove the namespace in the removal code path.

This fixes sporious I/O errors in path failover coditions where the controller reset is racing with the namespace scan work.

Reported-by Hannes Reinecke <hare at suse.de>
Signed-off-by Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/core.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +0 -0
index fa31da0762b9..5f6970e7ba73 100644

drivers/nvme/host/core.c static void nvme_validate_ns(struct nvme_ctrl *ctrl, unsigned nsid) · line 3433

Quoted 7 lines
drivers/nvme/host/core.c +2 -1
@@ -3428,7 +3428,8 @@ static void nvme_validate_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 
        ns = nvme_find_get_ns(ctrl, nsid);
        if (ns) {
-               if (ns->disk && revalidate_disk(ns->disk))
+               if (ns->disk && revalidate_disk(ns->disk) &&
+                   ctrl->state != NVME_CTRL_LIVE)
                        nvme_ns_remove(ns);

That should be '== NVME_CTRL_LIVE', right? You don't want to remove it for the resetting state, and the removing states take care of removal directly.

On the RESETTING state, shouldn't we avoid calling revalidate_disk() in the first place? An intermittent failure during reset could cause the disk to temporarily go to a zero-capacity, which may cause some filesystem issues that we could avoid if we know we're about to rescan this namespace.

Quoted 2 lines
drivers/nvme/host/core.c +0 -0
                nvme_put_ns(ns);
        } else
author maintainer
#3 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 7 lines
Quoted 7 lines
drivers/nvme/host/core.c +0 -0
index fa31da0762b9..5f6970e7ba73 100644
@@ -3428,7 +3428,8 @@ static void nvme_validate_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 
Quoted 10 lines

drivers/nvme/host/core.c static void nvme_validate_ns(struct nvme_ctrl *ctrl, unsigned nsid) · line 3433

Quoted 6 lines
drivers/nvme/host/core.c +2 -1
         ns = nvme_find_get_ns(ctrl, nsid);
         if (ns) {
-               if (ns->disk && revalidate_disk(ns->disk))
+               if (ns->disk && revalidate_disk(ns->disk) &&
+                   ctrl->state != NVME_CTRL_LIVE)
                         nvme_ns_remove(ns);

That should be '== NVME_CTRL_LIVE', right? You don't want to remove it for the resetting state, and the removing states take care of removal directly.

rrr... you're right...

Quoted 5 lines

On the RESETTING state, shouldn't we avoid calling revalidate_disk() in the first place? An intermittent failure during reset could cause the disk to temporarily go to a zero-capacity, which may cause some filesystem issues that we could avoid if we know we're about to rescan this namespace.

Yea, this should do the trick I guess:
--

drivers/nvme/host/core.c +0 -0
index fa31da0762b9..d01976c93160 100644
@@ -3428,7 +3428,8 @@ static void nvme_validate_ns(struct nvme_ctrl 

*ctrl, unsigned nsid)

         ns = nvme_find_get_ns(ctrl, nsid);
         if (ns) {
-               if (ns->disk && revalidate_disk(ns->disk))
+               if (ns->disk && ctrl->state == NVME_CTRL_LIVE &&
+                   revalidate_disk(ns->disk)
                         nvme_ns_remove(ns);
                 nvme_put_ns(ns);
         } else

--

#4 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

In reply to Sagi Grimberg
Quoted 3 lines · Sagi Grimberg

If a controller reset is racing with a namespace revalidation, the revalidation I/O will surely fail, but we should not remove the

No, there is sync IO request in revalidation, and the sync IO can't be completed during resetting.

Thanks,
Ming Lei

author maintainer
#5 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 6 lines
Quoted 3 lines

If a controller reset is racing with a namespace revalidation, the revalidation I/O will surely fail, but we should not remove the

No, there is sync IO request in revalidation, and the sync IO can't be completed during resetting.

Why? of course it can. The controller reset is responsible to quiesce + abort + unquiesce both I/O and admin commands.

#6 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 10 lines · Sagi Grimberg
Quoted 7 lines
Quoted 4 lines

If a controller reset is racing with a namespace revalidation, the revalidation I/O will surely fail, but we should not remove the

No, there is sync IO request in revalidation, and the sync IO can't be completed during resetting.

Why? of course it can. The controller reset is responsible to quiesce + abort + unquiesce both I/O and admin commands.

Abort simply cancels the in-flight requests, which will be retried & re-queued into
blk-mq queues, and will be dispatched again after reset is done. That is why the revalidation IO won't be failed.

Thanks,
Ming Lei

author maintainer
#7 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

If a controller reset is racing with a namespace revalidation, the revalidation I/O will surely fail, but we should not remove the

No, there is sync IO request in revalidation, and the sync IO can't be completed during resetting.

Quoted 7 lines
Quoted 2 lines

Why? of course it can. The controller reset is responsible to quiesce + abort + unquiesce both I/O and admin commands.

Abort simply cancels the in-flight requests, which will be retried & re-queued into
blk-mq queues, and will be dispatched again after reset is done. That is why the revalidation IO won't be failed.

These I/Os are admin which will not be retried (because we mark it with REQ_FAILFAST_DRIVER). As for normal I/O that meant to be retried, it still must either failfast or failover to another path.

(see the patch I sent Logan for an issue in this area:
[PATCH v3] nvme: fix controller removal race with scan work)

#8 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 11 lines · Sagi Grimberg
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines

If a controller reset is racing with a namespace revalidation, the revalidation I/O will surely fail, but we should not remove the

No, there is sync IO request in revalidation, and the sync IO can't be completed during resetting.

Why? of course it can. The controller reset is responsible to quiesce + abort + unquiesce both I/O and admin commands.

Quoted 7 lines
Quoted 4 lines

Abort simply cancels the in-flight requests, which will be retried & re-queued into
blk-mq queues, and will be dispatched again after reset is done. That is why the revalidation IO won't be failed.

These I/Os are admin which will not be retried (because we mark it with REQ_FAILFAST_DRIVER).

Yeah, this way is fine since data loss won't be caused.

Quoted 2 lines

As for normal I/O that meant to be retried, it still must either failfast or failover to another path.

Why? If the IO is failed, dirty pages may be lost, and it is one generic issue in either multipath or not, and I don't think the normal IO should be failed at least in case of non-multipath.

Quoted 3 lines

(see the patch I sent Logan for an issue in this area:
[PATCH v3] nvme: fix controller removal race with scan work)

The above patch is only for multipath.

thanks,
Ming Lei

author maintainer
#9 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 20 lines · Ming Lei
Quoted 11 lines · Sagi Grimberg
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines

If a controller reset is racing with a namespace revalidation, the revalidation I/O will surely fail, but we should not remove the

No, there is sync IO request in revalidation, and the sync IO can't be completed during resetting.

Why? of course it can. The controller reset is responsible to quiesce + abort + unquiesce both I/O and admin commands.

Quoted 7 lines
Quoted 4 lines

Abort simply cancels the in-flight requests, which will be retried & re-queued into
blk-mq queues, and will be dispatched again after reset is done. That is why the revalidation IO won't be failed.

These I/Os are admin which will not be retried (because we mark it with REQ_FAILFAST_DRIVER).

Quoted 8 lines

Yeah, this way is fine since data loss won't be caused.

Quoted 2 lines

As for normal I/O that meant to be retried, it still must either failfast or failover to another path.

Why? If the IO is failed, dirty pages may be lost, and it is one generic issue in either multipath or not, and I don't think the normal IO should be failed at least in case of non-multipath.

This was meant only when the namespace is going away.
Yea, if the namespace is not going away as described in this patch, then yes, it will block until reset is completed.

Quoted 4 lines
Quoted 2 lines

(see the patch I sent Logan for an issue in this area:
[PATCH v3] nvme: fix controller removal race with scan work)

The above patch is only for multipath.

Yes, and again, addresses the case that the namespace is going away.

So I think we are in agreement? I only need to change the commit message from: "the revalidation I/O" to "the admin I/O" ?

#10 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 40 lines · Sagi Grimberg
Quoted 20 lines · Ming Lei
Quoted 11 lines · Sagi Grimberg
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines

If a controller reset is racing with a namespace revalidation, the revalidation I/O will surely fail, but we should not remove the

No, there is sync IO request in revalidation, and the sync IO can't be completed during resetting.

Why? of course it can. The controller reset is responsible to quiesce + abort + unquiesce both I/O and admin commands.

Quoted 7 lines
Quoted 4 lines

Abort simply cancels the in-flight requests, which will be retried & re-queued into
blk-mq queues, and will be dispatched again after reset is done. That is why the revalidation IO won't be failed.

These I/Os are admin which will not be retried (because we mark it with REQ_FAILFAST_DRIVER).

Quoted 8 lines

Yeah, this way is fine since data loss won't be caused.

Quoted 2 lines

As for normal I/O that meant to be retried, it still must either failfast or failover to another path.

Why? If the IO is failed, dirty pages may be lost, and it is one generic issue in either multipath or not, and I don't think the normal IO should be failed at least in case of non-multipath.

This was meant only when the namespace is going away.
Yea, if the namespace is not going away as described in this patch, then yes, it will block until reset is completed.

Quoted 3 lines
Quoted 2 lines

(see the patch I sent Logan for an issue in this area:
[PATCH v3] nvme: fix controller removal race with scan work)

Quoted 6 lines
Quoted 1 line

The above patch is only for multipath.

Yes, and again, addresses the case that the namespace is going away.

So I think we are in agreement? I only need to change the commit message from: "the revalidation I/O" to "the admin I/O" ?

That words of 'admin I/O' isn't related with the patch or issue.

Quoted 16 lines

Yea, this should do the trick I guess:
--

drivers/nvme/host/core.c +2 -1
index fa31da0762b9..d01976c93160 100644
@@ -3428,7 +3428,8 @@ static void nvme_validate_ns(struct nvme_ctrl
 *ctrl, unsigned nsid)
 
         ns = nvme_find_get_ns(ctrl, nsid);
         if (ns) {
-               if (ns->disk && revalidate_disk(ns->disk))
+               if (ns->disk && ctrl->state == NVME_CTRL_LIVE &&
+                   revalidate_disk(ns->disk)
                         nvme_ns_remove(ns);
                 nvme_put_ns(ns);
Quoted 1 line
         } else

If RESET is triggered just inside revalidate_disk(), and not done after revalidate_disk() returns, there is still race between reset and scan work.

For example, nvme_ns_remove() may hang for ever since the ns may be quiesced in RESET, and can't be un-quiesced any more.

Thanks,
Ming Lei

author maintainer
#11 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 7 lines
Quoted 5 lines

Yes, and again, addresses the case that the namespace is going away.

So I think we are in agreement? I only need to change the commit message from: "the revalidation I/O" to "the admin I/O" ?

That words of 'admin I/O' isn't related with the patch or issue.

But it is, the original issue was due to the fact that nvme_revalidate_disk() I/Os such as nvme_identify_ns() or nvme_identify_ns_descs(). This was the original issue.

Quoted 9 lines
Quoted 9 lines

Yea, this should do the trick I guess:
--

drivers/nvme/host/core.c +0 -0
index fa31da0762b9..d01976c93160 100644
@@ -3428,7 +3428,8 @@ static void nvme_validate_ns(struct nvme_ctrl
 *ctrl, unsigned nsid)
 
Quoted 12 lines
Quoted 8 lines
drivers/nvme/host/core.c +2 -1
          ns = nvme_find_get_ns(ctrl, nsid);
          if (ns) {
-               if (ns->disk && revalidate_disk(ns->disk))
+               if (ns->disk && ctrl->state == NVME_CTRL_LIVE &&
+                   revalidate_disk(ns->disk)
                          nvme_ns_remove(ns);
                  nvme_put_ns(ns);
          } else

If RESET is triggered just inside revalidate_disk(), and not done after revalidate_disk() returns, there is still race between reset and scan work.

You are correct, this was why I had the ctrl->state check after revalidate_disk so if it failed because we are in a reset we should not remove the namespace.

We need a reliable way to NOT remove the namespace if revalidate_disk failed because the controller is resetting and we don't have a channel to the controller at this very moment...

Keith,

As for the failure during reset scenario, this is happening only when the namespace is about to go away or something is seriously wrong right (looking from where nvme_kill_queues is called).

Do you still think we should avoid calling the revalidate_disk if the controller is resetting?

maintainer
#12 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 42 lines · Sagi Grimberg
Quoted 7 lines
Quoted 5 lines

Yes, and again, addresses the case that the namespace is going away.

So I think we are in agreement? I only need to change the commit message from: "the revalidation I/O" to "the admin I/O" ?

That words of 'admin I/O' isn't related with the patch or issue.

But it is, the original issue was due to the fact that nvme_revalidate_disk() I/Os such as nvme_identify_ns() or nvme_identify_ns_descs(). This was the original issue.

Quoted 9 lines
Quoted 9 lines

Yea, this should do the trick I guess:
--

drivers/nvme/host/core.c +0 -0
index fa31da0762b9..d01976c93160 100644
@@ -3428,7 +3428,8 @@ static void nvme_validate_ns(struct nvme_ctrl
 *ctrl, unsigned nsid)
 
Quoted 12 lines
Quoted 8 lines
drivers/nvme/host/core.c +2 -1
          ns = nvme_find_get_ns(ctrl, nsid);
          if (ns) {
-               if (ns->disk && revalidate_disk(ns->disk))
+               if (ns->disk && ctrl->state == NVME_CTRL_LIVE &&
+                   revalidate_disk(ns->disk)
                          nvme_ns_remove(ns);
                  nvme_put_ns(ns);
          } else

If RESET is triggered just inside revalidate_disk(), and not done after revalidate_disk() returns, there is still race between reset and scan work.

You are correct, this was why I had the ctrl->state check after revalidate_disk so if it failed because we are in a reset we should not remove the namespace.

We need a reliable way to NOT remove the namespace if revalidate_disk failed because the controller is resetting and we don't have a channel to the controller at this very moment...

Quoted 8 lines

Keith,

As for the failure during reset scenario, this is happening only when the namespace is about to go away or something is seriously wrong right (looking from where nvme_kill_queues is called).

Do you still think we should avoid calling the revalidate_disk if the controller is resetting?

I was considering if a reset happens to trigger when nvme's revalidate_disk tries to read identify namespace. It's possible that command gets aborted, and we don't retry admin commands, so we'd return -ENODEV and nvme_validate_ns() removes an otherwise healthy namespace.

I'm not too concerned about this corner case actually occuring in practice, though.

author maintainer
#13 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 7 lines
Quoted 7 lines

Keith,

As for the failure during reset scenario, this is happening only when the namespace is about to go away or something is seriously wrong right (looking from where nvme_kill_queues is called).

Quoted 7 lines
Quoted 2 lines

Do you still think we should avoid calling the revalidate_disk if the controller is resetting?

I was considering if a reset happens to trigger when nvme's revalidate_disk tries to read identify namespace. It's possible that command gets aborted, and we don't retry admin commands, so we'd return -ENODEV and nvme_validate_ns() removes an otherwise healthy namespace.

But if the ctrl->state condition is after the revalidate_disk call it won't remove the namespace.
--

                 if (ns->disk && revalidate_disk(ns->disk &&
                     ctrl->state == NVME_CTRL_LIVE)
                         nvme_ns_remove(ns);

--

Quoted 3 lines

I'm not too concerned about this corner case actually occuring in practice, though.

Say, why are we removing the namepsace if the revalidate failed? Don't see an equivalent behavior in scsi or others in drivers/block..

#14 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 10 lines · Sagi Grimberg
Quoted 10 lines
Quoted 9 lines

Keith,

As for the failure during reset scenario, this is happening only when the namespace is about to go away or something is seriously wrong right (looking from where nvme_kill_queues is called).

Do you still think we should avoid calling the revalidate_disk if the controller is resetting?

Quoted 19 lines
Quoted 4 lines

I was considering if a reset happens to trigger when nvme's revalidate_disk tries to read identify namespace. It's possible that command gets aborted, and we don't retry admin commands, so we'd return -ENODEV and nvme_validate_ns() removes an otherwise healthy namespace.

But if the ctrl->state condition is after the revalidate_disk call it won't remove the namespace.

Signature · 4 lines
??????????????? if (ns->disk && revalidate_disk(ns->disk &&
??????????????????? ctrl->state == NVME_CTRL_LIVE)
??????????????????????? nvme_ns_remove(ns);
-- 
Quoted 3 lines

I'm not too concerned about this corner case actually occuring in practice, though.

Say, why are we removing the namepsace if the revalidate failed? Don't see an equivalent behavior in scsi or others in drivers/block..

That probably due to the convoluted logic in nvme_scan_ns_list(). That scans all namespaces in the list, and removes those which fails. Which seems sensible, but really shouldn't be called during reset as we cannot get any sensible results for any of those calls.

This was the reason for my original patch to flush the scanning thread before entering reset. Maybe we should revive it...

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG N?rnberg)
author maintainer
#15 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 30 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines

Keith,

As for the failure during reset scenario, this is happening only when the namespace is about to go away or something is seriously wrong right (looking from where nvme_kill_queues is called).

Do you still think we should avoid calling the revalidate_disk if the controller is resetting?

Quoted 19 lines
Quoted 4 lines

I was considering if a reset happens to trigger when nvme's revalidate_disk tries to read identify namespace. It's possible that command gets aborted, and we don't retry admin commands, so we'd return -ENODEV and nvme_validate_ns() removes an otherwise healthy namespace.

But if the ctrl->state condition is after the revalidate_disk call it won't remove the namespace.

Signature · 4 lines
 ??????????????? if (ns->disk && revalidate_disk(ns->disk &&
 ??????????????????? ctrl->state == NVME_CTRL_LIVE)
 ??????????????????????? nvme_ns_remove(ns);
-- 
Quoted 3 lines

I'm not too concerned about this corner case actually occuring in practice, though.

Say, why are we removing the namepsace if the revalidate failed? Don't see an equivalent behavior in scsi or others in drivers/block..

Quoted 7 lines

That probably due to the convoluted logic in nvme_scan_ns_list(). That scans all namespaces in the list, and removes those which fails. Which seems sensible, but really shouldn't be called during reset as we cannot get any sensible results for any of those calls.

This was the reason for my original patch to flush the scanning thread before entering reset. Maybe we should revive it...

I stand firm that controller resets should not attempt to flush the scan_work with some hooks to try to make it bail out early. I just don't think its the correct solution. Note that the scan flush existed in the reset flow before (nvme_stop_ctrl) but moved away because it caused issues before and probably will again if its restored.

When the controller resets, I/Os fail (fast-fail or block), the model is fine in my mind, but needs some hashing out to be more aware of that.

#16 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

In reply to Keith Busch
Quoted 52 lines · Keith Busch
Quoted 42 lines · Sagi Grimberg
Quoted 7 lines
Quoted 5 lines

Yes, and again, addresses the case that the namespace is going away.

So I think we are in agreement? I only need to change the commit message from: "the revalidation I/O" to "the admin I/O" ?

That words of 'admin I/O' isn't related with the patch or issue.

But it is, the original issue was due to the fact that nvme_revalidate_disk() I/Os such as nvme_identify_ns() or nvme_identify_ns_descs(). This was the original issue.

Quoted 9 lines
Quoted 9 lines

Yea, this should do the trick I guess:
--

drivers/nvme/host/core.c +0 -0
index fa31da0762b9..d01976c93160 100644
@@ -3428,7 +3428,8 @@ static void nvme_validate_ns(struct nvme_ctrl
 *ctrl, unsigned nsid)
 
Quoted 12 lines
Quoted 8 lines
drivers/nvme/host/core.c +2 -1
          ns = nvme_find_get_ns(ctrl, nsid);
          if (ns) {
-               if (ns->disk && revalidate_disk(ns->disk))
+               if (ns->disk && ctrl->state == NVME_CTRL_LIVE &&
+                   revalidate_disk(ns->disk)
                          nvme_ns_remove(ns);
                  nvme_put_ns(ns);
          } else

If RESET is triggered just inside revalidate_disk(), and not done after revalidate_disk() returns, there is still race between reset and scan work.

You are correct, this was why I had the ctrl->state check after revalidate_disk so if it failed because we are in a reset we should not remove the namespace.

We need a reliable way to NOT remove the namespace if revalidate_disk failed because the controller is resetting and we don't have a channel to the controller at this very moment...

Quoted 8 lines

Keith,

As for the failure during reset scenario, this is happening only when the namespace is about to go away or something is seriously wrong right (looking from where nvme_kill_queues is called).

Do you still think we should avoid calling the revalidate_disk if the controller is resetting?

Quoted 8 lines

I was considering if a reset happens to trigger when nvme's revalidate_disk tries to read identify namespace. It's possible that command gets aborted, and we don't retry admin commands, so we'd return -ENODEV and nvme_validate_ns() removes an otherwise healthy namespace.

I'm not too concerned about this corner case actually occuring in practice, though.

... discarding those poor folks having to hunt down this very same issue for several months now ...

Yes, it _does_ occur.
Not on PCI, mind, but definitely for FC. RDMA might have a slightly better chance of not hitting it, but even there we have seen it.

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG N?rnberg)
maintainer
#17 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 9 lines · Hannes Reinecke
Quoted 3 lines

I'm not too concerned about this corner case actually occuring in practice, though.

... discarding those poor folks having to hunt down this very same issue for several months now ...

Yes, it _does_ occur.
Not on PCI, mind, but definitely for FC. RDMA might have a slightly better chance of not hitting it, but even there we have seen it.

Fair enough! I'll re-read your patches with this better understanding on the context. :)

author maintainer
#18 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

In reply to Hannes Reinecke
Quoted 9 lines
Quoted 9 lines

I was considering if a reset happens to trigger when nvme's revalidate_disk tries to read identify namespace. It's possible that command gets aborted, and we don't retry admin commands, so we'd return -ENODEV and nvme_validate_ns() removes an otherwise healthy namespace.

I'm not too concerned about this corner case actually occuring in practice, though.

Quoted 6 lines

... discarding those poor folks having to hunt down this very same issue for several months now ...

Yes, it _does_ occur.
Not on PCI, mind, but definitely for FC. RDMA might have a slightly better chance of not hitting it, but even there we have seen it.

And this patch prevents from the namespaces being removed, which _is_ the wrong behavior we need to prevent. As I said, we should not remove that namespace instead of trying to synchronize the remove with everything else...

I think I asked this but was not answered, why are we removing the namespace at all? do others do the same thing (remove the disk if revalidation fails)?

maintainer
#19 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 4 lines · Sagi Grimberg

I think I asked this but was not answered, why are we removing the namespace at all? do others do the same thing (remove the disk if revalidation fails)?

If a namespace no longer exists, what do you want to do with it instead of removing it?

author maintainer
#20 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 6 lines
Quoted 4 lines

I think I asked this but was not answered, why are we removing the namespace at all? do others do the same thing (remove the disk if revalidation fails)?

If a namespace no longer exists,

Why is it no longer exists? it failed revalidate..

Quoted 1 line

what do you want to do with it instead of removing it?

Well, I don't see anyone else even checking the return status of revalidate_disk.. Perhaps Hannes can share more on how scsi handles this?

maintainer
#21 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 8 lines · Sagi Grimberg
Quoted 6 lines
Quoted 4 lines

I think I asked this but was not answered, why are we removing the namespace at all? do others do the same thing (remove the disk if revalidation fails)?

If a namespace no longer exists,

Why is it no longer exists? it failed revalidate..

One way it fails to validate is if it doesn't exist, i.e., the controller returned an error when attempting to identify it.

The other way it may fail to revalidate is if its identify has changed since we last discovered it, so removal is better than data corruption.

Either scenario could happen from administrative namespace management commands to any controller in the subsystem.

Quoted 5 lines
Quoted 1 line

what do you want to do with it instead of removing it?

Well, I don't see anyone else even checking the return status of revalidate_disk.. Perhaps Hannes can share more on how scsi handles this?

Other drivers may have a different path to this, but I'd imagine they handle a LUN deletion somehow.

author maintainer
#22 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines

I think I asked this but was not answered, why are we removing the namespace at all? do others do the same thing (remove the disk if revalidation fails)?

If a namespace no longer exists,

Quoted 7 lines
Quoted 1 line

Why is it no longer exists? it failed revalidate..

One way it fails to validate is if it doesn't exist, i.e., the controller returned an error when attempting to identify it.

The other way it may fail to revalidate is if its identify has changed since we last discovered it, so removal is better than data corruption.

Well, perhaps we can mark failures resulting from reset with a transport error.

For example, nvme_cancel_request is setting:NVME_SC_ABORT_REQ, perhaps we can modify nvme_error_status to set that into BLK_STS_TRANSPORT and check for that as the return code for revalidate_disk?

Thoughts?

Quoted 11 lines

Either scenario could happen from administrative namespace management commands to any controller in the subsystem.

Quoted 5 lines
Quoted 1 line

what do you want to do with it instead of removing it?

Well, I don't see anyone else even checking the return status of revalidate_disk.. Perhaps Hannes can share more on how scsi handles this?

Other drivers may have a different path to this, but I'd imagine they handle a LUN deletion somehow.

Well, its not coming from revalidate_disk failure as the return status is not checked...

maintainer
#23 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 15 lines · Sagi Grimberg
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines

I think I asked this but was not answered, why are we removing the namespace at all? do others do the same thing (remove the disk if revalidation fails)?

If a namespace no longer exists,

Quoted 7 lines
Quoted 1 line

Why is it no longer exists? it failed revalidate..

One way it fails to validate is if it doesn't exist, i.e., the controller returned an error when attempting to identify it.

The other way it may fail to revalidate is if its identify has changed since we last discovered it, so removal is better than data corruption.

Quoted 8 lines

Well, perhaps we can mark failures resulting from reset with a transport error.

For example, nvme_cancel_request is setting:NVME_SC_ABORT_REQ, perhaps we can modify nvme_error_status to set that into BLK_STS_TRANSPORT and check for that as the return code for revalidate_disk?

Thoughts?

Would it be sufficient to let these admin commands requeue? Instead of flushing the scan work, we can let it block for IO on a reset, and the IO will resume when the reset completes.

author maintainer
#24 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 11 lines
Quoted 11 lines
Quoted 3 lines

The other way it may fail to revalidate is if its identify has changed since we last discovered it, so removal is better than data corruption.

Well, perhaps we can mark failures resulting from reset with a transport error.

For example, nvme_cancel_request is setting:NVME_SC_ABORT_REQ, perhaps we can modify nvme_error_status to set that into BLK_STS_TRANSPORT and check for that as the return code for revalidate_disk?

Quoted 5 lines
Quoted 1 line

Thoughts?

Would it be sufficient to let these admin commands requeue? Instead of flushing the scan work, we can let it block for IO on a reset, and the IO will resume when the reset completes.

Well, I don't think we should do that. Unlike I/O commands, which can failover to a different path, these admin commands are bound to the specific controller. In case it takes minutes/hours/days for the controller to restore normal operation, it will be unpleasant to say the least to have admin operations get stuck for so long.

maintainer
#25 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 17 lines · Sagi Grimberg
Quoted 11 lines
Quoted 11 lines
Quoted 3 lines

The other way it may fail to revalidate is if its identify has changed since we last discovered it, so removal is better than data corruption.

Well, perhaps we can mark failures resulting from reset with a transport error.

For example, nvme_cancel_request is setting:NVME_SC_ABORT_REQ, perhaps we can modify nvme_error_status to set that into BLK_STS_TRANSPORT and check for that as the return code for revalidate_disk?

Quoted 5 lines
Quoted 1 line

Thoughts?

Would it be sufficient to let these admin commands requeue? Instead of flushing the scan work, we can let it block for IO on a reset, and the IO will resume when the reset completes.

Quoted 5 lines

Well, I don't think we should do that. Unlike I/O commands, which can failover to a different path, these admin commands are bound to the specific controller. In case it takes minutes/hours/days for the controller to restore normal operation, it will be unpleasant to say the least to have admin operations get stuck for so long.

Unpleasant for who? The scan_work is the only thing waiting for these commands, no one else should care because you can't run IO if you're stuck in very long reset anyway.

I think the main point is that we don't want to take a delete action on a transient condition, but sprinkling NVME_CTRL_LIVE checks is open to many other races.

author maintainer
#26 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 10 lines
Quoted 6 lines

Well, I don't think we should do that. Unlike I/O commands, which can failover to a different path, these admin commands are bound to the specific controller. In case it takes minutes/hours/days for the controller to restore normal operation, it will be unpleasant to say the least to have admin operations get stuck for so long.

Unpleasant for who? The scan_work is the only thing waiting for these commands, no one else should care because you can't run IO if you're stuck in very long reset anyway.

The hung task detector would care, and a user who will attempt to issue a passthru command, and the rest of the system that have one of the kworkers sacrificed for a significant amount of time...

Quoted 3 lines

I think the main point is that we don't want to take a delete action on a transient condition, but sprinkling NVME_CTRL_LIVE checks is open to many other races.

Hence I suggested the transport error code...

maintainer
#27 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 7 lines · Sagi Grimberg
Quoted 7 lines
Quoted 6 lines

Well, I don't think we should do that. Unlike I/O commands, which can failover to a different path, these admin commands are bound to the specific controller. In case it takes minutes/hours/days for the controller to restore normal operation, it will be unpleasant to say the least to have admin operations get stuck for so long.

Quoted 7 lines
Quoted 3 lines

Unpleasant for who? The scan_work is the only thing waiting for these commands, no one else should care because you can't run IO if you're stuck in very long reset anyway.

The hung task detector would care, and a user who will attempt to issue a passthru command, and the rest of the system that have one of the kworkers sacrificed for a significant amount of time...

blk_execute_rq already defeats hung task detection for stalled IO.

My point, though, was passthru doesn't care about scan_work. A submitted passthru command is blocked for reset, so blocking scan_work doesn't make that situation any better or worse.

Quoted 5 lines
Quoted 3 lines

I think the main point is that we don't want to take a delete action on a transient condition, but sprinkling NVME_CTRL_LIVE checks is open to many other races.

Hence I suggested the transport error code...

That should work too.

author maintainer
#28 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 6 lines

Well, I don't think we should do that. Unlike I/O commands, which can failover to a different path, these admin commands are bound to the specific controller. In case it takes minutes/hours/days for the controller to restore normal operation, it will be unpleasant to say the least to have admin operations get stuck for so long.

Unpleasant for who? The scan_work is the only thing waiting for these commands, no one else should care because you can't run IO if you're stuck in very long reset anyway.

Quoted 8 lines
Quoted 3 lines

The hung task detector would care, and a user who will attempt to issue a passthru command, and the rest of the system that have one of the kworkers sacrificed for a significant amount of time...

blk_execute_rq already defeats hung task detection for stalled IO.

My point, though, was passthru doesn't care about scan_work. A submitted passthru command is blocked for reset,

Not in fabrics drivers (unless I'm missing something that changed).

Quoted 1 line

so blocking scan_work doesn't make that situation any better or worse.

I think that when we talk about reset in fabrics, we have in mind a long process (mainly because of the fact that network port failures are a lot more frequent and span some amount of time). This is why fabric drivers, when they get a transport error, they go into the reset flow and they quiesce+terminate+unquiesce to fastfail admin commands.

#29 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

In reply to Sagi Grimberg
Quoted 35 lines · Sagi Grimberg
Quoted 7 lines
Quoted 5 lines

Yes, and again, addresses the case that the namespace is going away.

So I think we are in agreement? I only need to change the commit message from: "the revalidation I/O" to "the admin I/O" ?

That words of 'admin I/O' isn't related with the patch or issue.

But it is, the original issue was due to the fact that nvme_revalidate_disk() I/Os such as nvme_identify_ns() or nvme_identify_ns_descs(). This was the original issue.

Quoted 9 lines
Quoted 9 lines

Yea, this should do the trick I guess:

drivers/nvme/host/core.c +0 -0
index fa31da0762b9..d01976c93160 100644
@@ -3428,7 +3428,8 @@ static void nvme_validate_ns(struct nvme_ctrl
 *ctrl, unsigned nsid)
 
Quoted 13 lines
Quoted 8 lines
drivers/nvme/host/core.c +2 -1
 ????????? ns = nvme_find_get_ns(ctrl, nsid);
 ????????? if (ns) {
-?????????????? if (ns->disk && revalidate_disk(ns->disk))
+?????????????? if (ns->disk && ctrl->state == NVME_CTRL_LIVE &&
+?????????????????? revalidate_disk(ns->disk)
 ????????????????????????? nvme_ns_remove(ns);
 ????????????????? nvme_put_ns(ns);

????????? } else

If RESET is triggered just inside revalidate_disk(), and not done after revalidate_disk() returns,? there is still race between reset and scan work.

Quoted 8 lines

You are correct, this was why I had the ctrl->state check after revalidate_disk so if it failed because we are in a reset we should not remove the namespace.

We need a reliable way to NOT remove the namespace if revalidate_disk failed because the controller is resetting and we don't have a channel to the controller at this very moment...

A similar thing is true for the 'ns_head' structure; I could easily envision a scenario where all paths are dropped (due to revalidate failures or whatnot) with all controllers in reset.
But as all paths are dropped struct ns_head doesn't have any reference to any namespaces (and consequently, to any controller), and will be failing all I/O before itself being removed.
It will be re-created after resetting completes and all namespaces are re-attached, for sure, but it still means that we're getting I/O errors where we really shouldn't.
One is tempted to say 'queue_if_no_path' here ...

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG N?rnberg)
author maintainer
#30 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 5 lines
Quoted 5 lines

You are correct, this was why I had the ctrl->state check after revalidate_disk so if it failed because we are in a reset we should not remove the namespace.

Quoted 7 lines
Quoted 4 lines

We need a reliable way to NOT remove the namespace if revalidate_disk failed because the controller is resetting and we don't have a channel to the controller at this very moment...

A similar thing is true for the 'ns_head' structure; I could easily envision a scenario where all paths are dropped (due to revalidate failures or whatnot) with all controllers in reset.

Hannes, paths should not be dropped in controller resets, only when the controller is going away. Its simple as that.

Quoted 6 lines

But as all paths are dropped struct ns_head doesn't have any reference to any namespaces (and consequently, to any controller), and will be failing all I/O before itself being removed.
It will be re-created after resetting completes and all namespaces are re-attached, for sure, but it still means that we're getting I/O errors where we really shouldn't.

Paths should only be removed when the controller is going away, not when it is temporarily disconnected.

#31 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 9 lines · Sagi Grimberg
Quoted 9 lines
Quoted 9 lines

You are correct, this was why I had the ctrl->state check after revalidate_disk so if it failed because we are in a reset we should not remove the namespace.

We need a reliable way to NOT remove the namespace if revalidate_disk failed because the controller is resetting and we don't have a channel to the controller at this very moment...

Quoted 7 lines
Quoted 3 lines

A similar thing is true for the 'ns_head' structure; I could easily envision a scenario where all paths are dropped (due to revalidate failures or whatnot) with all controllers in reset.

Hannes, paths should not be dropped in controller resets, only when the controller is going away. Its simple as that.

Fully agree. But so far we don't seem to be there yet.

Quoted 9 lines
Quoted 6 lines

But as all paths are dropped struct ns_head doesn't have any reference to any namespaces (and consequently, to any controller), and will be failing all I/O before itself being removed.
It will be re-created after resetting completes and all namespaces are re-attached, for sure, but it still means that we're getting I/O errors where we really shouldn't.

Paths should only be removed when the controller is going away, not when it is temporarily disconnected.

I'm fully in line with that.
However, it looks as if this isn't the case currently :-)
And whenever I try to come up with a solution here I always come back to the thought: It would've been easier if we just could flush scan_work ...

And my main worry here is: surely we'll manage to hash out the situation eventually, and fixup all code paths to avoid dropping paths during reset. But the solution will be very fragile, as any innocent change there might break things again. So I'm not convinced that asynchronous reset and rescan is a feature which should be kept.
(And honestly, we're working towards a solution where scanning during reset will become a no-op ...)

But anyway, prime objective is to get this mess fixed :-)

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke            Teamlead Storage & Networking
hare at suse.de                              +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG N?rnberg)
author maintainer
#32 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 2 lines

Fully agree. But so far we don't seem to be there yet.

Right.

Quoted 15 lines
Quoted 9 lines
Quoted 6 lines

But as all paths are dropped struct ns_head doesn't have any reference to any namespaces (and consequently, to any controller), and will be failing all I/O before itself being removed.
It will be re-created after resetting completes and all namespaces are re-attached, for sure, but it still means that we're getting I/O errors where we really shouldn't.

Paths should only be removed when the controller is going away, not when it is temporarily disconnected.

I'm fully in line with that.
However, it looks as if this isn't the case currently :-)
And whenever I try to come up with a solution here I always come back to the thought: It would've been easier if we just could flush scan_work ...

Quoted 7 lines

And my main worry here is: surely we'll manage to hash out the situation eventually, and fixup all code paths to avoid dropping paths during reset. But the solution will be very fragile, as any innocent change there might break things again. So I'm not convinced that asynchronous reset and rescan is a feature which should be kept.
(And honestly, we're working towards a solution where scanning during reset will become a no-op ...)

But flushing the scan_work is even more fragile.. the scan_work is a non-trivial work that may have dependencies with other processes.

For example, passthru commands with effects take the scan lock during its lifetime, which means that it may be blocked even before it checks the ctrl state (which is again, unreliable with respect to resets). So do resets flush all outstanding passthru commands now?

So I think that flushing the scan_work might bring us new issues because of this dependency. That is why I think we should address the root cause which is we're wrongly removing the namespace, not the fact that we failed to scan it.

#33 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

In reply to Sagi Grimberg
Quoted 22 lines · Sagi Grimberg

If a controller reset is racing with a namespace revalidation, the revalidation I/O will surely fail, but we should not remove the namespace as we will execute the I/O when the controller is LIVE again. Specifically check if the controller is LIVE because as RESETTING/CONNECTING are transient and DELETING/DEAD will eventually remove the namespace in the removal code path.

This fixes sporious I/O errors in path failover coditions where the controller reset is racing with the namespace scan work.

Reported-by Hannes Reinecke <hare at suse.de>
Signed-off-by Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/core.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +0 -0
index fa31da0762b9..5f6970e7ba73 100644

drivers/nvme/host/core.c static void nvme_validate_ns(struct nvme_ctrl *ctrl, unsigned nsid) · line 3435

Quoted 9 lines
drivers/nvme/host/core.c +2 -1
@@ -3428,7 +3428,8 @@ static void nvme_validate_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 
 	ns = nvme_find_get_ns(ctrl, nsid);
 	if (ns) {
-		if (ns->disk && revalidate_disk(ns->disk))
+		if (ns->disk && revalidate_disk(ns->disk) &&
+		    ctrl->state != NVME_CTRL_LIVE)
 			nvme_ns_remove(ns);
 		nvme_put_ns(ns);
 	} else

The thing I'm worried about here is concurrency (this was also what caused my earlier attempts to be rejected).
Thing is, the controller state can be set asynchronously. Which in itself is protected by ctrl->lock, but here we're not taking the lock at all. Hence we might be seeing a stale value, causing us to makt the wrong decision.

I have a patchset checking 'ctrl->state' under lock; will be posting it later.

And another thing: where's the point in 'revalidate_disk()' to be called if the controller is not live? At best it won't do anything; at worst we'll stall if we have to do more than one I/O (the reset cycle will abort _one_ I/O, but if we have to submit more than one we'll be stuck, too).

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG N?rnberg)
author maintainer
#34 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 7 lines
Quoted 7 lines
drivers/nvme/host/core.c +0 -0
index fa31da0762b9..5f6970e7ba73 100644
@@ -3428,7 +3428,8 @@ static void nvme_validate_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 
Quoted 15 lines

drivers/nvme/host/core.c static void nvme_validate_ns(struct nvme_ctrl *ctrl, unsigned nsid) · line 3435

Quoted 9 lines
drivers/nvme/host/core.c +2 -1
  	ns = nvme_find_get_ns(ctrl, nsid);
  	if (ns) {
-		if (ns->disk && revalidate_disk(ns->disk))
+		if (ns->disk && revalidate_disk(ns->disk) &&
+		    ctrl->state != NVME_CTRL_LIVE)
  			nvme_ns_remove(ns);
  		nvme_put_ns(ns);
  	} else

The thing I'm worried about here is concurrency (this was also what caused my earlier attempts to be rejected).
Thing is, the controller state can be set asynchronously. Which in itself is protected by ctrl->lock, but here we're not taking the lock at all. Hence we might be seeing a stale value, causing us to makt the wrong decision.

This is why the state check is _after_ the failure, if we failed to do I/O because of a reset, it will be reflected in the state.

Quoted 2 lines

I have a patchset checking 'ctrl->state' under lock; will be posting it later.

Unless the wraps your actions according the state, the state still advisory.

Quoted 5 lines

And another thing: where's the point in 'revalidate_disk()' to be called if the controller is not live? At best it won't do anything; at worst we'll stall if we have to do more than one I/O (the reset cycle will abort _one_ I/O, but if we have to submit more than one we'll be stuck, too).

Exactly because of the race that you are indicating. We can't reliably know if the controller is resetting (or will be starting a reset soon) so we go ahead with the revalidate knowing that if the controller is resetting it will fail. That is why the state check is after (this tells us if the revalidation failed because of a reset or something else).

#35 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 38 lines · Sagi Grimberg
Quoted 14 lines
Quoted 14 lines
drivers/nvme/host/core.c +2 -1
index fa31da0762b9..5f6970e7ba73 100644
@@ -3428,7 +3428,8 @@ static void nvme_validate_ns(struct nvme_ctrl 
 *ctrl, unsigned nsid)
 ????? ns = nvme_find_get_ns(ctrl, nsid);
 ????? if (ns) {
-??????? if (ns->disk && revalidate_disk(ns->disk))
+??????? if (ns->disk && revalidate_disk(ns->disk) &&
+??????????? ctrl->state != NVME_CTRL_LIVE)
 ????????????? nvme_ns_remove(ns);
 ????????? nvme_put_ns(ns);
Quoted 8 lines
Quoted 2 lines

????? } else

The thing I'm worried about here is concurrency (this was also what caused my earlier attempts to be rejected).
Thing is, the controller state can be set asynchronously. Which in itself is protected by ctrl->lock, but here we're not taking the lock at all. Hence we might be seeing a stale value, causing us to makt the wrong decision.

This is why the state check is _after_ the failure, if we failed to do I/O because of a reset, it will be reflected in the state.

Quoted 2 lines

I have a patchset checking 'ctrl->state' under lock; will be posting it later.

Unless the wraps your actions according the state, the state still advisory.

Quoted 5 lines

And another thing: where's the point in 'revalidate_disk()' to be called if the controller is not live? At best it won't do anything; at worst we'll stall if we have to do more than one I/O (the reset cycle will abort _one_ I/O, but if we have to submit more than one we'll be stuck, too).

Quoted 5 lines

Exactly because of the race that you are indicating. We can't reliably know if the controller is resetting (or will be starting a reset soon) so we go ahead with the revalidate knowing that if the controller is resetting it will fail. That is why the state check is after (this tells us if the revalidation failed because of a reset or something else).

Okay, fair enough.

Reviewed-by Hannes Reinecke <hare at suse.com>

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke            Teamlead Storage & Networking
hare at suse.de                              +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG N?rnberg)
author maintainer
#36 lore raw

RFC PATCH 1/2 nvme: don't remove namespace if revalidate failed because of controller reset

Quoted 10 lines
Quoted 6 lines

Exactly because of the race that you are indicating. We can't reliably know if the controller is resetting (or will be starting a reset soon) so we go ahead with the revalidate knowing that if the controller is resetting it will fail. That is why the state check is after (this tells us if the revalidation failed because of a reset or something else).

Okay, fair enough.

Reviewed-by Hannes Reinecke <hare at suse.com>

Hannes,

Can we understand if this addresses the issues you have reported?

From the cover-letter:
Hey Hannes,

Here is two patches that to my understanding of the issues you describe in your patchset: "nvme: flush rescan worker before resetting" and your report of "spurious I/O errors during failover".

[...]

Can you please apply the following two patches and report if they address the issues you are seeing? And if not, can you please report a call trace of the hanged threads?

And, given that your are in a multipath environment, can you apply these on top of: "nvme: fix controller removal race with scan work"?

author maintainer
#37 lore raw

RFC PATCH 2/2 nvme: fix possible use-after-free condition when controller reset is racing namespace scanning

In reply to Sagi Grimberg

remove ns from ctrl namespaces list before destroying the request queue as the reset work will unquisce for all the controller namespaces, while an online scan might remove invalid namespaces resulting from a concurrent AEN.

While we're at it, move nvme_check_last_path to right after we modify the multipath reated cleanups.

Reported-by Hannes Reinecke <hare at suse.de>
Signed-off-by Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/core.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
drivers/nvme/host/core.c +5 -5
index 5f6970e7ba73..9f8f8f5feeae 100644
@@ -3406,6 +3406,11 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 	synchronize_rcu(); /* guarantee not available in head->list */
 	nvme_mpath_clear_current_path(ns);
 	synchronize_srcu(&ns->head->srcu); /* wait for concurrent submissions */
+	nvme_mpath_check_last_path(ns);
+
+	down_write(&ns->ctrl->namespaces_rwsem);
+	list_del_init(&ns->list);
+	up_write(&ns->ctrl->namespaces_rwsem);
 
 	if (ns->disk && ns->disk->flags & GENHD_FL_UP) {
 		del_gendisk(ns->disk);
@@ -3414,11 +3419,6 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 			blk_integrity_unregister(ns->disk);
 	}
 
-	down_write(&ns->ctrl->namespaces_rwsem);
-	list_del_init(&ns->list);
-	up_write(&ns->ctrl->namespaces_rwsem);
-
-	nvme_mpath_check_last_path(ns);
 	nvme_put_ns(ns);
 }
 
Signature · 1 line
2.17.1
#38 lore raw

RFC PATCH 2/2 nvme: fix possible use-after-free condition when controller reset is racing namespace scanning

Quoted 33 lines · Sagi Grimberg

remove ns from ctrl namespaces list before destroying the request queue as the reset work will unquisce for all the controller namespaces, while an online scan might remove invalid namespaces resulting from a concurrent AEN.

While we're at it, move nvme_check_last_path to right after we modify the multipath reated cleanups.

Reported-by Hannes Reinecke <hare at suse.de>
Signed-off-by Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/core.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
drivers/nvme/host/core.c +5 -0
index 5f6970e7ba73..9f8f8f5feeae 100644
@@ -3406,6 +3406,11 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 	synchronize_rcu(); /* guarantee not available in head->list */
 	nvme_mpath_clear_current_path(ns);
 	synchronize_srcu(&ns->head->srcu); /* wait for concurrent submissions */
+	nvme_mpath_check_last_path(ns);
+
+	down_write(&ns->ctrl->namespaces_rwsem);
+	list_del_init(&ns->list);
+	up_write(&ns->ctrl->namespaces_rwsem);
 
 	if (ns->disk && ns->disk->flags & GENHD_FL_UP) {
 		del_gendisk(ns->disk);
@@ -3414,11 +3419,6 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 			blk_integrity_unregister(ns->disk);

drivers/nvme/host/core.c static void nvme_ns_remove(struct nvme_ns *ns) · line 3424

Quoted 10 lines
drivers/nvme/host/core.c +0 -5
 	}
 
-	down_write(&ns->ctrl->namespaces_rwsem);
-	list_del_init(&ns->list);
-	up_write(&ns->ctrl->namespaces_rwsem);
-
-	nvme_mpath_check_last_path(ns);
 	nvme_put_ns(ns);
 }
 

How does this one play with nvme_stop_queues()/nvme_start_queues() we're doing during reset?
IE what happens if this is called after nvme_stop_queues(), but before nvme_start_queues()?
We'll end up having a stopped queue when deleting the disk; from what I've seen blk_cleanup_queue() will just freeze the queue and wait for I/O to complete, which it'll never will as the queue is stopped ...

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG N?rnberg)
author maintainer
#39 lore raw

RFC PATCH 2/2 nvme: fix possible use-after-free condition when controller reset is racing namespace scanning

Quoted 21 lines
Quoted 21 lines
drivers/nvme/host/core.c +5 -0
index 5f6970e7ba73..9f8f8f5feeae 100644
@@ -3406,6 +3406,11 @@ static void nvme_ns_remove(struct nvme_ns *ns)
  	synchronize_rcu(); /* guarantee not available in head->list */
  	nvme_mpath_clear_current_path(ns);
  	synchronize_srcu(&ns->head->srcu); /* wait for concurrent submissions */
+	nvme_mpath_check_last_path(ns);
+
+	down_write(&ns->ctrl->namespaces_rwsem);
+	list_del_init(&ns->list);
+	up_write(&ns->ctrl->namespaces_rwsem);
 
  	if (ns->disk && ns->disk->flags & GENHD_FL_UP) {
  		del_gendisk(ns->disk);
@@ -3414,11 +3419,6 @@ static void nvme_ns_remove(struct nvme_ns *ns)
  			blk_integrity_unregister(ns->disk);
  	}
 
Quoted 16 lines

drivers/nvme/host/core.c static void nvme_ns_remove(struct nvme_ns *ns) · line 3424

Quoted 9 lines
drivers/nvme/host/core.c +0 -5
-	down_write(&ns->ctrl->namespaces_rwsem);
-	list_del_init(&ns->list);
-	up_write(&ns->ctrl->namespaces_rwsem);
-
-	nvme_mpath_check_last_path(ns);
  	nvme_put_ns(ns);
  }
 

How does this one play with nvme_stop_queues()/nvme_start_queues() we're doing during reset?
IE what happens if this is called after nvme_stop_queues(), but before nvme_start_queues()?
We'll end up having a stopped queue when deleting the disk; from what I've seen blk_cleanup_queue() will just freeze the queue and wait for I/O to complete, which it'll never will as the queue is stopped ...

nvme_stop/start_queue are quiescing the queue, not freezing it which does not block requests in blk_queue_enter.
So I don't think that blk_cleanup_queue will hang for unquiesced queues.

#40 lore raw

RFC PATCH 2/2 nvme: fix possible use-after-free condition when controller reset is racing namespace scanning

Quoted 36 lines · Sagi Grimberg
Quoted 27 lines
Quoted 27 lines
drivers/nvme/host/core.c +0 -0
index 5f6970e7ba73..9f8f8f5feeae 100644
@@ -3406,6 +3406,11 @@ static void nvme_ns_remove(struct nvme_ns *ns)

????? synchronize_rcu(); /* guarantee not available in head->list */ ????? nvme_mpath_clear_current_path(ns);
????? synchronize_srcu(&ns->head->srcu); /* wait for concurrent submissions */
+??? nvme_mpath_check_last_path(ns);
+
+??? down_write(&ns->ctrl->namespaces_rwsem);
+??? list_del_init(&ns->list);
+??? up_write(&ns->ctrl->namespaces_rwsem);
? ????? if (ns->disk && ns->disk->flags & GENHD_FL_UP) { ????????? del_gendisk(ns->disk);

drivers/nvme/host/core.c +0 -4
@@ -3414,11 +3419,6 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 ????????????? blk_integrity_unregister(ns->disk);
 ????? }
 ? -??? down_write(&ns->ctrl->namespaces_rwsem);
-??? list_del_init(&ns->list);
-??? up_write(&ns->ctrl->namespaces_rwsem);
-
-??? nvme_mpath_check_last_path(ns);
 ????? nvme_put_ns(ns);
 ? }
Quoted 8 lines
Quoted 1 line

?

How does this one play with nvme_stop_queues()/nvme_start_queues() we're doing during reset?
IE what happens if this is called after nvme_stop_queues(), but before nvme_start_queues()?
We'll end up having a stopped queue when deleting the disk; from what I've seen blk_cleanup_queue() will just freeze the queue and wait for I/O to complete, which it'll never will as the queue is stopped ...

Quoted 3 lines

nvme_stop/start_queue are quiescing the queue, not freezing it which does not block requests in blk_queue_enter.
So I don't think that blk_cleanup_queue will hang for unquiesced queues.

My worry here is for quiesced queues; they are not unquiesced when blk_clean_queue() is running. So I'm not sure if we're able to flush all commands then.

But be it as it may, I've sent a patchset for checking the NS_REMOVING flag when traversing the namespace list. That should take care of this issue, and should render this patch obsolete.

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG N?rnberg)
author maintainer
#41 lore raw

RFC PATCH 2/2 nvme: fix possible use-after-free condition when controller reset is racing namespace scanning

Quoted 8 lines
Quoted 4 lines

nvme_stop/start_queue are quiescing the queue, not freezing it which does not block requests in blk_queue_enter.
So I don't think that blk_cleanup_queue will hang for unquiesced queues.

My worry here is for quiesced queues; they are not unquiesced when blk_clean_queue() is running. So I'm not sure if we're able to flush all commands then.

blk_cleanup_queue does not block on quiesced queues, it blocks on frozen queues.

Quoted 3 lines

But be it as it may, I've sent a patchset for checking the NS_REMOVING flag when traversing the namespace list. That should take care of this issue, and should render this patch obsolete.

I've shared feedback on that one, I need to understand what is the crash that you are talking about.. and why not promoting the list removal in nvme_ns_remove() doesn't address it.