Skip to contentSkip to conversation

nvme: make NVMe freeze API reliably

PATCH In review Ming Lei 7 participants 24 messages Aug 21, 2022 – Sep 20, 2022 NVMe

lore
Share & tools
lore

Ming Lei

author
#0 lore raw

cover nvme: make NVMe freeze API reliably

Hello,

The 1st patch is from Keith, which makes NVMe freeze APIs more reliably.

The 2nd patch removes nvme_wait_freeze() from nvme-pci's nvme_reset_work().

Yi, please test and see if the issue you reported in [1] can be fixed.

[1] https://lore.kernel.org/linux-block/CAHj4cs--KPTAGP=jj+7KMe=arDv=HeGeOgs1T8vbusyk=EjXow@mail.gmail.com/#r

Keith Busch (1):
nvme: make NVMe freeze API reliably

Ming Lei (1):
nvme-pci: don't wait freeze during resetting

 drivers/nvme/host/core.c | 16 +++++++++++++---
 drivers/nvme/host/nvme.h |  1 +
 drivers/nvme/host/pci.c  |  1 -
 3 files changed, 14 insertions(+), 4 deletions(-)
Signature · 1 line
2.31.1
author
#1 lore raw

PATCH 1/2 nvme: make NVMe freeze API reliably

From: Keith Busch <[email protected]>

In some corner cases[1], freeze wait and unfreeze API may be called on unfrozen queue, add one per-ns flag of NVME_NS_FREEZE to make these freeze APIs more reliably, then this kind of issues can be avoided. And similar approach has been applied on stopping/quiescing nvme queues.

[1] https://lore.kernel.org/linux-nvme/[email protected]/

Add comment log.

---
 drivers/nvme/host/core.c | 16 +++++++++++++---
 drivers/nvme/host/nvme.h |  1 +
 2 files changed, 14 insertions(+), 3 deletions(-)
drivers/nvme/host/core.c +13 -3
index af367b22871b..fe4ae3616ed1 100644
@@ -5028,8 +5028,11 @@ void nvme_unfreeze(struct nvme_ctrl *ctrl)
 	struct nvme_ns *ns;
 
 	down_read(&ctrl->namespaces_rwsem);
-	list_for_each_entry(ns, &ctrl->namespaces, list)
+	list_for_each_entry(ns, &ctrl->namespaces, list) {
+		if (!test_and_clear_bit(NVME_NS_FREEZE, &ns->flags))
+			continue;
 		blk_mq_unfreeze_queue(ns->queue);
+	}
 	up_read(&ctrl->namespaces_rwsem);
 }
 EXPORT_SYMBOL_GPL(nvme_unfreeze);
@@ -5040,6 +5043,8 @@ int nvme_wait_freeze_timeout(struct nvme_ctrl *ctrl, long timeout)
 
 	down_read(&ctrl->namespaces_rwsem);
 	list_for_each_entry(ns, &ctrl->namespaces, list) {
+		if (!test_bit(NVME_NS_FREEZE, &ns->flags))
+			continue;
 		timeout = blk_mq_freeze_queue_wait_timeout(ns->queue, timeout);
 		if (timeout <= 0)
 			break;
@@ -5054,8 +5059,11 @@ void nvme_wait_freeze(struct nvme_ctrl *ctrl)
 	struct nvme_ns *ns;
 
 	down_read(&ctrl->namespaces_rwsem);
-	list_for_each_entry(ns, &ctrl->namespaces, list)
+	list_for_each_entry(ns, &ctrl->namespaces, list) {
+		if (!test_bit(NVME_NS_FREEZE, &ns->flags))
+			continue;
 		blk_mq_freeze_queue_wait(ns->queue);
+	}
 	up_read(&ctrl->namespaces_rwsem);
 }
 EXPORT_SYMBOL_GPL(nvme_wait_freeze);
@@ -5065,8 +5073,10 @@ void nvme_start_freeze(struct nvme_ctrl *ctrl)
 	struct nvme_ns *ns;
 
 	down_read(&ctrl->namespaces_rwsem);
-	list_for_each_entry(ns, &ctrl->namespaces, list)
+	list_for_each_entry(ns, &ctrl->namespaces, list) {
+		set_bit(NVME_NS_FREEZE, &ns->flags);
 		blk_freeze_queue_start(ns->queue);
+	}
 	up_read(&ctrl->namespaces_rwsem);
 }
 EXPORT_SYMBOL_GPL(nvme_start_freeze);
drivers/nvme/host/nvme.h +1 -0
index 1bdf714dcd9e..702ba19b5647 100644
@@ -484,6 +484,7 @@ struct nvme_ns {
 #define NVME_NS_FORCE_RO	3
 #define NVME_NS_READY		4
 #define NVME_NS_STOPPED		5
+#define NVME_NS_FREEZE		6
 
 	struct cdev		cdev;
 	struct device		cdev_device;
Signature · 1 line
2.31.1
#2 lore raw
Quoted 59 lines · Ming Lei

From: Keith Busch <[email protected]>

In some corner cases[1], freeze wait and unfreeze API may be called on unfrozen queue, add one per-ns flag of NVME_NS_FREEZE to make these freeze APIs more reliably, then this kind of issues can be avoided. And similar approach has been applied on stopping/quiescing nvme queues.

[1] https://lore.kernel.org/linux-nvme/[email protected]/

Add comment log.

---
  drivers/nvme/host/core.c | 16 +++++++++++++---
  drivers/nvme/host/nvme.h |  1 +
  2 files changed, 14 insertions(+), 3 deletions(-)
drivers/nvme/host/core.c +10 -2
index af367b22871b..fe4ae3616ed1 100644
@@ -5028,8 +5028,11 @@ void nvme_unfreeze(struct nvme_ctrl *ctrl)
  	struct nvme_ns *ns;
 
  	down_read(&ctrl->namespaces_rwsem);
-	list_for_each_entry(ns, &ctrl->namespaces, list)
+	list_for_each_entry(ns, &ctrl->namespaces, list) {
+		if (!test_and_clear_bit(NVME_NS_FREEZE, &ns->flags))
+			continue;
  		blk_mq_unfreeze_queue(ns->queue);
+	}
  	up_read(&ctrl->namespaces_rwsem);
  }
  EXPORT_SYMBOL_GPL(nvme_unfreeze);
@@ -5040,6 +5043,8 @@ int nvme_wait_freeze_timeout(struct nvme_ctrl *ctrl, long timeout)
 
  	down_read(&ctrl->namespaces_rwsem);
  	list_for_each_entry(ns, &ctrl->namespaces, list) {
+		if (!test_bit(NVME_NS_FREEZE, &ns->flags))
+			continue;
  		timeout = blk_mq_freeze_queue_wait_timeout(ns->queue, timeout);
  		if (timeout <= 0)
  			break;
@@ -5054,8 +5059,11 @@ void nvme_wait_freeze(struct nvme_ctrl *ctrl)
  	struct nvme_ns *ns;
 
  	down_read(&ctrl->namespaces_rwsem);
-	list_for_each_entry(ns, &ctrl->namespaces, list)
+	list_for_each_entry(ns, &ctrl->namespaces, list) {
+		if (!test_bit(NVME_NS_FREEZE, &ns->flags))
+			continue;
  		blk_mq_freeze_queue_wait(ns->queue);
+	}
  	up_read(&ctrl->namespaces_rwsem);
  }
  EXPORT_SYMBOL_GPL(nvme_wait_freeze);

drivers/nvme/host/core.c void nvme_start_freeze(struct nvme_ctrl *ctrl) · line 5076

Quoted 5 lines
drivers/nvme/host/core.c +1 -1
@@ -5065,8 +5073,10 @@ void nvme_start_freeze(struct nvme_ctrl *ctrl)
  	struct nvme_ns *ns;
 
  	down_read(&ctrl->namespaces_rwsem);
-	list_for_each_entry(ns, &ctrl->namespaces, list)
+	list_for_each_entry(ns, &ctrl->namespaces, list) { > +		set_bit(NVME_NS_FREEZE, &ns->flags);

Why not test_and_set_bit()?
Which would have the nice effect of adding a memory barrier ...

drivers/nvme/host/core.c void nvme_start_freeze(struct nvme_ctrl *ctrl) · line 5082

Quoted 5 lines
drivers/nvme/host/core.c +1 -0
  		blk_freeze_queue_start(ns->queue);
+	}
  	up_read(&ctrl->namespaces_rwsem);
  }
  EXPORT_SYMBOL_GPL(nvme_start_freeze);

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke		           Kernel Storage Architect
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer
maintainer
#3 lore raw
Quoted 10 lines · Hannes Reinecke

drivers/nvme/host/core.c void nvme_start_freeze(struct nvme_ctrl *ctrl) · line 5076

Quoted 6 lines · Ming Lei
drivers/nvme/host/core.c +2 -1
@@ -5065,8 +5073,10 @@ void nvme_start_freeze(struct nvme_ctrl *ctrl)
  	struct nvme_ns *ns;
  	down_read(&ctrl->namespaces_rwsem);
-	list_for_each_entry(ns, &ctrl->namespaces, list)
+	list_for_each_entry(ns, &ctrl->namespaces, list) {
+		set_bit(NVME_NS_FREEZE, &ns->flags);

Why not test_and_set_bit()?
Which would have the nice effect of adding a memory barrier ...

Sure, I think you can even make a new WARN if the test fails. It should never fail in this path, otherwise we could mess up the freeze depth.

#4 lore raw
In reply to Ming Lei
Quoted 6 lines · Ming Lei

From: Keith Busch <[email protected]>

In some corner cases[1], freeze wait and unfreeze API may be called on unfrozen queue, add one per-ns flag of NVME_NS_FREEZE to make these freeze APIs more reliably, then this kind of issues can be avoided. And similar approach has been applied on stopping/quiescing nvme queues.

This leads to another problem: the process that needs to be in the frozen state is not actually frozen.
It's not safe.
There is the same risk on stopping/quiescing nvme queues.

Quoted 77 lines

[1] https://lore.kernel.org/linux-nvme/[email protected]/

Add comment log.

---
  drivers/nvme/host/core.c | 16 +++++++++++++---
  drivers/nvme/host/nvme.h |  1 +
  2 files changed, 14 insertions(+), 3 deletions(-)
drivers/nvme/host/core.c +13 -3
index af367b22871b..fe4ae3616ed1 100644
@@ -5028,8 +5028,11 @@ void nvme_unfreeze(struct nvme_ctrl *ctrl)
  	struct nvme_ns *ns;
 
  	down_read(&ctrl->namespaces_rwsem);
-	list_for_each_entry(ns, &ctrl->namespaces, list)
+	list_for_each_entry(ns, &ctrl->namespaces, list) {
+		if (!test_and_clear_bit(NVME_NS_FREEZE, &ns->flags))
+			continue;
  		blk_mq_unfreeze_queue(ns->queue);
+	}
  	up_read(&ctrl->namespaces_rwsem);
  }
  EXPORT_SYMBOL_GPL(nvme_unfreeze);
@@ -5040,6 +5043,8 @@ int nvme_wait_freeze_timeout(struct nvme_ctrl *ctrl, long timeout)
 
  	down_read(&ctrl->namespaces_rwsem);
  	list_for_each_entry(ns, &ctrl->namespaces, list) {
+		if (!test_bit(NVME_NS_FREEZE, &ns->flags))
+			continue;
  		timeout = blk_mq_freeze_queue_wait_timeout(ns->queue, timeout);
  		if (timeout <= 0)
  			break;
@@ -5054,8 +5059,11 @@ void nvme_wait_freeze(struct nvme_ctrl *ctrl)
  	struct nvme_ns *ns;
 
  	down_read(&ctrl->namespaces_rwsem);
-	list_for_each_entry(ns, &ctrl->namespaces, list)
+	list_for_each_entry(ns, &ctrl->namespaces, list) {
+		if (!test_bit(NVME_NS_FREEZE, &ns->flags))
+			continue;
  		blk_mq_freeze_queue_wait(ns->queue);
+	}
  	up_read(&ctrl->namespaces_rwsem);
  }
  EXPORT_SYMBOL_GPL(nvme_wait_freeze);
@@ -5065,8 +5073,10 @@ void nvme_start_freeze(struct nvme_ctrl *ctrl)
  	struct nvme_ns *ns;
 
  	down_read(&ctrl->namespaces_rwsem);
-	list_for_each_entry(ns, &ctrl->namespaces, list)
+	list_for_each_entry(ns, &ctrl->namespaces, list) {
+		set_bit(NVME_NS_FREEZE, &ns->flags);
  		blk_freeze_queue_start(ns->queue);
+	}
  	up_read(&ctrl->namespaces_rwsem);
  }
  EXPORT_SYMBOL_GPL(nvme_start_freeze);
drivers/nvme/host/nvme.h +1 -0
index 1bdf714dcd9e..702ba19b5647 100644
@@ -484,6 +484,7 @@ struct nvme_ns {
  #define NVME_NS_FORCE_RO	3
  #define NVME_NS_READY		4
  #define NVME_NS_STOPPED		5
+#define NVME_NS_FREEZE		6
 
  	struct cdev		cdev;
  	struct device		cdev_device;
#5 lore raw
Quoted 11 lines · Chao Leng
Quoted 6 lines · Ming Lei

From: Keith Busch <[email protected]>

In some corner cases[1], freeze wait and unfreeze API may be called on unfrozen queue, add one per-ns flag of NVME_NS_FREEZE to make these freeze APIs more reliably, then this kind of issues can be avoided. And similar approach has been applied on stopping/quiescing nvme queues.

This leads to another problem: the process that needs to be in the frozen state is not actually frozen.

Where do frozen processes come into play here?

author
#6 lore raw
In reply to Chao Leng
Quoted 5 lines · Chao Leng
Quoted 2 lines · Ming Lei

From: Keith Busch <[email protected]>

Quoted 7 lines
Quoted 4 lines

In some corner cases[1], freeze wait and unfreeze API may be called on unfrozen queue, add one per-ns flag of NVME_NS_FREEZE to make these freeze APIs more reliably, then this kind of issues can be avoided. And similar approach has been applied on stopping/quiescing nvme queues.

This leads to another problem: the process that needs to be in the frozen state is not actually frozen.
It's not safe.

The flag is just to control if queue wait is needed, blk_mq_freeze_queue_wait can be done only the flag is set. Not sure how it isn't safe.

Meantime calling blk_mq_freeze_queue_wait() on queue not being started to freeze is usually a bug, and I think WARN_ON_ONCE() can be added in nvme_wait_freeze().

Thanks,
Ming

#7 lore raw
Quoted 10 lines · Ming Lei
Quoted 9 lines · Chao Leng
Quoted 6 lines · Ming Lei

From: Keith Busch <[email protected]>

In some corner cases[1], freeze wait and unfreeze API may be called on unfrozen queue, add one per-ns flag of NVME_NS_FREEZE to make these freeze APIs more reliably, then this kind of issues can be avoided. And similar approach has been applied on stopping/quiescing nvme queues.

Quoted 6 lines
Quoted 3 lines

This leads to another problem: the process that needs to be in the frozen state is not actually frozen.
It's not safe.

The flag is just to control if queue wait is needed, blk_mq_freeze_queue_wait can be done only the flag is set. Not sure how it isn't safe.

I thought that the use of NVME_NS_FREEZE was the same as NVME_NS_STOPPED. If just set_bit in nvme_start_freeze, it will cause another problem in below scenario.
A: start freeze and set the bit;B:start freeze and set the bit; and then
A:test and clear the bit, and unfreeze;B: test and skip;
The queue will be frozen for ever.

In addition, I think patch 2/2 can fix the bug well, patch 1/2 is not necessary. No matter how to use NVME_NS_FREEZE , it may cause problems.
The freeze mechanism is perfect, and no additional protection mechanism is required.

Quoted 11 lines

Meantime calling blk_mq_freeze_queue_wait() on queue not being started to freeze is usually a bug, and I think WARN_ON_ONCE() can be added in nvme_wait_freeze().

Thanks,
Ming

.

author
#8 lore raw
Quoted 19 lines · Chao Leng
Quoted 10 lines · Ming Lei
Quoted 9 lines · Chao Leng
Quoted 6 lines · Ming Lei

From: Keith Busch <[email protected]>

In some corner cases[1], freeze wait and unfreeze API may be called on unfrozen queue, add one per-ns flag of NVME_NS_FREEZE to make these freeze APIs more reliably, then this kind of issues can be avoided. And similar approach has been applied on stopping/quiescing nvme queues.

Quoted 6 lines
Quoted 3 lines

This leads to another problem: the process that needs to be in the frozen state is not actually frozen.
It's not safe.

The flag is just to control if queue wait is needed, blk_mq_freeze_queue_wait can be done only the flag is set. Not sure how it isn't safe.

Quoted 7 lines

I thought that the use of NVME_NS_FREEZE was the same as NVME_NS_STOPPED. If just set_bit in nvme_start_freeze, it will cause another problem in below scenario.
A: start freeze and set the bit;B:start freeze and set the bit; and then
A:test and clear the bit, and unfreeze;B: test and skip;
The queue will be frozen for ever.

One simple approach is to replace down_read(->namespaces_rwsem) with down_write(->namespaces_rwsem) in nvme_start_freeze() and nvme_unfreeze().

Quoted 4 lines

In addition, I think patch 2/2 can fix the bug well, patch 1/2 is not necessary. No matter how to use NVME_NS_FREEZE , it may cause problems.
The freeze mechanism is perfect, and no additional protection mechanism is required.

block layer requires queue freeze and unfreeze APIs to be called in pair strictly, that is why I add the 1st patch.

Thanks,
Ming

#9 lore raw
Quoted 33 lines · Ming Lei
Quoted 19 lines · Chao Leng
Quoted 10 lines · Ming Lei
Quoted 9 lines · Chao Leng
Quoted 6 lines · Ming Lei

From: Keith Busch <[email protected]>

In some corner cases[1], freeze wait and unfreeze API may be called on unfrozen queue, add one per-ns flag of NVME_NS_FREEZE to make these freeze APIs more reliably, then this kind of issues can be avoided. And similar approach has been applied on stopping/quiescing nvme queues.

Quoted 6 lines
Quoted 3 lines

This leads to another problem: the process that needs to be in the frozen state is not actually frozen.
It's not safe.

The flag is just to control if queue wait is needed, blk_mq_freeze_queue_wait can be done only the flag is set. Not sure how it isn't safe.

Quoted 7 lines

I thought that the use of NVME_NS_FREEZE was the same as NVME_NS_STOPPED. If just set_bit in nvme_start_freeze, it will cause another problem in below scenario.
A: start freeze and set the bit;B:start freeze and set the bit; and then
A:test and clear the bit, and unfreeze;B: test and skip;
The queue will be frozen for ever.

One simple approach is to replace down_read(->namespaces_rwsem) with down_write(->namespaces_rwsem) in nvme_start_freeze() and nvme_unfreeze().

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

In addition, I think patch 2/2 can fix the bug well, patch 1/2 is not necessary. No matter how to use NVME_NS_FREEZE , it may cause problems.
The freeze mechanism is perfect, and no additional protection mechanism is required.

block layer requires queue freeze and unfreeze APIs to be called in pair strictly, that is why I add the 1st patch.

From your bug analysis, the reason is that nvme_wait_freeze is called without nvme_start_freeze. patch 2/2 is already delete the nvme_wait_freeze.
If there is another bug of unmatched freeze and unfreeze,
can you describe the analysis of unmatched freeze and unfreeze?
The current patch 1/2 will introduce the unmatched freeze and unfreeze rather than solved it. Maybe another patch is needed to fix the bug.

Quoted 8 lines

Thanks,
Ming

.

author
#10 lore raw
Quoted 40 lines · Chao Leng
Quoted 37 lines · Ming Lei
Quoted 19 lines · Chao Leng
Quoted 10 lines · Ming Lei
Quoted 9 lines · Chao Leng
Quoted 6 lines · Ming Lei

From: Keith Busch <[email protected]>

In some corner cases[1], freeze wait and unfreeze API may be called on unfrozen queue, add one per-ns flag of NVME_NS_FREEZE to make these freeze APIs more reliably, then this kind of issues can be avoided. And similar approach has been applied on stopping/quiescing nvme queues.

Quoted 6 lines
Quoted 3 lines

This leads to another problem: the process that needs to be in the frozen state is not actually frozen.
It's not safe.

The flag is just to control if queue wait is needed, blk_mq_freeze_queue_wait can be done only the flag is set. Not sure how it isn't safe.

Quoted 7 lines

I thought that the use of NVME_NS_FREEZE was the same as NVME_NS_STOPPED. If just set_bit in nvme_start_freeze, it will cause another problem in below scenario.
A: start freeze and set the bit;B:start freeze and set the bit; and then
A:test and clear the bit, and unfreeze;B: test and skip;
The queue will be frozen for ever.

One simple approach is to replace down_read(->namespaces_rwsem) with down_write(->namespaces_rwsem) in nvme_start_freeze() and nvme_unfreeze().

Quoted 4 lines

In addition, I think patch 2/2 can fix the bug well, patch 1/2 is not necessary. No matter how to use NVME_NS_FREEZE , it may cause problems.
The freeze mechanism is perfect, and no additional protection mechanism is required.

Quoted 8 lines
Quoted 2 lines

block layer requires queue freeze and unfreeze APIs to be called in pair strictly, that is why I add the 1st patch.

From your bug analysis, the reason is that nvme_wait_freeze is called without nvme_start_freeze. patch 2/2 is already delete the nvme_wait_freeze.
If there is another bug of unmatched freeze and unfreeze,
can you describe the analysis of unmatched freeze and unfreeze?
The current patch 1/2 will introduce the unmatched freeze and unfreeze rather than solved it. Maybe another patch is needed to fix the bug.

Please look at nvme_reset_work():

		...
		if (dev->online_queues < 2) {
			...
        } else {
                nvme_start_queues(&dev->ctrl);
				nvme_wait_freeze(&dev->ctrl);
                if (!dev->ctrl.tagset)
                        nvme_pci_alloc_tag_set(dev);
                else
                        nvme_pci_update_nr_queues(dev);
                nvme_dbbuf_set(dev);
                nvme_unfreeze(&dev->ctrl);
		}
		...

nvme_wait_freeze() is called on unfrozen queue, so io hang is caused.

If nvme_unfreeze() is called on unfrozen queue, WARN_ON_ONCE() in __blk_mq_unfreeze_queue() will be triggered.

Thanks,
Ming

#11 lore raw
Quoted 66 lines · Ming Lei
Quoted 40 lines · Chao Leng
Quoted 37 lines · Ming Lei
Quoted 19 lines · Chao Leng
Quoted 10 lines · Ming Lei
Quoted 9 lines · Chao Leng
Quoted 6 lines · Ming Lei

From: Keith Busch <[email protected]>

In some corner cases[1], freeze wait and unfreeze API may be called on unfrozen queue, add one per-ns flag of NVME_NS_FREEZE to make these freeze APIs more reliably, then this kind of issues can be avoided. And similar approach has been applied on stopping/quiescing nvme queues.

Quoted 6 lines
Quoted 3 lines

This leads to another problem: the process that needs to be in the frozen state is not actually frozen.
It's not safe.

The flag is just to control if queue wait is needed, blk_mq_freeze_queue_wait can be done only the flag is set. Not sure how it isn't safe.

Quoted 7 lines

I thought that the use of NVME_NS_FREEZE was the same as NVME_NS_STOPPED. If just set_bit in nvme_start_freeze, it will cause another problem in below scenario.
A: start freeze and set the bit;B:start freeze and set the bit; and then
A:test and clear the bit, and unfreeze;B: test and skip;
The queue will be frozen for ever.

One simple approach is to replace down_read(->namespaces_rwsem) with down_write(->namespaces_rwsem) in nvme_start_freeze() and nvme_unfreeze().

Quoted 4 lines

In addition, I think patch 2/2 can fix the bug well, patch 1/2 is not necessary. No matter how to use NVME_NS_FREEZE , it may cause problems.
The freeze mechanism is perfect, and no additional protection mechanism is required.

Quoted 8 lines
Quoted 2 lines

block layer requires queue freeze and unfreeze APIs to be called in pair strictly, that is why I add the 1st patch.

From your bug analysis, the reason is that nvme_wait_freeze is called without nvme_start_freeze.
patch 2/2 is already delete the nvme_wait_freeze.
If there is another bug of unmatched freeze and unfreeze,
can you describe the analysis of unmatched freeze and unfreeze?
The current patch 1/2 will introduce the unmatched freeze and unfreeze rather than solved it. Maybe another patch is needed to fix the bug.

Please look at nvme_reset_work():

		...
		if (dev->online_queues < 2) {
			...
         } else {
                 nvme_start_queues(&dev->ctrl);
				nvme_wait_freeze(&dev->ctrl);
                 if (!dev->ctrl.tagset)
                         nvme_pci_alloc_tag_set(dev);
                 else
                         nvme_pci_update_nr_queues(dev);
                 nvme_dbbuf_set(dev);
                 nvme_unfreeze(&dev->ctrl);
		}
		...
Quoted 4 lines

nvme_wait_freeze() is called on unfrozen queue, so io hang is caused.

If nvme_unfreeze() is called on unfrozen queue, WARN_ON_ONCE() in __blk_mq_unfreeze_queue() will be triggered.

The bug is specific to the PCI transport layer.
It might be a better option to fix this bug by adding flag for normal connection and abnormal recovery like other transport layers, and then perform different processing. Different processes are processed differently to avoid unexpected nvme_unfreeze.

Quoted 8 lines

Thanks,
Ming

.

author
#13 lore raw

PATCH 2/2 nvme-pci: don't wait freeze during resetting

In reply to Ming Lei

First it isn't necessary to call nvme_wait_freeze in nvme_reset_work(). If tagset isn't allocated, there can't be any inflight IOs; otherwise blk_mq_update_nr_hw_queues can freeze & wait queues.

Second, since commit bdd6316094e0 ("block: Allow unfreezing of a queue while requests are in progress"), it is fine to unfreeze queue without draining inflight IOs.

---
 drivers/nvme/host/pci.c | 1 -
 1 file changed, 1 deletion(-)
drivers/nvme/host/pci.c +0 -1
index 3a1c37f32f30..91b2903fcc24 100644
@@ -2910,7 +2910,6 @@ static void nvme_reset_work(struct work_struct *work)
 		nvme_free_tagset(dev);
 	} else {
 		nvme_start_queues(&dev->ctrl);
-		nvme_wait_freeze(&dev->ctrl);
 		if (!dev->ctrl.tagset)
 			nvme_pci_alloc_tag_set(dev);
 		else
Signature · 1 line
2.31.1
#14 lore raw

Re: [PATCH 2/2] nvme-pci: don't wait freeze during resetting

Quoted 18 lines · Ming Lei

First it isn't necessary to call nvme_wait_freeze in nvme_reset_work(). If tagset isn't allocated, there can't be any inflight IOs; otherwise blk_mq_update_nr_hw_queues can freeze & wait queues.

Second, since commit bdd6316094e0 ("block: Allow unfreezing of a queue while requests are in progress"), it is fine to unfreeze queue without draining inflight IOs.

---
  drivers/nvme/host/pci.c | 1 -
  1 file changed, 1 deletion(-)
drivers/nvme/host/pci.c +0 -0
index 3a1c37f32f30..91b2903fcc24 100644

drivers/nvme/host/pci.c static void nvme_reset_work(struct work_struct *work) · line 2913

Quoted 4 lines
drivers/nvme/host/pci.c +0 -1
@@ -2910,7 +2910,6 @@ static void nvme_reset_work(struct work_struct *work)
  		nvme_free_tagset(dev);
  	} else {
  		nvme_start_queues(&dev->ctrl);
-		nvme_wait_freeze(&dev->ctrl);

It is not safe.
nvme_dev_add may call blk_mq_update_nr_hw_queues, blk_mq_update_nr_hw_queues require the state is frozen.

Quoted 4 lines
drivers/nvme/host/pci.c +0 -0
  		if (!dev->ctrl.tagset)
  			nvme_pci_alloc_tag_set(dev);
  		else
author
#15 lore raw

Re: [PATCH 2/2] nvme-pci: don't wait freeze during resetting

Quoted 16 lines · Chao Leng
Quoted 13 lines · Ming Lei

First it isn't necessary to call nvme_wait_freeze in nvme_reset_work(). If tagset isn't allocated, there can't be any inflight IOs; otherwise blk_mq_update_nr_hw_queues can freeze & wait queues.

Second, since commit bdd6316094e0 ("block: Allow unfreezing of a queue while requests are in progress"), it is fine to unfreeze queue without draining inflight IOs.

---
  drivers/nvme/host/pci.c | 1 -
  1 file changed, 1 deletion(-)
Quoted 12 lines

drivers/nvme/host/pci.c static void nvme_reset_work(struct work_struct *work) · line 2913

Quoted 9 lines
drivers/nvme/host/pci.c +0 -1
index 3a1c37f32f30..91b2903fcc24 100644
@@ -2910,7 +2910,6 @@ static void nvme_reset_work(struct work_struct *work)
  		nvme_free_tagset(dev);
  	} else {
  		nvme_start_queues(&dev->ctrl);
-		nvme_wait_freeze(&dev->ctrl);

It is not safe.
nvme_dev_add may call blk_mq_update_nr_hw_queues, blk_mq_update_nr_hw_queues require the state is frozen.

blk_mq_update_nr_hw_queues() calls blk_freeze_queue() for every queue in this tagset, so it needn't nvme's freeze wait.

Thanks,
Ming

maintainer Reviewed-by
#16 lore raw

Re: [PATCH 2/2] nvme-pci: don't wait freeze during resetting

Quoted 30 lines · Ming Lei
Quoted 16 lines · Chao Leng
Quoted 13 lines · Ming Lei

First it isn't necessary to call nvme_wait_freeze in nvme_reset_work(). If tagset isn't allocated, there can't be any inflight IOs; otherwise blk_mq_update_nr_hw_queues can freeze & wait queues.

Second, since commit bdd6316094e0 ("block: Allow unfreezing of a queue while requests are in progress"), it is fine to unfreeze queue without draining inflight IOs.

---
  drivers/nvme/host/pci.c | 1 -
  1 file changed, 1 deletion(-)
Quoted 12 lines

drivers/nvme/host/pci.c static void nvme_reset_work(struct work_struct *work) · line 2913

Quoted 9 lines
drivers/nvme/host/pci.c +0 -1
index 3a1c37f32f30..91b2903fcc24 100644
@@ -2910,7 +2910,6 @@ static void nvme_reset_work(struct work_struct *work)
  		nvme_free_tagset(dev);
  	} else {
  		nvme_start_queues(&dev->ctrl);
-		nvme_wait_freeze(&dev->ctrl);

It is not safe.
nvme_dev_add may call blk_mq_update_nr_hw_queues, blk_mq_update_nr_hw_queues require the state is frozen.

Quoted 2 lines

blk_mq_update_nr_hw_queues() calls blk_freeze_queue() for every queue in this tagset, so it needn't nvme's freeze wait.

Right, the blk-mq realloc should be fine, and it's really a very uncommon event that nvme needs to realloc the tag set.

I'm trying to remember why I added the wait freeze here in the first place, and I don't recall a good reason for it. I think I had a controller that always timed out IO, and the resets would just continue indefinitely. Waiting for freeze within a reset causes the next IO timeout to trigger a driver detach. That doesn't sound like a good idea, though, since it tears down the admin queue too.

Anyway, this looks good to me.

author
#17 lore raw

Re: [PATCH 2/2] nvme-pci: don't wait freeze during resetting

Quoted 37 lines · Keith Busch
Quoted 30 lines · Ming Lei
Quoted 16 lines · Chao Leng
Quoted 13 lines · Ming Lei

First it isn't necessary to call nvme_wait_freeze in nvme_reset_work(). If tagset isn't allocated, there can't be any inflight IOs; otherwise blk_mq_update_nr_hw_queues can freeze & wait queues.

Second, since commit bdd6316094e0 ("block: Allow unfreezing of a queue while requests are in progress"), it is fine to unfreeze queue without draining inflight IOs.

---
  drivers/nvme/host/pci.c | 1 -
  1 file changed, 1 deletion(-)
Quoted 12 lines

drivers/nvme/host/pci.c static void nvme_reset_work(struct work_struct *work) · line 2913

Quoted 9 lines
drivers/nvme/host/pci.c +0 -1
index 3a1c37f32f30..91b2903fcc24 100644
@@ -2910,7 +2910,6 @@ static void nvme_reset_work(struct work_struct *work)
  		nvme_free_tagset(dev);
  	} else {
  		nvme_start_queues(&dev->ctrl);
-		nvme_wait_freeze(&dev->ctrl);

It is not safe.
nvme_dev_add may call blk_mq_update_nr_hw_queues, blk_mq_update_nr_hw_queues require the state is frozen.

Quoted 2 lines

blk_mq_update_nr_hw_queues() calls blk_freeze_queue() for every queue in this tagset, so it needn't nvme's freeze wait.

Right, the blk-mq realloc should be fine, and it's really a very uncommon event that nvme needs to realloc the tag set.

Quoted 6 lines

I'm trying to remember why I added the wait freeze here in the first place, and I don't recall a good reason for it. I think I had a controller that always timed out IO, and the resets would just continue indefinitely. Waiting for freeze within a reset causes the next IO timeout to trigger a driver detach. That doesn't sound like a good idea, though, since it tears down the admin queue too.

The wait becomes not necessary since commit bdd6316094e0 ("block: Allow unfreezing of a queue while requests are in progress").

Thanks,
Ming

Reviewed-by
#18 lore raw

Re: [PATCH 2/2] nvme-pci: don't wait freeze during resetting

Quoted 45 lines · Ming Lei
Quoted 37 lines · Keith Busch
Quoted 30 lines · Ming Lei
Quoted 16 lines · Chao Leng
Quoted 13 lines · Ming Lei

First it isn't necessary to call nvme_wait_freeze in nvme_reset_work(). If tagset isn't allocated, there can't be any inflight IOs; otherwise blk_mq_update_nr_hw_queues can freeze & wait queues.

Second, since commit bdd6316094e0 ("block: Allow unfreezing of a queue while requests are in progress"), it is fine to unfreeze queue without draining inflight IOs.

---
   drivers/nvme/host/pci.c | 1 -
   1 file changed, 1 deletion(-)
Quoted 12 lines

drivers/nvme/host/pci.c static void nvme_reset_work(struct work_struct *work) · line 2913

Quoted 9 lines
drivers/nvme/host/pci.c +0 -1
index 3a1c37f32f30..91b2903fcc24 100644
@@ -2910,7 +2910,6 @@ static void nvme_reset_work(struct work_struct *work)
   		nvme_free_tagset(dev);
   	} else {
   		nvme_start_queues(&dev->ctrl);
-		nvme_wait_freeze(&dev->ctrl);

It is not safe.
nvme_dev_add may call blk_mq_update_nr_hw_queues, blk_mq_update_nr_hw_queues require the state is frozen.

Quoted 2 lines

blk_mq_update_nr_hw_queues() calls blk_freeze_queue() for every queue in this tagset, so it needn't nvme's freeze wait.

Right, the blk-mq realloc should be fine, and it's really a very uncommon event that nvme needs to realloc the tag set.

Quoted 6 lines

I'm trying to remember why I added the wait freeze here in the first place, and I don't recall a good reason for it. I think I had a controller that always timed out IO, and the resets would just continue indefinitely. Waiting for freeze within a reset causes the next IO timeout to trigger a driver detach. That doesn't sound like a good idea, though, since it tears down the admin queue too.

Quoted 2 lines

The wait becomes not necessary since commit bdd6316094e0 ("block: Allow unfreezing of a queue while requests are in progress").

Yes, I didn't notice this before. The wait freeze in nvme is redundant.

Quoted 8 lines

Thanks,
Ming

.

#20 lore raw

Re: [PATCH 2/2] nvme-pci: don't wait freeze during resetting

In reply to Ming Lei
Quoted 7 lines · Ming Lei

First it isn't necessary to call nvme_wait_freeze in nvme_reset_work(). If tagset isn't allocated, there can't be any inflight IOs; otherwise blk_mq_update_nr_hw_queues can freeze & wait queues.

Second, since commit bdd6316094e0 ("block: Allow unfreezing of a queue while requests are in progress"), it is fine to unfreeze queue without draining inflight IOs.

The apple, RDMA, and TCP drivers seem to follow the same wait_free pattern, so I think we'll need patches for them as well.

Tested-by on the v1 series
#21 lore raw
In reply to Ming Lei
Quoted 8 lines · Ming Lei

Hello,

The 1st patch is from Keith, which makes NVMe freeze APIs more reliably.

The 2nd patch removes nvme_wait_freeze() from nvme-pci's nvme_reset_work().

Yi, please test and see if the issue you reported in [1] can be fixed.

Hi Ming
The issue cannot be reproduced now after applying the two patch, feel free to add

Quoted 17 lines

[1] https://lore.kernel.org/linux-block/CAHj4cs--KPTAGP=jj+7KMe=arDv=HeGeOgs1T8vbusyk=EjXow@mail.gmail.com/#r

Keith Busch (1):
nvme: make NVMe freeze API reliably

Ming Lei (1):
nvme-pci: don't wait freeze during resetting

 drivers/nvme/host/core.c | 16 +++++++++++++---
 drivers/nvme/host/nvme.h |  1 +
 drivers/nvme/host/pci.c  |  1 -
 3 files changed, 14 insertions(+), 4 deletions(-)

--
2.31.1

Signature · 2 lines
Best Regards,
  Yi Zhang
#22 lore raw
In reply to Ming Lei

Just trying to get back into this: I think we're all in agreement that the approach is fine now, aren't we?

Ming, can you respin the series to cover all transport and not just PCIe, please?

author
#23 lore raw
Quoted 2 lines · Christoph Hellwig

Just trying to get back into this: I think we're all in agreement that the approach is fine now, aren't we?

Accurately the approach in patch 2 is fine, but there is still problem in patch 1.

Quoted 3 lines

Ming, can you respin the series to cover all transport and not just PCIe, please?

OK.

Thanks,
Ming