Skip to contentSkip to conversation

nvme: don't call blk_mq_{,un}quiesce_tagset when ctrl->tagset is NULL

PATCH In review Christoph Hellwig 4 participants 13 messages Nov 16, 2022 – Nov 30, 2022 NVMe

lore
Share & tools
lore

Christoph Hellwig

author maintainer
#0 lore raw

PATCH 1/1 nvme: don't call blk_mq_{,un}quiesce_tagset when ctrl->tagset is NULL

The NVMe drivers support a mode where no tagset is allocated for the I/O queues and only the admin queue is usable. In that case ctrl->tagset is NULL and we must not call the block per-tagset quiesce helpers that dereference it.

Fixes 98d81f0df70c ("nvme: use blk_mq_[un]quiesce_tagset")

---

It turns out we have a bunch of other quiesce calls that can happen when only the admin queue is live. So bring back the original quick fix to check for the tagset pointer in the quiesce helpers.

 drivers/nvme/host/core.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/nvme/host/core.c +4 -0
index 3195ae17df309..fd2e26cb7a688 100644
@@ -5215,6 +5215,8 @@ EXPORT_SYMBOL_GPL(nvme_start_freeze);
 
 void nvme_quiesce_io_queues(struct nvme_ctrl *ctrl)
 {
+	if (!ctrl->tagset)
+		return;
 	if (!test_and_set_bit(NVME_CTRL_STOPPED, &ctrl->flags))
 		blk_mq_quiesce_tagset(ctrl->tagset);
 	else
@@ -5224,6 +5226,8 @@ EXPORT_SYMBOL_GPL(nvme_quiesce_io_queues);
 
 void nvme_unquiesce_io_queues(struct nvme_ctrl *ctrl)
 {
+	if (!ctrl->tagset)
+		return;
 	if (test_and_clear_bit(NVME_CTRL_STOPPED, &ctrl->flags))
 		blk_mq_unquiesce_tagset(ctrl->tagset);
 }
Signature · 1 line
2.30.2
Reviewed-by on v1 1/1
#1 lore raw
Quoted 40 lines · Christoph Hellwig

The NVMe drivers support a mode where no tagset is allocated for the I/O queues and only the admin queue is usable. In that case ctrl->tagset is NULL and we must not call the block per-tagset quiesce helpers that dereference it.

Fixes 98d81f0df70c ("nvme: use blk_mq_[un]quiesce_tagset")

---

It turns out we have a bunch of other quiesce calls that can happen when only the admin queue is live. So bring back the original quick fix to check for the tagset pointer in the quiesce helpers.

  drivers/nvme/host/core.c | 4 ++++
  1 file changed, 4 insertions(+)
drivers/nvme/host/core.c +4 -0
index 3195ae17df309..fd2e26cb7a688 100644
@@ -5215,6 +5215,8 @@ EXPORT_SYMBOL_GPL(nvme_start_freeze);
 
  void nvme_quiesce_io_queues(struct nvme_ctrl *ctrl)
  {
+	if (!ctrl->tagset)
+		return;
  	if (!test_and_set_bit(NVME_CTRL_STOPPED, &ctrl->flags))
  		blk_mq_quiesce_tagset(ctrl->tagset);
  	else
@@ -5224,6 +5226,8 @@ EXPORT_SYMBOL_GPL(nvme_quiesce_io_queues);
 
  void nvme_unquiesce_io_queues(struct nvme_ctrl *ctrl)
  {
+	if (!ctrl->tagset)
+		return;
  	if (test_and_clear_bit(NVME_CTRL_STOPPED, &ctrl->flags))
  		blk_mq_unquiesce_tagset(ctrl->tagset);
  }
maintainer
#3 lore raw
In reply to Christoph Hellwig
Quoted 22 lines · Christoph Hellwig

The NVMe drivers support a mode where no tagset is allocated for the I/O queues and only the admin queue is usable. In that case ctrl->tagset is NULL and we must not call the block per-tagset quiesce helpers that dereference it.

Fixes 98d81f0df70c ("nvme: use blk_mq_[un]quiesce_tagset")

---

It turns out we have a bunch of other quiesce calls that can happen when only the admin queue is live. So bring back the original quick fix to check for the tagset pointer in the quiesce helpers.

  drivers/nvme/host/core.c | 4 ++++
  1 file changed, 4 insertions(+)
drivers/nvme/host/core.c +0 -0
index 3195ae17df309..fd2e26cb7a688 100644

drivers/nvme/host/core.c EXPORT_SYMBOL_GPL(nvme_start_freeze); · line 5219

Quoted 5 lines
drivers/nvme/host/core.c +2 -0
@@ -5215,6 +5215,8 @@ EXPORT_SYMBOL_GPL(nvme_start_freeze);
 
  void nvme_quiesce_io_queues(struct nvme_ctrl *ctrl)
  {
+	if (!ctrl->tagset)
+		return;

Can this be placed in the pci code instead?

Quoted 12 lines
drivers/nvme/host/core.c +2 -0
  	if (!test_and_set_bit(NVME_CTRL_STOPPED, &ctrl->flags))
  		blk_mq_quiesce_tagset(ctrl->tagset);
  	else
@@ -5224,6 +5226,8 @@ EXPORT_SYMBOL_GPL(nvme_quiesce_io_queues);
 
  void nvme_unquiesce_io_queues(struct nvme_ctrl *ctrl)
  {
+	if (!ctrl->tagset)
+		return;
  	if (test_and_clear_bit(NVME_CTRL_STOPPED, &ctrl->flags))
  		blk_mq_unquiesce_tagset(ctrl->tagset);
  }
author maintainer
#4 lore raw
Quoted 5 lines · Sagi Grimberg

Review

Quoted 3 lines
  {
(quoted diff) +2 -0
+	if (!ctrl->tagset)
+		return;

Can this be placed in the pci code instead?

There's nothing PCIe-specific about this case. All fabrics drivers support controllers with I/O queues and thus the I/O tagset, e.g. discovery controllers.

maintainer
#5 lore raw
Quoted 10 lines
Quoted 6 lines

Review

Quoted 4 lines
   {
(quoted diff) +2 -0
+	if (!ctrl->tagset)
+		return;

Can this be placed in the pci code instead?

There's nothing PCIe-specific about this case. All fabrics drivers support controllers with I/O queues and thus the I/O tagset, e.g. discovery controllers.

All the other drivers check that they have IO queues before quiescing/unquiescing them.

author maintainer
#6 lore raw
Quoted 2 lines · Sagi Grimberg

All the other drivers check that they have IO queues before quiescing/unquiescing them.

The core firmware activation code does not.

maintainer
#7 lore raw
Quoted 5 lines
Quoted 3 lines

All the other drivers check that they have IO queues before quiescing/unquiescing them.

The core firmware activation code does not.

Well, ideally it should, but maybe it would be more readable if we condition on ctrl->queue_count > 1.

author maintainer
#8 lore raw
Quoted 8 lines · Sagi Grimberg
Quoted 5 lines
Quoted 3 lines

All the other drivers check that they have IO queues before quiescing/unquiescing them.

The core firmware activation code does not.

Well, ideally it should, but maybe it would be more readable if we condition on ctrl->queue_count > 1.

I find this more confusing to be honest, but if we get a consensus for that I can live with it.

author maintainer
#9 lore raw
In reply to Sagi Grimberg
Quoted 8 lines · Sagi Grimberg
Quoted 5 lines
Quoted 3 lines

All the other drivers check that they have IO queues before quiescing/unquiescing them.

The core firmware activation code does not.

Well, ideally it should, but maybe it would be more readable if we condition on ctrl->queue_count > 1.

This would be the queue_count version. I have to say I much prefer the tagset check, as it is self-documenting and matches what nvme_queue_scan, nvme_cancel_tagset and nvme_cancel_admin_tagset do.

author maintainer
#10 lore raw
Quoted 4 lines · Christoph Hellwig

This would be the queue_count version. I have to say I much prefer the tagset check, as it is self-documenting and matches what nvme_queue_scan, nvme_cancel_tagset and nvme_cancel_admin_tagset do.

And here is the actual patch:

---
From 167d14d9613014142ee285c2912dadc089d80e86 Mon Sep 17 00:00:00 2001
From: Christoph Hellwig <[email protected]>
Date: Wed, 16 Nov 2022 08:14:46 +0100
Subject: nvme: don't call blk_mq_{,un}quiesce_tagset when ctrl->tagset is NULL

The NVMe drivers support a mode where no tagset is allocated for the I/O queues and only the admin queue is usable. In that case ctrl->queue_count is smaller than two and we must not call the block per-tagset quiesce helpers that dereference it.

Fixes 98d81f0df70c ("nvme: use blk_mq_[un]quiesce_tagset")
---
 drivers/nvme/host/core.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/nvme/host/core.c +4 -0
index 3195ae17df309..6c8a110d6738a 100644
@@ -5215,6 +5215,8 @@ EXPORT_SYMBOL_GPL(nvme_start_freeze);
 
 void nvme_quiesce_io_queues(struct nvme_ctrl *ctrl)
 {
+	if (ctrl->queue_count < 2)
+		return;
 	if (!test_and_set_bit(NVME_CTRL_STOPPED, &ctrl->flags))
 		blk_mq_quiesce_tagset(ctrl->tagset);
 	else
@@ -5224,6 +5226,8 @@ EXPORT_SYMBOL_GPL(nvme_quiesce_io_queues);
 
 void nvme_unquiesce_io_queues(struct nvme_ctrl *ctrl)
 {
+	if (ctrl->queue_count < 2)
+		return;
 	if (test_and_clear_bit(NVME_CTRL_STOPPED, &ctrl->flags))
 		blk_mq_unquiesce_tagset(ctrl->tagset);
 }
Signature · 1 line
2.30.2
author maintainer
#11 lore raw
In reply to Christoph Hellwig
Quoted 4 lines · Christoph Hellwig

This would be the queue_count version. I have to say I much prefer the tagset check, as it is self-documenting and matches what nvme_queue_scan, nvme_cancel_tagset and nvme_cancel_admin_tagset do.

So no comments so far. Unless I hear more feedback I'd prefer to go with the ->tagset checks that matches what the other code is doing and also is more readable.

maintainer
#12 lore raw
Quoted 9 lines
Quoted 5 lines

This would be the queue_count version. I have to say I much prefer the tagset check, as it is self-documenting and matches what nvme_queue_scan, nvme_cancel_tagset and nvme_cancel_admin_tagset do.

So no comments so far. Unless I hear more feedback I'd prefer to go with the ->tagset checks that matches what the other code is doing and also is more readable.

OK, I'm not opposed to that anymore. We can take care of this area on a different occasion.