Skip to contentSkip to conversation

nvme-loop: fixes for concurrent reset and delete

PATCH Applied Hannes Reinecke 3 participants 9 messages May 26, 2021 – Jun 2, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover nvme-loop: fixes for concurrent reset and delete

Hi all,

during concurrent reset and delete a crash might happen as we're not resetting all fields in the loop driver upon reset.

As usual, comments and reviews are welcome.

Hannes Reinecke (4):
nvme/loop: reset queue count to 1 in nvme_loop_destroy_io_queues() nvme/loop: clear NVME_LOOP_Q_LIVE when
nvme_loop_configure_admin_queue() fails
nvme/loop: check for NVME_LOOP_Q_LIVE in
nvme_loop_destroy_admin_queue()
nvme/loop: Do not warn for deleted controllers during reset

 drivers/nvme/target/loop.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
Signature · 7 lines
2.29.2


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#1 lore raw

PATCH 1/4 nvme/loop: reset queue count to 1 in nvme_loop_destroy_io_queues()

The queue count is increased in nvme_loop_init_io_queues(), so we need to reset it to 1 at the end of nvme_loop_destroy_io_queues(). Otherwise the function is not re-entrant safe, and crash will happen during concurrent reset and remove calls.

---
 drivers/nvme/target/loop.c | 1 +
 1 file changed, 1 insertion(+)
drivers/nvme/target/loop.c +1 -0
index 74b3b150e1a5..f96ec84db8c2 100644
@@ -299,6 +299,7 @@ static void nvme_loop_destroy_io_queues(struct nvme_loop_ctrl *ctrl)
 		clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[i].flags);
 		nvmet_sq_destroy(&ctrl->queues[i].nvme_sq);
 	}
+	ctrl->ctrl.queue_count = 1;
 }
 
 static int nvme_loop_init_io_queues(struct nvme_loop_ctrl *ctrl)
Signature · 7 lines
2.29.2


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Reviewed-by on v1 1/4
#2 lore raw

Re: [PATCH 1/4] nvme/loop: reset queue count to 1 in nvme_loop_destroy_io_queues()

Quoted 6 lines · Hannes Reinecke

The queue count is increased in nvme_loop_init_io_queues(), so we need to reset it to 1 at the end of nvme_loop_destroy_io_queues(). Otherwise the function is not re-entrant safe, and crash will happen during concurrent reset and remove calls.

Looks good.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#3 lore raw

PATCH 2/4 nvme/loop: clear NVME_LOOP_Q_LIVE when nvme_loop_configure_admin_queue() fails

In reply to Hannes Reinecke

When the call to nvme_enable_ctrl() in nvme_loop_configure_admin_queue() fails the NVME_LOOP_Q_LIVE flag is not cleared.

---
 drivers/nvme/target/loop.c | 1 +
 1 file changed, 1 insertion(+)
drivers/nvme/target/loop.c +1 -0
index f96ec84db8c2..8e6d1ea3fd9e 100644
@@ -406,6 +406,7 @@ static int nvme_loop_configure_admin_queue(struct nvme_loop_ctrl *ctrl)
 	return 0;
 
 out_cleanup_queue:
+	clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags);
 	blk_cleanup_queue(ctrl->ctrl.admin_q);
 out_cleanup_fabrics_q:
 	blk_cleanup_queue(ctrl->ctrl.fabrics_q);
Signature · 7 lines
2.29.2


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Reviewed-by on v1 2/4
#4 lore raw

Re: [PATCH 2/4] nvme/loop: clear NVME_LOOP_Q_LIVE when nvme_loop_configure_admin_queue() fails

Quoted 4 lines · Hannes Reinecke

When the call to nvme_enable_ctrl() in nvme_loop_configure_admin_queue() fails the NVME_LOOP_Q_LIVE flag is not cleared.

Looks good.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#5 lore raw

PATCH 3/4 nvme/loop: check for NVME_LOOP_Q_LIVE in nvme_loop_destroy_admin_queue()

In reply to Hannes Reinecke

We need to check the NVME_LOOP_Q_LIVE flag in nvme_loop_destroy_admin_queue() to protect against duplicate invocations eg during concurrent reset and remove calls.

---
 drivers/nvme/target/loop.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/target/loop.c +2 -1
index 8e6d1ea3fd9e..c2f3e0b7c4ea 100644
@@ -263,7 +263,8 @@ static const struct blk_mq_ops nvme_loop_admin_mq_ops = {
 
 static void nvme_loop_destroy_admin_queue(struct nvme_loop_ctrl *ctrl)
 {
-	clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags);
+	if (!test_and_clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags))
+		return;
 	nvmet_sq_destroy(&ctrl->queues[0].nvme_sq);
 	blk_cleanup_queue(ctrl->ctrl.admin_q);
 	blk_cleanup_queue(ctrl->ctrl.fabrics_q);
Signature · 7 lines
2.29.2


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Reviewed-by on v1 3/4
#6 lore raw

Re: [PATCH 3/4] nvme/loop: check for NVME_LOOP_Q_LIVE in nvme_loop_destroy_admin_queue()

Quoted 5 lines · Hannes Reinecke

We need to check the NVME_LOOP_Q_LIVE flag in nvme_loop_destroy_admin_queue() to protect against duplicate invocations eg during concurrent reset and remove calls.

It will be great if we can document here what is the concurrent reset scenario is make understanding of the patch much easy.

Looks good.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#7 lore raw

PATCH 4/4 nvme/loop: Do not warn for deleted controllers during reset

In reply to Hannes Reinecke

During concurrent reset and delete calls the reset workqueue is flushed, causing nvme_loop_reset_ctrl_work() to be executed when the controller is in state DELETING or DELETING_NOIO. But this is expected, so we shouldn't issue a WARN_ON here.

---
 drivers/nvme/target/loop.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/nvme/target/loop.c +4 -2
index c2f3e0b7c4ea..9838a7d27bc1 100644
@@ -465,8 +465,10 @@ static void nvme_loop_reset_ctrl_work(struct work_struct *work)
 	nvme_loop_shutdown_ctrl(ctrl);
 
 	if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_CONNECTING)) {
-		/* state change failure should never happen */
-		WARN_ON_ONCE(1);
+		if (ctrl->ctrl.state != NVME_CTRL_DELETING &&
+		    ctrl->ctrl.state != NVME_CTRL_DELETING_NOIO)
+			/* state change failure for non-deleted ctrl? */
+			WARN_ON_ONCE(1);
 		return;
 	}
 
Signature · 7 lines
2.29.2


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme