Skip to contentSkip to conversation

nvme: Block ns inventory changes during admin

PATCH New Keith Busch 2 participants 8 messages Jan 25, 2019 – Jan 28, 2019 2 messages not in the archive NVMe

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 nvme: Block ns inventory changes during admin

The driver quiesces IO when the admin command effects say it may alter namespaces.

A device that supports nvme namespace management may trigger a namespace scan automatically. If this changes the namespace inventory by adding more namespace, the driver's unquiesce sequence at the end of sending the admin command may try to unfreeze a queue that isn't frozen, breaking the freeze depth for future use.

Fix that by preventing scan work from altering the namespace list while we are pairing freeze and unfreeze operations.

Reported-by Wen Xiong <wenxiong at us.ibm.com>
Signed-off-by Keith Busch <keith.busch at intel.com>
---
 drivers/nvme/host/core.c | 8 +++++++-
 drivers/nvme/host/nvme.h | 1 +
 2 files changed, 8 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +7 -1
index 150e49723c15..6a9dd68c0f4f 100644
@@ -1253,6 +1253,7 @@ static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	 * effects say only one namespace is affected.
 	 */
 	if (effects & (NVME_CMD_EFFECTS_LBCC | NVME_CMD_EFFECTS_CSE_MASK)) {
+		mutex_lock(&ctrl->scan_lock);
 		nvme_start_freeze(ctrl);
 		nvme_wait_freeze(ctrl);
 	}
@@ -1281,8 +1282,10 @@ static void nvme_passthru_end(struct nvme_ctrl *ctrl, u32 effects)
 	 */
 	if (effects & NVME_CMD_EFFECTS_LBCC)
 		nvme_update_formats(ctrl);
-	if (effects & (NVME_CMD_EFFECTS_LBCC | NVME_CMD_EFFECTS_CSE_MASK))
+	if (effects & (NVME_CMD_EFFECTS_LBCC | NVME_CMD_EFFECTS_CSE_MASK)) {
 		nvme_unfreeze(ctrl);
+		mutex_unlock(&ctrl->scan_lock);
+	}
 	if (effects & NVME_CMD_EFFECTS_CCC)
 		nvme_init_identify(ctrl);
 	if (effects & (NVME_CMD_EFFECTS_NIC | NVME_CMD_EFFECTS_NCC))
@@ -3401,6 +3404,7 @@ static void nvme_scan_work(struct work_struct *work)
 	if (nvme_identify_ctrl(ctrl, &id))
 		return;
 
+	mutex_lock(&ctrl->scan_lock);
 	nn = le32_to_cpu(id->nn);
 	if (ctrl->vs >= NVME_VS(1, 1, 0) &&
 	    !(ctrl->quirks & NVME_QUIRK_IDENTIFY_CNS)) {
@@ -3409,6 +3413,7 @@ static void nvme_scan_work(struct work_struct *work)
 	}
 	nvme_scan_ns_sequential(ctrl, nn);
 out_free_id:
+	mutex_unlock(&ctrl->scan_lock);
 	kfree(id);
 	down_write(&ctrl->namespaces_rwsem);
 	list_sort(NULL, &ctrl->namespaces, ns_cmp);
@@ -3652,6 +3657,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 
 	ctrl->state = NVME_CTRL_NEW;
 	spin_lock_init(&ctrl->lock);
+	mutex_init(&ctrl->scan_lock);
 	INIT_LIST_HEAD(&ctrl->namespaces);
 	init_rwsem(&ctrl->namespaces_rwsem);
 	ctrl->dev = dev;
drivers/nvme/host/nvme.h +1 -0
index ab961bdeea89..c4a1bb41abf0 100644
@@ -154,6 +154,7 @@ struct nvme_ctrl {
 	enum nvme_ctrl_state state;
 	bool identified;
 	spinlock_t lock;
+	struct mutex scan_lock;
 	const struct nvme_ctrl_ops *ops;
 	struct request_queue *admin_q;
 	struct request_queue *connect_q;
Signature · 1 line
2.14.4

A message that is not in the archive (sent off-list or not yet received): OF5147C249.BC64CC03-ON0025838D.007F28EE-8625838D.007F6785@notes.na.collabserv.com

#2 lore raw

PATCH 1/1 nvme: Block ns inventory changes during admin

[replying in plain-text for mailing list posterity]

Quoted 11 lines · Wen Xiong

Hi Keith,

I have tested the patch and fixed the two hung issues on our system.

Quoted 2 lines
Quoted 2 lines
Reported-by Wen Xiong <wenxiong at us.ibm.com>
Signed-off-by Keith Busch <keith.busch at intel.com>
Tested-by Wen Xiong<wenxiong at us.ibm.com>

Thanks,
Wendy

Thank you for confirming this fixes your issue.

A message that is not in the archive (sent off-list or not yet received): BYAPR04MB45021C05936FF7EF982D579686940@BYAPR04MB4502.namprd04.prod.outlook.com

#4 lore raw

PATCH 1/1 nvme: Block ns inventory changes during admin

Quoted 3 lines · Chaitanya Kulkarni

Looks good to me. Just thinking out loud we can only unfreeze the namespace if that is frozen in nvme_unfreeze(ctrl)? We can mark it freeze in the nvme_wait_freeze().

Adding a freeze state to namespaces may work too if there are no callers making use of a layer freeze depth.

Quoted 25 lines
Reviewed-by Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

???????????????????????????????????????????????????????????????????????????????
From: Linux-nvme <linux-nvme-bounces at lists.infradead.org> on behalf of Keith Busch <keith.busch at intel.com>
Sent: Friday, January 25, 2019 3:37:43 PM
To: Wen Xiong
Cc: Christoph Hellwig; linux-nvme at lists.infradead.org; Sagi Grimberg
Subject: Re: [PATCH] nvme: Block ns inventory changes during admin

[replying in plain-text for mailing list posterity]

Quoted 11 lines · Wen Xiong

Hi Keith,

I have tested the patch and fixed the two hung issues on our system.

Quoted 2 lines
Quoted 2 lines
Reported-by Wen Xiong <wenxiong at us.ibm.com>
Signed-off-by Keith Busch <keith.busch at intel.com>
Tested-by Wen Xiong<wenxiong at us.ibm.com>

Thanks,
Wendy

Thank you for confirming this fixes your issue.

#5 lore raw

PATCH 1/1 nvme: Block ns inventory changes during admin

In reply to Keith Busch
Quoted 11 lines · Keith Busch

The driver quiesces IO when the admin command effects say it may alter namespaces.

A device that supports nvme namespace management may trigger a namespace scan automatically. If this changes the namespace inventory by adding more namespace, the driver's unquiesce sequence at the end of sending the admin command may try to unfreeze a queue that isn't frozen, breaking the freeze depth for future use.

Fix that by preventing scan work from altering the namespace list while we are pairing freeze and unfreeze operations.

Do you have a reference for the bug report?

#6 lore raw

PATCH 1/1 nvme: Block ns inventory changes during admin

Quoted 10 lines · Christoph Hellwig
Quoted 9 lines · Keith Busch

The driver quiesces IO when the admin command effects say it may alter namespaces.

A device that supports nvme namespace management may trigger a namespace scan automatically. If this changes the namespace inventory by adding more namespace, the driver's unquiesce sequence at the end of sending the admin command may try to unfreeze a queue that isn't frozen, breaking the freeze depth for future use.

Quoted 4 lines
Quoted 2 lines

Fix that by preventing scan work from altering the namespace list while we are pairing freeze and unfreeze operations.

Do you have a reference for the bug report?

This was reported internally from Wen. If you prefer, we can open a public bz sighting and append the kernel logs for a changelog Link:.

#7 lore raw

PATCH 1/1 nvme: Block ns inventory changes during admin

Quoted 2 lines · Keith Busch

This was reported internally from Wen. If you prefer, we can open a public bz sighting and append the kernel logs for a changelog Link:.

I don't really need a bugzilla, but I'd like to understand the reported problem a little better.

#8 lore raw

PATCH 1/1 nvme: Block ns inventory changes during admin

Quoted 6 lines · Christoph Hellwig
Quoted 2 lines · Keith Busch

This was reported internally from Wen. If you prefer, we can open a public bz sighting and append the kernel logs for a changelog Link:.

I don't really need a bugzilla, but I'd like to understand the reported problem a little better.

Sure thing. The reporter is attaching namespaces in a loop. The controller does support the NS Change Notification, so that automatically triggers the namespace scan_work.

Occasionally the namespace scan_work appends the new namespace to the list before the admin command effects handling is completed. The effects handling attempts to unfreeze the namespace, but if it unfreezes the new one, that request_queue freeze depth will be off and we'll hit the warning in blk_mq_unfreeze_queue().

On the next namespace add, we fail to freeze that queue due to the previous bad accounting and deadlock waiting for frozen.

#9 lore raw

PATCH 1/1 nvme: Block ns inventory changes during admin

Quoted 4 lines · Keith Busch

Sure thing. The reporter is attaching namespaces in a loop. The controller does support the NS Change Notification, so that automatically triggers the namespace scan_work.

Quoted 8 lines

Occasionally the namespace scan_work appends the new namespace to the list before the admin command effects handling is completed. The effects handling attempts to unfreeze the namespace, but if it unfreezes the new one, that request_queue freeze depth will be off and we'll hit the warning in blk_mq_unfreeze_queue().

On the next namespace add, we fail to freeze that queue due to the previous bad accounting and deadlock waiting for frozen.

Ok, makes sense. Can you fold this into the patch description?