Skip to contentSkip to conversation

nvme: fix reconnection fail due to reserved tag allocation

PATCH Superseded brookxu.cn 2 participants 4 messages Feb 28, 2024 – Mar 7, 2024 NVMe

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

brookxu.cn

author
#0 lore raw

PATCH 1/1 nvme: fix reconnection fail due to reserved tag allocation

From: Chunguang Xu <[email protected]>

We found a issue on production environment while using NVMe over RDMA, admin_q reconnect failed forever while remote target and network is ok. After dig into it, we found it may caused by a ABBA deadlock due to tag allocation. In my case, the tag was hold by a keep alive request waiting inside admin_q, as we quiesced admin_q while reset ctrl, so the request maked as idle and will not process before reset success. As fabric_q shares tagset with admin_q, while reconnect remote target, we need a tag for connect command, but the only one reserved tag was held by keep alive command which waiting inside admin_q. As a result, we failed to reconnect admin_q forever.

In order to workaround this issue, I think we should not retry keep alive request while controller reconnecting, as we have stopped keep alive while resetting controller, and will start it again while init finish, so it maybe ok to drop it.

---
 drivers/nvme/host/core.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
drivers/nvme/host/core.c +9 -2
index 0a96362912ce..07ed2b6a75fb 100644
@@ -371,6 +371,8 @@ enum nvme_disposition {
 
 static inline enum nvme_disposition nvme_decide_disposition(struct request *req)
 {
+	struct nvme_ctrl *ctrl = nvme_req(req)->ctrl;
+
 	if (likely(nvme_req(req)->status == 0))
 		return COMPLETE;
 
@@ -382,6 +384,12 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req)
 	    nvme_req(req)->retries >= nvme_max_retries)
 		return COMPLETE;
 
+	if (nvme_ctrl_state(ctrl) != NVME_CTRL_LIVE) {
+		if (nvme_req(req)->cmd->common.opcode ==
+		    nvme_admin_keep_alive)
+			return COMPLETE;
+	}
+
 	if (req->cmd_flags & REQ_NVME_MPATH) {
 		if (nvme_is_path_error(nvme_req(req)->status) ||
 		    blk_queue_dying(req->q))
@@ -1296,8 +1304,7 @@ static enum rq_end_io_ret nvme_keep_alive_end_io(struct request *rq,
 	ctrl->ka_last_check_time = jiffies;
 	ctrl->comp_seen = false;
 	spin_lock_irqsave(&ctrl->lock, flags);
-	if (ctrl->state == NVME_CTRL_LIVE ||
-	    ctrl->state == NVME_CTRL_CONNECTING)
+	if (ctrl->state == NVME_CTRL_LIVE)
 		startka = true;
 	spin_unlock_irqrestore(&ctrl->lock, flags);
 	if (startka)
Signature · 1 line
2.25.1
maintainer
#1 lore raw
Quoted 15 lines · brookxu.cn

From: Chunguang Xu <[email protected]>

We found a issue on production environment while using NVMe over RDMA, admin_q reconnect failed forever while remote target and network is ok. After dig into it, we found it may caused by a ABBA deadlock due to tag allocation. In my case, the tag was hold by a keep alive request waiting inside admin_q, as we quiesced admin_q while reset ctrl, so the request maked as idle and will not process before reset success. As fabric_q shares tagset with admin_q, while reconnect remote target, we need a tag for connect command, but the only one reserved tag was held by keep alive command which waiting inside admin_q. As a result, we failed to reconnect admin_q forever.

Quoted 5 lines

In order to workaround this issue, I think we should not retry keep alive request while controller reconnecting, as we have stopped keep alive while resetting controller, and will start it again while init finish, so it maybe ok to drop it.

This is the wrong fix.
First we should note that this is a regression caused by:
ed01fee283a0 ("nvme-fabrics: only reserve a single tag")

Then, you need to restore reserving two tags for the admin tagset.

author
#2 lore raw

Thanks for review, seems that we should revert this patch ed01fee283a0, ed01fee283a0 seems just a alone 'optimization'. If no double, I will send another patch.

Thanks

Quoted 31 lines · Sagi Grimberg <[email protected]> 于2024年3月7日周四 17:36写道:
Quoted 15 lines · brookxu.cn

From: Chunguang Xu <[email protected]>

We found a issue on production environment while using NVMe over RDMA, admin_q reconnect failed forever while remote target and network is ok. After dig into it, we found it may caused by a ABBA deadlock due to tag allocation. In my case, the tag was hold by a keep alive request waiting inside admin_q, as we quiesced admin_q while reset ctrl, so the request maked as idle and will not process before reset success. As fabric_q shares tagset with admin_q, while reconnect remote target, we need a tag for connect command, but the only one reserved tag was held by keep alive command which waiting inside admin_q. As a result, we failed to reconnect admin_q forever.

Quoted 5 lines

In order to workaround this issue, I think we should not retry keep alive request while controller reconnecting, as we have stopped keep alive while resetting controller, and will start it again while init finish, so it maybe ok to drop it.

This is the wrong fix.
First we should note that this is a regression caused by:
ed01fee283a0 ("nvme-fabrics: only reserve a single tag")

Then, you need to restore reserving two tags for the admin tagset.

maintainer
#3 lore raw
Quoted 3 lines · 许春光

Thanks for review, seems that we should revert this patch ed01fee283a0, ed01fee283a0 seems just a alone 'optimization'. If no double, I will send another patch.

Not a revert, but a fix with a Fixes tag. Just use NVMF_ADMIN_RESERVED_TAGS and NVMF_IO_RESERVED_TAGS.

Quoted 33 lines

Thanks

Quoted 29 lines · Sagi Grimberg <[email protected]> 于2024年3月7日周四 17:36写道:
Quoted 15 lines · brookxu.cn

From: Chunguang Xu <[email protected]>

We found a issue on production environment while using NVMe over RDMA, admin_q reconnect failed forever while remote target and network is ok. After dig into it, we found it may caused by a ABBA deadlock due to tag allocation. In my case, the tag was hold by a keep alive request waiting inside admin_q, as we quiesced admin_q while reset ctrl, so the request maked as idle and will not process before reset success. As fabric_q shares tagset with admin_q, while reconnect remote target, we need a tag for connect command, but the only one reserved tag was held by keep alive command which waiting inside admin_q. As a result, we failed to reconnect admin_q forever.

Quoted 5 lines

In order to workaround this issue, I think we should not retry keep alive request while controller reconnecting, as we have stopped keep alive while resetting controller, and will start it again while init finish, so it maybe ok to drop it.

This is the wrong fix.
First we should note that this is a regression caused by:
ed01fee283a0 ("nvme-fabrics: only reserve a single tag")

Then, you need to restore reserving two tags for the admin tagset.