nvme: fix reconnection fail due to reserved tag allocation
PATCH 1/1 nvme: fix reconnection fail due to reserved tag allocation
From: Chunguang Xu <[email protected]>
We found a issue on production environment while using NVMe over RDMA, admin_q reconnect failed forever while remote target and network is ok. After dig into it, we found it may caused by a ABBA deadlock due to tag allocation. In my case, the tag was hold by a keep alive request waiting inside admin_q, as we quiesced admin_q while reset ctrl, so the request maked as idle and will not process before reset success. As fabric_q shares tagset with admin_q, while reconnect remote target, we need a tag for connect command, but the only one reserved tag was held by keep alive command which waiting inside admin_q. As a result, we failed to reconnect admin_q forever.
In order to workaround this issue, I think we should not retry keep alive request while controller reconnecting, as we have stopped keep alive while resetting controller, and will start it again while init finish, so it maybe ok to drop it.
--- drivers/nvme/host/core.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
drivers/nvme/host/core.c +9 -2
@@ -371,6 +371,8 @@ enum nvme_disposition {static inline enum nvme_disposition nvme_decide_disposition (struct request *req) { +struct nvme_ctrl *ctrl =nvme_req (req)->ctrl; +if (likely (nvme_req (req)->status ==0 ))return COMPLETE ;
@@ -382,6 +384,12 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req) nvme_req(req)->retries >= nvme_max_retries)return COMPLETE ; +if (nvme_ctrl_state (ctrl) !=NVME_CTRL_LIVE ) { +if (nvme_req (req)->cmd->common.opcode == + nvme_admin_keep_alive) +return COMPLETE ; + } +if (req->cmd_flags &REQ_NVME_MPATH ) {if (nvme_is_path_error (nvme_req (req)->status) ||blk_queue_dying (req->q))
@@ -1296,8 +1304,7 @@ static enum rq_end_io_ret nvme_keep_alive_end_io(struct request *rq, ctrl->ka_last_check_time = jiffies; ctrl->comp_seen = false;spin_lock_irqsave (&ctrl->lock, flags); -if (ctrl->state ==NVME_CTRL_LIVE || - ctrl->state ==NVME_CTRL_CONNECTING ) +if (ctrl->state ==NVME_CTRL_LIVE ) startka = true;spin_unlock_irqrestore (&ctrl->lock, flags);if (startka)
Signature · 1 line
2.25.1
Sagi Grimberg
maintainerQuoted 15 lines · brookxu.cn
From: Chunguang Xu <[email protected]>
We found a issue on production environment while using NVMe over RDMA,
admin_qreconnect failed forever while remote target and network is ok. After dig into it, we found it may caused by a ABBA deadlock due to tag allocation. In my case, the tag was hold by a keep alive request waiting insideadmin_q, as we quiescedadmin_qwhile reset ctrl, so the request maked as idle and will not process before reset success. Asfabric_qshares tagset withadmin_q, while reconnect remote target, we need a tag for connect command, but the only one reserved tag was held by keep alive command which waiting insideadmin_q. As a result, we failed to reconnectadmin_qforever.
Quoted 5 lines
In order to workaround this issue, I think we should not retry keep alive request while controller reconnecting, as we have stopped keep alive while resetting controller, and will start it again while init finish, so it maybe ok to drop it.
This is the wrong fix.
First we should note that this is a regression caused by:ed01fee283a0 ("nvme-fabrics: only reserve a single tag")
Then, you need to restore reserving two tags for the admin tagset.
Thanks for review, seems that we should revert this patch ed01fee283a0, ed01fee283a0 seems just a alone 'optimization'. If no double, I will send another patch.
Thanks
Quoted 31 lines · Sagi Grimberg <[email protected]> 于2024年3月7日周四 17:36写道:
Quoted 15 lines · brookxu.cn
From: Chunguang Xu <[email protected]>
We found a issue on production environment while using NVMe over RDMA,
admin_qreconnect failed forever while remote target and network is ok. After dig into it, we found it may caused by a ABBA deadlock due to tag allocation. In my case, the tag was hold by a keep alive request waiting insideadmin_q, as we quiescedadmin_qwhile reset ctrl, so the request maked as idle and will not process before reset success. Asfabric_qshares tagset withadmin_q, while reconnect remote target, we need a tag for connect command, but the only one reserved tag was held by keep alive command which waiting insideadmin_q. As a result, we failed to reconnectadmin_qforever.Quoted 5 lines
In order to workaround this issue, I think we should not retry keep alive request while controller reconnecting, as we have stopped keep alive while resetting controller, and will start it again while init finish, so it maybe ok to drop it.
This is the wrong fix.
First we should note that this is a regression caused by:ed01fee283a0("nvme-fabrics: only reserve a single tag")Then, you need to restore reserving two tags for the admin tagset.
Sagi Grimberg
maintainerQuoted 3 lines · 许春光
Thanks for review, seems that we should revert this patch
ed01fee283a0,ed01fee283a0seems just a alone 'optimization'. If no double, I will send another patch.
Not a revert, but a fix with a Fixes tag. Just use NVMF_ADMIN_RESERVED_TAGS and NVMF_IO_RESERVED_TAGS.
Quoted 33 lines
Thanks
Quoted 29 lines · Sagi Grimberg <[email protected]> 于2024年3月7日周四 17:36写道:
Quoted 15 lines · brookxu.cn
From: Chunguang Xu <[email protected]>
We found a issue on production environment while using NVMe over RDMA,
admin_qreconnect failed forever while remote target and network is ok. After dig into it, we found it may caused by a ABBA deadlock due to tag allocation. In my case, the tag was hold by a keep alive request waiting insideadmin_q, as we quiescedadmin_qwhile reset ctrl, so the request maked as idle and will not process before reset success. Asfabric_qshares tagset withadmin_q, while reconnect remote target, we need a tag for connect command, but the only one reserved tag was held by keep alive command which waiting insideadmin_q. As a result, we failed to reconnectadmin_qforever.Quoted 5 lines
In order to workaround this issue, I think we should not retry keep alive request while controller reconnecting, as we have stopped keep alive while resetting controller, and will start it again while init finish, so it maybe ok to drop it.
This is the wrong fix.
First we should note that this is a regression caused by:ed01fee283a0("nvme-fabrics: only reserve a single tag")Then, you need to restore reserving two tags for the admin tagset.