nvme: fix reconnection fail due to reserved tag allocation
PATCH v2 1/1 nvme: fix reconnection fail due to reserved tag allocation
From: Chunguang Xu <[email protected]>
We found a issue on production environment while using NVMe over RDMA, admin_q reconnect failed forever while remote target and network is ok. After dig into it, we found it may caused by a ABBA deadlock due to tag allocation. In my case, the tag was hold by a keep alive request waiting inside admin_q, as we quiesced admin_q while reset ctrl, so the request maked as idle and will not process before reset success. As fabric_q shares tagset with admin_q, while reconnect remote target, we need a tag for connect command, but the only one reserved tag was held by keep alive command which waiting inside admin_q. As a result, we failed to reconnect admin_q forever. In order to fix this issue, I think we should keep two reserved tags for admin queue.
--- drivers/nvme/host/core.c | 4 ++-- drivers/nvme/host/fabrics.h | 10 ++++------ 2 files changed, 6 insertions(+), 8 deletions(-)
drivers/nvme/host/core.c +2 -2
@@ -4359,7 +4359,7 @@ int nvme_alloc_admin_tag_set(struct nvme_ctrl *ctrl, struct blk_mq_tag_set *set, set->ops = ops; set->queue_depth =NVME_AQ_MQ_TAG_DEPTH ;if (ctrl->ops->flags &NVME_F_FABRICS ) - set->reserved_tags =NVMF_RESERVED_TAGS ; + set->reserved_tags =NVMF_ADMIN_RESERVED_TAGS ; set->numa_node = ctrl->numa_node; set->flags =BLK_MQ_F_NO_SCHED ;if (ctrl->ops->flags &NVME_F_BLOCKING )
@@ -4428,7 +4428,7 @@ int nvme_alloc_io_tag_set(struct nvme_ctrl *ctrl, struct blk_mq_tag_set *set,if (ctrl->quirks &NVME_QUIRK_SHARED_TAGS ) set->reserved_tags =NVME_AQ_DEPTH ;else if (ctrl->ops->flags &NVME_F_FABRICS ) - set->reserved_tags =NVMF_RESERVED_TAGS ; + set->reserved_tags =NVMF_IO_RESERVED_TAGS ; set->numa_node = ctrl->numa_node; set->flags =BLK_MQ_F_SHOULD_MERGE ;if (ctrl->ops->flags &NVME_F_BLOCKING )
drivers/nvme/host/fabrics.h +4 -6
@@ -18,12 +18,10 @@/* default is -1: the fail fast mechanism is disabled */ #define NVMF_DEF_FAIL_FAST_TMO -1 -/* -* Reserved one command for internal usage. This command is used for sending -* the connect command, as well as for the keep alive command on the admin -* queue once live. -*/ -#define NVMF_RESERVED_TAGS 1 +/* Reserved for connect */ +#define NVMF_IO_RESERVED_TAGS 1 +/* Reserved for connect and keep alive */ +#define NVMF_ADMIN_RESERVED_TAGS 2/* * Define a host as seen by the target. We allocate one at boot, but also
Signature · 1 line
2.25.1
Quoted 17 lines · brookxu.cn
From: Chunguang Xu <[email protected]>
We found a issue on production environment while using NVMe over RDMA,
admin_qreconnect failed forever while remote target and network is ok. After dig into it, we found it may caused by a ABBA deadlock due to tag allocation. In my case, the tag was hold by a keep alive request waiting insideadmin_q, as we quiescedadmin_qwhile reset ctrl, so the request maked as idle and will not process before reset success. Asfabric_qshares tagset withadmin_q, while reconnect remote target, we need a tag for connect command, but the only one reserved tag was held by keep alive command which waiting insideadmin_q. As a result, we failed to reconnectadmin_qforever. In order to fix this issue, I think we should keep two reserved tags for admin queue.
Quoted 2 lines
Fixes ed01fee283a0 ("nvme-fabrics: only reserve a single tag")Signed-off-by Chunguang Xu <[email protected]>
Quoted 16 lines · brookxu.cn
From: Chunguang Xu <[email protected]>
We found a issue on production environment while using NVMe over RDMA,
admin_qreconnect failed forever while remote target and network is ok. After dig into it, we found it may caused by a ABBA deadlock due to tag allocation. In my case, the tag was hold by a keep alive request waiting insideadmin_q, as we quiescedadmin_qwhile reset ctrl, so the request maked as idle and will not process before reset success. Asfabric_qshares tagset withadmin_q, while reconnect remote target, we need a tag for connect command, but the only one reserved tag was held by keep alive command which waiting insideadmin_q. As a result, we failed to reconnectadmin_qforever. In order to fix this issue, I think we should keep two reserved tags for admin queue.
plz consider rearranged line length, no change in wording to use the full length :-
We found a issue on production environment while using NVMe over RDMA, admin_q reconnect failed forever while remote target and network is ok. After dig into it, we found it may caused by a ABBA deadlock due to tag allocation. In my case, the tag was hold by a keep alive request waiting inside admin_q, as we quiesced admin_q while reset ctrl, so the request maked as idle and will not process before reset success. As fabric_q shares tagset with admin_q, while reconnect remote target, we need a tag for connect command, but the only one reserved tag was held by keep alive command which waiting inside admin_q. As a result, we failed to reconnect admin_q forever. In order to fix this issue, I think we should keep two reserved tags for admin queue.
Rest of the patch looks good and follows the discussion on V1.
-ck