RDMA QP leak on spdk nvmf target

Hi All,
With recent spdk code, RDMA QPs are not destroyed on the nvmf target after each spdk perf run until target is cleared/stopped.

SPDK nvmf target is using drain WR completion logic to destroy the RDMA QP. But even after send/recv drain completions are received RDMA QP is not destroyed as the rqpair->refcnt is 0 in spdk_nvmf_rdma_qpair_destroy().

I believe rqpair refcnt needs to be incremented before spdk_nvmf_rdma_qpair_destroy(). Here is my experimental patch that fixes the issue, Please let me know if this qualifies for a patch.

@@ -2586,9 +2595,10 @@ spdk_nvmf_rdma_poller_poll(struct spdk_nvmf_rdma_transport *rtransport,

	case RDMA_WR_TYPE_DRAIN_RECV:
		rqpair = SPDK_CONTAINEROF(rdma_wr, struct spdk_nvmf_rdma_qpair, drain_recv_wr);
		assert(rqpair->disconnect_flags & RDMA_QP_DISCONNECTING);
		SPDK_DEBUGLOG(SPDK_LOG_RDMA, "Drained QP RECV %u (%p)\n", rqpair->qpair.qid, rqpair);
		              rqpair->disconnect_flags |= RDMA_QP_RECV_DRAINED;
		if (rqpair->disconnect_flags & RDMA_QP_SEND_DRAINED) {
+			spdk_nvmf_rdma_qpair_inc_refcnt(rqpair);
			spdk_nvmf_rdma_qpair_destroy(rqpair);
		}
		/* Continue so that this does not trigger the disconnect path below. */
@@ -2596,9 +2606,10 @@ spdk_nvmf_rdma_poller_poll(struct spdk_nvmf_rdma_transport *rtransport,
	case RDMA_WR_TYPE_DRAIN_SEND:
		rqpair = SPDK_CONTAINEROF(rdma_wr, struct spdk_nvmf_rdma_qpair, drain_send_wr);
		assert(rqpair->disconnect_flags & RDMA_QP_DISCONNECTING);
		SPDK_DEBUGLOG(SPDK_LOG_RDMA, "Drained QP SEND %u (%p)\n", rqpair->qpair.qid, rqpair);
		              rqpair->disconnect_flags |= RDMA_QP_SEND_DRAINED;
		if (rqpair->disconnect_flags & RDMA_QP_RECV_DRAINED) {
+			spdk_nvmf_rdma_qpair_inc_refcnt(rqpair);
			spdk_nvmf_rdma_qpair_destroy(rqpair);
		}
		/* Continue so that this does not trigger the disconnect path below. */

----

Thanks,
Bharat.

Hi,

I believe,
https://github.com/spdk/spdk/commit/7da9f8faba8712e655fc75240b2a2dc6660e0e61
recently merged into "master" should fix the problem.

Best regards

Sasha

Yes. That should fix.
Thanks!

From: Sasha Kotchubievsky
Sent: Tuesday, December 18, 12:51 AM
Subject: Re: [SPDK] RDMA QP leak on spdk nvmf target
To: spdk(a)lists.01.org

Hi, I believe, https://github.com/spdk/spdk/commit/7da9f8faba8712e655fc75240b2a2dc6660e0e61 recently merged into "master" should fix the problem. Best regards Sasha On 12/17/2018 9:02 PM, Potnuri Bharat Teja wrote: > Hi All, > With recent spdk code, RDMA QPs are not destroyed on the nvmf target after each > spdk perf run until target is cleared/stopped. > > SPDK nvmf target is using drain WR completion logic to destroy the RDMA QP. But > even after send/recv drain completions are received RDMA QP is not destroyed as > the rqpair->refcnt is 0 in spdk_nvmf_rdma_qpair_destroy(). > > I believe rqpair refcnt needs to be incremented before spdk_nvmf_rdma_qpair_destroy(). > Here is my experimental patch that fixes the issue, Please let me know if this > qualifies for a patch. > --- a/lib/nvmf/rdma.c > +++ b/lib/nvmf/rdma.c > @@ -2586,9 +2595,10 @@ spdk_nvmf_rdma_poller_poll(struct spdk_nvmf_rdma_transport *rtransport, > case RDMA_WR_TYPE_DRAIN_RECV: > rqpair = SPDK_CONTAINEROF(rdma_wr, struct spdk_nvmf_rdma_qpair, drain_recv_wr); > assert(rqpair->disconnect_flags & RDMA_QP_DISCONNECTING); > SPDK_DEBUGLOG(SPDK_LOG_RDMA, "Drained QP RECV %u (%p)\n", rqpair->qpair.qid, rqpair); > rqpair->disconnect_flags |= RDMA_QP_RECV_DRAINED; > if (rqpair->disconnect_flags & RDMA_QP_SEND_DRAINED) { > + spdk_nvmf_rdma_qpair_inc_refcnt(rqpair); > spdk_nvmf_rdma_qpair_destroy(rqpair); > } > /* Continue so that this does not trigger the disconnect path below. */ > @@ -2596,9 +2606,10 @@ spdk_nvmf_rdma_poller_poll(struct spdk_nvmf_rdma_transport *rtransport, > case RDMA_WR_TYPE_DRAIN_SEND: > rqpair = SPDK_CONTAINEROF(rdma_wr, struct spdk_nvmf_rdma_qpair, drain_send_wr); > assert(rqpair->disconnect_flags & RDMA_QP_DISCONNECTING); > SPDK_DEBUGLOG(SPDK_LOG_RDMA, "Drained QP SEND %u (%p)\n", rqpair->qpair.qid, rqpair); > rqpair->disconnect_flags |= RDMA_QP_SEND_DRAINED; > if (rqpair->disconnect_flags & RDMA_QP_RECV_DRAINED) { > + spdk_nvmf_rdma_qpair_inc_refcnt(rqpair); > spdk_nvmf_rdma_qpair_destroy(rqpair); > } > /* Continue so that this does not trigger the disconnect path below. */ > ---- > > Thanks, > Bharat. > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk