Skip to contentSkip to conversation

RDMA/mana_ib: UC QP support for UAPI

PATCH Superseded Konstantin Taranov 4 participants 5 messages May 20 – Jun 9

Subsystems (2)

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 RDMA/mana_ib: UC QP support for UAPI

From: Konstantin Taranov <[email protected]>

Implement UC QP creation in the RNIC HW for user API. An UC QP is exposed as three work queues: send, receive, and memory management. The latter is used for bind and invalidate WQEs to support memory windows.

---
It is a resend of v3
v3: use new udata helpers
v2: removed udata check and removed enum from mana-abi.h

 drivers/infiniband/hw/mana/main.c    | 41 ++++++++++++-
 drivers/infiniband/hw/mana/mana_ib.h | 41 ++++++++++++-
 drivers/infiniband/hw/mana/qp.c      | 89 ++++++++++++++++++++++++++--
 include/uapi/rdma/mana-abi.h         | 11 ++++
 4 files changed, 173 insertions(+), 9 deletions(-)
drivers/infiniband/hw/mana/main.c +40 -1
index ac5e75dd3..42567c30e 100644
@@ -978,7 +978,46 @@ int mana_ib_gd_create_rc_qp(struct mana_ib_dev *mdev, struct mana_ib_qp *qp,
 	return 0;
 }
 
-int mana_ib_gd_destroy_rc_qp(struct mana_ib_dev *mdev, struct mana_ib_qp *qp)
+int mana_ib_gd_create_uc_qp(struct mana_ib_dev *mdev, struct mana_ib_qp *qp,
+			    struct ib_qp_init_attr *attr, u32 doorbell, u64 flags)
+{
+	struct mana_ib_cq *send_cq = container_of(qp->ibqp.send_cq, struct mana_ib_cq, ibcq);
+	struct mana_ib_cq *recv_cq = container_of(qp->ibqp.recv_cq, struct mana_ib_cq, ibcq);
+	struct mana_ib_pd *pd = container_of(qp->ibqp.pd, struct mana_ib_pd, ibpd);
+	struct gdma_context *gc = mdev_to_gc(mdev);
+	struct mana_rnic_create_uc_qp_resp resp = {};
+	struct mana_rnic_create_uc_qp_req req = {};
+	int err, i;
+
+	mana_gd_init_req_hdr(&req.hdr, MANA_IB_CREATE_UC_QP, sizeof(req), sizeof(resp));
+	req.hdr.dev_id = mdev->gdma_dev->dev_id;
+	req.adapter = mdev->adapter_handle;
+	req.pd_handle = pd->pd_handle;
+	req.send_cq_handle = send_cq->cq_handle;
+	req.recv_cq_handle = recv_cq->cq_handle;
+	for (i = 0; i < MANA_UC_QUEUE_TYPE_MAX; i++)
+		req.dma_region[i] = qp->uc_qp.queues[i].gdma_region;
+	req.doorbell_page = doorbell;
+	req.max_send_wr = attr->cap.max_send_wr;
+	req.max_recv_wr = attr->cap.max_recv_wr;
+	req.max_send_sge = attr->cap.max_send_sge;
+	req.max_recv_sge = attr->cap.max_recv_sge;
+	req.flags = flags;
+
+	err = mana_gd_send_request(gc, sizeof(req), &req, sizeof(resp), &resp);
+	if (err)
+		return err;
+
+	qp->qp_handle = resp.qp_handle;
+	for (i = 0; i < MANA_UC_QUEUE_TYPE_MAX; i++) {
+		qp->uc_qp.queues[i].id = resp.queue_ids[i];
+		/* The GDMA regions are now owned by the RNIC QP handle */
+		qp->uc_qp.queues[i].gdma_region = GDMA_INVALID_DMA_REGION;
+	}
+	return 0;
+}
+
+int mana_ib_gd_destroy_rnic_qp(struct mana_ib_dev *mdev, struct mana_ib_qp *qp)
 {
 	struct mana_rnic_destroy_rc_qp_resp resp = {0};
 	struct mana_rnic_destroy_rc_qp_req req = {0};
drivers/infiniband/hw/mana/mana_ib.h +39 -2

71 lines not on this page · raw message

drivers/infiniband/hw/mana/qp.c +83 -6

154 lines not on this page · raw message

include/uapi/rdma/mana-abi.h +11 -0
index a75bf32b8..f844afb6b 100644
@@ -57,6 +57,17 @@ struct mana_ib_create_rc_qp_resp {
 	__u32 queue_id[4];
 };
 
+struct mana_ib_create_uc_qp {
+	__aligned_u64 queue_buf[3];
+	__u32 queue_size[3];
+	__u32 reserved;
+};
+
+struct mana_ib_create_uc_qp_resp {
+	__u32 queue_id[3];
+	__u32 reserved;
+};
+
 struct mana_ib_create_wq {
 	__aligned_u64 wq_buf_addr;
 	__u32 wq_buf_size;
Signature · 1 line
2.43.0
#1 lore raw

Review

Quoted 2 lines
(quoted diff) +2 -0
+int mana_ib_gd_destroy_rnic_qp(struct mana_ib_dev *mdev, struct
+mana_ib_qp *qp)

The function is renamed to _rnic_ to be shared between RC and UC, but
the request/response structs are still named _rc_. Should these be renamed to _rnic_ as well for consistency? Or does the firmware use the same destroy command for both QP types?

Quoted 3 lines

{

(quoted diff) +0 -0
 	struct mana_rnic_destroy_rc_qp_resp resp = {0};
 	struct mana_rnic_destroy_rc_qp_req req = {0}; diff --git
#2 lore raw
Quoted 7 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+int mana_ib_gd_destroy_rnic_qp(struct mana_ib_dev *mdev, struct
+mana_ib_qp *qp)

The function is renamed to _rnic_ to be shared between RC and UC, but the request/response structs are still named _rc_. Should these be renamed to _rnic_ as well for consistency? Or does the firmware use the same destroy command for both QP types?

Sure, they can be renamed. In the firmware, it is still called RC though, but it can be used for all QP types (even UD). So, the name _rc_ is just historical.

Long, do you want me to rename it?

Jason, can the patch series accepted, or do I need to send IB_UVERBS_CORE_SUPPORT_ROBUST_UDATA first?

Thanks

Quoted 4 lines
Quoted 4 lines

{

(quoted diff) +0 -0
 	struct mana_rnic_destroy_rc_qp_resp resp = {0};
 	struct mana_rnic_destroy_rc_qp_req req = {0}; diff --git
#3 lore raw
Quoted 8 lines · Konstantin Taranov
Quoted 7 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+int mana_ib_gd_destroy_rnic_qp(struct mana_ib_dev *mdev, struct
+mana_ib_qp *qp)

The function is renamed to _rnic_ to be shared between RC and UC, but the request/response structs are still named _rc_. Should these be renamed to _rnic_ as well for consistency? Or does the firmware use the same destroy command for both QP types?

Quoted 7 lines

Sure, they can be renamed. In the firmware, it is still called RC though, but it can be used for all QP types (even UD). So, the name _rc_ is just historical.

Long, do you want me to rename it?

Jason, can the patch series accepted, or do I need to send IB_UVERBS_CORE_SUPPORT_ROBUST_UDATA first?

Yeah, that's what I want to start seeing

And use the proper helpers too:

+ err = ib_copy_to_udata(udata, &resp, min(sizeof(resp), udata->outlen));
+ if (err)
+ goto destroy_qp;

Thats ib_respond_udata() now too

Jason

#4 lore raw
In reply to Konstantin Taranov
Quoted 11 lines

Subject: RE: [PATCH rdma-next v3 1/1] RDMA/mana_ib: UC QP support for UAPI

Quoted 7 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+int mana_ib_gd_destroy_rnic_qp(struct mana_ib_dev *mdev, struct
+mana_ib_qp *qp)

The function is renamed to _rnic_ to be shared between RC and UC, but the request/response structs are still named _rc_. Should these be renamed to _rnic_ as well for consistency? Or does the firmware use the same destroy command for both QP types?

Quoted 4 lines

Sure, they can be renamed. In the firmware, it is still called RC though, but it can be used for all QP types (even UD). So, the name _rc_ is just historical.

Long, do you want me to rename it?

Yes, it's better to rename them to avoid confusions.

Thanks,
Long