Skip to contentSkip to conversation

RDMA/mana_ib: Add support of RC QPs

PATCH Superseded Konstantin Taranov 4 participants 11 messages May 7, 2024 – May 21, 2024

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover RDMA/mana_ib: Add support of RC QPs

From: Konstantin Taranov <[email protected]>

This patch series enables creation and destruction of RC QPs. The RC QP can be transitioned to RTS and be used by rdma-core.

Later I will submit rdma-core patches with fully working RC QPs.

Konstantin Taranov (3):
RDMA/mana_ib: Create and destroy RC QP RDMA/mana_ib: Implement uapi to create and destroy RC QP RDMA/mana_ib: Modify QP state

 drivers/infiniband/hw/mana/main.c    |  59 ++++++++++
 drivers/infiniband/hw/mana/mana_ib.h |  99 +++++++++++++++-
 drivers/infiniband/hw/mana/qp.c      | 165 ++++++++++++++++++++++++++-
 include/uapi/rdma/mana-abi.h         |   9 ++
 4 files changed, 328 insertions(+), 4 deletions(-)
Signature · 1 line
2.43.0
#1 lore raw

PATCH 1/3 RDMA/mana_ib: Create and destroy RC QP

From: Konstantin Taranov <[email protected]>

Implement HW requests to create and destroy an RC QP. An RC QP may have 5 queues.

---
 drivers/infiniband/hw/mana/main.c    | 59 ++++++++++++++++++++++++++++
 drivers/infiniband/hw/mana/mana_ib.h | 58 ++++++++++++++++++++++++++-
 2 files changed, 116 insertions(+), 1 deletion(-)
drivers/infiniband/hw/mana/main.c +59 -0
index 2a41135..6bd6072 100644
@@ -888,3 +888,62 @@ int mana_ib_gd_destroy_cq(struct mana_ib_dev *mdev, struct mana_ib_cq *cq)
 
 	return 0;
 }
+
+int mana_ib_gd_create_rc_qp(struct mana_ib_dev *mdev, struct mana_ib_qp *qp,
+			    struct ib_qp_init_attr *attr, u32 doorbell, u64 flags)
+{
+	struct mana_ib_cq *send_cq = container_of(qp->ibqp.send_cq, struct mana_ib_cq, ibcq);
+	struct mana_ib_cq *recv_cq = container_of(qp->ibqp.recv_cq, struct mana_ib_cq, ibcq);
+	struct mana_ib_pd *pd = container_of(qp->ibqp.pd, struct mana_ib_pd, ibpd);
+	struct gdma_context *gc = mdev_to_gc(mdev);
+	struct mana_rnic_create_qp_resp resp = {};
+	struct mana_rnic_create_qp_req req = {};
+	int err, i;
+
+	mana_gd_init_req_hdr(&req.hdr, MANA_IB_CREATE_RC_QP, sizeof(req), sizeof(resp));
+	req.hdr.dev_id = gc->mana_ib.dev_id;
+	req.adapter = mdev->adapter_handle;
+	req.pd_handle = pd->pd_handle;
+	req.send_cq_handle = send_cq->cq_handle;
+	req.recv_cq_handle = recv_cq->cq_handle;
+	for (i = 0; i < MANA_RC_QUEUE_TYPE_MAX; i++)
+		req.dma_region[i] = qp->rc_qp.queues[i].gdma_region;
+	req.doorbell_page = doorbell;
+	req.max_send_wr = attr->cap.max_send_wr;
+	req.max_recv_wr = attr->cap.max_recv_wr;
+	req.max_send_sge = attr->cap.max_send_sge;
+	req.max_recv_sge = attr->cap.max_recv_sge;
+	req.flags = flags;
+
+	err = mana_gd_send_request(gc, sizeof(req), &req, sizeof(resp), &resp);
+	if (err) {
+		ibdev_err(&mdev->ib_dev, "Failed to create rc qp err %d", err);
+		return err;
+	}
+	qp->qp_handle = resp.rc_qp_handle;
+	for (i = 0; i < MANA_RC_QUEUE_TYPE_MAX; i++) {
+		qp->rc_qp.queues[i].id = resp.queue_ids[i];
+		/* The GDMA regions are now owned by the RNIC QP handle */
+		qp->rc_qp.queues[i].gdma_region = GDMA_INVALID_DMA_REGION;
+	}
+	return 0;
+}
+
+int mana_ib_gd_destroy_rc_qp(struct mana_ib_dev *mdev, struct mana_ib_qp *qp)
+{
+	struct mana_rnic_destroy_rc_qp_resp resp = {0};
+	struct mana_rnic_destroy_rc_qp_req req = {0};
+	struct gdma_context *gc = mdev_to_gc(mdev);
+	int err;
+
+	mana_gd_init_req_hdr(&req.hdr, MANA_IB_DESTROY_RC_QP, sizeof(req), sizeof(resp));
+	req.hdr.dev_id = gc->mana_ib.dev_id;
+	req.adapter = mdev->adapter_handle;
+	req.rc_qp_handle = qp->qp_handle;
+	err = mana_gd_send_request(gc, sizeof(req), &req, sizeof(resp), &resp);
+	if (err) {
+		ibdev_err(&mdev->ib_dev, "Failed to destroy rc qp err %d", err);
+		return err;
+	}
+	return 0;
+}
drivers/infiniband/hw/mana/mana_ib.h +57 -1

84 lines not on this page · raw message

Signature · 1 line
2.43.0
#3 lore raw

PATCH 2/3 RDMA/mana_ib: Implement uapi to create and destroy RC QP

In reply to Konstantin Taranov

From: Konstantin Taranov <[email protected]>

Implement user requests to create and destroy an RC QP.
As the user does not have an FMR queue, it is skipped and NO_FMR flag is used.

---
 drivers/infiniband/hw/mana/mana_ib.h |  4 ++
 drivers/infiniband/hw/mana/qp.c      | 93 +++++++++++++++++++++++++++-
 include/uapi/rdma/mana-abi.h         |  9 +++
 3 files changed, 105 insertions(+), 1 deletion(-)
drivers/infiniband/hw/mana/mana_ib.h +4 -0
index a3e229c..5cccbe3 100644
@@ -248,6 +248,10 @@ struct mana_rnic_destroy_cq_resp {
 	struct gdma_resp_hdr hdr;
 }; /* HW Data */
 
+enum mana_rnic_create_rc_flags {
+	MANA_RC_FLAG_NO_FMR = 2,
+};
+
 struct mana_rnic_create_qp_req {
 	struct gdma_req_hdr hdr;
 	mana_handle_t adapter;
drivers/infiniband/hw/mana/qp.c +92 -1

117 lines not on this page · raw message

include/uapi/rdma/mana-abi.h +9 -0
index 2c41cc3..45c2df6 100644
@@ -45,6 +45,15 @@ struct mana_ib_create_qp_resp {
 	__u32 reserved;
 };
 
+struct mana_ib_create_rc_qp {
+	__aligned_u64 queue_buf[4];
+	__u32 queue_size[4];
+};
+
+struct mana_ib_create_rc_qp_resp {
+	__u32 queue_id[4];
+};
+
 struct mana_ib_create_wq {
 	__aligned_u64 wq_buf_addr;
 	__u32 wq_buf_size;
Signature · 1 line
2.43.0
#4 lore raw

RE: [PATCH rdma-next 2/3] RDMA/mana_ib: Implement uapi to create and destroy RC QP

Quoted 133 lines

Subject: [PATCH rdma-next 2/3] RDMA/mana_ib: Implement uapi to create and destroy RC QP

From: Konstantin Taranov <[email protected]>

Implement user requests to create and destroy an RC QP.
As the user does not have an FMR queue, it is skipped and NO_FMR flag is used.

---
 drivers/infiniband/hw/mana/mana_ib.h |  4 ++
 drivers/infiniband/hw/mana/qp.c      | 93 +++++++++++++++++++++++++++-
 include/uapi/rdma/mana-abi.h         |  9 +++
 3 files changed, 105 insertions(+), 1 deletion(-)
drivers/infiniband/hw/mana/mana_ib.h +4 -0
index a3e229c..5cccbe3 100644
@@ -248,6 +248,10 @@ struct mana_rnic_destroy_cq_resp {
 	struct gdma_resp_hdr hdr;
 }; /* HW Data */
 
+enum mana_rnic_create_rc_flags {
+	MANA_RC_FLAG_NO_FMR = 2,
+};
+
 struct mana_rnic_create_qp_req {
 	struct gdma_req_hdr hdr;
 	mana_handle_t adapter;
drivers/infiniband/hw/mana/qp.c +18 -0
index ba13c5a..14e6adb 100644
@@ -398,6 +398,78 @@ err_free_vport:
 	return err;
 }
 
+static int mana_ib_create_rc_qp(struct ib_qp *ibqp, struct ib_pd *ibpd,
+				struct ib_qp_init_attr *attr, struct ib_udata
 *udata) {
+	struct mana_ib_dev *mdev = container_of(ibpd->device, struct
 mana_ib_dev, ib_dev);
+	struct mana_ib_qp *qp = container_of(ibqp, struct mana_ib_qp, ibqp);
+	struct mana_ib_create_rc_qp_resp resp = {};
+	struct mana_ib_ucontext *mana_ucontext;
+	struct mana_ib_create_rc_qp ucmd = {};
+	int i, err, j;
+	u64 flags = 0;
+	u32 doorbell;
+
+	if (!udata || udata->inlen < sizeof(ucmd))
+		return -EINVAL;
+
+	mana_ucontext = rdma_udata_to_drv_context(udata, struct
 mana_ib_ucontext, ibucontext);
+	doorbell = mana_ucontext->doorbell;
+	flags = MANA_RC_FLAG_NO_FMR;
+	err = ib_copy_from_udata(&ucmd, udata, min(sizeof(ucmd), udata-
Quoted 1 line

inlen));

drivers/infiniband/hw/mana/qp.c +2 -0
+	if (err) {
+		ibdev_dbg(&mdev->ib_dev, "Failed to copy from udata, %d\n",

err);

drivers/infiniband/hw/mana/qp.c +8 -0
+		return err;
+	}
+
+	for (i = 0, j = 0; i < MANA_RC_QUEUE_TYPE_MAX; ++i) {
+		// skip FMR for user-level RC QPs
+		if (i == MANA_RC_SEND_QUEUE_FMR) {
+			qp->rc_qp.queues[i].id = INVALID_QUEUE_ID;
+			qp->rc_qp.queues[i].gdma_region =

GDMA_INVALID_DMA_REGION;

drivers/infiniband/hw/mana/qp.c +3 -0
+			continue;
+		}
+		err = mana_ib_create_queue(mdev, ucmd.queue_buf[j],

ucmd.queue_size[j],

drivers/infiniband/hw/mana/qp.c +3 -0
+					   &qp->rc_qp.queues[i]);
+		if (err) {
+			ibdev_err(&mdev->ib_dev, "Failed to create queue %d,

err %d\n", i, err);

drivers/infiniband/hw/mana/qp.c +11 -0
+			goto destroy_queues;
+		}
+		j++;
+	}
+
+	err = mana_ib_gd_create_rc_qp(mdev, qp, attr, doorbell, flags);
+	if (err) {
+		ibdev_err(&mdev->ib_dev, "Failed to create rc qp  %d\n", err);
+		goto destroy_queues;
+	}
+	qp->ibqp.qp_num = qp-
Quoted 1 line

rc_qp.queues[MANA_RC_RECV_QUEUE_RESPONDER].id;

drivers/infiniband/hw/mana/qp.c +10 -0
+	qp->port = attr->port_num;
+
+	if (udata) {
+		for (i = 0, j = 0; i < MANA_RC_QUEUE_TYPE_MAX; ++i) {
+			if (i == MANA_RC_SEND_QUEUE_FMR)
+				continue;
+			resp.queue_id[j] = qp->rc_qp.queues[i].id;
+			j++;
+		}
+		err = ib_copy_to_udata(udata, &resp, min(sizeof(resp), udata-
Quoted 1 line

outlen));

drivers/infiniband/hw/mana/qp.c +2 -0
+		if (err) {
+			ibdev_dbg(&mdev->ib_dev, "Failed to copy to

udata, %d\n", err);

drivers/infiniband/hw/mana/qp.c +16 -0
+			goto destroy_qp;
+		}
+	}
+
+	return 0;
+
+destroy_qp:
+	mana_ib_gd_destroy_rc_qp(mdev, qp);
+destroy_queues:
+	while (i-- > 0)
+		mana_ib_destroy_queue(mdev, &qp->rc_qp.queues[i]);
+	return err;
+}
+
 int mana_ib_create_qp(struct ib_qp *ibqp, struct ib_qp_init_attr *attr,
 		      struct ib_udata *udata)
 {
@@ -409,6 +481,8 @@ int mana_ib_create_qp(struct ib_qp *ibqp, struct
 ib_qp_init_attr *attr,
 						     udata);
 
 		return mana_ib_create_qp_raw(ibqp, ibqp->pd, attr, udata);
+	case IB_QPT_RC:
+		return mana_ib_create_rc_qp(ibqp, ibqp->pd, attr, udata);
 	default:
 		/* Creating QP other than IB_QPT_RAW_PACKET is not
Quoted 1 line

supported */

Need to change this comment as RC is supported now.

Quoted 42 lines
drivers/infiniband/hw/mana/qp.c +18 -1
 		ibdev_dbg(ibqp->device, "Creating QP type %u not supported\n",
@@ -473,6 +547,22 @@ static int mana_ib_destroy_qp_raw(struct mana_ib_qp
 *qp, struct ib_udata *udata)
 	return 0;
 }
 
+static int mana_ib_destroy_rc_qp(struct mana_ib_qp *qp, struct ib_udata
+*udata) {
+	struct mana_ib_dev *mdev =
+		container_of(qp->ibqp.device, struct mana_ib_dev, ib_dev);
+	int i;
+
+	/* Ignore return code as there is not much we can do about it.
+	 * The error message is printed inside.
+	 */
+	mana_ib_gd_destroy_rc_qp(mdev, qp);
+	for (i = 0; i < MANA_RC_QUEUE_TYPE_MAX; ++i)
+		mana_ib_destroy_queue(mdev, &qp->rc_qp.queues[i]);
+
+	return 0;
+}
+
 int mana_ib_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)  {
 	struct mana_ib_qp *qp = container_of(ibqp, struct mana_ib_qp, ibqp);
@@ -484,7 +574,8 @@ int mana_ib_destroy_qp(struct ib_qp *ibqp, struct
 ib_udata *udata)
 						      udata);
 
 		return mana_ib_destroy_qp_raw(qp, udata);
-
+	case IB_QPT_RC:
+		return mana_ib_destroy_rc_qp(qp, udata);
 	default:
 		ibdev_dbg(ibqp->device, "Unexpected QP type %u\n",
drivers/infiniband/hw/mana/qp.c +0 -0
 			  ibqp->qp_type);
include/uapi/rdma/mana-abi.h +0 -0
index 2c41cc3..45c2df6 100644
@@ -45,6 +45,15 @@ struct mana_ib_create_qp_resp {
 	__u32 reserved;
 };

include/uapi/rdma/mana-abi.h struct mana_ib_create_qp_resp { · line 56

Quoted 10 lines
include/uapi/rdma/mana-abi.h +9 -0
 
+struct mana_ib_create_rc_qp {
+	__aligned_u64 queue_buf[4];
+	__u32 queue_size[4];
+};
+
+struct mana_ib_create_rc_qp_resp {
+	__u32 queue_id[4];
+};
+

You are adding new UAPI without changing MANA_IB_UVERBS_ABI_VERSION.

For this use-case, I think it's okay because it will fail to create CQ before this. But it may not be a good customer experience for RC usage.

#5 lore raw

RE: [PATCH rdma-next 2/3] RDMA/mana_ib: Implement uapi to create and destroy RC QP

Quoted 3 lines

Need to change this comment as RC is supported now.

Thanks, fixed.

Quoted 6 lines

You are adding new UAPI without changing
MANA_IB_UVERBS_ABI_VERSION.

For this use-case, I think it's okay because it will fail to create CQ before this. But it may not be a good customer experience for RC usage.

ABI versions are for resolving incompatibilities in API, and not for tracking features. So If a user fails to create an RC QP it means the kernel is old and does not have this feature. The customer experience is not affected.

Increase in the abi version would invalidate all previous kernels, preventing clients to use the many versions of kernels and rdma-core together. As a result, breaking the customer experience.

Konstantin

#6 lore raw

PATCH 3/3 RDMA/mana_ib: Modify QP state

In reply to Konstantin Taranov

From: Konstantin Taranov <[email protected]>

Implement modify QP state for RC QPs.

---
 drivers/infiniband/hw/mana/mana_ib.h | 37 ++++++++++++++
 drivers/infiniband/hw/mana/qp.c      | 72 +++++++++++++++++++++++++++-
 2 files changed, 107 insertions(+), 2 deletions(-)
drivers/infiniband/hw/mana/mana_ib.h +37 -0
index 5cccbe3..d29dee7 100644
@@ -140,6 +140,7 @@ enum mana_ib_command_code {
 	MANA_IB_DESTROY_CQ      = 0x30009,
 	MANA_IB_CREATE_RC_QP    = 0x3000a,
 	MANA_IB_DESTROY_RC_QP   = 0x3000b,
+	MANA_IB_SET_QP_STATE	= 0x3000d,
 };
 
 struct mana_ib_query_adapter_caps_req {
@@ -286,6 +287,42 @@ struct mana_rnic_destroy_rc_qp_resp {
 	struct gdma_resp_hdr hdr;
 }; /* HW Data */
 
+struct mana_ib_ah_attr {
+	u8 src_addr[16];
+	u8 dest_addr[16];
+	u8 src_mac[ETH_ALEN];
+	u8 dest_mac[ETH_ALEN];
+	u8 src_addr_type;
+	u8 dest_addr_type;
+	u8 hop_limit;
+	u8 traffic_class;
+	u16 src_port;
+	u16 dest_port;
+	u32 reserved;
+};
+
+struct mana_rnic_set_qp_state_req {
+	struct gdma_req_hdr hdr;
+	mana_handle_t adapter;
+	mana_handle_t qp_handle;
+	u64 attr_mask;
+	u32 qp_state;
+	u32 path_mtu;
+	u32 rq_psn;
+	u32 sq_psn;
+	u32 dest_qpn;
+	u32 max_dest_rd_atomic;
+	u32 retry_cnt;
+	u32 rnr_retry;
+	u32 min_rnr_timer;
+	u32 reserved;
+	struct mana_ib_ah_attr ah_attr;
+}; /* HW Data */
+
+struct mana_rnic_set_qp_state_resp {
+	struct gdma_resp_hdr hdr;
+}; /* HW Data */
+
 static inline struct gdma_context *mdev_to_gc(struct mana_ib_dev *mdev)
 {
 	return mdev->gdma_dev->gdma_context;
drivers/infiniband/hw/mana/qp.c +70 -2

81 lines not on this page · raw message

Signature · 1 line
2.43.0
#8 lore raw
In reply to Konstantin Taranov
Quoted 6 lines · Konstantin Taranov

From: Konstantin Taranov <[email protected]>

This patch series enables creation and destruction of RC QPs. The RC QP can be transitioned to RTS and be used by rdma-core.

Later I will submit rdma-core patches with fully working RC QPs.

Did it happen?

I want to remind that we are not merging UAPI changes without relevant userspace part.

Thanks

Quoted 15 lines

Konstantin Taranov (3):
RDMA/mana_ib: Create and destroy RC QP RDMA/mana_ib: Implement uapi to create and destroy RC QP RDMA/mana_ib: Modify QP state

 drivers/infiniband/hw/mana/main.c    |  59 ++++++++++
 drivers/infiniband/hw/mana/mana_ib.h |  99 +++++++++++++++-
 drivers/infiniband/hw/mana/qp.c      | 165 ++++++++++++++++++++++++++-
 include/uapi/rdma/mana-abi.h         |   9 ++
 4 files changed, 328 insertions(+), 4 deletions(-)
Signature · 1 line
2.43.0
#9 lore raw
Quoted 16 lines

-----Original Message-----
From: Leon Romanovsky <[email protected]>
Sent: Sunday, 12 May 2024 11:36
To: Konstantin Taranov <[email protected]>
Cc: Konstantin Taranov <[email protected]>; [email protected]; Long Li <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: [EXTERNAL] Re: [PATCH rdma-next 0/3] RDMA/mana_ib: Add support of RC QPs

Quoted 5 lines · Konstantin Taranov

From: Konstantin Taranov <[email protected]>

This patch series enables creation and destruction of RC QPs. The RC QP can be transitioned to RTS and be used by rdma-core.

Quoted 6 lines
Quoted 1 line

Later I will submit rdma-core patches with fully working RC QPs.

Did it happen?

I want to remind that we are not merging UAPI changes without relevant userspace part.

Sorry, I missed this requirement. Thanks for informing! I will submit it within next 2 days.

Konstantin

Quoted 19 lines

Thanks

Quoted 9 lines

Konstantin Taranov (3):
RDMA/mana_ib: Create and destroy RC QP RDMA/mana_ib: Implement uapi to create and destroy RC QP RDMA/mana_ib: Modify QP state

 drivers/infiniband/hw/mana/main.c    |  59 ++++++++++
 drivers/infiniband/hw/mana/mana_ib.h |  99 +++++++++++++++-
 drivers/infiniband/hw/mana/qp.c      | 165

++++++++++++++++++++++++++-

Quoted 6 lines
 include/uapi/rdma/mana-abi.h         |   9 ++
 4 files changed, 328 insertions(+), 4 deletions(-)

--
2.43.0

#10 lore raw
Quoted 9 lines
Quoted 6 lines
Quoted 1 line

Later I will submit rdma-core patches with fully working RC QPs.

Did it happen?

I want to remind that we are not merging UAPI changes without relevant userspace part.

Sorry, I missed this requirement. Thanks for informing! I will submit it within next 2 days.

Here is the PR to the rdma-core:
https://github.com/linux-rdma/rdma-core/pull/1461

Thanks