Skip to contentSkip to conversation

RDMA/mana_ib: return PD number to the user

PATCH Superseded Konstantin Taranov 4 participants 5 messages Feb 4 – Feb 6

Subsystems (4)

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 RDMA/mana_ib: return PD number to the user

From: Konstantin Taranov <[email protected]>

Implement returning to userspace applications PDNs of created PDs. Allow users to request short PDNs which are 16 bits.

---
 drivers/infiniband/hw/mana/main.c | 19 ++++++++++++++++++-
 include/net/mana/gdma.h           |  4 ++--
 include/uapi/rdma/mana-abi.h      | 14 ++++++++++++++
 3 files changed, 34 insertions(+), 3 deletions(-)
drivers/infiniband/hw/mana/main.c +18 -1
index fac159f71..7ee4493cb 100644
@@ -69,9 +69,11 @@ int mana_ib_cfg_vport(struct mana_ib_dev *dev, u32 port, struct mana_ib_pd *pd,
 int mana_ib_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
 {
 	struct mana_ib_pd *pd = container_of(ibpd, struct mana_ib_pd, ibpd);
+	struct mana_ib_alloc_pd_resp cmd_resp = {};
 	struct ib_device *ibdev = ibpd->device;
 	struct gdma_create_pd_resp resp = {};
 	struct gdma_create_pd_req req = {};
+	struct mana_ib_alloc_pd ucmd = {};
 	enum gdma_pd_flags flags = 0;
 	struct mana_ib_dev *dev;
 	struct gdma_context *gc;
@@ -83,8 +85,15 @@ int mana_ib_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
 	mana_gd_init_req_hdr(&req.hdr, GDMA_CREATE_PD, sizeof(req),
 			     sizeof(resp));
 
-	if (!udata)
+	if (!udata) {
 		flags |= GDMA_PD_FLAG_ALLOW_GPA_MR;
+	} else {
+		err = ib_copy_from_udata(&ucmd, udata, min(sizeof(ucmd), udata->inlen));
+		if (err)
+			return err;
+		if (ucmd.flags & MANA_IB_PD_SHORT_PDN)
+			flags |= GDMA_PD_FLAG_SHORT_PDN;
+	}
 
 	req.flags = flags;
 	err = mana_gd_send_request(gc, sizeof(req), &req,
@@ -107,6 +116,14 @@ int mana_ib_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
 
 	mutex_init(&pd->vport_mutex);
 	pd->vport_use_count = 0;
+
+	if (udata) {
+		cmd_resp.pdn = resp.pd_id;
+		err = ib_copy_to_udata(udata, &cmd_resp, min(sizeof(cmd_resp), udata->outlen));
+		if (err)
+			return err;
+	}
+
 	return 0;
 }
 
include/net/mana/gdma.h +2 -2
index 8649eb789..cebb9b2bd 100644
@@ -824,8 +824,8 @@ struct gdma_destroy_dma_region_req {
 }; /* HW DATA */
 
 enum gdma_pd_flags {
-	GDMA_PD_FLAG_INVALID = 0,
-	GDMA_PD_FLAG_ALLOW_GPA_MR = 1,
+	GDMA_PD_FLAG_ALLOW_GPA_MR = BIT(0),
+	GDMA_PD_FLAG_SHORT_PDN = BIT(2),
 };
 
 struct gdma_create_pd_req {
include/uapi/rdma/mana-abi.h +14 -0
index a75bf32b8..88b24ae50 100644
@@ -87,4 +87,18 @@ struct mana_ib_create_qp_rss_resp {
 	struct rss_resp_entry entries[64];
 };
 
+enum mana_ib_create_pd_flags {
+	MANA_IB_PD_SHORT_PDN = 1 << 0,
+};
+
+struct mana_ib_alloc_pd {
+	__u32 flags;
+	__u32 reserved;
+};
+
+struct mana_ib_alloc_pd_resp {
+	__u32 pdn;
+	__u32 reserved;
+};
+
 #endif
Signature · 1 line
2.43.0
#1 lore raw
Quoted 4 lines · Konstantin Taranov

From: Konstantin Taranov <[email protected]>

Implement returning to userspace applications PDNs of created PDs. Allow users to request short PDNs which are 16 bits.

Why does userspace ever need to see a PDN? Please justify that in the commit message

Jason

#2 lore raw
Quoted 8 lines · Jason Gunthorpe
Quoted 4 lines · Konstantin Taranov

From: Konstantin Taranov <[email protected]>

Implement returning to userspace applications PDNs of created PDs. Allow users to request short PDNs which are 16 bits.

Why does userspace ever need to see a PDN? Please justify that in the commit message

Probably for the debug and we have restrack for it.

Thanks

Quoted 2 lines

Jason

#3 lore raw
Quoted 13 lines · Jason Gunthorpe
Quoted 8 lines · Konstantin Taranov
Quoted 4 lines

From: Konstantin Taranov <[email protected]>

Implement returning to userspace applications PDNs of created PDs. Allow users to request short PDNs which are 16 bits.

Why does userspace ever need to see a PDN? Please justify that in the commit message

Probably for the debug and we have restrack for it.

Sure, I will add the explanation in v2. Overall, it is for applications working on top of the rdma-core (e.g., mana DPDK). The use-case is similar to what mlx4 and mthca have for address vectors in rdma-core for isolation.
As the whole process of working with WQs and CQs is implemented in that applications (e.g., mana DPDK), they need to know PDN to build a correct request. What is more, the HW folks put a limit of 16 bits to the PDN field, requiring a flag to ensure that we get a PDN that fits into the field.

I hope that it justifies the change as most ib providers have pdn in the user-space.

- Konstantin

Quoted 4 lines

Thanks

Quoted 2 lines

Jason

#4 lore raw
Quoted 23 lines · Konstantin Taranov
Quoted 13 lines · Jason Gunthorpe
Quoted 8 lines · Konstantin Taranov
Quoted 4 lines

From: Konstantin Taranov <[email protected]>

Implement returning to userspace applications PDNs of created PDs. Allow users to request short PDNs which are 16 bits.

Why does userspace ever need to see a PDN? Please justify that in the commit message

Probably for the debug and we have restrack for it.

Sure, I will add the explanation in v2. Overall, it is for applications working on top of the rdma-core (e.g., mana DPDK). The use-case is similar to what mlx4 and mthca have for address vectors in rdma-core for isolation. As the whole process of working with WQs and CQs is implemented in that applications (e.g., mana DPDK), they need to know PDN to build a correct request. What is more, the HW folks put a limit of 16 bits to the PDN field, requiring a flag to ensure that we get a PDN that fits into the field.

Quoted 2 lines

I hope that it justifies the change as most ib providers have pdn in the user-space.

But they don't put it in a WQE and don't check in HW it is exactly the same as the PDN the WQ already has.

You have PDs in AHs and other related objects which make sense, but a WQ only has one PD, it is illogical to pass in a PDN in a WQE, because it can never take on a different value.

If it can take on a different value then your HW's security model is broken.

Jason