Skip to contentSkip to conversation

ODP support for mlx5 DC QPs

PATCH In review Leon Romanovsky 2 participants 7 messages Aug 19, 2019 – Aug 28, 2019

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

v3 cover ODP support for mlx5 DC QPs

From: Leon Romanovsky <[email protected]>

Changelog
v3:
* Rewrote patches to expose through DEVX without need to change mlx5-abi.h at all.
v2: https://lore.kernel.org/linux-rdma/[email protected]
* Fixed reserved_* field wrong name (Saeed M.)
* Split first patch to two patches, one for mlx5-next and one for rdma-next. (Saeed M.)
v1: https://lore.kernel.org/linux-rdma/[email protected]
* Fixed alignment to u64 in mlx5-abi.h (Gal P.)
v0: https://lore.kernel.org/linux-rdma/[email protected]

---------------------------------------------------------------------------------
From Michael,

The series adds support for on-demand paging for DC transport.

As DC is mlx-only transport, the capabilities are exposed
to the user using DEVX objects and later on through mlx5dv_query_device.

Thanks

Michael Guralnik (3):
net/mlx5: Set ODP capabilities for DC transport to max IB/mlx5: Remove check of FW capabilities in ODP page fault handling IB/mlx5: Add page fault handler for DC initiator WQE

 drivers/infiniband/hw/mlx5/odp.c              | 51 ++-----------------
 .../net/ethernet/mellanox/mlx5/core/main.c    |  6 +++
 include/linux/mlx5/mlx5_ifc.h                 |  4 +-
 3 files changed, 12 insertions(+), 49 deletions(-)

--
2.20.1

#1 lore raw

PATCH v3 1/3 net/mlx5: Set ODP capabilities for DC transport to max

From: Michael Guralnik <[email protected]>

In mlx5_core initialization, query max ODP capabilities for DC transport from FW and set as current capabilities.

---
 drivers/net/ethernet/mellanox/mlx5/core/main.c | 6 ++++++
 include/linux/mlx5/mlx5_ifc.h                  | 4 +++-
 2 files changed, 9 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/main.c +6 -0
index fa0e991f1983..7f70ecb1db6d 100644
@@ -495,6 +495,12 @@ static int handle_hca_cap_odp(struct mlx5_core_dev *dev)
 	ODP_CAP_SET_MAX(dev, xrc_odp_caps.write);
 	ODP_CAP_SET_MAX(dev, xrc_odp_caps.read);
 	ODP_CAP_SET_MAX(dev, xrc_odp_caps.atomic);
+	ODP_CAP_SET_MAX(dev, dc_odp_caps.srq_receive);
+	ODP_CAP_SET_MAX(dev, dc_odp_caps.send);
+	ODP_CAP_SET_MAX(dev, dc_odp_caps.receive);
+	ODP_CAP_SET_MAX(dev, dc_odp_caps.write);
+	ODP_CAP_SET_MAX(dev, dc_odp_caps.read);
+	ODP_CAP_SET_MAX(dev, dc_odp_caps.atomic);
 
 	if (do_set)
 		err = set_caps(dev, set_ctx, set_sz,
include/linux/mlx5/mlx5_ifc.h +3 -1
index ab6ae723aae6..f037f8d5970e 100644
@@ -946,7 +946,9 @@ struct mlx5_ifc_odp_cap_bits {
 
 	struct mlx5_ifc_odp_per_transport_service_cap_bits xrc_odp_caps;
 
-	u8         reserved_at_100[0x700];
+	struct mlx5_ifc_odp_per_transport_service_cap_bits dc_odp_caps;
+
+	u8         reserved_at_120[0x6E0];
 };
 
 struct mlx5_ifc_calc_op {
Signature · 1 line
2.20.1
#2 lore raw

PATCH v3 2/3 IB/mlx5: Remove check of FW capabilities in ODP page fault handling

In reply to Leon Romanovsky

From: Michael Guralnik <[email protected]>

As page fault handling is initiated by FW, there is no need to check that the ODP supports the operation and transport.

---
 drivers/infiniband/hw/mlx5/odp.c | 48 +-------------------------------
 1 file changed, 1 insertion(+), 47 deletions(-)
drivers/infiniband/hw/mlx5/odp.c +1 -47
index 8b155a1f0b38..e7a4ea979209 100644
@@ -986,17 +986,6 @@ static int pagefault_data_segments(struct mlx5_ib_dev *dev,
 	return ret < 0 ? ret : npages;
 }
 
-static const u32 mlx5_ib_odp_opcode_cap[] = {
-	[MLX5_OPCODE_SEND]	       = IB_ODP_SUPPORT_SEND,
-	[MLX5_OPCODE_SEND_IMM]	       = IB_ODP_SUPPORT_SEND,
-	[MLX5_OPCODE_SEND_INVAL]       = IB_ODP_SUPPORT_SEND,
-	[MLX5_OPCODE_RDMA_WRITE]       = IB_ODP_SUPPORT_WRITE,
-	[MLX5_OPCODE_RDMA_WRITE_IMM]   = IB_ODP_SUPPORT_WRITE,
-	[MLX5_OPCODE_RDMA_READ]	       = IB_ODP_SUPPORT_READ,
-	[MLX5_OPCODE_ATOMIC_CS]	       = IB_ODP_SUPPORT_ATOMIC,
-	[MLX5_OPCODE_ATOMIC_FA]	       = IB_ODP_SUPPORT_ATOMIC,
-};
-
 /*
  * Parse initiator WQE. Advances the wqe pointer to point at the
  * scatter-gather list, and set wqe_end to the end of the WQE.
@@ -1007,7 +996,6 @@ static int mlx5_ib_mr_initiator_pfault_handler(
 {
 	struct mlx5_wqe_ctrl_seg *ctrl = *wqe;
 	u16 wqe_index = pfault->wqe.wqe_index;
-	u32 transport_caps;
 	struct mlx5_base_av *av;
 	unsigned ds, opcode;
 	u32 qpn = qp->trans_qp.base.mqp.qpn;
@@ -1031,29 +1019,8 @@ static int mlx5_ib_mr_initiator_pfault_handler(
 	opcode = be32_to_cpu(ctrl->opmod_idx_opcode) &
 		 MLX5_WQE_CTRL_OPCODE_MASK;
 
-	switch (qp->ibqp.qp_type) {
-	case IB_QPT_XRC_INI:
+	if (qp->ibqp.qp_type == IB_QPT_XRC_INI)
 		*wqe += sizeof(struct mlx5_wqe_xrc_seg);
-		transport_caps = dev->odp_caps.per_transport_caps.xrc_odp_caps;
-		break;
-	case IB_QPT_RC:
-		transport_caps = dev->odp_caps.per_transport_caps.rc_odp_caps;
-		break;
-	case IB_QPT_UD:
-		transport_caps = dev->odp_caps.per_transport_caps.ud_odp_caps;
-		break;
-	default:
-		mlx5_ib_err(dev, "ODP fault on QP of an unsupported transport 0x%x\n",
-			    qp->ibqp.qp_type);
-		return -EFAULT;
-	}
-
-	if (unlikely(opcode >= ARRAY_SIZE(mlx5_ib_odp_opcode_cap) ||
-		     !(transport_caps & mlx5_ib_odp_opcode_cap[opcode]))) {
-		mlx5_ib_err(dev, "ODP fault on QP of an unsupported opcode 0x%x\n",
-			    opcode);
-		return -EFAULT;
-	}
 
 	if (qp->ibqp.qp_type == IB_QPT_UD) {
 		av = *wqe;
@@ -1118,19 +1085,6 @@ static int mlx5_ib_mr_responder_pfault_handler_rq(struct mlx5_ib_dev *dev,
 		return -EFAULT;
 	}
 
-	switch (qp->ibqp.qp_type) {
-	case IB_QPT_RC:
-		if (!(dev->odp_caps.per_transport_caps.rc_odp_caps &
-		      IB_ODP_SUPPORT_RECV))
-			goto invalid_transport_or_opcode;
-		break;
-	default:
-invalid_transport_or_opcode:
-		mlx5_ib_err(dev, "ODP fault on QP of an unsupported transport. transport: 0x%x\n",
-			    qp->ibqp.qp_type);
-		return -EFAULT;
-	}
-
 	*wqe_end = wqe + wqe_size;
 
 	return 0;
Signature · 1 line
2.20.1
#3 lore raw

PATCH v3 3/3 IB/mlx5: Add page fault handler for DC initiator WQE

In reply to Leon Romanovsky

From: Michael Guralnik <[email protected]>

Parsing DC initiator WQEs upon page fault requires skipping an address vector segment, as in UD WQEs.

---
 drivers/infiniband/hw/mlx5/odp.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/infiniband/hw/mlx5/odp.c +2 -1
index e7a4ea979209..e6903e90aaf1 100644
@@ -1022,7 +1022,8 @@ static int mlx5_ib_mr_initiator_pfault_handler(
 	if (qp->ibqp.qp_type == IB_QPT_XRC_INI)
 		*wqe += sizeof(struct mlx5_wqe_xrc_seg);
 
-	if (qp->ibqp.qp_type == IB_QPT_UD) {
+	if (qp->ibqp.qp_type == IB_QPT_UD ||
+	    qp->qp_sub_type == MLX5_IB_QPT_DCI) {
 		av = *wqe;
 		if (av->dqp_dct & cpu_to_be32(MLX5_EXTENDED_UD_AV))
 			*wqe += sizeof(struct mlx5_av);
Signature · 1 line
2.20.1
#4 lore raw
In reply to Leon Romanovsky
Quoted 19 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Changelog
v3:
* Rewrote patches to expose through DEVX without need to change mlx5-abi.h at all.
v2: https://lore.kernel.org/linux-rdma/[email protected]
* Fixed reserved_* field wrong name (Saeed M.)
* Split first patch to two patches, one for mlx5-next and one for rdma-next. (Saeed M.)
v1: https://lore.kernel.org/linux-rdma/[email protected]
* Fixed alignment to u64 in mlx5-abi.h (Gal P.)
v0: https://lore.kernel.org/linux-rdma/[email protected]

Quoted 1 line

From Michael,

The series adds support for on-demand paging for DC transport.

As DC is mlx-only transport, the capabilities are exposed
to the user using DEVX objects and later on through mlx5dv_query_device.

Quoted 6 lines

Thanks

Michael Guralnik (3):
net/mlx5: Set ODP capabilities for DC transport to max IB/mlx5: Remove check of FW capabilities in ODP page fault handling IB/mlx5: Add page fault handler for DC initiator WQE

This seems fine, can you put the commit on the shared branch?

Thanks,
Jason

author Applied
#5 lore raw
Quoted 20 lines · Jason Gunthorpe
Quoted 19 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Changelog
v3:
* Rewrote patches to expose through DEVX without need to change mlx5-abi.h at all.
v2: https://lore.kernel.org/linux-rdma/[email protected]
* Fixed reserved_* field wrong name (Saeed M.)
* Split first patch to two patches, one for mlx5-next and one for rdma-next. (Saeed M.)
v1: https://lore.kernel.org/linux-rdma/[email protected]
* Fixed alignment to u64 in mlx5-abi.h (Gal P.)
v0: https://lore.kernel.org/linux-rdma/[email protected]

Quoted 1 line

From Michael,

The series adds support for on-demand paging for DC transport.

As DC is mlx-only transport, the capabilities are exposed
to the user using DEVX objects and later on through mlx5dv_query_device.

Quoted 8 lines
Quoted 6 lines

Thanks

Michael Guralnik (3):
net/mlx5: Set ODP capabilities for DC transport to max IB/mlx5: Remove check of FW capabilities in ODP page fault handling IB/mlx5: Add page fault handler for DC initiator WQE

This seems fine, can you put the commit on the shared branch?

Thanks, applied to mlx5-next
00679b631edd net/mlx5: Set ODP capabilities for DC transport to max

Quoted 3 lines

Thanks,
Jason

#6 lore raw
Quoted 28 lines · Leon Romanovsky
Quoted 27 lines · Jason Gunthorpe
Quoted 25 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Changelog
v3:
* Rewrote patches to expose through DEVX without need to change mlx5-abi.h at all.
v2: https://lore.kernel.org/linux-rdma/[email protected]
* Fixed reserved_* field wrong name (Saeed M.)
* Split first patch to two patches, one for mlx5-next and one for rdma-next. (Saeed M.)
v1: https://lore.kernel.org/linux-rdma/[email protected]
* Fixed alignment to u64 in mlx5-abi.h (Gal P.)
v0: https://lore.kernel.org/linux-rdma/[email protected]

Quoted 1 line

From Michael,

The series adds support for on-demand paging for DC transport.

As DC is mlx-only transport, the capabilities are exposed
to the user using DEVX objects and later on through mlx5dv_query_device.

Thanks

Michael Guralnik (3):
net/mlx5: Set ODP capabilities for DC transport to max IB/mlx5: Remove check of FW capabilities in ODP page fault handling IB/mlx5: Add page fault handler for DC initiator WQE

Quoted 4 lines
Quoted 1 line

This seems fine, can you put the commit on the shared branch?

Thanks, applied to mlx5-next
00679b631edd net/mlx5: Set ODP capabilities for DC transport to max

Done, thanks

Jason