ODP support for mlx5 DC QPs
Subsystems (5)
v3 cover ODP support for mlx5 DC QPs
From: Leon Romanovsky <[email protected]>
Changelog
v3:
* Rewrote patches to expose through DEVX without need to change mlx5-abi.h at all.
v2: https://lore.kernel.org/linux-rdma/[email protected]
* Fixed reserved_* field wrong name (Saeed M.)
* Split first patch to two patches, one for mlx5-next and one for rdma-next. (Saeed M.)
v1: https://lore.kernel.org/linux-rdma/[email protected]
* Fixed alignment to u64 in mlx5-abi.h (Gal P.)
v0: https://lore.kernel.org/linux-rdma/[email protected]
---------------------------------------------------------------------------------
From Michael,
The series adds support for on-demand paging for DC transport.
As DC is mlx-only transport, the capabilities are exposed
to the user using DEVX objects and later on through mlx5dv_query_device.
Thanks
Michael Guralnik (3):
net/mlx5: Set ODP capabilities for DC transport to max IB/mlx5: Remove check of FW capabilities in ODP page fault handling IB/mlx5: Add page fault handler for DC initiator WQE
drivers/infiniband/hw/mlx5/odp.c | 51 ++----------------- .../net/ethernet/mellanox/mlx5/core/main.c | 6 +++ include/linux/mlx5/mlx5_ifc.h | 4 +- 3 files changed, 12 insertions(+), 49 deletions(-)
--
2.20.1
PATCH v3 1/3 net/mlx5: Set ODP capabilities for DC transport to max
From: Michael Guralnik <[email protected]>
In mlx5_core initialization, query max ODP capabilities for DC transport from FW and set as current capabilities.
--- drivers/net/ethernet/mellanox/mlx5/core/main.c | 6 ++++++ include/linux/mlx5/mlx5_ifc.h | 4 +++- 2 files changed, 9 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/main.c +6 -0
@@ -495,6 +495,12 @@ static int handle_hca_cap_odp(struct mlx5_core_dev *dev)ODP_CAP_SET_MAX (dev, xrc_odp_caps.write);ODP_CAP_SET_MAX (dev, xrc_odp_caps.read);ODP_CAP_SET_MAX (dev, xrc_odp_caps.atomic); +ODP_CAP_SET_MAX (dev, dc_odp_caps.srq_receive); +ODP_CAP_SET_MAX (dev, dc_odp_caps.send); +ODP_CAP_SET_MAX (dev, dc_odp_caps.receive); +ODP_CAP_SET_MAX (dev, dc_odp_caps.write); +ODP_CAP_SET_MAX (dev, dc_odp_caps.read); +ODP_CAP_SET_MAX (dev, dc_odp_caps.atomic);if (do_set) err = set_caps(dev, set_ctx, set_sz,
include/linux/mlx5/mlx5_ifc.h +3 -1
@@ -946,7 +946,9 @@ struct mlx5_ifc_odp_cap_bits {struct mlx5_ifc_odp_per_transport_service_cap_bits xrc_odp_caps; -u8 reserved_at_100[0x700 ]; +struct mlx5_ifc_odp_per_transport_service_cap_bits dc_odp_caps; + +u8 reserved_at_120[0x6E0 ]; };struct mlx5_ifc_calc_op {
Signature · 1 line
2.20.1
PATCH v3 2/3 IB/mlx5: Remove check of FW capabilities in ODP page fault handling
In reply to Leon RomanovskyFrom: Michael Guralnik <[email protected]>
As page fault handling is initiated by FW, there is no need to check that the ODP supports the operation and transport.
--- drivers/infiniband/hw/mlx5/odp.c | 48 +------------------------------- 1 file changed, 1 insertion(+), 47 deletions(-)
drivers/infiniband/hw/mlx5/odp.c +1 -47
@@ -986,17 +986,6 @@ static int pagefault_data_segments(struct mlx5_ib_dev *dev,return ret <0 ? ret : npages; } -static const u32 mlx5_ib_odp_opcode_cap[] = { - [MLX5_OPCODE_SEND ] =IB_ODP_SUPPORT_SEND , - [MLX5_OPCODE_SEND_IMM ] =IB_ODP_SUPPORT_SEND , - [MLX5_OPCODE_SEND_INVAL ] =IB_ODP_SUPPORT_SEND , - [MLX5_OPCODE_RDMA_WRITE ] =IB_ODP_SUPPORT_WRITE , - [MLX5_OPCODE_RDMA_WRITE_IMM ] =IB_ODP_SUPPORT_WRITE , - [MLX5_OPCODE_RDMA_READ ] =IB_ODP_SUPPORT_READ , - [MLX5_OPCODE_ATOMIC_CS ] =IB_ODP_SUPPORT_ATOMIC , - [MLX5_OPCODE_ATOMIC_FA ] =IB_ODP_SUPPORT_ATOMIC , -}; -/* * Parse initiator WQE. Advances the wqe pointer to point at the * scatter-gather list, and set wqe_end to the end of the WQE.
@@ -1007,7 +996,6 @@ static int mlx5_ib_mr_initiator_pfault_handler( {struct mlx5_wqe_ctrl_seg *ctrl = *wqe;u16 wqe_index = pfault->wqe.wqe_index; -u32 transport_caps;struct mlx5_base_av *av;unsigned ds, opcode;u32 qpn = qp->trans_qp.base.mqp.qpn;
@@ -1031,29 +1019,8 @@ static int mlx5_ib_mr_initiator_pfault_handler( opcode =be32_to_cpu (ctrl->opmod_idx_opcode) &MLX5_WQE_CTRL_OPCODE_MASK ; -switch (qp->ibqp.qp_type) { -case IB_QPT_XRC_INI : +if (qp->ibqp.qp_type ==IB_QPT_XRC_INI ) *wqe +=sizeof (struct mlx5_wqe_xrc_seg ); - transport_caps = dev->odp_caps.per_transport_caps.xrc_odp_caps; -break ; -case IB_QPT_RC : - transport_caps = dev->odp_caps.per_transport_caps.rc_odp_caps; -break ; -case IB_QPT_UD : - transport_caps = dev->odp_caps.per_transport_caps.ud_odp_caps; -break ; -default : -mlx5_ib_err (dev,"ODP fault on QP of an unsupported transport 0x%x\n" , - qp->ibqp.qp_type); -return -EFAULT ; - } - -if (unlikely (opcode >=ARRAY_SIZE (mlx5_ib_odp_opcode_cap) || - !(transport_caps & mlx5_ib_odp_opcode_cap[opcode]))) { -mlx5_ib_err (dev,"ODP fault on QP of an unsupported opcode 0x%x\n" , - opcode); -return -EFAULT ; - }if (qp->ibqp.qp_type ==IB_QPT_UD ) { av = *wqe;
@@ -1118,19 +1085,6 @@ static int mlx5_ib_mr_responder_pfault_handler_rq(struct mlx5_ib_dev *dev,return -EFAULT ; } -switch (qp->ibqp.qp_type) { -case IB_QPT_RC : -if (!(dev->odp_caps.per_transport_caps.rc_odp_caps & -IB_ODP_SUPPORT_RECV )) - goto invalid_transport_or_opcode; -break ; -default : -invalid_transport_or_opcode: -mlx5_ib_err (dev,"ODP fault on QP of an unsupported transport. transport: 0x%x\n" , - qp->ibqp.qp_type); -return -EFAULT ; - } - *wqe_end = wqe + wqe_size;return 0 ;
Signature · 1 line
2.20.1
PATCH v3 3/3 IB/mlx5: Add page fault handler for DC initiator WQE
In reply to Leon RomanovskyFrom: Michael Guralnik <[email protected]>
Parsing DC initiator WQEs upon page fault requires skipping an address vector segment, as in UD WQEs.
--- drivers/infiniband/hw/mlx5/odp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/infiniband/hw/mlx5/odp.c +2 -1
@@ -1022,7 +1022,8 @@ static int mlx5_ib_mr_initiator_pfault_handler(if (qp->ibqp.qp_type ==IB_QPT_XRC_INI ) *wqe +=sizeof (struct mlx5_wqe_xrc_seg ); -if (qp->ibqp.qp_type ==IB_QPT_UD ) { +if (qp->ibqp.qp_type ==IB_QPT_UD || + qp->qp_sub_type ==MLX5_IB_QPT_DCI ) { av = *wqe;if (av->dqp_dct &cpu_to_be32 (MLX5_EXTENDED_UD_AV )) *wqe +=sizeof (struct mlx5_av );
Signature · 1 line
2.20.1
Quoted 19 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Changelog
v3:
* Rewrote patches to expose through DEVX without need to changemlx5-abi.hat all.
v2: https://lore.kernel.org/linux-rdma/[email protected]
* Fixed reserved_* field wrong name (Saeed M.)
* Split first patch to two patches, one for mlx5-next and one for rdma-next. (Saeed M.)
v1: https://lore.kernel.org/linux-rdma/[email protected]
* Fixed alignment to u64 inmlx5-abi.h(Gal P.)
v0: https://lore.kernel.org/linux-rdma/[email protected]Quoted 1 line
From Michael,
The series adds support for on-demand paging for DC transport.
As DC is mlx-only transport, the capabilities are exposed
to the user using DEVX objects and later on throughmlx5dv_query_device.
Quoted 6 lines
Thanks
Michael Guralnik (3):
net/mlx5: Set ODP capabilities for DC transport to max IB/mlx5: Remove check of FW capabilities in ODP page fault handling IB/mlx5: Add page fault handler for DC initiator WQE
This seems fine, can you put the commit on the shared branch?
Thanks,
Jason
Leon Romanovsky
AppliedQuoted 20 lines · Jason Gunthorpe
Quoted 19 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Changelog
v3:
* Rewrote patches to expose through DEVX without need to changemlx5-abi.hat all.
v2: https://lore.kernel.org/linux-rdma/[email protected]
* Fixed reserved_* field wrong name (Saeed M.)
* Split first patch to two patches, one for mlx5-next and one for rdma-next. (Saeed M.)
v1: https://lore.kernel.org/linux-rdma/[email protected]
* Fixed alignment to u64 inmlx5-abi.h(Gal P.)
v0: https://lore.kernel.org/linux-rdma/[email protected]Quoted 1 line
From Michael,
The series adds support for on-demand paging for DC transport.
As DC is mlx-only transport, the capabilities are exposed
to the user using DEVX objects and later on throughmlx5dv_query_device.
Quoted 8 lines
Quoted 6 lines
Thanks
Michael Guralnik (3):
net/mlx5: Set ODP capabilities for DC transport to max IB/mlx5: Remove check of FW capabilities in ODP page fault handling IB/mlx5: Add page fault handler for DC initiator WQEThis seems fine, can you put the commit on the shared branch?
Thanks, applied to mlx5-next00679b631edd net/mlx5: Set ODP capabilities for DC transport to max
Quoted 3 lines
Thanks,
Jason
Quoted 28 lines · Leon Romanovsky
Quoted 27 lines · Jason Gunthorpe
Quoted 25 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Changelog
v3:
* Rewrote patches to expose through DEVX without need to changemlx5-abi.hat all.
v2: https://lore.kernel.org/linux-rdma/[email protected]
* Fixed reserved_* field wrong name (Saeed M.)
* Split first patch to two patches, one for mlx5-next and one for rdma-next. (Saeed M.)
v1: https://lore.kernel.org/linux-rdma/[email protected]
* Fixed alignment to u64 inmlx5-abi.h(Gal P.)
v0: https://lore.kernel.org/linux-rdma/[email protected]Quoted 1 line
From Michael,
The series adds support for on-demand paging for DC transport.
As DC is mlx-only transport, the capabilities are exposed
to the user using DEVX objects and later on throughmlx5dv_query_device.Thanks
Michael Guralnik (3):
net/mlx5: Set ODP capabilities for DC transport to max IB/mlx5: Remove check of FW capabilities in ODP page fault handling IB/mlx5: Add page fault handler for DC initiator WQE
Quoted 4 lines
Quoted 1 line
This seems fine, can you put the commit on the shared branch?
Thanks, applied to mlx5-next
00679b631eddnet/mlx5: Set ODP capabilities for DC transport to max
Done, thanks
Jason