Packet based credit mode

PATCH Applied Leon Romanovsky 2 participants 7 messages Nov 30, 2018 – Dec 7, 2018

Subsystems (5)

Networking Network drivers InfiniBand Mellanox mlx5 core +1 more subsystems

lore

Share & tools

lore

Leon Romanovsky

author

2018-11-30 11:22 UTC #0 lore raw

cover Packet based credit mode

From: Leon Romanovsky <[email protected]>

Quoted 1 line

From Danit,

Packet based credit mode is an alternative end-to-end credit mode for QPs set during their creation. Credits are transported from the responder to the requester to optimize the use of its receive resources. In packet-based credit mode, credits are issued on a per packet basis.

The advantage of this feature comes while sending large RDMA messages through switches that are short in memory.

The first commit exposes QP creation flag and the HCA capability. The second commit adds support for a new DV QP creation flag. The last commit report packet based credit mode capability via the MLX5DV device capabilities.

Thanks

Danit Goldberg (3):
net/mlx5: Expose packet based credit mode IB/mlx5: Add packet based credit mode support IB/mlx5: Report packet based credit mode device capability

 drivers/infiniband/hw/mlx5/main.c    |  3 +++
 drivers/infiniband/hw/mlx5/mlx5_ib.h |  1 +
 drivers/infiniband/hw/mlx5/qp.c      | 15 +++++++++++++--
 include/linux/mlx5/mlx5_ifc.h        |  6 ++++--
 include/uapi/rdma/mlx5-abi.h         |  2 ++
 5 files changed, 23 insertions(+), 4 deletions(-)

Leon Romanovsky

author

2018-11-30 11:22 UTC #1 lore raw

PATCH 1/3 net/mlx5: Expose packet based credit mode

From: Danit Goldberg <[email protected]>

Packet based credit mode bit determines whether the credit mode is done per message or packet. Expose the QP creation flag and the HCA capability.

Signed-off-by Danit Goldberg <[email protected]>

Reviewed-by Majd Dibbiny <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 include/linux/mlx5/mlx5_ifc.h | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

include/linux/mlx5/mlx5_ifc.h +4 -2

index 171d68663640..52393fbcf3b4 100644
@@ -1057,7 +1057,8 @@ struct mlx5_ifc_cmd_hca_cap_bits {
 	u8         vector_calc[0x1];
 	u8         umr_ptr_rlky[0x1];
 	u8	   imaicl[0x1];
-	u8         reserved_at_232[0x4];
+	u8	   qp_packet_based[0x1];
+	u8         reserved_at_233[0x3];
 	u8         qkv[0x1];
 	u8         pkv[0x1];
 	u8         set_deth_sqpn[0x1];

@@ -2269,7 +2270,8 @@ struct mlx5_ifc_qpc_bits {
 	u8         st[0x8];
 	u8         reserved_at_10[0x3];
 	u8         pm_state[0x2];
-	u8         reserved_at_15[0x3];
+	u8         reserved_at_15[0x1];
+	u8         req_e2e_credit_mode[0x2];
 	u8         offload_type[0x4];
 	u8         end_padding_mode[0x2];
 	u8         reserved_at_1e[0x2];

Signature · 1 line

2.19.1

Leon Romanovsky

author

2018-11-30 11:22 UTC #2 lore raw

PATCH 2/3 IB/mlx5: Add packet based credit mode support

In reply to Leon Romanovsky

From: Danit Goldberg <[email protected]>

The device can support two credit modes, message based (default) and packet based. In order to enable packet based mode, the QP should be created with special flag that indicates this.

This patch adds support for the new DV QP creation flag that can be used for RC QPs in order to change the credit mode.

Signed-off-by Danit Goldberg <[email protected]>

Reviewed-by Majd Dibbiny <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/infiniband/hw/mlx5/mlx5_ib.h |  1 +
 drivers/infiniband/hw/mlx5/qp.c      | 15 +++++++++++++--
 include/uapi/rdma/mlx5-abi.h         |  1 +
 3 files changed, 15 insertions(+), 2 deletions(-)

drivers/infiniband/hw/mlx5/mlx5_ib.h +1 -0

index 4ad493f581de..95652ef8d89b 100644
@@ -462,6 +462,7 @@ enum mlx5_ib_qp_flags {
 	MLX5_IB_QP_UNDERLAY			= 1 << 10,
 	MLX5_IB_QP_PCI_WRITE_END_PADDING	= 1 << 11,
 	MLX5_IB_QP_TUNNEL_OFFLOAD		= 1 << 12,
+	MLX5_IB_QP_PACKET_BASED_CREDIT		= 1 << 13,
 };
 
 struct mlx5_umr_wr {

drivers/infiniband/hw/mlx5/qp.c +13 -2

index 369db954edbe..485d3c838bf2 100644
@@ -1917,7 +1917,8 @@ static int create_qp_common(struct mlx5_ib_dev *dev, struct ib_pd *pd,
 					      MLX5_QP_FLAG_BFREG_INDEX |
 					      MLX5_QP_FLAG_TYPE_DCT |
 					      MLX5_QP_FLAG_TYPE_DCI |
-					      MLX5_QP_FLAG_ALLOW_SCATTER_CQE))
+					      MLX5_QP_FLAG_ALLOW_SCATTER_CQE |
+					      MLX5_QP_FLAG_PACKET_BASED_CREDIT_MODE))
 			return -EINVAL;
 
 		err = get_qp_user_index(to_mucontext(pd->uobject->context),

@@ -1953,6 +1954,15 @@ static int create_qp_common(struct mlx5_ib_dev *dev, struct ib_pd *pd,
 			qp->flags_en |= MLX5_QP_FLAG_TIR_ALLOW_SELF_LB_MC;
 		}
 
+		if (ucmd.flags & MLX5_QP_FLAG_PACKET_BASED_CREDIT_MODE) {
+			if (init_attr->qp_type != IB_QPT_RC ||
+				!MLX5_CAP_GEN(dev->mdev, qp_packet_based)) {
+				mlx5_ib_dbg(dev, "packet based credit mode isn't supported\n");
+				return -EOPNOTSUPP;
+			}
+			qp->flags |= MLX5_IB_QP_PACKET_BASED_CREDIT;
+		}
+
 		if (init_attr->create_flags & IB_QP_CREATE_SOURCE_QPN) {
 			if (init_attr->qp_type != IB_QPT_UD ||
 			    (MLX5_CAP_GEN(dev->mdev, port_type) !=

@@ -2049,7 +2059,8 @@ static int create_qp_common(struct mlx5_ib_dev *dev, struct ib_pd *pd,
 		MLX5_SET(qpc, qpc, cd_slave_send, 1);
 	if (qp->flags & MLX5_IB_QP_MANAGED_RECV)
 		MLX5_SET(qpc, qpc, cd_slave_receive, 1);
-
+	if (qp->flags & MLX5_IB_QP_PACKET_BASED_CREDIT)
+		MLX5_SET(qpc, qpc, req_e2e_credit_mode, 1);
 	if (qp->scat_cqe && is_connected(init_attr->qp_type)) {
 		configure_responder_scat_cqe(init_attr, qpc);
 		configure_requester_scat_cqe(dev, init_attr,

include/uapi/rdma/mlx5-abi.h +1 -0

index 8fa9f90e2bb1..4af581170f62 100644
@@ -48,6 +48,7 @@ enum {
 	MLX5_QP_FLAG_TIR_ALLOW_SELF_LB_UC = 1 << 6,
 	MLX5_QP_FLAG_TIR_ALLOW_SELF_LB_MC = 1 << 7,
 	MLX5_QP_FLAG_ALLOW_SCATTER_CQE	= 1 << 8,
+	MLX5_QP_FLAG_PACKET_BASED_CREDIT_MODE	= 1 << 9,
 };
 
 enum {

Signature · 1 line

2.19.1

Leon Romanovsky

author

2018-11-30 11:22 UTC #3 lore raw

PATCH 3/3 IB/mlx5: Report packet based credit mode device capability

In reply to Leon Romanovsky

From: Danit Goldberg <[email protected]>

Report packet based credit mode capability via the mlx5 DV interface.

Signed-off-by Danit Goldberg <[email protected]>

Reviewed-by Majd Dibbiny <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/infiniband/hw/mlx5/main.c | 3 +++
 include/uapi/rdma/mlx5-abi.h      | 1 +
 2 files changed, 4 insertions(+)

drivers/infiniband/hw/mlx5/main.c +3 -0

index 0acf6a4d8f4b..1ad5e35663af 100644
@@ -1018,6 +1018,9 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
 
 		if (MLX5_CAP_GEN(mdev, cqe_128_always))
 			resp.flags |= MLX5_IB_QUERY_DEV_RESP_FLAGS_CQE_128B_PAD;
+		if (MLX5_CAP_GEN(mdev, qp_packet_based))
+			resp.flags |=
+				MLX5_IB_QUERY_DEV_RESP_PACKET_BASED_CREDIT_MODE;
 	}
 
 	if (field_avail(typeof(resp), sw_parsing_caps,

include/uapi/rdma/mlx5-abi.h +1 -0

index 4af581170f62..87b3198f4b5d 100644
@@ -237,6 +237,7 @@ enum mlx5_ib_query_dev_resp_flags {
 	/* Support 128B CQE compression */
 	MLX5_IB_QUERY_DEV_RESP_FLAGS_CQE_128B_COMP = 1 << 0,
 	MLX5_IB_QUERY_DEV_RESP_FLAGS_CQE_128B_PAD  = 1 << 1,
+	MLX5_IB_QUERY_DEV_RESP_PACKET_BASED_CREDIT_MODE = 1 << 2,
 };
 
 enum mlx5_ib_tunnel_offloads {

Signature · 1 line

2.19.1

Jason Gunthorpe

2018-12-07 03:27 UTC #4 lore raw

In reply to Leon Romanovsky

Quoted 17 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>
Quoted 1 line
From Danit,
Packet based credit mode is an alternative end-to-end credit mode for QPs set during their creation. Credits are transported from the responder to the requester to optimize the use of its receive resources. In packet-based credit mode, credits are issued on a per packet basis.
The advantage of this feature comes while sending large RDMA messages through switches that are short in memory.
The first commit exposes QP creation flag and the HCA capability. The second commit adds support for a new DV QP creation flag. The last commit report packet based credit mode capability via the MLX5DV device capabilities.

Quoted 6 lines

Thanks
Danit Goldberg (3):
net/mlx5: Expose packet based credit mode IB/mlx5: Add packet based credit mode support IB/mlx5: Report packet based credit mode device capability

This looks fine to me, can you update the shared branch please

Thanks,
Jason

Leon Romanovsky

author

2018-12-07 06:04 UTC #5 lore raw

Quoted 18 lines · Jason Gunthorpe

Quoted 17 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Quoted 1 line
From Danit,
Packet based credit mode is an alternative end-to-end credit mode for QPs set during their creation. Credits are transported from the responder to the requester to optimize the use of its receive resources. In packet-based credit mode, credits are issued on a per packet basis.
The advantage of this feature comes while sending large RDMA messages through switches that are short in memory.
The first commit exposes QP creation flag and the HCA capability. The second commit adds support for a new DV QP creation flag. The last commit report packet based credit mode capability via the MLX5DV device capabilities.

Quoted 8 lines

Quoted 6 lines
Thanks
Danit Goldberg (3):
net/mlx5: Expose packet based credit mode IB/mlx5: Add packet based credit mode support IB/mlx5: Report packet based credit mode device capability
This looks fine to me, can you update the shared branch please

Done, thanks
3fd3c80acc17 net/mlx5: Expose packet based credit mode

Quoted 3 lines

Thanks,
Jason

Jason Gunthorpe

Applied

2018-12-07 20:29 UTC #6 lore raw

Quoted 26 lines · Leon Romanovsky

Quoted 25 lines · Jason Gunthorpe
Quoted 23 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Quoted 1 line
From Danit,
Packet based credit mode is an alternative end-to-end credit mode for QPs set during their creation. Credits are transported from the responder to the requester to optimize the use of its receive resources. In packet-based credit mode, credits are issued on a per packet basis.
The advantage of this feature comes while sending large RDMA messages through switches that are short in memory.
The first commit exposes QP creation flag and the HCA capability. The second commit adds support for a new DV QP creation flag. The last commit report packet based credit mode capability via the MLX5DV device capabilities.
Thanks
Danit Goldberg (3):
net/mlx5: Expose packet based credit mode IB/mlx5: Add packet based credit mode support IB/mlx5: Report packet based credit mode device capability

Quoted 4 lines

Quoted 1 line
This looks fine to me, can you update the shared branch please
Done, thanks
3fd3c80acc17 net/mlx5: Expose packet based credit mode

Applied to for-next

Thanks,
Jason