Skip to contentSkip to conversation

Enable IB out-of-order by default in mlx5

PATCH Applied Leon Romanovsky 1 participant 5 messages Mar 19, 2023 – Mar 23, 2023

Subsystems (5)
Networking Network drivers InfiniBand Mellanox mlx5 core +1 more subsystems
lore
Share & tools
lore

Leon Romanovsky

author
#0 lore raw

cover Enable IB out-of-order by default in mlx5

From: Leon Romanovsky <[email protected]>

Hi,

This series from Or changes default of IB out-of-order feature and allows to the RDMA users to decide if they need to wait for completion for all segments or it is enough to wait for last segment completion only.

Thanks

Or Har-Toov (3):
net/mlx5: Expose bits for enabling out-of-order by default RDMA/mlx5: Disable out-of-order in integrity enabled QPs net/mlx5: Set out of order (ooo) by default

 drivers/infiniband/hw/mlx5/qp.c                |  8 ++++++++
 drivers/net/ethernet/mellanox/mlx5/core/main.c |  3 +++
 include/linux/mlx5/mlx5_ifc.h                  | 10 +++++++---
 3 files changed, 18 insertions(+), 3 deletions(-)
Signature · 1 line
2.39.2
author
#1 lore raw

PATCH 1/3 net/mlx5: Expose bits for enabling out-of-order by default

From: Or Har-Toov <[email protected]>

Add needed HW bits for enabling out-of-order by default and use go_back_n when out-of-order is not needed.

Signed-off-by Or Har-Toov <[email protected]>
Signed-off-by Leon Romanovsky <[email protected]>
---
 include/linux/mlx5/mlx5_ifc.h | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
include/linux/mlx5/mlx5_ifc.h +7 -3
index 15a850f52ef2..e4306cd87cd7 100644
@@ -1077,7 +1077,9 @@ struct mlx5_ifc_roce_cap_bits {
 	u8         sw_r_roce_src_udp_port[0x1];
 	u8         fl_rc_qp_when_roce_disabled[0x1];
 	u8         fl_rc_qp_when_roce_enabled[0x1];
-	u8         reserved_at_7[0x17];
+	u8         reserved_at_7[0x1];
+	u8	   qp_ooo_transmit_default[0x1];
+	u8         reserved_at_9[0x15];
 	u8	   qp_ts_format[0x2];
 
 	u8         reserved_at_20[0x60];

@@ -1493,7 +1495,8 @@ struct mlx5_ifc_cmd_hca_cap_bits {
 	u8         reserved_at_b0[0x1];
 	u8         uplink_follow[0x1];
 	u8         ts_cqe_to_dest_cqn[0x1];
-	u8         reserved_at_b3[0x7];
+	u8         reserved_at_b3[0x6];
+	u8         go_back_n[0x1];
 	u8         shampo[0x1];
 	u8         reserved_at_bb[0x5];
 

@@ -3263,7 +3266,8 @@ struct mlx5_ifc_qpc_bits {
 	u8         log_rq_stride[0x3];
 	u8         no_sq[0x1];
 	u8         log_sq_size[0x4];
-	u8         reserved_at_55[0x3];
+	u8         reserved_at_55[0x1];
+	u8	   retry_mode[0x2];
 	u8	   ts_format[0x2];
 	u8         reserved_at_5a[0x1];
 	u8         rlky[0x1];
Signature · 1 line
2.39.2
author
#2 lore raw

PATCH 2/3 RDMA/mlx5: Disable out-of-order in integrity enabled QPs

In reply to Leon Romanovsky

From: Or Har-Toov <[email protected]>

Set retry_mode to GO_BACK_N when qp is created with INTEGRITY_EN flag because out-of-order is not supported when doing HW offload of signature operations.

Signed-off-by Or Har-Toov <[email protected]>
Signed-off-by Leon Romanovsky <[email protected]>
---
 drivers/infiniband/hw/mlx5/qp.c | 8 ++++++++
 1 file changed, 8 insertions(+)
drivers/infiniband/hw/mlx5/qp.c +8 -0
index f04adc18e63b..d32b644885b3 100644
@@ -60,6 +60,10 @@ enum raw_qp_set_mask_map {
 	MLX5_RAW_QP_RATE_LIMIT			= 1UL << 1,
 };
 
+enum {
+	MLX5_QP_RM_GO_BACK_N			= 0x1,
+};
+
 struct mlx5_modify_raw_qp_param {
 	u16 operation;
 

@@ -2521,6 +2525,10 @@ static int create_kernel_qp(struct mlx5_ib_dev *dev, struct ib_pd *pd,
 	if (qp->flags & IB_QP_CREATE_IPOIB_UD_LSO)
 		MLX5_SET(qpc, qpc, ulp_stateless_offload_mode, 1);
 
+	if (qp->flags & IB_QP_CREATE_INTEGRITY_EN &&
+	    MLX5_CAP_GEN(mdev, go_back_n))
+		MLX5_SET(qpc, qpc, retry_mode, MLX5_QP_RM_GO_BACK_N);
+
 	err = mlx5_qpc_create_qp(dev, &base->mqp, in, inlen, out);
 	kvfree(in);
 	if (err)
Signature · 1 line
2.39.2
author
#3 lore raw

PATCH 3/3 net/mlx5: Set out of order (ooo) by default

In reply to Leon Romanovsky

From: Or Har-Toov <[email protected]>

When FW supports ooo by default, enable the cap.

Signed-off-by Or Har-Toov <[email protected]>
Signed-off-by Leon Romanovsky <[email protected]>
---
 drivers/net/ethernet/mellanox/mlx5/core/main.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/main.c +3 -0
index 540840e80493..564a82ac3787 100644
@@ -684,6 +684,9 @@ static int handle_hca_cap_roce(struct mlx5_core_dev *dev, void *set_ctx)
 	       MLX5_ST_SZ_BYTES(roce_cap));
 	MLX5_SET(roce_cap, set_hca_cap, sw_r_roce_src_udp_port, 1);
 
+	if (MLX5_CAP_ROCE_MAX(dev, qp_ooo_transmit_default))
+		MLX5_SET(roce_cap, set_hca_cap, qp_ooo_transmit_default, 1);
+
 	err = set_caps(dev, set_ctx, MLX5_SET_HCA_CAP_OP_MOD_ROCE);
 	return err;
 }
Signature · 1 line
2.39.2
author Applied
#4 lore raw
In reply to Leon Romanovsky
Quoted 9 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Hi,

This series from Or changes default of IB out-of-order feature and allows to the RDMA users to decide if they need to wait for completion for all segments or it is enough to wait for last segment completion only.

[...]

Applied, thanks!

[1/3] net/mlx5: Expose bits for enabling out-of-order by default

      https://git.kernel.org/rdma/rdma/c/6e2a3a324aab9d
[2/3] RDMA/mlx5: Disable out-of-order in integrity enabled QPs
      https://git.kernel.org/rdma/rdma/c/742948cc02d523
[3/3] net/mlx5: Set out of order (ooo) by default
      https://git.kernel.org/rdma/rdma/c/f4244e55e4c3a1

Best regards,

Signature · 1 line
Leon Romanovsky <[email protected]>