Skip to contentSkip to conversation

Create UMR with relaxed ordering

PATCH Applied Leon Romanovsky 4 participants 7 messages Jul 16, 2020 – Jul 27, 2020

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

cover Create UMR with relaxed ordering

From: Leon Romanovsky <[email protected]>

Hi,

ConnectX-7 supports setting relaxed ordering read/write mkey attribute by UMR, indicated by new HCA capabilities, so extend mlx5_ib driver to configure UMR control segment based on those capabilities.

Thanks

Meir Lichtinger (3):
RDMA/mlx5: ConnectX-7 new capabilities to set relaxed ordering by UMR RDMA/mlx5: Use MLX5_SET macro instead of local structure RDMA/mlx5: Set mkey relaxed ordering by UMR with ConnectX-7

 drivers/infiniband/hw/mlx5/mlx5_ib.h | 18 +++-----
 drivers/infiniband/hw/mlx5/wr.c      | 68 ++++++++++++++++++++--------
 include/linux/mlx5/device.h          |  5 +-
 include/linux/mlx5/mlx5_ifc.h        |  4 +-
 4 files changed, 63 insertions(+), 32 deletions(-)

--
2.26.2

#1 lore raw

PATCH 1/3 RDMA/mlx5: ConnectX-7 new capabilities to set relaxed ordering by UMR

From: Meir Lichtinger <[email protected]>

Up to ConnectX-7 setting mkey relaxed ordering read/write attributes by UMR is not supported. ConnectX-7 supports this option, which is indicated by two new HCA capabilities - relaxed_ordering_write_umr and relaxed_ordering_read_umr.

---
Not based on latest mlx5-next.

---
 include/linux/mlx5/mlx5_ifc.h | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
include/linux/mlx5/mlx5_ifc.h +3 -1
index d22b3ff99b7a..2999a026938a 100644
@@ -1216,7 +1216,9 @@ struct mlx5_ifc_cmd_hca_cap_bits {
 
 	u8         max_sgl_for_optimized_performance[0x8];
 	u8         log_max_cq_sz[0x8];
-	u8         reserved_at_d0[0xb];
+	u8         relaxed_ordering_write_umr[0x1];
+	u8         relaxed_ordering_read_umr[0x1];
+	u8         reserved_at_d2[0x9];
 	u8         log_max_cq[0x5];
 
 	u8         log_max_eq_sz[0x8];

--
2.26.2

#2 lore raw

PATCH 2/3 RDMA/mlx5: Use MLX5_SET macro instead of local structure

In reply to Leon Romanovsky

From: Meir Lichtinger <[email protected]>

Use generic mlx5 structure defined in mlx5_ifc.h to represent ConnectX device data structures instead of using structure defined specifically for mlx5_ib module

---
 drivers/infiniband/hw/mlx5/mlx5_ib.h |  9 ---------
 drivers/infiniband/hw/mlx5/wr.c      | 26 ++++++++++++++++----------
 include/linux/mlx5/device.h          |  1 -
 3 files changed, 16 insertions(+), 20 deletions(-)
drivers/infiniband/hw/mlx5/mlx5_ib.h +0 -9
index 13a2d1f3f14d..52c384214f26 100644
@@ -1361,15 +1361,6 @@ static inline void init_query_mad(struct ib_smp *mad)
 	mad->method	   = IB_MGMT_METHOD_GET;
 }
 
-static inline u8 convert_access(int acc)
-{
-	return (acc & IB_ACCESS_REMOTE_ATOMIC ? MLX5_PERM_ATOMIC       : 0) |
-	       (acc & IB_ACCESS_REMOTE_WRITE  ? MLX5_PERM_REMOTE_WRITE : 0) |
-	       (acc & IB_ACCESS_REMOTE_READ   ? MLX5_PERM_REMOTE_READ  : 0) |
-	       (acc & IB_ACCESS_LOCAL_WRITE   ? MLX5_PERM_LOCAL_WRITE  : 0) |
-	       MLX5_PERM_LOCAL_READ;
-}
-
 static inline int is_qp1(enum ib_qp_type qp_type)
 {
 	return qp_type == MLX5_IB_QPT_HW_GSI;
drivers/infiniband/hw/mlx5/wr.c +16 -10
index 2c6df1c43b55..e58ecb46f8e3 100644
@@ -383,20 +383,26 @@ static void set_reg_mkey_segment(struct mlx5_mkey_seg *seg,
 
 	memset(seg, 0, sizeof(*seg));
 	if (wr->send_flags & MLX5_IB_SEND_UMR_DISABLE_MR)
-		seg->status = MLX5_MKEY_STATUS_FREE;
-
-	seg->flags = convert_access(umrwr->access_flags);
+		MLX5_SET(mkc, seg, free, 1);
+
+	MLX5_SET(mkc, seg, a,
+		 !!(umrwr->access_flags & IB_ACCESS_REMOTE_ATOMIC));
+	MLX5_SET(mkc, seg, rw,
+		 !!(umrwr->access_flags & IB_ACCESS_REMOTE_WRITE));
+	MLX5_SET(mkc, seg, rr, !!(umrwr->access_flags & IB_ACCESS_REMOTE_READ));
+	MLX5_SET(mkc, seg, lw, !!(umrwr->access_flags & IB_ACCESS_LOCAL_WRITE));
+	MLX5_SET(mkc, seg, lr, 1);
 	if (umrwr->pd)
-		seg->flags_pd = cpu_to_be32(to_mpd(umrwr->pd)->pdn);
+		MLX5_SET(mkc, seg, pd, to_mpd(umrwr->pd)->pdn);
 	if (wr->send_flags & MLX5_IB_SEND_UMR_UPDATE_TRANSLATION &&
 	    !umrwr->length)
-		seg->flags_pd |= cpu_to_be32(MLX5_MKEY_LEN64);
+		MLX5_SET(mkc, seg, length64, 1);
 
-	seg->start_addr = cpu_to_be64(umrwr->virt_addr);
-	seg->len = cpu_to_be64(umrwr->length);
-	seg->log2_page_size = umrwr->page_shift;
-	seg->qpn_mkey7_0 = cpu_to_be32(0xffffff00 |
-				       mlx5_mkey_variant(umrwr->mkey));
+	MLX5_SET64(mkc, seg, start_addr, umrwr->virt_addr);
+	MLX5_SET64(mkc, seg, len, umrwr->length);
+	MLX5_SET(mkc, seg, log_page_size, umrwr->page_shift);
+	MLX5_SET(mkc, seg, qpn, 0xffffff);
+	MLX5_SET(mkc, seg, mkey_7_0, mlx5_mkey_variant(umrwr->mkey));
 }
 
 static void set_reg_data_seg(struct mlx5_wqe_data_seg *dseg,
include/linux/mlx5/device.h +0 -1
index 57db125e5802..7940a574a618 100644
@@ -1007,7 +1007,6 @@ enum {
 	MLX5_MKEY_REMOTE_INVAL	= 1 << 24,
 	MLX5_MKEY_FLAG_SYNC_UMR = 1 << 29,
 	MLX5_MKEY_BSF_EN	= 1 << 30,
-	MLX5_MKEY_LEN64		= 1 << 31,
 };
 
 struct mlx5_mkey_seg {

--
2.26.2

#3 lore raw

PATCH 3/3 RDMA/mlx5: Set mkey relaxed ordering by UMR with ConnectX-7

In reply to Leon Romanovsky

From: Meir Lichtinger <[email protected]>

Up to ConnectX-7 UMR is not used when user passes relaxed ordering access flag. ConnectX-7 supports setting relaxed ordering read/write mkey attribute by UMR, indicated by new HCA capabilities.

With ConnectX-7 driver uses UMR when user set relaxed ordering access flag, in contrast to previous silicon models. Specifically it includes setting relvant flags of mkey context mask in UMR control segment, and relaxed ordering write and read flags in UMR mkey context segment.

---
 drivers/infiniband/hw/mlx5/mlx5_ib.h |  9 ++++--
 drivers/infiniband/hw/mlx5/wr.c      | 44 ++++++++++++++++++++++------
 include/linux/mlx5/device.h          |  4 ++-
 3 files changed, 45 insertions(+), 12 deletions(-)
drivers/infiniband/hw/mlx5/mlx5_ib.h +7 -2
index 52c384214f26..06ee2322052d 100644
@@ -1459,8 +1459,13 @@ static inline bool mlx5_ib_can_use_umr(struct mlx5_ib_dev *dev,
 		return false;
 
 	if (access_flags & IB_ACCESS_RELAXED_ORDERING &&
-	    (MLX5_CAP_GEN(dev->mdev, relaxed_ordering_write) ||
-	     MLX5_CAP_GEN(dev->mdev, relaxed_ordering_read)))
+	    MLX5_CAP_GEN(dev->mdev, relaxed_ordering_write) &&
+	    !MLX5_CAP_GEN(dev->mdev, relaxed_ordering_write_umr))
+		return false;
+
+	if (access_flags & IB_ACCESS_RELAXED_ORDERING &&
+	     MLX5_CAP_GEN(dev->mdev, relaxed_ordering_read) &&
+	     !MLX5_CAP_GEN(dev->mdev, relaxed_ordering_read_umr))
 		return false;
 
 	return true;
drivers/infiniband/hw/mlx5/wr.c +35 -9

86 lines not on this page · raw message

include/linux/mlx5/device.h +3 -1
index 7940a574a618..b1a891576fbe 100644
@@ -276,7 +276,9 @@ enum {
 	MLX5_MKEY_MASK_RW		= 1ull << 20,
 	MLX5_MKEY_MASK_A		= 1ull << 21,
 	MLX5_MKEY_MASK_SMALL_FENCE	= 1ull << 23,
-	MLX5_MKEY_MASK_FREE		= 1ull << 29,
+	MLX5_MKEY_MASK_RELAXED_ORDERING_WRITE	= 1ull << 25,
+	MLX5_MKEY_MASK_FREE			= 1ull << 29,
+	MLX5_MKEY_MASK_RELAXED_ORDERING_READ	= 1ull << 47,
 };
 
 enum {

--
2.26.2

maintainer
#4 lore raw
In reply to Leon Romanovsky
Quoted 8 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Hi,

ConnectX-7 supports setting relaxed ordering read/write mkey attribute by UMR, indicated by new HCA capabilities, so extend mlx5_ib driver to configure UMR control segment based on those capabilities.

Quoted 6 lines

Thanks

Meir Lichtinger (3):
RDMA/mlx5: ConnectX-7 new capabilities to set relaxed ordering by UMR RDMA/mlx5: Use MLX5_SET macro instead of local structure RDMA/mlx5: Set mkey relaxed ordering by UMR with ConnectX-7

These look OK, can you apply the first patch to the shared branch please?

Jason

#5 lore raw
In reply to Leon Romanovsky
Quoted 15 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Hi,

ConnectX-7 supports setting relaxed ordering read/write mkey attribute by UMR, indicated by new HCA capabilities, so extend mlx5_ib driver to configure UMR control segment based on those capabilities.

Thanks

Meir Lichtinger (3):
RDMA/mlx5: ConnectX-7 new capabilities to set relaxed ordering by UMR RDMA/mlx5: Use MLX5_SET macro instead of local structure RDMA/mlx5: Set mkey relaxed ordering by UMR with ConnectX-7

Quoted 5 lines
 drivers/infiniband/hw/mlx5/mlx5_ib.h | 18 +++-----
 drivers/infiniband/hw/mlx5/wr.c      | 68 ++++++++++++++++++++--------
 include/linux/mlx5/device.h          |  5 +-
 include/linux/mlx5/mlx5_ifc.h        |  4 +-
 4 files changed, 63 insertions(+), 32 deletions(-)

Thanks, first patch is applied to mlx5-next.
042dd05bddbd RDMA/mlx5: ConnectX-7 new capabilities to set relaxed ordering by UMR

Quoted 4 lines

--
2.26.2

#6 lore raw
Quoted 16 lines · Leon Romanovsky
Quoted 15 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Hi,

ConnectX-7 supports setting relaxed ordering read/write mkey attribute by UMR, indicated by new HCA capabilities, so extend mlx5_ib driver to configure UMR control segment based on those capabilities.

Thanks

Meir Lichtinger (3):
RDMA/mlx5: ConnectX-7 new capabilities to set relaxed ordering by UMR RDMA/mlx5: Use MLX5_SET macro instead of local structure RDMA/mlx5: Set mkey relaxed ordering by UMR with ConnectX-7

Quoted 8 lines
Quoted 5 lines
 drivers/infiniband/hw/mlx5/mlx5_ib.h | 18 +++-----
 drivers/infiniband/hw/mlx5/wr.c      | 68 ++++++++++++++++++++--------
 include/linux/mlx5/device.h          |  5 +-
 include/linux/mlx5/mlx5_ifc.h        |  4 +-
 4 files changed, 63 insertions(+), 32 deletions(-)

Thanks, first patch is applied to mlx5-next.
042dd05bddbd RDMA/mlx5: ConnectX-7 new capabilities to set relaxed ordering by UMR

Applied to for-next, thanks

Jason