Skip to contentSkip to conversation

RDMA/hns: Updates for 5.12

PATCH In review Weihang Li 2 participants 16 messages Feb 5, 2021 – Feb 9, 2021

Subsystems (2)
lore
Share & tools
lore

Weihang Li

author
#0 lore raw

cover RDMA/hns: Updates for 5.12

As usual, this series collects some miscellaneous fixes and cleanups at the end of 5.12 for the hns driver:
* #1 ~ #3 fix some non-urgent issues.
* #4 ~ #6 make some changes on existing features.
* #7 ~ #12 are cleanups, just do some refactor and delete dead code.

This series is made based on the series named "RDMA/hns: Several fixes and cleanups of RQ/SRQ" which is still being reviewed:
https://patchwork.kernel.org/project/linux-rdma/cover/[email protected]/

I'm worried that it will be too late for 5.12 if any patch of this series needs changes, so I send it before the previous one is merged. If there is any comment or merge conflict on it, I will fix them as soon as possible, thanks.

Lang Cheng (3):
RDMA/hns: Replace wmb&__raw_writeq with writeq RDMA/hns: Move HIP06 related definitions into hns_roce_hw_v1.h RDMA/hns: Avoid unnecessary memset on WQEs in post_send

Lijun Ou (1):
RDMA/hns: Disable RQ inline by default

Weihang Li (2):
RDMA/hns: Avoid filling sgid index when modifying QP to RTR RDMA/hns: Fix type of sq_signal_bits

Xi Wang (1):
RDMA/hns: Add mapped page count checking for MTR

Xinhao Liu (2):
RDMA/hns: Remove some magic numbers
RDMA/hns: Delete redundant judgment when preparing descriptors

Yixian Liu (1):
RDMA/hns: Remove unnecessary wrap around for EQ's consumer index

Yixing Liu (2):
RDMA/hns: Adjust definition of FRMR fields
RDMA/hns: Skip qp_flow_control_init() for HIP09

 drivers/infiniband/hw/hns/hns_roce_device.h |  45 +---------
 drivers/infiniband/hw/hns/hns_roce_hem.c    |   9 +-
 drivers/infiniband/hw/hns/hns_roce_hw_v1.c  |  31 +++----
 drivers/infiniband/hw/hns/hns_roce_hw_v1.h  |  43 ++++++++++
 drivers/infiniband/hw/hns/hns_roce_hw_v2.c  | 125 +++++++++-------------------
 drivers/infiniband/hw/hns/hns_roce_hw_v2.h  |  12 +--
 drivers/infiniband/hw/hns/hns_roce_main.c   |  16 ----
 drivers/infiniband/hw/hns/hns_roce_mr.c     |  56 ++++++++-----
 8 files changed, 148 insertions(+), 189 deletions(-)
Signature · 1 line
2.8.1
author
#1 lore raw

PATCH 1/12 RDMA/hns: Avoid filling sgid index when modifying QP to RTR

ULP usually set IB(V)_QP_AV when trying to modify QP to RTR if they want to record sgid index into QPC. For UD QPs, it is useless because it will be included in WQE. For RC QPs, it will be filled in hns_roce_set_path(). So sgid index shouldn't be filled by default. Then hns_get_gid_index() is moved to hns_roce_hw_v1.c because it is only called in it.

Fixes 926a01dc000d ("RDMA/hns: Add QP operations support for hip08 SoC")
---
 drivers/infiniband/hw/hns/hns_roce_hw_v1.c | 16 ++++++++++++++++
 drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 11 -----------
 drivers/infiniband/hw/hns/hns_roce_main.c  | 16 ----------------
 3 files changed, 16 insertions(+), 27 deletions(-)
drivers/infiniband/hw/hns/hns_roce_hw_v1.c +16 -0
index 23fe8e9..262ad58 100644
@@ -43,6 +43,22 @@
 #include "hns_roce_hem.h"
 #include "hns_roce_hw_v1.h"
 
+/**
+ * hns_get_gid_index - Get gid index.
+ * @hr_dev: pointer to structure hns_roce_dev.
+ * @port:  port, value range: 0 ~ MAX
+ * @gid_index:  gid_index, value range: 0 ~ MAX
+ * Description:
+ *    N ports shared gids, allocation method as follow:
+ *		GID[0][0], GID[1][0],.....GID[N - 1][0],
+ *		GID[0][0], GID[1][0],.....GID[N - 1][0],
+ *		And so on
+ */
+u8 hns_get_gid_index(struct hns_roce_dev *hr_dev, u8 port, int gid_index)
+{
+	return gid_index * hr_dev->caps.num_ports + port;
+}
+
 static void set_data_seg(struct hns_roce_wqe_data_seg *dseg, struct ib_sge *sg)
 {
 	dseg->lkey = cpu_to_le32(sg->lkey);
drivers/infiniband/hw/hns/hns_roce_hw_v2.c +0 -11
index c962f26..f0691f4 100644
@@ -4308,7 +4308,6 @@ static int modify_qp_init_to_rtr(struct ib_qp *ibqp,
 				 struct hns_roce_v2_qp_context *context,
 				 struct hns_roce_v2_qp_context *qpc_mask)
 {
-	const struct ib_global_route *grh = rdma_ah_read_grh(&attr->ah_attr);
 	struct hns_roce_dev *hr_dev = to_hr_dev(ibqp->device);
 	struct hns_roce_qp *hr_qp = to_hr_qp(ibqp);
 	struct ib_device *ibdev = &hr_dev->ib_dev;
@@ -4316,7 +4315,6 @@ static int modify_qp_init_to_rtr(struct ib_qp *ibqp,
 	dma_addr_t irrl_ba;
 	enum ib_mtu mtu;
 	u8 lp_pktn_ini;
-	u8 port_num;
 	u64 *mtts;
 	u8 *dmac;
 	u8 *smac;
@@ -4397,15 +4395,6 @@ static int modify_qp_init_to_rtr(struct ib_qp *ibqp,
 			       V2_QPC_BYTE_56_DQPN_M, V2_QPC_BYTE_56_DQPN_S, 0);
 	}
 
-	/* Configure GID index */
-	port_num = rdma_ah_get_port_num(&attr->ah_attr);
-	roce_set_field(context->byte_20_smac_sgid_idx,
-		       V2_QPC_BYTE_20_SGID_IDX_M, V2_QPC_BYTE_20_SGID_IDX_S,
-		       hns_get_gid_index(hr_dev, port_num - 1,
-					 grh->sgid_index));
-	roce_set_field(qpc_mask->byte_20_smac_sgid_idx,
-		       V2_QPC_BYTE_20_SGID_IDX_M, V2_QPC_BYTE_20_SGID_IDX_S, 0);
-
 	memcpy(&(context->dmac), dmac, sizeof(u32));
 	roce_set_field(context->byte_52_udpspn_dmac, V2_QPC_BYTE_52_DMAC_M,
 		       V2_QPC_BYTE_52_DMAC_S, *((u16 *)(&dmac[4])));
drivers/infiniband/hw/hns/hns_roce_main.c +0 -16
index 7978220..c29215a 100644
@@ -42,22 +42,6 @@
 #include "hns_roce_device.h"
 #include "hns_roce_hem.h"
 
-/**
- * hns_get_gid_index - Get gid index.
- * @hr_dev: pointer to structure hns_roce_dev.
- * @port:  port, value range: 0 ~ MAX
- * @gid_index:  gid_index, value range: 0 ~ MAX
- * Description:
- *    N ports shared gids, allocation method as follow:
- *		GID[0][0], GID[1][0],.....GID[N - 1][0],
- *		GID[0][0], GID[1][0],.....GID[N - 1][0],
- *		And so on
- */
-u8 hns_get_gid_index(struct hns_roce_dev *hr_dev, u8 port, int gid_index)
-{
-	return gid_index * hr_dev->caps.num_ports + port;
-}
-
 static int hns_roce_set_mac(struct hns_roce_dev *hr_dev, u8 port, u8 *addr)
 {
 	u8 phy_port;
Signature · 1 line
2.8.1
author
#2 lore raw

PATCH 2/12 RDMA/hns: Fix type of sq_signal_bits

In reply to Weihang Li

This bit should be in type of enum ib_sig_type, or there will be a sparse warning.

Fixes bfe860351e31 ("RDMA/hns: Fix cast from or to restricted __le32 for driver")
---
 drivers/infiniband/hw/hns/hns_roce_device.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/hw/hns/hns_roce_device.h +1 -1
index c9a9e80..1f94154 100644
@@ -661,7 +661,7 @@ struct hns_roce_qp {
 	struct hns_roce_db	sdb;
 	unsigned long		en_flags;
 	u32			doorbell_qpn;
-	u32			sq_signal_bits;
+	enum ib_sig_type	sq_signal_bits;
 	struct hns_roce_wq	sq;
 
 	struct hns_roce_mtr	mtr;
Signature · 1 line
2.8.1
author
#3 lore raw

PATCH 3/12 RDMA/hns: Add mapped page count checking for MTR

In reply to Weihang Li

From: Xi Wang <[email protected]>

Add the mapped page count checking flow to avoid invalid page size when creating MTR.

Fixes 38389eaa4db1 ("RDMA/hns: Add mtr support for mixed multihop addressing")
---
 drivers/infiniband/hw/hns/hns_roce_hem.c |  9 +++--
 drivers/infiniband/hw/hns/hns_roce_mr.c  | 56 +++++++++++++++++++-------------
 2 files changed, 40 insertions(+), 25 deletions(-)
drivers/infiniband/hw/hns/hns_roce_hem.c +6 -3
index edc9d6b..cfd2e1b 100644
@@ -1075,9 +1075,8 @@ static struct roce_hem_item *hem_list_alloc_item(struct hns_roce_dev *hr_dev,
 		return NULL;
 
 	if (exist_bt) {
-		hem->addr = dma_alloc_coherent(hr_dev->dev,
-						   count * BA_BYTE_LEN,
-						   &hem->dma_addr, GFP_KERNEL);
+		hem->addr = dma_alloc_coherent(hr_dev->dev, count * BA_BYTE_LEN,
+					       &hem->dma_addr, GFP_KERNEL);
 		if (!hem->addr) {
 			kfree(hem);
 			return NULL;
@@ -1336,6 +1335,10 @@ static int hem_list_alloc_root_bt(struct hns_roce_dev *hr_dev,
 	if (ba_num < 1)
 		return -ENOMEM;
 
+	if (ba_num > unit)
+		return -ENOBUFS;
+
+	ba_num = min_t(int, ba_num, unit);
 	INIT_LIST_HEAD(&temp_root);
 	offset = r->offset;
 	/* indicate to last region */
drivers/infiniband/hw/hns/hns_roce_mr.c +34 -22

105 lines not on this page · raw message

Signature · 1 line
2.8.1
author
#4 lore raw

PATCH 4/12 RDMA/hns: Disable RQ inline by default

In reply to Weihang Li

From: Lijun Ou <[email protected]>

This feature should only be enabled by querying capability from firmware.

Fixes ba6bb7e97421 ("RDMA/hns: Add interfaces to get pf capabilities from firmware")
---
 drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 1 -
 1 file changed, 1 deletion(-)
drivers/infiniband/hw/hns/hns_roce_hw_v2.c +0 -1
index f0691f4..3ba6783 100644
@@ -1949,7 +1949,6 @@ static void set_default_caps(struct hns_roce_dev *hr_dev)
 
 	caps->flags		= HNS_ROCE_CAP_FLAG_REREG_MR |
 				  HNS_ROCE_CAP_FLAG_ROCE_V1_V2 |
-				  HNS_ROCE_CAP_FLAG_RQ_INLINE |
 				  HNS_ROCE_CAP_FLAG_RECORD_DB |
 				  HNS_ROCE_CAP_FLAG_SQ_RECORD_DB;
 
Signature · 1 line
2.8.1
author
#5 lore raw

PATCH 5/12 RDMA/hns: Adjust definition of FRMR fields

In reply to Weihang Li

From: Yixing Liu <[email protected]>

The position of the FRMR-related fields has changed, change them to fit the new design.

---
 drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 26 ++++++++++++--------------
 drivers/infiniband/hw/hns/hns_roce_hw_v2.h | 12 ++++++------
 2 files changed, 18 insertions(+), 20 deletions(-)
drivers/infiniband/hw/hns/hns_roce_hw_v2.c +12 -14
index 3ba6783..18a8ac9 100644
@@ -99,16 +99,16 @@ static void set_frmr_seg(struct hns_roce_v2_rc_send_wqe *rc_sq_wqe,
 	u64 pbl_ba;
 
 	/* use ib_access_flags */
-	roce_set_bit(rc_sq_wqe->byte_4, V2_RC_FRMR_WQE_BYTE_4_BIND_EN_S,
-		     wr->access & IB_ACCESS_MW_BIND ? 1 : 0);
-	roce_set_bit(rc_sq_wqe->byte_4, V2_RC_FRMR_WQE_BYTE_4_ATOMIC_S,
-		     wr->access & IB_ACCESS_REMOTE_ATOMIC ? 1 : 0);
-	roce_set_bit(rc_sq_wqe->byte_4, V2_RC_FRMR_WQE_BYTE_4_RR_S,
-		     wr->access & IB_ACCESS_REMOTE_READ ? 1 : 0);
-	roce_set_bit(rc_sq_wqe->byte_4, V2_RC_FRMR_WQE_BYTE_4_RW_S,
-		     wr->access & IB_ACCESS_REMOTE_WRITE ? 1 : 0);
-	roce_set_bit(rc_sq_wqe->byte_4, V2_RC_FRMR_WQE_BYTE_4_LW_S,
-		     wr->access & IB_ACCESS_LOCAL_WRITE ? 1 : 0);
+	roce_set_bit(fseg->byte_40, V2_RC_FRMR_WQE_BYTE_40_BIND_EN_S,
+		     !!(wr->access & IB_ACCESS_MW_BIND));
+	roce_set_bit(fseg->byte_40, V2_RC_FRMR_WQE_BYTE_40_ATOMIC_S,
+		     !!(wr->access & IB_ACCESS_REMOTE_ATOMIC));
+	roce_set_bit(fseg->byte_40, V2_RC_FRMR_WQE_BYTE_40_RR_S,
+		     !!(wr->access & IB_ACCESS_REMOTE_READ));
+	roce_set_bit(fseg->byte_40, V2_RC_FRMR_WQE_BYTE_40_RW_S,
+		     !!(wr->access & IB_ACCESS_REMOTE_WRITE));
+	roce_set_bit(fseg->byte_40, V2_RC_FRMR_WQE_BYTE_40_LW_S,
+		     !!(wr->access & IB_ACCESS_LOCAL_WRITE));
 
 	/* Data structure reuse may lead to confusion */
 	pbl_ba = mr->pbl_mtr.hem_cfg.root_ba;
@@ -121,12 +121,10 @@ static void set_frmr_seg(struct hns_roce_v2_rc_send_wqe *rc_sq_wqe,
 	rc_sq_wqe->va = cpu_to_le64(wr->mr->iova);
 
 	fseg->pbl_size = cpu_to_le32(mr->npages);
-	roce_set_field(fseg->mode_buf_pg_sz,
-		       V2_RC_FRMR_WQE_BYTE_40_PBL_BUF_PG_SZ_M,
+	roce_set_field(fseg->byte_40, V2_RC_FRMR_WQE_BYTE_40_PBL_BUF_PG_SZ_M,
 		       V2_RC_FRMR_WQE_BYTE_40_PBL_BUF_PG_SZ_S,
 		       to_hr_hw_page_shift(mr->pbl_mtr.hem_cfg.buf_pg_shift));
-	roce_set_bit(fseg->mode_buf_pg_sz,
-		     V2_RC_FRMR_WQE_BYTE_40_BLK_MODE_S, 0);
+	roce_set_bit(fseg->byte_40, V2_RC_FRMR_WQE_BYTE_40_BLK_MODE_S, 0);
 }
 
 static void set_atomic_seg(const struct ib_send_wr *wr,
drivers/infiniband/hw/hns/hns_roce_hw_v2.h +6 -6
index f29438c..1da980c 100644
@@ -1255,15 +1255,15 @@ struct hns_roce_v2_rc_send_wqe {
 
 #define V2_RC_SEND_WQE_BYTE_4_INLINE_S 12
 
-#define V2_RC_FRMR_WQE_BYTE_4_BIND_EN_S 19
+#define V2_RC_FRMR_WQE_BYTE_40_BIND_EN_S 10
 
-#define V2_RC_FRMR_WQE_BYTE_4_ATOMIC_S 20
+#define V2_RC_FRMR_WQE_BYTE_40_ATOMIC_S 11
 
-#define V2_RC_FRMR_WQE_BYTE_4_RR_S 21
+#define V2_RC_FRMR_WQE_BYTE_40_RR_S 12
 
-#define V2_RC_FRMR_WQE_BYTE_4_RW_S 22
+#define V2_RC_FRMR_WQE_BYTE_40_RW_S 13
 
-#define V2_RC_FRMR_WQE_BYTE_4_LW_S 23
+#define V2_RC_FRMR_WQE_BYTE_40_LW_S 14
 
 #define V2_RC_SEND_WQE_BYTE_4_FLAG_S 31
 
@@ -1280,7 +1280,7 @@ struct hns_roce_v2_rc_send_wqe {
 
 struct hns_roce_wqe_frmr_seg {
 	__le32	pbl_size;
-	__le32	mode_buf_pg_sz;
+	__le32	byte_40;
 };
 
 #define V2_RC_FRMR_WQE_BYTE_40_PBL_BUF_PG_SZ_S	4
Signature · 1 line
2.8.1
maintainer
#6 lore raw

Re: [PATCH for-next 05/12] RDMA/hns: Adjust definition of FRMR fields

Quoted 25 lines · Weihang Li
drivers/infiniband/hw/hns/hns_roce_hw_v2.h +6 -5
index f29438c..1da980c 100644
+++ b/drivers/infiniband/hw/hns/hns_roce_hw_v2.h
@@ -1255,15 +1255,15 @@ struct hns_roce_v2_rc_send_wqe {
 
 #define V2_RC_SEND_WQE_BYTE_4_INLINE_S 12
 
-#define V2_RC_FRMR_WQE_BYTE_4_BIND_EN_S 19
+#define V2_RC_FRMR_WQE_BYTE_40_BIND_EN_S 10
 
-#define V2_RC_FRMR_WQE_BYTE_4_ATOMIC_S 20
+#define V2_RC_FRMR_WQE_BYTE_40_ATOMIC_S 11
 
-#define V2_RC_FRMR_WQE_BYTE_4_RR_S 21
+#define V2_RC_FRMR_WQE_BYTE_40_RR_S 12
 
-#define V2_RC_FRMR_WQE_BYTE_4_RW_S 22
+#define V2_RC_FRMR_WQE_BYTE_40_RW_S 13
 
-#define V2_RC_FRMR_WQE_BYTE_4_LW_S 23
+#define V2_RC_FRMR_WQE_BYTE_40_LW_S 14
 
 #define V2_RC_SEND_WQE_BYTE_4_FLAG_S 31
 

drivers/infiniband/hw/hns/hns_roce_hw_v2.h struct hns_roce_v2_rc_send_wqe { · line 1284

Quoted 6 lines
drivers/infiniband/hw/hns/hns_roce_hw_v2.h +1 -1
@@ -1280,7 +1280,7 @@ struct hns_roce_v2_rc_send_wqe {
 
 struct hns_roce_wqe_frmr_seg {
 	__le32	pbl_size;
-	__le32	mode_buf_pg_sz;
+	__le32	byte_40;
 };

This stuff is HW API isn't it?

I didn't see anything to negotiate compatability with existing HW? What happens if the kernel is updated and run on old HW/FW?

If you tightly couple you still need to check and refuse to load the driver.

Jason

author
#7 lore raw

Re: [PATCH for-next 05/12] RDMA/hns: Adjust definition of FRMR fields

Quoted 35 lines · Jason Gunthorpe
Quoted 25 lines · Weihang Li
drivers/infiniband/hw/hns/hns_roce_hw_v2.h +6 -5
index f29438c..1da980c 100644
+++ b/drivers/infiniband/hw/hns/hns_roce_hw_v2.h
@@ -1255,15 +1255,15 @@ struct hns_roce_v2_rc_send_wqe {
 
 #define V2_RC_SEND_WQE_BYTE_4_INLINE_S 12
 
-#define V2_RC_FRMR_WQE_BYTE_4_BIND_EN_S 19
+#define V2_RC_FRMR_WQE_BYTE_40_BIND_EN_S 10
 
-#define V2_RC_FRMR_WQE_BYTE_4_ATOMIC_S 20
+#define V2_RC_FRMR_WQE_BYTE_40_ATOMIC_S 11
 
-#define V2_RC_FRMR_WQE_BYTE_4_RR_S 21
+#define V2_RC_FRMR_WQE_BYTE_40_RR_S 12
 
-#define V2_RC_FRMR_WQE_BYTE_4_RW_S 22
+#define V2_RC_FRMR_WQE_BYTE_40_RW_S 13
 
-#define V2_RC_FRMR_WQE_BYTE_4_LW_S 23
+#define V2_RC_FRMR_WQE_BYTE_40_LW_S 14
 
 #define V2_RC_SEND_WQE_BYTE_4_FLAG_S 31
 

drivers/infiniband/hw/hns/hns_roce_hw_v2.h struct hns_roce_v2_rc_send_wqe { · line 1284

Quoted 6 lines
drivers/infiniband/hw/hns/hns_roce_hw_v2.h +1 -1
@@ -1280,7 +1280,7 @@ struct hns_roce_v2_rc_send_wqe {
 
 struct hns_roce_wqe_frmr_seg {
 	__le32	pbl_size;
-	__le32	mode_buf_pg_sz;
+	__le32	byte_40;
 };

This stuff is HW API isn't it?

Quoted 8 lines

I didn't see anything to negotiate compatability with existing HW? What happens if the kernel is updated and run on old HW/FW?

If you tightly couple you still need to check and refuse to load the driver.

Jason

Thank you, FRMR is not well-supported on HIP08, so we re-design it on HIP09. I will add a check to avoid ULPs using FRMR on HIP08.

Weihang

author
#8 lore raw

PATCH 6/12 RDMA/hns: Skip qp_flow_control_init() for HIP09

In reply to Weihang Li

From: Yixing Liu <[email protected]>

Since HIP09 does not require this function, it should be masked.

---
 drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/infiniband/hw/hns/hns_roce_hw_v2.c +3 -0
index 18a8ac9..6c9dbe2 100644
@@ -5233,6 +5233,9 @@ static int hns_roce_v2_qp_flow_control_init(struct hns_roce_dev *hr_dev,
 	struct hns_roce_cmq_desc desc;
 	int ret, i;
 
+	if (hr_dev->pci_dev->revision >= PCI_REVISION_ID_HIP09)
+		return 0;
+
 	mutex_lock(&hr_dev->qp_table.scc_mutex);
 
 	/* set scc ctx clear done flag */
Signature · 1 line
2.8.1
author
#9 lore raw

PATCH 7/12 RDMA/hns: Replace wmb&__raw_writeq with writeq

In reply to Weihang Li

From: Lang Cheng <[email protected]>

Currently, the driver updates doorbell looks like this:
post()

{
	wqe.field = 0x111;
	wmb();
	update_wq_db();
}

update_wq_db()

{
	db.field = 0x222;
	__raw_writeq(db, db_reg);
}

writeq() is a better choice than __raw_writeq() because it calls dma_wmb() to barrier in ARM64, and dma_wmb() is better than wmb() for ROCEE device.

This patch removes all wmb() before updating doorbell of SQ/RQ/CQ/SRQ by replacing __raw_writeq() with writeq() to improve performence. The new process looks like this:
post()

{
	wqe.field = 0x111;
	update_wq_db();
}

update_wq_db()

{
	db.field = 0x222;
	writeq(db, db_reg);
}
---
 drivers/infiniband/hw/hns/hns_roce_device.h |  2 +-
 drivers/infiniband/hw/hns/hns_roce_hw_v1.c  | 15 ---------------
 drivers/infiniband/hw/hns/hns_roce_hw_v2.c  | 20 +-------------------
 3 files changed, 2 insertions(+), 35 deletions(-)
drivers/infiniband/hw/hns/hns_roce_device.h +1 -1
index 1f94154..74eb08f 100644
@@ -1077,7 +1077,7 @@ static inline struct hns_roce_srq *to_hr_srq(struct ib_srq *ibsrq)
 
 static inline void hns_roce_write64_k(__le32 val[2], void __iomem *dest)
 {
-	__raw_writeq(*(u64 *) val, dest);
+	writeq(*(u64 *)val, dest);
 }
 
 static inline struct hns_roce_qp
drivers/infiniband/hw/hns/hns_roce_hw_v1.c +0 -15
index 262ad58..5346fdc 100644
@@ -330,8 +330,6 @@ static int hns_roce_v1_post_send(struct ib_qp *ibqp,
 	/* Set DB return */
 	if (likely(nreq)) {
 		qp->sq.head += nreq;
-		/* Memory barrier */
-		wmb();
 
 		roce_set_field(sq_db.u32_4, SQ_DOORBELL_U32_4_SQ_HEAD_M,
 			       SQ_DOORBELL_U32_4_SQ_HEAD_S,
@@ -411,8 +409,6 @@ static int hns_roce_v1_post_recv(struct ib_qp *ibqp,
 out:
 	if (likely(nreq)) {
 		hr_qp->rq.head += nreq;
-		/* Memory barrier */
-		wmb();
 
 		if (ibqp->qp_type == IB_QPT_GSI) {
 			__le32 tmp;
@@ -1984,12 +1980,6 @@ static void __hns_roce_v1_cq_clean(struct hns_roce_cq *hr_cq, u32 qpn,
 
 	if (nfreed) {
 		hr_cq->cons_index += nfreed;
-		/*
-		 * Make sure update of buffer contents is done before
-		 * updating consumer index.
-		 */
-		wmb();
-
 		hns_roce_v1_cq_set_ci(hr_cq, hr_cq->cons_index);
 	}
 }
@@ -2330,8 +2320,6 @@ int hns_roce_v1_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc)
 		*hr_cq->tptr_addr = hr_cq->cons_index &
 			((hr_cq->cq_depth << 1) - 1);
 
-		/* Memroy barrier */
-		wmb();
 		hns_roce_v1_cq_set_ci(hr_cq, hr_cq->cons_index);
 	}
 
@@ -3220,9 +3208,6 @@ static int hns_roce_v1_m_qp(struct ib_qp *ibqp, const struct ib_qp_attr *attr,
 	 * need to hw to flash RQ HEAD by DB again
 	 */
 	if (cur_state == IB_QPS_INIT && new_state == IB_QPS_INIT) {
-		/* Memory barrier */
-		wmb();
-
 		roce_set_field(doorbell[0], RQ_DOORBELL_U32_4_RQ_HEAD_M,
 			       RQ_DOORBELL_U32_4_RQ_HEAD_S, hr_qp->rq.head);
 		roce_set_field(doorbell[1], RQ_DOORBELL_U32_8_QPN_M,
drivers/infiniband/hw/hns/hns_roce_hw_v2.c +1 -19
index 6c9dbe2..a5e304a 100644
@@ -742,8 +742,6 @@ static int hns_roce_v2_post_send(struct ib_qp *ibqp,
 	if (likely(nreq)) {
 		qp->sq.head += nreq;
 		qp->next_sge = sge_idx;
-		/* Memory barrier */
-		wmb();
 
 		if (nreq == 1 && qp->sq.head == qp->sq.tail + 1 &&
 		    (qp->en_flags & HNS_ROCE_QP_CAP_DIRECT_WQE))
@@ -873,8 +871,6 @@ static int hns_roce_v2_post_recv(struct ib_qp *ibqp,
 out:
 	if (likely(nreq)) {
 		hr_qp->rq.head += nreq;
-		/* Memory barrier */
-		wmb();
 
 		/*
 		 * Hip08 hardware cannot flush the WQEs in RQ if the QP state
@@ -1013,12 +1009,6 @@ static int hns_roce_v2_post_srq_recv(struct ib_srq *ibsrq,
 	}
 
 	if (likely(nreq)) {
-		/*
-		 * Make sure that descriptors are written before
-		 * doorbell record.
-		 */
-		wmb();
-
 		srq_db.byte_4 =
 			cpu_to_le32(HNS_ROCE_V2_SRQ_DB << V2_DB_BYTE_4_CMD_S |
 				    (srq->srqn & V2_DB_BYTE_4_TAG_M));
@@ -3196,11 +3186,6 @@ static void __hns_roce_v2_cq_clean(struct hns_roce_cq *hr_cq, u32 qpn,
 
 	if (nfreed) {
 		hr_cq->cons_index += nfreed;
-		/*
-		 * Make sure update of buffer contents is done before
-		 * updating consumer index.
-		 */
-		wmb();
 		hns_roce_v2_cq_set_ci(hr_cq, hr_cq->cons_index);
 	}
 }
@@ -3709,11 +3694,8 @@ static int hns_roce_v2_poll_cq(struct ib_cq *ibcq, int num_entries,
 			break;
 	}
 
-	if (npolled) {
-		/* Memory barrier */
-		wmb();
+	if (npolled)
 		hns_roce_v2_cq_set_ci(hr_cq, hr_cq->cons_index);
-	}
 
 out:
 	spin_unlock_irqrestore(&hr_cq->lock, flags);
Signature · 1 line
2.8.1
author
#10 lore raw

PATCH 8/12 RDMA/hns: Move HIP06 related definitions into hns_roce_hw_v1.h

In reply to Weihang Li

From: Lang Cheng <[email protected]>

hns_roce_device.h is not specific to hardware, some definitions are only used for HIP06, they should be moved into hns_roce_hw_v1.h.

---
 drivers/infiniband/hw/hns/hns_roce_device.h | 41 ---------------------------
 drivers/infiniband/hw/hns/hns_roce_hw_v1.h  | 43 +++++++++++++++++++++++++++++
 2 files changed, 43 insertions(+), 41 deletions(-)
drivers/infiniband/hw/hns/hns_roce_device.h +0 -41
index 74eb08f..315c013 100644
@@ -170,44 +170,6 @@ enum hns_roce_event {
 	HNS_ROCE_EVENT_TYPE_FLR			      = 0x15,
 };
 
-/* Local Work Queue Catastrophic Error,SUBTYPE 0x5 */
-enum {
-	HNS_ROCE_LWQCE_QPC_ERROR		= 1,
-	HNS_ROCE_LWQCE_MTU_ERROR		= 2,
-	HNS_ROCE_LWQCE_WQE_BA_ADDR_ERROR	= 3,
-	HNS_ROCE_LWQCE_WQE_ADDR_ERROR		= 4,
-	HNS_ROCE_LWQCE_SQ_WQE_SHIFT_ERROR	= 5,
-	HNS_ROCE_LWQCE_SL_ERROR			= 6,
-	HNS_ROCE_LWQCE_PORT_ERROR		= 7,
-};
-
-/* Local Access Violation Work Queue Error,SUBTYPE 0x7 */
-enum {
-	HNS_ROCE_LAVWQE_R_KEY_VIOLATION		= 1,
-	HNS_ROCE_LAVWQE_LENGTH_ERROR		= 2,
-	HNS_ROCE_LAVWQE_VA_ERROR		= 3,
-	HNS_ROCE_LAVWQE_PD_ERROR		= 4,
-	HNS_ROCE_LAVWQE_RW_ACC_ERROR		= 5,
-	HNS_ROCE_LAVWQE_KEY_STATE_ERROR		= 6,
-	HNS_ROCE_LAVWQE_MR_OPERATION_ERROR	= 7,
-};
-
-/* DOORBELL overflow subtype */
-enum {
-	HNS_ROCE_DB_SUBTYPE_SDB_OVF		= 1,
-	HNS_ROCE_DB_SUBTYPE_SDB_ALM_OVF		= 2,
-	HNS_ROCE_DB_SUBTYPE_ODB_OVF		= 3,
-	HNS_ROCE_DB_SUBTYPE_ODB_ALM_OVF		= 4,
-	HNS_ROCE_DB_SUBTYPE_SDB_ALM_EMP		= 5,
-	HNS_ROCE_DB_SUBTYPE_ODB_ALM_EMP		= 6,
-};
-
-enum {
-	/* RQ&SRQ related operations */
-	HNS_ROCE_OPCODE_SEND_DATA_RECEIVE	= 0x06,
-	HNS_ROCE_OPCODE_RDMA_WITH_IMM_RECEIVE	= 0x07,
-};
-
 #define HNS_ROCE_CAP_FLAGS_EX_SHIFT 12
 
 enum {
@@ -260,9 +222,6 @@ enum {
 
 #define HNS_ROCE_CMD_SUCCESS			1
 
-#define HNS_ROCE_PORT_DOWN			0
-#define HNS_ROCE_PORT_UP			1
-
 /* The minimum page size is 4K for hardware */
 #define HNS_HW_PAGE_SHIFT			12
 #define HNS_HW_PAGE_SIZE			(1 << HNS_HW_PAGE_SHIFT)
drivers/infiniband/hw/hns/hns_roce_hw_v1.h +43 -0

49 lines not on this page · raw message

Signature · 1 line
2.8.1
author
#11 lore raw

PATCH 9/12 RDMA/hns: Remove some magic numbers

In reply to Weihang Li

From: Xinhao Liu <[email protected]>

Use macros instead of magic numbers to represent shift of dma_handle_wqe, dma_handle_idx and UDP destination port number of RoCEv2.

---
 drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)
drivers/infiniband/hw/hns/hns_roce_hw_v2.c +11 -5
index a5e304a..1bff432 100644
@@ -1601,7 +1601,8 @@ static int hns_roce_config_global_param(struct hns_roce_dev *hr_dev)
 		       CFG_GLOBAL_PARAM_DATA_0_ROCEE_TIME_1US_CFG_S, 0x3e8);
 	roce_set_field(req->time_cfg_udp_port,
 		       CFG_GLOBAL_PARAM_DATA_0_ROCEE_UDP_PORT_M,
-		       CFG_GLOBAL_PARAM_DATA_0_ROCEE_UDP_PORT_S, 0x12b7);
+		       CFG_GLOBAL_PARAM_DATA_0_ROCEE_UDP_PORT_S,
+		       ROCE_V2_UDP_DPORT);
 
 	return hns_roce_cmq_send(hr_dev, &desc, 1);
 }
@@ -5264,6 +5265,9 @@ static int hns_roce_v2_qp_flow_control_init(struct hns_roce_dev *hr_dev,
 	return ret;
 }
 
+#define DMA_IDX_SHIFT 3
+#define DMA_WQE_SHIFT 3
+
 static int hns_roce_v2_write_srqc_index_queue(struct hns_roce_srq *srq,
 					      struct hns_roce_srq_context *ctx)
 {
@@ -5286,8 +5290,9 @@ static int hns_roce_v2_write_srqc_index_queue(struct hns_roce_srq *srq,
 	hr_reg_write(ctx, SRQC_IDX_HOP_NUM,
 		     to_hr_hem_hopnum(hr_dev->caps.idx_hop_num, srq->wqe_cnt));
 
-	hr_reg_write(ctx, SRQC_IDX_BT_BA_L, dma_handle_idx >> 3);
-	hr_reg_write(ctx, SRQC_IDX_BT_BA_H, upper_32_bits(dma_handle_idx >> 3));
+	hr_reg_write(ctx, SRQC_IDX_BT_BA_L, dma_handle_idx >> DMA_IDX_SHIFT);
+	hr_reg_write(ctx, SRQC_IDX_BT_BA_H,
+		     upper_32_bits(dma_handle_idx >> DMA_IDX_SHIFT));
 
 	hr_reg_write(ctx, SRQC_IDX_BA_PG_SZ,
 		     to_hr_hw_page_shift(idx_que->mtr.hem_cfg.ba_pg_shift));
@@ -5340,8 +5345,9 @@ static int hns_roce_v2_write_srqc(struct hns_roce_srq *srq, void *mb_buf)
 		     to_hr_hem_hopnum(hr_dev->caps.srqwqe_hop_num,
 				      srq->wqe_cnt));
 
-	hr_reg_write(ctx, SRQC_WQE_BT_BA_L, dma_handle_wqe >> 3);
-	hr_reg_write(ctx, SRQC_WQE_BT_BA_H, upper_32_bits(dma_handle_wqe >> 3));
+	hr_reg_write(ctx, SRQC_WQE_BT_BA_L, dma_handle_wqe >> DMA_WQE_SHIFT);
+	hr_reg_write(ctx, SRQC_WQE_BT_BA_H,
+		     upper_32_bits(dma_handle_wqe >> DMA_WQE_SHIFT));
 
 	hr_reg_write(ctx, SRQC_WQE_BA_PG_SZ,
 		     to_hr_hw_page_shift(srq->buf_mtr.hem_cfg.ba_pg_shift));
Signature · 1 line
2.8.1
author
#12 lore raw

PATCH 10/12 RDMA/hns: Avoid unnecessary memset on WQEs in post_send

In reply to Weihang Li

From: Lang Cheng <[email protected]>

All fields of WQE will be rewrote, so the memset is unnecessary. And when SQ is working in OWNER mode, the pipeline may prefetch the WQEs beyond PI, the memset operation may flip the owner bit too early, then the pipeline may get a wrong WQ.

---
 drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 2 --
 1 file changed, 2 deletions(-)
drivers/infiniband/hw/hns/hns_roce_hw_v2.c +0 -2
index 1bff432..55dfd44 100644
@@ -467,7 +467,6 @@ static inline int set_ud_wqe(struct hns_roce_qp *qp,
 	int ret;
 
 	valid_num_sge = calc_wr_sge_num(wr, &msg_len);
-	memset(ud_sq_wqe, 0, sizeof(*ud_sq_wqe));
 
 	ret = set_ud_opcode(ud_sq_wqe, wr);
 	if (WARN_ON(ret))
@@ -573,7 +572,6 @@ static inline int set_rc_wqe(struct hns_roce_qp *qp,
 	int ret;
 
 	valid_num_sge = calc_wr_sge_num(wr, &msg_len);
-	memset(rc_sq_wqe, 0, sizeof(*rc_sq_wqe));
 
 	rc_sq_wqe->msg_len = cpu_to_le32(msg_len);
 
Signature · 1 line
2.8.1
author
#13 lore raw

PATCH 11/12 RDMA/hns: Remove unnecessary wrap around for EQ's consumer index

In reply to Weihang Li

From: Yixian Liu <[email protected]>

The hns driver wrap around the consumer index of AEQ and CEQ when they reach to two times of queue entries number for owner mechanism, actually, it is unnecessary to wrap around since the hardware itself will mask it before use.

---
 drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 6 ------
 1 file changed, 6 deletions(-)
drivers/infiniband/hw/hns/hns_roce_hw_v2.c +0 -6
index 55dfd44..9c6fe32 100644
@@ -5667,9 +5667,6 @@ static int hns_roce_v2_aeq_int(struct hns_roce_dev *hr_dev,
 		++eq->cons_index;
 		aeqe_found = 1;
 
-		if (eq->cons_index > (2 * eq->entries - 1))
-			eq->cons_index = 0;
-
 		hns_roce_v2_init_irq_work(hr_dev, eq, queue_num);
 
 		aeqe = next_aeqe_sw_v2(eq);
@@ -5712,9 +5709,6 @@ static int hns_roce_v2_ceq_int(struct hns_roce_dev *hr_dev,
 		++eq->cons_index;
 		ceqe_found = 1;
 
-		if (eq->cons_index > (EQ_DEPTH_COEFF * eq->entries - 1))
-			eq->cons_index = 0;
-
 		ceqe = next_ceqe_sw_v2(eq);
 	}
 
Signature · 1 line
2.8.1
author
#14 lore raw

PATCH 12/12 RDMA/hns: Delete redundant judgment when preparing descriptors

In reply to Weihang Li

From: Xinhao Liu <[email protected]>

There is no need to use a for loop to assign values for an array of cmd descriptors which has only two elements.

---
 drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 40 +++++++++++-------------------
 1 file changed, 14 insertions(+), 26 deletions(-)
drivers/infiniband/hw/hns/hns_roce_hw_v2.c +14 -26
index 9c6fe32..1071b3b 100644
@@ -1611,17 +1611,13 @@ static int hns_roce_query_pf_resource(struct hns_roce_dev *hr_dev)
 	struct hns_roce_pf_res_a *req_a;
 	struct hns_roce_pf_res_b *req_b;
 	int ret;
-	int i;
 
-	for (i = 0; i < 2; i++) {
-		hns_roce_cmq_setup_basic_desc(&desc[i],
-					      HNS_ROCE_OPC_QUERY_PF_RES, true);
+	hns_roce_cmq_setup_basic_desc(&desc[0], HNS_ROCE_OPC_QUERY_PF_RES,
+				      true);
+	desc[0].flag |= cpu_to_le16(HNS_ROCE_CMD_FLAG_NEXT);
 
-		if (i == 0)
-			desc[i].flag |= cpu_to_le16(HNS_ROCE_CMD_FLAG_NEXT);
-		else
-			desc[i].flag &= ~cpu_to_le16(HNS_ROCE_CMD_FLAG_NEXT);
-	}
+	hns_roce_cmq_setup_basic_desc(&desc[1], HNS_ROCE_OPC_QUERY_PF_RES,
+				      true);
 
 	ret = hns_roce_cmq_send(hr_dev, desc, 2);
 	if (ret)
@@ -1714,19 +1710,16 @@ static int hns_roce_alloc_vf_resource(struct hns_roce_dev *hr_dev)
 	struct hns_roce_cmq_desc desc[2];
 	struct hns_roce_vf_res_a *req_a;
 	struct hns_roce_vf_res_b *req_b;
-	int i;
 
 	req_a = (struct hns_roce_vf_res_a *)desc[0].data;
 	req_b = (struct hns_roce_vf_res_b *)desc[1].data;
-	for (i = 0; i < 2; i++) {
-		hns_roce_cmq_setup_basic_desc(&desc[i],
-					      HNS_ROCE_OPC_ALLOC_VF_RES, false);
 
-		if (i == 0)
-			desc[i].flag |= cpu_to_le16(HNS_ROCE_CMD_FLAG_NEXT);
-		else
-			desc[i].flag &= ~cpu_to_le16(HNS_ROCE_CMD_FLAG_NEXT);
-	}
+	hns_roce_cmq_setup_basic_desc(&desc[0], HNS_ROCE_OPC_ALLOC_VF_RES,
+				      false);
+	desc[0].flag |= cpu_to_le16(HNS_ROCE_CMD_FLAG_NEXT);
+
+	hns_roce_cmq_setup_basic_desc(&desc[1], HNS_ROCE_OPC_ALLOC_VF_RES,
+				      false);
 
 	roce_set_field(req_a->vf_qpc_bt_idx_num,
 		       VF_RES_A_DATA_1_VF_QPC_BT_IDX_M,
@@ -2407,7 +2400,6 @@ static int hns_roce_config_link_table(struct hns_roce_dev *hr_dev,
 	struct hns_roce_link_table_entry *entry;
 	enum hns_roce_opcode_type opcode;
 	u32 page_num;
-	int i;
 
 	switch (type) {
 	case TSQ_LINK_TABLE:
@@ -2425,14 +2417,10 @@ static int hns_roce_config_link_table(struct hns_roce_dev *hr_dev,
 	page_num = link_tbl->npages;
 	entry = link_tbl->table.buf;
 
-	for (i = 0; i < 2; i++) {
-		hns_roce_cmq_setup_basic_desc(&desc[i], opcode, false);
+	hns_roce_cmq_setup_basic_desc(&desc[0], opcode, false);
+	desc[0].flag |= cpu_to_le16(HNS_ROCE_CMD_FLAG_NEXT);
 
-		if (i == 0)
-			desc[i].flag |= cpu_to_le16(HNS_ROCE_CMD_FLAG_NEXT);
-		else
-			desc[i].flag &= ~cpu_to_le16(HNS_ROCE_CMD_FLAG_NEXT);
-	}
+	hns_roce_cmq_setup_basic_desc(&desc[1], opcode, false);
 
 	req_a->base_addr_l = cpu_to_le32(link_tbl->table.map & 0xffffffff);
 	req_a->base_addr_h = cpu_to_le32(link_tbl->table.map >> 32);
Signature · 1 line
2.8.1
maintainer
#15 lore raw
In reply to Weihang Li
Quoted 10 lines · Weihang Li

As usual, this series collects some miscellaneous fixes and cleanups at the end of 5.12 for the hns driver:
* #1 ~ #3 fix some non-urgent issues.
* #4 ~ #6 make some changes on existing features.
* #7 ~ #12 are cleanups, just do some refactor and delete dead code.

This series is made based on the series named "RDMA/hns: Several fixes and cleanups of RQ/SRQ" which is still being reviewed:
https://patchwork.kernel.org/project/linux-rdma/cover/[email protected]/

Quoted 7 lines

I'm worried that it will be too late for 5.12 if any patch of this series needs changes, so I send it before the previous one is merged. If there is any comment or merge conflict on it, I will fix them as soon as possible, thanks.

Lang Cheng (3):
RDMA/hns: Replace wmb&__raw_writeq with writeq

This one is quite a good idea

Quoted 17 lines

RDMA/hns: Move HIP06 related definitions into hns_roce_hw_v1.h RDMA/hns: Avoid unnecessary memset on WQEs in post_send

Lijun Ou (1):
RDMA/hns: Disable RQ inline by default

Weihang Li (2):
RDMA/hns: Avoid filling sgid index when modifying QP to RTR RDMA/hns: Fix type of sq_signal_bits

Xi Wang (1):
RDMA/hns: Add mapped page count checking for MTR

Xinhao Liu (2):
RDMA/hns: Remove some magic numbers
RDMA/hns: Delete redundant judgment when preparing descriptors

Quoted 8 lines

Yixian Liu (1):
RDMA/hns: Remove unnecessary wrap around for EQ's consumer index

Yixing Liu (2):
RDMA/hns: Skip qp_flow_control_init() for HIP09

RDMA/hns: Adjust definition of FRMR fields

I didn't apply this one because of my question, but did take the rest to for-next, thanks

Jason