RDMA/hns: Updates for 5.12
Subsystems (2)
cover RDMA/hns: Updates for 5.12
As usual, this series collects some miscellaneous fixes and cleanups at the end of 5.12 for the hns driver:
* #1 ~ #3 fix some non-urgent issues.
* #4 ~ #6 make some changes on existing features.
* #7 ~ #12 are cleanups, just do some refactor and delete dead code.
This series is made based on the series named "RDMA/hns: Several fixes and cleanups of RQ/SRQ" which is still being reviewed:
https://patchwork.kernel.org/project/linux-rdma/cover/[email protected]/
I'm worried that it will be too late for 5.12 if any patch of this series needs changes, so I send it before the previous one is merged. If there is any comment or merge conflict on it, I will fix them as soon as possible, thanks.
Lang Cheng (3):
RDMA/hns: Replace wmb&__raw_writeq with writeq RDMA/hns: Move HIP06 related definitions into hns_roce_hw_v1.h RDMA/hns: Avoid unnecessary memset on WQEs in post_send
Lijun Ou (1):
RDMA/hns: Disable RQ inline by default
Weihang Li (2):
RDMA/hns: Avoid filling sgid index when modifying QP to RTR RDMA/hns: Fix type of sq_signal_bits
Xi Wang (1):
RDMA/hns: Add mapped page count checking for MTR
Xinhao Liu (2):
RDMA/hns: Remove some magic numbers
RDMA/hns: Delete redundant judgment when preparing descriptors
Yixian Liu (1):
RDMA/hns: Remove unnecessary wrap around for EQ's consumer index
Yixing Liu (2):
RDMA/hns: Adjust definition of FRMR fields
RDMA/hns: Skip qp_flow_control_init() for HIP09
drivers/infiniband/hw/hns/hns_roce_device.h | 45 +--------- drivers/infiniband/hw/hns/hns_roce_hem.c | 9 +- drivers/infiniband/hw/hns/hns_roce_hw_v1.c | 31 +++---- drivers/infiniband/hw/hns/hns_roce_hw_v1.h | 43 ++++++++++ drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 125 +++++++++------------------- drivers/infiniband/hw/hns/hns_roce_hw_v2.h | 12 +-- drivers/infiniband/hw/hns/hns_roce_main.c | 16 ---- drivers/infiniband/hw/hns/hns_roce_mr.c | 56 ++++++++----- 8 files changed, 148 insertions(+), 189 deletions(-)
Signature · 1 line
2.8.1
PATCH 1/12 RDMA/hns: Avoid filling sgid index when modifying QP to RTR
ULP usually set IB(V)_QP_AV when trying to modify QP to RTR if they want to record sgid index into QPC. For UD QPs, it is useless because it will be included in WQE. For RC QPs, it will be filled in hns_roce_set_path(). So sgid index shouldn't be filled by default. Then hns_get_gid_index() is moved to hns_roce_hw_v1.c because it is only called in it.
--- drivers/infiniband/hw/hns/hns_roce_hw_v1.c | 16 ++++++++++++++++ drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 11 ----------- drivers/infiniband/hw/hns/hns_roce_main.c | 16 ---------------- 3 files changed, 16 insertions(+), 27 deletions(-)
drivers/infiniband/hw/hns/hns_roce_hw_v1.c +16 -0
@@ -43,6 +43,22 @@#include "hns_roce_hem.h" #include "hns_roce_hw_v1.h" +/** +* hns_get_gid_index - Get gid index. +* @hr_dev: pointer to structure hns_roce_dev. +* @port: port, value range: 0 ~ MAX +* @gid_index: gid_index, value range: 0 ~ MAX +* Description: +* N ports shared gids, allocation method as follow: +* GID[0][0], GID[1][0],.....GID[N - 1][0], +* GID[0][0], GID[1][0],.....GID[N - 1][0], +* And so on +*/ +u8 hns_get_gid_index (struct hns_roce_dev *hr_dev,u8 port,int gid_index) +{ +return gid_index * hr_dev->caps.num_ports + port; +} +static void set_data_seg (struct hns_roce_wqe_data_seg *dseg,struct ib_sge *sg) { dseg->lkey =cpu_to_le32 (sg->lkey);
drivers/infiniband/hw/hns/hns_roce_hw_v2.c +0 -11
@@ -4308,7 +4308,6 @@ static int modify_qp_init_to_rtr(struct ib_qp *ibqp,struct hns_roce_v2_qp_context *context, struct hns_roce_v2_qp_context *qpc_mask) { - const struct ib_global_route *grh = rdma_ah_read_grh(&attr->ah_attr); struct hns_roce_dev *hr_dev = to_hr_dev(ibqp->device);struct hns_roce_qp *hr_qp =to_hr_qp (ibqp);struct ib_device *ibdev = &hr_dev->ib_dev;
@@ -4316,7 +4315,6 @@ static int modify_qp_init_to_rtr(struct ib_qp *ibqp,dma_addr_t irrl_ba;enum ib_mtu mtu;u8 lp_pktn_ini; -u8 port_num;u64 *mtts;u8 *dmac;u8 *smac;
@@ -4397,15 +4395,6 @@ static int modify_qp_init_to_rtr(struct ib_qp *ibqp, V2_QPC_BYTE_56_DQPN_M, V2_QPC_BYTE_56_DQPN_S, 0); } -/* Configure GID index */ - port_num =rdma_ah_get_port_num (&attr->ah_attr); -roce_set_field (context->byte_20_smac_sgid_idx, -V2_QPC_BYTE_20_SGID_IDX_M ,V2_QPC_BYTE_20_SGID_IDX_S , -hns_get_gid_index (hr_dev, port_num -1 , - grh->sgid_index)); -roce_set_field (qpc_mask->byte_20_smac_sgid_idx, -V2_QPC_BYTE_20_SGID_IDX_M ,V2_QPC_BYTE_20_SGID_IDX_S ,0 ); -memcpy (&(context->dmac), dmac,sizeof (u32));roce_set_field (context->byte_52_udpspn_dmac,V2_QPC_BYTE_52_DMAC_M ,V2_QPC_BYTE_52_DMAC_S , *((u16 *)(&dmac[4 ])));
drivers/infiniband/hw/hns/hns_roce_main.c +0 -16
@@ -42,22 +42,6 @@#include "hns_roce_device.h" #include "hns_roce_hem.h" -/** -* hns_get_gid_index - Get gid index. -* @hr_dev: pointer to structure hns_roce_dev. -* @port: port, value range: 0 ~ MAX -* @gid_index: gid_index, value range: 0 ~ MAX -* Description: -* N ports shared gids, allocation method as follow: -* GID[0][0], GID[1][0],.....GID[N - 1][0], -* GID[0][0], GID[1][0],.....GID[N - 1][0], -* And so on -*/ -u8 hns_get_gid_index (struct hns_roce_dev *hr_dev,u8 port,int gid_index) -{ -return gid_index * hr_dev->caps.num_ports + port; -} -static int hns_roce_set_mac (struct hns_roce_dev *hr_dev,u8 port,u8 *addr) {u8 phy_port;
Signature · 1 line
2.8.1
PATCH 2/12 RDMA/hns: Fix type of sq_signal_bits
In reply to Weihang LiThis bit should be in type of enum ib_sig_type, or there will be a sparse warning.
--- drivers/infiniband/hw/hns/hns_roce_device.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/hw/hns/hns_roce_device.h +1 -1
@@ -661,7 +661,7 @@ struct hns_roce_qp {struct hns_roce_db sdb;unsigned long en_flags;u32 doorbell_qpn; -u32 sq_signal_bits; +enum ib_sig_type sq_signal_bits;struct hns_roce_wq sq;struct hns_roce_mtr mtr;
Signature · 1 line
2.8.1
PATCH 3/12 RDMA/hns: Add mapped page count checking for MTR
In reply to Weihang LiFrom: Xi Wang <[email protected]>
Add the mapped page count checking flow to avoid invalid page size when creating MTR.
--- drivers/infiniband/hw/hns/hns_roce_hem.c | 9 +++-- drivers/infiniband/hw/hns/hns_roce_mr.c | 56 +++++++++++++++++++------------- 2 files changed, 40 insertions(+), 25 deletions(-)
drivers/infiniband/hw/hns/hns_roce_hem.c +6 -3
@@ -1075,9 +1075,8 @@ static struct roce_hem_item *hem_list_alloc_item(struct hns_roce_dev *hr_dev,return NULL ;if (exist_bt) { - hem->addr =dma_alloc_coherent (hr_dev->dev, - count *BA_BYTE_LEN , - &hem->dma_addr,GFP_KERNEL ); + hem->addr =dma_alloc_coherent (hr_dev->dev, count *BA_BYTE_LEN , + &hem->dma_addr,GFP_KERNEL );if (!hem->addr) {kfree (hem);return NULL ;
@@ -1336,6 +1335,10 @@ static int hem_list_alloc_root_bt(struct hns_roce_dev *hr_dev,if (ba_num <1 )return -ENOMEM ; +if (ba_num > unit) +return -ENOBUFS ; + + ba_num =min_t (int, ba_num, unit);INIT_LIST_HEAD (&temp_root); offset = r->offset;/* indicate to last region */
drivers/infiniband/hw/hns/hns_roce_mr.c +34 -22
105 lines not on this page · raw message
Signature · 1 line
2.8.1
PATCH 4/12 RDMA/hns: Disable RQ inline by default
In reply to Weihang LiFrom: Lijun Ou <[email protected]>
This feature should only be enabled by querying capability from firmware.
--- drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 1 - 1 file changed, 1 deletion(-)
drivers/infiniband/hw/hns/hns_roce_hw_v2.c +0 -1
@@ -1949,7 +1949,6 @@ static void set_default_caps(struct hns_roce_dev *hr_dev) caps->flags =HNS_ROCE_CAP_FLAG_REREG_MR |HNS_ROCE_CAP_FLAG_ROCE_V1_V2 | -HNS_ROCE_CAP_FLAG_RQ_INLINE |HNS_ROCE_CAP_FLAG_RECORD_DB |HNS_ROCE_CAP_FLAG_SQ_RECORD_DB ;
Signature · 1 line
2.8.1
PATCH 5/12 RDMA/hns: Adjust definition of FRMR fields
In reply to Weihang LiFrom: Yixing Liu <[email protected]>
The position of the FRMR-related fields has changed, change them to fit the new design.
--- drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 26 ++++++++++++-------------- drivers/infiniband/hw/hns/hns_roce_hw_v2.h | 12 ++++++------ 2 files changed, 18 insertions(+), 20 deletions(-)
drivers/infiniband/hw/hns/hns_roce_hw_v2.c +12 -14
@@ -99,16 +99,16 @@ static void set_frmr_seg(struct hns_roce_v2_rc_send_wqe *rc_sq_wqe,u64 pbl_ba;/* use ib_access_flags */ -roce_set_bit (rc_sq_wqe->byte_4,V2_RC_FRMR_WQE_BYTE_4_BIND_EN_S , - wr->access &IB_ACCESS_MW_BIND ?1 :0 ); -roce_set_bit (rc_sq_wqe->byte_4,V2_RC_FRMR_WQE_BYTE_4_ATOMIC_S , - wr->access &IB_ACCESS_REMOTE_ATOMIC ?1 :0 ); -roce_set_bit (rc_sq_wqe->byte_4,V2_RC_FRMR_WQE_BYTE_4_RR_S , - wr->access &IB_ACCESS_REMOTE_READ ?1 :0 ); -roce_set_bit (rc_sq_wqe->byte_4,V2_RC_FRMR_WQE_BYTE_4_RW_S , - wr->access &IB_ACCESS_REMOTE_WRITE ?1 :0 ); -roce_set_bit (rc_sq_wqe->byte_4,V2_RC_FRMR_WQE_BYTE_4_LW_S , - wr->access &IB_ACCESS_LOCAL_WRITE ?1 :0 ); +roce_set_bit (fseg->byte_40,V2_RC_FRMR_WQE_BYTE_40_BIND_EN_S , + !!(wr->access &IB_ACCESS_MW_BIND )); +roce_set_bit (fseg->byte_40,V2_RC_FRMR_WQE_BYTE_40_ATOMIC_S , + !!(wr->access &IB_ACCESS_REMOTE_ATOMIC )); +roce_set_bit (fseg->byte_40,V2_RC_FRMR_WQE_BYTE_40_RR_S , + !!(wr->access &IB_ACCESS_REMOTE_READ )); +roce_set_bit (fseg->byte_40,V2_RC_FRMR_WQE_BYTE_40_RW_S , + !!(wr->access &IB_ACCESS_REMOTE_WRITE )); +roce_set_bit (fseg->byte_40,V2_RC_FRMR_WQE_BYTE_40_LW_S , + !!(wr->access &IB_ACCESS_LOCAL_WRITE ));/* Data structure reuse may lead to confusion */ pbl_ba = mr->pbl_mtr.hem_cfg.root_ba;
@@ -121,12 +121,10 @@ static void set_frmr_seg(struct hns_roce_v2_rc_send_wqe *rc_sq_wqe, rc_sq_wqe->va =cpu_to_le64 (wr->mr->iova); fseg->pbl_size =cpu_to_le32 (mr->npages); -roce_set_field (fseg->mode_buf_pg_sz, -V2_RC_FRMR_WQE_BYTE_40_PBL_BUF_PG_SZ_M , +roce_set_field (fseg->byte_40,V2_RC_FRMR_WQE_BYTE_40_PBL_BUF_PG_SZ_M ,V2_RC_FRMR_WQE_BYTE_40_PBL_BUF_PG_SZ_S ,to_hr_hw_page_shift (mr->pbl_mtr.hem_cfg.buf_pg_shift)); -roce_set_bit (fseg->mode_buf_pg_sz, -V2_RC_FRMR_WQE_BYTE_40_BLK_MODE_S ,0 ); +roce_set_bit (fseg->byte_40,V2_RC_FRMR_WQE_BYTE_40_BLK_MODE_S ,0 ); } static void set_atomic_seg(const struct ib_send_wr *wr,
drivers/infiniband/hw/hns/hns_roce_hw_v2.h +6 -6
@@ -1255,15 +1255,15 @@ struct hns_roce_v2_rc_send_wqe {#define V2_RC_SEND_WQE_BYTE_4_INLINE_S 12 -#define V2_RC_FRMR_WQE_BYTE_4_BIND_EN_S 19 +#define V2_RC_FRMR_WQE_BYTE_40_BIND_EN_S 10 -#define V2_RC_FRMR_WQE_BYTE_4_ATOMIC_S 20 +#define V2_RC_FRMR_WQE_BYTE_40_ATOMIC_S 11 -#define V2_RC_FRMR_WQE_BYTE_4_RR_S 21 +#define V2_RC_FRMR_WQE_BYTE_40_RR_S 12 -#define V2_RC_FRMR_WQE_BYTE_4_RW_S 22 +#define V2_RC_FRMR_WQE_BYTE_40_RW_S 13 -#define V2_RC_FRMR_WQE_BYTE_4_LW_S 23 +#define V2_RC_FRMR_WQE_BYTE_40_LW_S 14#define V2_RC_SEND_WQE_BYTE_4_FLAG_S 31
@@ -1280,7 +1280,7 @@ struct hns_roce_v2_rc_send_wqe {struct hns_roce_wqe_frmr_seg {__le32 pbl_size; -__le32 mode_buf_pg_sz; +__le32 byte_40; };#define V2_RC_FRMR_WQE_BYTE_40_PBL_BUF_PG_SZ_S 4
Signature · 1 line
2.8.1
Jason Gunthorpe
maintainerRe: [PATCH for-next 05/12] RDMA/hns: Adjust definition of FRMR fields
Quoted 25 lines · Weihang Li
drivers/infiniband/hw/hns/hns_roce_hw_v2.h +6
-5+++ b/drivers/infiniband/hw/hns/hns_roce_hw_v2.h@@ -1255,15 +1255,15 @@ struct hns_roce_v2_rc_send_wqe {#define V2_RC_SEND_WQE_BYTE_4_INLINE_S 12 -#define V2_RC_FRMR_WQE_BYTE_4_BIND_EN_S 19 +#define V2_RC_FRMR_WQE_BYTE_40_BIND_EN_S 10 -#define V2_RC_FRMR_WQE_BYTE_4_ATOMIC_S 20 +#define V2_RC_FRMR_WQE_BYTE_40_ATOMIC_S 11 -#define V2_RC_FRMR_WQE_BYTE_4_RR_S 21 +#define V2_RC_FRMR_WQE_BYTE_40_RR_S 12 -#define V2_RC_FRMR_WQE_BYTE_4_RW_S 22 +#define V2_RC_FRMR_WQE_BYTE_40_RW_S 13 -#define V2_RC_FRMR_WQE_BYTE_4_LW_S 23 +#define V2_RC_FRMR_WQE_BYTE_40_LW_S 14#define V2_RC_SEND_WQE_BYTE_4_FLAG_S 31
drivers/infiniband/hw/hns/hns_roce_hw_v2.h struct hns_roce_v2_rc_send_wqe { · line 1284
Quoted 6 lines
drivers/infiniband/hw/hns/hns_roce_hw_v2.h +1
-1@@ -1280,7 +1280,7 @@ struct hns_roce_v2_rc_send_wqe {struct hns_roce_wqe_frmr_seg {__le32 pbl_size; -__le32 mode_buf_pg_sz; +__le32 byte_40; };
This stuff is HW API isn't it?
I didn't see anything to negotiate compatability with existing HW? What happens if the kernel is updated and run on old HW/FW?
If you tightly couple you still need to check and refuse to load the driver.
Jason
Re: [PATCH for-next 05/12] RDMA/hns: Adjust definition of FRMR fields
Quoted 35 lines · Jason Gunthorpe
Quoted 25 lines · Weihang Li
drivers/infiniband/hw/hns/hns_roce_hw_v2.h +6
-5+++ b/drivers/infiniband/hw/hns/hns_roce_hw_v2.h@@ -1255,15 +1255,15 @@ struct hns_roce_v2_rc_send_wqe {#define V2_RC_SEND_WQE_BYTE_4_INLINE_S 12 -#define V2_RC_FRMR_WQE_BYTE_4_BIND_EN_S 19 +#define V2_RC_FRMR_WQE_BYTE_40_BIND_EN_S 10 -#define V2_RC_FRMR_WQE_BYTE_4_ATOMIC_S 20 +#define V2_RC_FRMR_WQE_BYTE_40_ATOMIC_S 11 -#define V2_RC_FRMR_WQE_BYTE_4_RR_S 21 +#define V2_RC_FRMR_WQE_BYTE_40_RR_S 12 -#define V2_RC_FRMR_WQE_BYTE_4_RW_S 22 +#define V2_RC_FRMR_WQE_BYTE_40_RW_S 13 -#define V2_RC_FRMR_WQE_BYTE_4_LW_S 23 +#define V2_RC_FRMR_WQE_BYTE_40_LW_S 14#define V2_RC_SEND_WQE_BYTE_4_FLAG_S 31drivers/infiniband/hw/hns/hns_roce_hw_v2.h struct hns_roce_v2_rc_send_wqe { · line 1284
Quoted 6 lines
drivers/infiniband/hw/hns/hns_roce_hw_v2.h +1
-1@@ -1280,7 +1280,7 @@ struct hns_roce_v2_rc_send_wqe {struct hns_roce_wqe_frmr_seg {__le32 pbl_size; -__le32 mode_buf_pg_sz; +__le32 byte_40; };This stuff is HW API isn't it?
Quoted 8 lines
I didn't see anything to negotiate compatability with existing HW? What happens if the kernel is updated and run on old HW/FW?
If you tightly couple you still need to check and refuse to load the driver.
Jason
Thank you, FRMR is not well-supported on HIP08, so we re-design it on HIP09. I will add a check to avoid ULPs using FRMR on HIP08.
Weihang
PATCH 6/12 RDMA/hns: Skip qp_flow_control_init() for HIP09
In reply to Weihang LiFrom: Yixing Liu <[email protected]>
Since HIP09 does not require this function, it should be masked.
--- drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/infiniband/hw/hns/hns_roce_hw_v2.c +3 -0
@@ -5233,6 +5233,9 @@ static int hns_roce_v2_qp_flow_control_init(struct hns_roce_dev *hr_dev,struct hns_roce_cmq_desc desc;int ret, i; +if (hr_dev->pci_dev->revision >=PCI_REVISION_ID_HIP09 ) +return 0 ; +mutex_lock (&hr_dev->qp_table.scc_mutex);/* set scc ctx clear done flag */
Signature · 1 line
2.8.1
PATCH 7/12 RDMA/hns: Replace wmb&__raw_writeq with writeq
In reply to Weihang LiFrom: Lang Cheng <[email protected]>
Currently, the driver updates doorbell looks like this:post()
{
wqe.field = 0x111;
wmb();
update_wq_db();
}update_wq_db()
{
db.field = 0x222;
__raw_writeq(db, db_reg);
}writeq() is a better choice than __raw_writeq() because it calls dma_wmb() to barrier in ARM64, and dma_wmb() is better than wmb() for ROCEE device.
This patch removes all wmb() before updating doorbell of SQ/RQ/CQ/SRQ by replacing __raw_writeq() with writeq() to improve performence. The new process looks like this:post()
{
wqe.field = 0x111;
update_wq_db();
}update_wq_db()
{
db.field = 0x222;
writeq(db, db_reg);
}--- drivers/infiniband/hw/hns/hns_roce_device.h | 2 +- drivers/infiniband/hw/hns/hns_roce_hw_v1.c | 15 --------------- drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 20 +------------------- 3 files changed, 2 insertions(+), 35 deletions(-)
drivers/infiniband/hw/hns/hns_roce_device.h +1 -1
@@ -1077,7 +1077,7 @@ static inline struct hns_roce_srq *to_hr_srq(struct ib_srq *ibsrq)static inline void hns_roce_write64_k (__le32 val[2 ],void __iomem *dest) { -__raw_writeq (*(u64 *) val, dest); +writeq (*(u64 *)val, dest); } static inline struct hns_roce_qp
drivers/infiniband/hw/hns/hns_roce_hw_v1.c +0 -15
@@ -330,8 +330,6 @@ static int hns_roce_v1_post_send(struct ib_qp *ibqp,/* Set DB return */ if (likely (nreq)) { qp->sq.head += nreq; -/* Memory barrier */ -wmb (); roce_set_field(sq_db.u32_4, SQ_DOORBELL_U32_4_SQ_HEAD_M, SQ_DOORBELL_U32_4_SQ_HEAD_S,
@@ -411,8 +409,6 @@ static int hns_roce_v1_post_recv(struct ib_qp *ibqp, out:if (likely (nreq)) { hr_qp->rq.head += nreq; -/* Memory barrier */ -wmb ();if (ibqp->qp_type ==IB_QPT_GSI ) {__le32 tmp;
@@ -1984,12 +1980,6 @@ static void __hns_roce_v1_cq_clean(struct hns_roce_cq *hr_cq, u32 qpn,if (nfreed) { hr_cq->cons_index += nfreed; -/* -* Make sure update of buffer contents is done before -* updating consumer index. -*/ -wmb (); -hns_roce_v1_cq_set_ci (hr_cq, hr_cq->cons_index); } }
@@ -2330,8 +2320,6 @@ int hns_roce_v1_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc) *hr_cq->tptr_addr = hr_cq->cons_index & ((hr_cq->cq_depth <<1 ) -1 ); -/* Memroy barrier */ -wmb ();hns_roce_v1_cq_set_ci (hr_cq, hr_cq->cons_index); }
@@ -3220,9 +3208,6 @@ static int hns_roce_v1_m_qp(struct ib_qp *ibqp, const struct ib_qp_attr *attr,* need to hw to flash RQ HEAD by DB again */ if (cur_state ==IB_QPS_INIT && new_state ==IB_QPS_INIT ) { -/* Memory barrier */ -wmb (); -roce_set_field (doorbell[0 ],RQ_DOORBELL_U32_4_RQ_HEAD_M ,RQ_DOORBELL_U32_4_RQ_HEAD_S , hr_qp->rq.head); roce_set_field(doorbell[1], RQ_DOORBELL_U32_8_QPN_M,
drivers/infiniband/hw/hns/hns_roce_hw_v2.c +1 -19
@@ -742,8 +742,6 @@ static int hns_roce_v2_post_send(struct ib_qp *ibqp,if (likely (nreq)) { qp->sq.head += nreq; qp->next_sge = sge_idx; -/* Memory barrier */ -wmb ();if (nreq ==1 && qp->sq.head == qp->sq.tail +1 && (qp->en_flags &HNS_ROCE_QP_CAP_DIRECT_WQE ))
@@ -873,8 +871,6 @@ static int hns_roce_v2_post_recv(struct ib_qp *ibqp, out:if (likely (nreq)) { hr_qp->rq.head += nreq; -/* Memory barrier */ -wmb ();/* * Hip08 hardware cannot flush the WQEs in RQ if the QP state
@@ -1013,12 +1009,6 @@ static int hns_roce_v2_post_srq_recv(struct ib_srq *ibsrq, }if (likely (nreq)) { -/* -* Make sure that descriptors are written before -* doorbell record. -*/ -wmb (); - srq_db.byte_4 =cpu_to_le32 (HNS_ROCE_V2_SRQ_DB <<V2_DB_BYTE_4_CMD_S | (srq->srqn &V2_DB_BYTE_4_TAG_M ));
@@ -3196,11 +3186,6 @@ static void __hns_roce_v2_cq_clean(struct hns_roce_cq *hr_cq, u32 qpn,if (nfreed) { hr_cq->cons_index += nfreed; -/* -* Make sure update of buffer contents is done before -* updating consumer index. -*/ -wmb ();hns_roce_v2_cq_set_ci (hr_cq, hr_cq->cons_index); } }
@@ -3709,11 +3694,8 @@ static int hns_roce_v2_poll_cq(struct ib_cq *ibcq, int num_entries,break ; } -if (npolled) { -/* Memory barrier */ -wmb (); +if (npolled)hns_roce_v2_cq_set_ci (hr_cq, hr_cq->cons_index); - } out:spin_unlock_irqrestore (&hr_cq->lock, flags);
Signature · 1 line
2.8.1
PATCH 8/12 RDMA/hns: Move HIP06 related definitions into hns_roce_hw_v1.h
In reply to Weihang LiFrom: Lang Cheng <[email protected]>
hns_roce_device.h is not specific to hardware, some definitions are only used for HIP06, they should be moved into hns_roce_hw_v1.h.
--- drivers/infiniband/hw/hns/hns_roce_device.h | 41 --------------------------- drivers/infiniband/hw/hns/hns_roce_hw_v1.h | 43 +++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+), 41 deletions(-)
drivers/infiniband/hw/hns/hns_roce_device.h +0 -41
@@ -170,44 +170,6 @@ enum hns_roce_event { HNS_ROCE_EVENT_TYPE_FLR = 0x15, }; -/* Local Work Queue Catastrophic Error,SUBTYPE 0x5 */ -enum { -HNS_ROCE_LWQCE_QPC_ERROR =1 , -HNS_ROCE_LWQCE_MTU_ERROR =2 , -HNS_ROCE_LWQCE_WQE_BA_ADDR_ERROR =3 , -HNS_ROCE_LWQCE_WQE_ADDR_ERROR =4 , -HNS_ROCE_LWQCE_SQ_WQE_SHIFT_ERROR =5 , -HNS_ROCE_LWQCE_SL_ERROR =6 , -HNS_ROCE_LWQCE_PORT_ERROR =7 , -}; - -/* Local Access Violation Work Queue Error,SUBTYPE 0x7 */ -enum { -HNS_ROCE_LAVWQE_R_KEY_VIOLATION =1 , -HNS_ROCE_LAVWQE_LENGTH_ERROR =2 , -HNS_ROCE_LAVWQE_VA_ERROR =3 , -HNS_ROCE_LAVWQE_PD_ERROR =4 , -HNS_ROCE_LAVWQE_RW_ACC_ERROR =5 , -HNS_ROCE_LAVWQE_KEY_STATE_ERROR =6 , -HNS_ROCE_LAVWQE_MR_OPERATION_ERROR =7 , -}; - -/* DOORBELL overflow subtype */ -enum { -HNS_ROCE_DB_SUBTYPE_SDB_OVF =1 , -HNS_ROCE_DB_SUBTYPE_SDB_ALM_OVF =2 , -HNS_ROCE_DB_SUBTYPE_ODB_OVF =3 , -HNS_ROCE_DB_SUBTYPE_ODB_ALM_OVF =4 , -HNS_ROCE_DB_SUBTYPE_SDB_ALM_EMP =5 , -HNS_ROCE_DB_SUBTYPE_ODB_ALM_EMP =6 , -}; - -enum { -/* RQ&SRQ related operations */ -HNS_ROCE_OPCODE_SEND_DATA_RECEIVE =0x06 , -HNS_ROCE_OPCODE_RDMA_WITH_IMM_RECEIVE =0x07 , -}; -#define HNS_ROCE_CAP_FLAGS_EX_SHIFT 12enum {
@@ -260,9 +222,6 @@ enum {#define HNS_ROCE_CMD_SUCCESS 1 -#define HNS_ROCE_PORT_DOWN 0 -#define HNS_ROCE_PORT_UP 1 -/* The minimum page size is 4K for hardware */ #define HNS_HW_PAGE_SHIFT 12#define HNS_HW_PAGE_SIZE (1 << HNS_HW_PAGE_SHIFT)
drivers/infiniband/hw/hns/hns_roce_hw_v1.h +43 -0
49 lines not on this page · raw message
Signature · 1 line
2.8.1
PATCH 9/12 RDMA/hns: Remove some magic numbers
In reply to Weihang LiFrom: Xinhao Liu <[email protected]>
Use macros instead of magic numbers to represent shift of dma_handle_wqe, dma_handle_idx and UDP destination port number of RoCEv2.
--- drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)
drivers/infiniband/hw/hns/hns_roce_hw_v2.c +11 -5
@@ -1601,7 +1601,8 @@ static int hns_roce_config_global_param(struct hns_roce_dev *hr_dev) CFG_GLOBAL_PARAM_DATA_0_ROCEE_TIME_1US_CFG_S, 0x3e8);roce_set_field (req->time_cfg_udp_port,CFG_GLOBAL_PARAM_DATA_0_ROCEE_UDP_PORT_M , -CFG_GLOBAL_PARAM_DATA_0_ROCEE_UDP_PORT_S ,0x12b7 ); +CFG_GLOBAL_PARAM_DATA_0_ROCEE_UDP_PORT_S , +ROCE_V2_UDP_DPORT );return hns_roce_cmq_send (hr_dev, &desc,1 ); }
@@ -5264,6 +5265,9 @@ static int hns_roce_v2_qp_flow_control_init(struct hns_roce_dev *hr_dev,return ret; } +#define DMA_IDX_SHIFT 3 +#define DMA_WQE_SHIFT 3 +static int hns_roce_v2_write_srqc_index_queue (struct hns_roce_srq *srq,struct hns_roce_srq_context *ctx) {
@@ -5286,8 +5290,9 @@ static int hns_roce_v2_write_srqc_index_queue(struct hns_roce_srq *srq,hr_reg_write (ctx,SRQC_IDX_HOP_NUM ,to_hr_hem_hopnum (hr_dev->caps.idx_hop_num, srq->wqe_cnt)); -hr_reg_write (ctx,SRQC_IDX_BT_BA_L , dma_handle_idx >>3 ); -hr_reg_write (ctx,SRQC_IDX_BT_BA_H ,upper_32_bits (dma_handle_idx >>3 )); +hr_reg_write (ctx,SRQC_IDX_BT_BA_L , dma_handle_idx >>DMA_IDX_SHIFT ); +hr_reg_write (ctx,SRQC_IDX_BT_BA_H , +upper_32_bits (dma_handle_idx >>DMA_IDX_SHIFT ));hr_reg_write (ctx,SRQC_IDX_BA_PG_SZ ,to_hr_hw_page_shift (idx_que->mtr.hem_cfg.ba_pg_shift));
@@ -5340,8 +5345,9 @@ static int hns_roce_v2_write_srqc(struct hns_roce_srq *srq, void *mb_buf)to_hr_hem_hopnum (hr_dev->caps.srqwqe_hop_num, srq->wqe_cnt)); -hr_reg_write (ctx,SRQC_WQE_BT_BA_L , dma_handle_wqe >>3 ); -hr_reg_write (ctx,SRQC_WQE_BT_BA_H ,upper_32_bits (dma_handle_wqe >>3 )); +hr_reg_write (ctx,SRQC_WQE_BT_BA_L , dma_handle_wqe >>DMA_WQE_SHIFT ); +hr_reg_write (ctx,SRQC_WQE_BT_BA_H , +upper_32_bits (dma_handle_wqe >>DMA_WQE_SHIFT ));hr_reg_write (ctx,SRQC_WQE_BA_PG_SZ ,to_hr_hw_page_shift (srq->buf_mtr.hem_cfg.ba_pg_shift));
Signature · 1 line
2.8.1
PATCH 10/12 RDMA/hns: Avoid unnecessary memset on WQEs in post_send
In reply to Weihang LiFrom: Lang Cheng <[email protected]>
All fields of WQE will be rewrote, so the memset is unnecessary. And when SQ is working in OWNER mode, the pipeline may prefetch the WQEs beyond PI, the memset operation may flip the owner bit too early, then the pipeline may get a wrong WQ.
--- drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 2 -- 1 file changed, 2 deletions(-)
drivers/infiniband/hw/hns/hns_roce_hw_v2.c +0 -2
@@ -467,7 +467,6 @@ static inline int set_ud_wqe(struct hns_roce_qp *qp,int ret; valid_num_sge =calc_wr_sge_num (wr, &msg_len); -memset (ud_sq_wqe,0 ,sizeof (*ud_sq_wqe)); ret =set_ud_opcode (ud_sq_wqe, wr);if (WARN_ON (ret))
@@ -573,7 +572,6 @@ static inline int set_rc_wqe(struct hns_roce_qp *qp,int ret; valid_num_sge =calc_wr_sge_num (wr, &msg_len); -memset (rc_sq_wqe,0 ,sizeof (*rc_sq_wqe)); rc_sq_wqe->msg_len =cpu_to_le32 (msg_len);
Signature · 1 line
2.8.1
PATCH 11/12 RDMA/hns: Remove unnecessary wrap around for EQ's consumer index
In reply to Weihang LiFrom: Yixian Liu <[email protected]>
The hns driver wrap around the consumer index of AEQ and CEQ when they reach to two times of queue entries number for owner mechanism, actually, it is unnecessary to wrap around since the hardware itself will mask it before use.
--- drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 6 ------ 1 file changed, 6 deletions(-)
drivers/infiniband/hw/hns/hns_roce_hw_v2.c +0 -6
@@ -5667,9 +5667,6 @@ static int hns_roce_v2_aeq_int(struct hns_roce_dev *hr_dev, ++eq->cons_index; aeqe_found =1 ; -if (eq->cons_index > (2 * eq->entries -1 )) - eq->cons_index =0 ; -hns_roce_v2_init_irq_work (hr_dev, eq, queue_num); aeqe =next_aeqe_sw_v2 (eq);
@@ -5712,9 +5709,6 @@ static int hns_roce_v2_ceq_int(struct hns_roce_dev *hr_dev, ++eq->cons_index; ceqe_found =1 ; -if (eq->cons_index > (EQ_DEPTH_COEFF * eq->entries -1 )) - eq->cons_index =0 ; - ceqe =next_ceqe_sw_v2 (eq); }
Signature · 1 line
2.8.1
PATCH 12/12 RDMA/hns: Delete redundant judgment when preparing descriptors
In reply to Weihang LiFrom: Xinhao Liu <[email protected]>
There is no need to use a for loop to assign values for an array of cmd descriptors which has only two elements.
--- drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 40 +++++++++++------------------- 1 file changed, 14 insertions(+), 26 deletions(-)
drivers/infiniband/hw/hns/hns_roce_hw_v2.c +14 -26
@@ -1611,17 +1611,13 @@ static int hns_roce_query_pf_resource(struct hns_roce_dev *hr_dev)struct hns_roce_pf_res_a *req_a;struct hns_roce_pf_res_b *req_b;int ret; -int i; -for (i =0 ; i <2 ; i++) { -hns_roce_cmq_setup_basic_desc (&desc[i], -HNS_ROCE_OPC_QUERY_PF_RES , true); +hns_roce_cmq_setup_basic_desc (&desc[0 ],HNS_ROCE_OPC_QUERY_PF_RES , + true); + desc[0 ].flag |=cpu_to_le16 (HNS_ROCE_CMD_FLAG_NEXT ); -if (i ==0 ) - desc[i].flag |=cpu_to_le16 (HNS_ROCE_CMD_FLAG_NEXT ); -else - desc[i].flag &= ~cpu_to_le16 (HNS_ROCE_CMD_FLAG_NEXT ); - } +hns_roce_cmq_setup_basic_desc (&desc[1 ],HNS_ROCE_OPC_QUERY_PF_RES , + true); ret =hns_roce_cmq_send (hr_dev, desc,2 );if (ret)
@@ -1714,19 +1710,16 @@ static int hns_roce_alloc_vf_resource(struct hns_roce_dev *hr_dev)struct hns_roce_cmq_desc desc[2 ];struct hns_roce_vf_res_a *req_a;struct hns_roce_vf_res_b *req_b; -int i; req_a = (struct hns_roce_vf_res_a *)desc[0 ].data; req_b = (struct hns_roce_vf_res_b *)desc[1 ].data; -for (i =0 ; i <2 ; i++) { -hns_roce_cmq_setup_basic_desc (&desc[i], -HNS_ROCE_OPC_ALLOC_VF_RES , false); -if (i ==0 ) - desc[i].flag |=cpu_to_le16 (HNS_ROCE_CMD_FLAG_NEXT ); -else - desc[i].flag &= ~cpu_to_le16 (HNS_ROCE_CMD_FLAG_NEXT ); - } +hns_roce_cmq_setup_basic_desc (&desc[0 ],HNS_ROCE_OPC_ALLOC_VF_RES , + false); + desc[0 ].flag |=cpu_to_le16 (HNS_ROCE_CMD_FLAG_NEXT ); + +hns_roce_cmq_setup_basic_desc (&desc[1 ],HNS_ROCE_OPC_ALLOC_VF_RES , + false); roce_set_field(req_a->vf_qpc_bt_idx_num, VF_RES_A_DATA_1_VF_QPC_BT_IDX_M,
@@ -2407,7 +2400,6 @@ static int hns_roce_config_link_table(struct hns_roce_dev *hr_dev,struct hns_roce_link_table_entry *entry;enum hns_roce_opcode_type opcode;u32 page_num; -int i;switch (type) {case TSQ_LINK_TABLE :
@@ -2425,14 +2417,10 @@ static int hns_roce_config_link_table(struct hns_roce_dev *hr_dev, page_num = link_tbl->npages; entry = link_tbl->table.buf; -for (i =0 ; i <2 ; i++) { -hns_roce_cmq_setup_basic_desc (&desc[i], opcode, false); +hns_roce_cmq_setup_basic_desc (&desc[0 ], opcode, false); + desc[0 ].flag |=cpu_to_le16 (HNS_ROCE_CMD_FLAG_NEXT ); -if (i ==0 ) - desc[i].flag |=cpu_to_le16 (HNS_ROCE_CMD_FLAG_NEXT ); -else - desc[i].flag &= ~cpu_to_le16 (HNS_ROCE_CMD_FLAG_NEXT ); - } +hns_roce_cmq_setup_basic_desc (&desc[1 ], opcode, false); req_a->base_addr_l =cpu_to_le32 (link_tbl->table.map &0xffffffff ); req_a->base_addr_h =cpu_to_le32 (link_tbl->table.map >>32 );
Signature · 1 line
2.8.1
Jason Gunthorpe
maintainerQuoted 10 lines · Weihang Li
As usual, this series collects some miscellaneous fixes and cleanups at the end of 5.12 for the hns driver:
* #1 ~ #3 fix some non-urgent issues.
* #4 ~ #6 make some changes on existing features.
* #7 ~ #12 are cleanups, just do some refactor and delete dead code.This series is made based on the series named "RDMA/hns: Several fixes and cleanups of RQ/SRQ" which is still being reviewed:
https://patchwork.kernel.org/project/linux-rdma/cover/[email protected]/
Quoted 7 lines
I'm worried that it will be too late for 5.12 if any patch of this series needs changes, so I send it before the previous one is merged. If there is any comment or merge conflict on it, I will fix them as soon as possible, thanks.
Lang Cheng (3):
RDMA/hns: Replace wmb&__raw_writeq with writeq
This one is quite a good idea
Quoted 17 lines
RDMA/hns: Move HIP06 related definitions into
hns_roce_hw_v1.hRDMA/hns: Avoid unnecessary memset on WQEs inpost_sendLijun Ou (1):
RDMA/hns: Disable RQ inline by defaultWeihang Li (2):
RDMA/hns: Avoid filling sgid index when modifying QP to RTR RDMA/hns: Fix type ofsq_signal_bitsXi Wang (1):
RDMA/hns: Add mapped page count checking for MTRXinhao Liu (2):
RDMA/hns: Remove some magic numbers
RDMA/hns: Delete redundant judgment when preparing descriptors
Quoted 8 lines
Yixian Liu (1):
RDMA/hns: Remove unnecessary wrap around for EQ's consumer indexYixing Liu (2):
RDMA/hns: Skipqp_flow_control_init()for HIP09RDMA/hns: Adjust definition of FRMR fields
I didn't apply this one because of my question, but did take the rest to for-next, thanks
Jason