RDMA/erdma: erdma fixes 3-20-2023

PATCH Applied Cheng Xu 2 participants 6 messages Mar 20, 2023

Subsystems (2)

InfiniBand Alibaba elastic RDMA driver

lore

Share & tools

lore

Cheng Xu

author maintainer

2023-03-20 08:47 UTC #0 lore raw

cover RDMA/erdma: erdma fixes 3-20-2023

Though there is an ongoing patch waiting Jason to take a look, this series with small changes may be accepted with less discussion. So I submit them first.

This series has some fixes for erdma driver:
- #1 fixes some typos.
- #2 updates the default EQ depth and max_send_wr for SQ.
- #3 inlines 4 mtt entries for FRMR to achieve the HW limitation.
- #4 defers probing flow if netdevice does not exist temporarily.

Thanks,
Cheng Xu

Cheng Xu (4):
RDMA/erdma: Fix some typos
RDMA/erdma: Update default EQ depth to 4096 and max_send_wr to 8192 RDMA/erdma: Inline mtt entries into WQE if supported RDMA/erdma: Defer probing if netdevice can not be found

 drivers/infiniband/hw/erdma/erdma_cq.c    | 2 +-
 drivers/infiniband/hw/erdma/erdma_hw.h    | 4 ++--
 drivers/infiniband/hw/erdma/erdma_main.c  | 2 +-
 drivers/infiniband/hw/erdma/erdma_qp.c    | 4 ++--
 drivers/infiniband/hw/erdma/erdma_verbs.h | 2 +-
 5 files changed, 7 insertions(+), 7 deletions(-)

Signature · 1 line

2.31.1

Cheng Xu

author maintainer

2023-03-20 08:47 UTC #1 lore raw

PATCH 1/4 RDMA/erdma: Fix some typos

FAA is short for atomic fetch and add, not FAD. Fix this.

Fixes 0ca9c2e2844a ("RDMA/erdma: Implement atomic operations support")

Signed-off-by Cheng Xu <[email protected]>

---
 drivers/infiniband/hw/erdma/erdma_cq.c | 2 +-
 drivers/infiniband/hw/erdma/erdma_hw.h | 2 +-
 drivers/infiniband/hw/erdma/erdma_qp.c | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

drivers/infiniband/hw/erdma/erdma_cq.c +1 -1

index cabd8678b355..7bc354273d4e 100644
@@ -65,7 +65,7 @@ static const enum ib_wc_opcode wc_mapping_table[ERDMA_NUM_OPCODES] = {
 	[ERDMA_OP_LOCAL_INV] = IB_WC_LOCAL_INV,
 	[ERDMA_OP_READ_WITH_INV] = IB_WC_RDMA_READ,
 	[ERDMA_OP_ATOMIC_CAS] = IB_WC_COMP_SWAP,
-	[ERDMA_OP_ATOMIC_FAD] = IB_WC_FETCH_ADD,
+	[ERDMA_OP_ATOMIC_FAA] = IB_WC_FETCH_ADD,
 };
 
 static const struct {

drivers/infiniband/hw/erdma/erdma_hw.h +1 -1

index 4c38d99c73f1..5d3a541db941 100644
@@ -491,7 +491,7 @@ enum erdma_opcode {
 	ERDMA_OP_LOCAL_INV = 15,
 	ERDMA_OP_READ_WITH_INV = 16,
 	ERDMA_OP_ATOMIC_CAS = 17,
-	ERDMA_OP_ATOMIC_FAD = 18,
+	ERDMA_OP_ATOMIC_FAA = 18,
 	ERDMA_NUM_OPCODES = 19,
 	ERDMA_OP_INVALID = ERDMA_NUM_OPCODES + 1
 };

drivers/infiniband/hw/erdma/erdma_qp.c +1 -1

index d088d6bef431..ff473b208acf 100644
@@ -439,7 +439,7 @@ static int erdma_push_one_sqe(struct erdma_qp *qp, u16 *pi,
 				cpu_to_le64(atomic_wr(send_wr)->compare_add);
 		} else {
 			wqe_hdr |= FIELD_PREP(ERDMA_SQE_HDR_OPCODE_MASK,
-					      ERDMA_OP_ATOMIC_FAD);
+					      ERDMA_OP_ATOMIC_FAA);
 			atomic_sqe->fetchadd_swap_data =
 				cpu_to_le64(atomic_wr(send_wr)->compare_add);
 		}

Signature · 1 line

2.31.1

Cheng Xu

author maintainer

2023-03-20 08:47 UTC #2 lore raw

PATCH 2/4 RDMA/erdma: Update default EQ depth to 4096 and max_send_wr to 8192

In reply to Cheng Xu

Max EQ depth of hardware is 32K, the current default EQ depth is too small for some applications, so change the default depth to 4096. Max send WRs the hardware can support is 8K, but the driver limits the value to 4K. Remove this limitation.

Fixes be3cff0f242d ("RDMA/erdma: Add the hardware related definitions")

Fixes db23ae64caac ("RDMA/erdma: Add verbs header file")

Signed-off-by Cheng Xu <[email protected]>

---
 drivers/infiniband/hw/erdma/erdma_hw.h    | 2 +-
 drivers/infiniband/hw/erdma/erdma_verbs.h | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

drivers/infiniband/hw/erdma/erdma_hw.h +1 -1

index 5d3a541db941..37ad1bb1917c 100644
@@ -441,7 +441,7 @@ struct erdma_reg_mr_sqe {
 };
 
 /* EQ related. */
-#define ERDMA_DEFAULT_EQ_DEPTH 256
+#define ERDMA_DEFAULT_EQ_DEPTH 4096
 
 /* ceqe */
 #define ERDMA_CEQE_HDR_DB_MASK BIT_ULL(63)

drivers/infiniband/hw/erdma/erdma_verbs.h +1 -1

index e0a993bc032a..131cf5f40982 100644
@@ -11,7 +11,7 @@
 
 /* RDMA Capability. */
 #define ERDMA_MAX_PD (128 * 1024)
-#define ERDMA_MAX_SEND_WR 4096
+#define ERDMA_MAX_SEND_WR 8192
 #define ERDMA_MAX_ORD 128
 #define ERDMA_MAX_IRD 128
 #define ERDMA_MAX_SGE_RD 1

Signature · 1 line

2.31.1

Cheng Xu

author maintainer

2023-03-20 08:47 UTC #3 lore raw

PATCH 3/4 RDMA/erdma: Inline mtt entries into WQE if supported

In reply to Cheng Xu

The max inline mtt count supported is ERDMA_MAX_INLINE_MTT_ENTRIES. When mr->mem.mtt_nents == ERDMA_MAX_INLINE_MTT_ENTRIES, inline mtt is also supported, fix it.

Fixes 155055771704 ("RDMA/erdma: Add verbs implementation")

Signed-off-by Cheng Xu <[email protected]>

---
 drivers/infiniband/hw/erdma/erdma_qp.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

drivers/infiniband/hw/erdma/erdma_qp.c +1 -1

index ff473b208acf..44923c51a01b 100644
@@ -405,7 +405,7 @@ static int erdma_push_one_sqe(struct erdma_qp *qp, u16 *pi,
 			FIELD_PREP(ERDMA_SQE_MR_MTT_CNT_MASK,
 				   mr->mem.mtt_nents);
 
-		if (mr->mem.mtt_nents < ERDMA_MAX_INLINE_MTT_ENTRIES) {
+		if (mr->mem.mtt_nents <= ERDMA_MAX_INLINE_MTT_ENTRIES) {
 			attrs |= FIELD_PREP(ERDMA_SQE_MR_MTT_TYPE_MASK, 0);
 			/* Copy SGLs to SQE content to accelerate */
 			memcpy(get_queue_entry(qp->kern_qp.sq_buf, idx + 1,

Signature · 1 line

2.31.1

Cheng Xu

author maintainer

2023-03-20 08:47 UTC #4 lore raw

PATCH 4/4 RDMA/erdma: Defer probing if netdevice can not be found

In reply to Cheng Xu

ERDMA device may be probed before its associated netdevice, returning -EPROBE_DEFER allows OS try to probe erdma device later.

Fixes d55e6fb4803c ("RDMA/erdma: Add the erdma module")

Signed-off-by Cheng Xu <[email protected]>

---
 drivers/infiniband/hw/erdma/erdma_main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

drivers/infiniband/hw/erdma/erdma_main.c +1 -1

index 5dc31e5df5cb..4a29a53a6652 100644
@@ -56,7 +56,7 @@ static int erdma_netdev_event(struct notifier_block *nb, unsigned long event,
 static int erdma_enum_and_get_netdev(struct erdma_dev *dev)
 {
 	struct net_device *netdev;
-	int ret = -ENODEV;
+	int ret = -EPROBE_DEFER;
 
 	/* Already binded to a net_device, so we skip. */
 	if (dev->netdev)

Signature · 1 line

2.31.1

Leon Romanovsky

Applied

2023-03-20 10:57 UTC #5 lore raw

In reply to Cheng Xu

Quoted 11 lines · Cheng Xu

Though there is an ongoing patch waiting Jason to take a look, this series with small changes may be accepted with less discussion. So I submit them first.
This series has some fixes for erdma driver:
- #1 fixes some typos.
- #2 updates the default EQ depth and max_send_wr for SQ.
- #3 inlines 4 mtt entries for FRMR to achieve the HW limitation.
- #4 defers probing flow if netdevice does not exist temporarily.
[...]

Applied, thanks!

[1/4] RDMA/erdma: Fix some typos

      https://git.kernel.org/rdma/rdma/c/3fe26c0493e4c2
[2/4] RDMA/erdma: Update default EQ depth to 4096 and max_send_wr to 8192
      https://git.kernel.org/rdma/rdma/c/6256aa9ae955d1
[3/4] RDMA/erdma: Inline mtt entries into WQE if supported
      https://git.kernel.org/rdma/rdma/c/0dd83a4d775671
[4/4] RDMA/erdma: Defer probing if netdevice can not be found
      https://git.kernel.org/rdma/rdma/c/6bd1bca858f173

Best regards,

Signature · 1 line

Leon Romanovsky <[email protected]>