Skip to contentSkip to conversation

RDMA/mana_ib: Streamline kernel UD/GSI posting and completion handling

PATCH Applied Konstantin Taranov 3 participants 12 messages Sep 23 – Sep 28

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

v4 cover RDMA/mana_ib: Streamline kernel UD/GSI posting and completion handling

From: Konstantin Taranov <[email protected]>

This series updates the kernel UD/GSI work-request and completion paths in the MANA RDMA driver. This series is required for upcoming kernel RC QP support.

The main idea is to fill in WCs directly from CQEs and do not use shadow Qs as proxy. The older code, a CQ had lists of QPs using it, which adds overhead on polling when CQs are shared. Instead, CQs now maintain only the list of QPs in error state.

The series also revises posting WQEs to reuse it for RC QPs in the next series. We also add GDMA_WR_IB_SGL flag for posting to remove sge copying for RQs. The RC QPs will use it for SQs and RQs.

Imporve CQ arming to avoid doorbell drops in hw. MANA hw requires occasional arming to update tail offest in hw.

Changes in v4:
- re-write all commit messages

Changes in v3:
- Patch 5: Track unsignaled send WRs in the shadow queue.
- Patch 5: Validate the CQE target before retiring a shadow wqe.
- Patch 5: Notify the send CQ after posting to an already-flushing QP, outside the CQ lock, so GSI draining does not need another hardware CQE.
- Patches 1-4 are unchanged.

Changes in v2:
- Patch 5: Hold the corresponding CQ lock while resetting each shadow queue and removing the QP from the error list, serializing RESET with hardware polling and software flushing.
- Patch 5: Return the unmasked WQE start offset from GDMA posting to discard mismatching CQEs without producing a WC.
- Patches 1-4 are unchanged.

Konstantin Taranov (5):
RDMA/mana_ib: Optimize shadow queue bookkeeping RDMA/mana_ib: Revise UD send posting and WQE definitions RDMA/mana_ib: Revise UD receive posting with GDMA_WR_IB_SGL RDMA/mana_ib: Make kernel CQ arming robust RDMA/mana_ib: Poll UD completions and flush software error QPs

 drivers/infiniband/hw/mana/cq.c               | 344 +++++++++++++-----
 drivers/infiniband/hw/mana/mana_ib.h          | 105 +++++-
 drivers/infiniband/hw/mana/qp.c               |  84 +++--
 drivers/infiniband/hw/mana/shadow_queue.h     |  61 ++--
 drivers/infiniband/hw/mana/wr.c               | 158 ++++----
 .../net/ethernet/microsoft/mana/gdma_main.c   |  49 ++-
 include/net/mana/gdma.h                       |  11 +-
 7 files changed, 586 insertions(+), 226 deletions(-)

base-commit: 63367df6e7255067ad6a83abe0d2799dfa491876

Signature · 1 line
2.43.0
#1 lore raw

PATCH v4 1/5 RDMA/mana_ib: Optimize shadow queue bookkeeping

From: Konstantin Taranov <[email protected]>

Pack the posted WQE size and send opcode into compact bitfields. Move the UD receive shadow fields into the common shadow_wqe_header. This simplifies the code and lets all QP and WQ types share one structure.

Use release and acquire operations so posting and polling can safely run on different CPUs.

---
 drivers/infiniband/hw/mana/cq.c           | 35 ++++++++++++-----------
 drivers/infiniband/hw/mana/qp.c           |  4 +--
 drivers/infiniband/hw/mana/shadow_queue.h | 29 +++++++++----------
 drivers/infiniband/hw/mana/wr.c           | 15 +++++-----
 4 files changed, 40 insertions(+), 43 deletions(-)
drivers/infiniband/hw/mana/cq.c +18 -17
index 6f9ac8b4a..14a48409e 100644
@@ -190,15 +190,15 @@ static inline void handle_ud_sq_cqe(struct mana_ib_qp *qp, struct gdma_comp *cqe
 {
 	struct mana_rdma_cqe *rdma_cqe = (struct mana_rdma_cqe *)cqe->cqe_data;
 	struct gdma_queue *wq = qp->ud_qp.queues[MANA_UD_SEND_QUEUE].kmem;
-	struct ud_sq_shadow_wqe *shadow_wqe;
+	struct shadow_wqe_header *shadow_wqe;
 
 	shadow_wqe = shadow_queue_get_next_to_complete(&qp->shadow_sq);
 	if (!shadow_wqe)
 		return;
 
-	shadow_wqe->header.error_code = rdma_cqe->ud_send.vendor_error;
+	shadow_wqe->error_code = rdma_cqe->ud_send.vendor_error;
 
-	wq->tail += shadow_wqe->header.posted_wqe_size;
+	wq->tail += shadow_wqe->wqe_size_in_bu;
 	shadow_queue_advance_next_to_complete(&qp->shadow_sq);
 }
 
@@ -206,7 +206,7 @@ static inline void handle_ud_rq_cqe(struct mana_ib_qp *qp, struct gdma_comp *cqe
 {
 	struct mana_rdma_cqe *rdma_cqe = (struct mana_rdma_cqe *)cqe->cqe_data;
 	struct gdma_queue *wq = qp->ud_qp.queues[MANA_UD_RECV_QUEUE].kmem;
-	struct ud_rq_shadow_wqe *shadow_wqe;
+	struct shadow_wqe_header *shadow_wqe;
 
 	shadow_wqe = shadow_queue_get_next_to_complete(&qp->shadow_rq);
 	if (!shadow_wqe)
@@ -214,9 +214,9 @@ static inline void handle_ud_rq_cqe(struct mana_ib_qp *qp, struct gdma_comp *cqe
 
 	shadow_wqe->byte_len = rdma_cqe->ud_recv.msg_len;
 	shadow_wqe->src_qpn = rdma_cqe->ud_recv.src_qpn;
-	shadow_wqe->header.error_code = IB_WC_SUCCESS;
+	shadow_wqe->error_code = IB_WC_SUCCESS;
 
-	wq->tail += shadow_wqe->header.posted_wqe_size;
+	wq->tail += shadow_wqe->wqe_size_in_bu;
 	shadow_queue_advance_next_to_complete(&qp->shadow_rq);
 }
 
@@ -238,21 +238,20 @@ static void mana_handle_cqe(struct mana_ib_dev *mdev, struct gdma_comp *cqe)
 }
 
 static void fill_verbs_from_shadow_wqe(struct mana_ib_qp *qp, struct ib_wc *wc,
-				       const struct shadow_wqe_header *shadow_wqe)
+				       const struct shadow_wqe_header *shadow_wqe,
+				       enum ib_wc_opcode opcode)
 {
-	const struct ud_rq_shadow_wqe *ud_wqe = (const struct ud_rq_shadow_wqe *)shadow_wqe;
-
 	wc->wr_id = shadow_wqe->wr_id;
 	wc->status = shadow_wqe->error_code;
-	wc->opcode = shadow_wqe->opcode;
+	wc->opcode = opcode;
 	wc->vendor_err = shadow_wqe->error_code;
 	wc->wc_flags = 0;
 	wc->qp = &qp->ibqp;
 	wc->pkey_index = 0;
 
-	if (shadow_wqe->opcode == IB_WC_RECV) {
-		wc->byte_len = ud_wqe->byte_len;
-		wc->src_qp = ud_wqe->src_qpn;
+	if (opcode == IB_WC_RECV) {
+		wc->byte_len = shadow_wqe->byte_len;
+		wc->src_qp = shadow_wqe->src_qpn;
 		wc->wc_flags |= IB_WC_GRH;
 	}
 }
@@ -270,7 +269,8 @@ static int mana_process_completions(struct mana_ib_cq *cq, int nwc, struct ib_wc
 			if (wc_index >= nwc)
 				goto out;
 
-			fill_verbs_from_shadow_wqe(qp, &wc[wc_index], shadow_wqe);
+			fill_verbs_from_shadow_wqe(qp, &wc[wc_index], shadow_wqe,
+						   shadow_wqe->send_opcode);
 			shadow_queue_advance_consumer(&qp->shadow_sq);
 			wc_index++;
 		}
@@ -283,7 +283,8 @@ static int mana_process_completions(struct mana_ib_cq *cq, int nwc, struct ib_wc
 			if (wc_index >= nwc)
 				goto out;
 
-			fill_verbs_from_shadow_wqe(qp, &wc[wc_index], shadow_wqe);
+			fill_verbs_from_shadow_wqe(qp, &wc[wc_index], shadow_wqe,
+						   IB_WC_RECV);
 			shadow_queue_advance_consumer(&qp->shadow_rq);
 			wc_index++;
 		}
@@ -296,13 +297,13 @@ static int mana_process_completions(struct mana_ib_cq *cq, int nwc, struct ib_wc
 static void mana_drain_gsi_sq(struct mana_ib_qp *qp)
 {
 	struct mana_ib_cq *cq = container_of(qp->ibqp.send_cq, struct mana_ib_cq, ibcq);
-	struct ud_sq_shadow_wqe *shadow_wqe;
+	struct shadow_wqe_header *shadow_wqe;
 	unsigned long flags;
 
 	spin_lock_irqsave(&cq->cq_lock, flags);
 	while ((shadow_wqe = shadow_queue_get_next_to_complete(&qp->shadow_sq))
 			!= NULL) {
-		shadow_wqe->header.error_code = IB_WC_GENERAL_ERR;
+		shadow_wqe->error_code = IB_WC_GENERAL_ERR;
 		shadow_queue_advance_next_to_complete(&qp->shadow_sq);
 	}
 	spin_unlock_irqrestore(&cq->cq_lock, flags);
drivers/infiniband/hw/mana/qp.c +2 -2
index fac43b3a5..79c9cc944 100644
@@ -748,13 +748,13 @@ static int mana_ib_create_ud_qp(struct ib_qp *ibqp, struct ib_pd *ibpd,
 	doorbell = mdev->gdma_dev->doorbell;
 
 	err = create_shadow_queue(&qp->shadow_rq, attr->cap.max_recv_wr,
-				  sizeof(struct ud_rq_shadow_wqe));
+				  sizeof(struct shadow_wqe_header));
 	if (err) {
 		ibdev_err(&mdev->ib_dev, "Failed to create shadow rq err %d\n", err);
 		goto destroy_queues;
 	}
 	err = create_shadow_queue(&qp->shadow_sq, attr->cap.max_send_wr,
-				  sizeof(struct ud_sq_shadow_wqe));
+				  sizeof(struct shadow_wqe_header));
 	if (err) {
 		ibdev_err(&mdev->ib_dev, "Failed to create shadow sq err %d\n", err);
 		goto destroy_shadow_queues;
drivers/infiniband/hw/mana/shadow_queue.h +13 -16
index a4b3818f9..a127edf7a 100644
@@ -7,22 +7,14 @@
 #define _MANA_SHADOW_QUEUE_H_
 
 struct shadow_wqe_header {
-	u16 opcode;
-	u16 error_code;
-	u32 posted_wqe_size;
 	u64 wr_id;
-};
-
-struct ud_rq_shadow_wqe {
-	struct shadow_wqe_header header;
+	u64 wqe_size_in_bu : 8;
+	u64 send_opcode : 4;
+	u64 error_code : 16;
 	u32 byte_len;
 	u32 src_qpn;
 };
 
-struct ud_sq_shadow_wqe {
-	struct shadow_wqe_header header;
-};
-
 struct shadow_queue {
 	/* Unmasked producer index, Incremented on wqe posting */
 	u64 prod_idx;
@@ -57,12 +49,14 @@ static inline void destroy_shadow_queue(struct shadow_queue *queue)
 
 static inline bool shadow_queue_full(struct shadow_queue *queue)
 {
-	return (queue->prod_idx - queue->cons_idx) >= queue->length;
+	/* Do not reuse an entry until the poller has finished reading it. */
+	return (queue->prod_idx - smp_load_acquire(&queue->cons_idx)) >= queue->length;
 }
 
 static inline bool shadow_queue_empty(struct shadow_queue *queue)
 {
-	return queue->prod_idx == queue->cons_idx;
+	/* Pair with posting's release of the initialized shadow WQE. */
+	return smp_load_acquire(&queue->prod_idx) == queue->cons_idx;
 }
 
 static inline void *
@@ -91,7 +85,8 @@ shadow_queue_get_next_to_consume(const struct shadow_queue *queue)
 static inline void *
 shadow_queue_get_next_to_complete(struct shadow_queue *queue)
 {
-	if (queue->next_to_complete_idx == queue->prod_idx)
+	/* Observe initialized WQE fields published by the posting CPU. */
+	if (queue->next_to_complete_idx == smp_load_acquire(&queue->prod_idx))
 		return NULL;
 
 	return shadow_queue_get_element(queue, queue->next_to_complete_idx);
@@ -99,12 +94,14 @@ shadow_queue_get_next_to_complete(struct shadow_queue *queue)
 
 static inline void shadow_queue_advance_producer(struct shadow_queue *queue)
 {
-	queue->prod_idx++;
+	/* Publish all WQE fields to CQ polling on another CPU. */
+	smp_store_release(&queue->prod_idx, queue->prod_idx + 1);
 }
 
 static inline void shadow_queue_advance_consumer(struct shadow_queue *queue)
 {
-	queue->cons_idx++;
+	/* Finish WC generation and queue-tail updates before allowing reuse. */
+	smp_store_release(&queue->cons_idx, queue->cons_idx + 1);
 }
 
 static inline void shadow_queue_advance_next_to_complete(struct shadow_queue *queue)
drivers/infiniband/hw/mana/wr.c +7 -8
index 36a1d506f..8cd2980d4 100644
@@ -14,7 +14,7 @@ static int mana_ib_post_recv_ud(struct mana_ib_qp *qp, const struct ib_recv_wr *
 	struct gdma_posted_wqe_info wqe_info = {0};
 	struct gdma_sge gdma_sgl[MAX_WR_SGL_NUM];
 	struct gdma_wqe_request wqe_req = {0};
-	struct ud_rq_shadow_wqe *shadow_wqe;
+	struct shadow_wqe_header *shadow_wqe;
 	int err, i;
 
 	if (shadow_queue_full(&qp->shadow_rq))
@@ -37,9 +37,8 @@ static int mana_ib_post_recv_ud(struct mana_ib_qp *qp, const struct ib_recv_wr *
 
 	shadow_wqe = shadow_queue_producer_entry(&qp->shadow_rq);
 	memset(shadow_wqe, 0, sizeof(*shadow_wqe));
-	shadow_wqe->header.opcode = IB_WC_RECV;
-	shadow_wqe->header.wr_id = wr->wr_id;
-	shadow_wqe->header.posted_wqe_size = wqe_info.wqe_size_in_bu;
+	shadow_wqe->wr_id = wr->wr_id;
+	shadow_wqe->wqe_size_in_bu = wqe_info.wqe_size_in_bu;
 	shadow_queue_advance_producer(&qp->shadow_rq);
 
 	mana_gd_wq_ring_doorbell(mdev_to_gc(mdev), queue);
@@ -82,7 +81,7 @@ static int mana_ib_post_send_ud(struct mana_ib_qp *qp, const struct ib_ud_wr *wr
 	struct gdma_posted_wqe_info wqe_info = {0};
 	struct gdma_wqe_request wqe_req = {0};
 	struct rdma_send_oob send_oob = {0};
-	struct ud_sq_shadow_wqe *shadow_wqe;
+	struct shadow_wqe_header *shadow_wqe;
 	int err, i;
 
 	if (!ndev) {
@@ -132,9 +131,9 @@ static int mana_ib_post_send_ud(struct mana_ib_qp *qp, const struct ib_ud_wr *wr
 	qp->ud_qp.sq_psn++;
 	shadow_wqe = shadow_queue_producer_entry(&qp->shadow_sq);
 	memset(shadow_wqe, 0, sizeof(*shadow_wqe));
-	shadow_wqe->header.opcode = IB_WC_SEND;
-	shadow_wqe->header.wr_id = wr->wr.wr_id;
-	shadow_wqe->header.posted_wqe_size = wqe_info.wqe_size_in_bu;
+	shadow_wqe->send_opcode = IB_WC_SEND;
+	shadow_wqe->wr_id = wr->wr.wr_id;
+	shadow_wqe->wqe_size_in_bu = wqe_info.wqe_size_in_bu;
 	shadow_queue_advance_producer(&qp->shadow_sq);
 
 	mana_gd_wq_ring_doorbell(mdev_to_gc(mdev), queue);
Signature · 1 line
2.43.0
#2 lore raw

Re: [PATCH rdma-next v4 1/5] RDMA/mana_ib: Optimize shadow queue bookkeeping

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 10 lines

From: Konstantin Taranov <[email protected]>

Pack the posted WQE size and send opcode into compact bitfields. Move the UD receive shadow fields into the common shadow_wqe_header. This simplifies the code and lets all QP and WQ types share one structure.

Use release and acquire operations so posting and polling can safely run on different CPUs.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#3 lore raw

PATCH v4 2/5 RDMA/mana_ib: Revise UD send posting and WQE definitions

In reply to Konstantin Taranov

From: Konstantin Taranov <[email protected]>

Batch multiple send requests under a single doorbell. Move the send PSN to the common QP structure so it can be shared by all QP types. Define new WQE formats for UD and RC QPs.

---
 drivers/infiniband/hw/mana/mana_ib.h | 40 ++++++++++++++++++-
 drivers/infiniband/hw/mana/wr.c      | 58 ++++++++++++++++------------
 2 files changed, 72 insertions(+), 26 deletions(-)
drivers/infiniband/hw/mana/mana_ib.h +39 -1
index 4da110b30..619578f16 100644
@@ -206,7 +206,6 @@ enum mana_ud_queue_type {
 
 struct mana_ib_ud_qp {
 	struct mana_ib_queue queues[MANA_UD_QUEUE_TYPE_MAX];
-	u32 sq_psn;
 };
 
 struct mana_ib_qp {
@@ -222,6 +221,7 @@ struct mana_ib_qp {
 
 	/* The port on the IB device, starting with 1 */
 	u32 port;
+	u32 sq_psn;
 
 	struct list_head cq_send_list;
 	struct list_head cq_recv_list;
@@ -525,7 +525,16 @@ struct mana_rnic_set_qp_state_resp {
 
 enum WQE_OPCODE_TYPES {
 	WQE_TYPE_UD_SEND = 0,
+	WQE_TYPE_RC_SEND = 2,
+	WQE_TYPE_RC_SEND_IMM = 3,
+	WQE_TYPE_RC_SEND_INV = 4,
+	WQE_TYPE_WRITE = 5,
+	WQE_TYPE_WRITE_IMM = 6,
+	WQE_TYPE_READ = 7,
 	WQE_TYPE_UD_RECV = 8,
+	WQE_TYPE_RC_RECV = 9,
+	WQE_TYPE_REG_MR = 10,
+	WQE_TYPE_LOCAL_INV = 12,
 }; /* HW DATA */
 
 struct rdma_send_oob {
@@ -544,7 +553,36 @@ struct rdma_send_oob {
 			u32 reserved1;
 			u32 reserved2;
 		} ud_send;
+		union {
+			u32 immediate;
+			u32 invalidate_key;
+		} rc_send;
+		struct {
+			u32 address_hi;
+			u32 address_low;
+			u32 rkey;
+			u32 dma_len;
+		} rdma;
+		struct {
+			u32 mkey;
+		} mm;
 	};
+	union {
+		u32 immediate_ext;
+		struct {
+			u16 rsn;
+			u16 reserved;
+		} read;
+	};
+	u32 fsn : 24;
+	u32 reserved2   : 8;
+}; /* HW DATA */
+
+struct rdma_recv_oob {
+	u32 psn_start   : 24;
+	u32 reserved1   : 8;
+	u32 msn         : 24;
+	u32 reserved2   : 8;
 }; /* HW DATA */
 
 struct mana_rdma_cqe {
drivers/infiniband/hw/mana/wr.c +33 -25

105 lines not on this page · raw message

Signature · 1 line
2.43.0
#4 lore raw

Re: [PATCH rdma-next v4 2/5] RDMA/mana_ib: Revise UD send posting and WQE definitions

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 7 lines

From: Konstantin Taranov <[email protected]>

Batch multiple send requests under a single doorbell. Move the send PSN to the common QP structure so it can be shared by all QP types. Define new WQE formats for UD and RC QPs.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#5 lore raw

PATCH v4 3/5 RDMA/mana_ib: Revise UD receive posting with GDMA_WR_IB_SGL

In reply to Konstantin Taranov

From: Konstantin Taranov <[email protected]>

Decouple RQ WQE posting from the QP structure so it can be reused by other QP types. Batch multiple receive WQEs under a single doorbell.

Introduce GDMA_WR_IB_SGL to pass SGEs directly to GDMA instead of copying them into a small stack array. Support receive WQEs with num_sge set to zero.

---
 drivers/infiniband/hw/mana/wr.c               | 70 +++++++++++--------
 .../net/ethernet/microsoft/mana/gdma_main.c   | 23 +++++-
 include/net/mana/gdma.h                       |  7 +-
 3 files changed, 70 insertions(+), 30 deletions(-)
drivers/infiniband/hw/mana/wr.c +42 -28
index f25df2fad..eb671b45b 100644
@@ -7,29 +7,30 @@
 
 #define MAX_WR_SGL_NUM (2)
 
-static int mana_ib_post_recv_ud(struct mana_ib_qp *qp, const struct ib_recv_wr *wr)
+static int mana_ib_post_rq(struct mana_ib_qp *qp, const struct ib_recv_wr *wr)
 {
-	struct mana_ib_dev *mdev = container_of(qp->ibqp.device, struct mana_ib_dev, ib_dev);
-	struct gdma_queue *queue = qp->ud_qp.queues[MANA_UD_RECV_QUEUE].kmem;
+	struct ib_sge mana_ib_dummy_sge = {.addr = 1, .length = 0, .lkey = 0};
+	struct mana_ib_queue *ib_rq = mana_qp_get_rq(qp);
 	struct gdma_posted_wqe_info wqe_info = {0};
-	struct gdma_sge gdma_sgl[MAX_WR_SGL_NUM];
+	struct gdma_queue *queue = ib_rq->kmem;
 	struct gdma_wqe_request wqe_req = {0};
 	struct shadow_wqe_header *shadow_wqe;
-	int err, i;
+	int err;
 
 	if (shadow_queue_full(&qp->shadow_rq))
 		return -EINVAL;
 
-	if (wr->num_sge > MAX_WR_SGL_NUM)
+	if (wr->num_sge > MAX_RX_WQE_SGL_ENTRIES)
 		return -EINVAL;
 
-	for (i = 0; i < wr->num_sge; ++i) {
-		gdma_sgl[i].address = wr->sg_list[i].addr;
-		gdma_sgl[i].mem_key = wr->sg_list[i].lkey;
-		gdma_sgl[i].size = wr->sg_list[i].length;
-	}
 	wqe_req.num_sge = wr->num_sge;
-	wqe_req.sgl = gdma_sgl;
+	wqe_req.ib_sgl = wr->sg_list;
+	wqe_req.flags = GDMA_WR_IB_SGL;
+
+	if (wr->num_sge == 0) {
+		wqe_req.ib_sgl = &mana_ib_dummy_sge;
+		wqe_req.num_sge = 1;
+	}
 
 	err = mana_gd_post_work_request(queue, &wqe_req, &wqe_info);
 	if (err)
@@ -41,36 +42,49 @@ static int mana_ib_post_recv_ud(struct mana_ib_qp *qp, const struct ib_recv_wr *
 	shadow_wqe->wqe_size_in_bu = wqe_info.wqe_size_in_bu;
 	shadow_queue_advance_producer(&qp->shadow_rq);
 
-	mana_gd_wq_ring_doorbell(mdev_to_gc(mdev), queue);
 	return 0;
 }
 
-int mana_ib_post_recv(struct ib_qp *ibqp, const struct ib_recv_wr *wr,
-		      const struct ib_recv_wr **bad_wr)
+static int mana_ib_post_recv_ud(struct mana_ib_qp *qp, const struct ib_recv_wr *wr,
+				const struct ib_recv_wr **bad_wr)
 {
-	struct mana_ib_qp *qp = container_of(ibqp, struct mana_ib_qp, ibqp);
+	struct mana_ib_dev *mdev = container_of(qp->ibqp.device, struct mana_ib_dev, ib_dev);
+	struct mana_ib_queue *ib_rq = mana_qp_get_rq(qp);
+	struct gdma_queue *rq = ib_rq->kmem;
+	bool ring_rq = false;
 	int err = 0;
 
 	for (; wr; wr = wr->next) {
-		switch (ibqp->qp_type) {
-		case IB_QPT_UD:
-		case IB_QPT_GSI:
-			err = mana_ib_post_recv_ud(qp, wr);
-			if (unlikely(err)) {
-				*bad_wr = wr;
-				return err;
-			}
+		err = mana_ib_post_rq(qp, wr);
+		if (unlikely(err)) {
+			*bad_wr = wr;
 			break;
-		default:
-			ibdev_dbg(ibqp->device, "Posting recv wr on qp type %u is not supported\n",
-				  ibqp->qp_type);
-			return -EINVAL;
 		}
+		ring_rq = true;
 	}
 
+	if (ring_rq)
+		mana_gd_wq_ring_doorbell(mdev_to_gc(mdev), rq);
+
 	return err;
 }
 
+int mana_ib_post_recv(struct ib_qp *ibqp, const struct ib_recv_wr *wr,
+		      const struct ib_recv_wr **bad_wr)
+{
+	struct mana_ib_qp *qp = container_of(ibqp, struct mana_ib_qp, ibqp);
+
+	switch (ibqp->qp_type) {
+	case IB_QPT_UD:
+	case IB_QPT_GSI:
+		return mana_ib_post_recv_ud(qp, wr, bad_wr);
+	default:
+		/* Unsupported QP type */
+		*bad_wr = wr;
+		return -EINVAL;
+	}
+}
+
 static int mana_ib_post_send_ud_one(struct mana_ib_qp *qp, const struct ib_ud_wr *wr)
 {
 	struct mana_ib_dev *mdev = container_of(qp->ibqp.device, struct mana_ib_dev, ib_dev);
drivers/net/ethernet/microsoft/mana/gdma_main.c +22 -1
index f92b2d0bf..542491a6c 100644
@@ -1738,6 +1738,24 @@ static void mana_gd_write_sgl(struct gdma_queue *wq, u32 sgl_offset,
 	memcpy(mana_gd_ring_ptr(wq, sgl_offset), address, sgl_size);
 }
 
+static void mana_gd_write_ib_sgl(struct gdma_queue *wq, u32 sgl_offset,
+				 const struct gdma_wqe_request *wqe_req)
+{
+	const struct ib_sge *sge = wqe_req->ib_sgl;
+	struct gdma_sge *gdma_sgl;
+	u32 i;
+
+	for (i = 0; i < wqe_req->num_sge; ++i, ++sge) {
+		gdma_sgl = mana_gd_ring_ptr(wq, sgl_offset);
+		gdma_sgl->address = sge->addr;
+		gdma_sgl->size = sge->length;
+		gdma_sgl->mem_key = sge->lkey;
+		sgl_offset += sizeof(*gdma_sgl);
+		if (sgl_offset == wq->queue_size)
+			sgl_offset = 0;
+	}
+}
+
 int mana_gd_post_work_request(struct gdma_queue *wq,
 			      const struct gdma_wqe_request *wqe_req,
 			      struct gdma_posted_wqe_info *wqe_info)
@@ -1792,7 +1810,10 @@ int mana_gd_post_work_request(struct gdma_queue *wq,
 	if (sgl_offset >= wq->queue_size)
 		sgl_offset -= wq->queue_size;
 
-	mana_gd_write_sgl(wq, sgl_offset, wqe_req);
+	if (wqe_req->flags & GDMA_WR_IB_SGL)
+		mana_gd_write_ib_sgl(wq, sgl_offset, wqe_req);
+	else
+		mana_gd_write_sgl(wq, sgl_offset, wqe_req);
 
 	wq->head += wqe_size / GDMA_WQE_BU_SIZE;
 
include/net/mana/gdma.h +6 -1
index 06af65d50..8ccbe2d57 100644
@@ -6,6 +6,7 @@
 
 #include <linux/dma-mapping.h>
 #include <linux/netdevice.h>
+#include <rdma/ib_verbs.h>
 
 #include "shm_channel.h"
 
@@ -54,6 +55,7 @@ enum gdma_work_request_flags {
 	GDMA_WR_NONE			= 0,
 	GDMA_WR_OOB_IN_SGL		= BIT(0),
 	GDMA_WR_PAD_BY_SGE0		= BIT(1),
+	GDMA_WR_IB_SGL			= BIT(2),
 };
 
 enum gdma_eqe_type {
@@ -212,7 +214,10 @@ struct gdma_sge {
 }; /* HW DATA */
 
 struct gdma_wqe_request {
-	struct gdma_sge *sgl;
+	union {
+		struct gdma_sge *sgl;
+		struct ib_sge *ib_sgl;
+	};
 	u32 num_sge;
 
 	u32 inline_oob_size;
Signature · 1 line
2.43.0
#6 lore raw

Re: [PATCH rdma-next v4 3/5] RDMA/mana_ib: Revise UD receive posting with GDMA_WR_IB_SGL

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 10 lines

From: Konstantin Taranov <[email protected]>

Decouple RQ WQE posting from the QP structure so it can be reused by other QP types. Batch multiple receive WQEs under a single doorbell.

Introduce GDMA_WR_IB_SGL to pass SGEs directly to GDMA instead of copying them into a small stack array. Support receive WQEs with num_sge set to zero.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
#7 lore raw

PATCH v4 4/5 RDMA/mana_ib: Make kernel CQ arming robust

In reply to Konstantin Taranov

From: Konstantin Taranov <[email protected]>

Track CQ polling credits and ring the doorbell promptly to advance the hardware CQ tail pointer. The distance between the new and previously armed tails must remain below four times the CQ length.

Expose CQ arming to mana_ib through mana_gd_wq_ring_doorbell_ext(). RC QPs will also use this helper to ring special doorbell offsets.

MANA hardware ignores duplicate doorbells. Calculate the arming offset from the previously armed offset and poll_credit to avoid duplicates. Do not arm the CQ if no CQE was produced, because the CQ remains armed.

---
 drivers/infiniband/hw/mana/cq.c               | 55 ++++++++++++++++++-
 drivers/infiniband/hw/mana/mana_ib.h          |  1 +
 .../net/ethernet/microsoft/mana/gdma_main.c   | 20 +++++--
 include/net/mana/gdma.h                       |  2 +
 4 files changed, 72 insertions(+), 6 deletions(-)
drivers/infiniband/hw/mana/cq.c +54 -1
index 14a48409e..6764e2de0 100644
@@ -64,6 +64,9 @@ int mana_ib_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
 		doorbell = mdev->gdma_dev->doorbell;
 	}
 
+	ibcq->cqe = cq->cqe;
+	cq->poll_credit = (cq->cqe << (GDMA_CQE_OWNER_BITS - 1)) - 1;
+
 	if (is_rnic_cq) {
 		err = mana_ib_gd_create_cq(mdev, cq, doorbell);
 		if (err) {
@@ -174,15 +177,62 @@ void mana_ib_remove_cq_cb(struct mana_ib_dev *mdev, struct mana_ib_cq *cq)
 	gc->cq_table[cq->queue.id] = NULL;
 }
 
+static inline bool gdma_cq_idx_produced(struct gdma_queue *gdma_cq, uint32_t idx)
+{
+	struct gdma_mem_info *gmi = &gdma_cq->mem_info;
+	u32 num_cqe = gdma_cq->queue_size / GDMA_CQE_SIZE;
+	u32 expected_bits = (idx / num_cqe) & GDMA_CQE_OWNER_MASK;
+	u32 offset = (idx % num_cqe) * GDMA_CQE_SIZE;
+	struct gdma_cqe *cqe;
+
+	if (gmi->nr_pages)
+		cqe = gmi->pages_va[offset / PAGE_SIZE] +
+		      (offset & (PAGE_SIZE - 1));
+	else
+		cqe = gdma_cq->queue_mem_ptr + offset;
+
+	return cqe->cqe_info.owner_bits == expected_bits;
+}
+
+static inline void mana_ib_cq_doorbell(struct mana_ib_cq *cq, uint8_t arm)
+{
+	struct mana_ib_dev *mdev = container_of(cq->ibcq.device, struct mana_ib_dev, ib_dev);
+	struct gdma_queue *gdma_cq = cq->queue.kmem;
+	u32 num_cqe, max_credit, idx;
+
+	num_cqe = gdma_cq->queue_size / GDMA_CQE_SIZE;
+	max_credit = num_cqe << (GDMA_CQE_OWNER_BITS - 1);
+	idx = gdma_cq->head;
+
+	if (cq->poll_credit >= max_credit) {
+		if (gdma_cq_idx_produced(gdma_cq, idx + cq->poll_credit - max_credit))
+			cq->poll_credit++;
+		else
+			return;
+	} else {
+		/* Set index of already polled CQE for unarm */
+		cq->poll_credit = max_credit - (arm ? 0 : 1);
+	}
+
+	idx += (cq->poll_credit - max_credit);
+	idx %= (num_cqe << GDMA_CQE_OWNER_BITS);
+
+	mana_gd_wq_ring_doorbell_ext(mdev_to_gc(mdev), gdma_cq, idx, arm, 0);
+}
+
 int mana_ib_arm_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags flags)
 {
 	struct mana_ib_cq *cq = container_of(ibcq, struct mana_ib_cq, ibcq);
 	struct gdma_queue *gdma_cq = cq->queue.kmem;
+	unsigned long irq_flags;
 
 	if (!gdma_cq)
 		return -EINVAL;
 
-	mana_gd_ring_cq(gdma_cq, SET_ARM_BIT);
+	spin_lock_irqsave(&cq->cq_lock, irq_flags);
+	mana_ib_cq_doorbell(cq, SET_ARM_BIT);
+	spin_unlock_irqrestore(&cq->cq_lock, irq_flags);
+
 	return 0;
 }
 
@@ -343,6 +393,9 @@ int mana_ib_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc)
 		comp_read = mana_gd_poll_cq(queue, &gdma_cqe, 1);
 		if (comp_read < 1)
 			break;
+		cq->poll_credit--;
+		if (!cq->poll_credit)
+			mana_ib_cq_doorbell(cq, 0);
 		mana_handle_cqe(mdev, &gdma_cqe);
 	}
 
drivers/infiniband/hw/mana/mana_ib.h +1 -0
index 619578f16..0ad757990 100644
@@ -170,6 +170,7 @@ struct mana_ib_cq {
 	struct list_head list_recv_qp;
 	int cqe;
 	u32 comp_vector;
+	u32 poll_credit;
 	mana_handle_t  cq_handle;
 };
 
drivers/net/ethernet/microsoft/mana/gdma_main.c +15 -5
index 542491a6c..cb51fcf57 100644
@@ -559,11 +559,13 @@ static int mana_gd_disable_queue(struct gdma_queue *queue)
 
 static void mana_gd_ring_doorbell(struct gdma_context *gc, u32 db_index,
 				  enum gdma_queue_type q_type, u32 qid,
-				  u32 tail_ptr, u8 num_req)
+				  u32 tail_ptr, u8 num_req, u8 client_offset)
 {
 	void __iomem *addr = gc->db_page_base + gc->db_page_size * db_index;
 	union gdma_doorbell_entry e = {};
 
+	addr += client_offset;
+
 	switch (q_type) {
 	case GDMA_EQ:
 		e.eq.id = qid;
@@ -623,10 +625,18 @@ void mana_gd_wq_ring_doorbell(struct gdma_context *gc, struct gdma_queue *queue)
 	 * wqe_cnt for Receive Queues. This value is not used in Send Queues.
 	 */
 	mana_gd_ring_doorbell(gc, queue->gdma_dev->doorbell, queue->type,
-			      queue->id, queue->head * GDMA_WQE_BU_SIZE, 0);
+			      queue->id, queue->head * GDMA_WQE_BU_SIZE, 0, 0);
 }
 EXPORT_SYMBOL_NS(mana_gd_wq_ring_doorbell, "NET_MANA");
 
+void mana_gd_wq_ring_doorbell_ext(struct gdma_context *gc, struct gdma_queue *queue,
+				  u32 tail_ptr, u8 wqe_cnt, u8 client_offset)
+{
+	mana_gd_ring_doorbell(gc, queue->gdma_dev->doorbell, queue->type,
+			      queue->id, tail_ptr, wqe_cnt, client_offset);
+}
+EXPORT_SYMBOL_NS(mana_gd_wq_ring_doorbell_ext, "NET_MANA");
+
 void mana_gd_ring_cq(struct gdma_queue *cq, u8 arm_bit)
 {
 	struct gdma_context *gc = cq->gdma_dev->gdma_context;
@@ -636,7 +646,7 @@ void mana_gd_ring_cq(struct gdma_queue *cq, u8 arm_bit)
 	u32 head = cq->head % (num_cqe << GDMA_CQE_OWNER_BITS);
 
 	mana_gd_ring_doorbell(gc, cq->gdma_dev->doorbell, cq->type, cq->id,
-			      head, arm_bit);
+			      head, arm_bit, 0);
 }
 EXPORT_SYMBOL_NS(mana_gd_ring_cq, "NET_MANA");
 
@@ -653,7 +663,7 @@ void mana_gd_ring_dim(struct gdma_queue *cq, u32 mod_usec, bool mod_usec_vld,
 		dim_val |= MANA_INTR_MODR_USEC_VLD;
 
 	mana_gd_ring_doorbell(gc, cq->gdma_dev->doorbell, GDMA_DIM, cq->id,
-			      dim_val, mod_comps_vld);
+			      dim_val, mod_comps_vld, 0);
 }
 EXPORT_SYMBOL_NS(mana_gd_ring_dim, "NET_MANA");
 
@@ -1023,7 +1033,7 @@ static void mana_gd_process_eq_events(void *arg)
 	head = eq->head % (num_eqe << GDMA_EQE_OWNER_BITS);
 
 	mana_gd_ring_doorbell(gc, eq->gdma_dev->doorbell, eq->type, eq->id,
-			      head, SET_ARM_BIT);
+			      head, SET_ARM_BIT, 0);
 }
 
 static int mana_gd_register_irq(struct gdma_queue *queue,
include/net/mana/gdma.h +2 -0
index 8ccbe2d57..02632acd7 100644
@@ -1068,6 +1068,8 @@ void mana_gd_free_res_map(struct gdma_resource *r);
 
 void mana_gd_wq_ring_doorbell(struct gdma_context *gc,
 			      struct gdma_queue *queue);
+void mana_gd_wq_ring_doorbell_ext(struct gdma_context *gc, struct gdma_queue *queue,
+				  u32 tail_ptr, u8 wqe_cnt, u8 client_offset);
 
 int mana_gd_alloc_memory(struct gdma_context *gc, unsigned int length,
 			 struct gdma_mem_info *gmi, bool allow_scatter);
Signature · 1 line
2.43.0
#8 lore raw

Re: [PATCH rdma-next v4 4/5] RDMA/mana_ib: Make kernel CQ arming robust

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 6 lines

From: Konstantin Taranov <[email protected]>

Track CQ polling credits and ring the doorbell promptly to advance the hardware CQ tail pointer. The distance between the new and previously armed tails must remain below four times the CQ length.

Quoted 8 lines

Expose CQ arming to mana_ib through mana_gd_wq_ring_doorbell_ext(). RC QPs will also use this helper to ring special doorbell offsets.

MANA hardware ignores duplicate doorbells. Calculate the arming offset from the previously armed offset and poll_credit to avoid duplicates. Do not arm the CQ if no CQE was produced, because the CQ remains armed.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
#9 lore raw

PATCH v4 5/5 RDMA/mana_ib: Poll UD completions and flush software error QPs

In reply to Konstantin Taranov

From: Konstantin Taranov <[email protected]>

Build WCs directly from CQEs and their corresponding published shadow entries. This avoids iterating over all QPs that share a CQ. Track only QPs in the error state instead of every QP associated with the CQ.

Keep the explicit GSI send-drain path responsible for adding the QP to the send error list and invoking its CQ handler. To invoke the handler for WQEs posted after GSI draining, check the QP state after posting WQEs. This is required because MANA hardware ignores doorbells for QPs in the error state.

Introduce a WC-budgeted polling context that caches CQEs representing multiple WCs.

Before consuming a shadow entry, match the CQE against the low 24 bits of the posted WQE offset to filter out stale CQEs. Implement WR signaling to skip WC generation for unsignaled WQEs.

---
Changes in v3:
- Honor IB_SIGNAL_ALL_WR and set MANA_WQ_NO_SIGNAL_WC for unsignaled sends.
- Find the matching CQE target before retiring its unsignaled prefix and the target with a wrap-safe inclusive loop, preserving stale-CQE checks.
- Notify the send CQ after posting to an already-flushing QP, outside the CQ lock, so later sends are flushed without another hardware CQE.

Changes in v2:
- Serialize each shadow queue reset and error-list removal with CQ polling and software flushing by holding the corresponding CQ lock.
- Return the unmasked WQE start offset from GDMA posting to ignore CQEs with mismatched low 24-bit WQE offsets.

 drivers/infiniband/hw/mana/cq.c               | 298 ++++++++++++------
 drivers/infiniband/hw/mana/mana_ib.h          |  64 +++-
 drivers/infiniband/hw/mana/qp.c               |  80 +++--
 drivers/infiniband/hw/mana/shadow_queue.h     |  38 +--
 drivers/infiniband/hw/mana/wr.c               |  19 +-
 .../net/ethernet/microsoft/mana/gdma_main.c   |   6 +-
 include/net/mana/gdma.h                       |   2 +
 7 files changed, 359 insertions(+), 148 deletions(-)
drivers/infiniband/hw/mana/cq.c +205 -93
index 6764e2de0..3b8044b7d 100644
@@ -5,6 +5,58 @@
 
 #include "mana_ib.h"
 
+static enum ib_wc_status vendor_error_to_wc_error(uint32_t vendor_error)
+{
+	switch (vendor_error) {
+	case VENDOR_ERR_OK:
+		return IB_WC_SUCCESS;
+	case VENDOR_ERR_RX_PKT_LEN:
+	case VENDOR_ERR_RX_MSG_LEN_OVFL:
+		return IB_WC_LOC_LEN_ERR;
+	case VENDOR_ERR_TX_GDMA_CORRUPTED_WQE:
+	case VENDOR_ERR_TX_PCIE_WQE:
+	case VENDOR_ERR_TX_PCIE_MSG:
+	case VENDOR_ERR_RX_MALFORMED_WQE:
+	case VENDOR_ERR_TX_GDMA_INVALID_STATE:
+	case VENDOR_ERR_TX_MISBEHAVING_CLIENT:
+	case VENDOR_ERR_TX_RDMA_MALFORMED_WQE_SIZE:
+	case VENDOR_ERR_TX_RDMA_MALFORMED_WQE_FIELD:
+	case VENDOR_ERR_TX_RDMA_WQE_UNSUPPORTED:
+	case VENDOR_ERR_TX_RDMA_WQE_LEN_ERR:
+	case VENDOR_ERR_TX_RDMA_MTU_ERR:
+		return IB_WC_LOC_QP_OP_ERR;
+	case VENDOR_ERR_TX_ATB_MSG_ACCESS_VIOLATION:
+	case VENDOR_ERR_TX_ATB_MSG_ADDR_RANGE:
+	case VENDOR_ERR_TX_ATB_MSG_CONFIG_ERR:
+	case VENDOR_ERR_TX_ATB_WQE_ACCESS_VIOLATION:
+	case VENDOR_ERR_TX_ATB_WQE_ADDR_RANGE:
+	case VENDOR_ERR_TX_ATB_WQE_CONFIG_ERR:
+	case VENDOR_ERR_RX_ATB_SGE_ADDR_RANGE:
+	case VENDOR_ERR_RX_ATB_SGE_MISSCONFIG:
+		return IB_WC_LOC_PROT_ERR;
+	case VENDOR_ERR_RX_ATB_SGE_ADDR_RIGHT:
+	case VENDOR_ERR_RX_GFID:
+		return IB_WC_LOC_ACCESS_ERR;
+	case VENDOR_ERR_RX_MISBEHAVING_CLIENT:
+	case VENDOR_ERR_RX_CLIENT_ID:
+	case VENDOR_ERR_RX_PCIE:
+	case VENDOR_ERR_RX_NO_AVAIL_WQE:
+	case VENDOR_ERR_RX_ATB_WQE_MISCONFIG:
+	case VENDOR_ERR_RX_ATB_WQE_ADDR_RIGHT:
+	case VENDOR_ERR_RX_ATB_WQE_ADDR_RANGE:
+	case VENDOR_ERR_TX_RDMA_INVALID_STATE:
+	case VENDOR_ERR_TX_RDMA_INVALID_NPT:
+	case VENDOR_ERR_TX_RDMA_INVALID_SGID:
+	case VENDOR_ERR_TX_RDMA_VFID_MISMATCH:
+		return IB_WC_FATAL_ERR;
+	case VENDOR_ERR_RX_NOT_EMPTY_ON_DISABLE:
+	case VENDOR_ERR_SW_FLUSHED:
+		return IB_WC_WR_FLUSH_ERR;
+	default:
+		return IB_WC_GENERAL_ERR;
+	}
+}
+
 int mana_ib_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
 		      struct uverbs_attr_bundle *attrs)
 {
@@ -89,8 +141,8 @@ int mana_ib_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
 	}
 
 	spin_lock_init(&cq->cq_lock);
-	INIT_LIST_HEAD(&cq->list_send_qp);
-	INIT_LIST_HEAD(&cq->list_recv_qp);
+	INIT_LIST_HEAD(&cq->send_err_qp_list);
+	INIT_LIST_HEAD(&cq->recv_err_qp_list);
 
 	return 0;
 
@@ -236,126 +288,172 @@ int mana_ib_arm_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags flags)
 	return 0;
 }
 
-static inline void handle_ud_sq_cqe(struct mana_ib_qp *qp, struct gdma_comp *cqe)
-{
-	struct mana_rdma_cqe *rdma_cqe = (struct mana_rdma_cqe *)cqe->cqe_data;
-	struct gdma_queue *wq = qp->ud_qp.queues[MANA_UD_SEND_QUEUE].kmem;
-	struct shadow_wqe_header *shadow_wqe;
+struct mana_cq_poll {
+	struct ib_wc *wc;
+	int budget;
+	int produced;
+};
 
-	shadow_wqe = shadow_queue_get_next_to_complete(&qp->shadow_sq);
-	if (!shadow_wqe)
-		return;
+static struct ib_wc *mana_fill_wc(struct mana_ib_qp *qp,
+				  struct mana_cq_poll *poll,
+				  const struct shadow_wqe_header *wqe,
+				  enum ib_wc_opcode opcode, u32 vendor_error)
+{
+	struct ib_wc *wc = &poll->wc[poll->produced++];
 
-	shadow_wqe->error_code = rdma_cqe->ud_send.vendor_error;
+	memset(wc, 0, sizeof(*wc));
+	wc->wr_id = wqe->wr_id;
+	wc->status = vendor_error_to_wc_error(vendor_error);
+	wc->opcode = opcode;
+	wc->vendor_err = vendor_error;
+	wc->qp = &qp->ibqp;
 
-	wq->tail += shadow_wqe->wqe_size_in_bu;
-	shadow_queue_advance_next_to_complete(&qp->shadow_sq);
+	return wc;
 }
 
-static inline void handle_ud_rq_cqe(struct mana_ib_qp *qp, struct gdma_comp *cqe)
+static void mana_complete_send(struct mana_ib_qp *qp,
+			       struct mana_cq_poll *poll, u32 vendor_error)
 {
-	struct mana_rdma_cqe *rdma_cqe = (struct mana_rdma_cqe *)cqe->cqe_data;
-	struct gdma_queue *wq = qp->ud_qp.queues[MANA_UD_RECV_QUEUE].kmem;
-	struct shadow_wqe_header *shadow_wqe;
+	struct shadow_queue *shadow = &qp->shadow_sq;
+	struct shadow_wqe_header *wqe = shadow_queue_get_next_to_consume(shadow);
+	struct gdma_queue *queue;
 
-	shadow_wqe = shadow_queue_get_next_to_complete(&qp->shadow_rq);
-	if (!shadow_wqe)
+	if (!wqe)
 		return;
 
-	shadow_wqe->byte_len = rdma_cqe->ud_recv.msg_len;
-	shadow_wqe->src_qpn = rdma_cqe->ud_recv.src_qpn;
-	shadow_wqe->error_code = IB_WC_SUCCESS;
+	if (vendor_error || !(wqe->flags & MANA_WQ_NO_SIGNAL_WC))
+		mana_fill_wc(qp, poll, wqe, wqe->send_opcode, vendor_error);
 
-	wq->tail += shadow_wqe->wqe_size_in_bu;
-	shadow_queue_advance_next_to_complete(&qp->shadow_rq);
+	queue = mana_qp_get_sq(qp)->kmem;
+	queue->tail += wqe->wqe_size_in_bu;
+	shadow_queue_advance_consumer(shadow);
 }
 
-static void mana_handle_cqe(struct mana_ib_dev *mdev, struct gdma_comp *cqe)
+static void handle_ud_sq_cqe(struct mana_ib_qp *qp, struct mana_rdma_cqe *rdma_cqe,
+			     struct mana_cq_poll *poll)
 {
-	struct mana_ib_qp *qp = mana_get_qp_ref(mdev, cqe->wq_num, cqe->is_sq);
+	u32 offset = rdma_cqe->ud_send.tx_wqe_offset & MANA_WQE_OFFSET_MASK;
+	struct shadow_queue *shadow = &qp->shadow_sq;
+	struct shadow_wqe_header *wqe;
+	u64 idx = shadow->cons_idx;
+	u32 to_complete = 0;
+	u64 prod_idx;
+
+	/* Pair with posting's release of the initialized shadow entries. */
+	prod_idx = smp_load_acquire(&shadow->prod_idx);
+	/* Find the target before retiring any entries: the CQE may be stale. */
+	for (; idx != prod_idx; idx++) {
+		wqe = shadow_queue_get_element(shadow, idx);
+		to_complete++;
+		if (wqe->wqe_offset_or_psn == offset)
+			break;
+		if (!(wqe->flags & MANA_WQ_NO_SIGNAL_WC))
+			return;
+	}
+	if (idx == prod_idx)
+		return;
 
-	if (!qp)
+	for (; to_complete; to_complete--)
+		mana_complete_send(qp, poll, VENDOR_ERR_OK);
+}
+
+static void handle_rq_cqe(struct mana_ib_qp *qp, struct gdma_comp *cqe,
+			  struct mana_cq_poll *poll)
+{
+	struct mana_rdma_cqe *rdma_cqe = (struct mana_rdma_cqe *)cqe->cqe_data;
+	u32 offset = rdma_cqe->ud_recv.rx_wqe_offset / GDMA_WQE_BU_SIZE;
+	struct mana_ib_queue *rq = mana_qp_get_rq(qp);
+	struct gdma_queue *wq = rq->kmem;
+	struct shadow_wqe_header *wqe;
+	struct ib_wc *wc;
+
+	wqe = shadow_queue_get_next_to_consume(&qp->shadow_rq);
+	if (!wqe || wqe->wqe_offset_or_psn != (offset & MANA_WQE_OFFSET_MASK))
 		return;
 
-	if (qp->ibqp.qp_type == IB_QPT_GSI || qp->ibqp.qp_type == IB_QPT_UD) {
-		if (cqe->is_sq)
-			handle_ud_sq_cqe(qp, cqe);
-		else
-			handle_ud_rq_cqe(qp, cqe);
+	wc = mana_fill_wc(qp, poll, wqe, IB_WC_RECV, VENDOR_ERR_OK);
+	switch (rdma_cqe->cqe_type) {
+	case CQE_TYPE_UD_SEND_IMM:
+		wc->ex.imm_data = cpu_to_be32(rdma_cqe->ud_recv.imm_data);
+		wc->wc_flags |= IB_WC_WITH_IMM;
+		fallthrough;
+	case CQE_TYPE_UD_SEND:
+		wc->byte_len = rdma_cqe->ud_recv.msg_len;
+		wc->src_qp = rdma_cqe->ud_recv.src_qpn;
+		wc->wc_flags |= IB_WC_GRH;
+		break;
+	default:
+		break;
 	}
 
-	mana_put_qp_ref(qp);
+	wq->tail += wqe->wqe_size_in_bu;
+	shadow_queue_advance_consumer(&qp->shadow_rq);
 }
 
-static void fill_verbs_from_shadow_wqe(struct mana_ib_qp *qp, struct ib_wc *wc,
-				       const struct shadow_wqe_header *shadow_wqe,
-				       enum ib_wc_opcode opcode)
+static bool mana_handle_cqe(struct mana_ib_cq *cq, struct mana_ib_dev *mdev,
+			    struct mana_cq_poll *poll)
 {
-	wc->wr_id = shadow_wqe->wr_id;
-	wc->status = shadow_wqe->error_code;
-	wc->opcode = opcode;
-	wc->vendor_err = shadow_wqe->error_code;
-	wc->wc_flags = 0;
-	wc->qp = &qp->ibqp;
-	wc->pkey_index = 0;
+	struct gdma_comp *cqe = &cq->pending_cqe;
+	struct mana_rdma_cqe *rdma_cqe = (struct mana_rdma_cqe *)cqe->cqe_data;
+	struct mana_ib_qp *qp = mana_get_qp_ref(mdev, cqe->wq_num, cqe->is_sq);
 
-	if (opcode == IB_WC_RECV) {
-		wc->byte_len = shadow_wqe->byte_len;
-		wc->src_qp = shadow_wqe->src_qpn;
-		wc->wc_flags |= IB_WC_GRH;
+	if (!qp)
+		return true;
+
+	switch (rdma_cqe->cqe_type) {
+	case CQE_TYPE_UD_SEND:
+		if (cqe->is_sq) {
+			handle_ud_sq_cqe(qp, rdma_cqe, poll);
+			break;
+		}
+		fallthrough;
+	case CQE_TYPE_UD_SEND_IMM:
+		handle_rq_cqe(qp, cqe, poll);
+		break;
+	default:
+		ibdev_warn_ratelimited(qp->ibqp.device, "Unexpected CQE type %u\n",
+				       rdma_cqe->cqe_type);
+		break;
 	}
+	mana_put_qp_ref(qp);
+	return true;
 }
 
-static int mana_process_completions(struct mana_ib_cq *cq, int nwc, struct ib_wc *wc)
+static void mana_flush_completions(struct mana_ib_cq *cq, struct mana_cq_poll *poll)
 {
-	struct shadow_wqe_header *shadow_wqe;
+	struct shadow_wqe_header *wqe;
 	struct mana_ib_qp *qp;
-	int wc_index = 0;
 
-	/* process send shadow queue completions  */
-	list_for_each_entry(qp, &cq->list_send_qp, cq_send_list) {
-		while ((shadow_wqe = shadow_queue_get_next_to_consume(&qp->shadow_sq))
-				!= NULL) {
-			if (wc_index >= nwc)
-				goto out;
+	if (poll->produced >= poll->budget)
+		return;
 
-			fill_verbs_from_shadow_wqe(qp, &wc[wc_index], shadow_wqe,
-						   shadow_wqe->send_opcode);
-			shadow_queue_advance_consumer(&qp->shadow_sq);
-			wc_index++;
-		}
+	list_for_each_entry(qp, &cq->send_err_qp_list, send_err_node) {
+		while (poll->produced < poll->budget &&
+		       shadow_queue_get_next_to_consume(&qp->shadow_sq))
+			mana_complete_send(qp, poll, VENDOR_ERR_SW_FLUSHED);
+		if (poll->produced == poll->budget)
+			return;
 	}
 
-	/* process recv shadow queue completions */
-	list_for_each_entry(qp, &cq->list_recv_qp, cq_recv_list) {
-		while ((shadow_wqe = shadow_queue_get_next_to_consume(&qp->shadow_rq))
-				!= NULL) {
-			if (wc_index >= nwc)
-				goto out;
-
-			fill_verbs_from_shadow_wqe(qp, &wc[wc_index], shadow_wqe,
-						   IB_WC_RECV);
+	list_for_each_entry(qp, &cq->recv_err_qp_list, recv_err_node) {
+		while (poll->produced < poll->budget &&
+		       (wqe = shadow_queue_get_next_to_consume(&qp->shadow_rq))) {
+			mana_fill_wc(qp, poll, wqe, IB_WC_RECV, VENDOR_ERR_SW_FLUSHED);
 			shadow_queue_advance_consumer(&qp->shadow_rq);
-			wc_index++;
 		}
+		if (poll->produced == poll->budget)
+			return;
 	}
-
-out:
-	return wc_index;
 }
 
 static void mana_drain_gsi_sq(struct mana_ib_qp *qp)
 {
 	struct mana_ib_cq *cq = container_of(qp->ibqp.send_cq, struct mana_ib_cq, ibcq);
-	struct shadow_wqe_header *shadow_wqe;
 	unsigned long flags;
 
 	spin_lock_irqsave(&cq->cq_lock, flags);
-	while ((shadow_wqe = shadow_queue_get_next_to_complete(&qp->shadow_sq))
-			!= NULL) {
-		shadow_wqe->error_code = IB_WC_GENERAL_ERR;
-		shadow_queue_advance_next_to_complete(&qp->shadow_sq);
-	}
+	if (list_empty(&qp->send_err_node))
+		list_add_tail(&qp->send_err_node, &cq->send_err_qp_list);
 	spin_unlock_irqrestore(&cq->cq_lock, flags);
 
 	if (cq->ibcq.comp_handler)
@@ -382,25 +480,39 @@ int mana_ib_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc)
 {
 	struct mana_ib_cq *cq = container_of(ibcq, struct mana_ib_cq, ibcq);
 	struct mana_ib_dev *mdev = container_of(ibcq->device, struct mana_ib_dev, ib_dev);
+	struct mana_cq_poll poll = { .wc = wc, .budget = num_entries, .produced = 0 };
 	struct gdma_queue *queue = cq->queue.kmem;
-	struct gdma_comp gdma_cqe;
 	unsigned long flags;
-	int num_polled = 0;
-	int comp_read, i;
+	bool consumed;
+	int comp_read;
+
+	if (!queue)
+		return -EINVAL;
 
 	spin_lock_irqsave(&cq->cq_lock, flags);
-	for (i = 0; i < num_entries; i++) {
-		comp_read = mana_gd_poll_cq(queue, &gdma_cqe, 1);
-		if (comp_read < 1)
-			break;
-		cq->poll_credit--;
-		if (!cq->poll_credit)
-			mana_ib_cq_doorbell(cq, 0);
-		mana_handle_cqe(mdev, &gdma_cqe);
+	while (poll.produced < poll.budget) {
+		if (!cq->has_pending_cqe) {
+			comp_read = mana_gd_poll_cq(queue, &cq->pending_cqe, 1);
+			if (comp_read < 0) {
+				if (!poll.produced)
+					poll.produced = comp_read;
+				goto out;
+			}
+			if (!comp_read)
+				break;
+
+			cq->poll_credit--;
+			if (!cq->poll_credit)
+				mana_ib_cq_doorbell(cq, 0);
+		}
+
+		consumed = mana_handle_cqe(cq, mdev, &poll);
+		cq->has_pending_cqe = !consumed;
 	}
 
-	num_polled = mana_process_completions(cq, num_entries, wc);
+	mana_flush_completions(cq, &poll);
+out:
 	spin_unlock_irqrestore(&cq->cq_lock, flags);
 
-	return num_polled;
+	return poll.produced;
 }
drivers/infiniband/hw/mana/mana_ib.h +58 -6

89 lines not on this page · raw message

drivers/infiniband/hw/mana/qp.c +56 -24

130 lines not on this page · raw message

drivers/infiniband/hw/mana/shadow_queue.h +17 -21
index a127edf7a..b5d6bb28b 100644
@@ -6,13 +6,19 @@
 #ifndef _MANA_SHADOW_QUEUE_H_
 #define _MANA_SHADOW_QUEUE_H_
 
+#include <linux/build_bug.h>
+
+#define MANA_WQ_FENCE_WC		BIT(0)
+#define MANA_WQ_NO_SIGNAL_WC		BIT(1)
+#define MANA_WQE_OFFSET_MASK	GENMASK(23, 0)
+
 struct shadow_wqe_header {
 	u64 wr_id;
+	u64 wqe_offset_or_psn : 24;
 	u64 wqe_size_in_bu : 8;
+	u64 fsn : 24;
 	u64 send_opcode : 4;
-	u64 error_code : 16;
-	u32 byte_len;
-	u32 src_qpn;
+	u64 flags : 2;
 };
 
 struct shadow_queue {
@@ -20,8 +26,6 @@ struct shadow_queue {
 	u64 prod_idx;
 	/* Unmasked consumer index, Incremented on cq polling */
 	u64 cons_idx;
-	/* Unmasked index of next-to-complete (from HW) shadow WQE */
-	u64 next_to_complete_idx;
 	/* queue size in wqes */
 	u32 length;
 	/* distance between elements in bytes */
@@ -42,6 +46,12 @@ static inline int create_shadow_queue(struct shadow_queue *queue, uint32_t lengt
 	return 0;
 }
 
+static inline void reset_shadow_queue(struct shadow_queue *queue)
+{
+	queue->prod_idx = 0;
+	queue->cons_idx = 0;
+}
+
 static inline void destroy_shadow_queue(struct shadow_queue *queue)
 {
 	kvfree(queue->buffer);
@@ -76,22 +86,13 @@ shadow_queue_producer_entry(struct shadow_queue *queue)
 static inline void *
 shadow_queue_get_next_to_consume(const struct shadow_queue *queue)
 {
-	if (queue->cons_idx == queue->next_to_complete_idx)
+	/* The producer publishes the WQE before advancing prod_idx. */
+	if (queue->cons_idx == smp_load_acquire(&queue->prod_idx))
 		return NULL;
 
 	return shadow_queue_get_element(queue, queue->cons_idx);
 }
 
-static inline void *
-shadow_queue_get_next_to_complete(struct shadow_queue *queue)
-{
-	/* Observe initialized WQE fields published by the posting CPU. */
-	if (queue->next_to_complete_idx == smp_load_acquire(&queue->prod_idx))
-		return NULL;
-
-	return shadow_queue_get_element(queue, queue->next_to_complete_idx);
-}
-
 static inline void shadow_queue_advance_producer(struct shadow_queue *queue)
 {
 	/* Publish all WQE fields to CQ polling on another CPU. */
@@ -104,9 +105,4 @@ static inline void shadow_queue_advance_consumer(struct shadow_queue *queue)
 	smp_store_release(&queue->cons_idx, queue->cons_idx + 1);
 }
 
-static inline void shadow_queue_advance_next_to_complete(struct shadow_queue *queue)
-{
-	queue->next_to_complete_idx++;
-}
-
 #endif
drivers/infiniband/hw/mana/wr.c +17 -2
index eb671b45b..250d673f5 100644
@@ -40,6 +40,7 @@ static int mana_ib_post_rq(struct mana_ib_qp *qp, const struct ib_recv_wr *wr)
 	memset(shadow_wqe, 0, sizeof(*shadow_wqe));
 	shadow_wqe->wr_id = wr->wr_id;
 	shadow_wqe->wqe_size_in_bu = wqe_info.wqe_size_in_bu;
+	shadow_wqe->wqe_offset_or_psn = wqe_info.wqe_offset;
 	shadow_queue_advance_producer(&qp->shadow_rq);
 
 	return 0;
@@ -96,6 +97,7 @@ static int mana_ib_post_send_ud_one(struct mana_ib_qp *qp, const struct ib_ud_wr
 	struct gdma_wqe_request wqe_req = {0};
 	struct rdma_send_oob send_oob = {0};
 	struct shadow_wqe_header *shadow_wqe;
+	bool signaled = qp->sq_sig_all || (wr->wr.send_flags & IB_SEND_SIGNALED);
 	int err, i;
 
 	if (!ndev)
@@ -128,7 +130,7 @@ static int mana_ib_post_send_ud_one(struct mana_ib_qp *qp, const struct ib_ud_wr
 
 	send_oob.wqe_type = WQE_TYPE_UD_SEND;
 	send_oob.fence = !!(wr->wr.send_flags & IB_SEND_FENCE);
-	send_oob.signaled = !!(wr->wr.send_flags & IB_SEND_SIGNALED);
+	send_oob.signaled = signaled;
 	send_oob.solicited = !!(wr->wr.send_flags & IB_SEND_SOLICITED);
 	send_oob.psn = qp->sq_psn;
 	send_oob.ssn_or_rqpn = wr->remote_qpn;
@@ -143,7 +145,9 @@ static int mana_ib_post_send_ud_one(struct mana_ib_qp *qp, const struct ib_ud_wr
 	shadow_wqe = shadow_queue_producer_entry(&qp->shadow_sq);
 	memset(shadow_wqe, 0, sizeof(*shadow_wqe));
 	shadow_wqe->wr_id = wr->wr.wr_id;
+	shadow_wqe->flags = signaled ? 0 : MANA_WQ_NO_SIGNAL_WC;
 	shadow_wqe->wqe_size_in_bu = wqe_info.wqe_size_in_bu;
+	shadow_wqe->wqe_offset_or_psn = wqe_info.wqe_offset;
 	shadow_queue_advance_producer(&qp->shadow_sq);
 
 	return 0;
@@ -153,8 +157,11 @@ static int mana_ib_post_send_ud(struct mana_ib_qp *qp, const struct ib_send_wr *
 				const struct ib_send_wr **bad_wr)
 {
 	struct mana_ib_dev *mdev = container_of(qp->ibqp.device, struct mana_ib_dev, ib_dev);
+	struct mana_ib_cq *cq = container_of(qp->ibqp.send_cq, struct mana_ib_cq, ibcq);
 	struct gdma_queue *queue = qp->ud_qp.queues[MANA_UD_SEND_QUEUE].kmem;
+	unsigned long flags;
 	bool ring_sq = false;
+	bool flush_send;
 	int err = 0;
 
 	for (; wr; wr = wr->next) {
@@ -166,9 +173,17 @@ static int mana_ib_post_send_ud(struct mana_ib_qp *qp, const struct ib_send_wr *
 		ring_sq = true;
 	}
 
-	if (ring_sq)
+	if (ring_sq) {
 		mana_gd_wq_ring_doorbell(mdev_to_gc(mdev), queue);
 
+		spin_lock_irqsave(&cq->cq_lock, flags);
+		flush_send = !list_empty(&qp->send_err_node);
+		spin_unlock_irqrestore(&cq->cq_lock, flags);
+
+		if (flush_send && cq->ibcq.comp_handler)
+			cq->ibcq.comp_handler(&cq->ibcq, cq->ibcq.cq_context);
+	}
+
 	return err;
 }
 
drivers/net/ethernet/microsoft/mana/gdma_main.c +4 -2
index cb51fcf57..05ba70554 100644
@@ -1807,10 +1807,12 @@ int mana_gd_post_work_request(struct gdma_queue *wq,
 	if (wq->monitor_avl_buf && wqe_size > mana_gd_wq_avail_space(wq))
 		return -ENOSPC;
 
-	if (wqe_info)
+	head = wq->head;
+	if (wqe_info) {
 		wqe_info->wqe_size_in_bu = wqe_size / GDMA_WQE_BU_SIZE;
+		wqe_info->wqe_offset = head;
+	}
 
-	head = wq->head;
 	wqe_offset = (head * GDMA_WQE_BU_SIZE) & (wq->queue_size - 1);
 	wqe_ptr = mana_gd_get_wqe_ptr(wq, head);
 	oob_len = mana_gd_write_client_oob(wqe_req, wq->type, client_oob_size,
include/net/mana/gdma.h +2 -0
index 02632acd7..2a28fcaa6 100644
@@ -630,6 +630,8 @@ struct gdma_eqe {
 
 struct gdma_posted_wqe_info {
 	u32 wqe_size_in_bu;
+	/* Unmasked WQE start offset in GDMA basic units. */
+	u32 wqe_offset;
 };
 
 /* GDMA_GENERATE_TEST_EQE */
Signature · 1 line
2.43.0
#10 lore raw

Re: [PATCH rdma-next v4 5/5] RDMA/mana_ib: Poll UD completions and flush software error QPs

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] RDMA/mana_ib: missing QP error state transitions prevent SW flush and hang queues - [High]…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] RDMA/mana_ib: missing QP error state transitions prevent SW flush and hang queues
- [High] RDMA/mana_ib: hardware error CQEs are silently dropped or treated as SUCCESS
- [Medium] RDMA/mana_ib: missing queue tail update during RECV SW flush causes resource leak --

commit 1efb216b21b0bbf2212163e8d48231dab5d0c94a
Author: Konstantin Taranov <[email protected]>
Subject: RDMA/mana_ib: Poll UD completions and flush software error QPs

This commit introduces a WC-budgeted polling context that builds WCs directly from CQEs and published shadow entries. It also adds a software flush path for QPs in the error state, since MANA hardware ignores doorbells for them.

Quoted 4 lines
drivers/infiniband/hw/mana/cq.c +0 -0
index 6764e2de0c9f9..3b8044b7dc8fa 100644

[ ... ]

Quoted 4 lines
drivers/infiniband/hw/mana/cq.c +2 -0
@@ -236,126 +288,172 @@ int mana_ib_arm_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags flags)
+static void handle_ud_sq_cqe(struct mana_ib_qp *qp, struct mana_rdma_cqe *rdma_cqe,
+			     struct mana_cq_poll *poll)
 {

[ ... ]

drivers/infiniband/hw/mana/cq.c int mana_ib_arm_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags flags) · line 358

Quoted 3 lines
drivers/infiniband/hw/mana/cq.c +3 -0
+	for (; to_complete; to_complete--)
+		mana_complete_send(qp, poll, VENDOR_ERR_OK);
+}

[Severity: High]
Does this code drop hardware error signals by hardcoding VENDOR_ERR_OK?

Previously, the driver extracted the error code from the hardware CQE, but now it unconditionally reports success. If the hardware delivers an error, it seems it will be incorrectly reported to the application as IB_WC_SUCCESS.

For other CQE types, mana_handle_cqe() might hit the default case and return true without advancing the shadow queue, which could stall it.

[ ... ]

Quoted 2 lines

+static void mana_flush_completions(struct mana_ib_cq *cq, struct mana_cq_poll *poll)
{

[ ... ]

drivers/infiniband/hw/mana/cq.c int mana_ib_arm_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags flags) · line 438

Quoted 1 line
drivers/infiniband/hw/mana/cq.c +1 -0
+	list_for_each_entry(qp, &cq->recv_err_qp_list, recv_err_node) {

[Severity: High]
Is this receive flush loop unreachable?

It iterates over recv_err_qp_list using recv_err_node, but looking at the rest of the code, recv_err_node is never added to any list. If a receive queue goes into the error state, its WQEs might hang forever because this flush path is dead code.

drivers/infiniband/hw/mana/cq.c int mana_ib_arm_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags flags) · line 443

Quoted 5 lines
drivers/infiniband/hw/mana/cq.c +3 -0
+		while (poll->produced < poll->budget &&
+		       (wqe = shadow_queue_get_next_to_consume(&qp->shadow_rq))) {
+			mana_fill_wc(qp, poll, wqe, IB_WC_RECV, VENDOR_ERR_SW_FLUSHED);
 			shadow_queue_advance_consumer(&qp->shadow_rq);
 		}

[Severity: Medium]
Does this receive flush path miss a queue tail update?

In the normal hardware completion path, wq->tail is advanced to free space:

handle_rq_cqe()
wq->tail += wqe->wqe_size_in_bu;

However, here we call shadow_queue_advance_consumer() without advancing the underlying wq->tail. If the QP is recovered and reused, mana_gd_wq_avail_space() might incorrectly see the queue as full, leading to -ENOSPC errors.

Quoted 9 lines
drivers/infiniband/hw/mana/cq.c +2 -0
+		if (poll->produced == poll->budget)
+			return;
 	}
 }
drivers/infiniband/hw/mana/mana_ib.h +0 -0
index 0ad75799062e8..74edb6e707a17 100644

[ ... ]

drivers/infiniband/hw/mana/mana_ib.h struct mana_rdma_cqe { · line 653

Quoted 10 lines
drivers/infiniband/hw/mana/mana_ib.h +1 -2
@@ -594,8 +647,7 @@ struct mana_rdma_cqe {
 		};
 		struct {
 			u32 cqe_type		: 8;
-			u32 vendor_error	: 9;
-			u32 reserved1		: 15;
+			u32 reserved1		: 24;
 			u32 sge_offset		: 5;
 			u32 tx_wqe_offset	: 27;
 		} ud_send;

[Severity: High]
By removing the vendor_error field, the driver seems to lose the ability to parse inline hardware errors. The vendor_error_to_wc_error() switch statement added in cq.c appears to be unreachable for actual hardware errors. How should the hardware communicate these errors without this field?

Quoted 4 lines
drivers/infiniband/hw/mana/qp.c +0 -0
index 79c9cc9443bb6..ae2ebbcc759a2 100644

[ ... ]

Quoted 15 lines
drivers/infiniband/hw/mana/qp.c +11 -0
@@ -891,19 +886,56 @@ static int mana_ib_gd_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
 	return 0;
 }
 
+static void mana_ib_modify_qp_state(struct ib_qp *ibqp, struct ib_qp_attr *attr,
+				    int attr_mask, struct ib_udata *udata)
+{
+	struct mana_ib_qp *qp = container_of(ibqp, struct mana_ib_qp, ibqp);
+
+	if (udata)
+		return;
+
+	if (attr_mask & IB_QP_STATE) {
+		switch (attr->qp_state) {
+		case IB_QPS_RESET:

drivers/infiniband/hw/mana/qp.c static int mana_ib_gd_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, · line 909

Quoted 10 lines
drivers/infiniband/hw/mana/qp.c +10 -0
+			mana_remove_qp_from_cqs(qp, true);
+			break;
+		default:
+			break;
+		}
+	}
+
+	if (attr_mask & IB_QP_SQ_PSN)
+		qp->sq_psn = attr->sq_psn;
+}

[Severity: High]
Does mana_ib_modify_qp_state() need to handle transitions to IB_QPS_ERR?

Since neither send_err_node nor recv_err_node are added to the CQ error lists here (or elsewhere during an error state transition for standard UD QPs), the software flushing mechanism in mana_flush_completions() will not process WQEs for QPs in the error state, causing them to hang.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
#11 lore raw
In reply to Konstantin Taranov
Quoted 11 lines · Konstantin Taranov

From: Konstantin Taranov <[email protected]>

This series updates the kernel UD/GSI work-request and completion paths in the MANA RDMA driver. This series is required for upcoming kernel RC QP support.

The main idea is to fill in WCs directly from CQEs and do not use shadow Qs as proxy. The older code, a CQ had lists of QPs using it, which adds overhead on polling when CQs are shared. Instead, CQs now maintain only the list of QPs in error state.

[...]

Applied, thanks!

[1/5] RDMA/mana_ib: Optimize shadow queue bookkeeping

      https://git.kernel.org/rdma/rdma/c/8b55ae7c81f038
[2/5] RDMA/mana_ib: Revise UD send posting and WQE definitions
      https://git.kernel.org/rdma/rdma/c/afcebf3cfa0e84
[3/5] RDMA/mana_ib: Revise UD receive posting with GDMA_WR_IB_SGL
      https://git.kernel.org/rdma/rdma/c/387c51aa0ee0ae
[4/5] RDMA/mana_ib: Make kernel CQ arming robust
      https://git.kernel.org/rdma/rdma/c/9d3fccc58efe4a
[5/5] RDMA/mana_ib: Poll UD completions and flush software error QPs
      https://git.kernel.org/rdma/rdma/c/1d99b7cea29c31

Best regards,

Signature · 1 line
Leon Romanovsky <[email protected]>