Skip to contentSkip to conversation

More 5.3 patches

PATCH In review Mike Marciniszyn 2 participants 12 messages Jul 15, 2019 – Jul 22, 2019

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover More 5.3 patches

The following series contains fixes and a cleanup.

I noticed that 5.3 rc1 hasn't happened yet? So I'm not quite sure of the destination here.

5 of the patches are stable, and should be held for the rc or pulled for 5.3.

Deleting the unused define can wait if necessary.

---

John Fleck (1):
IB/hfi1: Check for error on call to alloc_rsm_map_table

Kaike Wan (4):
IB/hfi1: Unreserve a flushed OPFN request
IB/hfi1: Field not zero-ed when allocating TID flow memory IB/hfi1: Drop all TID RDMA READ RESP packets after r_next_psn IB/hfi1: Do not update hcrc for a KDETH packet during fault injection

Mike Marciniszyn (1):
IB/hfi1: Remove unused define

 drivers/infiniband/hw/hfi1/chip.c      |   11 +++++++-
 drivers/infiniband/hw/hfi1/rc.c        |    2 -
 drivers/infiniband/hw/hfi1/tid_rdma.c  |   43 +-------------------------------
 drivers/infiniband/hw/hfi1/user_sdma.h |    6 ----
 drivers/infiniband/hw/hfi1/verbs.c     |   17 +++++++------
 include/rdma/rdmavt_qp.h               |    9 +++----
 6 files changed, 24 insertions(+), 64 deletions(-)
Signature · 1 line
Mike
#1 lore raw

PATCH 1/6 IB/hfi1: Check for error on call to alloc_rsm_map_table

From: John Fleck <[email protected]>

The call to alloc_rsm_map_table does not check if the kmalloc fails. Check for a NULL on alloc, and bail if it fails.

Fixes 372cc85a13c9 ("IB/hfi1: Extract RSM map table init from QOS")
---
 drivers/infiniband/hw/hfi1/chip.c |   11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
drivers/infiniband/hw/hfi1/chip.c +9 -2
index d5b643a..67052dc 100644
@@ -14452,7 +14452,7 @@ void hfi1_deinit_vnic_rsm(struct hfi1_devdata *dd)
 		clear_rcvctrl(dd, RCV_CTRL_RCV_RSM_ENABLE_SMASK);
 }
 
-static void init_rxe(struct hfi1_devdata *dd)
+static int init_rxe(struct hfi1_devdata *dd)
 {
 	struct rsm_map_table *rmt;
 	u64 val;
@@ -14461,6 +14461,9 @@ static void init_rxe(struct hfi1_devdata *dd)
 	write_csr(dd, RCV_ERR_MASK, ~0ull);
 
 	rmt = alloc_rsm_map_table(dd);
+	if (!rmt)
+		return -ENOMEM;
+
 	/* set up QOS, including the QPN map table */
 	init_qos(dd, rmt);
 	init_fecn_handling(dd, rmt);
@@ -14487,6 +14490,7 @@ static void init_rxe(struct hfi1_devdata *dd)
 	val |= ((4ull & RCV_BYPASS_HDR_SIZE_MASK) <<
 		RCV_BYPASS_HDR_SIZE_SHIFT);
 	write_csr(dd, RCV_BYPASS, val);
+	return 0;
 }
 
 static void init_other(struct hfi1_devdata *dd)
@@ -15024,7 +15028,10 @@ int hfi1_init_dd(struct hfi1_devdata *dd)
 		goto bail_cleanup;
 
 	/* set initial RXE CSRs */
-	init_rxe(dd);
+	ret = init_rxe(dd);
+	if (ret)
+		goto bail_cleanup;
+
 	/* set initial TXE CSRs */
 	init_txe(dd);
 	/* set initial non-RXE, non-TXE CSRs */
#2 lore raw

PATCH 2/6 IB/hfi1: Unreserve a flushed OPFN request

In reply to Mike Marciniszyn

From: Kaike Wan <[email protected]>

When an OPFN request is flushed, the request is completed without unreserving itself from the send queue. Subsequently, when a new request is post sent, the following warning will be triggered:

WARNING: CPU: 4 PID: 8130 at rdmavt/qp.c:1761 rvt_post_send+0x72a/0x880 [rdmavt]
Call Trace:
[<ffffffffbbb61e41>] dump_stack+0x19/0x1b
[<ffffffffbb497688>] __warn+0xd8/0x100
[<ffffffffbb4977cd>] warn_slowpath_null+0x1d/0x20
[<ffffffffc01c941a>] rvt_post_send+0x72a/0x880 [rdmavt]
[<ffffffffbb4dcabe>] ? account_entity_dequeue+0xae/0xd0
[<ffffffffbb61d645>] ? __kmalloc+0x55/0x230
[<ffffffffc04e1a4c>] ib_uverbs_post_send+0x37c/0x5d0 [ib_uverbs]
[<ffffffffc04e5e36>] ? rdma_lookup_put_uobject+0x26/0x60 [ib_uverbs]
[<ffffffffc04dbce6>] ib_uverbs_write+0x286/0x460 [ib_uverbs]
[<ffffffffbb6f9457>] ? security_file_permission+0x27/0xa0
[<ffffffffbb641650>] vfs_write+0xc0/0x1f0
[<ffffffffbb64246f>] SyS_write+0x7f/0xf0
[<ffffffffbbb74ddb>] system_call_fastpath+0x22/0x27

This patch fixes the problem by moving rvt_qp_wqe_unreserve() into rvt_qp_complete_swqe() to simplify the code and make it less error-prone.

Fixes ca95f802ef51 ("IB/hfi1: Unreserve a reserved request when it is completed")
---
 drivers/infiniband/hw/hfi1/rc.c |    2 --
 include/rdma/rdmavt_qp.h        |    9 ++++-----
 2 files changed, 4 insertions(+), 7 deletions(-)
drivers/infiniband/hw/hfi1/rc.c +0 -2
index 0477c14..024a7c2 100644
@@ -1835,7 +1835,6 @@ void hfi1_rc_send_complete(struct rvt_qp *qp, struct hfi1_opa_header *opah)
 		    cmp_psn(qp->s_sending_psn, qp->s_sending_hpsn) <= 0)
 			break;
 		trdma_clean_swqe(qp, wqe);
-		rvt_qp_wqe_unreserve(qp, wqe);
 		trace_hfi1_qp_send_completion(qp, wqe, qp->s_last);
 		rvt_qp_complete_swqe(qp,
 				     wqe,
@@ -1882,7 +1881,6 @@ struct rvt_swqe *do_rc_completion(struct rvt_qp *qp,
 	if (cmp_psn(wqe->lpsn, qp->s_sending_psn) < 0 ||
 	    cmp_psn(qp->s_sending_psn, qp->s_sending_hpsn) > 0) {
 		trdma_clean_swqe(qp, wqe);
-		rvt_qp_wqe_unreserve(qp, wqe);
 		trace_hfi1_qp_send_completion(qp, wqe, qp->s_last);
 		rvt_qp_complete_swqe(qp,
 				     wqe,
include/rdma/rdmavt_qp.h +4 -5
index 0eeea52..e06c77d 100644
@@ -608,7 +608,7 @@ static inline void rvt_qp_wqe_reserve(
 /**
  * rvt_qp_wqe_unreserve - clean reserved operation
  * @qp - the rvt qp
- * @wqe - the send wqe
+ * @flags - send wqe flags
  *
  * This decrements the reserve use count.
  *
@@ -620,11 +620,9 @@ static inline void rvt_qp_wqe_reserve(
  * the compiler does not juggle the order of the s_last
  * ring index and the decrementing of s_reserved_used.
  */
-static inline void rvt_qp_wqe_unreserve(
-	struct rvt_qp *qp,
-	struct rvt_swqe *wqe)
+static inline void rvt_qp_wqe_unreserve(struct rvt_qp *qp, int flags)
 {
-	if (unlikely(wqe->wr.send_flags & RVT_SEND_RESERVE_USED)) {
+	if (unlikely(flags & RVT_SEND_RESERVE_USED)) {
 		atomic_dec(&qp->s_reserved_used);
 		/* insure no compiler re-order up to s_last change */
 		smp_mb__after_atomic();
@@ -853,6 +851,7 @@ static inline void rvt_send_cq(struct rvt_qp *qp, struct ib_wc *wc,
 	u32 byte_len, last;
 	int flags = wqe->wr.send_flags;
 
+	rvt_qp_wqe_unreserve(qp, flags);
 	rvt_put_qp_swqe(qp, wqe);
 
 	need_completion =
#3 lore raw

PATCH 3/6 IB/hfi1: Field not zero-ed when allocating TID flow memory

In reply to Mike Marciniszyn

From: Kaike Wan <[email protected]>

The field flow->resync_npkts is added for TID RDMA WRITE request and zero-ed when a TID RDMA WRITE RESP packet is received by the requester. This field is used to rewind a request during retry in the function hfi1_tid_rdma_restart_req() shared by both TID RDMA WRITE and TID RDMA READ requests. Therefore, when a TID RDMA READ request is retried, this field may not be initialized at all, which causes the retry to start at an incorrect psn, leading to the drop of the retry request by the responder.

This patch fixes the problem by zeroing out the field when the flow memory is allocated.

Fixes 838b6fd2d9ca ("IB/hfi1: TID RDMA RcvArray programming and TID allocation")
---
 drivers/infiniband/hw/hfi1/tid_rdma.c |    1 +
 1 file changed, 1 insertion(+)
drivers/infiniband/hw/hfi1/tid_rdma.c +1 -0
index 92accca..7fcbeee 100644
@@ -1620,6 +1620,7 @@ static int hfi1_kern_exp_rcv_alloc_flows(struct tid_rdma_request *req,
 		flows[i].req = req;
 		flows[i].npagesets = 0;
 		flows[i].pagesets[0].mapped =  0;
+		flows[i].resync_npkts = 0;
 	}
 	req->flows = flows;
 	return 0;
#4 lore raw

PATCH 4/6 IB/hfi1: Drop all TID RDMA READ RESP packets after r_next_psn

In reply to Mike Marciniszyn

From: Kaike Wan <[email protected]>

When a TID sequence error occurs while receiving TID RDMA READ RESP packets, all packets after flow->flow_state.r_next_psn should be dropped, including those response packets for subsequent segments.

The current implementation will drop the subsequent response packets for the segment to complete next, but may accept packets for subsequent segments and therefore mistakenly advance the r_next_psn fields for the corresponding software flows. This may result in failures to complete subsequent segments after the current segment is completed.

The fix is to only use the flow pointed by req->clear_tail for checking KDETH PSN instead of finding a flow from the request's flow array.

Fixes b885d5be9ca1 ("IB/hfi1: Unify the software PSN check for TID RDMA READ/WRITE")
---
 drivers/infiniband/hw/hfi1/tid_rdma.c |   42 +--------------------------------
 1 file changed, 1 insertion(+), 41 deletions(-)
drivers/infiniband/hw/hfi1/tid_rdma.c +1 -41
index 7fcbeee..996fc2982 100644
@@ -1674,34 +1674,6 @@ static struct tid_rdma_flow *find_flow_ib(struct tid_rdma_request *req,
 	return NULL;
 }
 
-static struct tid_rdma_flow *
-__find_flow_ranged(struct tid_rdma_request *req, u16 head, u16 tail,
-		   u32 psn, u16 *fidx)
-{
-	for ( ; CIRC_CNT(head, tail, MAX_FLOWS);
-	      tail = CIRC_NEXT(tail, MAX_FLOWS)) {
-		struct tid_rdma_flow *flow = &req->flows[tail];
-		u32 spsn, lpsn;
-
-		spsn = full_flow_psn(flow, flow->flow_state.spsn);
-		lpsn = full_flow_psn(flow, flow->flow_state.lpsn);
-
-		if (cmp_psn(psn, spsn) >= 0 && cmp_psn(psn, lpsn) <= 0) {
-			if (fidx)
-				*fidx = tail;
-			return flow;
-		}
-	}
-	return NULL;
-}
-
-static struct tid_rdma_flow *find_flow(struct tid_rdma_request *req,
-				       u32 psn, u16 *fidx)
-{
-	return __find_flow_ranged(req, req->setup_head, req->clear_tail, psn,
-				  fidx);
-}
-
 /* TID RDMA READ functions */
 u32 hfi1_build_tid_rdma_read_packet(struct rvt_swqe *wqe,
 				    struct ib_other_headers *ohdr, u32 *bth1,
@@ -2789,19 +2761,7 @@ static bool handle_read_kdeth_eflags(struct hfi1_ctxtdata *rcd,
 			 * to prevent continuous Flow Sequence errors for any
 			 * packets that could be still in the fabric.
 			 */
-			flow = find_flow(req, psn, NULL);
-			if (!flow) {
-				/*
-				 * We can't find the IB PSN matching the
-				 * received KDETH PSN. The only thing we can
-				 * do at this point is report the error to
-				 * the QP.
-				 */
-				hfi1_kern_read_tid_flow_free(qp);
-				spin_unlock(&qp->s_lock);
-				rvt_rc_error(qp, IB_WC_LOC_QP_OP_ERR);
-				return ret;
-			}
+			flow = &req->flows[req->clear_tail];
 			if (priv->s_flags & HFI1_R_TID_SW_PSN) {
 				diff = cmp_psn(psn,
 					       flow->flow_state.r_next_psn);
#5 lore raw

PATCH 5/6 IB/hfi1: Do not update hcrc for a KDETH packet during fault injection

In reply to Mike Marciniszyn

From: Kaike Wan <[email protected]>

When a KDETH packet is subject to fault injection during transmission, HCRC is supposed to be omitted from the packet so that the hardware on the receiver side would drop the packet. When creating pbc, the PbcInsertHcrc field is set to be PBC_IHCRC_NONE if the KDETH packet is subject to fault injection, but overwritten with PBC_IHCRC_LKDETH when update_hcrc() is called later.

This problem is fixed by not calling update_hcrc() when the packet is subject to fault injection.

Fixes 6b6cf9357f78 ("IB/hfi1: Set PbcInsertHcrc for TID RDMA packets")
---
 drivers/infiniband/hw/hfi1/verbs.c |   17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)
drivers/infiniband/hw/hfi1/verbs.c +9 -8
index c4b243f..f4ca436 100644
@@ -873,16 +873,17 @@ int hfi1_verbs_send_dma(struct rvt_qp *qp, struct hfi1_pkt_state *ps,
 			else
 				pbc |= (ib_is_sc5(sc5) << PBC_DC_INFO_SHIFT);
 
-			if (unlikely(hfi1_dbg_should_fault_tx(qp, ps->opcode)))
-				pbc = hfi1_fault_tx(qp, ps->opcode, pbc);
 			pbc = create_pbc(ppd,
 					 pbc,
 					 qp->srate_mbps,
 					 vl,
 					 plen);
 
-			/* Update HCRC based on packet opcode */
-			pbc = update_hcrc(ps->opcode, pbc);
+			if (unlikely(hfi1_dbg_should_fault_tx(qp, ps->opcode)))
+				pbc = hfi1_fault_tx(qp, ps->opcode, pbc);
+			else
+				/* Update HCRC based on packet opcode */
+				pbc = update_hcrc(ps->opcode, pbc);
 		}
 		tx->wqe = qp->s_wqe;
 		ret = build_verbs_tx_desc(tx->sde, len, tx, ahg_info, pbc);
@@ -1029,12 +1030,12 @@ int hfi1_verbs_send_pio(struct rvt_qp *qp, struct hfi1_pkt_state *ps,
 		else
 			pbc |= (ib_is_sc5(sc5) << PBC_DC_INFO_SHIFT);
 
+		pbc = create_pbc(ppd, pbc, qp->srate_mbps, vl, plen);
 		if (unlikely(hfi1_dbg_should_fault_tx(qp, ps->opcode)))
 			pbc = hfi1_fault_tx(qp, ps->opcode, pbc);
-		pbc = create_pbc(ppd, pbc, qp->srate_mbps, vl, plen);
-
-		/* Update HCRC based on packet opcode */
-		pbc = update_hcrc(ps->opcode, pbc);
+		else
+			/* Update HCRC based on packet opcode */
+			pbc = update_hcrc(ps->opcode, pbc);
 	}
 	if (cb)
 		iowait_pio_inc(&priv->s_iowait);
#6 lore raw

PATCH 6/6 IB/hfi1: Remove unused define

In reply to Mike Marciniszyn

The patch noted in Fixes missed deleting the define it obsoleted.

Delete it here.

Fixes da9de5f8527f ("IB/hfi1: Close PSM sdma_progress sleep window")
---
 drivers/infiniband/hw/hfi1/user_sdma.h |    6 ------
 1 file changed, 6 deletions(-)
drivers/infiniband/hw/hfi1/user_sdma.h +0 -6
index 4d8510b0..9972e0e 100644
@@ -110,12 +110,6 @@ enum pkt_q_sdma_state {
 	SDMA_PKT_Q_DEFERRED,
 };
 
-/*
- * Maximum retry attempts to submit a TX request
- * before putting the process to sleep.
- */
-#define MAX_DEFER_RETRY_COUNT 1
-
 #define SDMA_IOWAIT_TIMEOUT 1000 /* in milliseconds */
 
 #define SDMA_DBG(req, fmt, ...)				     \
#7 lore raw
In reply to Mike Marciniszyn
Quoted 4 lines · Mike Marciniszyn

The following series contains fixes and a cleanup.

I noticed that 5.3 rc1 hasn't happened yet? So I'm not quite sure of the destination here.

You shouldn't send patches during the merge window. If they don't apply cleanly to rc1 they will need resend.

Jason

#8 lore raw
Quoted 11 lines

Subject: Re: [PATCH 0/6] More 5.3 patches

Quoted 4 lines · Mike Marciniszyn

The following series contains fixes and a cleanup.

I noticed that 5.3 rc1 hasn't happened yet? So I'm not quite sure of the destination here.

You shouldn't send patches during the merge window. If they don't apply cleanly to rc1 they will need resend.

They apply cleanly to for-next and rdma/for-next.

Mike

#9 lore raw
Quoted 5 lines · Mike
Quoted 5 lines

Subject: Re: [PATCH 0/6] More 5.3 patches

Quoted 2 lines · Mike Marciniszyn

The following series contains fixes and a cleanup.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

I noticed that 5.3 rc1 hasn't happened yet? So I'm not quite sure of the destination here.

You shouldn't send patches during the merge window. If they don't apply cleanly to rc1 they will need resend.

They apply cleanly to for-next and rdma/for-next.

Those branches are done - we are jumping to 5.3-rc1 next Monday.

Jason

#10 lore raw
Quoted 5 lines
Quoted 1 line

They apply cleanly to for-next and rdma/for-next.

Those branches are done - we are jumping to 5.3-rc1 next Monday.

Jason

Ok. I will resubmit on that base.

Mike

#11 lore raw
In reply to Mike Marciniszyn
Quoted 10 lines · Mike Marciniszyn

The following series contains fixes and a cleanup.

I noticed that 5.3 rc1 hasn't happened yet? So I'm not quite sure of the destination here.

5 of the patches are stable, and should be held for the rc or pulled for 5.3.

Deleting the unused define can wait if necessary.

Quoted 7 lines

John Fleck (1):
IB/hfi1: Check for error on call to alloc_rsm_map_table

Kaike Wan (4):
IB/hfi1: Unreserve a flushed OPFN request
IB/hfi1: Field not zero-ed when allocating TID flow memory IB/hfi1: Drop all TID RDMA READ RESP packets after r_next_psn

These four look OK for -rc, applied

Quoted 1 line

IB/hfi1: Do not update hcrc for a KDETH packet during fault injection

This looks like it is just debugging fixing, it can go to -next

So all 6 applied

Thanks,
Jason