More 5.3 patches
Subsystems (2)
cover More 5.3 patches
The following series contains fixes and a cleanup.
I noticed that 5.3 rc1 hasn't happened yet? So I'm not quite sure of the destination here.
5 of the patches are stable, and should be held for the rc or pulled for 5.3.
Deleting the unused define can wait if necessary.
---
John Fleck (1):
IB/hfi1: Check for error on call to alloc_rsm_map_table
Kaike Wan (4):
IB/hfi1: Unreserve a flushed OPFN request
IB/hfi1: Field not zero-ed when allocating TID flow memory IB/hfi1: Drop all TID RDMA READ RESP packets after r_next_psn IB/hfi1: Do not update hcrc for a KDETH packet during fault injection
Mike Marciniszyn (1):
IB/hfi1: Remove unused define
drivers/infiniband/hw/hfi1/chip.c | 11 +++++++- drivers/infiniband/hw/hfi1/rc.c | 2 - drivers/infiniband/hw/hfi1/tid_rdma.c | 43 +------------------------------- drivers/infiniband/hw/hfi1/user_sdma.h | 6 ---- drivers/infiniband/hw/hfi1/verbs.c | 17 +++++++------ include/rdma/rdmavt_qp.h | 9 +++---- 6 files changed, 24 insertions(+), 64 deletions(-)
Signature · 1 line
Mike
PATCH 1/6 IB/hfi1: Check for error on call to alloc_rsm_map_table
From: John Fleck <[email protected]>
The call to alloc_rsm_map_table does not check if the kmalloc fails. Check for a NULL on alloc, and bail if it fails.
--- drivers/infiniband/hw/hfi1/chip.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
drivers/infiniband/hw/hfi1/chip.c +9 -2
@@ -14452,7 +14452,7 @@ void hfi1_deinit_vnic_rsm(struct hfi1_devdata *dd)clear_rcvctrl (dd,RCV_CTRL_RCV_RSM_ENABLE_SMASK ); } -static void init_rxe (struct hfi1_devdata *dd) +static int init_rxe (struct hfi1_devdata *dd) {struct rsm_map_table *rmt;u64 val;
@@ -14461,6 +14461,9 @@ static void init_rxe(struct hfi1_devdata *dd)write_csr (dd,RCV_ERR_MASK , ~0ull ); rmt =alloc_rsm_map_table (dd); +if (!rmt) +return -ENOMEM ; +/* set up QOS, including the QPN map table */ init_qos (dd, rmt);init_fecn_handling (dd, rmt);
@@ -14487,6 +14490,7 @@ static void init_rxe(struct hfi1_devdata *dd) val |= ((4ull &RCV_BYPASS_HDR_SIZE_MASK ) <<RCV_BYPASS_HDR_SIZE_SHIFT );write_csr (dd,RCV_BYPASS , val); +return 0 ; }static void init_other (struct hfi1_devdata *dd)
@@ -15024,7 +15028,10 @@ int hfi1_init_dd(struct hfi1_devdata *dd) goto bail_cleanup;/* set initial RXE CSRs */ -init_rxe (dd); + ret =init_rxe (dd); +if (ret) + goto bail_cleanup; +/* set initial TXE CSRs */ init_txe (dd);/* set initial non-RXE, non-TXE CSRs */
PATCH 2/6 IB/hfi1: Unreserve a flushed OPFN request
In reply to Mike MarciniszynFrom: Kaike Wan <[email protected]>
When an OPFN request is flushed, the request is completed without unreserving itself from the send queue. Subsequently, when a new request is post sent, the following warning will be triggered:
WARNING: CPU: 4 PID: 8130 at rdmavt/qp.c:1761 rvt_post_send+0x72a/0x880 [rdmavt] Call Trace: [<ffffffffbbb61e41>] dump_stack+0x19/0x1b [<ffffffffbb497688>] __warn+0xd8/0x100 [<ffffffffbb4977cd>] warn_slowpath_null+0x1d/0x20 [<ffffffffc01c941a>] rvt_post_send+0x72a/0x880 [rdmavt] [<ffffffffbb4dcabe>] ? account_entity_dequeue+0xae/0xd0 [<ffffffffbb61d645>] ? __kmalloc+0x55/0x230 [<ffffffffc04e1a4c>] ib_uverbs_post_send+0x37c/0x5d0 [ib_uverbs] [<ffffffffc04e5e36>] ? rdma_lookup_put_uobject+0x26/0x60 [ib_uverbs] [<ffffffffc04dbce6>] ib_uverbs_write+0x286/0x460 [ib_uverbs] [<ffffffffbb6f9457>] ? security_file_permission+0x27/0xa0 [<ffffffffbb641650>] vfs_write+0xc0/0x1f0 [<ffffffffbb64246f>] SyS_write+0x7f/0xf0 [<ffffffffbbb74ddb>] system_call_fastpath+0x22/0x27
This patch fixes the problem by moving rvt_qp_wqe_unreserve() into rvt_qp_complete_swqe() to simplify the code and make it less error-prone.
--- drivers/infiniband/hw/hfi1/rc.c | 2 -- include/rdma/rdmavt_qp.h | 9 ++++----- 2 files changed, 4 insertions(+), 7 deletions(-)
drivers/infiniband/hw/hfi1/rc.c +0 -2
@@ -1835,7 +1835,6 @@ void hfi1_rc_send_complete(struct rvt_qp *qp, struct hfi1_opa_header *opah) cmp_psn(qp->s_sending_psn, qp->s_sending_hpsn) <= 0)break ;trdma_clean_swqe (qp, wqe); -rvt_qp_wqe_unreserve (qp, wqe);trace_hfi1_qp_send_completion (qp, wqe, qp->s_last); rvt_qp_complete_swqe(qp, wqe,
@@ -1882,7 +1881,6 @@ struct rvt_swqe *do_rc_completion(struct rvt_qp *qp,if (cmp_psn (wqe->lpsn, qp->s_sending_psn) <0 ||cmp_psn (qp->s_sending_psn, qp->s_sending_hpsn) >0 ) {trdma_clean_swqe (qp, wqe); -rvt_qp_wqe_unreserve (qp, wqe);trace_hfi1_qp_send_completion (qp, wqe, qp->s_last); rvt_qp_complete_swqe(qp, wqe,
include/rdma/rdmavt_qp.h +4 -5
@@ -608,7 +608,7 @@ static inline void rvt_qp_wqe_reserve(/** * rvt_qp_wqe_unreserve - clean reserved operation * @qp - the rvt qp -* @wqe - the send wqe +* @flags - send wqe flags * * This decrements the reserve use count. *
@@ -620,11 +620,9 @@ static inline void rvt_qp_wqe_reserve(* the compiler does not juggle the order of the s_last * ring index and the decrementing of s_reserved_used. */ -static inline void rvt_qp_wqe_unreserve ( -struct rvt_qp *qp, -struct rvt_swqe *wqe) +static inline void rvt_qp_wqe_unreserve (struct rvt_qp *qp,int flags) { -if (unlikely (wqe->wr.send_flags &RVT_SEND_RESERVE_USED )) { +if (unlikely (flags &RVT_SEND_RESERVE_USED )) {atomic_dec (&qp->s_reserved_used);/* insure no compiler re-order up to s_last change */ smp_mb__after_atomic ();
@@ -853,6 +851,7 @@ static inline void rvt_send_cq(struct rvt_qp *qp, struct ib_wc *wc,u32 byte_len, last;int flags = wqe->wr.send_flags; +rvt_qp_wqe_unreserve (qp, flags);rvt_put_qp_swqe (qp, wqe); need_completion =
PATCH 3/6 IB/hfi1: Field not zero-ed when allocating TID flow memory
In reply to Mike MarciniszynFrom: Kaike Wan <[email protected]>
The field flow->resync_npkts is added for TID RDMA WRITE request and zero-ed when a TID RDMA WRITE RESP packet is received by the requester. This field is used to rewind a request during retry in the function hfi1_tid_rdma_restart_req() shared by both TID RDMA WRITE and TID RDMA READ requests. Therefore, when a TID RDMA READ request is retried, this field may not be initialized at all, which causes the retry to start at an incorrect psn, leading to the drop of the retry request by the responder.
This patch fixes the problem by zeroing out the field when the flow memory is allocated.
--- drivers/infiniband/hw/hfi1/tid_rdma.c | 1 + 1 file changed, 1 insertion(+)
drivers/infiniband/hw/hfi1/tid_rdma.c +1 -0
@@ -1620,6 +1620,7 @@ static int hfi1_kern_exp_rcv_alloc_flows(struct tid_rdma_request *req, flows[i].req = req; flows[i].npagesets =0 ; flows[i].pagesets[0 ].mapped =0 ; + flows[i].resync_npkts =0 ; } req->flows = flows;return 0 ;
PATCH 4/6 IB/hfi1: Drop all TID RDMA READ RESP packets after r_next_psn
In reply to Mike MarciniszynFrom: Kaike Wan <[email protected]>
When a TID sequence error occurs while receiving TID RDMA READ RESP packets, all packets after flow->flow_state.r_next_psn should be dropped, including those response packets for subsequent segments.
The current implementation will drop the subsequent response packets for the segment to complete next, but may accept packets for subsequent segments and therefore mistakenly advance the r_next_psn fields for the corresponding software flows. This may result in failures to complete subsequent segments after the current segment is completed.
The fix is to only use the flow pointed by req->clear_tail for checking KDETH PSN instead of finding a flow from the request's flow array.
--- drivers/infiniband/hw/hfi1/tid_rdma.c | 42 +-------------------------------- 1 file changed, 1 insertion(+), 41 deletions(-)
drivers/infiniband/hw/hfi1/tid_rdma.c +1 -41
@@ -1674,34 +1674,6 @@ static struct tid_rdma_flow *find_flow_ib(struct tid_rdma_request *req,return NULL ; } -static struct tid_rdma_flow * -__find_flow_ranged (struct tid_rdma_request *req,u16 head,u16 tail, -u32 psn,u16 *fidx) -{ -for ( ;CIRC_CNT (head, tail,MAX_FLOWS ); - tail =CIRC_NEXT (tail,MAX_FLOWS )) { -struct tid_rdma_flow *flow = &req->flows[tail]; -u32 spsn, lpsn; - - spsn =full_flow_psn (flow, flow->flow_state.spsn); - lpsn =full_flow_psn (flow, flow->flow_state.lpsn); - -if (cmp_psn (psn, spsn) >=0 &&cmp_psn (psn, lpsn) <=0 ) { -if (fidx) - *fidx = tail; -return flow; - } - } -return NULL ; -} - -static struct tid_rdma_flow *find_flow (struct tid_rdma_request *req, -u32 psn,u16 *fidx) -{ -return __find_flow_ranged (req, req->setup_head, req->clear_tail, psn, - fidx); -} -/* TID RDMA READ functions */ u32 hfi1_build_tid_rdma_read_packet(struct rvt_swqe *wqe, struct ib_other_headers *ohdr, u32 *bth1,
@@ -2789,19 +2761,7 @@ static bool handle_read_kdeth_eflags(struct hfi1_ctxtdata *rcd,* to prevent continuous Flow Sequence errors for any * packets that could be still in the fabric. */ - flow =find_flow (req, psn,NULL ); -if (!flow) { -/* -* We can't find the IB PSN matching the -* received KDETH PSN. The only thing we can -* do at this point is report the error to -* the QP. -*/ -hfi1_kern_read_tid_flow_free (qp); -spin_unlock (&qp->s_lock); -rvt_rc_error (qp,IB_WC_LOC_QP_OP_ERR ); -return ret; - } + flow = &req->flows[req->clear_tail];if (priv->s_flags &HFI1_R_TID_SW_PSN ) { diff =cmp_psn (psn, flow->flow_state.r_next_psn);
PATCH 5/6 IB/hfi1: Do not update hcrc for a KDETH packet during fault injection
In reply to Mike MarciniszynFrom: Kaike Wan <[email protected]>
When a KDETH packet is subject to fault injection during transmission, HCRC is supposed to be omitted from the packet so that the hardware on the receiver side would drop the packet. When creating pbc, the PbcInsertHcrc field is set to be PBC_IHCRC_NONE if the KDETH packet is subject to fault injection, but overwritten with PBC_IHCRC_LKDETH when update_hcrc() is called later.
This problem is fixed by not calling update_hcrc() when the packet is subject to fault injection.
--- drivers/infiniband/hw/hfi1/verbs.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-)
drivers/infiniband/hw/hfi1/verbs.c +9 -8
@@ -873,16 +873,17 @@ int hfi1_verbs_send_dma(struct rvt_qp *qp, struct hfi1_pkt_state *ps,else pbc |= (ib_is_sc5 (sc5) <<PBC_DC_INFO_SHIFT ); -if (unlikely (hfi1_dbg_should_fault_tx (qp, ps->opcode))) - pbc =hfi1_fault_tx (qp, ps->opcode, pbc); pbc =create_pbc (ppd, pbc, qp->srate_mbps, vl, plen); -/* Update HCRC based on packet opcode */ - pbc =update_hcrc (ps->opcode, pbc); +if (unlikely (hfi1_dbg_should_fault_tx (qp, ps->opcode))) + pbc =hfi1_fault_tx (qp, ps->opcode, pbc); +else +/* Update HCRC based on packet opcode */ + pbc =update_hcrc (ps->opcode, pbc); } tx->wqe = qp->s_wqe; ret =build_verbs_tx_desc (tx->sde, len, tx, ahg_info, pbc);
@@ -1029,12 +1030,12 @@ int hfi1_verbs_send_pio(struct rvt_qp *qp, struct hfi1_pkt_state *ps,else pbc |= (ib_is_sc5 (sc5) <<PBC_DC_INFO_SHIFT ); + pbc =create_pbc (ppd, pbc, qp->srate_mbps, vl, plen);if (unlikely (hfi1_dbg_should_fault_tx (qp, ps->opcode))) pbc =hfi1_fault_tx (qp, ps->opcode, pbc); - pbc =create_pbc (ppd, pbc, qp->srate_mbps, vl, plen); - -/* Update HCRC based on packet opcode */ - pbc =update_hcrc (ps->opcode, pbc); +else +/* Update HCRC based on packet opcode */ + pbc =update_hcrc (ps->opcode, pbc); }if (cb)iowait_pio_inc (&priv->s_iowait);
PATCH 6/6 IB/hfi1: Remove unused define
In reply to Mike MarciniszynThe patch noted in Fixes missed deleting the define it obsoleted.
Delete it here.
--- drivers/infiniband/hw/hfi1/user_sdma.h | 6 ------ 1 file changed, 6 deletions(-)
drivers/infiniband/hw/hfi1/user_sdma.h +0 -6
@@ -110,12 +110,6 @@ enum pkt_q_sdma_state { SDMA_PKT_Q_DEFERRED, }; -/* -* Maximum retry attempts to submit a TX request -* before putting the process to sleep. -*/ -#define MAX_DEFER_RETRY_COUNT 1 -#define SDMA_IOWAIT_TIMEOUT 1000/* in milliseconds */ #define SDMA_DBG (req, fmt, ...) \
Quoted 4 lines · Mike Marciniszyn
The following series contains fixes and a cleanup.
I noticed that 5.3 rc1 hasn't happened yet? So I'm not quite sure of the destination here.
You shouldn't send patches during the merge window. If they don't apply cleanly to rc1 they will need resend.
Jason
Quoted 11 lines
Subject: Re: [PATCH 0/6] More 5.3 patches
Quoted 4 lines · Mike Marciniszyn
The following series contains fixes and a cleanup.
I noticed that 5.3 rc1 hasn't happened yet? So I'm not quite sure of the destination here.
You shouldn't send patches during the merge window. If they don't apply cleanly to rc1 they will need resend.
They apply cleanly to for-next and rdma/for-next.
Mike
Quoted 5 lines · Mike
Quoted 5 lines
Subject: Re: [PATCH 0/6] More 5.3 patches
Quoted 2 lines · Mike Marciniszyn
The following series contains fixes and a cleanup.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
I noticed that 5.3 rc1 hasn't happened yet? So I'm not quite sure of the destination here.
You shouldn't send patches during the merge window. If they don't apply cleanly to rc1 they will need resend.
They apply cleanly to for-next and rdma/for-next.
Those branches are done - we are jumping to 5.3-rc1 next Monday.
Jason
Quoted 5 lines
Quoted 1 line
They apply cleanly to for-next and rdma/for-next.
Those branches are done - we are jumping to 5.3-rc1 next Monday.
Jason
Ok. I will resubmit on that base.
Mike
Quoted 10 lines · Mike Marciniszyn
The following series contains fixes and a cleanup.
I noticed that 5.3 rc1 hasn't happened yet? So I'm not quite sure of the destination here.
5 of the patches are stable, and should be held for the rc or pulled for 5.3.
Deleting the unused define can wait if necessary.
Quoted 7 lines
John Fleck (1):
IB/hfi1: Check for error on call toalloc_rsm_map_tableKaike Wan (4):
IB/hfi1: Unreserve a flushed OPFN request
IB/hfi1: Field not zero-ed when allocating TID flow memory IB/hfi1: Drop all TID RDMA READ RESP packets afterr_next_psn
These four look OK for -rc, applied
Quoted 1 line
IB/hfi1: Do not update hcrc for a KDETH packet during fault injection
This looks like it is just debugging fixing, it can go to -next
So all 6 applied
Thanks,
Jason