RDMA/rxe: Various fixes and cleanups
Subsystems (2)
v3 cover RDMA/rxe: Various fixes and cleanups
This series of patches is the result of high scale testing on a large HPC system with a large attached Lustre file system. Several errors were found which had not been previously seen at smaller scales. In this case up to 1600 QPs on 1024 compute nodes attached to about 100 flash storage nodes. Each patch has it's own description.
v3
Fixed an error in "Don't call rxe_requester from rxe_completer" Moved run_requester_again from a global to rxe_req_info.again. The control parameter has to be local to each qp.
v2
Minor edits to some of the commit messages.
Added a missing change to "Don't schedule rxe_completer...". Added a missing change to "Git rid of pkt resend on err". Added one additional commit.
Bob Pearson (12):
RDMA/rxe: Fix seg fault in rxe_comp_queue_pkt RDMA/rxe: Allow good work requests to be executed RDMA/rxe: Remove redundant scheduling of rxe_completer RDMA/rxe: Merge request and complete tasks RDMA/rxe: Remove save/rollback_state in rxe_requester RDMA/rxe: Don't schedule rxe_completer from rxe_requester RDMA/rxe: Don't call rxe_requester from rxe_completer RDMA/rxe: Don't call direct between tasks RDMA/rxe: Fix incorrect rxe_put in error path RDMA/rxe: Make rxe_loopback match rxe_send behavior RDMA/rxe: Get rid of pkt resend on err
RDMA/rxe: Let destroy qp succeed with stuck packet
drivers/infiniband/sw/rxe/rxe_comp.c | 32 ++++---- drivers/infiniband/sw/rxe/rxe_hw_counters.c | 2 +- drivers/infiniband/sw/rxe/rxe_hw_counters.h | 2 +- drivers/infiniband/sw/rxe/rxe_loc.h | 3 +- drivers/infiniband/sw/rxe/rxe_net.c | 69 +++++++++-------- drivers/infiniband/sw/rxe/rxe_qp.c | 46 +++++------- drivers/infiniband/sw/rxe/rxe_req.c | 82 ++++++--------------- drivers/infiniband/sw/rxe/rxe_resp.c | 14 +--- drivers/infiniband/sw/rxe/rxe_verbs.c | 17 ++--- drivers/infiniband/sw/rxe/rxe_verbs.h | 7 +- 10 files changed, 111 insertions(+), 163 deletions(-)
Signature · 1 line
2.43.0
PATCH v3 1/12 RDMA/rxe: Fix seg fault in rxe_comp_queue_pkt
In rxe_comp_queue_pkt() an incoming response packet skb is enqueued to the resp_pkts queue and then a decision is made whether to run the completer task inline or schedule it. Finally the skb is dereferenced to bump a 'hw' performance counter. This is wrong because if the completer task is already running in a separate thread it may have already processed the skb and freed it which can cause a seg fault. This has been observed infrequently in testing at high scale.
This patch fixes this by changing the order of enqueuing the packet until after the counter is accessed.
--- drivers/infiniband/sw/rxe/rxe_comp.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/infiniband/sw/rxe/rxe_comp.c +3 -3
@@ -131,12 +131,12 @@ void rxe_comp_queue_pkt(struct rxe_qp *qp, struct sk_buff *skb) {int must_sched; -skb_queue_tail (&qp->resp_pkts, skb); - - must_sched =skb_queue_len (&qp->resp_pkts) >1 ; + must_sched =skb_queue_len (&qp->resp_pkts) >0 ;if (must_sched !=0 )rxe_counter_inc (SKB_TO_PKT (skb)->rxe,RXE_CNT_COMPLETER_SCHED ); +skb_queue_tail (&qp->resp_pkts, skb); +if (must_sched)rxe_sched_task (&qp->comp.task);else
Signature · 1 line
2.43.0
PATCH v3 2/12 RDMA/rxe: Allow good work requests to be executed
In reply to Bob PearsonA previous commit incorrectly added an 'if(!err)' before scheduling the requester task in rxe_post_send_kernel(). But if there were send wrs successfully added to the send queue before a bad wr they might never get executed.
This commit fixes this by scheduling the requester task if any wqes were successfully posted in rxe_post_send_kernel() in rxe_verbs.c.
--- drivers/infiniband/sw/rxe/rxe_verbs.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/infiniband/sw/rxe/rxe_verbs.c +5 -1
@@ -888,6 +888,7 @@ static int rxe_post_send_kernel(struct rxe_qp *qp, {int err =0 ;unsigned long flags; +int good =0 ;spin_lock_irqsave (&qp->sq.sq_lock, flags);while (ibwr) {
@@ -895,12 +896,15 @@ static int rxe_post_send_kernel(struct rxe_qp *qp,if (err) { *bad_wr = ibwr;break ; + }else { + good++; } ibwr = ibwr->next; }spin_unlock_irqrestore (&qp->sq.sq_lock, flags); -if (!err) +/* kickoff processing of any posted wqes */ +if (good)rxe_sched_task (&qp->req.task);spin_lock_irqsave (&qp->state_lock, flags);
Signature · 1 line
2.43.0
Re: [PATCH for-next v3 02/12] RDMA/rxe: Allow good work requests to be executed
Quoted 19 lines · Bob Pearson
A previous commit incorrectly added an 'if(!err)' before scheduling the requester task in
rxe_post_send_kernel(). But if there were send wrs successfully added to the send queue before a bad wr they might never get executed.This commit fixes this by scheduling the requester task if any wqes were successfully posted in
rxe_post_send_kernel()inrxe_verbs.c.Signed-off-by Bob Pearson <[email protected]>Fixes 5bf944f24129 ("RDMA/rxe: Add error messages")--- drivers/infiniband/sw/rxe/rxe_verbs.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)drivers/infiniband/sw/rxe/rxe_verbs.c +0
-0
drivers/infiniband/sw/rxe/rxe_verbs.c static int rxe_post_send_kernel(struct rxe_qp *qp, · line 891
Quoted 4 lines
drivers/infiniband/sw/rxe/rxe_verbs.c +1
-0@@ -888,6 +888,7 @@ static int rxe_post_send_kernel(struct rxe_qp *qp, {int err =0 ;unsigned long flags; +int good =0 ;
If we use RCT (Reverse Christmas Tree) to sort out the above variable declaration, it is better.
Thanks,
Zhu Yanjun
Quoted 20 lines
drivers/infiniband/sw/rxe/rxe_verbs.c +4
-1spin_lock_irqsave (&qp->sq.sq_lock, flags);while (ibwr) {@@ -895,12 +896,15 @@ static int rxe_post_send_kernel(struct rxe_qp *qp,if (err) { *bad_wr = ibwr;break ; + }else { + good++; } ibwr = ibwr->next; }spin_unlock_irqrestore (&qp->sq.sq_lock, flags); -if (!err) +/* kickoff processing of any posted wqes */ +if (good)rxe_sched_task (&qp->req.task);spin_lock_irqsave (&qp->state_lock, flags);
PATCH v3 3/12 RDMA/rxe: Remove redundant scheduling of rxe_completer
In reply to Bob PearsonIn rxe_post_send_kernel() if the qp is in the error state after posting the work requests the rxe_completer() task is scheduled.
But, the only way to move the qp into the error state is to call rxe_qp_error() which also schedules the rxe_completer() task to drain the queues. Calling it a second time has no effect. This commit removes the redundant call.
--- drivers/infiniband/sw/rxe/rxe_verbs.c | 5 ----- 1 file changed, 5 deletions(-)
drivers/infiniband/sw/rxe/rxe_verbs.c +0 -5
@@ -907,11 +907,6 @@ static int rxe_post_send_kernel(struct rxe_qp *qp,if (good)rxe_sched_task (&qp->req.task); -spin_lock_irqsave (&qp->state_lock, flags); -if (qp_state (qp) ==IB_QPS_ERR ) -rxe_sched_task (&qp->comp.task); -spin_unlock_irqrestore (&qp->state_lock, flags); -return err; }
Signature · 1 line
2.43.0
PATCH v3 4/12 RDMA/rxe: Merge request and complete tasks
In reply to Bob PearsonCurrently the rxe driver has three work queue tasks per qp. These are the req.task, comp.task and resp.task which call rxe_requester(), rxe_completer() and rxe_responder() respectively directly or on work queues. Each of these subroutines checks to see if there is work to be performed on the send queue or on the response packet queue or the request packet queue and will run until there is no work remaining or yield the cpu and reschedule itself until there is no work remaining.
This commit combines the req.task and comp.task into a single send.task and renames the resp.task to the recv.task. The combined send.task calls rxe_requester() and rxe_completer() serially and continues until all work on both the send queue and the response packet queue are done.
In various benchmarks the performance is either improved or left the same. At high scale there is a significant reduction in the load on the cpu.
This is the first step in combining these two tasks. Once they are serialized cross rescheduling of req.task and comp.task can be more efficiently handled by just letting the send.task continue to run. This will be done in the next several patches.
--- drivers/infiniband/sw/rxe/rxe_comp.c | 20 +++++----- drivers/infiniband/sw/rxe/rxe_hw_counters.c | 2 +- drivers/infiniband/sw/rxe/rxe_hw_counters.h | 2 +- drivers/infiniband/sw/rxe/rxe_loc.h | 3 +- drivers/infiniband/sw/rxe/rxe_net.c | 4 +- drivers/infiniband/sw/rxe/rxe_qp.c | 44 ++++++++------------- drivers/infiniband/sw/rxe/rxe_req.c | 25 ++++++++++-- drivers/infiniband/sw/rxe/rxe_resp.c | 6 +-- drivers/infiniband/sw/rxe/rxe_verbs.c | 6 +-- drivers/infiniband/sw/rxe/rxe_verbs.h | 6 +-- 10 files changed, 63 insertions(+), 55 deletions(-)
drivers/infiniband/sw/rxe/rxe_comp.c +10 -10
@@ -122,7 +122,7 @@ void retransmit_timer(struct timer_list *t)spin_lock_irqsave (&qp->state_lock, flags);if (qp->valid) { qp->comp.timeout =1 ; -rxe_sched_task (&qp->comp.task); +rxe_sched_task (&qp->send_task); }spin_unlock_irqrestore (&qp->state_lock, flags); }
@@ -133,14 +133,14 @@ void rxe_comp_queue_pkt(struct rxe_qp *qp, struct sk_buff *skb) must_sched =skb_queue_len (&qp->resp_pkts) >0 ;if (must_sched !=0 ) -rxe_counter_inc (SKB_TO_PKT (skb)->rxe,RXE_CNT_COMPLETER_SCHED ); +rxe_counter_inc (SKB_TO_PKT (skb)->rxe,RXE_CNT_SENDER_SCHED );skb_queue_tail (&qp->resp_pkts, skb);if (must_sched) -rxe_sched_task (&qp->comp.task); +rxe_sched_task (&qp->send_task);else -rxe_run_task (&qp->comp.task); +rxe_run_task (&qp->send_task); } static inline enum comp_state get_wqe(struct rxe_qp *qp,
@@ -325,7 +325,7 @@ static inline enum comp_state check_ack(struct rxe_qp *qp, qp->comp.psn = pkt->psn;if (qp->req.wait_psn) { qp->req.wait_psn =0 ; -rxe_sched_task (&qp->req.task); +rxe_sched_task (&qp->send_task); } }return COMPST_ERROR_RETRY ;
@@ -476,7 +476,7 @@ static void do_complete(struct rxe_qp *qp, struct rxe_send_wqe *wqe)*/ if (qp->req.wait_fence) { qp->req.wait_fence =0 ; -rxe_sched_task (&qp->req.task); +rxe_sched_task (&qp->send_task); } }
@@ -515,7 +515,7 @@ static inline enum comp_state complete_ack(struct rxe_qp *qp,if (qp->req.need_rd_atomic) { qp->comp.timeout_retry =0 ; qp->req.need_rd_atomic =0 ; -rxe_sched_task (&qp->req.task); +rxe_sched_task (&qp->send_task); } }
@@ -541,7 +541,7 @@ static inline enum comp_state complete_wqe(struct rxe_qp *qp,if (qp->req.wait_psn) { qp->req.wait_psn =0 ; -rxe_sched_task (&qp->req.task); +rxe_sched_task (&qp->send_task); } }
@@ -737,7 +737,7 @@ int rxe_completer(struct rxe_qp *qp)if (qp->req.wait_psn) { qp->req.wait_psn =0 ; -rxe_sched_task (&qp->req.task); +rxe_sched_task (&qp->send_task); } state =COMPST_DONE ;
@@ -792,7 +792,7 @@ int rxe_completer(struct rxe_qp *qp) RXE_CNT_COMP_RETRY); qp->req.need_retry =1 ; qp->comp.started_retry =1 ; -rxe_sched_task (&qp->req.task); +rxe_sched_task (&qp->send_task); } goto done;
drivers/infiniband/sw/rxe/rxe_hw_counters.c +1 -1
@@ -14,7 +14,7 @@ static const struct rdma_stat_desc rxe_counter_descs[] = { [RXE_CNT_RCV_RNR].name = "rcvd_rnr_err", [RXE_CNT_SND_RNR].name = "send_rnr_err", [RXE_CNT_RCV_SEQ_ERR].name = "rcvd_seq_err", - [RXE_CNT_COMPLETER_SCHED].name = "ack_deferred", + [RXE_CNT_SENDER_SCHED].name = "ack_deferred", [RXE_CNT_RETRY_EXCEEDED].name = "retry_exceeded_err", [RXE_CNT_RNR_RETRY_EXCEEDED].name = "retry_rnr_exceeded_err", [RXE_CNT_COMP_RETRY].name = "completer_retry_err",
drivers/infiniband/sw/rxe/rxe_hw_counters.h +1 -1
@@ -18,7 +18,7 @@ enum rxe_counters {RXE_CNT_RCV_RNR ,RXE_CNT_SND_RNR ,RXE_CNT_RCV_SEQ_ERR , -RXE_CNT_COMPLETER_SCHED , +RXE_CNT_SENDER_SCHED ,RXE_CNT_RETRY_EXCEEDED ,RXE_CNT_RNR_RETRY_EXCEEDED , RXE_CNT_COMP_RETRY,
drivers/infiniband/sw/rxe/rxe_loc.h +2 -1
@@ -164,7 +164,8 @@ void rxe_dealloc(struct ib_device *ib_dev);int rxe_completer (struct rxe_qp *qp);int rxe_requester (struct rxe_qp *qp); -int rxe_responder (struct rxe_qp *qp); +int rxe_sender (struct rxe_qp *qp); +int rxe_receiver (struct rxe_qp *qp);/* rxe_icrc.c */ int rxe_icrc_init (struct rxe_dev *rxe);
drivers/infiniband/sw/rxe/rxe_net.c +2 -2
@@ -351,7 +351,7 @@ static void rxe_skb_tx_dtor(struct sk_buff *skb)if (unlikely (qp->need_req_skb && skb_out <RXE_INFLIGHT_SKBS_PER_QP_LOW )) -rxe_sched_task (&qp->req.task); +rxe_sched_task (&qp->send_task);rxe_put (qp); }
@@ -443,7 +443,7 @@ int rxe_xmit_packet(struct rxe_qp *qp, struct rxe_pkt_info *pkt,if ((qp_type (qp) !=IB_QPT_RC ) && (pkt->mask &RXE_END_MASK )) { pkt->wqe->state = wqe_state_done; -rxe_sched_task (&qp->comp.task); +rxe_sched_task (&qp->send_task); }rxe_counter_inc (rxe,RXE_CNT_SENT_PKTS );
drivers/infiniband/sw/rxe/rxe_qp.c +17 -27
91 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_req.c +21 -4
@@ -108,7 +108,7 @@ void rnr_nak_timer(struct timer_list *t)/* request a send queue retry */ qp->req.need_retry =1 ; qp->req.wait_for_rnr_timer =0 ; -rxe_sched_task (&qp->req.task); +rxe_sched_task (&qp->send_task); }spin_unlock_irqrestore (&qp->state_lock, flags); }
@@ -659,7 +659,7 @@ static int rxe_do_local_ops(struct rxe_qp *qp, struct rxe_send_wqe *wqe)* which can lead to a deadlock. So go ahead and complete * it now. */ -rxe_sched_task (&qp->comp.task); +rxe_sched_task (&qp->send_task);return 0 ; }
@@ -786,7 +786,7 @@ int rxe_requester(struct rxe_qp *qp) qp->req.wqe_index); wqe->state = wqe_state_done; wqe->status =IB_WC_SUCCESS ; -rxe_sched_task (&qp->comp.task); +rxe_sched_task (&qp->send_task); goto done; } payload = mtu;
@@ -855,7 +855,7 @@ int rxe_requester(struct rxe_qp *qp)*/ qp->need_req_skb =1 ; -rxe_sched_task (&qp->req.task); +rxe_sched_task (&qp->send_task); goto exit; }
@@ -878,3 +878,20 @@ int rxe_requester(struct rxe_qp *qp) out:return ret; } + +int rxe_sender (struct rxe_qp *qp) +{ +int req_ret; +int comp_ret; + +/* process the send queue */ + req_ret =rxe_requester (qp); + +/* process the response queue */ + comp_ret =rxe_completer (qp); + +/* exit the task loop if both requester and completer +* are ready +*/ +return (req_ret && comp_ret) ? -EAGAIN :0 ; +}
drivers/infiniband/sw/rxe/rxe_resp.c +3 -3
@@ -58,9 +58,9 @@ void rxe_resp_queue_pkt(struct rxe_qp *qp, struct sk_buff *skb) (skb_queue_len (&qp->req_pkts) >1 );if (must_sched) -rxe_sched_task (&qp->resp.task); +rxe_sched_task (&qp->recv_task);else -rxe_run_task (&qp->resp.task); +rxe_run_task (&qp->recv_task); } static inline enum resp_states get_req(struct rxe_qp *qp,
@@ -1485,7 +1485,7 @@ static void flush_recv_queue(struct rxe_qp *qp, bool notify) qp->resp.wqe =NULL ; } -int rxe_responder (struct rxe_qp *qp) +int rxe_receiver (struct rxe_qp *qp) {struct rxe_dev *rxe =to_rdev (qp->ibqp.device);enum resp_states state;
drivers/infiniband/sw/rxe/rxe_verbs.c +3 -3
@@ -905,7 +905,7 @@ static int rxe_post_send_kernel(struct rxe_qp *qp,/* kickoff processing of any posted wqes */ if (good) -rxe_sched_task (&qp->req.task); +rxe_sched_task (&qp->send_task);return err; }
@@ -935,7 +935,7 @@ static int rxe_post_send(struct ib_qp *ibqp, const struct ib_send_wr *wr,if (qp->is_user) {/* Utilize process context to do protocol processing */ -rxe_run_task (&qp->req.task); +rxe_run_task (&qp->send_task); }else { err =rxe_post_send_kernel (qp, wr, bad_wr);if (err)
@@ -1045,7 +1045,7 @@ static int rxe_post_recv(struct ib_qp *ibqp, const struct ib_recv_wr *wr,spin_lock_irqsave (&qp->state_lock, flags);if (qp_state (qp) ==IB_QPS_ERR ) -rxe_sched_task (&qp->resp.task); +rxe_sched_task (&qp->recv_task);spin_unlock_irqrestore (&qp->state_lock, flags);return err;
drivers/infiniband/sw/rxe/rxe_verbs.h +3 -3
@@ -113,7 +113,6 @@ struct rxe_req_info {int need_retry;int wait_for_rnr_timer;int noack_pkts; -struct rxe_task task; };struct rxe_comp_info {
@@ -124,7 +123,6 @@ struct rxe_comp_info {int started_retry;u32 retry_cnt;u32 rnr_retry; -struct rxe_task task; };enum rdatm_res_state {
@@ -196,7 +194,6 @@ struct rxe_resp_info {unsigned int res_head;unsigned int res_tail;struct resp_res *res; -struct rxe_task task; };struct rxe_qp {
@@ -229,6 +226,9 @@ struct rxe_qp {struct sk_buff_head req_pkts;struct sk_buff_head resp_pkts; +struct rxe_task send_task; +struct rxe_task recv_task; +struct rxe_req_info req;struct rxe_comp_info comp;struct rxe_resp_info resp;
Signature · 1 line
2.43.0
PATCH v3 5/12 RDMA/rxe: Remove save/rollback_state in rxe_requester
In reply to Bob PearsonNow that req.task and comp.task are merged it is no longer necessary to call save_state() before calling rxe_xmit_pkt() and rollback_state() if rxe_xmit_pkt() fails. This was done originally to prevent races between rxe_completer() and rxe_requester() which now cannot happen.
--- drivers/infiniband/sw/rxe/rxe_req.c | 40 ++--------------------------- 1 file changed, 2 insertions(+), 38 deletions(-)
drivers/infiniband/sw/rxe/rxe_req.c +2 -38
@@ -573,30 +573,6 @@ static void update_wqe_psn(struct rxe_qp *qp, qp->req.psn = (qp->req.psn +1 ) &BTH_PSN_MASK ; } -static void save_state (struct rxe_send_wqe *wqe, -struct rxe_qp *qp, -struct rxe_send_wqe *rollback_wqe, -u32 *rollback_psn) -{ - rollback_wqe->state = wqe->state; - rollback_wqe->first_psn = wqe->first_psn; - rollback_wqe->last_psn = wqe->last_psn; - rollback_wqe->dma = wqe->dma; - *rollback_psn = qp->req.psn; -} - -static void rollback_state (struct rxe_send_wqe *wqe, -struct rxe_qp *qp, -struct rxe_send_wqe *rollback_wqe, -u32 rollback_psn) -{ - wqe->state = rollback_wqe->state; - wqe->first_psn = rollback_wqe->first_psn; - wqe->last_psn = rollback_wqe->last_psn; - wqe->dma = rollback_wqe->dma; - qp->req.psn = rollback_psn; -} -static void update_state (struct rxe_qp *qp,struct rxe_pkt_info *pkt) { qp->req.opcode = pkt->opcode;
@@ -676,8 +652,6 @@ int rxe_requester(struct rxe_qp *qp)int opcode;int err;int ret; -struct rxe_send_wqe rollback_wqe; -u32 rollback_psn;struct rxe_queue *q = qp->sq.queue;struct rxe_ah *ah;struct rxe_av *av;
@@ -799,9 +773,6 @@ int rxe_requester(struct rxe_qp *qp) pkt.mask = rxe_opcode[opcode].mask; pkt.wqe = wqe; -/* save wqe state before we build and send packet */ -save_state (wqe, qp, &rollback_wqe, &rollback_psn); - av =rxe_get_av (&pkt, &ah);if (unlikely (!av)) {rxe_dbg_qp (qp,"Failed no address vector\n" );
@@ -834,10 +805,6 @@ int rxe_requester(struct rxe_qp *qp)if (ah)rxe_put (ah); -/* update wqe state as though we had sent it */ -update_wqe_state (qp, wqe, &pkt); -update_wqe_psn (qp, wqe, &pkt, payload); - err =rxe_xmit_packet (qp, &pkt, skb);if (err) {if (err != -EAGAIN ) {
@@ -845,11 +812,6 @@ int rxe_requester(struct rxe_qp *qp) goto err; } -/* the packet was dropped so reset wqe to the state -* before we sent it so we can try to resend -*/ -rollback_state (wqe, qp, &rollback_wqe, rollback_psn); -/* force a delay until the dropped packet is freed and * the send queue is drained below the low water mark */
@@ -859,6 +821,8 @@ int rxe_requester(struct rxe_qp *qp) goto exit; } +update_wqe_state (qp, wqe, &pkt); +update_wqe_psn (qp, wqe, &pkt, payload);update_state (qp, &pkt);/* A non-zero return value will cause rxe_do_task to
Signature · 1 line
2.43.0
PATCH v3 6/12 RDMA/rxe: Don't schedule rxe_completer from rxe_requester
In reply to Bob PearsonNow that rxe_completer() is always called serially after rxe_requester() there is no reason to schedule rxe_completer() from rxe_requester().
--- drivers/infiniband/sw/rxe/rxe_net.c | 6 ------ drivers/infiniband/sw/rxe/rxe_req.c | 9 ++------- 2 files changed, 2 insertions(+), 13 deletions(-)
drivers/infiniband/sw/rxe/rxe_net.c +0 -6
@@ -440,12 +440,6 @@ int rxe_xmit_packet(struct rxe_qp *qp, struct rxe_pkt_info *pkt,return err; } -if ((qp_type (qp) !=IB_QPT_RC ) && - (pkt->mask &RXE_END_MASK )) { - pkt->wqe->state = wqe_state_done; -rxe_sched_task (&qp->send_task); - } -rxe_counter_inc (rxe,RXE_CNT_SENT_PKTS ); goto done;
drivers/infiniband/sw/rxe/rxe_req.c +2 -7
@@ -545,6 +545,8 @@ static void update_wqe_state(struct rxe_qp *qp,if (pkt->mask &RXE_END_MASK ) {if (qp_type (qp) ==IB_QPT_RC ) wqe->state = wqe_state_pending; +else + wqe->state = wqe_state_done; }else { wqe->state = wqe_state_processing; }
@@ -631,12 +633,6 @@ static int rxe_do_local_ops(struct rxe_qp *qp, struct rxe_send_wqe *wqe) wqe->status =IB_WC_SUCCESS ; qp->req.wqe_index =queue_next_index (qp->sq.queue, qp->req.wqe_index); -/* There is no ack coming for local work requests -* which can lead to a deadlock. So go ahead and complete -* it now. -*/ -rxe_sched_task (&qp->send_task); -return 0 ; }
@@ -760,7 +756,6 @@ int rxe_requester(struct rxe_qp *qp) qp->req.wqe_index); wqe->state = wqe_state_done; wqe->status =IB_WC_SUCCESS ; -rxe_sched_task (&qp->send_task); goto done; } payload = mtu;
Signature · 1 line
2.43.0
PATCH v3 7/12 RDMA/rxe: Don't call rxe_requester from rxe_completer
In reply to Bob PearsonInstead of rescheduling rxe_requester from rxe_completer() just extend the duration of rxe_sender() by one pass. Setting run_requester_again forces rxe_completer() to return 0 which will cause rxe_sender() to be called at least one more time.
--- drivers/infiniband/sw/rxe/rxe_comp.c | 17 ++++++++++------- drivers/infiniband/sw/rxe/rxe_verbs.h | 1 + 2 files changed, 11 insertions(+), 7 deletions(-)
drivers/infiniband/sw/rxe/rxe_comp.c +10 -7
@@ -325,7 +325,7 @@ static inline enum comp_state check_ack(struct rxe_qp *qp, qp->comp.psn = pkt->psn;if (qp->req.wait_psn) { qp->req.wait_psn =0 ; -rxe_sched_task (&qp->send_task); + qp->req.again =1 ; } }return COMPST_ERROR_RETRY ;
@@ -476,7 +476,7 @@ static void do_complete(struct rxe_qp *qp, struct rxe_send_wqe *wqe)*/ if (qp->req.wait_fence) { qp->req.wait_fence =0 ; -rxe_sched_task (&qp->send_task); + qp->req.again =1 ; } }
@@ -515,7 +515,7 @@ static inline enum comp_state complete_ack(struct rxe_qp *qp,if (qp->req.need_rd_atomic) { qp->comp.timeout_retry =0 ; qp->req.need_rd_atomic =0 ; -rxe_sched_task (&qp->send_task); + qp->req.again =1 ; } }
@@ -541,7 +541,7 @@ static inline enum comp_state complete_wqe(struct rxe_qp *qp,if (qp->req.wait_psn) { qp->req.wait_psn =0 ; -rxe_sched_task (&qp->send_task); + qp->req.again =1 ; } }
@@ -654,6 +654,8 @@ int rxe_completer(struct rxe_qp *qp)int ret;unsigned long flags; + qp->req.again =0 ; +spin_lock_irqsave (&qp->state_lock, flags);if (!qp->valid ||qp_state (qp) ==IB_QPS_ERR ||qp_state (qp) ==IB_QPS_RESET ) {
@@ -737,7 +739,7 @@ int rxe_completer(struct rxe_qp *qp)if (qp->req.wait_psn) { qp->req.wait_psn =0 ; -rxe_sched_task (&qp->send_task); + qp->req.again =1 ; } state =COMPST_DONE ;
@@ -792,7 +794,7 @@ int rxe_completer(struct rxe_qp *qp) RXE_CNT_COMP_RETRY); qp->req.need_retry =1 ; qp->comp.started_retry =1 ; -rxe_sched_task (&qp->send_task); + qp->req.again =1 ; } goto done;
@@ -843,8 +845,9 @@ int rxe_completer(struct rxe_qp *qp) ret =0 ; goto out; exit: - ret = -EAGAIN ; + ret = (qp->req.again) ?0 : -EAGAIN ; out: + qp->req.again =0 ;if (pkt)free_pkt (pkt);return ret;
drivers/infiniband/sw/rxe/rxe_verbs.h +1 -0
@@ -113,6 +113,7 @@ struct rxe_req_info {int need_retry;int wait_for_rnr_timer;int noack_pkts; +int again; };struct rxe_comp_info {
Signature · 1 line
2.43.0
PATCH v3 8/12 RDMA/rxe: Don't call direct between tasks
In reply to Bob PearsonReplace calls to rxe_run_task() with rxe_sched_task(). This prevents the tasks from all running on the same cpu.
This change slightly reduces performance for single qp send and write benchmarks in loopback mode but greatly improves the performance with multiple qps because if run task is used all the work tends to be performed on one cpu. For actual on the wire benchmarks there is no noticeable performance change.
--- drivers/infiniband/sw/rxe/rxe_comp.c | 13 ++----------- drivers/infiniband/sw/rxe/rxe_resp.c | 12 +----------- drivers/infiniband/sw/rxe/rxe_verbs.c | 2 +- 3 files changed, 4 insertions(+), 23 deletions(-)
drivers/infiniband/sw/rxe/rxe_comp.c +2 -11
@@ -129,18 +129,9 @@ void retransmit_timer(struct timer_list *t)void rxe_comp_queue_pkt (struct rxe_qp *qp,struct sk_buff *skb) { -int must_sched; - - must_sched =skb_queue_len (&qp->resp_pkts) >0 ; -if (must_sched !=0 ) -rxe_counter_inc (SKB_TO_PKT (skb)->rxe,RXE_CNT_SENDER_SCHED ); - +rxe_counter_inc (SKB_TO_PKT (skb)->rxe,RXE_CNT_SENDER_SCHED );skb_queue_tail (&qp->resp_pkts, skb); - -if (must_sched) -rxe_sched_task (&qp->send_task); -else -rxe_run_task (&qp->send_task); +rxe_sched_task (&qp->send_task); } static inline enum comp_state get_wqe(struct rxe_qp *qp,
drivers/infiniband/sw/rxe/rxe_resp.c +1 -11
@@ -49,18 +49,8 @@ static char *resp_state_name[] = {/* rxe_recv calls here to add a request packet to the input queue */ void rxe_resp_queue_pkt (struct rxe_qp *qp,struct sk_buff *skb) { -int must_sched; -struct rxe_pkt_info *pkt =SKB_TO_PKT (skb); -skb_queue_tail (&qp->req_pkts, skb); - - must_sched = (pkt->opcode ==IB_OPCODE_RC_RDMA_READ_REQUEST ) || - (skb_queue_len (&qp->req_pkts) >1 ); - -if (must_sched) -rxe_sched_task (&qp->recv_task); -else -rxe_run_task (&qp->recv_task); +rxe_sched_task (&qp->recv_task); } static inline enum resp_states get_req(struct rxe_qp *qp,
drivers/infiniband/sw/rxe/rxe_verbs.c +1 -1
@@ -935,7 +935,7 @@ static int rxe_post_send(struct ib_qp *ibqp, const struct ib_send_wr *wr,if (qp->is_user) {/* Utilize process context to do protocol processing */ -rxe_run_task (&qp->send_task); +rxe_sched_task (&qp->send_task); }else { err =rxe_post_send_kernel (qp, wr, bad_wr);if (err)
Signature · 1 line
2.43.0
PATCH v3 9/12 RDMA/rxe: Fix incorrect rxe_put in error path
In reply to Bob PearsonIn rxe_send() a ref is taken on the qp to keep it alive until the kfree_skb() has a chance to call the skb destructor rxe_skb_tx_dtor() which drops the reference. If the packet has an incorrect protocol the error path just calls kfree_skb() which will call the destructor which will drop the ref. Currently the driver also calls rxe_put() which is incorrect. Additionally since the packets sent to rxe_send() are under the control of the driver and it only ever produces IPV4 or IPV6 packets the simplest fix is to remove all the code in this block.
--- drivers/infiniband/sw/rxe/rxe_net.c | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-)
drivers/infiniband/sw/rxe/rxe_net.c +2 -10
@@ -366,18 +366,10 @@ static int rxe_send(struct sk_buff *skb, struct rxe_pkt_info *pkt)rxe_get (pkt->qp);atomic_inc (&pkt->qp->skb_out); -if (skb->protocol ==htons (ETH_P_IP )) { +if (skb->protocol ==htons (ETH_P_IP )) err =ip_local_out (dev_net (skb_dst (skb)->dev), skb->sk, skb); - }else if (skb->protocol ==htons (ETH_P_IPV6 )) { +else err =ip6_local_out (dev_net (skb_dst (skb)->dev), skb->sk, skb); - }else { -rxe_dbg_qp (pkt->qp,"Unknown layer 3 protocol: %d\n" , - skb->protocol); -atomic_dec (&pkt->qp->skb_out); -rxe_put (pkt->qp); -kfree_skb (skb); -return -EINVAL ; - }if (unlikely (net_xmit_eval (err))) {rxe_dbg_qp (pkt->qp,"error sending packet: %d\n" , err);
Signature · 1 line
2.43.0
PATCH v3 10/12 RDMA/rxe: Make rxe_loopback match rxe_send behavior
In reply to Bob PearsonThe rxe send path currently counts the number of skbs outstanding between the rxe driver and the ethernet driver to prevent too many packets to accumulate waiting to send. This patch makes the local loopback path behave the same way. The loopback path forwards the packets to the receive path which will eventually call kfree_skb on all packets and drop the qp references. This makes the loopback path more useful for software testing.
--- drivers/infiniband/sw/rxe/rxe_net.c | 6 ++++++ 1 file changed, 6 insertions(+)
drivers/infiniband/sw/rxe/rxe_net.c +6 -0
@@ -386,6 +386,12 @@ static int rxe_loopback(struct sk_buff *skb, struct rxe_pkt_info *pkt) {memcpy (SKB_TO_PKT (skb), pkt,sizeof (*pkt)); + skb->destructor = rxe_skb_tx_dtor; + skb->sk = pkt->qp->sk->sk; + +rxe_get (pkt->qp); +atomic_inc (&pkt->qp->skb_out); +if (skb->protocol ==htons (ETH_P_IP ))skb_pull (skb,sizeof (struct iphdr ));else
Signature · 1 line
2.43.0
PATCH v3 11/12 RDMA/rxe: Get rid of pkt resend on err
In reply to Bob PearsonCurrently the rxe_driver detects packet drops by ip_local_out() which occur before the packet is sent on the wire and attempts to resend them. This is redundant with the usual retry mechanism which covers packets that get dropped in transit to or from the remote node.
The way this is implemented is not robust since it sets need_req_skb and waits for the number of local skbs outstanding for this qp to drop below a low water mark. This is racy since the skb may be sent to the destructor before the requester can set the need_req_skb flag. This will cause a deadlock in the send path for that qp.
This patch removes this mechanism since the normal retry path will correct the error and resend the packet and it makes no difference if the packet is dropped locally or later.
--- drivers/infiniband/sw/rxe/rxe_net.c | 7 +------ drivers/infiniband/sw/rxe/rxe_req.c | 14 ++------------ 2 files changed, 3 insertions(+), 18 deletions(-)
drivers/infiniband/sw/rxe/rxe_net.c +1 -6
@@ -371,12 +371,7 @@ static int rxe_send(struct sk_buff *skb, struct rxe_pkt_info *pkt)else err =ip6_local_out (dev_net (skb_dst (skb)->dev), skb->sk, skb); -if (unlikely (net_xmit_eval (err))) { -rxe_dbg_qp (pkt->qp,"error sending packet: %d\n" , err); -return -EAGAIN ; - } - -return 0 ; +return err; }/* fix up a send packet to match the packets
drivers/infiniband/sw/rxe/rxe_req.c +2 -12
@@ -802,18 +802,8 @@ int rxe_requester(struct rxe_qp *qp) err =rxe_xmit_packet (qp, &pkt, skb);if (err) { -if (err != -EAGAIN ) { - wqe->status =IB_WC_LOC_QP_OP_ERR ; - goto err; - } - -/* force a delay until the dropped packet is freed and -* the send queue is drained below the low water mark -*/ - qp->need_req_skb =1 ; - -rxe_sched_task (&qp->send_task); - goto exit; + wqe->status =IB_WC_LOC_QP_OP_ERR ; + goto err; }update_wqe_state (qp, wqe, &pkt);
Signature · 1 line
2.43.0
PATCH v3 12/12 RDMA/rxe: Let destroy qp succeed with stuck packet
In reply to Bob PearsonIn some situations a sent packet may get queued in the NIC longer than than timeout of a ULP. Currently if this happens the ULP may try to reset the link by destroying the qp and setting up an alternate connection but will fail because the rxe driver is waiting for the packet to finish getting sent and be returned to the skb destructor function where the qp reference holding things up will be dropped. This patch modifies the way that the qp is passed to the destructor to pass the qp index and not a qp pointer. Then the destructor will attempt to lookup the qp from its index and if it fails exit early. This requires taking a reference on the struct sock rather than the qp allowing the qp to be destroyed while the sk is still around waiting for the packet to finish.
--- drivers/infiniband/sw/rxe/rxe_net.c | 42 +++++++++++++++++++++-------- drivers/infiniband/sw/rxe/rxe_qp.c | 2 +- 2 files changed, 32 insertions(+), 12 deletions(-)
drivers/infiniband/sw/rxe/rxe_net.c +31 -11
@@ -345,25 +345,44 @@ int rxe_prepare(struct rxe_av *av, struct rxe_pkt_info *pkt,static void rxe_skb_tx_dtor (struct sk_buff *skb) { -struct sock *sk = skb->sk; -struct rxe_qp *qp = sk->sk_user_data; -int skb_out =atomic_dec_return (&qp->skb_out); +struct net_device *ndev = skb->dev; +struct rxe_dev *rxe; +unsigned int qp_index; +struct rxe_qp *qp; +int skb_out; + + rxe =rxe_get_dev_from_net (ndev); +if (!rxe &&is_vlan_dev (ndev)) + rxe =rxe_get_dev_from_net (vlan_dev_real_dev (ndev)); +if (WARN_ON (!rxe)) +return ; -if (unlikely (qp->need_req_skb && - skb_out <RXE_INFLIGHT_SKBS_PER_QP_LOW )) + qp_index = (int )(uintptr_t )skb->sk->sk_user_data; +if (!qp_index) +return ; + + qp =rxe_pool_get_index (&rxe->qp_pool, qp_index); +if (!qp) + goto put_dev; + + skb_out =atomic_dec_return (&qp->skb_out); +if (qp->need_req_skb && skb_out <RXE_INFLIGHT_SKBS_PER_QP_LOW )rxe_sched_task (&qp->send_task);rxe_put (qp); +put_dev: +ib_device_put (&rxe->ib_dev); +sock_put (skb->sk); }static int rxe_send (struct sk_buff *skb,struct rxe_pkt_info *pkt) {int err; +struct sock *sk = pkt->qp->sk->sk; +sock_hold (sk); + skb->sk = sk; skb->destructor = rxe_skb_tx_dtor; - skb->sk = pkt->qp->sk->sk; - -rxe_get (pkt->qp);atomic_inc (&pkt->qp->skb_out);if (skb->protocol ==htons (ETH_P_IP ))
@@ -379,12 +398,13 @@ static int rxe_send(struct sk_buff *skb, struct rxe_pkt_info *pkt)*/ static int rxe_loopback (struct sk_buff *skb,struct rxe_pkt_info *pkt) { +struct sock *sk = pkt->qp->sk->sk; +memcpy (SKB_TO_PKT (skb), pkt,sizeof (*pkt)); +sock_hold (sk); + skb->sk = sk; skb->destructor = rxe_skb_tx_dtor; - skb->sk = pkt->qp->sk->sk; - -rxe_get (pkt->qp);atomic_inc (&pkt->qp->skb_out);if (skb->protocol ==htons (ETH_P_IP ))
drivers/infiniband/sw/rxe/rxe_qp.c +1 -1
@@ -244,7 +244,7 @@ static int rxe_qp_init_req(struct rxe_dev *rxe, struct rxe_qp *qp, err =sock_create_kern (&init_net,AF_INET ,SOCK_DGRAM ,0 , &qp->sk);if (err <0 )return err; - qp->sk->sk->sk_user_data = qp; + qp->sk->sk->sk_user_data = (void *)(uintptr_t )qp->elem.index;/* pick a source UDP port number for this QP based on * the source QPN. this spreads traffic for different QPs
Signature · 1 line
2.43.0
Quoted 16 lines · Bob Pearson
This series of patches is the result of high scale testing on a large HPC system with a large attached Lustre file system. Several errors were found which had not been previously seen at smaller scales. In this case up to 1600 QPs on 1024 compute nodes attached to about 100 flash storage nodes. Each patch has it's own description.
v3
Fixed an error in "Don't callrxe_requesterfromrxe_completer" Movedrun_requester_againfrom a global to rxe_req_info.again. The control parameter has to be local to each qp.
v2
Minor edits to some of the commit messages.
Added a missing change to "Don't schedulerxe_completer...". Added a missing change to "Git rid of pkt resend on err". Added one additional commit.
Quoted 13 lines
Bob Pearson (12):
RDMA/rxe: Fix seg fault inrxe_comp_queue_pktRDMA/rxe: Allow good work requests to be executed RDMA/rxe: Remove redundant scheduling ofrxe_completerRDMA/rxe: Merge request and complete tasks RDMA/rxe: Remove save/rollback_state inrxe_requesterRDMA/rxe: Don't schedulerxe_completerfromrxe_requesterRDMA/rxe: Don't callrxe_requesterfromrxe_completerRDMA/rxe: Don't call direct between tasks RDMA/rxe: Fix incorrectrxe_putin error path RDMA/rxe: Makerxe_loopbackmatchrxe_sendbehavior RDMA/rxe: Get rid of pkt resend on err
RDMA/rxe: Let destroy qp succeed with stuck packet
Applied to for-next, thanks
Jason