Three rxe bug fixes
Subsystems (2)
v2 cover Three rxe bug fixes
These patches fix three bugs currently in the rxe driver. One of these was released earlier. In this v2 version two other bug fixes are added.
Bob Pearson (3):
RDMA/rxe: Fix bug in get_srq_wqe() in rxe_resp.c RDMA/rxe: Fix bug in rxe_net.c
RDMA/rxe: Add memory barriers to kernel queues
drivers/infiniband/sw/rxe/rxe_comp.c | 10 +--- drivers/infiniband/sw/rxe/rxe_cq.c | 25 ++------- drivers/infiniband/sw/rxe/rxe_net.c | 1 + drivers/infiniband/sw/rxe/rxe_qp.c | 10 ++-- drivers/infiniband/sw/rxe/rxe_queue.h | 73 ++++++++------------------- drivers/infiniband/sw/rxe/rxe_req.c | 21 ++------ drivers/infiniband/sw/rxe/rxe_resp.c | 40 ++++----------- drivers/infiniband/sw/rxe/rxe_srq.c | 2 +- drivers/infiniband/sw/rxe/rxe_verbs.c | 53 ++++--------------- 9 files changed, 57 insertions(+), 178 deletions(-)
Signature · 1 line
2.30.2
PATCH v2 1/3 RDMA/rxe: Fix bug in get_srq_wqe() in rxe_resp.c
The memcpy() that copies a WQE from a SRQ the QP uses an incorrect size. The size should have been the size of the rxe_send_wqe struct not the size of a pointer to it. The result is that IO operations using a SRQ on the responder side will fail.
--- drivers/infiniband/sw/rxe/rxe_resp.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/sw/rxe/rxe_resp.c +1 -1
@@ -318,7 +318,7 @@ static enum resp_states get_srq_wqe(struct rxe_qp *qp)pr_warn ("%s: invalid num_sge in SRQ entry\n" , __func__);return RESPST_ERR_MALFORMED_WQE ; } - size =sizeof (wqe) + wqe->dma.num_sge*sizeof (struct rxe_sge ); + size =sizeof (*wqe) + wqe->dma.num_sge*sizeof (struct rxe_sge );memcpy (&qp->resp.srq_wqe, wqe, size); qp->resp.wqe = &qp->resp.srq_wqe.wqe;
Signature · 1 line
2.30.2
PATCH v2 2/3 RDMA/rxe: Fix bug in rxe_net.c
In reply to Bob PearsonAn earlier patch removed setting of tot_len in IPV4 headers because it was also set in ip_local_out. However, this change resulted in an incorrect ICRC being computed because the tot_len field is not masked out. This patch restores that line. This fixes the bug reported by Zhu Yanjun. This bug affects anyone using rxe which is currently broken.
--- drivers/infiniband/sw/rxe/rxe_net.c | 1 + 1 file changed, 1 insertion(+)
drivers/infiniband/sw/rxe/rxe_net.c +1 -0
@@ -259,6 +259,7 @@ static void prepare_ipv4_hdr(struct dst_entry *dst, struct sk_buff *skb, iph->version =IPVERSION ; iph->ihl =sizeof (struct iphdr ) >>2 ; + iph->tot_len =htons (skb->len); iph->frag_off = df; iph->protocol = proto; iph->tos = tos;
Signature · 1 line
2.30.2
Zhu Yanjun
maintainerRe: [PATCH for-next v2 2/3] RDMA/rxe: Fix bug in rxe_net.c
Quoted 10 lines · Bob Pearson
An earlier patch removed setting of
tot_lenin IPV4 headers because it was also set inip_local_out. However, this change resulted in an incorrect ICRC being computed because thetot_lenfield is not masked out. This patch restores that line. This fixes the bug reported by Zhu Yanjun. This bug affects anyone using rxe which is currently broken.Fixes 230bb836ee88 ("RDMA/rxe: Fix redundant call to ip_send_check")Reported-by Zhu Yanjun <[email protected]>Signed-off-by Bob Pearson <[email protected]>
This commit is needed.
Without this commit, on some different kernel versions, SoftRoCE can not connect to each other.
Thanks a lot.
Zhu Yanjun
Quoted 19 lines
--- drivers/infiniband/sw/rxe/rxe_net.c | 1 + 1 file changed, 1 insertion(+)drivers/infiniband/sw/rxe/rxe_net.c +1
-1@@ -259,6 +259,7 @@ static void prepare_ipv4_hdr(struct dst_entry *dst, struct sk_buff *skb, iph->version =IPVERSION ; iph->ihl =sizeof (struct iphdr ) >>2 ; + iph->tot_len =htons (skb->len); iph->frag_off = df; iph->protocol = proto; iph->tos = tos; --2.30.2
PATCH v2 3/3 RDMA/rxe: Add memory barriers to kernel queues
In reply to Bob PearsonEarlier patches added memory barriers to protect user space to kernel space communications. This patch extends that to the case where queues are used between kernel space threads.
--- drivers/infiniband/sw/rxe/rxe_comp.c | 10 +--- drivers/infiniband/sw/rxe/rxe_cq.c | 25 ++------- drivers/infiniband/sw/rxe/rxe_qp.c | 10 ++-- drivers/infiniband/sw/rxe/rxe_queue.h | 73 ++++++++------------------- drivers/infiniband/sw/rxe/rxe_req.c | 21 ++------ drivers/infiniband/sw/rxe/rxe_resp.c | 38 ++++---------- drivers/infiniband/sw/rxe/rxe_srq.c | 2 +- drivers/infiniband/sw/rxe/rxe_verbs.c | 53 ++++--------------- 8 files changed, 55 insertions(+), 177 deletions(-)
drivers/infiniband/sw/rxe/rxe_comp.c +2 -8
@@ -142,10 +142,7 @@ static inline enum comp_state get_wqe(struct rxe_qp *qp,/* we come here whether or not we found a response packet to see if * there are any posted WQEs */ -if (qp->is_user) - wqe =queue_head (qp->sq.queue,QUEUE_TYPE_FROM_USER ); -else - wqe =queue_head (qp->sq.queue,QUEUE_TYPE_KERNEL ); + wqe =queue_head (qp->sq.queue,QUEUE_TYPE_FROM_CLIENT ); *wqe_p = wqe;/* no WQE or requester has not started it yet */
@@ -432,10 +429,7 @@ static void do_complete(struct rxe_qp *qp, struct rxe_send_wqe *wqe)if (post)make_send_cqe (qp, wqe, &cqe); -if (qp->is_user) -advance_consumer (qp->sq.queue,QUEUE_TYPE_FROM_USER ); -else -advance_consumer (qp->sq.queue,QUEUE_TYPE_KERNEL ); +advance_consumer (qp->sq.queue,QUEUE_TYPE_FROM_CLIENT );if (post)rxe_cq_post (qp->scq, &cqe,0 );
drivers/infiniband/sw/rxe/rxe_cq.c +5 -20
@@ -25,11 +25,7 @@ int rxe_cq_chk_attr(struct rxe_dev *rxe, struct rxe_cq *cq, }if (cq) { -if (cq->is_user) - count =queue_count (cq->queue,QUEUE_TYPE_TO_USER ); -else - count =queue_count (cq->queue,QUEUE_TYPE_KERNEL ); - + count =queue_count (cq->queue,QUEUE_TYPE_TO_CLIENT );if (cqe < count) {pr_warn ("cqe(%d) < current # elements in queue (%d)" , cqe, count);
@@ -65,7 +61,7 @@ int rxe_cq_from_init(struct rxe_dev *rxe, struct rxe_cq *cq, int cqe,int err;enum queue_type type; - type = uresp ?QUEUE_TYPE_TO_USER :QUEUE_TYPE_KERNEL ; + type =QUEUE_TYPE_TO_CLIENT ; cq->queue =rxe_queue_init (rxe, &cqe,sizeof (struct rxe_cqe ), type);if (!cq->queue) {
@@ -117,11 +113,7 @@ int rxe_cq_post(struct rxe_cq *cq, struct rxe_cqe *cqe, int solicited)spin_lock_irqsave (&cq->cq_lock, flags); -if (cq->is_user) - full =queue_full (cq->queue,QUEUE_TYPE_TO_USER ); -else - full =queue_full (cq->queue,QUEUE_TYPE_KERNEL ); - + full =queue_full (cq->queue,QUEUE_TYPE_TO_CLIENT );if (unlikely (full)) {spin_unlock_irqrestore (&cq->cq_lock, flags);if (cq->ibcq.event_handler) {
@@ -134,17 +126,10 @@ int rxe_cq_post(struct rxe_cq *cq, struct rxe_cqe *cqe, int solicited)return -EBUSY ; } -if (cq->is_user) - addr =producer_addr (cq->queue,QUEUE_TYPE_TO_USER ); -else - addr =producer_addr (cq->queue,QUEUE_TYPE_KERNEL ); - + addr =producer_addr (cq->queue,QUEUE_TYPE_TO_CLIENT );memcpy (addr, cqe,sizeof (*cqe)); -if (cq->is_user) -advance_producer (cq->queue,QUEUE_TYPE_TO_USER ); -else -advance_producer (cq->queue,QUEUE_TYPE_KERNEL ); +advance_producer (cq->queue,QUEUE_TYPE_TO_CLIENT );spin_unlock_irqrestore (&cq->cq_lock, flags);
drivers/infiniband/sw/rxe/rxe_qp.c +3 -7
@@ -231,7 +231,7 @@ static int rxe_qp_init_req(struct rxe_dev *rxe, struct rxe_qp *qp, qp->sq.max_inline = init->cap.max_inline_data = wqe_size; wqe_size +=sizeof (struct rxe_send_wqe ); - type = uresp ?QUEUE_TYPE_FROM_USER :QUEUE_TYPE_KERNEL ; + type =QUEUE_TYPE_FROM_CLIENT ; qp->sq.queue =rxe_queue_init (rxe, &qp->sq.max_wr, wqe_size, type);if (!qp->sq.queue)
@@ -248,12 +248,8 @@ static int rxe_qp_init_req(struct rxe_dev *rxe, struct rxe_qp *qp,return err; } -if (qp->is_user) qp->req.wqe_index =producer_index (qp->sq.queue, -QUEUE_TYPE_FROM_USER ); -else - qp->req.wqe_index =producer_index (qp->sq.queue, -QUEUE_TYPE_KERNEL ); +QUEUE_TYPE_FROM_CLIENT ); qp->req.state =QP_STATE_RESET ; qp->req.opcode =-1 ;
@@ -293,7 +289,7 @@ static int rxe_qp_init_resp(struct rxe_dev *rxe, struct rxe_qp *qp,pr_debug ("qp#%d max_wr = %d, max_sge = %d, wqe_size = %d\n" ,qp_num (qp), qp->rq.max_wr, qp->rq.max_sge, wqe_size); - type = uresp ?QUEUE_TYPE_FROM_USER :QUEUE_TYPE_KERNEL ; + type =QUEUE_TYPE_FROM_CLIENT ; qp->rq.queue =rxe_queue_init (rxe, &qp->rq.max_wr, wqe_size, type);if (!qp->rq.queue)
drivers/infiniband/sw/rxe/rxe_queue.h +20 -53
201 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_req.c +5 -16
@@ -49,13 +49,8 @@ static void req_retry(struct rxe_qp *qp)unsigned int cons;unsigned int prod; -if (qp->is_user) { - cons =consumer_index (q,QUEUE_TYPE_FROM_USER ); - prod =producer_index (q,QUEUE_TYPE_FROM_USER ); - }else { - cons =consumer_index (q,QUEUE_TYPE_KERNEL ); - prod =producer_index (q,QUEUE_TYPE_KERNEL ); - } + cons =consumer_index (q,QUEUE_TYPE_FROM_CLIENT ); + prod =producer_index (q,QUEUE_TYPE_FROM_CLIENT ); qp->req.wqe_index = cons; qp->req.psn = qp->comp.psn;
@@ -121,15 +116,9 @@ static struct rxe_send_wqe *req_next_wqe(struct rxe_qp *qp)unsigned int cons;unsigned int prod; -if (qp->is_user) { - wqe =queue_head (q,QUEUE_TYPE_FROM_USER ); - cons =consumer_index (q,QUEUE_TYPE_FROM_USER ); - prod =producer_index (q,QUEUE_TYPE_FROM_USER ); - }else { - wqe =queue_head (q,QUEUE_TYPE_KERNEL ); - cons =consumer_index (q,QUEUE_TYPE_KERNEL ); - prod =producer_index (q,QUEUE_TYPE_KERNEL ); - } + wqe =queue_head (q,QUEUE_TYPE_FROM_CLIENT ); + cons =consumer_index (q,QUEUE_TYPE_FROM_CLIENT ); + prod =producer_index (q,QUEUE_TYPE_FROM_CLIENT );if (unlikely (qp->req.state ==QP_STATE_DRAIN )) {/* check to see if we are drained;
drivers/infiniband/sw/rxe/rxe_resp.c +9 -29
@@ -303,10 +303,7 @@ static enum resp_states get_srq_wqe(struct rxe_qp *qp)spin_lock_bh (&srq->rq.consumer_lock); -if (qp->is_user) - wqe =queue_head (q,QUEUE_TYPE_FROM_USER ); -else - wqe =queue_head (q,QUEUE_TYPE_KERNEL ); + wqe =queue_head (q,QUEUE_TYPE_FROM_CLIENT );if (!wqe) {spin_unlock_bh (&srq->rq.consumer_lock);return RESPST_ERR_RNR ;
@@ -322,13 +319,8 @@ static enum resp_states get_srq_wqe(struct rxe_qp *qp)memcpy (&qp->resp.srq_wqe, wqe, size); qp->resp.wqe = &qp->resp.srq_wqe.wqe; -if (qp->is_user) { -advance_consumer (q,QUEUE_TYPE_FROM_USER ); - count =queue_count (q,QUEUE_TYPE_FROM_USER ); - }else { -advance_consumer (q,QUEUE_TYPE_KERNEL ); - count =queue_count (q,QUEUE_TYPE_KERNEL ); - } +advance_consumer (q,QUEUE_TYPE_FROM_CLIENT ); + count =queue_count (q,QUEUE_TYPE_FROM_CLIENT );if (srq->limit && srq->ibsrq.event_handler && (count < srq->limit)) { srq->limit =0 ;
@@ -357,12 +349,8 @@ static enum resp_states check_resource(struct rxe_qp *qp, qp->resp.status =IB_WC_WR_FLUSH_ERR ;return RESPST_COMPLETE ; }else if (!srq) { -if (qp->is_user) - qp->resp.wqe =queue_head (qp->rq.queue, -QUEUE_TYPE_FROM_USER ); -else - qp->resp.wqe =queue_head (qp->rq.queue, -QUEUE_TYPE_KERNEL ); + qp->resp.wqe =queue_head (qp->rq.queue, +QUEUE_TYPE_FROM_CLIENT );if (qp->resp.wqe) { qp->resp.status =IB_WC_WR_FLUSH_ERR ;return RESPST_COMPLETE ;
@@ -389,12 +377,8 @@ static enum resp_states check_resource(struct rxe_qp *qp,if (srq)return get_srq_wqe (qp); -if (qp->is_user) - qp->resp.wqe =queue_head (qp->rq.queue, -QUEUE_TYPE_FROM_USER ); -else - qp->resp.wqe =queue_head (qp->rq.queue, -QUEUE_TYPE_KERNEL ); + qp->resp.wqe =queue_head (qp->rq.queue, +QUEUE_TYPE_FROM_CLIENT );return (qp->resp.wqe) ?RESPST_CHK_LENGTH :RESPST_ERR_RNR ; }
@@ -936,12 +920,8 @@ static enum resp_states do_complete(struct rxe_qp *qp, }/* have copy for srq and reference for !srq */ -if (!qp->srq) { -if (qp->is_user) -advance_consumer (qp->rq.queue,QUEUE_TYPE_FROM_USER ); -else -advance_consumer (qp->rq.queue,QUEUE_TYPE_KERNEL ); - } +if (!qp->srq) +advance_consumer (qp->rq.queue,QUEUE_TYPE_FROM_CLIENT ); qp->resp.wqe =NULL ;
drivers/infiniband/sw/rxe/rxe_srq.c +1 -1
@@ -93,7 +93,7 @@ int rxe_srq_from_init(struct rxe_dev *rxe, struct rxe_srq *srq,spin_lock_init (&srq->rq.producer_lock);spin_lock_init (&srq->rq.consumer_lock); - type = uresp ?QUEUE_TYPE_FROM_USER :QUEUE_TYPE_KERNEL ; + type =QUEUE_TYPE_FROM_CLIENT ; q =rxe_queue_init (rxe, &srq->rq.max_wr, srq_wqe_size, type);if (!q) {
drivers/infiniband/sw/rxe/rxe_verbs.c +10 -43
107 lines not on this page · raw message
Signature · 1 line
2.30.2
Jason Gunthorpe
maintainerQuoted 7 lines · Bob Pearson
These patches fix three bugs currently in the rxe driver. One of these was released earlier. In this v2 version two other bug fixes are added.
Bob Pearson (3):
RDMA/rxe: Fix bug inget_srq_wqe()inrxe_resp.cRDMA/rxe: Fix bug inrxe_net.c
I fixed these up and applied them to for-rc
Please be careful that the fixes lines are correct and subjects should be self descriptive not 'fix a bug in x'
Quoted 1 line
RDMA/rxe: Add memory barriers to kernel queues
This one can go to for-next
Jason
Zhu Yanjun
maintainerQuoted 11 lines · Jason Gunthorpe
Quoted 7 lines · Bob Pearson
These patches fix three bugs currently in the rxe driver. One of these was released earlier. In this v2 version two other bug fixes are added.
Bob Pearson (3):
RDMA/rxe: Fix bug inget_srq_wqe()inrxe_resp.cRDMA/rxe: Fix bug inrxe_net.cI fixed these up and applied them to for-rc
I will focus on the other 2 commits except the commit "RDMA/rxe: Fix bug in rxe_net.c" since this commit is reviewed and tested.
Zhu Yanjun
Quoted 9 lines
Please be careful that the fixes lines are correct and subjects should be self descriptive not 'fix a bug in x'
Quoted 1 line
RDMA/rxe: Add memory barriers to kernel queues
This one can go to for-next
Jason