Correct race conditions in rdma_rxe
Subsystems (2)
v4 cover Correct race conditions in rdma_rxe
There are several race conditions discovered in the current rdma_rxe driver. They mostly relate to races between normal operations and destroying objects. This patch series makes several minor cleanups in rxe_pool.[ch] and replaces the red-black trees currently used by xarrays which have better atomic behavior.
This patch series applies cleanly to current for-next. commit 6a463bc9d999 ("Merge branch 'for-rc' into rdma.git for-next")
---
v4
Restructured patch series to change to xarray earlier which greatly simplified the changes.
Rebased to current for-next
v3
Changed rxe_alloc to use GFP_KERNEL
Addressed other comments by Jason Gunthorp
Merged the previous 06/10 and 07/10 patches into one since they overlapped Added some minor cleanups as 10/10
v2
Rebased to current for-next.
Added 4 additional patches
Bob Pearson (13):
RDMA/rxe: Replace irqsave locks with bh locks RDMA/rxe: Cleanup rxe_pool_entry
RDMA/rxe: Copy setup parameters into rxe_pool RDMA/rxe: Save object pointer in pool element RDMA/rxe: Replace RB tree by xarray for indexes RDMA/rxe: Remove #include "rxe_loc.h" from rxe_pool.c RDMA/rxe: Remove some #defines from rxe_pool.h RDMA/rxe: Reverse the sense of RXE_POOL_NO_ALLOC RDMA/rxe: Replaced keyed rxe objects by indexed objects RDMA/rxe: Prevent taking references to dead objects RDMA/rxe: Fix ref error in rxe_av.c
RDMA/rxe: Replace mr by rkey in responder resources RDMA/rxe: Protect against race between get_index and drop_ref
drivers/infiniband/sw/rxe/rxe.c | 100 +----- drivers/infiniband/sw/rxe/rxe_av.c | 19 +- drivers/infiniband/sw/rxe/rxe_comp.c | 8 +- drivers/infiniband/sw/rxe/rxe_cq.c | 24 +- drivers/infiniband/sw/rxe/rxe_loc.h | 25 +- drivers/infiniband/sw/rxe/rxe_mcast.c | 167 ++++++--- drivers/infiniband/sw/rxe/rxe_mr.c | 7 +- drivers/infiniband/sw/rxe/rxe_mw.c | 25 +- drivers/infiniband/sw/rxe/rxe_net.c | 17 +- drivers/infiniband/sw/rxe/rxe_pool.c | 479 ++++++++------------------ drivers/infiniband/sw/rxe/rxe_pool.h | 129 ++----- drivers/infiniband/sw/rxe/rxe_qp.c | 16 +- drivers/infiniband/sw/rxe/rxe_queue.c | 9 +- drivers/infiniband/sw/rxe/rxe_recv.c | 3 +- drivers/infiniband/sw/rxe/rxe_req.c | 66 ++-- drivers/infiniband/sw/rxe/rxe_resp.c | 125 ++++--- drivers/infiniband/sw/rxe/rxe_srq.c | 2 +- drivers/infiniband/sw/rxe/rxe_task.c | 18 +- drivers/infiniband/sw/rxe/rxe_verbs.c | 73 ++-- drivers/infiniband/sw/rxe/rxe_verbs.h | 23 +- 20 files changed, 542 insertions(+), 793 deletions(-)
Signature · 1 line
2.30.2
PATCH v4 1/13 RDMA/rxe: Replace irqsave locks with bh locks
Most of the locks in the rxe driver are _irqsave/restore locks but in fact there are no interrupt threads that run rxe code or share data with rxe. There are softirq threads and data sharing so the appropriate lock type is _bh. This patch replaces all irqsave type locks with bh type locks.
--- drivers/infiniband/sw/rxe/rxe_comp.c | 8 +++---- drivers/infiniband/sw/rxe/rxe_cq.c | 20 +++++++----------- drivers/infiniband/sw/rxe/rxe_mcast.c | 7 +++---- drivers/infiniband/sw/rxe/rxe_mw.c | 15 ++++++-------- drivers/infiniband/sw/rxe/rxe_pool.c | 30 +++++++++++---------------- drivers/infiniband/sw/rxe/rxe_queue.c | 9 ++++---- drivers/infiniband/sw/rxe/rxe_req.c | 11 ++++------ drivers/infiniband/sw/rxe/rxe_task.c | 18 +++++++--------- drivers/infiniband/sw/rxe/rxe_verbs.c | 27 ++++++++++-------------- 9 files changed, 59 insertions(+), 86 deletions(-)
drivers/infiniband/sw/rxe/rxe_comp.c +3 -5
@@ -458,8 +458,6 @@ static inline enum comp_state complete_ack(struct rxe_qp *qp, struct rxe_pkt_info *pkt, struct rxe_send_wqe *wqe) { - unsigned long flags; -if (wqe->has_rd_atomic) { wqe->has_rd_atomic =0 ;atomic_inc (&qp->req.rd_atomic);
@@ -472,11 +470,11 @@ static inline enum comp_state complete_ack(struct rxe_qp *qp,if (unlikely (qp->req.state ==QP_STATE_DRAIN )) {/* state_lock used by requester & completer */ -spin_lock_irqsave (&qp->state_lock, flags); +spin_lock_bh (&qp->state_lock);if ((qp->req.state ==QP_STATE_DRAIN ) && (qp->comp.psn == qp->req.psn)) { qp->req.state =QP_STATE_DRAINED ; -spin_unlock_irqrestore (&qp->state_lock, flags); +spin_unlock_bh (&qp->state_lock);if (qp->ibqp.event_handler) {struct ib_event ev;
@@ -488,7 +486,7 @@ static inline enum comp_state complete_ack(struct rxe_qp *qp, qp->ibqp.qp_context); } }else { -spin_unlock_irqrestore (&qp->state_lock, flags); +spin_unlock_bh (&qp->state_lock); } }
drivers/infiniband/sw/rxe/rxe_cq.c +8 -12
@@ -42,14 +42,13 @@ int rxe_cq_chk_attr(struct rxe_dev *rxe, struct rxe_cq *cq,static void rxe_send_complete (struct tasklet_struct *t) {struct rxe_cq *cq =from_tasklet (cq, t, comp_task); -unsigned long flags; -spin_lock_irqsave (&cq->cq_lock, flags); +spin_lock_bh (&cq->cq_lock);if (cq->is_dying) { -spin_unlock_irqrestore (&cq->cq_lock, flags); +spin_unlock_bh (&cq->cq_lock);return ; } -spin_unlock_irqrestore (&cq->cq_lock, flags); +spin_unlock_bh (&cq->cq_lock); cq->ibcq.comp_handler (&cq->ibcq, cq->ibcq.cq_context); }
@@ -106,15 +105,14 @@ int rxe_cq_resize_queue(struct rxe_cq *cq, int cqe,int rxe_cq_post (struct rxe_cq *cq,struct rxe_cqe *cqe,int solicited) {struct ib_event ev; -unsigned long flags;int full;void *addr; -spin_lock_irqsave (&cq->cq_lock, flags); +spin_lock_bh (&cq->cq_lock); full =queue_full (cq->queue,QUEUE_TYPE_TO_CLIENT );if (unlikely (full)) { -spin_unlock_irqrestore (&cq->cq_lock, flags); +spin_unlock_bh (&cq->cq_lock);if (cq->ibcq.event_handler) { ev.device = cq->ibcq.device; ev.element.cq = &cq->ibcq;
@@ -130,7 +128,7 @@ int rxe_cq_post(struct rxe_cq *cq, struct rxe_cqe *cqe, int solicited)queue_advance_producer (cq->queue,QUEUE_TYPE_TO_CLIENT ); -spin_unlock_irqrestore (&cq->cq_lock, flags); +spin_unlock_bh (&cq->cq_lock);if ((cq->notify ==IB_CQ_NEXT_COMP ) || (cq->notify ==IB_CQ_SOLICITED && solicited)) {
@@ -143,11 +141,9 @@ int rxe_cq_post(struct rxe_cq *cq, struct rxe_cqe *cqe, int solicited)void rxe_cq_disable (struct rxe_cq *cq) { -unsigned long flags; - -spin_lock_irqsave (&cq->cq_lock, flags); +spin_lock_bh (&cq->cq_lock); cq->is_dying = true; -spin_unlock_irqrestore (&cq->cq_lock, flags); +spin_unlock_bh (&cq->cq_lock); }void rxe_cq_cleanup (struct rxe_pool_entry *arg)
drivers/infiniband/sw/rxe/rxe_mcast.c +3 -4
@@ -40,12 +40,11 @@ int rxe_mcast_get_grp(struct rxe_dev *rxe, union ib_gid *mgid,int err;struct rxe_mc_grp *grp;struct rxe_pool *pool = &rxe->mc_grp_pool; -unsigned long flags;if (rxe->attr.max_mcast_qp_attach ==0 )return -EINVAL ; -write_lock_irqsave (&pool->pool_lock, flags); +write_lock_bh (&pool->pool_lock); grp =rxe_pool_get_key_locked (pool, mgid);if (grp)
@@ -53,13 +52,13 @@ int rxe_mcast_get_grp(struct rxe_dev *rxe, union ib_gid *mgid, grp =create_grp (rxe, pool, mgid);if (IS_ERR (grp)) { -write_unlock_irqrestore (&pool->pool_lock, flags); +write_unlock_bh (&pool->pool_lock); err =PTR_ERR (grp);return err; } done: -write_unlock_irqrestore (&pool->pool_lock, flags); +write_unlock_bh (&pool->pool_lock); *grp_p = grp;return 0 ; }
drivers/infiniband/sw/rxe/rxe_mw.c +6 -9
@@ -56,11 +56,10 @@ int rxe_dealloc_mw(struct ib_mw *ibmw) {struct rxe_mw *mw =to_rmw (ibmw);struct rxe_pd *pd =to_rpd (ibmw->pd); -unsigned long flags; -spin_lock_irqsave (&mw->lock, flags); +spin_lock_bh (&mw->lock);rxe_do_dealloc_mw (mw); -spin_unlock_irqrestore (&mw->lock, flags); +spin_unlock_bh (&mw->lock);rxe_drop_ref (mw);rxe_drop_ref (pd);
@@ -197,7 +196,6 @@ int rxe_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe)struct rxe_dev *rxe =to_rdev (qp->ibqp.device);u32 mw_rkey = wqe->wr.wr.mw.mw_rkey;u32 mr_lkey = wqe->wr.wr.mw.mr_lkey; -unsigned long flags; mw =rxe_pool_get_index (&rxe->mw_pool, mw_rkey >>8 );if (unlikely (!mw)) {
@@ -225,7 +223,7 @@ int rxe_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe) mr =NULL ; } -spin_lock_irqsave (&mw->lock, flags); +spin_lock_bh (&mw->lock); ret =rxe_check_bind_mw (qp, wqe, mw, mr);if (ret)
@@ -233,7 +231,7 @@ int rxe_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe)rxe_do_bind_mw (qp, wqe, mw, mr); err_unlock: -spin_unlock_irqrestore (&mw->lock, flags); +spin_unlock_bh (&mw->lock); err_drop_mr:if (mr)rxe_drop_ref (mr);
@@ -280,7 +278,6 @@ static void rxe_do_invalidate_mw(struct rxe_mw *mw)int rxe_invalidate_mw (struct rxe_qp *qp,u32 rkey) {struct rxe_dev *rxe =to_rdev (qp->ibqp.device); -unsigned long flags;struct rxe_mw *mw;int ret;
@@ -295,7 +292,7 @@ int rxe_invalidate_mw(struct rxe_qp *qp, u32 rkey) goto err_drop_ref; } -spin_lock_irqsave (&mw->lock, flags); +spin_lock_bh (&mw->lock); ret =rxe_check_invalidate_mw (qp, mw);if (ret)
@@ -303,7 +300,7 @@ int rxe_invalidate_mw(struct rxe_qp *qp, u32 rkey)rxe_do_invalidate_mw (mw); err_unlock: -spin_unlock_irqrestore (&mw->lock, flags); +spin_unlock_bh (&mw->lock); err_drop_ref:rxe_drop_ref (mw); err:
drivers/infiniband/sw/rxe/rxe_pool.c +12 -18
@@ -261,12 +261,11 @@ int __rxe_add_key_locked(struct rxe_pool_entry *elem, void *key)int __rxe_add_key (struct rxe_pool_entry *elem,void *key) {struct rxe_pool *pool = elem->pool; -unsigned long flags;int err; -write_lock_irqsave (&pool->pool_lock, flags); +write_lock_bh (&pool->pool_lock); err =__rxe_add_key_locked (elem, key); -write_unlock_irqrestore (&pool->pool_lock, flags); +write_unlock_bh (&pool->pool_lock);return err; }
@@ -281,11 +280,10 @@ void __rxe_drop_key_locked(struct rxe_pool_entry *elem)void __rxe_drop_key (struct rxe_pool_entry *elem) {struct rxe_pool *pool = elem->pool; -unsigned long flags; -write_lock_irqsave (&pool->pool_lock, flags); +write_lock_bh (&pool->pool_lock);__rxe_drop_key_locked (elem); -write_unlock_irqrestore (&pool->pool_lock, flags); +write_unlock_bh (&pool->pool_lock); }int __rxe_add_index_locked (struct rxe_pool_entry *elem)
@@ -302,12 +300,11 @@ int __rxe_add_index_locked(struct rxe_pool_entry *elem)int __rxe_add_index (struct rxe_pool_entry *elem) {struct rxe_pool *pool = elem->pool; -unsigned long flags;int err; -write_lock_irqsave (&pool->pool_lock, flags); +write_lock_bh (&pool->pool_lock); err =__rxe_add_index_locked (elem); -write_unlock_irqrestore (&pool->pool_lock, flags); +write_unlock_bh (&pool->pool_lock);return err; }
@@ -323,11 +320,10 @@ void __rxe_drop_index_locked(struct rxe_pool_entry *elem)void __rxe_drop_index (struct rxe_pool_entry *elem) {struct rxe_pool *pool = elem->pool; -unsigned long flags; -write_lock_irqsave (&pool->pool_lock, flags); +write_lock_bh (&pool->pool_lock);__rxe_drop_index_locked (elem); -write_unlock_irqrestore (&pool->pool_lock, flags); +write_unlock_bh (&pool->pool_lock); }void *rxe_alloc_locked (struct rxe_pool *pool)
@@ -447,11 +443,10 @@ void *rxe_pool_get_index_locked(struct rxe_pool *pool, u32 index)void *rxe_pool_get_index (struct rxe_pool *pool,u32 index) {u8 *obj; -unsigned long flags; -read_lock_irqsave (&pool->pool_lock, flags); +read_lock_bh (&pool->pool_lock); obj =rxe_pool_get_index_locked (pool, index); -read_unlock_irqrestore (&pool->pool_lock, flags); +read_unlock_bh (&pool->pool_lock);return obj; }
@@ -493,11 +488,10 @@ void *rxe_pool_get_key_locked(struct rxe_pool *pool, void *key)void *rxe_pool_get_key (struct rxe_pool *pool,void *key) {u8 *obj; -unsigned long flags; -read_lock_irqsave (&pool->pool_lock, flags); +read_lock_bh (&pool->pool_lock); obj =rxe_pool_get_key_locked (pool, key); -read_unlock_irqrestore (&pool->pool_lock, flags); +read_unlock_bh (&pool->pool_lock);return obj; }
drivers/infiniband/sw/rxe/rxe_queue.c +4 -5
@@ -151,7 +151,6 @@ int rxe_queue_resize(struct rxe_queue *q, unsigned int *num_elem_p,struct rxe_queue *new_q;unsigned int num_elem = *num_elem_p;int err; -unsigned long flags =0 , flags1; new_q =rxe_queue_init (q->rxe, &num_elem, elem_size, q->type);if (!new_q)
@@ -165,17 +164,17 @@ int rxe_queue_resize(struct rxe_queue *q, unsigned int *num_elem_p, goto err1; } -spin_lock_irqsave (consumer_lock, flags1); +spin_lock_bh (consumer_lock);if (producer_lock) { -spin_lock_irqsave (producer_lock, flags); +spin_lock_bh (producer_lock); err =resize_finish (q, new_q, num_elem); -spin_unlock_irqrestore (producer_lock, flags); +spin_unlock_bh (producer_lock); }else { err =resize_finish (q, new_q, num_elem); } -spin_unlock_irqrestore (consumer_lock, flags1); +spin_unlock_bh (consumer_lock);rxe_queue_cleanup (new_q);/* new/old dep on err */ if (err)
drivers/infiniband/sw/rxe/rxe_req.c +4 -7
@@ -110,7 +110,6 @@ void rnr_nak_timer(struct timer_list *t)static struct rxe_send_wqe *req_next_wqe (struct rxe_qp *qp) {struct rxe_send_wqe *wqe; -unsigned long flags;struct rxe_queue *q = qp->sq.queue;unsigned int index = qp->req.wqe_index;unsigned int cons;
@@ -124,25 +123,23 @@ static struct rxe_send_wqe *req_next_wqe(struct rxe_qp *qp)/* check to see if we are drained; * state_lock used by requester and completer */ -spin_lock_irqsave (&qp->state_lock, flags); +spin_lock_bh (&qp->state_lock); do { if (qp->req.state != QP_STATE_DRAIN) { /* comp just finished */ - spin_unlock_irqrestore(&qp->state_lock, - flags); + spin_unlock_bh(&qp->state_lock); break; } if (wqe && ((index != cons) || (wqe->state != wqe_state_posted))) { /* comp not done yet */ - spin_unlock_irqrestore(&qp->state_lock, - flags); + spin_unlock_bh(&qp->state_lock); break; } qp->req.state = QP_STATE_DRAINED; - spin_unlock_irqrestore(&qp->state_lock, flags); + spin_unlock_bh(&qp->state_lock); if (qp->ibqp.event_handler) { struct ib_event ev;
drivers/infiniband/sw/rxe/rxe_task.c +8 -10
@@ -32,25 +32,24 @@ void rxe_do_task(struct tasklet_struct *t) {int cont;int ret; -unsigned long flags;struct rxe_task *task =from_tasklet (task, t, tasklet); -spin_lock_irqsave (&task->state_lock, flags); +spin_lock_bh (&task->state_lock);switch (task->state) {case TASK_STATE_START : task->state =TASK_STATE_BUSY ; -spin_unlock_irqrestore (&task->state_lock, flags); +spin_unlock_bh (&task->state_lock);break ;case TASK_STATE_BUSY : task->state =TASK_STATE_ARMED ; fallthrough;case TASK_STATE_ARMED : -spin_unlock_irqrestore (&task->state_lock, flags); +spin_unlock_bh (&task->state_lock);return ;default : -spin_unlock_irqrestore (&task->state_lock, flags); +spin_unlock_bh (&task->state_lock);pr_warn ("%s failed with bad state %d\n" , __func__, task->state);return ; }
@@ -59,7 +58,7 @@ void rxe_do_task(struct tasklet_struct *t) cont =0 ; ret = task->func (task->arg); -spin_lock_irqsave (&task->state_lock, flags); +spin_lock_bh (&task->state_lock);switch (task->state) {case TASK_STATE_BUSY :if (ret)
@@ -81,7 +80,7 @@ void rxe_do_task(struct tasklet_struct *t)pr_warn ("%s failed with bad state %d\n" , __func__, task->state); } -spin_unlock_irqrestore (&task->state_lock, flags); +spin_unlock_bh (&task->state_lock); }while (cont); task->ret = ret;
@@ -106,7 +105,6 @@ int rxe_init_task(void *obj, struct rxe_task *task,void rxe_cleanup_task (struct rxe_task *task) { -unsigned long flags;bool idle;/*
@@ -116,9 +114,9 @@ void rxe_cleanup_task(struct rxe_task *task) task->destroyed = true;do { -spin_lock_irqsave (&task->state_lock, flags); +spin_lock_bh (&task->state_lock); idle = (task->state ==TASK_STATE_START ); -spin_unlock_irqrestore (&task->state_lock, flags); +spin_unlock_bh (&task->state_lock); }while (!idle);tasklet_kill (&task->tasklet);
drivers/infiniband/sw/rxe/rxe_verbs.c +11 -16
@@ -383,10 +383,9 @@ static int rxe_post_srq_recv(struct ib_srq *ibsrq, const struct ib_recv_wr *wr, const struct ib_recv_wr **bad_wr) {int err =0 ; -unsigned long flags;struct rxe_srq *srq =to_rsrq (ibsrq); -spin_lock_irqsave (&srq->rq.producer_lock, flags); +spin_lock_bh (&srq->rq.producer_lock);while (wr) { err =post_one_recv (&srq->rq, wr);
@@ -395,7 +394,7 @@ static int rxe_post_srq_recv(struct ib_srq *ibsrq, const struct ib_recv_wr *wr, wr = wr->next; } -spin_unlock_irqrestore (&srq->rq.producer_lock, flags); +spin_unlock_bh (&srq->rq.producer_lock);if (err) *bad_wr = wr;
@@ -634,19 +633,18 @@ static int post_one_send(struct rxe_qp *qp, const struct ib_send_wr *ibwr,int err;struct rxe_sq *sq = &qp->sq;struct rxe_send_wqe *send_wqe; -unsigned long flags;int full; err =validate_send_wr (qp, ibwr, mask, length);if (err)return err; -spin_lock_irqsave (&qp->sq.sq_lock, flags); +spin_lock_bh (&qp->sq.sq_lock); full =queue_full (sq->queue,QUEUE_TYPE_TO_DRIVER );if (unlikely (full)) { -spin_unlock_irqrestore (&qp->sq.sq_lock, flags); +spin_unlock_bh (&qp->sq.sq_lock);return -ENOMEM ; }
@@ -655,7 +653,7 @@ static int post_one_send(struct rxe_qp *qp, const struct ib_send_wr *ibwr,queue_advance_producer (sq->queue,QUEUE_TYPE_TO_DRIVER ); -spin_unlock_irqrestore (&qp->sq.sq_lock, flags); +spin_unlock_bh (&qp->sq.sq_lock);return 0 ; }
@@ -735,7 +733,6 @@ static int rxe_post_recv(struct ib_qp *ibqp, const struct ib_recv_wr *wr,int err =0 ;struct rxe_qp *qp =to_rqp (ibqp);struct rxe_rq *rq = &qp->rq; -unsigned long flags;if (unlikely ((qp_state (qp) <IB_QPS_INIT ) || !qp->valid)) { *bad_wr = wr;
@@ -749,7 +746,7 @@ static int rxe_post_recv(struct ib_qp *ibqp, const struct ib_recv_wr *wr, goto err1; } -spin_lock_irqsave (&rq->producer_lock, flags); +spin_lock_bh (&rq->producer_lock);while (wr) { err =post_one_recv (rq, wr);
@@ -760,7 +757,7 @@ static int rxe_post_recv(struct ib_qp *ibqp, const struct ib_recv_wr *wr, wr = wr->next; } -spin_unlock_irqrestore (&rq->producer_lock, flags); +spin_unlock_bh (&rq->producer_lock);if (qp->resp.state ==QP_STATE_ERROR )rxe_run_task (&qp->resp.task,1 );
@@ -841,9 +838,8 @@ static int rxe_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc)int i;struct rxe_cq *cq =to_rcq (ibcq);struct rxe_cqe *cqe; -unsigned long flags; -spin_lock_irqsave (&cq->cq_lock, flags); +spin_lock_bh (&cq->cq_lock);for (i =0 ; i < num_entries; i++) { cqe =queue_head (cq->queue,QUEUE_TYPE_FROM_DRIVER );if (!cqe)
@@ -852,7 +848,7 @@ static int rxe_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc)memcpy (wc++, &cqe->ibwc,sizeof (*wc));queue_advance_consumer (cq->queue,QUEUE_TYPE_FROM_DRIVER ); } -spin_unlock_irqrestore (&cq->cq_lock, flags); +spin_unlock_bh (&cq->cq_lock);return i; }
@@ -870,11 +866,10 @@ static int rxe_peek_cq(struct ib_cq *ibcq, int wc_cnt)static int rxe_req_notify_cq (struct ib_cq *ibcq,enum ib_cq_notify_flags flags) {struct rxe_cq *cq =to_rcq (ibcq); -unsigned long irq_flags;int ret =0 ;int empty; -spin_lock_irqsave (&cq->cq_lock, irq_flags); +spin_lock_bh (&cq->cq_lock);if (cq->notify !=IB_CQ_NEXT_COMP ) cq->notify = flags &IB_CQ_SOLICITED_MASK ;
@@ -883,7 +878,7 @@ static int rxe_req_notify_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags flags)if ((flags &IB_CQ_REPORT_MISSED_EVENTS ) && !empty) ret =1 ; -spin_unlock_irqrestore (&cq->cq_lock, irq_flags); +spin_unlock_bh (&cq->cq_lock);return ret; }
Signature · 1 line
2.30.2
PATCH v4 2/13 RDMA/rxe: Cleanup rxe_pool_entry
In reply to Bob PearsonCurrently three different names are used to describe rxe pool elements. They are referred to as entries, elems or pelems. This patch chooses one 'elem' and changes the other ones.
--- drivers/infiniband/sw/rxe/rxe_cq.c | 4 +- drivers/infiniband/sw/rxe/rxe_loc.h | 10 ++-- drivers/infiniband/sw/rxe/rxe_mcast.c | 4 +- drivers/infiniband/sw/rxe/rxe_mr.c | 6 +-- drivers/infiniband/sw/rxe/rxe_mw.c | 6 +-- drivers/infiniband/sw/rxe/rxe_pool.c | 72 +++++++++++++-------------- drivers/infiniband/sw/rxe/rxe_pool.h | 46 ++++++++--------- drivers/infiniband/sw/rxe/rxe_qp.c | 6 +-- drivers/infiniband/sw/rxe/rxe_srq.c | 2 +- drivers/infiniband/sw/rxe/rxe_verbs.c | 2 +- drivers/infiniband/sw/rxe/rxe_verbs.h | 22 ++++---- 11 files changed, 89 insertions(+), 91 deletions(-)
drivers/infiniband/sw/rxe/rxe_cq.c +2 -2
@@ -146,9 +146,9 @@ void rxe_cq_disable(struct rxe_cq *cq)spin_unlock_bh (&cq->cq_lock); } -void rxe_cq_cleanup (struct rxe_pool_entry *arg) +void rxe_cq_cleanup (struct rxe_pool_elem *elem) { -struct rxe_cq *cq =container_of (arg,typeof (*cq), pelem); +struct rxe_cq *cq =container_of (elem,typeof (*cq), elem);if (cq->queue)rxe_queue_cleanup (cq->queue);
drivers/infiniband/sw/rxe/rxe_loc.h +5 -5
@@ -37,7 +37,7 @@ int rxe_cq_post(struct rxe_cq *cq, struct rxe_cqe *cqe, int solicited);void rxe_cq_disable (struct rxe_cq *cq); -void rxe_cq_cleanup (struct rxe_pool_entry *arg); +void rxe_cq_cleanup (struct rxe_pool_elem *arg);/* rxe_mcast.c */ int rxe_mcast_get_grp(struct rxe_dev *rxe, union ib_gid *mgid,
@@ -51,7 +51,7 @@ int rxe_mcast_drop_grp_elem(struct rxe_dev *rxe, struct rxe_qp *qp,void rxe_drop_all_mcast_groups (struct rxe_qp *qp); -void rxe_mc_cleanup (struct rxe_pool_entry *arg); +void rxe_mc_cleanup (struct rxe_pool_elem *arg);/* rxe_mmap.c */ struct rxe_mmap_info {
@@ -89,7 +89,7 @@ int rxe_invalidate_mr(struct rxe_qp *qp, u32 rkey);int rxe_reg_fast_mr (struct rxe_qp *qp,struct rxe_send_wqe *wqe);int rxe_mr_set_page (struct ib_mr *ibmr,u64 addr);int rxe_dereg_mr (struct ib_mr *ibmr,struct ib_udata *udata); -void rxe_mr_cleanup (struct rxe_pool_entry *arg); +void rxe_mr_cleanup (struct rxe_pool_elem *arg);/* rxe_mw.c */ int rxe_alloc_mw (struct ib_mw *ibmw,struct ib_udata *udata);
@@ -97,7 +97,7 @@ int rxe_dealloc_mw(struct ib_mw *ibmw);int rxe_bind_mw (struct rxe_qp *qp,struct rxe_send_wqe *wqe);int rxe_invalidate_mw (struct rxe_qp *qp,u32 rkey);struct rxe_mw *rxe_lookup_mw (struct rxe_qp *qp,int access,u32 rkey); -void rxe_mw_cleanup (struct rxe_pool_entry *arg); +void rxe_mw_cleanup (struct rxe_pool_elem *arg);/* rxe_net.c */ struct sk_buff *rxe_init_packet(struct rxe_dev *rxe, struct rxe_av *av,
@@ -131,7 +131,7 @@ void rxe_qp_error(struct rxe_qp *qp);void rxe_qp_destroy (struct rxe_qp *qp); -void rxe_qp_cleanup (struct rxe_pool_entry *arg); +void rxe_qp_cleanup (struct rxe_pool_elem *elem);static inline int qp_num (struct rxe_qp *qp) {
drivers/infiniband/sw/rxe/rxe_mcast.c +2 -2
@@ -168,9 +168,9 @@ void rxe_drop_all_mcast_groups(struct rxe_qp *qp) } } -void rxe_mc_cleanup (struct rxe_pool_entry *arg) +void rxe_mc_cleanup (struct rxe_pool_elem *elem) { -struct rxe_mc_grp *grp =container_of (arg,typeof (*grp), pelem); +struct rxe_mc_grp *grp =container_of (elem,typeof (*grp), elem);struct rxe_dev *rxe = grp->rxe;rxe_drop_key (grp);
drivers/infiniband/sw/rxe/rxe_mr.c +3 -3
@@ -50,7 +50,7 @@ int mr_check_range(struct rxe_mr *mr, u64 iova, size_t length)static void rxe_mr_init (int access,struct rxe_mr *mr) { -u32 lkey = mr->pelem.index <<8 |rxe_get_next_key (-1 ); +u32 lkey = mr->elem.index <<8 |rxe_get_next_key (-1 );u32 rkey = (access &IB_ACCESS_REMOTE ) ? lkey :0 ;/* set ibmr->l/rkey and also copy into private l/rkey
@@ -699,9 +699,9 @@ int rxe_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)return 0 ; } -void rxe_mr_cleanup (struct rxe_pool_entry *arg) +void rxe_mr_cleanup (struct rxe_pool_elem *elem) { -struct rxe_mr *mr =container_of (arg,typeof (*mr), pelem); +struct rxe_mr *mr =container_of (elem,typeof (*mr), elem);ib_umem_release (mr->umem);
drivers/infiniband/sw/rxe/rxe_mw.c +3 -3
@@ -21,7 +21,7 @@ int rxe_alloc_mw(struct ib_mw *ibmw, struct ib_udata *udata) }rxe_add_index (mw); - mw->rkey = ibmw->rkey = (mw->pelem.index <<8 ) |rxe_get_next_key (-1 ); + mw->rkey = ibmw->rkey = (mw->elem.index <<8 ) |rxe_get_next_key (-1 ); mw->state = (mw->ibmw.type ==IB_MW_TYPE_2 ) ?RXE_MW_STATE_FREE :RXE_MW_STATE_VALID ;spin_lock_init (&mw->lock);
@@ -330,9 +330,9 @@ struct rxe_mw *rxe_lookup_mw(struct rxe_qp *qp, int access, u32 rkey)return mw; } -void rxe_mw_cleanup (struct rxe_pool_entry *elem) +void rxe_mw_cleanup (struct rxe_pool_elem *elem) { -struct rxe_mw *mw =container_of (elem,typeof (*mw), pelem); +struct rxe_mw *mw =container_of (elem,typeof (*mw), elem);rxe_drop_index (mw); }
drivers/infiniband/sw/rxe/rxe_pool.c +36 -36
268 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_pool.h +22 -24
98 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_qp.c +3 -3
@@ -167,7 +167,7 @@ static void rxe_qp_init_misc(struct rxe_dev *rxe, struct rxe_qp *qp, qp->attr.path_mtu =1 ; qp->mtu =ib_mtu_enum_to_int (qp->attr.path_mtu); - qpn = qp->pelem.index; + qpn = qp->elem.index; port = &rxe->port;switch (init->qp_type) {
@@ -831,9 +831,9 @@ static void rxe_qp_do_cleanup(struct work_struct *work) }/* called when the last reference to the qp is dropped */ -void rxe_qp_cleanup (struct rxe_pool_entry *arg) +void rxe_qp_cleanup (struct rxe_pool_elem *elem) { -struct rxe_qp *qp =container_of (arg,typeof (*qp), pelem); +struct rxe_qp *qp =container_of (elem,typeof (*qp), elem);execute_in_process_context (rxe_qp_do_cleanup, &qp->cleanup_work); }
drivers/infiniband/sw/rxe/rxe_srq.c +1 -1
@@ -83,7 +83,7 @@ int rxe_srq_from_init(struct rxe_dev *rxe, struct rxe_srq *srq, srq->ibsrq.event_handler = init->event_handler; srq->ibsrq.srq_context = init->srq_context; srq->limit = init->attr.srq_limit; - srq->srq_num = srq->pelem.index; + srq->srq_num = srq->elem.index; srq->rq.max_wr = init->attr.max_wr; srq->rq.max_sge = init->attr.max_sge;
drivers/infiniband/sw/rxe/rxe_verbs.c +1 -1
@@ -182,7 +182,7 @@ static int rxe_create_ah(struct ib_ah *ibah,/* create index > 0 */ rxe_add_index (ah); - ah->ah_num = ah->pelem.index; + ah->ah_num = ah->elem.index;if (uresp) {/* only if new user provider */
drivers/infiniband/sw/rxe/rxe_verbs.h +11 -11
@@ -35,17 +35,17 @@ static inline int psn_compare(u32 psn_a, u32 psn_b)struct rxe_ucontext {struct ib_ucontext ibuc; -struct rxe_pool_entry pelem; +struct rxe_pool_elem elem; };struct rxe_pd {struct ib_pd ibpd; -struct rxe_pool_entry pelem; +struct rxe_pool_elem elem; };struct rxe_ah {struct ib_ah ibah; -struct rxe_pool_entry pelem; +struct rxe_pool_elem elem;struct rxe_av av;bool is_user;int ah_num;
@@ -60,7 +60,7 @@ struct rxe_cqe {struct rxe_cq {struct ib_cq ibcq; -struct rxe_pool_entry pelem; +struct rxe_pool_elem elem;struct rxe_queue *queue;spinlock_t cq_lock;u8 notify;
@@ -95,7 +95,7 @@ struct rxe_rq {struct rxe_srq {struct ib_srq ibsrq; -struct rxe_pool_entry pelem; +struct rxe_pool_elem elem;struct rxe_pd *pd;struct rxe_rq rq;u32 srq_num;
@@ -209,7 +209,7 @@ struct rxe_resp_info {struct rxe_qp {struct ib_qp ibqp; -struct rxe_pool_entry pelem; +struct rxe_pool_elem elem;struct ib_qp_attr attr;unsigned int valid;unsigned int mtu;
@@ -309,7 +309,7 @@ static inline int rkey_is_mw(u32 rkey) }struct rxe_mr { -struct rxe_pool_entry pelem; +struct rxe_pool_elem elem;struct ib_mr ibmr;struct ib_umem *umem;
@@ -342,7 +342,7 @@ enum rxe_mw_state {struct rxe_mw {struct ib_mw ibmw; -struct rxe_pool_entry pelem; +struct rxe_pool_elem elem;spinlock_t lock;enum rxe_mw_state state;struct rxe_qp *qp;/* Type 2 only */
@@ -354,7 +354,7 @@ struct rxe_mw { };struct rxe_mc_grp { -struct rxe_pool_entry pelem; +struct rxe_pool_elem elem;spinlock_t mcg_lock;/* guard group */ struct rxe_dev *rxe;struct list_head qp_list;
@@ -365,7 +365,7 @@ struct rxe_mc_grp { };struct rxe_mc_elem { -struct rxe_pool_entry pelem; +struct rxe_pool_elem elem;struct list_head qp_list;struct list_head grp_list;struct rxe_qp *qp;
@@ -484,6 +484,6 @@ static inline struct rxe_pd *rxe_mw_pd(struct rxe_mw *mw)int rxe_register_device (struct rxe_dev *rxe,const char *ibdev_name); -void rxe_mc_cleanup (struct rxe_pool_entry *arg); +void rxe_mc_cleanup (struct rxe_pool_elem *elem);#endif /* RXE_VERBS_H */
Signature · 1 line
2.30.2
PATCH v4 3/13 RDMA/rxe: Copy setup parameters into rxe_pool
In reply to Bob PearsonIn rxe_pool.c copy remaining pool setup parameters from rxe_pool_info into rxe_pool. This saves looking up rxe_pool_info in the performance path.
--- drivers/infiniband/sw/rxe/rxe_pool.c | 56 ++++++++++++---------------- drivers/infiniband/sw/rxe/rxe_pool.h | 4 +- 2 files changed, 27 insertions(+), 33 deletions(-)
drivers/infiniband/sw/rxe/rxe_pool.c +24 -32
@@ -96,11 +96,6 @@ static const struct rxe_type_info { }, }; -static inline const char *pool_name (struct rxe_pool *pool) -{ -return rxe_type_info[pool->type].name; -} -static int rxe_pool_init_index (struct rxe_pool *pool,u32 max,u32 min) {int err =0 ;
@@ -130,35 +125,36 @@ int rxe_pool_init(enum rxe_elem_type type, unsigned int max_elem) { + const struct rxe_type_info *info = &rxe_type_info[type];int err =0 ; -size_t size = rxe_type_info[type].size;memset (pool,0 ,sizeof (*pool)); pool->rxe = rxe; + pool->name = info->name; pool->type = type; pool->max_elem = max_elem; - pool->elem_size =ALIGN (size,RXE_POOL_ALIGN ); - pool->flags = rxe_type_info[type].flags; - pool->index.tree =RB_ROOT ; - pool->key.tree =RB_ROOT ; - pool->cleanup = rxe_type_info[type].cleanup; + pool->elem_size =ALIGN (info->size,RXE_POOL_ALIGN ); + pool->elem_offset = info->elem_offset; + pool->flags = info->flags; + pool->cleanup = info->cleanup;atomic_set (&pool->num_elem,0 );rwlock_init (&pool->pool_lock); -if (rxe_type_info[type].flags &RXE_POOL_INDEX ) { - err =rxe_pool_init_index (pool, - rxe_type_info[type].max_index, - rxe_type_info[type].min_index); +if (pool->flags &RXE_POOL_INDEX ) { + pool->index.tree =RB_ROOT ; + err =rxe_pool_init_index (pool, info->max_index, + info->min_index);if (err) goto out; } -if (rxe_type_info[type].flags &RXE_POOL_KEY ) { - pool->key.key_offset = rxe_type_info[type].key_offset; - pool->key.key_size = rxe_type_info[type].key_size; +if (pool->flags &RXE_POOL_KEY ) { + pool->key.tree =RB_ROOT ; + pool->key.key_offset = info->key_offset; + pool->key.key_size = info->key_size; } out:
@@ -169,9 +165,10 @@ void rxe_pool_cleanup(struct rxe_pool *pool) {if (atomic_read (&pool->num_elem) >0 )pr_warn ("%s pool destroyed with unfree'd elem\n" , -pool_name (pool)); + pool->name); -bitmap_free (pool->index.table); +if (pool->flags &RXE_POOL_INDEX ) +bitmap_free (pool->index.table); }static u32 alloc_index (struct rxe_pool *pool)
@@ -328,18 +325,17 @@ void __rxe_drop_index(struct rxe_pool_elem *elem)void *rxe_alloc_locked (struct rxe_pool *pool) { -const struct rxe_type_info *info = &rxe_type_info[pool->type];struct rxe_pool_elem *elem;u8 *obj;if (atomic_inc_return (&pool->num_elem) > pool->max_elem) goto out_cnt; - obj =kzalloc (info->size,GFP_ATOMIC ); + obj =kzalloc (pool->elem_size,GFP_ATOMIC );if (!obj) goto out_cnt; - elem = (struct rxe_pool_elem *)(obj + info->elem_offset); + elem = (struct rxe_pool_elem *)(obj + pool->elem_offset); elem->pool = pool;kref_init (&elem->ref_cnt);
@@ -353,18 +349,17 @@ void *rxe_alloc_locked(struct rxe_pool *pool)void *rxe_alloc (struct rxe_pool *pool) { -const struct rxe_type_info *info = &rxe_type_info[pool->type];struct rxe_pool_elem *elem;u8 *obj;if (atomic_inc_return (&pool->num_elem) > pool->max_elem) goto out_cnt; - obj =kzalloc (info->size,GFP_KERNEL ); + obj =kzalloc (pool->elem_size,GFP_KERNEL );if (!obj) goto out_cnt; - elem = (struct rxe_pool_elem *)(obj + info->elem_offset); + elem = (struct rxe_pool_elem *)(obj + pool->elem_offset); elem->pool = pool;kref_init (&elem->ref_cnt);
@@ -396,14 +391,13 @@ void rxe_elem_release(struct kref *kref)struct rxe_pool_elem *elem = container_of(kref, struct rxe_pool_elem, ref_cnt);struct rxe_pool *pool = elem->pool; -const struct rxe_type_info *info = &rxe_type_info[pool->type];u8 *obj;if (pool->cleanup) pool->cleanup (elem);if (!(pool->flags &RXE_POOL_NO_ALLOC )) { - obj = (u8 *)elem - info->elem_offset; + obj = (u8 *)elem - pool->elem_offset;kfree (obj); }
@@ -412,7 +406,6 @@ void rxe_elem_release(struct kref *kref)void *rxe_pool_get_index_locked (struct rxe_pool *pool,u32 index) { -const struct rxe_type_info *info = &rxe_type_info[pool->type];struct rb_node *node;struct rxe_pool_elem *elem;u8 *obj;
@@ -432,7 +425,7 @@ void *rxe_pool_get_index_locked(struct rxe_pool *pool, u32 index)if (node) {kref_get (&elem->ref_cnt); - obj = (u8 *)elem - info->elem_offset; + obj = (u8 *)elem - pool->elem_offset; }else { obj =NULL ; }
@@ -453,7 +446,6 @@ void *rxe_pool_get_index(struct rxe_pool *pool, u32 index)void *rxe_pool_get_key_locked (struct rxe_pool *pool,void *key) { -const struct rxe_type_info *info = &rxe_type_info[pool->type];struct rb_node *node;struct rxe_pool_elem *elem;u8 *obj;
@@ -477,7 +469,7 @@ void *rxe_pool_get_key_locked(struct rxe_pool *pool, void *key)if (node) {kref_get (&elem->ref_cnt); - obj = (u8 *)elem - info->elem_offset; + obj = (u8 *)elem - pool->elem_offset; }else { obj =NULL ; }
drivers/infiniband/sw/rxe/rxe_pool.h +3 -1
@@ -45,14 +45,16 @@ struct rxe_pool_elem {struct rxe_pool {struct rxe_dev *rxe; +const char *name;rwlock_t pool_lock;/* protects pool add/del/search */ -size_t elem_size;void (*cleanup)(struct rxe_pool_elem *obj);enum rxe_pool_flags flags;enum rxe_elem_type type;unsigned int max_elem;atomic_t num_elem; +size_t elem_size; +size_t elem_offset;/* only used if indexed */ struct {
Signature · 1 line
2.30.2
PATCH v4 4/13 RDMA/rxe: Save object pointer in pool element
In reply to Bob PearsonIn rxe_pool.c currently there are many cases where it is necessary to compute the offset from a pool element struct to the object containing it in a type independent way where the offset is different for each type. By saving a pointer to the object when they are created extra work can be saved.
--- drivers/infiniband/sw/rxe/rxe_pool.c | 30 ++++++++++++++++------------ drivers/infiniband/sw/rxe/rxe_pool.h | 1 + 2 files changed, 18 insertions(+), 13 deletions(-)
drivers/infiniband/sw/rxe/rxe_pool.c +17 -13
@@ -225,7 +225,8 @@ static int rxe_insert_key(struct rxe_pool *pool, struct rxe_pool_elem *new) elem = rb_entry(parent, struct rxe_pool_elem, key_node); cmp =memcmp ((u8 *)elem + pool->key.key_offset, - (u8 *)new + pool->key.key_offset, pool->key.key_size); + (u8 *)new + pool->key.key_offset, + pool->key.key_size);if (cmp ==0 ) {pr_warn ("key already exists!\n" );
@@ -326,7 +327,7 @@ void __rxe_drop_index(struct rxe_pool_elem *elem)void *rxe_alloc_locked (struct rxe_pool *pool) {struct rxe_pool_elem *elem; -u8 *obj; +void *obj;if (atomic_inc_return (&pool->num_elem) > pool->max_elem) goto out_cnt;
@@ -335,9 +336,10 @@ void *rxe_alloc_locked(struct rxe_pool *pool)if (!obj) goto out_cnt; - elem = (struct rxe_pool_elem *)(obj + pool->elem_offset); + elem = (struct rxe_pool_elem *)((u8 *)obj + pool->elem_offset); elem->pool = pool; + elem->obj = obj;kref_init (&elem->ref_cnt);return obj;
@@ -350,7 +352,7 @@ void *rxe_alloc_locked(struct rxe_pool *pool)void *rxe_alloc (struct rxe_pool *pool) {struct rxe_pool_elem *elem; -u8 *obj; +void *obj;if (atomic_inc_return (&pool->num_elem) > pool->max_elem) goto out_cnt;
@@ -359,9 +361,10 @@ void *rxe_alloc(struct rxe_pool *pool)if (!obj) goto out_cnt; - elem = (struct rxe_pool_elem *)(obj + pool->elem_offset); + elem = (struct rxe_pool_elem *)((u8 *)obj + pool->elem_offset); elem->pool = pool; + elem->obj = obj;kref_init (&elem->ref_cnt);return obj;
@@ -377,6 +380,7 @@ int __rxe_add_to_pool(struct rxe_pool *pool, struct rxe_pool_elem *elem) goto out_cnt; elem->pool = pool; + elem->obj = (u8 *)elem - pool->elem_offset;kref_init (&elem->ref_cnt);return 0 ;
@@ -391,13 +395,13 @@ void rxe_elem_release(struct kref *kref)struct rxe_pool_elem *elem = container_of(kref, struct rxe_pool_elem, ref_cnt);struct rxe_pool *pool = elem->pool; -u8 *obj; +void *obj;if (pool->cleanup) pool->cleanup (elem);if (!(pool->flags &RXE_POOL_NO_ALLOC )) { - obj = (u8 *)elem - pool->elem_offset; + obj = elem->obj;kfree (obj); }
@@ -408,7 +412,7 @@ void *rxe_pool_get_index_locked(struct rxe_pool *pool, u32 index) {struct rb_node *node;struct rxe_pool_elem *elem; -u8 *obj; +void *obj; node = pool->index.tree.rb_node;
@@ -425,7 +429,7 @@ void *rxe_pool_get_index_locked(struct rxe_pool *pool, u32 index)if (node) {kref_get (&elem->ref_cnt); - obj = (u8 *)elem - pool->elem_offset; + obj = elem->obj; }else { obj =NULL ; }
@@ -435,7 +439,7 @@ void *rxe_pool_get_index_locked(struct rxe_pool *pool, u32 index)void *rxe_pool_get_index (struct rxe_pool *pool,u32 index) { -u8 *obj; +void *obj;read_lock_bh (&pool->pool_lock); obj =rxe_pool_get_index_locked (pool, index);
@@ -448,7 +452,7 @@ void *rxe_pool_get_key_locked(struct rxe_pool *pool, void *key) {struct rb_node *node;struct rxe_pool_elem *elem; -u8 *obj; +void *obj;int cmp; node = pool->key.tree.rb_node;
@@ -469,7 +473,7 @@ void *rxe_pool_get_key_locked(struct rxe_pool *pool, void *key)if (node) {kref_get (&elem->ref_cnt); - obj = (u8 *)elem - pool->elem_offset; + obj = elem->obj; }else { obj =NULL ; }
@@ -479,7 +483,7 @@ void *rxe_pool_get_key_locked(struct rxe_pool *pool, void *key)void *rxe_pool_get_key (struct rxe_pool *pool,void *key) { -u8 *obj; +void *obj;read_lock_bh (&pool->pool_lock); obj =rxe_pool_get_key_locked (pool, key);
drivers/infiniband/sw/rxe/rxe_pool.h +1 -0
@@ -32,6 +32,7 @@ enum rxe_elem_type {struct rxe_pool_elem {struct rxe_pool *pool; +void *obj;struct kref ref_cnt;struct list_head list;
Signature · 1 line
2.30.2
PATCH v4 5/13 RDMA/rxe: Replace RB tree by xarray for indexes
In reply to Bob PearsonCurrently the rxe driver uses red-black trees to add indices and keys to the rxe object pool. Linux xarrays provide a better way to implement the same functionality for indices but not keys. This patch replaces edd-black trees by xarrays for indexed objects. Since caller managed locks for indexed objects are not used these APIs were deleted as well.
--- drivers/infiniband/sw/rxe/rxe.c | 100 ++------------ drivers/infiniband/sw/rxe/rxe_mr.c | 1 - drivers/infiniband/sw/rxe/rxe_mw.c | 4 - drivers/infiniband/sw/rxe/rxe_pool.c | 189 ++++++-------------------- drivers/infiniband/sw/rxe/rxe_pool.h | 44 +----- drivers/infiniband/sw/rxe/rxe_verbs.c | 12 -- 6 files changed, 62 insertions(+), 288 deletions(-)
drivers/infiniband/sw/rxe/rxe.c +14 -86
@@ -116,97 +116,31 @@ static void rxe_init_ports(struct rxe_dev *rxe) }/* init pools of managed objects */ -static int rxe_init_pools (struct rxe_dev *rxe) +static void rxe_init_pools (struct rxe_dev *rxe) { -int err; - - err =rxe_pool_init (rxe, &rxe->uc_pool,RXE_TYPE_UC , - rxe->max_ucontext); -if (err) - goto err1; - - err =rxe_pool_init (rxe, &rxe->pd_pool,RXE_TYPE_PD , - rxe->attr.max_pd); -if (err) - goto err2; - - err =rxe_pool_init (rxe, &rxe->ah_pool,RXE_TYPE_AH , - rxe->attr.max_ah); -if (err) - goto err3; - - err =rxe_pool_init (rxe, &rxe->srq_pool,RXE_TYPE_SRQ , - rxe->attr.max_srq); -if (err) - goto err4; - - err =rxe_pool_init (rxe, &rxe->qp_pool,RXE_TYPE_QP , - rxe->attr.max_qp); -if (err) - goto err5; - - err =rxe_pool_init (rxe, &rxe->cq_pool,RXE_TYPE_CQ , - rxe->attr.max_cq); -if (err) - goto err6; - - err =rxe_pool_init (rxe, &rxe->mr_pool,RXE_TYPE_MR , - rxe->attr.max_mr); -if (err) - goto err7; - - err =rxe_pool_init (rxe, &rxe->mw_pool,RXE_TYPE_MW , - rxe->attr.max_mw); -if (err) - goto err8; - - err =rxe_pool_init (rxe, &rxe->mc_grp_pool,RXE_TYPE_MC_GRP , +rxe_pool_init (rxe, &rxe->uc_pool,RXE_TYPE_UC , rxe->max_ucontext); +rxe_pool_init (rxe, &rxe->pd_pool,RXE_TYPE_PD , rxe->attr.max_pd); +rxe_pool_init (rxe, &rxe->ah_pool,RXE_TYPE_AH , rxe->attr.max_ah); +rxe_pool_init (rxe, &rxe->srq_pool,RXE_TYPE_SRQ , rxe->attr.max_srq); +rxe_pool_init (rxe, &rxe->qp_pool,RXE_TYPE_QP , rxe->attr.max_qp); +rxe_pool_init (rxe, &rxe->cq_pool,RXE_TYPE_CQ , rxe->attr.max_cq); +rxe_pool_init (rxe, &rxe->mr_pool,RXE_TYPE_MR , rxe->attr.max_mr); +rxe_pool_init (rxe, &rxe->mw_pool,RXE_TYPE_MW , rxe->attr.max_mw); +rxe_pool_init (rxe, &rxe->mc_grp_pool,RXE_TYPE_MC_GRP , rxe->attr.max_mcast_grp); -if (err) - goto err9; - - err =rxe_pool_init (rxe, &rxe->mc_elem_pool,RXE_TYPE_MC_ELEM , +rxe_pool_init (rxe, &rxe->mc_elem_pool,RXE_TYPE_MC_ELEM , rxe->attr.max_total_mcast_qp_attach); -if (err) - goto err10; - -return 0 ; - -err10: -rxe_pool_cleanup (&rxe->mc_grp_pool); -err9: -rxe_pool_cleanup (&rxe->mw_pool); -err8: -rxe_pool_cleanup (&rxe->mr_pool); -err7: -rxe_pool_cleanup (&rxe->cq_pool); -err6: -rxe_pool_cleanup (&rxe->qp_pool); -err5: -rxe_pool_cleanup (&rxe->srq_pool); -err4: -rxe_pool_cleanup (&rxe->ah_pool); -err3: -rxe_pool_cleanup (&rxe->pd_pool); -err2: -rxe_pool_cleanup (&rxe->uc_pool); -err1: -return err; }/* initialize rxe device state */ -static int rxe_init (struct rxe_dev *rxe) +static void rxe_init (struct rxe_dev *rxe) { -int err; -/* init default device parameters */ rxe_init_device_param (rxe);rxe_init_ports (rxe); - err =rxe_init_pools (rxe); -if (err) -return err; +rxe_init_pools (rxe);/* init pending mmap list */ spin_lock_init (&rxe->mmap_offset_lock);
@@ -214,8 +148,6 @@ static int rxe_init(struct rxe_dev *rxe)INIT_LIST_HEAD (&rxe->pending_mmaps);mutex_init (&rxe->usdev_lock); - -return 0 ; }void rxe_set_mtu (struct rxe_dev *rxe,unsigned int ndev_mtu)
@@ -237,11 +169,7 @@ void rxe_set_mtu(struct rxe_dev *rxe, unsigned int ndev_mtu)*/ int rxe_add (struct rxe_dev *rxe,unsigned int mtu,const char *ibdev_name) { -int err; - - err =rxe_init (rxe); -if (err) -return err; +rxe_init (rxe);rxe_set_mtu (rxe, mtu);
drivers/infiniband/sw/rxe/rxe_mr.c +0 -1
@@ -693,7 +693,6 @@ int rxe_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata) mr->state =RXE_MR_STATE_INVALID ;rxe_drop_ref (mr_pd (mr)); -rxe_drop_index (mr);rxe_drop_ref (mr);return 0 ;
drivers/infiniband/sw/rxe/rxe_mw.c +0 -4
@@ -20,7 +20,6 @@ int rxe_alloc_mw(struct ib_mw *ibmw, struct ib_udata *udata)return ret; } -rxe_add_index (mw); mw->rkey = ibmw->rkey = (mw->elem.index <<8 ) |rxe_get_next_key (-1 ); mw->state = (mw->ibmw.type ==IB_MW_TYPE_2 ) ?RXE_MW_STATE_FREE :RXE_MW_STATE_VALID ;
@@ -332,7 +331,4 @@ struct rxe_mw *rxe_lookup_mw(struct rxe_qp *qp, int access, u32 rkey)void rxe_mw_cleanup (struct rxe_pool_elem *elem) { -struct rxe_mw *mw =container_of (elem,typeof (*mw), elem); - -rxe_drop_index (mw); }
drivers/infiniband/sw/rxe/rxe_pool.c +43 -146
289 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_pool.h +5 -39
74 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_verbs.c +0 -12
@@ -181,7 +181,6 @@ static int rxe_create_ah(struct ib_ah *ibah,return err;/* create index > 0 */ -rxe_add_index (ah); ah->ah_num = ah->elem.index;if (uresp) {
@@ -189,7 +188,6 @@ static int rxe_create_ah(struct ib_ah *ibah, err =copy_to_user (&uresp->ah_num, &ah->ah_num,sizeof (uresp->ah_num));if (err) { -rxe_drop_index (ah);rxe_drop_ref (ah);return -EFAULT ; }
@@ -230,7 +228,6 @@ static int rxe_destroy_ah(struct ib_ah *ibah, u32 flags) {struct rxe_ah *ah =to_rah (ibah); -rxe_drop_index (ah);rxe_drop_ref (ah);return 0 ; }
@@ -437,7 +434,6 @@ static int rxe_create_qp(struct ib_qp *ibqp, struct ib_qp_init_attr *init,if (err)return err; -rxe_add_index (qp); err =rxe_qp_from_init (rxe, qp, pd, init, uresp, ibqp->pd, udata);if (err) goto qp_init;
@@ -445,7 +441,6 @@ static int rxe_create_qp(struct ib_qp *ibqp, struct ib_qp_init_attr *init,return 0 ; qp_init: -rxe_drop_index (qp);rxe_drop_ref (qp);return err; }
@@ -490,7 +485,6 @@ static int rxe_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)struct rxe_qp *qp =to_rqp (ibqp);rxe_qp_destroy (qp); -rxe_drop_index (qp);rxe_drop_ref (qp);return 0 ; }
@@ -893,7 +887,6 @@ static struct ib_mr *rxe_get_dma_mr(struct ib_pd *ibpd, int access)if (!mr)return ERR_PTR (-ENOMEM ); -rxe_add_index (mr);rxe_add_ref (pd);rxe_mr_init_dma (pd, access, mr);
@@ -917,7 +910,6 @@ static struct ib_mr *rxe_reg_user_mr(struct ib_pd *ibpd, goto err2; } -rxe_add_index (mr);rxe_add_ref (pd);
@@ -929,7 +921,6 @@ static struct ib_mr *rxe_reg_user_mr(struct ib_pd *ibpd, err3:rxe_drop_ref (pd); -rxe_drop_index (mr);rxe_drop_ref (mr); err2:return ERR_PTR (err);
@@ -952,8 +943,6 @@ static struct ib_mr *rxe_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type mr_type, goto err1; } -rxe_add_index (mr); -rxe_add_ref (pd); err =rxe_mr_init_fast (pd, max_num_sg, mr);
@@ -964,7 +953,6 @@ static struct ib_mr *rxe_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type mr_type, err2:rxe_drop_ref (pd); -rxe_drop_index (mr);rxe_drop_ref (mr); err1:return ERR_PTR (err);
Signature · 1 line
2.30.2
Jason Gunthorpe
maintainerRe: [PATCH for-next v4 05/13] RDMA/rxe: Replace RB tree by xarray for indexes
drivers/infiniband/sw/rxe/rxe_pool.c void *rxe_alloc_locked(struct rxe_pool *pool) · line 235
Quoted 8 lines · Bob Pearson
drivers/infiniband/sw/rxe/rxe_pool.c +4
-0@@ -342,8 +229,18 @@ void *rxe_alloc_locked(struct rxe_pool *pool) elem->obj = obj;kref_init (&elem->ref_cnt); +if (pool->flags &RXE_POOL_INDEX ) { + err =xa_alloc_cyclic_bh (&pool->xarray.xa, &elem->index, elem, + pool->xarray.limit, + &pool->xarray.next,GFP_ATOMIC );
This uses the _bh lock
drivers/infiniband/sw/rxe/rxe_pool.c void rxe_elem_release(struct kref *kref) · line 319
Quoted 7 lines
drivers/infiniband/sw/rxe/rxe_pool.c +2
-0void rxe_elem_release (struct kref *kref)@@ -397,6 +315,9 @@ void rxe_elem_release(struct kref *kref)struct rxe_pool *pool = elem->pool;void *obj; +if (pool->flags &RXE_POOL_INDEX ) +xa_erase (&pool->xarray.xa, elem->index);
But this doesn't ?
Shouldn't they all be the same?
And why is it a bh lock anyhow? Can you add some comments saying which APIs here are called from the softirq?
drivers/infiniband/sw/rxe/rxe_pool.c void rxe_elem_release(struct kref *kref) · line 341
Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_pool.c +3
-0+void *rxe_pool_get_index (struct rxe_pool *pool,u32 index) {struct rxe_pool_elem *elem;void *obj; + elem =xa_load (&pool->xarray.xa, index); +if (elem) {kref_get (&elem->ref_cnt); obj = elem->obj; }else {
And why doesn't this use after free elem? This pattern is only safe when using RCU or when holding the spinlock across the kref_get.. Since you can't use rcu with the core allocated objects it seems this needs a xa_lock?
Jason
PATCH v4 6/13 RDMA/rxe: Remove #include "rxe_loc.h" from rxe_pool.c
In reply to Bob Pearsonrxe_loc.h is already included in rxe.h so do not include it in rxe_pool.c
--- drivers/infiniband/sw/rxe/rxe_pool.c | 1 - 1 file changed, 1 deletion(-)
drivers/infiniband/sw/rxe/rxe_pool.c +0 -1
@@ -5,7 +5,6 @@*/ #include "rxe.h" -#include "rxe_loc.h" static const struct rxe_type_info { const char *name;
Signature · 1 line
2.30.2
PATCH v4 7/13 RDMA/rxe: Remove some #defines from rxe_pool.h
In reply to Bob PearsonRXE_POOL_ALIGN is only used in rxe_pool.c so move RXE_POOL_ALIGN to rxe_pool.c from rxe_pool.h.RXE_POOL_CACHE_FLAGS is never used so it is deleted from rxe_pool.h
--- drivers/infiniband/sw/rxe/rxe_pool.c | 2 ++ drivers/infiniband/sw/rxe/rxe_pool.h | 3 --- 2 files changed, 2 insertions(+), 3 deletions(-)
drivers/infiniband/sw/rxe/rxe_pool.c +2 -0
@@ -6,6 +6,8 @@#include "rxe.h" +#define RXE_POOL_ALIGN (16) + static const struct rxe_type_info { const char *name; size_t size;
drivers/infiniband/sw/rxe/rxe_pool.h +0 -3
@@ -7,9 +7,6 @@#ifndef RXE_POOL_H #define RXE_POOL_H -#define RXE_POOL_ALIGN (16) -#define RXE_POOL_CACHE_FLAGS (0) -enum rxe_pool_flags {RXE_POOL_INDEX =BIT (1 ), RXE_POOL_KEY = BIT(2),
Signature · 1 line
2.30.2
PATCH v4 8/13 RDMA/rxe: Reverse the sense of RXE_POOL_NO_ALLOC
In reply to Bob PearsonSince most rxe objects are now allocated in rdma-core change the sense of RXE_POOL_NO_ALLOC to RXE_POOL_ALLOC. This makes the code easier to understand.
--- drivers/infiniband/sw/rxe/rxe_pool.c | 18 ++++++++---------- drivers/infiniband/sw/rxe/rxe_pool.h | 2 +- 2 files changed, 9 insertions(+), 11 deletions(-)
drivers/infiniband/sw/rxe/rxe_pool.c +8 -10
@@ -23,19 +23,17 @@ static const struct rxe_type_info { .name = "rxe-uc", .size = sizeof(struct rxe_ucontext), .elem_offset = offsetof(struct rxe_ucontext, elem), - .flags = RXE_POOL_NO_ALLOC, }, [RXE_TYPE_PD] = { .name = "rxe-pd", .size = sizeof(struct rxe_pd), .elem_offset = offsetof(struct rxe_pd, elem), - .flags = RXE_POOL_NO_ALLOC, }, [RXE_TYPE_AH] = { .name = "rxe-ah", .size = sizeof(struct rxe_ah), .elem_offset = offsetof(struct rxe_ah, elem), - .flags = RXE_POOL_INDEX | RXE_POOL_NO_ALLOC, + .flags = RXE_POOL_INDEX, .min_index = RXE_MIN_AH_INDEX, .max_index = RXE_MAX_AH_INDEX, },
@@ -43,7 +41,7 @@ static const struct rxe_type_info { .name = "rxe-srq", .size = sizeof(struct rxe_srq), .elem_offset = offsetof(struct rxe_srq, elem), - .flags = RXE_POOL_INDEX | RXE_POOL_NO_ALLOC, + .flags = RXE_POOL_INDEX, .min_index = RXE_MIN_SRQ_INDEX, .max_index = RXE_MAX_SRQ_INDEX, },
@@ -52,7 +50,7 @@ static const struct rxe_type_info { .size = sizeof(struct rxe_qp), .elem_offset = offsetof(struct rxe_qp, elem), .cleanup = rxe_qp_cleanup, - .flags = RXE_POOL_INDEX | RXE_POOL_NO_ALLOC, + .flags = RXE_POOL_INDEX, .min_index = RXE_MIN_QP_INDEX, .max_index = RXE_MAX_QP_INDEX, },
@@ -60,7 +58,6 @@ static const struct rxe_type_info { .name = "rxe-cq", .size = sizeof(struct rxe_cq), .elem_offset = offsetof(struct rxe_cq, elem), - .flags = RXE_POOL_NO_ALLOC, .cleanup = rxe_cq_cleanup, }, [RXE_TYPE_MR] = {
@@ -68,7 +65,7 @@ static const struct rxe_type_info { .size = sizeof(struct rxe_mr), .elem_offset = offsetof(struct rxe_mr, elem), .cleanup = rxe_mr_cleanup, - .flags = RXE_POOL_INDEX, + .flags = RXE_POOL_INDEX | RXE_POOL_ALLOC, .min_index = RXE_MIN_MR_INDEX, .max_index = RXE_MAX_MR_INDEX, },
@@ -77,7 +74,7 @@ static const struct rxe_type_info { .size = sizeof(struct rxe_mw), .elem_offset = offsetof(struct rxe_mw, elem), .cleanup = rxe_mw_cleanup, - .flags = RXE_POOL_INDEX | RXE_POOL_NO_ALLOC, + .flags = RXE_POOL_INDEX, .min_index = RXE_MIN_MW_INDEX, .max_index = RXE_MAX_MW_INDEX, },
@@ -86,7 +83,7 @@ static const struct rxe_type_info { .size = sizeof(struct rxe_mc_grp), .elem_offset = offsetof(struct rxe_mc_grp, elem), .cleanup = rxe_mc_cleanup, - .flags = RXE_POOL_KEY, + .flags = RXE_POOL_KEY | RXE_POOL_ALLOC, .key_offset = offsetof(struct rxe_mc_grp, mgid), .key_size = sizeof(union ib_gid), },
@@ -94,6 +91,7 @@ static const struct rxe_type_info { .name = "rxe-mc_elem", .size = sizeof(struct rxe_mc_elem), .elem_offset = offsetof(struct rxe_mc_elem, elem), + .flags = RXE_POOL_ALLOC, }, };
@@ -322,7 +320,7 @@ void rxe_elem_release(struct kref *kref)if (pool->cleanup) pool->cleanup (elem); -if (!(pool->flags &RXE_POOL_NO_ALLOC )) { +if (pool->flags &RXE_POOL_ALLOC ) { obj = elem->obj;kfree (obj); }
drivers/infiniband/sw/rxe/rxe_pool.h +1 -1
@@ -10,7 +10,7 @@enum rxe_pool_flags {RXE_POOL_INDEX =BIT (1 ),RXE_POOL_KEY =BIT (2 ), -RXE_POOL_NO_ALLOC =BIT (4 ), +RXE_POOL_ALLOC =BIT (4 ), };enum rxe_elem_type {
Signature · 1 line
2.30.2
PATCH v4 9/13 RDMA/rxe: Replaced keyed rxe objects by indexed objects
In reply to Bob PearsonReplace the keyed type rxe objects by xarrays with a 64 bit index. This was only used for mgids. Here construct an index for each mgid that is very likely to be unique. With this change there is no longer a requirement for xxx_locked versions of the pool APIs.
--- drivers/infiniband/sw/rxe/rxe_loc.h | 10 +- drivers/infiniband/sw/rxe/rxe_mcast.c | 162 +++++++++++----- drivers/infiniband/sw/rxe/rxe_mw.c | 2 +- drivers/infiniband/sw/rxe/rxe_pool.c | 256 +++++++++----------------- drivers/infiniband/sw/rxe/rxe_pool.h | 67 +------ drivers/infiniband/sw/rxe/rxe_recv.c | 3 +- drivers/infiniband/sw/rxe/rxe_verbs.c | 32 ++-- 7 files changed, 230 insertions(+), 302 deletions(-)
drivers/infiniband/sw/rxe/rxe_loc.h +3 -7
@@ -40,17 +40,13 @@ void rxe_cq_disable(struct rxe_cq *cq);void rxe_cq_cleanup (struct rxe_pool_elem *arg);/* rxe_mcast.c */ -int rxe_mcast_get_grp (struct rxe_dev *rxe,union ib_gid *mgid, -struct rxe_mc_grp **grp_p); - +struct rxe_mc_grp *rxe_mcast_get_grp (struct rxe_dev *rxe,union ib_gid *mgid, +int alloc);int rxe_mcast_add_grp_elem (struct rxe_dev *rxe,struct rxe_qp *qp,struct rxe_mc_grp *grp); -int rxe_mcast_drop_grp_elem (struct rxe_dev *rxe,struct rxe_qp *qp, -union ib_gid *mgid); - +struct rxe_mc_grp *grp);void rxe_drop_all_mcast_groups (struct rxe_qp *qp); -void rxe_mc_cleanup (struct rxe_pool_elem *arg);/* rxe_mmap.c */
drivers/infiniband/sw/rxe/rxe_mcast.c +111 -51
224 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_mw.c +1 -1
@@ -14,7 +14,7 @@ int rxe_alloc_mw(struct ib_mw *ibmw, struct ib_udata *udata)rxe_add_ref (pd); - ret =rxe_add_to_pool (&rxe->mw_pool, mw); + ret =rxe_pool_add (&rxe->mw_pool, mw);if (ret) {rxe_drop_ref (pd);return ret;
drivers/infiniband/sw/rxe/rxe_pool.c +84 -172
392 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_pool.h +10 -57
98 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_recv.c +1 -2
@@ -245,8 +245,7 @@ static void rxe_rcv_mcast_pkt(struct rxe_dev *rxe, struct sk_buff *skb)else if (skb->protocol ==htons (ETH_P_IPV6 ))memcpy (&dgid, &ipv6_hdr (skb)->daddr,sizeof (dgid)); -/* lookup mcast group corresponding to mgid, takes a ref */ - mcg =rxe_pool_get_key (&rxe->mc_grp_pool, &dgid); + mcg =rxe_mcast_get_grp (rxe, &dgid,0 );if (!mcg) goto drop;/* mcast group not registered */
drivers/infiniband/sw/rxe/rxe_verbs.c +20 -12
@@ -108,7 +108,7 @@ static int rxe_alloc_ucontext(struct ib_ucontext *ibuc, struct ib_udata *udata)struct rxe_dev *rxe =to_rdev (ibuc->device);struct rxe_ucontext *uc =to_ruc (ibuc); -return rxe_add_to_pool (&rxe->uc_pool, uc); +return rxe_pool_add (&rxe->uc_pool, uc); }static void rxe_dealloc_ucontext (struct ib_ucontext *ibuc)
@@ -142,7 +142,7 @@ static int rxe_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)struct rxe_dev *rxe =to_rdev (ibpd->device);struct rxe_pd *pd =to_rpd (ibpd); -return rxe_add_to_pool (&rxe->pd_pool, pd); +return rxe_pool_add (&rxe->pd_pool, pd); }static int rxe_dealloc_pd (struct ib_pd *ibpd,struct ib_udata *udata)
@@ -176,7 +176,7 @@ static int rxe_create_ah(struct ib_ah *ibah,if (err)return err; - err =rxe_add_to_pool (&rxe->ah_pool, ah); + err =rxe_pool_add (&rxe->ah_pool, ah);if (err)return err;
@@ -299,7 +299,7 @@ static int rxe_create_srq(struct ib_srq *ibsrq, struct ib_srq_init_attr *init,if (err) goto err1; - err =rxe_add_to_pool (&rxe->srq_pool, srq); + err =rxe_pool_add (&rxe->srq_pool, srq);if (err) goto err1;
@@ -430,7 +430,7 @@ static int rxe_create_qp(struct ib_qp *ibqp, struct ib_qp_init_attr *init, qp->is_user = false; } - err =rxe_add_to_pool (&rxe->qp_pool, qp); + err =rxe_pool_add (&rxe->qp_pool, qp);if (err)return err;
@@ -787,7 +787,7 @@ static int rxe_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,if (err)return err; -return rxe_add_to_pool (&rxe->cq_pool, cq); +return rxe_pool_add (&rxe->cq_pool, cq); }static int rxe_destroy_cq (struct ib_cq *ibcq,struct ib_udata *udata)
@@ -984,15 +984,14 @@ static int rxe_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg,static int rxe_attach_mcast (struct ib_qp *ibqp,union ib_gid *mgid,u16 mlid) { -int err;struct rxe_dev *rxe =to_rdev (ibqp->device);struct rxe_qp *qp =to_rqp (ibqp);struct rxe_mc_grp *grp; +int err; -/* takes a ref on grp if successful */ - err =rxe_mcast_get_grp (rxe, mgid, &grp); -if (err) -return err; + grp =rxe_mcast_get_grp (rxe, mgid,1 ); +if (!grp) +return -ENOMEM ; err =rxe_mcast_add_grp_elem (rxe, qp, grp);
@@ -1004,8 +1003,17 @@ static int rxe_detach_mcast(struct ib_qp *ibqp, union ib_gid *mgid, u16 mlid) {struct rxe_dev *rxe =to_rdev (ibqp->device);struct rxe_qp *qp =to_rqp (ibqp); +struct rxe_mc_grp *grp; +int err; + + grp =rxe_mcast_get_grp (rxe, mgid,0 ); +if (!grp) +return -EINVAL ; -return rxe_mcast_drop_grp_elem (rxe, qp, mgid); + err =rxe_mcast_drop_grp_elem (rxe, qp, grp); + +rxe_drop_ref (grp); +return err; } static ssize_t parent_show(struct device *device,
Signature · 1 line
2.30.2
Jason Gunthorpe
maintainerRe: [PATCH for-next v4 09/13] RDMA/rxe: Replaced keyed rxe objects by indexed objects
Quoted 11 lines · Bob Pearson
drivers/infiniband/sw/rxe/rxe_mcast.c +11
-0+if (mgid->raw[10 ] ==0xff && mgid->raw[11 ] ==0xff ) { +if ((mgid->raw[12 ] &0xf0 ) !=0xe0 ) +pr_info ("mgid is not an ipv4 mc address\n" ); + +/* mgid is a mapped IPV4 multicast address +* use the 32 bits as an index which will be +* unique +*/ + index =be32_to_cpu (val32[3 ]); + }else { +if (mgid->raw[0 ] !=0xff )
drivers/infiniband/sw/rxe/rxe_mcast.c line 42
Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_mcast.c +10
-0+pr_info ("mgid is not an ipv6 mc address\n" ); + +/* mgid is an IPV6 multicast address which won't +* fit into the index so construct the index +* from the four 32 bit words in mgid. +* If there is a collision treat it like +* no memory and return NULL +*/ + index =be32_to_cpu (val32[0 ] ^ val32[1 ]); + index = (index <<32 ) |be32_to_cpu (val32[2 ] ^ val32[3 ]);
I'm not sure failing is such a good thing, can it chain them on a linked list or something?
Jason
Re: [PATCH for-next v4 09/13] RDMA/rxe: Replaced keyed rxe objects by indexed objects
Quoted 26 lines · Jason Gunthorpe
Quoted 11 lines · Bob Pearson
drivers/infiniband/sw/rxe/rxe_mcast.c +11
-0+if (mgid->raw[10 ] ==0xff && mgid->raw[11 ] ==0xff ) { +if ((mgid->raw[12 ] &0xf0 ) !=0xe0 ) +pr_info ("mgid is not an ipv4 mc address\n" ); + +/* mgid is a mapped IPV4 multicast address +* use the 32 bits as an index which will be +* unique +*/ + index =be32_to_cpu (val32[3 ]); + }else { +if (mgid->raw[0 ] !=0xff )drivers/infiniband/sw/rxe/rxe_mcast.c line 42
Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_mcast.c +10
-0+pr_info ("mgid is not an ipv6 mc address\n" ); + +/* mgid is an IPV6 multicast address which won't +* fit into the index so construct the index +* from the four 32 bit words in mgid. +* If there is a collision treat it like +* no memory and return NULL +*/ + index =be32_to_cpu (val32[0 ] ^ val32[1 ]); + index = (index <<32 ) |be32_to_cpu (val32[2 ] ^ val32[3 ]);I'm not sure failing is such a good thing, can it chain them on a linked list or something?
Quoted 2 lines
Jason
I had developed a version of the proposed patch which did what you suggest and felt that it wasn't an improvement over the current code. Just too complicated. This patch is only marginally cleaner than what is already there. I am leaning towards just leaving the current red-black tree based code in place for mgids which don't fit nicely into xarrays which however are quite a bit better for 32 bit indices (qpns, qkeys, rkeys, etc.) Multicast is not an important use case for RoCE since plain IP is a better solution. Unless you think otherwise we should just drop this patch.
Bob
Re: [PATCH for-next v4 09/13] RDMA/rxe: Replaced keyed rxe objects by indexed objects
Hi Bob, good to hear from you
Quoted 1 line · Bob Pearson
Multicast is not an important use case for RoCE since plain IP is a better solution.
Multicast is an important use case for ROCE especially in the area of finance and there is commercial middleware that depends on it. Plain IP suffers from the high kernel overhead. High packet rates are only possible through the RDMA apis.
Greetings,
Christoph
PATCH v4 10/13 RDMA/rxe: Prevent taking references to dead objects
In reply to Bob PearsonCurrently rxe_add_ref() calls kref_get() which increments the reference count even if the object has already been released. Change this to refcount_inc_not_zero() which will only succeed if the current ref count is larger than zero. This change exposes some reference counting errors which will be fixed in the following patches.
--- drivers/infiniband/sw/rxe/rxe_pool.h | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
drivers/infiniband/sw/rxe/rxe_pool.h +9 -1
@@ -7,6 +7,8 @@#ifndef RXE_POOL_H #define RXE_POOL_H +#include <linux/refcount.h> +enum rxe_pool_flags {RXE_POOL_AUTO_INDEX =BIT (1 ), RXE_POOL_EXT_INDEX = BIT(2),
@@ -70,9 +72,15 @@ int __rxe_pool_add(struct rxe_pool *pool, struct rxe_pool_elem *elem);void *rxe_pool_get_index (struct rxe_pool *pool,unsigned long index); -#define rxe_add_ref (obj) kref_get(&(obj)->elem.ref_cnt) +static inline int __rxe_add_ref (struct rxe_pool_elem *elem) +{ +return refcount_inc_not_zero (&elem->ref_cnt.refcount); +} + +#define rxe_add_ref (obj) __rxe_add_ref(&(obj)->elem)void rxe_elem_release (struct kref *kref); +#define rxe_drop_ref (obj) kref_put(&(obj)->elem.ref_cnt, rxe_elem_release)#endif /* RXE_POOL_H */
Signature · 1 line
2.30.2
Jason Gunthorpe
maintainerRe: [PATCH for-next v4 10/13] RDMA/rxe: Prevent taking references to dead objects
Quoted 24 lines · Bob Pearson
Currently
rxe_add_ref()callskref_get()which increments the reference count even if the object has already been released. Change this torefcount_inc_not_zero()which will only succeed if the current ref count is larger than zero. This change exposes some reference counting errors which will be fixed in the following patches.Signed-off-by Bob Pearson <[email protected]>drivers/infiniband/sw/rxe/rxe_pool.h | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)drivers/infiniband/sw/rxe/rxe_pool.h +3
-0+++ b/drivers/infiniband/sw/rxe/rxe_pool.h@@ -7,6 +7,8 @@#ifndef RXE_POOL_H #define RXE_POOL_H +#include <linux/refcount.h> +enum rxe_pool_flags {RXE_POOL_AUTO_INDEX =BIT (1 ), RXE_POOL_EXT_INDEX = BIT(2),
drivers/infiniband/sw/rxe/rxe_pool.h int __rxe_pool_add(struct rxe_pool *pool, struct rxe_pool_elem *elem); · line 78
Quoted 8 lines
drivers/infiniband/sw/rxe/rxe_pool.h +4
-1@@ -70,9 +72,15 @@ int __rxe_pool_add(struct rxe_pool *pool, struct rxe_pool_elem *elem);void *rxe_pool_get_index (struct rxe_pool *pool,unsigned long index); -#define rxe_add_ref (obj) kref_get(&(obj)->elem.ref_cnt) +static inline int __rxe_add_ref (struct rxe_pool_elem *elem) +{ +return refcount_inc_not_zero (&elem->ref_cnt.refcount); +}
Don't reach inside a kref to touch the refcount. kref_get_unless_zero() is the api for krefs
I'm not sure how any of this works, ie rxe_create_qp() stuffs a core allocated object into the pool, and various places do refcounting on it
But then rxe_destroy_qp doesn't seem to do anything with the refcounts, it just blindly lets things go to kfree in the core.
Seems really confused..
Jason
Re: [PATCH for-next v4 10/13] RDMA/rxe: Prevent taking references to dead objects
Quoted 28 lines · Jason Gunthorpe
Quoted 27 lines · Bob Pearson
Currently
rxe_add_ref()callskref_get()which increments the reference count even if the object has already been released. Change this torefcount_inc_not_zero()which will only succeed if the current ref count is larger than zero. This change exposes some reference counting errors which will be fixed in the following patches.Signed-off-by Bob Pearson <[email protected]>drivers/infiniband/sw/rxe/rxe_pool.h | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)drivers/infiniband/sw/rxe/rxe_pool.h +3
-0+++ b/drivers/infiniband/sw/rxe/rxe_pool.h@@ -7,6 +7,8 @@#ifndef RXE_POOL_H #define RXE_POOL_H +#include <linux/refcount.h> +enum rxe_pool_flags {RXE_POOL_AUTO_INDEX =BIT (1 ), RXE_POOL_EXT_INDEX = BIT(2),@@ -70,9 +72,15 @@ int __rxe_pool_add(struct rxe_pool *pool, struct rxe_pool_elem *elem);void *rxe_pool_get_index (struct rxe_pool *pool,unsigned long index);
Quoted 8 lines
drivers/infiniband/sw/rxe/rxe_pool.h int __rxe_pool_add(struct rxe_pool *pool, struct rxe_pool_elem *elem); · line 78
Quoted 5 lines
drivers/infiniband/sw/rxe/rxe_pool.h +4
-1-#define rxe_add_ref (obj) kref_get(&(obj)->elem.ref_cnt) +static inline int __rxe_add_ref (struct rxe_pool_elem *elem) +{ +return refcount_inc_not_zero (&elem->ref_cnt.refcount); +}Don't reach inside a kref to touch the refcount.
kref_get_unless_zero()is the api for krefs
Yes. Thanks for pointing this out. I had started using the refcount APIs instead of the kref APIs for a different one that is not supported by kref and then quit using it. This is a cleaner solution.
Quoted 4 lines
I'm not sure how any of this works, ie
rxe_create_qp()stuffs a core allocated object into the pool, and various places do refcounting on it
I have been having a long running discussion with Leon about rxe's use of ref counting. His opinion is that it isn't required at all and that rdma-core is doing enough. I think that we either have to push the problem onto the apps (e.g. return EAGAIN or EBUSY) or let the driver sleep if there are currently active packets being processed with a mutex or semaphore. Additionally we need to cleanup some bad code that makes things not work as mentioned.
Quoted 8 lines
But then
rxe_destroy_qpdoesn't seem to do anything with the refcounts, it just blindly lets things go to kfree in the core.Seems really confused..
Jason
Jason Gunthorpe
maintainerRe: [PATCH for-next v4 10/13] RDMA/rxe: Prevent taking references to dead objects
Quoted 3 lines · Bob Pearson
let the driver sleep if there are currently active packets being processed with a mutex or semaphore. Additionally we need to cleanup some bad code that makes things not work as mentioned.
Drivers have to sleep in their destroy to fully fence any activity related to the destroying object.
It can't keep operating the object after destroy returns, that is not our semantic
Jason
PATCH v4 11/13 RDMA/rxe: Fix ref error in rxe_av.c
In reply to Bob PearsonThe commit referenced below can take a reference to the AH which is never dropped. This only happens in the UD request path. This patch optionally passes that AH back to the caller so that it can hold the reference while the AV is being accessed and then drop it. Code to do this is added to rxe_req.c. The AV is also passed to rxe_prepare in rxe_net.c as an optimization.
--- drivers/infiniband/sw/rxe/rxe_av.c | 19 +++++++++- drivers/infiniband/sw/rxe/rxe_loc.h | 5 ++- drivers/infiniband/sw/rxe/rxe_net.c | 17 +++++---- drivers/infiniband/sw/rxe/rxe_req.c | 55 +++++++++++++++++----------- drivers/infiniband/sw/rxe/rxe_resp.c | 2 +- 5 files changed, 63 insertions(+), 35 deletions(-)
drivers/infiniband/sw/rxe/rxe_av.c +17 -2
@@ -99,11 +99,14 @@ void rxe_av_fill_ip_info(struct rxe_av *av, struct rdma_ah_attr *attr) av->network_type = type; } -struct rxe_av *rxe_get_av (struct rxe_pkt_info *pkt) +struct rxe_av *rxe_get_av (struct rxe_pkt_info *pkt,struct rxe_ah **ahp) {struct rxe_ah *ah;u32 ah_num; +if (ahp) + *ahp =NULL ; +if (!pkt || !pkt->qp)return NULL ;
@@ -117,10 +120,22 @@ struct rxe_av *rxe_get_av(struct rxe_pkt_info *pkt)if (ah_num) {/* only new user provider or kernel client */ ah =rxe_pool_get_index (&pkt->rxe->ah_pool, ah_num); -if (!ah || ah->ah_num != ah_num ||rxe_ah_pd (ah) != pkt->qp->pd) { +if (!ah) {pr_warn ("Unable to find AH matching ah_num\n" );return NULL ; } + +if (rxe_ah_pd (ah) != pkt->qp->pd) { +pr_warn ("PDs don't match for AH and QP\n" ); +rxe_drop_ref (ah); +return NULL ; + } + +if (ahp) + *ahp = ah; +else +rxe_drop_ref (ah); +return &ah->av; }
drivers/infiniband/sw/rxe/rxe_loc.h +3 -2
@@ -19,7 +19,7 @@ void rxe_av_to_attr(struct rxe_av *av, struct rdma_ah_attr *attr);void rxe_av_fill_ip_info (struct rxe_av *av,struct rdma_ah_attr *attr); -struct rxe_av *rxe_get_av (struct rxe_pkt_info *pkt); +struct rxe_av *rxe_get_av (struct rxe_pkt_info *pkt,struct rxe_ah **ahp);/* rxe_cq.c */ int rxe_cq_chk_attr(struct rxe_dev *rxe, struct rxe_cq *cq,
@@ -98,7 +98,8 @@ void rxe_mw_cleanup(struct rxe_pool_elem *arg);/* rxe_net.c */ struct sk_buff *rxe_init_packet (struct rxe_dev *rxe,struct rxe_av *av,int paylen,struct rxe_pkt_info *pkt); -int rxe_prepare (struct rxe_pkt_info *pkt,struct sk_buff *skb); +int rxe_prepare (struct rxe_av *av,struct rxe_pkt_info *pkt, +struct sk_buff *skb);int rxe_xmit_packet (struct rxe_qp *qp,struct rxe_pkt_info *pkt,struct sk_buff *skb);const char *rxe_parent_name (struct rxe_dev *rxe,unsigned int port_num);
drivers/infiniband/sw/rxe/rxe_net.c +9 -8
@@ -293,13 +293,13 @@ static void prepare_ipv6_hdr(struct dst_entry *dst, struct sk_buff *skb, ip6h->payload_len =htons (skb->len -sizeof (*ip6h)); } -static int prepare4 (struct rxe_pkt_info *pkt,struct sk_buff *skb) +static int prepare4 (struct rxe_av *av,struct rxe_pkt_info *pkt, +struct sk_buff *skb) {struct rxe_qp *qp = pkt->qp;struct dst_entry *dst;bool xnet = false;__be16 df =htons (IP_DF ); -struct rxe_av *av =rxe_get_av (pkt);struct in_addr *saddr = &av->sgid_addr._sockaddr_in.sin_addr;struct in_addr *daddr = &av->dgid_addr._sockaddr_in.sin_addr;
@@ -319,11 +319,11 @@ static int prepare4(struct rxe_pkt_info *pkt, struct sk_buff *skb)return 0 ; } -static int prepare6 (struct rxe_pkt_info *pkt,struct sk_buff *skb) +static int prepare6 (struct rxe_av *av,struct rxe_pkt_info *pkt, +struct sk_buff *skb) {struct rxe_qp *qp = pkt->qp;struct dst_entry *dst; -struct rxe_av *av =rxe_get_av (pkt);struct in6_addr *saddr = &av->sgid_addr._sockaddr_in6.sin6_addr;struct in6_addr *daddr = &av->dgid_addr._sockaddr_in6.sin6_addr;
@@ -344,16 +344,17 @@ static int prepare6(struct rxe_pkt_info *pkt, struct sk_buff *skb)return 0 ; } -int rxe_prepare (struct rxe_pkt_info *pkt,struct sk_buff *skb) +int rxe_prepare (struct rxe_av *av,struct rxe_pkt_info *pkt, +struct sk_buff *skb) {int err =0 ;if (skb->protocol ==htons (ETH_P_IP )) - err =prepare4 (pkt, skb); + err =prepare4 (av, pkt, skb);else if (skb->protocol ==htons (ETH_P_IPV6 )) - err =prepare6 (pkt, skb); + err =prepare6 (av, pkt, skb); -if (ether_addr_equal (skb->dev->dev_addr,rxe_get_av (pkt)->dmac)) +if (ether_addr_equal (skb->dev->dev_addr, av->dmac)) pkt->mask |=RXE_LOOPBACK_MASK ;return err;
drivers/infiniband/sw/rxe/rxe_req.c +33 -22
121 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_resp.c +1 -1
@@ -632,7 +632,7 @@ static struct sk_buff *prepare_ack_packet(struct rxe_qp *qp,if (ack->mask &RXE_ATMACK_MASK )atmack_set_orig (ack, qp->resp.atomic_orig); - err =rxe_prepare (ack, skb); + err =rxe_prepare (&qp->pri_av, ack, skb);if (err) {kfree_skb (skb);return NULL ;
Signature · 1 line
2.30.2
PATCH v4 12/13 RDMA/rxe: Replace mr by rkey in responder resources
In reply to Bob PearsonCurrently rxe saves a copy of MR in responder resources for RDMA reads. Since the responder resources are never freed just over written if more are needed this MR may not have a reference freed until the QP is destroyed. This patch uses the rkey instead of the MR and on subsequent packets of a multipacket read reply message it looks up the MR from the rkey for each packet. This makes it possible for a user to deregister an MR or unbind a MW on the fly and get correct behaviour.
--- drivers/infiniband/sw/rxe/rxe_qp.c | 10 +-- drivers/infiniband/sw/rxe/rxe_resp.c | 123 ++++++++++++++++++-------- drivers/infiniband/sw/rxe/rxe_verbs.h | 1 - 3 files changed, 87 insertions(+), 47 deletions(-)
drivers/infiniband/sw/rxe/rxe_qp.c +2 -8
@@ -135,12 +135,8 @@ static void free_rd_atomic_resources(struct rxe_qp *qp)void free_rd_atomic_resource (struct rxe_qp *qp,struct resp_res *res) { -if (res->type ==RXE_ATOMIC_MASK ) { +if (res->type ==RXE_ATOMIC_MASK )kfree_skb (res->atomic.skb); - }else if (res->type ==RXE_READ_MASK ) { -if (res->read.mr) -rxe_drop_ref (res->read.mr); - } res->type =0 ; }
@@ -816,10 +812,8 @@ static void rxe_qp_do_cleanup(struct work_struct *work)if (qp->pd)rxe_drop_ref (qp->pd); -if (qp->resp.mr) { +if (qp->resp.mr)rxe_drop_ref (qp->resp.mr); - qp->resp.mr =NULL ; - }if (qp_type (qp) ==IB_QPT_RC )sk_dst_reset (qp->sk->sk);
drivers/infiniband/sw/rxe/rxe_resp.c +85 -38
154 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_verbs.h +0 -1
@@ -157,7 +157,6 @@ struct resp_res {struct sk_buff *skb; } atomic;struct { -struct rxe_mr *mr;u64 va_org;u32 rkey;u32 length;
Signature · 1 line
2.30.2
PATCH v4 13/13 RDMA/rxe: Protect against race between get_index and drop_ref
In reply to Bob PearsonUse refcount_inc_not_zero instead of kref_get to protect object pointer returned by rxe_pool_get_index() to prevent chance of a race between get_index and drop_ref by another thread.
--- drivers/infiniband/sw/rxe/rxe_pool.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
drivers/infiniband/sw/rxe/rxe_pool.c +6 -1
@@ -272,8 +272,13 @@ void *rxe_pool_get_index(struct rxe_pool *pool, unsigned long index) } elem =xa_load (&pool->xarray.xa, index); +if (elem) { -kref_get (&elem->ref_cnt); +/* protect against a race with someone else dropping +* the last reference to the object +*/ +if (!__rxe_add_ref (elem)) +return NULL ; obj = elem->obj; }else { obj =NULL ;
Signature · 1 line
2.30.2
Jason Gunthorpe
maintainerRe: [PATCH for-next v4 13/13] RDMA/rxe: Protect against race between get_index and drop_ref
Quoted 14 lines · Bob Pearson
Use
refcount_inc_not_zeroinstead ofkref_getto protect object pointer returned byrxe_pool_get_index()to prevent chance of a race betweenget_indexanddrop_refby another thread.Signed-off-by Bob Pearson <[email protected]>drivers/infiniband/sw/rxe/rxe_pool.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)drivers/infiniband/sw/rxe/rxe_pool.c +1
-0+++ b/drivers/infiniband/sw/rxe/rxe_pool.c@@ -272,8 +272,13 @@ void *rxe_pool_get_index(struct rxe_pool *pool, unsigned long index) }
drivers/infiniband/sw/rxe/rxe_pool.c void *rxe_pool_get_index(struct rxe_pool *pool, unsigned long index) · line 282
Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_pool.c +6
-1elem =xa_load (&pool->xarray.xa, index); +if (elem) { -kref_get (&elem->ref_cnt); +/* protect against a race with someone else dropping +* the last reference to the object +*/ +if (!__rxe_add_ref (elem)) +return NULL ; obj = elem->obj;
That doesn't really work without RCU, since now you just use after free on the ref_cnt atomic.
Jason
Jason Gunthorpe
maintainerQuoted 23 lines · Bob Pearson
There are several race conditions discovered in the current
rdma_rxedriver. They mostly relate to races between normal operations and destroying objects. This patch series makes several minor cleanups in rxe_pool.[ch] and replaces the red-black trees currently used by xarrays which have better atomic behavior.This patch series applies cleanly to current for-next. commit
6a463bc9d999("Merge branch 'for-rc' into rdma.git for-next")Signed-off-by Bob Pearson <[email protected]>v4
Restructured patch series to change to xarray earlier which greatly simplified the changes.
Rebased to current for-next
v3
Changedrxe_allocto useGFP_KERNEL
Addressed other comments by Jason Gunthorp
Merged the previous 06/10 and 07/10 patches into one since they overlapped Added some minor cleanups as 10/10
v2
Rebased to current for-next.
Added 4 additional patches
Quoted 6 lines
Bob Pearson (13):
RDMA/rxe: Replace irqsave locks with bh locks RDMA/rxe: Cleanuprxe_pool_entry
RDMA/rxe: Copy setup parameters intorxe_poolRDMA/rxe: Save object pointer in pool element RDMA/rxe: Remove #include "rxe_loc.h" fromrxe_pool.c
I took these patches
Quoted 8 lines
RDMA/rxe: Replace RB tree by xarray for indexes RDMA/rxe: Remove some #defines from
rxe_pool.hRDMA/rxe: Reverse the sense ofRXE_POOL_NO_ALLOCRDMA/rxe: Replaced keyed rxe objects by indexed objects RDMA/rxe: Prevent taking references to dead objects RDMA/rxe: Fix ref error inrxe_av.c
RDMA/rxe: Replace mr by rkey in responder resources RDMA/rxe: Protect against race betweenget_indexanddrop_ref
These need some fixing
Thanks,
Jason