RDMA/rxe: Correct race conditions
Subsystems (2)
v6 cover RDMA/rxe: Correct race conditions
There are several race conditions discovered in the current rdma_rxe driver. They mostly relate to races between normal operations and destroying objects. This patch series
- Makes several minor cleanups in rxe_pool.[ch]
- Replaces the red-black trees currently used by xarrays for indices
- Simplifies the API for keyed objects
- Corrects several reference counting errors
- Adds wait for completions to the paths in verbs APIs which destroy objects.
This patch series applies cleanly to current for-next.
commit 81ff48ddda0b ("RDMA/bnxt_re: Use bitmap_zalloc() when applicable")
---
v6
Fixed a kzalloc flags bug.
Fixed comment bug reported by 'Kernel Test Robot'.
Changed type of rxe_pool.c in __rxe_fini().
v5
Removed patches already accepted into for-next and addressed comments from Jason Gunthorpe.
v4
Restructured patch series to change to xarray earlier which greatly simplified the changes.
Rebased to current for-next
v3
Changed rxe_alloc to use GFP_KERNEL
Addressed other comments by Jason Gunthorp
Merged the previous 06/10 and 07/10 patches into one since they overlapped Added some minor cleanups as 10/10
v2
Rebased to current for-next.
Added 4 additional patches
Bob Pearson (8):
RDMA/rxe: Replace RB tree by xarray for indexes RDMA/rxe: Reverse the sense of RXE_POOL_NO_ALLOC RDMA/rxe: Cleanup pool APIs for keyed objects RDMA/rxe: Fix ref error in rxe_av.c
RDMA/rxe: Replace mr by rkey in responder resources RDMA/rxe: Minor cleanups in rxe_pool.c/rxe_pool.h RDMA/rxe: Replace rxe_alloc by kzalloc for rxe_mc_elem RDMA/rxe: Add wait for completion to obj destruct
drivers/infiniband/sw/rxe/rxe.c | 101 +----- drivers/infiniband/sw/rxe/rxe_av.c | 19 +- drivers/infiniband/sw/rxe/rxe_comp.c | 4 +- drivers/infiniband/sw/rxe/rxe_loc.h | 10 +- drivers/infiniband/sw/rxe/rxe_mcast.c | 70 ++-- drivers/infiniband/sw/rxe/rxe_mr.c | 3 +- drivers/infiniband/sw/rxe/rxe_mw.c | 18 +- drivers/infiniband/sw/rxe/rxe_net.c | 17 +- drivers/infiniband/sw/rxe/rxe_pool.c | 485 +++++++++++--------------- drivers/infiniband/sw/rxe/rxe_pool.h | 113 ++---- drivers/infiniband/sw/rxe/rxe_qp.c | 10 +- drivers/infiniband/sw/rxe/rxe_recv.c | 4 +- drivers/infiniband/sw/rxe/rxe_req.c | 66 ++-- drivers/infiniband/sw/rxe/rxe_resp.c | 131 ++++--- drivers/infiniband/sw/rxe/rxe_verbs.c | 96 +++-- drivers/infiniband/sw/rxe/rxe_verbs.h | 3 - 16 files changed, 515 insertions(+), 635 deletions(-)
Signature · 1 line
2.32.0
PATCH v6 1/8 RDMA/rxe: Replace RB tree by xarray for indexes
Currently the rxe driver uses red-black trees to add indices and keys to the rxe object pool. Linux xarrays provide a better way to implement the same functionality for indices but not keys. This patch replaces red-black trees by xarrays for indexed objects. Since caller managed locks for indexed objects are not used these APIs are deleted as well.
To avoid double locking since xarray already includes a spinlock replace the rxe_pool rwlock by the spinlock included in xarray.
The RDMA objects are created and destroyed by verbs calls from rdma_core but are looked up from indices or keys from soft IRQs so _bh style locks are the correct type to use.
---
v6
Minor fix to comment.
--- drivers/infiniband/sw/rxe/rxe.c | 100 ++---------- drivers/infiniband/sw/rxe/rxe_mcast.c | 6 +- drivers/infiniband/sw/rxe/rxe_mr.c | 1 - drivers/infiniband/sw/rxe/rxe_mw.c | 4 - drivers/infiniband/sw/rxe/rxe_pool.c | 221 ++++++++------------------ drivers/infiniband/sw/rxe/rxe_pool.h | 77 ++++----- drivers/infiniband/sw/rxe/rxe_verbs.c | 12 -- 7 files changed, 114 insertions(+), 307 deletions(-)
drivers/infiniband/sw/rxe/rxe.c +14 -86
@@ -116,97 +116,31 @@ static void rxe_init_ports(struct rxe_dev *rxe) }/* init pools of managed objects */ -static int rxe_init_pools (struct rxe_dev *rxe) +static void rxe_init_pools (struct rxe_dev *rxe) { -int err; - - err =rxe_pool_init (rxe, &rxe->uc_pool,RXE_TYPE_UC , - rxe->max_ucontext); -if (err) - goto err1; - - err =rxe_pool_init (rxe, &rxe->pd_pool,RXE_TYPE_PD , - rxe->attr.max_pd); -if (err) - goto err2; - - err =rxe_pool_init (rxe, &rxe->ah_pool,RXE_TYPE_AH , - rxe->attr.max_ah); -if (err) - goto err3; - - err =rxe_pool_init (rxe, &rxe->srq_pool,RXE_TYPE_SRQ , - rxe->attr.max_srq); -if (err) - goto err4; - - err =rxe_pool_init (rxe, &rxe->qp_pool,RXE_TYPE_QP , - rxe->attr.max_qp); -if (err) - goto err5; - - err =rxe_pool_init (rxe, &rxe->cq_pool,RXE_TYPE_CQ , - rxe->attr.max_cq); -if (err) - goto err6; - - err =rxe_pool_init (rxe, &rxe->mr_pool,RXE_TYPE_MR , - rxe->attr.max_mr); -if (err) - goto err7; - - err =rxe_pool_init (rxe, &rxe->mw_pool,RXE_TYPE_MW , - rxe->attr.max_mw); -if (err) - goto err8; - - err =rxe_pool_init (rxe, &rxe->mc_grp_pool,RXE_TYPE_MC_GRP , +rxe_pool_init (rxe, &rxe->uc_pool,RXE_TYPE_UC , rxe->max_ucontext); +rxe_pool_init (rxe, &rxe->pd_pool,RXE_TYPE_PD , rxe->attr.max_pd); +rxe_pool_init (rxe, &rxe->ah_pool,RXE_TYPE_AH , rxe->attr.max_ah); +rxe_pool_init (rxe, &rxe->srq_pool,RXE_TYPE_SRQ , rxe->attr.max_srq); +rxe_pool_init (rxe, &rxe->qp_pool,RXE_TYPE_QP , rxe->attr.max_qp); +rxe_pool_init (rxe, &rxe->cq_pool,RXE_TYPE_CQ , rxe->attr.max_cq); +rxe_pool_init (rxe, &rxe->mr_pool,RXE_TYPE_MR , rxe->attr.max_mr); +rxe_pool_init (rxe, &rxe->mw_pool,RXE_TYPE_MW , rxe->attr.max_mw); +rxe_pool_init (rxe, &rxe->mc_grp_pool,RXE_TYPE_MC_GRP , rxe->attr.max_mcast_grp); -if (err) - goto err9; - - err =rxe_pool_init (rxe, &rxe->mc_elem_pool,RXE_TYPE_MC_ELEM , +rxe_pool_init (rxe, &rxe->mc_elem_pool,RXE_TYPE_MC_ELEM , rxe->attr.max_total_mcast_qp_attach); -if (err) - goto err10; - -return 0 ; - -err10: -rxe_pool_cleanup (&rxe->mc_grp_pool); -err9: -rxe_pool_cleanup (&rxe->mw_pool); -err8: -rxe_pool_cleanup (&rxe->mr_pool); -err7: -rxe_pool_cleanup (&rxe->cq_pool); -err6: -rxe_pool_cleanup (&rxe->qp_pool); -err5: -rxe_pool_cleanup (&rxe->srq_pool); -err4: -rxe_pool_cleanup (&rxe->ah_pool); -err3: -rxe_pool_cleanup (&rxe->pd_pool); -err2: -rxe_pool_cleanup (&rxe->uc_pool); -err1: -return err; }/* initialize rxe device state */ -static int rxe_init (struct rxe_dev *rxe) +static void rxe_init (struct rxe_dev *rxe) { -int err; -/* init default device parameters */ rxe_init_device_param (rxe);rxe_init_ports (rxe); - err =rxe_init_pools (rxe); -if (err) -return err; +rxe_init_pools (rxe);/* init pending mmap list */ spin_lock_init (&rxe->mmap_offset_lock);
@@ -214,8 +148,6 @@ static int rxe_init(struct rxe_dev *rxe)INIT_LIST_HEAD (&rxe->pending_mmaps);mutex_init (&rxe->usdev_lock); - -return 0 ; }void rxe_set_mtu (struct rxe_dev *rxe,unsigned int ndev_mtu)
@@ -237,11 +169,7 @@ void rxe_set_mtu(struct rxe_dev *rxe, unsigned int ndev_mtu)*/ int rxe_add (struct rxe_dev *rxe,unsigned int mtu,const char *ibdev_name) { -int err; - - err =rxe_init (rxe); -if (err) -return err; +rxe_init (rxe);rxe_set_mtu (rxe, mtu);
drivers/infiniband/sw/rxe/rxe_mcast.c +3 -3
@@ -44,7 +44,7 @@ int rxe_mcast_get_grp(struct rxe_dev *rxe, union ib_gid *mgid,if (rxe->attr.max_mcast_qp_attach ==0 )return -EINVAL ; -write_lock_bh (&pool->pool_lock); +rxe_pool_lock_bh (pool); grp =rxe_pool_get_key_locked (pool, mgid);if (grp)
@@ -52,13 +52,13 @@ int rxe_mcast_get_grp(struct rxe_dev *rxe, union ib_gid *mgid, grp =create_grp (rxe, pool, mgid);if (IS_ERR (grp)) { -write_unlock_bh (&pool->pool_lock); +rxe_pool_unlock_bh (pool); err =PTR_ERR (grp);return err; } done: -write_unlock_bh (&pool->pool_lock); +rxe_pool_unlock_bh (pool); *grp_p = grp;return 0 ; }
drivers/infiniband/sw/rxe/rxe_mr.c +0 -1
@@ -693,7 +693,6 @@ int rxe_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata) mr->state =RXE_MR_STATE_INVALID ;rxe_drop_ref (mr_pd (mr)); -rxe_drop_index (mr);rxe_drop_ref (mr);return 0 ;
drivers/infiniband/sw/rxe/rxe_mw.c +0 -4
@@ -20,7 +20,6 @@ int rxe_alloc_mw(struct ib_mw *ibmw, struct ib_udata *udata)return ret; } -rxe_add_index (mw); mw->rkey = ibmw->rkey = (mw->elem.index <<8 ) |rxe_get_next_key (-1 ); mw->state = (mw->ibmw.type ==IB_MW_TYPE_2 ) ?RXE_MW_STATE_FREE :RXE_MW_STATE_VALID ;
@@ -332,7 +331,4 @@ struct rxe_mw *rxe_lookup_mw(struct rxe_qp *qp, int access, u32 rkey)void rxe_mw_cleanup (struct rxe_pool_elem *elem) { -struct rxe_mw *mw =container_of (elem,typeof (*mw), elem); - -rxe_drop_index (mw); }
drivers/infiniband/sw/rxe/rxe_pool.c +64 -157
335 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_pool.h +33 -44
119 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_verbs.c +0 -12
@@ -181,7 +181,6 @@ static int rxe_create_ah(struct ib_ah *ibah,return err;/* create index > 0 */ -rxe_add_index (ah); ah->ah_num = ah->elem.index;if (uresp) {
@@ -189,7 +188,6 @@ static int rxe_create_ah(struct ib_ah *ibah, err =copy_to_user (&uresp->ah_num, &ah->ah_num,sizeof (uresp->ah_num));if (err) { -rxe_drop_index (ah);rxe_drop_ref (ah);return -EFAULT ; }
@@ -230,7 +228,6 @@ static int rxe_destroy_ah(struct ib_ah *ibah, u32 flags) {struct rxe_ah *ah =to_rah (ibah); -rxe_drop_index (ah);rxe_drop_ref (ah);return 0 ; }
@@ -437,7 +434,6 @@ static int rxe_create_qp(struct ib_qp *ibqp, struct ib_qp_init_attr *init,if (err)return err; -rxe_add_index (qp); err =rxe_qp_from_init (rxe, qp, pd, init, uresp, ibqp->pd, udata);if (err) goto qp_init;
@@ -445,7 +441,6 @@ static int rxe_create_qp(struct ib_qp *ibqp, struct ib_qp_init_attr *init,return 0 ; qp_init: -rxe_drop_index (qp);rxe_drop_ref (qp);return err; }
@@ -490,7 +485,6 @@ static int rxe_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)struct rxe_qp *qp =to_rqp (ibqp);rxe_qp_destroy (qp); -rxe_drop_index (qp);rxe_drop_ref (qp);return 0 ; }
@@ -893,7 +887,6 @@ static struct ib_mr *rxe_get_dma_mr(struct ib_pd *ibpd, int access)if (!mr)return ERR_PTR (-ENOMEM ); -rxe_add_index (mr);rxe_add_ref (pd);rxe_mr_init_dma (pd, access, mr);
@@ -917,7 +910,6 @@ static struct ib_mr *rxe_reg_user_mr(struct ib_pd *ibpd, goto err2; } -rxe_add_index (mr);rxe_add_ref (pd);
@@ -929,7 +921,6 @@ static struct ib_mr *rxe_reg_user_mr(struct ib_pd *ibpd, err3:rxe_drop_ref (pd); -rxe_drop_index (mr);rxe_drop_ref (mr); err2:return ERR_PTR (err);
@@ -952,8 +943,6 @@ static struct ib_mr *rxe_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type mr_type, goto err1; } -rxe_add_index (mr); -rxe_add_ref (pd); err =rxe_mr_init_fast (pd, max_num_sg, mr);
@@ -964,7 +953,6 @@ static struct ib_mr *rxe_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type mr_type, err2:rxe_drop_ref (pd); -rxe_drop_index (mr);rxe_drop_ref (mr); err1:return ERR_PTR (err);
Signature · 1 line
2.32.0
Jason Gunthorpe
maintainerRe: [PATCH for-next v6 1/8] RDMA/rxe: Replace RB tree by xarray for indexes
drivers/infiniband/sw/rxe/rxe_pool.c int rxe_pool_init( · line 127
Quoted 12 lines · Bob Pearson
drivers/infiniband/sw/rxe/rxe_pool.c +6
-5if (pool->flags &RXE_POOL_INDEX ) { - pool->index.tree =RB_ROOT ; - err =rxe_pool_init_index (pool, info->max_index, - info->min_index); -if (err) - goto out; +xa_init_flags (&pool->xarray.xa,XA_FLAGS_ALLOC ); + pool->xarray.limit.max = info->max_index; + pool->xarray.limit.min = info->min_index; + }else { +/* if pool not indexed just use xa spin_lock */ +spin_lock_init (&pool->xarray.xa.xa_lock);
xarray's don't cost anything to init, so there is no reason to do something like this.
drivers/infiniband/sw/rxe/rxe_pool.h void *rxe_pool_get_key(struct rxe_pool *pool, void *key); · line 141
Quoted 8 lines
drivers/infiniband/sw/rxe/rxe_pool.h +8
-0+/* drop a reference to an object */ +static inline bool __rxe_drop_ref (struct rxe_pool_elem *elem) +{ +bool ret; + +rxe_pool_lock_bh (elem->pool); + ret =kref_put (&elem->ref_cnt, rxe_elem_release); +rxe_pool_unlock_bh (elem->pool);
This is a bit strange, why does something need to hold a lock around a kref?
Jason
Re: [PATCH for-next v6 1/8] RDMA/rxe: Replace RB tree by xarray for indexes
Quoted 16 lines · Jason Gunthorpe
drivers/infiniband/sw/rxe/rxe_pool.c int rxe_pool_init( · line 127
Quoted 12 lines · Bob Pearson
drivers/infiniband/sw/rxe/rxe_pool.c +6
-5if (pool->flags &RXE_POOL_INDEX ) { - pool->index.tree =RB_ROOT ; - err =rxe_pool_init_index (pool, info->max_index, - info->min_index); -if (err) - goto out; +xa_init_flags (&pool->xarray.xa,XA_FLAGS_ALLOC ); + pool->xarray.limit.max = info->max_index; + pool->xarray.limit.min = info->min_index; + }else { +/* if pool not indexed just use xa spin_lock */ +spin_lock_init (&pool->xarray.xa.xa_lock);xarray's don't cost anything to init, so there is no reason to do something like this.
OK
Quoted 12 lines
drivers/infiniband/sw/rxe/rxe_pool.h void *rxe_pool_get_key(struct rxe_pool *pool, void *key); · line 141
Quoted 9 lines
drivers/infiniband/sw/rxe/rxe_pool.h +8
-0+/* drop a reference to an object */ +static inline bool __rxe_drop_ref (struct rxe_pool_elem *elem) +{ +bool ret; + +rxe_pool_lock_bh (elem->pool); + ret =kref_put (&elem->ref_cnt, rxe_elem_release); +rxe_pool_unlock_bh (elem->pool);This is a bit strange, why does something need to hold a lock around a kref?
This also relates to your comment on 8/8 patch. There seems to be a race opportunity between the call to kref_put(&obj->elem, rxe_elem_release) and the call in rxe_elem_release() to xa_erase(). If a duplicate or delayed packet arrives after the the final kref_put() and before the xa_erase() one can still lookup the object from its index (qpn, rkey, etc.) and take a reference to it. The use of kref_get_unless_zero and locking around kref_put and __xa_erase was an attempt to fix this. Once you call kref_put with the ref count going to zero there is no way to prevent the object getting its cleanup routine called.
Bob
Quoted 2 lines
Jason
Jason Gunthorpe
maintainerRe: [PATCH for-next v6 1/8] RDMA/rxe: Replace RB tree by xarray for indexes
Quoted 31 lines · Bob Pearson
Quoted 16 lines · Jason Gunthorpe
drivers/infiniband/sw/rxe/rxe_pool.c int rxe_pool_init( · line 127
Quoted 12 lines · Bob Pearson
drivers/infiniband/sw/rxe/rxe_pool.c +6
-5if (pool->flags &RXE_POOL_INDEX ) { - pool->index.tree =RB_ROOT ; - err =rxe_pool_init_index (pool, info->max_index, - info->min_index); -if (err) - goto out; +xa_init_flags (&pool->xarray.xa,XA_FLAGS_ALLOC ); + pool->xarray.limit.max = info->max_index; + pool->xarray.limit.min = info->min_index; + }else { +/* if pool not indexed just use xa spin_lock */ +spin_lock_init (&pool->xarray.xa.xa_lock);xarray's don't cost anything to init, so there is no reason to do something like this.
OK
Quoted 12 lines
drivers/infiniband/sw/rxe/rxe_pool.h void *rxe_pool_get_key(struct rxe_pool *pool, void *key); · line 141
Quoted 9 lines
drivers/infiniband/sw/rxe/rxe_pool.h +8
-0+/* drop a reference to an object */ +static inline bool __rxe_drop_ref (struct rxe_pool_elem *elem) +{ +bool ret; + +rxe_pool_lock_bh (elem->pool); + ret =kref_put (&elem->ref_cnt, rxe_elem_release); +rxe_pool_unlock_bh (elem->pool);This is a bit strange, why does something need to hold a lock around a kref?
Quoted 8 lines
This also relates to your comment on 8/8 patch. There seems to be a race opportunity between the call to
kref_put(&obj->elem, rxe_elem_release)and the call inrxe_elem_release()toxa_erase(). If a duplicate or delayed packet arrives after the the finalkref_put()and before thexa_erase()one can still lookup the object from its index (qpn, rkey, etc.) and take a reference to it. The use ofkref_get_unless_zeroand locking aroundkref_putand__xa_erasewas an attempt to fix this. Once you callkref_putwith the ref count going to zero there is no way to prevent the object getting its cleanup routine called.
This is why you use kref_get_not_zero only during the xa lookup, then during that race it returns failure
Costs nothing as the atomics are already all required.
Jason
Re: [PATCH for-next v6 1/8] RDMA/rxe: Replace RB tree by xarray for indexes
Quoted 41 lines · Jason Gunthorpe
Quoted 31 lines · Bob Pearson
Quoted 16 lines · Jason Gunthorpe
drivers/infiniband/sw/rxe/rxe_pool.c int rxe_pool_init( · line 127
Quoted 12 lines · Bob Pearson
drivers/infiniband/sw/rxe/rxe_pool.c +6
-5if (pool->flags &RXE_POOL_INDEX ) { - pool->index.tree =RB_ROOT ; - err =rxe_pool_init_index (pool, info->max_index, - info->min_index); -if (err) - goto out; +xa_init_flags (&pool->xarray.xa,XA_FLAGS_ALLOC ); + pool->xarray.limit.max = info->max_index; + pool->xarray.limit.min = info->min_index; + }else { +/* if pool not indexed just use xa spin_lock */ +spin_lock_init (&pool->xarray.xa.xa_lock);xarray's don't cost anything to init, so there is no reason to do something like this.
OK
Quoted 12 lines
drivers/infiniband/sw/rxe/rxe_pool.h void *rxe_pool_get_key(struct rxe_pool *pool, void *key); · line 141
Quoted 9 lines
drivers/infiniband/sw/rxe/rxe_pool.h +8
-0+/* drop a reference to an object */ +static inline bool __rxe_drop_ref (struct rxe_pool_elem *elem) +{ +bool ret; + +rxe_pool_lock_bh (elem->pool); + ret =kref_put (&elem->ref_cnt, rxe_elem_release); +rxe_pool_unlock_bh (elem->pool);This is a bit strange, why does something need to hold a lock around a kref?
Quoted 8 lines
This also relates to your comment on 8/8 patch. There seems to be a race opportunity between the call to
kref_put(&obj->elem, rxe_elem_release)and the call inrxe_elem_release()toxa_erase(). If a duplicate or delayed packet arrives after the the finalkref_put()and before thexa_erase()one can still lookup the object from its index (qpn, rkey, etc.) and take a reference to it. The use ofkref_get_unless_zeroand locking aroundkref_putand__xa_erasewas an attempt to fix this. Once you callkref_putwith the ref count going to zero there is no way to prevent the object getting its cleanup routine called.
Quoted 7 lines
This is why you use
kref_get_not_zeroonly during the xa lookup, then during that race it returns failureCosts nothing as the atomics are already all required.
Jason
Thanks. -- Bob
PATCH v6 2/8 RDMA/rxe: Reverse the sense of RXE_POOL_NO_ALLOC
In reply to Bob PearsonSince most rxe objects are now allocated in rdma-core change the sense of RXE_POOL_NO_ALLOC to RXE_POOL_ALLOC. This makes the code easier to understand.
--- drivers/infiniband/sw/rxe/rxe_pool.c | 18 ++++++++---------- drivers/infiniband/sw/rxe/rxe_pool.h | 2 +- 2 files changed, 9 insertions(+), 11 deletions(-)
drivers/infiniband/sw/rxe/rxe_pool.c +8 -10
@@ -23,19 +23,17 @@ static const struct rxe_type_info { .name = "rxe-uc", .size = sizeof(struct rxe_ucontext), .elem_offset = offsetof(struct rxe_ucontext, elem), - .flags = RXE_POOL_NO_ALLOC, }, [RXE_TYPE_PD] = { .name = "rxe-pd", .size = sizeof(struct rxe_pd), .elem_offset = offsetof(struct rxe_pd, elem), - .flags = RXE_POOL_NO_ALLOC, }, [RXE_TYPE_AH] = { .name = "rxe-ah", .size = sizeof(struct rxe_ah), .elem_offset = offsetof(struct rxe_ah, elem), - .flags = RXE_POOL_INDEX | RXE_POOL_NO_ALLOC, + .flags = RXE_POOL_INDEX, .min_index = RXE_MIN_AH_INDEX, .max_index = RXE_MAX_AH_INDEX, },
@@ -43,7 +41,7 @@ static const struct rxe_type_info { .name = "rxe-srq", .size = sizeof(struct rxe_srq), .elem_offset = offsetof(struct rxe_srq, elem), - .flags = RXE_POOL_INDEX | RXE_POOL_NO_ALLOC, + .flags = RXE_POOL_INDEX, .min_index = RXE_MIN_SRQ_INDEX, .max_index = RXE_MAX_SRQ_INDEX, },
@@ -52,7 +50,7 @@ static const struct rxe_type_info { .size = sizeof(struct rxe_qp), .elem_offset = offsetof(struct rxe_qp, elem), .cleanup = rxe_qp_cleanup, - .flags = RXE_POOL_INDEX | RXE_POOL_NO_ALLOC, + .flags = RXE_POOL_INDEX, .min_index = RXE_MIN_QP_INDEX, .max_index = RXE_MAX_QP_INDEX, },
@@ -60,7 +58,6 @@ static const struct rxe_type_info { .name = "rxe-cq", .size = sizeof(struct rxe_cq), .elem_offset = offsetof(struct rxe_cq, elem), - .flags = RXE_POOL_NO_ALLOC, .cleanup = rxe_cq_cleanup, }, [RXE_TYPE_MR] = {
@@ -68,7 +65,7 @@ static const struct rxe_type_info { .size = sizeof(struct rxe_mr), .elem_offset = offsetof(struct rxe_mr, elem), .cleanup = rxe_mr_cleanup, - .flags = RXE_POOL_INDEX, + .flags = RXE_POOL_INDEX | RXE_POOL_ALLOC, .min_index = RXE_MIN_MR_INDEX, .max_index = RXE_MAX_MR_INDEX, },
@@ -77,7 +74,7 @@ static const struct rxe_type_info { .size = sizeof(struct rxe_mw), .elem_offset = offsetof(struct rxe_mw, elem), .cleanup = rxe_mw_cleanup, - .flags = RXE_POOL_INDEX | RXE_POOL_NO_ALLOC, + .flags = RXE_POOL_INDEX, .min_index = RXE_MIN_MW_INDEX, .max_index = RXE_MAX_MW_INDEX, },
@@ -86,7 +83,7 @@ static const struct rxe_type_info { .size = sizeof(struct rxe_mc_grp), .elem_offset = offsetof(struct rxe_mc_grp, elem), .cleanup = rxe_mc_cleanup, - .flags = RXE_POOL_KEY, + .flags = RXE_POOL_KEY | RXE_POOL_ALLOC, .key_offset = offsetof(struct rxe_mc_grp, mgid), .key_size = sizeof(union ib_gid), },
@@ -94,6 +91,7 @@ static const struct rxe_type_info { .name = "rxe-mc_elem", .size = sizeof(struct rxe_mc_elem), .elem_offset = offsetof(struct rxe_mc_elem, elem), + .flags = RXE_POOL_ALLOC, }, };
@@ -323,7 +321,7 @@ void rxe_elem_release(struct kref *kref)if (pool->cleanup) pool->cleanup (elem); -if (!(pool->flags &RXE_POOL_NO_ALLOC )) { +if (pool->flags &RXE_POOL_ALLOC ) { obj = elem->obj;kfree (obj); }
drivers/infiniband/sw/rxe/rxe_pool.h +1 -1
@@ -10,7 +10,7 @@enum rxe_pool_flags {RXE_POOL_INDEX =BIT (1 ),RXE_POOL_KEY =BIT (2 ), -RXE_POOL_NO_ALLOC =BIT (4 ), +RXE_POOL_ALLOC =BIT (4 ), };enum rxe_elem_type {
Signature · 1 line
2.32.0
PATCH v6 3/8 RDMA/rxe: Cleanup pool APIs for keyed objects
In reply to Bob PearsonSimplify the rxe pool APIs for keyed objects. Eliminate xxx_locked() APIs. Merge rxe_drop_key into rxe_drop_ref. Replace separate rxe_get_key, and add_key by one call rxe_add_key which looks up and if necessary creates a new object.
---
v6
Changed gfp_t flag in __rxe_alloc() to flags so rxe_alloc() and rxe_add_to_pool() use GFP_KERNEL.
--- drivers/infiniband/sw/rxe/rxe_loc.h | 5 +- drivers/infiniband/sw/rxe/rxe_mcast.c | 46 ++--- drivers/infiniband/sw/rxe/rxe_pool.c | 255 +++++++++++++------------- drivers/infiniband/sw/rxe/rxe_pool.h | 50 +---- 4 files changed, 143 insertions(+), 213 deletions(-)
drivers/infiniband/sw/rxe/rxe_loc.h +1 -4
@@ -40,17 +40,14 @@ void rxe_cq_disable(struct rxe_cq *cq);void rxe_cq_cleanup (struct rxe_pool_elem *arg);/* rxe_mcast.c */ +int rxe_init_grp (struct rxe_pool_elem *elem);int rxe_mcast_get_grp (struct rxe_dev *rxe,union ib_gid *mgid,struct rxe_mc_grp **grp_p); -int rxe_mcast_add_grp_elem (struct rxe_dev *rxe,struct rxe_qp *qp,struct rxe_mc_grp *grp); -int rxe_mcast_drop_grp_elem (struct rxe_dev *rxe,struct rxe_qp *qp,union ib_gid *mgid); -void rxe_drop_all_mcast_groups (struct rxe_qp *qp); -void rxe_mc_cleanup (struct rxe_pool_elem *arg);/* rxe_mmap.c */
drivers/infiniband/sw/rxe/rxe_mcast.c +12 -34
84 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_pool.c +124 -131
340 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_pool.h +6 -44
87 lines not on this page · raw message
Signature · 1 line
2.32.0
Jason Gunthorpe
maintainerRe: [PATCH for-next v6 3/8] RDMA/rxe: Cleanup pool APIs for keyed objects
Quoted 10 lines · Bob Pearson
drivers/infiniband/sw/rxe/rxe_pool.c +10
-0+/** +* rxe_pool_add_key() - lookup or add object with key in pool +* @pool: the object pool +* @key: the key +* +* Returns: If object matching key is present in pool return +* its address and take a reference else allocate a +* new object to pool with key and return its address +* with one reference. +*/
drivers/infiniband/sw/rxe/rxe_pool.c void *rxe_pool_get_key_locked(struct rxe_pool *pool, void *key) · line 356
Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_pool.c +10
-0+void *rxe_pool_add_key (struct rxe_pool *pool,void *key) +{ +void *obj; + +rxe_pool_lock_bh (pool); + obj =__rxe_get_key (pool, key); +if (obj) + goto done; + + obj =__rxe_alloc (pool,GFP_ATOMIC );
Really try hard to avoid GFP_ATOMIC:
rxe_pool_lock_bh(pool);
obj = __rxe_get_key(pool, key);
rxe_pool_unlock_bh(pool);
if (obj)
return obj;
obj = __rxe_alloc(pool, GFP_KERNEL);
if (!obj)
return NULL;
rxe_pool_lock_bh(pool);
old_obj = __xa_cmpxchg(&pool->xarray.xa, key, NULL, entry, GFP_KERNEL);
if (xa_is_err(old_obj))
kfree(obj)
return NULL; if (old_obj) {
kfree(obj);
obj = old_obj;
}
kref_get(old_obj)
rxe_pool_unlock_bh(pool);
return obj; If it is very rare that this function would collide the index then forget the first lookup and use the cmpxchg
drivers/infiniband/sw/rxe/rxe_pool.c void *rxe_pool_get_key_locked(struct rxe_pool *pool, void *key) · line 383
Quoted 11 lines
drivers/infiniband/sw/rxe/rxe_pool.c +11
-0+void rxe_elem_release (struct kref *kref) +{ +struct rxe_pool_elem *elem = + container_of(kref, struct rxe_pool_elem, ref_cnt); +struct rxe_pool *pool = elem->pool; + +if (pool->flags &RXE_POOL_INDEX ) +__xa_erase (&pool->xarray.xa, elem->index); + +if (pool->flags &RXE_POOL_KEY ) +rb_erase (&elem->key_node, &pool->key.tree);
It is a bit jarring to see these as not exclusive ?
Jason
PATCH v6 4/8 RDMA/rxe: Fix ref error in rxe_av.c
In reply to Bob PearsonThe commit referenced below can take a reference to the AH which is never dropped. This only happens in the UD request path. This patch optionally passes that AH back to the caller so that it can hold the reference while the AV is being accessed and then drop it. Code to do this is added to rxe_req.c. The AV is also passed to rxe_prepare in rxe_net.c as an optimization.
--- drivers/infiniband/sw/rxe/rxe_av.c | 19 +++++++++- drivers/infiniband/sw/rxe/rxe_loc.h | 5 ++- drivers/infiniband/sw/rxe/rxe_net.c | 17 +++++---- drivers/infiniband/sw/rxe/rxe_req.c | 55 +++++++++++++++++----------- drivers/infiniband/sw/rxe/rxe_resp.c | 2 +- 5 files changed, 63 insertions(+), 35 deletions(-)
drivers/infiniband/sw/rxe/rxe_av.c +17 -2
@@ -99,11 +99,14 @@ void rxe_av_fill_ip_info(struct rxe_av *av, struct rdma_ah_attr *attr) av->network_type = type; } -struct rxe_av *rxe_get_av (struct rxe_pkt_info *pkt) +struct rxe_av *rxe_get_av (struct rxe_pkt_info *pkt,struct rxe_ah **ahp) {struct rxe_ah *ah;u32 ah_num; +if (ahp) + *ahp =NULL ; +if (!pkt || !pkt->qp)return NULL ;
@@ -117,10 +120,22 @@ struct rxe_av *rxe_get_av(struct rxe_pkt_info *pkt)if (ah_num) {/* only new user provider or kernel client */ ah =rxe_pool_get_index (&pkt->rxe->ah_pool, ah_num); -if (!ah || ah->ah_num != ah_num ||rxe_ah_pd (ah) != pkt->qp->pd) { +if (!ah) {pr_warn ("Unable to find AH matching ah_num\n" );return NULL ; } + +if (rxe_ah_pd (ah) != pkt->qp->pd) { +pr_warn ("PDs don't match for AH and QP\n" ); +rxe_drop_ref (ah); +return NULL ; + } + +if (ahp) + *ahp = ah; +else +rxe_drop_ref (ah); +return &ah->av; }
drivers/infiniband/sw/rxe/rxe_loc.h +3 -2
@@ -19,7 +19,7 @@ void rxe_av_to_attr(struct rxe_av *av, struct rdma_ah_attr *attr);void rxe_av_fill_ip_info (struct rxe_av *av,struct rdma_ah_attr *attr); -struct rxe_av *rxe_get_av (struct rxe_pkt_info *pkt); +struct rxe_av *rxe_get_av (struct rxe_pkt_info *pkt,struct rxe_ah **ahp);/* rxe_cq.c */ int rxe_cq_chk_attr(struct rxe_dev *rxe, struct rxe_cq *cq,
@@ -99,7 +99,8 @@ void rxe_mw_cleanup(struct rxe_pool_elem *arg);/* rxe_net.c */ struct sk_buff *rxe_init_packet (struct rxe_dev *rxe,struct rxe_av *av,int paylen,struct rxe_pkt_info *pkt); -int rxe_prepare (struct rxe_pkt_info *pkt,struct sk_buff *skb); +int rxe_prepare (struct rxe_av *av,struct rxe_pkt_info *pkt, +struct sk_buff *skb);int rxe_xmit_packet (struct rxe_qp *qp,struct rxe_pkt_info *pkt,struct sk_buff *skb);const char *rxe_parent_name (struct rxe_dev *rxe,unsigned int port_num);
drivers/infiniband/sw/rxe/rxe_net.c +9 -8
@@ -293,13 +293,13 @@ static void prepare_ipv6_hdr(struct dst_entry *dst, struct sk_buff *skb, ip6h->payload_len =htons (skb->len -sizeof (*ip6h)); } -static int prepare4 (struct rxe_pkt_info *pkt,struct sk_buff *skb) +static int prepare4 (struct rxe_av *av,struct rxe_pkt_info *pkt, +struct sk_buff *skb) {struct rxe_qp *qp = pkt->qp;struct dst_entry *dst;bool xnet = false;__be16 df =htons (IP_DF ); -struct rxe_av *av =rxe_get_av (pkt);struct in_addr *saddr = &av->sgid_addr._sockaddr_in.sin_addr;struct in_addr *daddr = &av->dgid_addr._sockaddr_in.sin_addr;
@@ -319,11 +319,11 @@ static int prepare4(struct rxe_pkt_info *pkt, struct sk_buff *skb)return 0 ; } -static int prepare6 (struct rxe_pkt_info *pkt,struct sk_buff *skb) +static int prepare6 (struct rxe_av *av,struct rxe_pkt_info *pkt, +struct sk_buff *skb) {struct rxe_qp *qp = pkt->qp;struct dst_entry *dst; -struct rxe_av *av =rxe_get_av (pkt);struct in6_addr *saddr = &av->sgid_addr._sockaddr_in6.sin6_addr;struct in6_addr *daddr = &av->dgid_addr._sockaddr_in6.sin6_addr;
@@ -344,16 +344,17 @@ static int prepare6(struct rxe_pkt_info *pkt, struct sk_buff *skb)return 0 ; } -int rxe_prepare (struct rxe_pkt_info *pkt,struct sk_buff *skb) +int rxe_prepare (struct rxe_av *av,struct rxe_pkt_info *pkt, +struct sk_buff *skb) {int err =0 ;if (skb->protocol ==htons (ETH_P_IP )) - err =prepare4 (pkt, skb); + err =prepare4 (av, pkt, skb);else if (skb->protocol ==htons (ETH_P_IPV6 )) - err =prepare6 (pkt, skb); + err =prepare6 (av, pkt, skb); -if (ether_addr_equal (skb->dev->dev_addr,rxe_get_av (pkt)->dmac)) +if (ether_addr_equal (skb->dev->dev_addr, av->dmac)) pkt->mask |=RXE_LOOPBACK_MASK ;return err;
drivers/infiniband/sw/rxe/rxe_req.c +33 -22
121 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_resp.c +1 -1
@@ -632,7 +632,7 @@ static struct sk_buff *prepare_ack_packet(struct rxe_qp *qp,if (ack->mask &RXE_ATMACK_MASK )atmack_set_orig (ack, qp->resp.atomic_orig); - err =rxe_prepare (ack, skb); + err =rxe_prepare (&qp->pri_av, ack, skb);if (err) {kfree_skb (skb);return NULL ;
Signature · 1 line
2.32.0
PATCH v6 5/8 RDMA/rxe: Replace mr by rkey in responder resources
In reply to Bob PearsonCurrently rxe saves a copy of MR in responder resources for RDMA reads. Since the responder resources are never freed just over written if more are needed this MR may not have a reference freed until the QP is destroyed. This patch uses the rkey instead of the MR and on subsequent packets of a multipacket read reply message it looks up the MR from the rkey for each packet. This makes it possible for a user to deregister an MR or unbind a MW on the fly and get correct behaviour.
--- drivers/infiniband/sw/rxe/rxe_qp.c | 10 +-- drivers/infiniband/sw/rxe/rxe_resp.c | 123 ++++++++++++++++++-------- drivers/infiniband/sw/rxe/rxe_verbs.h | 1 - 3 files changed, 87 insertions(+), 47 deletions(-)
drivers/infiniband/sw/rxe/rxe_qp.c +2 -8
@@ -135,12 +135,8 @@ static void free_rd_atomic_resources(struct rxe_qp *qp)void free_rd_atomic_resource (struct rxe_qp *qp,struct resp_res *res) { -if (res->type ==RXE_ATOMIC_MASK ) { +if (res->type ==RXE_ATOMIC_MASK )kfree_skb (res->atomic.skb); - }else if (res->type ==RXE_READ_MASK ) { -if (res->read.mr) -rxe_drop_ref (res->read.mr); - } res->type =0 ; }
@@ -816,10 +812,8 @@ static void rxe_qp_do_cleanup(struct work_struct *work)if (qp->pd)rxe_drop_ref (qp->pd); -if (qp->resp.mr) { +if (qp->resp.mr)rxe_drop_ref (qp->resp.mr); - qp->resp.mr =NULL ; - }if (qp_type (qp) ==IB_QPT_RC )sk_dst_reset (qp->sk->sk);
drivers/infiniband/sw/rxe/rxe_resp.c +85 -38
154 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_verbs.h +0 -1
@@ -157,7 +157,6 @@ struct resp_res {struct sk_buff *skb; } atomic;struct { -struct rxe_mr *mr;u64 va_org;u32 rkey;u32 length;
Signature · 1 line
2.32.0
PATCH v6 6/8 RDMA/rxe: Minor cleanups in rxe_pool.c/rxe_pool.h
In reply to Bob PearsonThis patch includes a couple of minor cleanups in rxe_pool.c and rxe_pool.h
--- drivers/infiniband/sw/rxe/rxe_pool.c | 10 +++------- drivers/infiniband/sw/rxe/rxe_pool.h | 1 - 2 files changed, 3 insertions(+), 8 deletions(-)
drivers/infiniband/sw/rxe/rxe_pool.c +3 -7
@@ -97,11 +97,8 @@ static const struct rxe_type_info { }, }; -void rxe_pool_init ( -struct rxe_dev *rxe, -struct rxe_pool *pool, -enum rxe_elem_type type, -unsigned int max_elem) +void rxe_pool_init (struct rxe_dev *rxe,struct rxe_pool *pool, +enum rxe_elem_type type,unsigned int max_elem) {const struct rxe_type_info *info = &rxe_type_info[type];
@@ -109,7 +106,6 @@ void rxe_pool_init( pool->rxe = rxe; pool->name = info->name; - pool->type = type; pool->max_elem = max_elem; pool->elem_size =ALIGN (info->size,RXE_POOL_ALIGN ); pool->elem_offset = info->elem_offset;
@@ -222,7 +218,7 @@ int __rxe_add_to_pool(struct rxe_pool *pool, struct rxe_pool_elem *elem) }/** -* rxe_pool_get_index - lookup object from index +* rxe_pool_get_index() - lookup object from index * @pool: the object pool * @index: the index of the object *
drivers/infiniband/sw/rxe/rxe_pool.h +0 -1
@@ -46,7 +46,6 @@ struct rxe_pool {int (*init)(struct rxe_pool_elem *elem);void (*cleanup)(struct rxe_pool_elem *elem);enum rxe_pool_flags flags; -enum rxe_elem_type type;unsigned int max_elem;atomic_t num_elem;
Signature · 1 line
2.32.0
PATCH v6 7/8 RDMA/rxe: Replace rxe_alloc by kzalloc for rxe_mc_elem
In reply to Bob PearsonCurrently rxe_mc_elem structs are treated as rdma objects which is unneeded. This patch replaces rxe_alloc and rxe_drop_ref by kzalloc and kfree for these structs which hold associatons between multicast groups and QPs.
--- drivers/infiniband/sw/rxe/rxe.c | 3 --- drivers/infiniband/sw/rxe/rxe_mcast.c | 22 ++++++++++++++-------- drivers/infiniband/sw/rxe/rxe_pool.c | 6 ------ drivers/infiniband/sw/rxe/rxe_pool.h | 1 - drivers/infiniband/sw/rxe/rxe_verbs.h | 2 -- 5 files changed, 14 insertions(+), 20 deletions(-)
drivers/infiniband/sw/rxe/rxe.c +0 -3
@@ -31,7 +31,6 @@ void rxe_dealloc(struct ib_device *ib_dev)rxe_pool_cleanup (&rxe->mr_pool);rxe_pool_cleanup (&rxe->mw_pool);rxe_pool_cleanup (&rxe->mc_grp_pool); -rxe_pool_cleanup (&rxe->mc_elem_pool);if (rxe->tfm)crypto_free_shash (rxe->tfm);
@@ -128,8 +127,6 @@ static void rxe_init_pools(struct rxe_dev *rxe)rxe_pool_init (rxe, &rxe->mw_pool,RXE_TYPE_MW , rxe->attr.max_mw);rxe_pool_init (rxe, &rxe->mc_grp_pool,RXE_TYPE_MC_GRP , rxe->attr.max_mcast_grp); -rxe_pool_init (rxe, &rxe->mc_elem_pool,RXE_TYPE_MC_ELEM , - rxe->attr.max_total_mcast_qp_attach); }/* initialize rxe device state */
drivers/infiniband/sw/rxe/rxe_mcast.c +14 -8
@@ -63,14 +63,15 @@ int rxe_mcast_add_grp_elem(struct rxe_dev *rxe, struct rxe_qp *qp, goto out; } - elem =rxe_alloc (&rxe->mc_elem_pool); + elem =kzalloc (sizeof (*elem),GFP_KERNEL );if (!elem) { err = -ENOMEM ; goto out; } -/* each qp holds a ref on the grp */ +/* each elem holds a ref on the grp and the qp */ rxe_add_ref (grp); +rxe_add_ref (qp); grp->num_qp++; elem->qp = qp;
@@ -91,6 +92,7 @@ int rxe_mcast_drop_grp_elem(struct rxe_dev *rxe, struct rxe_qp *qp, {struct rxe_mc_grp *grp;struct rxe_mc_elem *elem, *tmp; +int ret = -EINVAL ; grp =rxe_pool_get_key (&rxe->mc_grp_pool, mgid);if (!grp)
@@ -107,18 +109,21 @@ int rxe_mcast_drop_grp_elem(struct rxe_dev *rxe, struct rxe_qp *qp,spin_unlock_bh (&grp->mcg_lock);spin_unlock_bh (&qp->grp_lock); -rxe_drop_ref (elem); -rxe_drop_ref (grp);/* ref held by QP */ -rxe_drop_ref (grp);/* ref from get_key */ -return 0 ; +kfree (elem); +rxe_drop_ref (qp);/* ref held by elem */ +rxe_drop_ref (grp);/* ref held by elem */ + ret =0 ; + goto out_drop_ref; } }spin_unlock_bh (&grp->mcg_lock);spin_unlock_bh (&qp->grp_lock); + +out_drop_ref:rxe_drop_ref (grp);/* ref from get_key */ err1: -return -EINVAL ; +return ret; }void rxe_drop_all_mcast_groups (struct rxe_qp *qp)
@@ -142,8 +147,9 @@ void rxe_drop_all_mcast_groups(struct rxe_qp *qp)list_del (&elem->qp_list); grp->num_qp--;spin_unlock_bh (&grp->mcg_lock); +rxe_drop_ref (qp);rxe_drop_ref (grp); -rxe_drop_ref (elem); +kfree (elem); } }
drivers/infiniband/sw/rxe/rxe_pool.c +0 -6
@@ -89,12 +89,6 @@ static const struct rxe_type_info { .key_offset = offsetof(struct rxe_mc_grp, mgid), .key_size = sizeof(union ib_gid), }, - [RXE_TYPE_MC_ELEM] = { - .name = "rxe-mc_elem", - .size = sizeof(struct rxe_mc_elem), - .elem_offset = offsetof(struct rxe_mc_elem, elem), - .flags = RXE_POOL_ALLOC, - }, }; void rxe_pool_init(struct rxe_dev *rxe, struct rxe_pool *pool,
drivers/infiniband/sw/rxe/rxe_pool.h +0 -1
@@ -23,7 +23,6 @@ enum rxe_elem_type {RXE_TYPE_MR ,RXE_TYPE_MW ,RXE_TYPE_MC_GRP , -RXE_TYPE_MC_ELEM , RXE_NUM_TYPES, /* keep me last */ };
drivers/infiniband/sw/rxe/rxe_verbs.h +0 -2
@@ -364,7 +364,6 @@ struct rxe_mc_grp { };struct rxe_mc_elem { -struct rxe_pool_elem elem;struct list_head qp_list;struct list_head grp_list;struct rxe_qp *qp;
@@ -402,7 +401,6 @@ struct rxe_dev {struct rxe_pool mr_pool;struct rxe_pool mw_pool;struct rxe_pool mc_grp_pool; -struct rxe_pool mc_elem_pool;spinlock_t pending_lock;/* guard pending_mmaps */ struct list_head pending_mmaps;
Signature · 1 line
2.32.0
PATCH v6 8/8 RDMA/rxe: Add wait for completion to obj destruct
In reply to Bob PearsonThis patch adds code to wait until pending activity on RDMA objects has completed before freeing or returning to rdma-core where the object may be freed.
---
v6
Corrected incorrect comment before __rxe_fini()
Added a #define for complete timeout value.
Changed type of __rxe_fini to int to return value from wait_for_completion.
--- drivers/infiniband/sw/rxe/rxe_comp.c | 4 +- drivers/infiniband/sw/rxe/rxe_mcast.c | 4 ++ drivers/infiniband/sw/rxe/rxe_mr.c | 2 + drivers/infiniband/sw/rxe/rxe_mw.c | 14 +++-- drivers/infiniband/sw/rxe/rxe_pool.c | 31 +++++++++- drivers/infiniband/sw/rxe/rxe_pool.h | 4 ++ drivers/infiniband/sw/rxe/rxe_recv.c | 4 +- drivers/infiniband/sw/rxe/rxe_req.c | 11 ++-- drivers/infiniband/sw/rxe/rxe_resp.c | 6 +- drivers/infiniband/sw/rxe/rxe_verbs.c | 84 ++++++++++++++++++++------- 10 files changed, 126 insertions(+), 38 deletions(-)
drivers/infiniband/sw/rxe/rxe_comp.c +3 -1
@@ -562,7 +562,9 @@ int rxe_completer(void *arg)enum comp_state state;int ret =0 ; -rxe_add_ref (qp); +/* check qp pointer still valid */ +if (!rxe_add_ref (qp)) +return -EAGAIN ;if (!qp->valid || qp->req.state ==QP_STATE_ERROR || qp->req.state ==QP_STATE_RESET ) {
drivers/infiniband/sw/rxe/rxe_mcast.c +4 -0
@@ -122,6 +122,8 @@ int rxe_mcast_drop_grp_elem(struct rxe_dev *rxe, struct rxe_qp *qp, out_drop_ref:rxe_drop_ref (grp);/* ref from get_key */ +if (grp->elem.complete.done) +rxe_fini (grp); err1:return ret; }
@@ -149,6 +151,8 @@ void rxe_drop_all_mcast_groups(struct rxe_qp *qp)spin_unlock_bh (&grp->mcg_lock);rxe_drop_ref (qp);rxe_drop_ref (grp); +if (grp->elem.complete.done) +rxe_fini (grp);kfree (elem); } }
drivers/infiniband/sw/rxe/rxe_mr.c +2 -0
@@ -695,6 +695,8 @@ int rxe_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)rxe_drop_ref (mr_pd (mr));rxe_drop_ref (mr); +rxe_fini (mr); +return 0 ; }
drivers/infiniband/sw/rxe/rxe_mw.c +8 -6
@@ -12,7 +12,8 @@ int rxe_alloc_mw(struct ib_mw *ibmw, struct ib_udata *udata)struct rxe_dev *rxe =to_rdev (ibmw->device);int ret; -rxe_add_ref (pd); +if (!rxe_add_ref (pd)) +return -EINVAL ; ret =rxe_add_to_pool (&rxe->mw_pool, mw);if (ret) {
@@ -60,8 +61,9 @@ int rxe_dealloc_mw(struct ib_mw *ibmw)rxe_do_dealloc_mw (mw);spin_unlock_bh (&mw->lock); -rxe_drop_ref (mw);rxe_drop_ref (pd); +rxe_drop_ref (mw); +rxe_fini (mw);return 0 ; }
@@ -178,11 +180,11 @@ static void rxe_do_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe,if (mw->length) { mw->mr = mr;atomic_inc (&mr->num_mw); -rxe_add_ref (mr); +rxe_add_ref (mr);/* safe */ }if (mw->ibmw.type ==IB_MW_TYPE_2 ) { -rxe_add_ref (qp); +rxe_add_ref (qp);/* safe */ mw->qp = qp; } }
@@ -199,7 +201,7 @@ int rxe_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe) mw =rxe_pool_get_index (&rxe->mw_pool, mw_rkey >>8 );if (unlikely (!mw)) { ret = -EINVAL ; - goto err; + goto err_out; }if (unlikely (mw->rkey != mw_rkey)) {
@@ -236,7 +238,7 @@ int rxe_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe)rxe_drop_ref (mr); err_drop_mw:rxe_drop_ref (mw); -err: +err_out:return ret; }
drivers/infiniband/sw/rxe/rxe_pool.c +30 -1
@@ -6,6 +6,8 @@#include "rxe.h" +/* timeout in jiffies for pool element to complete */ +#define RXE_POOL_TIMEOUT (100)#define RXE_POOL_ALIGN (16) static const struct rxe_type_info {
@@ -150,6 +152,7 @@ static void *__rxe_alloc(struct rxe_pool *pool, gfp_t flags) elem->pool = pool; elem->obj = obj;kref_init (&elem->ref_cnt); +init_completion (&elem->complete);if (pool->init) { err = pool->init (elem);
@@ -189,6 +192,7 @@ int __rxe_add_to_pool(struct rxe_pool *pool, struct rxe_pool_elem *elem) elem->pool = pool; elem->obj = (u8 *)elem - pool->elem_offset;kref_init (&elem->ref_cnt); +init_completion (&elem->complete);if (pool->init) { err = pool->init (elem);
@@ -375,8 +379,33 @@ void rxe_elem_release(struct kref *kref)if (pool->cleanup) pool->cleanup (elem); +atomic_dec (&pool->num_elem); + +complete (&elem->complete); +} + +/** +* __rxe_fini() - wait for completion of pool element +* @elem: the pool elem +* +* Wait until the reference count of an object drops to zero when +* rxe_elem_release() will complete the object and then, if locally +* allocated, free the memory containing the object and return +* +* Returns: non-zero if the object completed successfully else zero +*/ +int __rxe_fini (struct rxe_pool_elem *elem) +{ +struct rxe_pool *pool = elem->pool; +int ret; + + ret =wait_for_completion_timeout (&elem->complete,RXE_POOL_TIMEOUT ); +if (!ret) +pr_warn_ratelimited ("Timed out waiting for %s#%d to complete\n" , + pool->name, elem->index); +if (pool->flags &RXE_POOL_ALLOC )kfree (elem->obj); -atomic_dec (&pool->num_elem); +return ret; }
drivers/infiniband/sw/rxe/rxe_pool.h +4 -0
@@ -30,6 +30,7 @@ struct rxe_pool_elem {struct rxe_pool *pool;void *obj;struct kref ref_cnt; +struct completion complete;struct list_head list;/* only used if keyed */
@@ -105,4 +106,7 @@ static inline bool __rxe_drop_ref(struct rxe_pool_elem *elem) }#define rxe_drop_ref (obj) __rxe_drop_ref(&(obj)->elem) +int __rxe_fini (struct rxe_pool_elem *elem); +#define rxe_fini (obj) __rxe_fini(&(obj)->elem) +#endif /* RXE_POOL_H */
drivers/infiniband/sw/rxe/rxe_recv.c +2 -2
@@ -288,11 +288,11 @@ static void rxe_rcv_mcast_pkt(struct rxe_dev *rxe, struct sk_buff *skb) cpkt =SKB_TO_PKT (cskb); cpkt->qp = qp; -rxe_add_ref (qp); +rxe_add_ref (qp);/* safe */ rxe_rcv_pkt (cpkt, cskb); }else { pkt->qp = qp; -rxe_add_ref (qp); +rxe_add_ref (qp);/* safe */ rxe_rcv_pkt (pkt, skb); skb =NULL ;/* mark consumed */ }
drivers/infiniband/sw/rxe/rxe_req.c +7 -4
@@ -614,9 +614,10 @@ int rxe_requester(void *arg)struct rxe_ah *ah;struct rxe_av *av; -rxe_add_ref (qp); +/* check qp pointer still valid */ +if (!rxe_add_ref (qp)) +return -EAGAIN ; -next_wqe:if (unlikely (!qp->valid || qp->req.state ==QP_STATE_ERROR )) goto exit;
@@ -644,7 +645,7 @@ int rxe_requester(void *arg)if (unlikely (ret)) goto err;else - goto next_wqe; + goto done; } if (unlikely(qp_type(qp) == IB_QPT_RC &&
@@ -760,7 +761,9 @@ int rxe_requester(void *arg)update_state (qp, wqe, &pkt, payload); - goto next_wqe; +done: +rxe_drop_ref (qp); +return 0 ; err_drop_ah:if (ah)
drivers/infiniband/sw/rxe/rxe_resp.c +4 -2
@@ -463,8 +463,8 @@ static enum resp_states check_rkey(struct rxe_qp *qp,if (mw->access &IB_ZERO_BASED ) qp->resp.offset = mw->addr; +rxe_add_ref (mr);/* safe */ rxe_drop_ref (mw); -rxe_add_ref (mr); }else { mr =lookup_mr (qp->pd, access, rkey,RXE_LOOKUP_REMOTE );if (!mr) {
@@ -1247,7 +1247,9 @@ int rxe_responder(void *arg)struct rxe_pkt_info *pkt =NULL ;int ret =0 ; -rxe_add_ref (qp); +/* check qp pointer still valid */ +if (!rxe_add_ref (qp)) +return -EAGAIN ; qp->resp.aeth_syndrome =AETH_ACK_UNLIMITED ;
drivers/infiniband/sw/rxe/rxe_verbs.c +62 -22
221 lines not on this page · raw message
Signature · 1 line
2.32.0
Jason Gunthorpe
maintainerRe: [PATCH for-next v6 8/8] RDMA/rxe: Add wait for completion to obj destruct
Quoted 26 lines · Bob Pearson
This patch adds code to wait until pending activity on RDMA objects has completed before freeing or returning to rdma-core where the object may be freed.
Reported-by kernel test robot <[email protected]>Signed-off-by Bob Pearson <[email protected]>v6
Corrected incorrect comment before__rxe_fini()
Added a #define for complete timeout value.
Changed type of__rxe_finito int to return value fromwait_for_completion.drivers/infiniband/sw/rxe/rxe_comp.c | 4 +- drivers/infiniband/sw/rxe/rxe_mcast.c | 4 ++ drivers/infiniband/sw/rxe/rxe_mr.c | 2 + drivers/infiniband/sw/rxe/rxe_mw.c | 14 +++-- drivers/infiniband/sw/rxe/rxe_pool.c | 31 +++++++++- drivers/infiniband/sw/rxe/rxe_pool.h | 4 ++ drivers/infiniband/sw/rxe/rxe_recv.c | 4 +- drivers/infiniband/sw/rxe/rxe_req.c | 11 ++-- drivers/infiniband/sw/rxe/rxe_resp.c | 6 +- drivers/infiniband/sw/rxe/rxe_verbs.c | 84 ++++++++++++++++++++------- 10 files changed, 126 insertions(+), 38 deletions(-)drivers/infiniband/sw/rxe/rxe_comp.c +1
-0+++ b/drivers/infiniband/sw/rxe/rxe_comp.c
drivers/infiniband/sw/rxe/rxe_comp.c int rxe_completer(void *arg) · line 567
Quoted 7 lines
drivers/infiniband/sw/rxe/rxe_comp.c +3
-1@@ -562,7 +562,9 @@ int rxe_completer(void *arg)enum comp_state state;int ret =0 ; -rxe_add_ref (qp); +/* check qp pointer still valid */ +if (!rxe_add_ref (qp)) +return -EAGAIN ;
This doesn't make sense..
If this already has a pointer to QP then it must already have a ref and add_ref cannot fail.
The kref_get_unless_zero() is a special case for something like a container where it is possible for the container to hold a 0 ref item in it.
The scheme you have makes that impossible because the container lock is held around the kref == 0 and list_del, so you never need unless_zero.
The optimization is to not hold the lock around the kref_get, allow the container to have a 0 ref until the release reaches list_del, and then lock and list_del. The reader side then needs the unless_zero
But the ONLY place unless_zero should be used is in a situation like that, and we should never see other failable refcount tests without some other clear explanation why the qp pointer with a 0 ref is not freed. The above doesn't qualify.
Further
+static inline bool __rxe_add_ref(struct rxe_pool_elem *elem) +{
+ return kref_get_unless_zero(&elem->ref_cnt);
+}
Just serves to defeat refcount debugging which checks that it is impossible for a 0 ref to be incr'd which is proof of a use after free when refcounts are implemetned properly.
So I don't know what this is all about here but it needs some fixing..
Jason
Re: [PATCH for-next v6 8/8] RDMA/rxe: Add wait for completion to obj destruct
Quoted 37 lines · Jason Gunthorpe
Quoted 26 lines · Bob Pearson
This patch adds code to wait until pending activity on RDMA objects has completed before freeing or returning to rdma-core where the object may be freed.
Reported-by kernel test robot <[email protected]>Signed-off-by Bob Pearson <[email protected]>v6
Corrected incorrect comment before__rxe_fini()
Added a #define for complete timeout value.
Changed type of__rxe_finito int to return value fromwait_for_completion.drivers/infiniband/sw/rxe/rxe_comp.c | 4 +- drivers/infiniband/sw/rxe/rxe_mcast.c | 4 ++ drivers/infiniband/sw/rxe/rxe_mr.c | 2 + drivers/infiniband/sw/rxe/rxe_mw.c | 14 +++-- drivers/infiniband/sw/rxe/rxe_pool.c | 31 +++++++++- drivers/infiniband/sw/rxe/rxe_pool.h | 4 ++ drivers/infiniband/sw/rxe/rxe_recv.c | 4 +- drivers/infiniband/sw/rxe/rxe_req.c | 11 ++-- drivers/infiniband/sw/rxe/rxe_resp.c | 6 +- drivers/infiniband/sw/rxe/rxe_verbs.c | 84 ++++++++++++++++++++------- 10 files changed, 126 insertions(+), 38 deletions(-)drivers/infiniband/sw/rxe/rxe_comp.c +1
-0+++ b/drivers/infiniband/sw/rxe/rxe_comp.cdrivers/infiniband/sw/rxe/rxe_comp.c int rxe_completer(void *arg) · line 567
Quoted 7 lines
drivers/infiniband/sw/rxe/rxe_comp.c +3
-1@@ -562,7 +562,9 @@ int rxe_completer(void *arg)enum comp_state state;int ret =0 ; -rxe_add_ref (qp); +/* check qp pointer still valid */ +if (!rxe_add_ref (qp)) +return -EAGAIN ;This doesn't make sense..
Quoted 2 lines
If this already has a pointer to QP then it must already have a ref and
add_refcannot fail.
Good point. My first reaction to using kref_get_unless_zero() which returns a value was to go around and check all the values to make sure no calls failed. But you are correct. All the main tasklets depend on qp still being around since the tasklet struct is contained inside of qp. Further all the qp references have to be freed and the ref from kref_init before qp is freed so this call is both safe from failing and completely unnecessary in the first place.
Bob
Quoted 33 lines
The
kref_get_unless_zero()is a special case for something like a container where it is possible for the container to hold a 0 ref item in it.The scheme you have makes that impossible because the container lock is held around the kref == 0 and
list_del, so you never needunless_zero.The optimization is to not hold the lock around the
kref_get, allow the container to have a 0 ref until the release reacheslist_del, and then lock andlist_del. The reader side then needs theunless_zeroBut the ONLY place
unless_zeroshould be used is in a situation like that, and we should never see other failable refcount tests without some other clear explanation why the qp pointer with a 0 ref is not freed. The above doesn't qualify.Further
(quoted diff) +4
-0+static inline bool __rxe_add_ref(struct rxe_pool_elem *elem) +{ + return kref_get_unless_zero(&elem->ref_cnt); +}Just serves to defeat refcount debugging which checks that it is impossible for a 0 ref to be incr'd which is proof of a use after free when refcounts are implemetned properly.
So I don't know what this is all about here but it needs some fixing..
Jason