Skip to contentSkip to conversation

RDMA/rxe: Set pd early in mr alloc routines

PATCH In review Bob Pearson 3 participants 7 messages Aug 5, 2022 – Aug 31, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH v5 1/2 RDMA/rxe: Set pd early in mr alloc routines

Move setting of pd in mr objects ahead of any possible errors so that it will always be set in rxe_mr_complete() to avoid seg faults when rxe_put(mr_pd(mr)) is called.

---
 drivers/infiniband/sw/rxe/rxe_loc.h   |  6 +++---
 drivers/infiniband/sw/rxe/rxe_mr.c    | 11 ++++-------
 drivers/infiniband/sw/rxe/rxe_verbs.c | 10 +++++++---
 3 files changed, 14 insertions(+), 13 deletions(-)
drivers/infiniband/sw/rxe/rxe_loc.h +3 -3
index 22f6cc31d1d6..c2a5c8814a48 100644
@@ -64,10 +64,10 @@ int rxe_mmap(struct ib_ucontext *context, struct vm_area_struct *vma);
 
 /* rxe_mr.c */
 u8 rxe_get_next_key(u32 last_key);
-void rxe_mr_init_dma(struct rxe_pd *pd, int access, struct rxe_mr *mr);
-int rxe_mr_init_user(struct rxe_pd *pd, u64 start, u64 length, u64 iova,
+void rxe_mr_init_dma(int access, struct rxe_mr *mr);
+int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
 		     int access, struct rxe_mr *mr);
-int rxe_mr_init_fast(struct rxe_pd *pd, int max_pages, struct rxe_mr *mr);
+int rxe_mr_init_fast(int max_pages, struct rxe_mr *mr);
 int rxe_mr_copy(struct rxe_mr *mr, u64 iova, void *addr, int length,
 		enum rxe_mr_copy_dir dir);
 int copy_data(struct rxe_pd *pd, int access, struct rxe_dma_info *dma,
drivers/infiniband/sw/rxe/rxe_mr.c +4 -7
index 850b80f5ad8b..af34f198e645 100644
@@ -103,17 +103,16 @@ static int rxe_mr_alloc(struct rxe_mr *mr, int num_buf)
 	return -ENOMEM;
 }
 
-void rxe_mr_init_dma(struct rxe_pd *pd, int access, struct rxe_mr *mr)
+void rxe_mr_init_dma(int access, struct rxe_mr *mr)
 {
 	rxe_mr_init(access, mr);
 
-	mr->ibmr.pd = &pd->ibpd;
 	mr->access = access;
 	mr->state = RXE_MR_STATE_VALID;
 	mr->type = IB_MR_TYPE_DMA;
 }
 
-int rxe_mr_init_user(struct rxe_pd *pd, u64 start, u64 length, u64 iova,
+int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
 		     int access, struct rxe_mr *mr)
 {
 	struct rxe_map		**map;
@@ -125,7 +124,7 @@ int rxe_mr_init_user(struct rxe_pd *pd, u64 start, u64 length, u64 iova,
 	int err;
 	int i;
 
-	umem = ib_umem_get(pd->ibpd.device, start, length, access);
+	umem = ib_umem_get(&rxe->ib_dev, start, length, access);
 	if (IS_ERR(umem)) {
 		pr_warn("%s: Unable to pin memory region err = %d\n",
 			__func__, (int)PTR_ERR(umem));
@@ -175,7 +174,6 @@ int rxe_mr_init_user(struct rxe_pd *pd, u64 start, u64 length, u64 iova,
 		}
 	}
 
-	mr->ibmr.pd = &pd->ibpd;
 	mr->umem = umem;
 	mr->access = access;
 	mr->length = length;
@@ -197,7 +195,7 @@ int rxe_mr_init_user(struct rxe_pd *pd, u64 start, u64 length, u64 iova,
 	return err;
 }
 
-int rxe_mr_init_fast(struct rxe_pd *pd, int max_pages, struct rxe_mr *mr)
+int rxe_mr_init_fast(int max_pages, struct rxe_mr *mr)
 {
 	int err;
 
@@ -208,7 +206,6 @@ int rxe_mr_init_fast(struct rxe_pd *pd, int max_pages, struct rxe_mr *mr)
 	if (err)
 		goto err1;
 
-	mr->ibmr.pd = &pd->ibpd;
 	mr->max_buf = max_pages;
 	mr->state = RXE_MR_STATE_FREE;
 	mr->type = IB_MR_TYPE_MEM_REG;
drivers/infiniband/sw/rxe/rxe_verbs.c +7 -3
index e264cf69bf55..6c13be14d723 100644
@@ -903,7 +903,9 @@ static struct ib_mr *rxe_get_dma_mr(struct ib_pd *ibpd, int access)
 		return ERR_PTR(-ENOMEM);
 
 	rxe_get(pd);
-	rxe_mr_init_dma(pd, access, mr);
+	mr->ibmr.pd = ibpd;
+
+	rxe_mr_init_dma(access, mr);
 	rxe_finalize(mr);
 
 	return &mr->ibmr;
@@ -928,8 +930,9 @@ static struct ib_mr *rxe_reg_user_mr(struct ib_pd *ibpd,
 
 
 	rxe_get(pd);
+	mr->ibmr.pd = ibpd;
 
-	err = rxe_mr_init_user(pd, start, length, iova, access, mr);
+	err = rxe_mr_init_user(rxe, start, length, iova, access, mr);
 	if (err)
 		goto err3;
 
@@ -962,8 +965,9 @@ static struct ib_mr *rxe_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type mr_type,
 	}
 
 	rxe_get(pd);
+	mr->ibmr.pd = ibpd;
 
-	err = rxe_mr_init_fast(pd, max_num_sg, mr);
+	err = rxe_mr_init_fast(max_num_sg, mr);
 	if (err)
 		goto err2;
 
Signature · 1 line
2.34.1
#2 lore raw

BTW: this patch should be for-rc instead.

Quoted 12 lines · On 22/08/2022 13:46, [email protected] wrote:

Bob,

Sorry for the late reply. Just back to office from a vacation :)

Quoted 5 lines

Move setting of pd in mr objects ahead of any possible errors so that it will always be set in rxe_mr_complete() to avoid seg faults when rxe_put(mr_pd(mr)) is called.

Looks good to me

#3 lore raw
Quoted 1 line · Li Zhijian

BTW: this patch should be for-rc instead.

It can't be in -rc without Fixes line and more clear description.

Thanks

#4 lore raw

Leon,

Quoted 3 lines · Leon Romanovsky
Quoted 1 line · Li Zhijian

BTW: this patch should be for-rc instead.

It can't be in -rc without Fixes line and more clear description.

Fixes cf40367961d8 ("RDMA/rxe: Move mr cleanup code to rxe_mr_cleanup()") ?

IIRC, This patch fixes a kernel crash newly introduced by v6.0 merge window
it's an alternative for my former patch: https://lore.kernel.org/all/[email protected]/
So if you agree it should go to for-rc, does Bob need to repost a new one with more details ?

Quoted 2 lines

Thanks

#5 lore raw
Quoted 5 lines · Li Zhijian

Leon,

Quoted 1 line · Leon Romanovsky

On Wed, Aug 31, 2022 at 01:48:08PM +0800, Li Zhijian wrote:

Quoted 8 lines
Quoted 2 lines
Quoted 1 line

BTW: this patch should be for-rc instead.

It can't be in -rc without Fixes line and more clear description.

Fixes cf40367961d8 ("RDMA/rxe: Move mr cleanup code to rxe_mr_cleanup()") ?

IIRC, This patch fixes a kernel crash newly introduced by v6.0 merge window
it's an alternative for my former patch: https://lore.kernel.org/all/[email protected]/
So if you agree it should go to for-rc, does Bob need to repost a new one with more details ?

Yes, please, especially kernel panic.

Quoted 5 lines
Quoted 4 lines

Thanks

#6 lore raw
In reply to Bob Pearson
Quoted 10 lines · Bob Pearson

Move setting of pd in mr objects ahead of any possible errors so that it will always be set in rxe_mr_complete() to avoid seg faults when rxe_put(mr_pd(mr)) is called.

---
 drivers/infiniband/sw/rxe/rxe_loc.h   |  6 +++---
 drivers/infiniband/sw/rxe/rxe_mr.c    | 11 ++++-------
 drivers/infiniband/sw/rxe/rxe_verbs.c | 10 +++++++---
 3 files changed, 14 insertions(+), 13 deletions(-)

I see only one patch out of two "v5 for-next 1/2". Where is the second one?

Thanks