Skip to contentSkip to conversation

RDMA/mana_ib: Fix bug in creation of dma regions

PATCH Superseded Konstantin Taranov 5 participants 10 messages Feb 7, 2024 – Feb 9, 2024

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 RDMA/mana_ib: Fix bug in creation of dma regions

From: Konstantin Taranov <[email protected]>

Dma registration was ignoring virtual addresses by setting it to 0. As a result, mana_ib could only register page-aligned memory. As well as, it could fail to produce dma regions with zero offset for WQs and CQs (e.g., page size is 8192 but address is only 4096 bytes aligned), which is required by hardware.

This patch takes into account the virtual address, allowing to create a dma region with any offset. For queues (e.g., WQs, CQs) that require dma regions with zero offset we add a flag to ensure zero offset.

---
 drivers/infiniband/hw/mana/cq.c      |  3 ++-
 drivers/infiniband/hw/mana/main.c    | 16 +++++++++++++---
 drivers/infiniband/hw/mana/mana_ib.h |  2 +-
 drivers/infiniband/hw/mana/mr.c      |  2 +-
 drivers/infiniband/hw/mana/qp.c      |  4 ++--
 drivers/infiniband/hw/mana/wq.c      |  3 ++-
 6 files changed, 21 insertions(+), 9 deletions(-)
drivers/infiniband/hw/mana/cq.c +2 -1
index 83d20c3f0..e35de6b92 100644
@@ -48,7 +48,8 @@ int mana_ib_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
 		return err;
 	}
 
-	err = mana_ib_gd_create_dma_region(mdev, cq->umem, &cq->gdma_region);
+	err = mana_ib_gd_create_dma_region(mdev, cq->umem, &cq->gdma_region,
+					   ucmd.buf_addr, true);
 	if (err) {
 		ibdev_dbg(ibdev,
 			  "Failed to create dma region for create cq, %d\n",
drivers/infiniband/hw/mana/main.c +13 -3
index 29dd2438d..13a4d5ab4 100644
@@ -302,7 +302,7 @@ mana_ib_gd_add_dma_region(struct mana_ib_dev *dev, struct gdma_context *gc,
 }
 
 int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,
-				 mana_handle_t *gdma_region)
+				 mana_handle_t *gdma_region, u64 virt, bool force_zero_offset)
 {
 	struct gdma_dma_region_add_pages_req *add_req = NULL;
 	size_t num_pages_processed = 0, num_pages_to_handle;
@@ -324,11 +324,21 @@ int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,
 	hwc = gc->hwc.driver_data;
 
 	/* Hardware requires dma region to align to chosen page size */
-	page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, 0);
+	page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, virt);
 	if (!page_sz) {
 		ibdev_dbg(&dev->ib_dev, "failed to find page size.\n");
 		return -ENOMEM;
 	}
+
+	if (force_zero_offset) {
+		while (ib_umem_dma_offset(umem, page_sz) && page_sz > PAGE_SIZE)
+			page_sz /= 2;
+		if (ib_umem_dma_offset(umem, page_sz) != 0) {
+			ibdev_dbg(&dev->ib_dev, "failed to find page size to force zero offset.\n");
+			return -ENOMEM;
+		}
+	}
+
 	num_pages_total = ib_umem_num_dma_blocks(umem, page_sz);
 
 	max_pgs_create_cmd =
@@ -348,7 +358,7 @@ int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,
 			     sizeof(struct gdma_create_dma_region_resp));
 
 	create_req->length = umem->length;
-	create_req->offset_in_page = umem->address & (page_sz - 1);
+	create_req->offset_in_page = ib_umem_dma_offset(umem, page_sz);
 	create_req->gdma_page_type = order_base_2(page_sz) - PAGE_SHIFT;
 	create_req->page_count = num_pages_total;
 
drivers/infiniband/hw/mana/mana_ib.h +1 -1
index 6a03ae645..0a5a8f3f8 100644
@@ -161,7 +161,7 @@ static inline struct net_device *mana_ib_get_netdev(struct ib_device *ibdev, u32
 int mana_ib_install_cq_cb(struct mana_ib_dev *mdev, struct mana_ib_cq *cq);
 
 int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,
-				 mana_handle_t *gdma_region);
+				 mana_handle_t *gdma_region, u64 virt, bool force_zero_offset);
 
 int mana_ib_gd_destroy_dma_region(struct mana_ib_dev *dev,
 				  mana_handle_t gdma_region);
drivers/infiniband/hw/mana/mr.c +1 -1
index ee4d4f834..856d73ea2 100644
@@ -127,7 +127,7 @@ struct ib_mr *mana_ib_reg_user_mr(struct ib_pd *ibpd, u64 start, u64 length,
 		goto err_free;
 	}
 
-	err = mana_ib_gd_create_dma_region(dev, mr->umem, &dma_region_handle);
+	err = mana_ib_gd_create_dma_region(dev, mr->umem, &dma_region_handle, iova, false);
 	if (err) {
 		ibdev_dbg(ibdev, "Failed create dma region for user-mr, %d\n",
 			  err);
drivers/infiniband/hw/mana/qp.c +2 -2
index 5d4c05dcd..02de90317 100644
@@ -357,8 +357,8 @@ static int mana_ib_create_qp_raw(struct ib_qp *ibqp, struct ib_pd *ibpd,
 	}
 	qp->sq_umem = umem;
 
-	err = mana_ib_gd_create_dma_region(mdev, qp->sq_umem,
-					   &qp->sq_gdma_region);
+	err = mana_ib_gd_create_dma_region(mdev, qp->sq_umem, &qp->sq_gdma_region,
+					   ucmd.sq_buf_addr, true);
 	if (err) {
 		ibdev_dbg(&mdev->ib_dev,
 			  "Failed to create dma region for create qp-raw, %d\n",
drivers/infiniband/hw/mana/wq.c +2 -1
index 372d36151..d9c1a2d5d 100644
@@ -46,7 +46,8 @@ struct ib_wq *mana_ib_create_wq(struct ib_pd *pd,
 	wq->wq_buf_size = ucmd.wq_buf_size;
 	wq->rx_object = INVALID_MANA_HANDLE;
 
-	err = mana_ib_gd_create_dma_region(mdev, wq->umem, &wq->gdma_region);
+	err = mana_ib_gd_create_dma_region(mdev, wq->umem, &wq->gdma_region,
+					   ucmd.wq_buf_addr, true);
 	if (err) {
 		ibdev_dbg(&mdev->ib_dev,
 			  "Failed to create dma region for create wq, %d\n",

base-commit: aafe4cc5096996873817ff4981a3744e8caf7808

Signature · 1 line
2.43.0
#1 lore raw
Quoted 13 lines · Konstantin Taranov

From: Konstantin Taranov <[email protected]>

Dma registration was ignoring virtual addresses by setting it to 0. As a result, mana_ib could only register page-aligned memory. As well as, it could fail to produce dma regions with zero offset for WQs and CQs (e.g., page size is 8192 but address is only 4096 bytes aligned), which is required by hardware.

This patch takes into account the virtual address, allowing to create a dma region with any offset. For queues (e.g., WQs, CQs) that require dma regions with zero offset we add a flag to ensure zero offset.

Quoted 8 lines
---
 drivers/infiniband/hw/mana/cq.c      |  3 ++-
 drivers/infiniband/hw/mana/main.c    | 16 +++++++++++++---
 drivers/infiniband/hw/mana/mana_ib.h |  2 +-
 drivers/infiniband/hw/mana/mr.c      |  2 +-
 drivers/infiniband/hw/mana/qp.c      |  4 ++--
 drivers/infiniband/hw/mana/wq.c      |  3 ++-
 6 files changed, 21 insertions(+), 9 deletions(-)

You definitely advised to look at the Documentation/process/submitting-patches.rst guide.
1. First revision doesn't need to be v1.
2. One logical fix/change == one patch.
3. Fixes should have Fixes: tag in the commit message.

And I'm confident that the force_zero_offset change is not correct.

Thanks

Quoted 120 lines
drivers/infiniband/hw/mana/cq.c +2 -1
index 83d20c3f0..e35de6b92 100644
@@ -48,7 +48,8 @@ int mana_ib_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
 		return err;
 	}
 
-	err = mana_ib_gd_create_dma_region(mdev, cq->umem, &cq->gdma_region);
+	err = mana_ib_gd_create_dma_region(mdev, cq->umem, &cq->gdma_region,
+					   ucmd.buf_addr, true);
 	if (err) {
 		ibdev_dbg(ibdev,
 			  "Failed to create dma region for create cq, %d\n",
drivers/infiniband/hw/mana/main.c +13 -3
index 29dd2438d..13a4d5ab4 100644
@@ -302,7 +302,7 @@ mana_ib_gd_add_dma_region(struct mana_ib_dev *dev, struct gdma_context *gc,
 }
 
 int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,
-				 mana_handle_t *gdma_region)
+				 mana_handle_t *gdma_region, u64 virt, bool force_zero_offset)
 {
 	struct gdma_dma_region_add_pages_req *add_req = NULL;
 	size_t num_pages_processed = 0, num_pages_to_handle;
@@ -324,11 +324,21 @@ int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,
 	hwc = gc->hwc.driver_data;
 
 	/* Hardware requires dma region to align to chosen page size */
-	page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, 0);
+	page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, virt);
 	if (!page_sz) {
 		ibdev_dbg(&dev->ib_dev, "failed to find page size.\n");
 		return -ENOMEM;
 	}
+
+	if (force_zero_offset) {
+		while (ib_umem_dma_offset(umem, page_sz) && page_sz > PAGE_SIZE)
+			page_sz /= 2;
+		if (ib_umem_dma_offset(umem, page_sz) != 0) {
+			ibdev_dbg(&dev->ib_dev, "failed to find page size to force zero offset.\n");
+			return -ENOMEM;
+		}
+	}
+
 	num_pages_total = ib_umem_num_dma_blocks(umem, page_sz);
 
 	max_pgs_create_cmd =
@@ -348,7 +358,7 @@ int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,
 			     sizeof(struct gdma_create_dma_region_resp));
 
 	create_req->length = umem->length;
-	create_req->offset_in_page = umem->address & (page_sz - 1);
+	create_req->offset_in_page = ib_umem_dma_offset(umem, page_sz);
 	create_req->gdma_page_type = order_base_2(page_sz) - PAGE_SHIFT;
 	create_req->page_count = num_pages_total;
 
drivers/infiniband/hw/mana/mana_ib.h +1 -1
index 6a03ae645..0a5a8f3f8 100644
@@ -161,7 +161,7 @@ static inline struct net_device *mana_ib_get_netdev(struct ib_device *ibdev, u32
 int mana_ib_install_cq_cb(struct mana_ib_dev *mdev, struct mana_ib_cq *cq);
 
 int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,
-				 mana_handle_t *gdma_region);
+				 mana_handle_t *gdma_region, u64 virt, bool force_zero_offset);
 
 int mana_ib_gd_destroy_dma_region(struct mana_ib_dev *dev,
 				  mana_handle_t gdma_region);
drivers/infiniband/hw/mana/mr.c +1 -1
index ee4d4f834..856d73ea2 100644
@@ -127,7 +127,7 @@ struct ib_mr *mana_ib_reg_user_mr(struct ib_pd *ibpd, u64 start, u64 length,
 		goto err_free;
 	}
 
-	err = mana_ib_gd_create_dma_region(dev, mr->umem, &dma_region_handle);
+	err = mana_ib_gd_create_dma_region(dev, mr->umem, &dma_region_handle, iova, false);
 	if (err) {
 		ibdev_dbg(ibdev, "Failed create dma region for user-mr, %d\n",
 			  err);
drivers/infiniband/hw/mana/qp.c +2 -2
index 5d4c05dcd..02de90317 100644
@@ -357,8 +357,8 @@ static int mana_ib_create_qp_raw(struct ib_qp *ibqp, struct ib_pd *ibpd,
 	}
 	qp->sq_umem = umem;
 
-	err = mana_ib_gd_create_dma_region(mdev, qp->sq_umem,
-					   &qp->sq_gdma_region);
+	err = mana_ib_gd_create_dma_region(mdev, qp->sq_umem, &qp->sq_gdma_region,
+					   ucmd.sq_buf_addr, true);
 	if (err) {
 		ibdev_dbg(&mdev->ib_dev,
 			  "Failed to create dma region for create qp-raw, %d\n",
drivers/infiniband/hw/mana/wq.c +2 -1
index 372d36151..d9c1a2d5d 100644
@@ -46,7 +46,8 @@ struct ib_wq *mana_ib_create_wq(struct ib_pd *pd,
 	wq->wq_buf_size = ucmd.wq_buf_size;
 	wq->rx_object = INVALID_MANA_HANDLE;
 
-	err = mana_ib_gd_create_dma_region(mdev, wq->umem, &wq->gdma_region);
+	err = mana_ib_gd_create_dma_region(mdev, wq->umem, &wq->gdma_region,
+					   ucmd.wq_buf_addr, true);
 	if (err) {
 		ibdev_dbg(&mdev->ib_dev,
 			  "Failed to create dma region for create wq, %d\n",

base-commit: aafe4cc5096996873817ff4981a3744e8caf7808

Signature · 1 line
2.43.0
#2 lore raw
Quoted 23 lines

From: Leon Romanovsky <[email protected]>

Quoted 13 lines

From: Konstantin Taranov <[email protected]>

Dma registration was ignoring virtual addresses by setting it to 0. As a result, mana_ib could only register page-aligned memory. As well as, it could fail to produce dma regions with zero offset for WQs and CQs (e.g., page size is 8192 but address is only 4096 bytes aligned), which is required by hardware.

This patch takes into account the virtual address, allowing to create a dma region with any offset. For queues (e.g., WQs, CQs) that require dma regions with zero offset we add a flag to ensure zero offset.

Quoted 8 lines
---
 drivers/infiniband/hw/mana/cq.c      |  3 ++-
 drivers/infiniband/hw/mana/main.c    | 16 +++++++++++++---
 drivers/infiniband/hw/mana/mana_ib.h |  2 +-
 drivers/infiniband/hw/mana/mr.c      |  2 +-
 drivers/infiniband/hw/mana/qp.c      |  4 ++--
 drivers/infiniband/hw/mana/wq.c      |  3 ++-
 6 files changed, 21 insertions(+), 9 deletions(-)
Quoted 3 lines

You definitely advised to look at the Documentation/process/submitting- patches.rst guide.
1. First revision doesn't need to be v1.

Thanks. I did not know that.

Quoted 1 line

2. One logical fix/change == one patch.

It is one fix. If I only replace 0 with virt, the code will stop working as the offset will not be zero quite often. That is why I need to make offset = 0 for queues.

Quoted 1 line

3. Fixes should have Fixes: tag in the commit message.

As existing applications were made to go around this limitation, I wanted this patch arrive to rdma-next. Or do you say that I cannot opt for rdma-next and must make it a "fix"?

Quoted 2 lines

And I'm confident that the force_zero_offset change is not correct.

It was tested with many page sizes and offsets. Could you elaborate why it is not correct?

Thanks!

Quoted 148 lines

Thanks

drivers/infiniband/hw/mana/cq.c int mana_ib_create_cq(struct ib_cq *ibcq, const struct

Quoted 6 lines
(quoted diff) +0 -0

b/drivers/infiniband/hw/mana/cq.c index 83d20c3f0..e35de6b92 100644

drivers/infiniband/hw/mana/cq.c +0 -0
@@ -48,7 +48,8 @@ int mana_ib_create_cq(struct ib_cq *ibcq, const struct

ib_cq_init_attr *attr,

drivers/infiniband/hw/mana/main.c mana_ib_gd_add_dma_region(struct mana_ib_dev

Quoted 17 lines
drivers/infiniband/hw/mana/cq.c +2 -1
              return err;
      }
 
-     err = mana_ib_gd_create_dma_region(mdev, cq->umem, &cq-
 gdma_region);
+     err = mana_ib_gd_create_dma_region(mdev, cq->umem, &cq-
 gdma_region,
+                                        ucmd.buf_addr, true);
      if (err) {

ibdev_dbg(ibdev,
"Failed to create dma region for create cq,
%d\n", diff --git a/drivers/infiniband/hw/mana/main.c b/drivers/infiniband/hw/mana/main.c
index 29dd2438d..13a4d5ab4 100644

drivers/infiniband/hw/mana/main.c +0 -0
@@ -302,7 +302,7 @@ mana_ib_gd_add_dma_region(struct mana_ib_dev

*dev,

drivers/infiniband/hw/mana/main.c mana_ib_gd_add_dma_region(struct mana_ib_dev · line 304

Quoted 3 lines
drivers/infiniband/hw/mana/main.c +0 -0
 struct gdma_context *gc,  }
 
 int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct

ib_umem *umem,

drivers/infiniband/hw/mana/main.c mana_ib_gd_add_dma_region(struct mana_ib_dev

Quoted 7 lines
drivers/infiniband/hw/mana/main.c +3 -1
-                              mana_handle_t *gdma_region)
+                              mana_handle_t *gdma_region, u64 virt,
+ bool force_zero_offset)
 {
      struct gdma_dma_region_add_pages_req *add_req = NULL;
      size_t num_pages_processed = 0, num_pages_to_handle; @@ -324,11
+324,21 @@ int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev,

struct ib_umem *umem,

drivers/infiniband/hw/mana/main.c int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,

Quoted 12 lines
      hwc = gc->hwc.driver_data;

      /* Hardware requires dma region to align to chosen page size */
drivers/infiniband/hw/mana/main.c +4 -1
-     page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, 0);
+     page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, virt);
      if (!page_sz) {
              ibdev_dbg(&dev->ib_dev, "failed to find page size.\n");
              return -ENOMEM;
      }
+
+     if (force_zero_offset) {
+             while (ib_umem_dma_offset(umem, page_sz) && page_sz >

PAGE_SIZE)

drivers/infiniband/hw/mana/main.c int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,

Quoted 3 lines
drivers/infiniband/hw/mana/main.c +3 -0
+                     page_sz /= 2;
+             if (ib_umem_dma_offset(umem, page_sz) != 0) {
+                     ibdev_dbg(&dev->ib_dev, "failed to find page size to force zero

offset.\n");

drivers/infiniband/hw/mana/main.c int mana_ib_gd_create_dma_region(struct

Quoted 8 lines
drivers/infiniband/hw/mana/main.c +4 -0
+                     return -ENOMEM;
+             }
+     }
+
      num_pages_total = ib_umem_num_dma_blocks(umem, page_sz);
 
      max_pgs_create_cmd =
@@ -348,7 +358,7 @@ int mana_ib_gd_create_dma_region(struct

mana_ib_dev *dev, struct ib_umem *umem,

Quoted 19 lines
drivers/infiniband/hw/mana/main.c +1 -1
                           sizeof(struct
 gdma_create_dma_region_resp));
 
      create_req->length = umem->length;
-     create_req->offset_in_page = umem->address & (page_sz - 1);
+     create_req->offset_in_page = ib_umem_dma_offset(umem, page_sz);
      create_req->gdma_page_type = order_base_2(page_sz) - PAGE_SHIFT;
      create_req->page_count = num_pages_total;
drivers/infiniband/hw/mana/mana_ib.h +0 -0
index 6a03ae645..0a5a8f3f8 100644
@@ -161,7 +161,7 @@ static inline struct net_device
 *mana_ib_get_netdev(struct ib_device *ibdev, u32  int

mana_ib_install_cq_cb(struct mana_ib_dev *mdev, struct mana_ib_cq *cq);

Quoted 1 line

int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct

ib_umem *umem,

drivers/infiniband/hw/mana/mr.c struct ib_mr *mana_ib_reg_user_mr(struct ib_pd

Quoted 11 lines
drivers/infiniband/hw/mana/mana_ib.h +2 -1
-                              mana_handle_t *gdma_region);
+                              mana_handle_t *gdma_region, u64 virt,
+ bool force_zero_offset);
 
 int mana_ib_gd_destroy_dma_region(struct mana_ib_dev *dev,
                                mana_handle_t gdma_region); diff --git

a/drivers/infiniband/hw/mana/mr.c b/drivers/infiniband/hw/mana/mr.c index ee4d4f834..856d73ea2 100644

drivers/infiniband/hw/mana/mr.c +0 -0
@@ -127,7 +127,7 @@ struct ib_mr *mana_ib_reg_user_mr(struct ib_pd

*ibpd, u64 start, u64 length,

drivers/infiniband/hw/mana/mr.c struct ib_mr *mana_ib_reg_user_mr(struct ib_pd *ibpd, u64 start, u64 length,

Quoted 4 lines
drivers/infiniband/hw/mana/mr.c +0 -1
              goto err_free;
      }
 
-     err = mana_ib_gd_create_dma_region(dev, mr->umem,

&dma_region_handle);

drivers/infiniband/hw/mana/qp.c static int mana_ib_create_qp_raw(struct ib_qp

Quoted 10 lines
drivers/infiniband/hw/mana/mr.c +2 -0
+     err = mana_ib_gd_create_dma_region(dev, mr->umem,
+ &dma_region_handle, iova, false);
      if (err) {
              ibdev_dbg(ibdev, "Failed create dma region for user-mr, %d\n",
                        err);
(quoted diff) +0 -0

b/drivers/infiniband/hw/mana/qp.c index 5d4c05dcd..02de90317 100644

drivers/infiniband/hw/mana/qp.c +0 -0
@@ -357,8 +357,8 @@ static int mana_ib_create_qp_raw(struct ib_qp

*ibqp, struct ib_pd *ibpd,

Quoted 33 lines
drivers/infiniband/hw/mana/qp.c +2 -2
      }
      qp->sq_umem = umem;
 
-     err = mana_ib_gd_create_dma_region(mdev, qp->sq_umem,
-                                        &qp->sq_gdma_region);
+     err = mana_ib_gd_create_dma_region(mdev, qp->sq_umem, &qp-
 sq_gdma_region,
+                                        ucmd.sq_buf_addr, true);
      if (err) {
              ibdev_dbg(&mdev->ib_dev,

"Failed to create dma region for create
qp-raw, %d\n", diff --git a/drivers/infiniband/hw/mana/wq.c b/drivers/infiniband/hw/mana/wq.c index 372d36151..d9c1a2d5d 100644

drivers/infiniband/hw/mana/wq.c +2 -1
@@ -46,7 +46,8 @@ struct ib_wq *mana_ib_create_wq(struct ib_pd *pd,
      wq->wq_buf_size = ucmd.wq_buf_size;
      wq->rx_object = INVALID_MANA_HANDLE;
 
-     err = mana_ib_gd_create_dma_region(mdev, wq->umem, &wq-
 gdma_region);
+     err = mana_ib_gd_create_dma_region(mdev, wq->umem, &wq-
 gdma_region,
+                                        ucmd.wq_buf_addr, true);
      if (err) {

ibdev_dbg(&mdev->ib_dev,
"Failed to create dma region for create wq,
%d\n",

base-commit: aafe4cc5096996873817ff4981a3744e8caf7808 --
2.43.0

#3 lore raw
Quoted 29 lines · Konstantin Taranov
Quoted 23 lines

From: Leon Romanovsky <[email protected]>

Quoted 13 lines

From: Konstantin Taranov <[email protected]>

Dma registration was ignoring virtual addresses by setting it to 0. As a result, mana_ib could only register page-aligned memory. As well as, it could fail to produce dma regions with zero offset for WQs and CQs (e.g., page size is 8192 but address is only 4096 bytes aligned), which is required by hardware.

This patch takes into account the virtual address, allowing to create a dma region with any offset. For queues (e.g., WQs, CQs) that require dma regions with zero offset we add a flag to ensure zero offset.

Quoted 8 lines
---
 drivers/infiniband/hw/mana/cq.c      |  3 ++-
 drivers/infiniband/hw/mana/main.c    | 16 +++++++++++++---
 drivers/infiniband/hw/mana/mana_ib.h |  2 +-
 drivers/infiniband/hw/mana/mr.c      |  2 +-
 drivers/infiniband/hw/mana/qp.c      |  4 ++--
 drivers/infiniband/hw/mana/wq.c      |  3 ++-
 6 files changed, 21 insertions(+), 9 deletions(-)
Quoted 3 lines

You definitely advised to look at the Documentation/process/submitting- patches.rst guide.
1. First revision doesn't need to be v1.

Thanks. I did not know that.

Quoted 8 lines
Quoted 1 line

2. One logical fix/change == one patch.

It is one fix. If I only replace 0 with virt, the code will stop working as the offset will not be zero quite often. That is why I need to make offset = 0 for queues.

Quoted 1 line

3. Fixes should have Fixes: tag in the commit message.

As existing applications were made to go around this limitation, I wanted this patch arrive to rdma-next. Or do you say that I cannot opt for rdma-next and must make it a "fix"?

Once you write "fix" word in the patch, the expectation is to have Fixes line. There is nothing wrong with applying patch with such tag to rdma-next and we are doing it all the time. Our policy is fluid here and can be summarized as follows:
1. Try to satisfy submitters request to put in specific target rdma-rc/rdma-next.
2. Very lax with taking patches to rdma-rc before -rc4.
3. In general, strict after -rc4, only patches with panics, build breakage and UAPI visible bugs.
4. More pedantic review of -rc material.

So if you write rdma-next in title, add Fixes line which points to "old" code, we will apply your patch to rdma-next.

Quoted 5 lines
Quoted 3 lines

And I'm confident that the force_zero_offset change is not correct.

It was tested with many page sizes and offsets. Could you elaborate why it is not correct?

I prefer that Jason will elaborate more on this, he will do it better than me.

Quoted 151 lines

Thanks!

Quoted 148 lines

Thanks

drivers/infiniband/hw/mana/cq.c int mana_ib_create_cq(struct ib_cq *ibcq, const struct

Quoted 6 lines
(quoted diff) +0 -0

b/drivers/infiniband/hw/mana/cq.c index 83d20c3f0..e35de6b92 100644

drivers/infiniband/hw/mana/cq.c +0 -0
@@ -48,7 +48,8 @@ int mana_ib_create_cq(struct ib_cq *ibcq, const struct

ib_cq_init_attr *attr,

drivers/infiniband/hw/mana/main.c mana_ib_gd_add_dma_region(struct mana_ib_dev

Quoted 17 lines
drivers/infiniband/hw/mana/cq.c +2 -1
              return err;
      }
 
-     err = mana_ib_gd_create_dma_region(mdev, cq->umem, &cq-
 gdma_region);
+     err = mana_ib_gd_create_dma_region(mdev, cq->umem, &cq-
 gdma_region,
+                                        ucmd.buf_addr, true);
      if (err) {

ibdev_dbg(ibdev,
"Failed to create dma region for create cq,
%d\n", diff --git a/drivers/infiniband/hw/mana/main.c b/drivers/infiniband/hw/mana/main.c
index 29dd2438d..13a4d5ab4 100644

drivers/infiniband/hw/mana/main.c +0 -0
@@ -302,7 +302,7 @@ mana_ib_gd_add_dma_region(struct mana_ib_dev

*dev,

drivers/infiniband/hw/mana/main.c mana_ib_gd_add_dma_region(struct mana_ib_dev · line 304

Quoted 3 lines
drivers/infiniband/hw/mana/main.c +0 -0
 struct gdma_context *gc,  }
 
 int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct

ib_umem *umem,

drivers/infiniband/hw/mana/main.c mana_ib_gd_add_dma_region(struct mana_ib_dev

Quoted 7 lines
drivers/infiniband/hw/mana/main.c +3 -1
-                              mana_handle_t *gdma_region)
+                              mana_handle_t *gdma_region, u64 virt,
+ bool force_zero_offset)
 {
      struct gdma_dma_region_add_pages_req *add_req = NULL;
      size_t num_pages_processed = 0, num_pages_to_handle; @@ -324,11
+324,21 @@ int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev,

struct ib_umem *umem,

drivers/infiniband/hw/mana/main.c int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,

Quoted 12 lines
      hwc = gc->hwc.driver_data;

      /* Hardware requires dma region to align to chosen page size */
drivers/infiniband/hw/mana/main.c +4 -1
-     page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, 0);
+     page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, virt);
      if (!page_sz) {
              ibdev_dbg(&dev->ib_dev, "failed to find page size.\n");
              return -ENOMEM;
      }
+
+     if (force_zero_offset) {
+             while (ib_umem_dma_offset(umem, page_sz) && page_sz >

PAGE_SIZE)

drivers/infiniband/hw/mana/main.c int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem,

Quoted 3 lines
drivers/infiniband/hw/mana/main.c +3 -0
+                     page_sz /= 2;
+             if (ib_umem_dma_offset(umem, page_sz) != 0) {
+                     ibdev_dbg(&dev->ib_dev, "failed to find page size to force zero

offset.\n");

drivers/infiniband/hw/mana/main.c int mana_ib_gd_create_dma_region(struct

Quoted 8 lines
drivers/infiniband/hw/mana/main.c +4 -0
+                     return -ENOMEM;
+             }
+     }
+
      num_pages_total = ib_umem_num_dma_blocks(umem, page_sz);
 
      max_pgs_create_cmd =
@@ -348,7 +358,7 @@ int mana_ib_gd_create_dma_region(struct

mana_ib_dev *dev, struct ib_umem *umem,

Quoted 19 lines
drivers/infiniband/hw/mana/main.c +1 -1
                           sizeof(struct
 gdma_create_dma_region_resp));
 
      create_req->length = umem->length;
-     create_req->offset_in_page = umem->address & (page_sz - 1);
+     create_req->offset_in_page = ib_umem_dma_offset(umem, page_sz);
      create_req->gdma_page_type = order_base_2(page_sz) - PAGE_SHIFT;
      create_req->page_count = num_pages_total;
drivers/infiniband/hw/mana/mana_ib.h +0 -0
index 6a03ae645..0a5a8f3f8 100644
@@ -161,7 +161,7 @@ static inline struct net_device
 *mana_ib_get_netdev(struct ib_device *ibdev, u32  int

mana_ib_install_cq_cb(struct mana_ib_dev *mdev, struct mana_ib_cq *cq);

Quoted 1 line

int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct

ib_umem *umem,

drivers/infiniband/hw/mana/mr.c struct ib_mr *mana_ib_reg_user_mr(struct ib_pd

Quoted 11 lines
drivers/infiniband/hw/mana/mana_ib.h +2 -1
-                              mana_handle_t *gdma_region);
+                              mana_handle_t *gdma_region, u64 virt,
+ bool force_zero_offset);
 
 int mana_ib_gd_destroy_dma_region(struct mana_ib_dev *dev,
                                mana_handle_t gdma_region); diff --git

a/drivers/infiniband/hw/mana/mr.c b/drivers/infiniband/hw/mana/mr.c index ee4d4f834..856d73ea2 100644

drivers/infiniband/hw/mana/mr.c +0 -0
@@ -127,7 +127,7 @@ struct ib_mr *mana_ib_reg_user_mr(struct ib_pd

*ibpd, u64 start, u64 length,

drivers/infiniband/hw/mana/mr.c struct ib_mr *mana_ib_reg_user_mr(struct ib_pd *ibpd, u64 start, u64 length,

Quoted 4 lines
drivers/infiniband/hw/mana/mr.c +0 -1
              goto err_free;
      }
 
-     err = mana_ib_gd_create_dma_region(dev, mr->umem,

&dma_region_handle);

drivers/infiniband/hw/mana/qp.c static int mana_ib_create_qp_raw(struct ib_qp

Quoted 10 lines
drivers/infiniband/hw/mana/mr.c +2 -0
+     err = mana_ib_gd_create_dma_region(dev, mr->umem,
+ &dma_region_handle, iova, false);
      if (err) {
              ibdev_dbg(ibdev, "Failed create dma region for user-mr, %d\n",
                        err);
(quoted diff) +0 -0

b/drivers/infiniband/hw/mana/qp.c index 5d4c05dcd..02de90317 100644

drivers/infiniband/hw/mana/qp.c +0 -0
@@ -357,8 +357,8 @@ static int mana_ib_create_qp_raw(struct ib_qp

*ibqp, struct ib_pd *ibpd,

Quoted 33 lines
drivers/infiniband/hw/mana/qp.c +2 -2
      }
      qp->sq_umem = umem;
 
-     err = mana_ib_gd_create_dma_region(mdev, qp->sq_umem,
-                                        &qp->sq_gdma_region);
+     err = mana_ib_gd_create_dma_region(mdev, qp->sq_umem, &qp-
 sq_gdma_region,
+                                        ucmd.sq_buf_addr, true);
      if (err) {
              ibdev_dbg(&mdev->ib_dev,

"Failed to create dma region for create
qp-raw, %d\n", diff --git a/drivers/infiniband/hw/mana/wq.c b/drivers/infiniband/hw/mana/wq.c index 372d36151..d9c1a2d5d 100644

drivers/infiniband/hw/mana/wq.c +2 -1
@@ -46,7 +46,8 @@ struct ib_wq *mana_ib_create_wq(struct ib_pd *pd,
      wq->wq_buf_size = ucmd.wq_buf_size;
      wq->rx_object = INVALID_MANA_HANDLE;
 
-     err = mana_ib_gd_create_dma_region(mdev, wq->umem, &wq-
 gdma_region);
+     err = mana_ib_gd_create_dma_region(mdev, wq->umem, &wq-
 gdma_region,
+                                        ucmd.wq_buf_addr, true);
      if (err) {

ibdev_dbg(&mdev->ib_dev,
"Failed to create dma region for create wq,
%d\n",

base-commit: aafe4cc5096996873817ff4981a3744e8caf7808 --
2.43.0

#4 lore raw
In reply to Konstantin Taranov

drivers/infiniband/hw/mana/main.c int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem, · line 331

Quoted 8 lines
drivers/infiniband/hw/mana/main.c +1 -1
 	/* Hardware requires dma region to align to chosen page size */
-	page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, 0);
+	page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, virt);
 	if (!page_sz) {
 		ibdev_dbg(&dev->ib_dev, "failed to find page size.\n");
 		return -ENOMEM;
 	}

How about doing:
page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, force_zero_offset ? 0 : virt);

Will this work? This can get rid of the following while loop.

Quoted 12 lines
drivers/infiniband/hw/mana/main.c +3 -0
+
+	if (force_zero_offset) {
+		while (ib_umem_dma_offset(umem, page_sz) && page_sz >

PAGE_SIZE)

drivers/infiniband/hw/mana/main.c +3 -0
+			page_sz /= 2;
+		if (ib_umem_dma_offset(umem, page_sz) != 0) {
+			ibdev_dbg(&dev->ib_dev, "failed to find page size to

force zero offset.\n");

drivers/infiniband/hw/mana/main.c +4 -0
+			return -ENOMEM;
+		}
+	}
+
#5 lore raw
Quoted 22 lines

From: Long Li <[email protected]>
Sent: Thursday, 8 February 2024 19:43
To: Konstantin Taranov <[email protected]>; Konstantin Taranov <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]
Subject: RE: [PATCH rdma-next v1 1/1] RDMA/mana_ib: Fix bug in creation of dma regions

drivers/infiniband/hw/mana/main.c int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem, · line 331

Quoted 8 lines
drivers/infiniband/hw/mana/main.c +1 -1
 	/* Hardware requires dma region to align to chosen page size */
-	page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, 0);
+	page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, virt);
 	if (!page_sz) {
 		ibdev_dbg(&dev->ib_dev, "failed to find page size.\n");
 		return -ENOMEM;
 	}

How about doing:
page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, force_zero_offset ? 0 : virt);

Quoted 2 lines

Will this work? This can get rid of the following while loop.

I do not think so. I mentioned once, that it was failing for me with existing code with the 4K-aligned addresses and 8K pages. In this case, we miscalculate the number of pages. So, we think that it is one 8K page, but it is in fact two.

Quoted 12 lines
Quoted 12 lines
drivers/infiniband/hw/mana/main.c +3 -0
+
+	if (force_zero_offset) {
+		while (ib_umem_dma_offset(umem, page_sz) && page_sz >

PAGE_SIZE)

drivers/infiniband/hw/mana/main.c +3 -0
+			page_sz /= 2;
+		if (ib_umem_dma_offset(umem, page_sz) != 0) {
+			ibdev_dbg(&dev->ib_dev, "failed to find page size to

force zero offset.\n");

drivers/infiniband/hw/mana/main.c +4 -0
+			return -ENOMEM;
+		}
+	}
+
#6 lore raw
Quoted 22 lines · Konstantin Taranov
Quoted 22 lines

From: Long Li <[email protected]>
Sent: Thursday, 8 February 2024 19:43
To: Konstantin Taranov <[email protected]>; Konstantin Taranov <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]
Subject: RE: [PATCH rdma-next v1 1/1] RDMA/mana_ib: Fix bug in creation of dma regions

drivers/infiniband/hw/mana/main.c int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem, · line 331

Quoted 8 lines
drivers/infiniband/hw/mana/main.c +1 -1
 	/* Hardware requires dma region to align to chosen page size */
-	page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, 0);
+	page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, virt);
 	if (!page_sz) {
 		ibdev_dbg(&dev->ib_dev, "failed to find page size.\n");
 		return -ENOMEM;
 	}

How about doing:
page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, force_zero_offset ? 0 : virt);

Quoted 6 lines
Quoted 2 lines

Will this work? This can get rid of the following while loop.

I do not think so. I mentioned once, that it was failing for me with existing code with the 4K-aligned addresses and 8K pages. In this case, we miscalculate the number of pages. So, we think that it is one 8K page, but it is in fact two.

That is a confusing statement.. What is "we" here?

ib_umem_dma_offset() is not always guaranteed to be zero, with a 0 iova. With higher order pages the offset can be within the page, it generates

offset = IOVA % pgsz

There are a couple places that do want the offset to be fixed to zero and have the loop, at this point it would be good to consolidate them into some common ib_umem_find_best_pgsz_zero_offset() or something.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/infiniband/hw/mana/main.c +3 -0
+
+	if (force_zero_offset) {
+		while (ib_umem_dma_offset(umem, page_sz) && page_sz >

PAGE_SIZE)

Quoted 8 lines
Quoted 8 lines

drivers/infiniband/hw/mana/main.c int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem, · line 341

Quoted 8 lines
drivers/infiniband/hw/mana/main.c +3 -0
+			page_sz /= 2;
+		if (ib_umem_dma_offset(umem, page_sz) != 0) {
+			ibdev_dbg(&dev->ib_dev, "failed to find page size to

force zero offset.\n");

drivers/infiniband/hw/mana/main.c +4 -0
+			return -ENOMEM;
+		}
+	}
+

Yes this doesn't look quite right..

It should flow from the HW capability, the helper you call should be tightly linked to what the HW can do.

ib_umem_find_best_pgsz() is used for MRs that have the usual
offset = IOVA % pgsz

We've always created other helpers for other restrictions.

So you should move your "force_zero_offset" into another helper and describe exactly how the HW works to support the calculation

It is odd to have the offset loop and be using ib_umem_find_best_pgsz() with some iova, usually you'd use ib_umem_find_best_pgoff() in those cases, see the other callers.

Jason

#7 lore raw
Quoted 28 lines

From: Jason Gunthorpe <[email protected]>

Quoted 18 lines · Konstantin Taranov
Quoted 18 lines

From: Long Li <[email protected]>
Sent: Thursday, 8 February 2024 19:43
To: Konstantin Taranov <[email protected]>; Konstantin Taranov <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]
Subject: RE: [PATCH rdma-next v1 1/1] RDMA/mana_ib: Fix bug in creation of dma regions

Quoted 8 lines

/* Hardware requires dma region to align to chosen page size */
- page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, 0);
+ page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, virt);

  if (!page_sz) {
          ibdev_dbg(&dev->ib_dev, "failed to find page size.\n");
          return -ENOMEM;

}

Quoted 2 lines
Quoted 2 lines

How about doing:
page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM,

force_zero_offset

Quoted 5 lines
Quoted 4 lines

? 0 : virt);

Will this work? This can get rid of the following while loop.

Quoted 6 lines
Quoted 3 lines

I do not think so. I mentioned once, that it was failing for me with existing code with the 4K-aligned addresses and 8K pages. In this case, we miscalculate the number of pages. So, we think that it is one 8K

page, but it is in fact two.

That is a confusing statement.. What is "we" here?

Sorry, I meant here the helper thinks that it is one 8K page for 8K buffer. But the offset will not not zero when the buffer is only 4K aligned. So the actual Number of pages is 2, but the helper thinks that it is one because of wrong virt.

Quoted 32 lines

ib_umem_dma_offset() is not always guaranteed to be zero, with a 0 iova. With higher order pages the offset can be within the page, it generates

offset = IOVA % pgsz

There are a couple places that do want the offset to be fixed to zero and have the loop, at this point it would be good to consolidate them into some common ib_umem_find_best_pgsz_zero_offset() or something.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/infiniband/hw/mana/main.c +3 -0
+
+ if (force_zero_offset) {
+         while (ib_umem_dma_offset(umem, page_sz) && page_sz >

PAGE_SIZE)
+ page_sz /= 2;

Quoted 8 lines
Quoted 8 lines

drivers/infiniband/hw/mana/main.c int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem, · line 341

Quoted 8 lines
(quoted diff) +3 -0
+         if (ib_umem_dma_offset(umem, page_sz) != 0) {
+                 ibdev_dbg(&dev->ib_dev, "failed to find page
+ size to

force zero offset.\n");

drivers/infiniband/hw/mana/main.c +4 -0
+                 return -ENOMEM;
+         }
+ }
+

Yes this doesn't look quite right..

It should flow from the HW capability, the helper you call should be tightly linked to what the HW can do.

ib_umem_find_best_pgsz() is used for MRs that have the usual
offset = IOVA % pgsz

Quoted 8 lines

We've always created other helpers for other restrictions.

So you should move your "force_zero_offset" into another helper and describe exactly how the HW works to support the calculation

It is odd to have the offset loop and be using ib_umem_find_best_pgsz() with some iova, usually you'd use ib_umem_find_best_pgoff() in those cases, see the other callers.

Hi Jason,
Thanks for the comments.

To be honest, I do not understand how I could employ ib_umem_find_best_pgoff for my purpose. As well as I do not see any mistake in the patch, and I think you neither.

I can explain the logic behind the code, and could you say what I need to change to get this patch accepted?

Our HW cannot create a work queue or a completion queue from a dma region that has non zero offset. As a result, applications must allocate at least 4K-aligned memory for such queues. As a queue can be long, the helper functions may suggest large page sizes (let's call it X), but compromise the zero offset. As we see that the offset is not zero, we half X till first page size that can offer us zero offset. This page size will be at least 4K. If we cannot find such page size, it means that the user did not provide 4K-aligned buffer.

I can make a special helper, but I do not think that it will be useful to anyone. Plus, there is no better approach then halving the page size, so the helper will end up with that loop under the hood. As I see mlnx also uses a loop with halving page_sz, but for a different purpose, I do not see why our code cannot do the same without a special helper.

Quoted 2 lines

Jason

#8 lore raw
Quoted 31 lines
Quoted 22 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/infiniband/hw/mana/main.c +3 -0
+
+ if (force_zero_offset) {
+         while (ib_umem_dma_offset(umem, page_sz) && page_sz >

PAGE_SIZE)
+ page_sz /= 2;

Quoted 8 lines
Quoted 8 lines

drivers/infiniband/hw/mana/main.c int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem, · line 341

Quoted 8 lines
(quoted diff) +3 -0
+         if (ib_umem_dma_offset(umem, page_sz) != 0) {
+                 ibdev_dbg(&dev->ib_dev, "failed to find page
+ size to

force zero offset.\n");

drivers/infiniband/hw/mana/main.c +4 -0
+                 return -ENOMEM;
+         }
+ }
+

Yes this doesn't look quite right..

It should flow from the HW capability, the helper you call should be tightly linked to what the HW can do.

ib_umem_find_best_pgsz() is used for MRs that have the usual
offset = IOVA % pgsz

Quoted 8 lines

We've always created other helpers for other restrictions.

So you should move your "force_zero_offset" into another helper and describe exactly how the HW works to support the calculation

It is odd to have the offset loop and be using ib_umem_find_best_pgsz() with some iova, usually you'd use ib_umem_find_best_pgoff() in those cases, see the other callers.

Quoted 5 lines

Hi Jason,
Thanks for the comments.

To be honest, I do not understand how I could employ ib_umem_find_best_pgoff for my purpose. As well as I do not see any mistake in the patch, and I think you neither.

It does exactly the same thing, it is just intended to be used by things that are not doing the IOVA calculation. It is a matter of documentation.

Quoted 4 lines

I can make a special helper, but I do not think that it will be useful to anyone. Plus, there is no better approach then halving the page size, so the helper will end up with that loop under the hood. As I see mlnx also uses a loop with halving page_sz, but for a different purpose, I do not see why our code cannot do the same without a special helper.

Are you sure you don't need the length check too? You have a granular size but not a granular offset?

In that case yes, a helper does not seem necessary

However, you should still be calling ib_umem_find_best_pgoff() for the initialize sizing as a matter of clarity since this is not a MR and does not use IOVA addressing.

Jason

#9 lore raw
Quoted 43 lines

From: Jason Gunthorpe <[email protected]>

Quoted 32 lines
Quoted 23 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
drivers/infiniband/hw/mana/main.c +4 -0
+
+ if (force_zero_offset) {
+         while (ib_umem_dma_offset(umem, page_sz) && page_sz
+ >

PAGE_SIZE)
+ page_sz /= 2;

Quoted 8 lines
Quoted 8 lines

drivers/infiniband/hw/mana/main.c int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem, · line 341

Quoted 8 lines
(quoted diff) +3 -0
+         if (ib_umem_dma_offset(umem, page_sz) != 0) {
+                 ibdev_dbg(&dev->ib_dev, "failed to find page
+ size to

force zero offset.\n");

drivers/infiniband/hw/mana/main.c +4 -0
+                 return -ENOMEM;
+         }
+ }
+

Yes this doesn't look quite right..

It should flow from the HW capability, the helper you call should be tightly linked to what the HW can do.

ib_umem_find_best_pgsz() is used for MRs that have the usual
offset = IOVA % pgsz

Quoted 8 lines

We've always created other helpers for other restrictions.

So you should move your "force_zero_offset" into another helper and describe exactly how the HW works to support the calculation

It is odd to have the offset loop and be using ib_umem_find_best_pgsz() with some iova, usually you'd use ib_umem_find_best_pgoff() in those cases, see the other callers.

Quoted 5 lines

Hi Jason,
Thanks for the comments.

To be honest, I do not understand how I could employ ib_umem_find_best_pgoff for my purpose. As well as I do not see any

mistake in the patch, and I think you neither.

It does exactly the same thing, it is just intended to be used by things that are not doing the IOVA calculation. It is a matter of documentation.

Quoted 8 lines
Quoted 4 lines

I can make a special helper, but I do not think that it will be useful to anyone. Plus, there is no better approach then halving the page size, so the helper will end up with that loop under the hood. As I see mlnx also uses a loop with halving page_sz, but for a different purpose,

I do not see why our code cannot do the same without a special helper.

Are you sure you don't need the length check too? You have a granular size but not a granular offset?

Yes, we do not have constraints on the size.

Quoted 6 lines

In that case yes, a helper does not seem necessary

However, you should still be calling ib_umem_find_best_pgoff() for the initialize sizing as a matter of clarity since this is not a MR and does not use IOVA addressing.

Thanks for the clarification! I agree that the use of ib_umem_find_best_pgoff() will make the code more understandable, even though it will do the same computation. I have already prepared the patch. I will send it next week after running tests.

Quoted 2 lines

Jason