Skip to contentSkip to conversation

mlx4: Limit num of fast reg WRs

PATCH In review Eli Cohen 3 participants 6 messages Oct 7, 2010 – Oct 12, 2010 2 messages not in the archive

Subsystems (3)
InfiniBand Mellanox MLX4 core VPI driver Mellanox MLX4 IB driver
lore
Share & tools
lore

Eli Cohen

author
#0 lore raw

PATCH 1/1 mlx4: Limit num of fast reg WRs

Fix the limit of max fast regisreation WRs that can be posted to CX to match hardware capabilities.

Signed-off-by Eli Cohen <eli-VPRAkNaXOzVS1MOuV/[email protected]>
---
 drivers/infiniband/hw/mlx4/main.c |    2 +-
 drivers/infiniband/hw/mlx4/mr.c   |    2 +-
 include/linux/mlx4/device.h       |    4 ++++
 3 files changed, 6 insertions(+), 2 deletions(-)
drivers/infiniband/hw/mlx4/main.c +1 -1
index 4e94e36..829225d 100644
@@ -135,7 +135,7 @@ static int mlx4_ib_query_device(struct ib_device *ibdev,
 	props->max_srq		   = dev->dev->caps.num_srqs - dev->dev->caps.reserved_srqs;
 	props->max_srq_wr	   = dev->dev->caps.max_srq_wqes - 1;
 	props->max_srq_sge	   = dev->dev->caps.max_srq_sge;
-	props->max_fast_reg_page_list_len = PAGE_SIZE / sizeof (u64);
+	props->max_fast_reg_page_list_len = MAX_FAST_REG_PAGES;
 	props->local_ca_ack_delay  = dev->dev->caps.local_ca_ack_delay;
 	props->atomic_cap	   = dev->dev->caps.flags & MLX4_DEV_CAP_FLAG_ATOMIC ?
 		IB_ATOMIC_HCA : IB_ATOMIC_NONE;
drivers/infiniband/hw/mlx4/mr.c +1 -1
index 1d27b9a..83e3cc7 100644
@@ -226,7 +226,7 @@ struct ib_fast_reg_page_list *mlx4_ib_alloc_fast_reg_page_list(struct ib_device
 	struct mlx4_ib_fast_reg_page_list *mfrpl;
 	int size = page_list_len * sizeof (u64);
 
-	if (size > PAGE_SIZE)
+	if (page_list_len > MAX_FAST_REG_PAGES)
 		return ERR_PTR(-EINVAL);
 
 	mfrpl = kmalloc(sizeof *mfrpl, GFP_KERNEL);
include/linux/mlx4/device.h +4 -0
index 7a7f9c1..69fe3f7 100644
@@ -171,6 +171,10 @@ enum {
 	MLX4_NUM_FEXCH          = 64 * 1024,
 };
 
+enum {
+	MAX_FAST_REG_PAGES = 511,
+};
+
 static inline u64 mlx4_fw_ver(u64 major, u64 minor, u64 subminor)
 {
 	return (major << 32) | (minor << 16) | subminor;
Signature · 6 lines
1.7.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#2 lore raw
In reply to Eli Cohen
Quoted 2 lines · Eli Cohen

Fix the limit of max fast regisreation WRs that can be posted to CX to match hardware capabilities.

Guys, can you clarify if the hardware limitation is 511 entries or its (PAGE_SIZE / sizeof(pointer)) - 1 which is 4096 / 8 - 1 = 511 but can change if the page size gets bigger or smaller?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): AANLkTinKiBHOT2sSA+_s5CGhqC8nEoDTp2_pGmo6MrQp-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

author
#4 lore raw
Quoted 4 lines · Or Gerlitz

Guys, can you clarify if the hardware limitation is 511 entries or its (PAGE_SIZE / sizeof(pointer)) - 1 which is 4096 / 8 - 1 = 511 but can change if the page size gets bigger or smaller?

The limit is 511 entries.
After I posted this patch, I was told that there is yet another constraint on the page list: The buffer containing the list must not cross a page boundary. So I was thinking what is the best way to deal with this. One way is to always allocate a whole page and map it using dma_map_page(page, DMA_TO_DEVICE), something like this (not a complete patch, just the idea).

drivers/infiniband/hw/mlx4/mr.c +10 -5
index 83e3cc7..e9b2c8a 100644
@@ -237,18 +237,23 @@ struct ib_fast_reg_page_list *mlx4_ib_alloc_fast_reg_page_list(struct ib_device
 	if (!mfrpl->ibfrpl.page_list)
 		goto err_free;
 
-	mfrpl->mapped_page_list = dma_alloc_coherent(&dev->dev->pdev->dev,
-						     size, &mfrpl->map,
-						     GFP_KERNEL);
+	mfrpl->mapped_page_list = (__be64 *)__get_free_page(GFP_KERNEL);
 	if (!mfrpl->mapped_page_list)
 		goto err_free;
 
-	WARN_ON(mfrpl->map & 0x3f);
+	mfrpl->map = dma_map_single(ibdev->dma_device, mfrpl->mapped_page_list,
+				    PAGE_SIZE, DMA_TO_DEVICE);
+	if (dma_mapping_error(ibdev->dma_device, mfrpl->map))
+		goto err_page;
+
+
 
 	return &mfrpl->ibfrpl;
 
+err_page:
+	free_page((unsigned long) mfrpl->mapped_page_list);
+
 err_free:
-	kfree(mfrpl->ibfrpl.page_list);
 	kfree(mfrpl);
 	return ERR_PTR(-ENOMEM);
 }

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#5 lore raw
Quoted 6 lines

After I posted this patch, I was told that there is yet another constraint on the page list: The buffer containing the list must not cross a page boundary. So I was thinking what is the best way to deal with this. One way is to always allocate a whole page and map it using dma_map_page(page, DMA_TO_DEVICE), something like this (not a complete patch, just the idea).

Is there any chance of the dma_alloc_coherent() in the current code allocating memory that crosses a page boundary?

- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

author
#7 lore raw
Quoted 4 lines · Roland Dreier

Is there any chance of the dma_alloc_coherent() in the current code allocating memory that crosses a page boundary?

You mean that the allocation is aligned at least to its size? I could not find any commitment to this anywhere.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html