Skip to contentSkip to conversation

RDMA/efa: Use __GFP_RETRY_MAYFAIL for ring allocations

PATCH Applied Michael Margolin 4 participants 6 messages Sep 17 – Sep 28

Subsystems (2)
InfiniBand Amazon RDMA efa driver
lore
Share & tools
lore

Michael Margolin

author maintainer
#0 lore raw

PATCH 1/1 RDMA/efa: Use __GFP_RETRY_MAYFAIL for ring allocations

RQ and CQ ring buffers require physically contiguous memory. On systems under memory pressure, high-order allocations can fail even when sufficient free memory exists because the allocator gives up after a single reclaim and compaction attempt for costly-order requests that lack __GFP_RETRY_MAYFAIL.

Add __GFP_RETRY_MAYFAIL to the GFP flags in efa_zalloc_mapped() to allow retries with escalating compaction priority and release of highatomic reserves before giving up.

Reviewed-by Firas Jahjah <[email protected]>
Reviewed-by Yonatan Nachum <[email protected]>
Signed-off-by Michael Margolin <[email protected]>
---
 drivers/infiniband/hw/efa/efa_verbs.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/hw/efa/efa_verbs.c +1 -1
index 2d28d68efe77..0b0396e190f2 100644
@@ -190,7 +190,7 @@ static void *efa_zalloc_mapped(struct efa_dev *dev, dma_addr_t *dma_addr,
 {
 	void *addr;
 
-	addr = alloc_pages_exact(size, GFP_KERNEL | __GFP_ZERO);
+	addr = alloc_pages_exact(size, GFP_KERNEL | __GFP_ZERO | __GFP_RETRY_MAYFAIL);
 	if (!addr)
 		return NULL;
Signature · 1 line
2.47.3
AI review
#1 lore raw
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 6 lines

RQ and CQ ring buffers require physically contiguous memory. On systems under memory pressure, high-order allocations can fail even when sufficient free memory exists because the allocator gives up after a single reclaim and compaction attempt for costly-order requests that lack __GFP_RETRY_MAYFAIL.

Quoted 7 lines

Add __GFP_RETRY_MAYFAIL to the GFP flags in efa_zalloc_mapped() to allow retries with escalating compaction priority and release of highatomic reserves before giving up.

Reviewed-by Firas Jahjah <[email protected]>
Reviewed-by Yonatan Nachum <[email protected]>
Signed-off-by Michael Margolin <[email protected]>

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#2 lore raw
In reply to Michael Margolin
Quoted 5 lines · Michael Margolin

RQ and CQ ring buffers require physically contiguous memory. On systems under memory pressure, high-order allocations can fail even when sufficient free memory exists because the allocator gives up after a single reclaim and compaction attempt for costly-order requests that lack __GFP_RETRY_MAYFAIL.

What sizes did you see fail, and in which flow?

Thanks

Quoted 28 lines

Add __GFP_RETRY_MAYFAIL to the GFP flags in efa_zalloc_mapped() to allow retries with escalating compaction priority and release of highatomic reserves before giving up.

Reviewed-by Firas Jahjah <[email protected]>
Reviewed-by Yonatan Nachum <[email protected]>
Signed-off-by Michael Margolin <[email protected]>
---
 drivers/infiniband/hw/efa/efa_verbs.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/hw/efa/efa_verbs.c +1 -1
index 2d28d68efe77..0b0396e190f2 100644
@@ -190,7 +190,7 @@ static void *efa_zalloc_mapped(struct efa_dev *dev, dma_addr_t *dma_addr,
 {
 	void *addr;
 
-	addr = alloc_pages_exact(size, GFP_KERNEL | __GFP_ZERO);
+	addr = alloc_pages_exact(size, GFP_KERNEL | __GFP_ZERO | __GFP_RETRY_MAYFAIL);
 	if (!addr)
 		return NULL;
Signature · 1 line
2.47.3
maintainer
#3 lore raw
Quoted 8 lines · Leon Romanovsky
Quoted 5 lines · Michael Margolin

RQ and CQ ring buffers require physically contiguous memory. On systems under memory pressure, high-order allocations can fail even when sufficient free memory exists because the allocator gives up after a single reclaim and compaction attempt for costly-order requests that lack __GFP_RETRY_MAYFAIL.

What sizes did you see fail, and in which flow?

.. and many other drivers has spent alot of effort not having high order allocations like this because they just don't work.

Jason

author maintainer
#4 lore raw
Quoted 9 lines · Jason Gunthorpe
Quoted 8 lines · Leon Romanovsky
Quoted 5 lines · Michael Margolin

RQ and CQ ring buffers require physically contiguous memory. On systems under memory pressure, high-order allocations can fail even when sufficient free memory exists because the allocator gives up after a single reclaim and compaction attempt for costly-order requests that lack __GFP_RETRY_MAYFAIL.

What sizes did you see fail, and in which flow?

In the recent cases we have seen (in customer/production environment) it during order 7 (512KB) allocations for 32K entries RQ.

Quoted 4 lines

.. and many other drivers has spent alot of effort not having high order allocations like this because they just don't work.

Unfortunately we cannot support non contiguous queues in some of the devices but we are going to add this support where possible.

Michael

Quoted 1 line

Jason

maintainer Applied
#5 lore raw
In reply to Michael Margolin
Quoted 10 lines · Michael Margolin

RQ and CQ ring buffers require physically contiguous memory. On systems under memory pressure, high-order allocations can fail even when sufficient free memory exists because the allocator gives up after a single reclaim and compaction attempt for costly-order requests that lack __GFP_RETRY_MAYFAIL.

Add __GFP_RETRY_MAYFAIL to the GFP flags in efa_zalloc_mapped() to allow retries with escalating compaction priority and release of highatomic reserves before giving up.

Quoted 6 lines
Reviewed-by Firas Jahjah <[email protected]>
Reviewed-by Yonatan Nachum <[email protected]>
Signed-off-by Michael Margolin <[email protected]>
---
 drivers/infiniband/hw/efa/efa_verbs.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Applied to for-next, thanks

Jason