Skip to contentSkip to conversation

mm/page_alloc: use existing highatomic reserves on the buddy fastpath

PATCH Superseded JP Kobryn 2 participants 3 messages Jun 17 – Jun 19

Subsystems (3)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

JP Kobryn

author
#0 lore raw

PATCH v2 1/1 mm/page_alloc: use existing highatomic reserves on the buddy fastpath

ALLOC_HIGHATOMIC currently provides both access to MIGRATE_HIGHATOMIC free pages and permission to create new highatomic pageblock reserves. This makes it unsuitable for the fastpath.

However, the fastpath can reach rmqueue_buddy() while MIGRATE_HIGHATOMIC reserves have free pages available. In this situation, the allocation can fall back to other migratetypes without trying those reserves first.

Allow high-priority non-blocking allocations above order-0 and up to the costly order to use existing MIGRATE_HIGHATOMIC reserves on the buddy fastpath. Change the semantics of ALLOC_HIGHATOMIC so that it only allows access to the reserves without permission to grow them. Add a new flag ALLOC_HIGHATOMIC_RESERVE that specifically allows growing the reserves.

A UDP receive workload was run with free MIGRATE_HIGHATOMIC pageblocks available in the target zone. Before this patch, the workload did not consume these blocks. With this patch, eligible order-1 allocations reaching the buddy path consumed existing MIGRATE_HIGHATOMIC pageblocks, with no highatomic misses observed. The workload did not grow highatomic reserves and NAPI page-frag allocations remained healthy with no failures or order-0 fallbacks.

---
v2:
- decouple use semantics from ALLOC_HIGHATOMIC_RESERVE
- update changelog to reflect above change and reword test paragraph
- adjust comment in PCP path
- rebase onto Linus' tree ~v7.2-rc1

v1: https://lore.kernel.org/linux-mm/[email protected]/

 mm/internal.h   |  1 +
 mm/page_alloc.c | 30 +++++++++++++++++++++++++-----
 2 files changed, 26 insertions(+), 5 deletions(-)
mm/internal.h +1 -0
index 5a2ddcf68e0b..6700659615e8 100644
@@ -1478,6 +1478,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,
 #define ALLOC_HIGHATOMIC	0x200 /* Allows access to MIGRATE_HIGHATOMIC */
 #define ALLOC_TRYLOCK		0x400 /* Only use spin_trylock in allocation path */
 #define ALLOC_KSWAPD		0x800 /* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */
+#define ALLOC_HIGHATOMIC_RESERVE	0x1000 /* Allows growing MIGRATE_HIGHATOMIC reserves */
 
 /* Flags that allow allocations below the min watermark. */
 #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
mm/page_alloc.c +25 -5
index d49c254174da..ed919e2ac99a 100644
@@ -3238,7 +3238,7 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
 	 * If this is a high-order atomic allocation then check
 	 * if the pageblock should be reserved for the future
 	 */
-	if (unlikely(alloc_flags & ALLOC_HIGHATOMIC))
+	if (unlikely(alloc_flags & ALLOC_HIGHATOMIC_RESERVE))
 		reserve_highatomic_pageblock(page, order, zone);
 
 	__count_zid_vm_events(PGALLOC, page_zonenum(page), 1 << order);
@@ -3320,8 +3320,9 @@ struct page *__rmqueue_pcplist(struct zone *zone, unsigned int order,
 			 *
 			 * Instead, direct it towards the reserves by
 			 * returning NULL, which will make the caller fall
-			 * back to rmqueue_buddy. This will try to use the
-			 * reserves first and grow them if needed.
+			 * back to rmqueue_buddy. There it will try to use
+			 * the reserves first and grow them if needed and
+			 * permitted by the ALLOC_HIGHATOMIC_RESERVE flag.
 			 */
 			if (alloc_flags & ALLOC_HIGHATOMIC)
 				return NULL;
@@ -3768,6 +3769,24 @@ alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask)
 	return alloc_flags;
 }
 
+/*
+ * Let high-priority non-blocking allocations above order-0 and up
+ * to the costly order try to use existing MIGRATE_HIGHATOMIC
+ * reserves on the fastpath.
+ */
+static inline unsigned int
+alloc_flags_highatomic_fastpath(gfp_t gfp_mask, unsigned int order)
+{
+	if (!order || order > PAGE_ALLOC_COSTLY_ORDER)
+		return 0;
+	if (!(gfp_mask & __GFP_HIGH))
+		return 0;
+	if (gfp_mask & (__GFP_DIRECT_RECLAIM | __GFP_NOMEMALLOC))
+		return 0;
+
+	return ALLOC_HIGHATOMIC;
+}
+
 /* Must be called after current_gfp_context() which can change gfp_mask */
 static inline unsigned int gfp_to_alloc_flags_cma(gfp_t gfp_mask,
 						  unsigned int alloc_flags)
@@ -4495,7 +4514,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order)
 			alloc_flags |= ALLOC_NON_BLOCK;
 
 			if (order > 0 && (alloc_flags & ALLOC_MIN_RESERVE))
-				alloc_flags |= ALLOC_HIGHATOMIC;
+				alloc_flags |= (ALLOC_HIGHATOMIC | ALLOC_HIGHATOMIC_RESERVE);
 		}
 
 		/*
@@ -5215,7 +5234,8 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,
 	 * Forbid the first pass from falling back to types that fragment
 	 * memory until all local zones are considered.
 	 */
-	alloc_flags |= alloc_flags_nofragment(zonelist_zone(ac.preferred_zoneref), gfp);
+	alloc_flags |= alloc_flags_nofragment(zonelist_zone(ac.preferred_zoneref), gfp) |
+			alloc_flags_highatomic_fastpath(alloc_gfp, order);
 
 	/* First allocation attempt */
 	page = get_page_from_freelist(alloc_gfp, order, alloc_flags, &ac);
Signature · 1 line
2.54.0
#1 lore raw
Quoted 14 lines · JP Kobryn

ALLOC_HIGHATOMIC currently provides both access to MIGRATE_HIGHATOMIC free pages and permission to create new highatomic pageblock reserves. This makes it unsuitable for the fastpath.

However, the fastpath can reach rmqueue_buddy() while MIGRATE_HIGHATOMIC reserves have free pages available. In this situation, the allocation can fall back to other migratetypes without trying those reserves first.

Allow high-priority non-blocking allocations above order-0 and up to the costly order to use existing MIGRATE_HIGHATOMIC reserves on the buddy fastpath. Change the semantics of ALLOC_HIGHATOMIC so that it only allows access to the reserves without permission to grow them. Add a new flag ALLOC_HIGHATOMIC_RESERVE that specifically allows growing the reserves.

Quoted 7 lines

A UDP receive workload was run with free MIGRATE_HIGHATOMIC pageblocks available in the target zone. Before this patch, the workload did not consume these blocks. With this patch, eligible order-1 allocations reaching the buddy path consumed existing MIGRATE_HIGHATOMIC pageblocks, with no highatomic misses observed. The workload did not grow highatomic reserves and NAPI page-frag allocations remained healthy with no failures or order-0 fallbacks.

Thanks for digging deeper into this! That's a great find.

Quoted 32 lines

---
v2:
- decouple use semantics from ALLOC_HIGHATOMIC_RESERVE
- update changelog to reflect above change and reword test paragraph
- adjust comment in PCP path
- rebase onto Linus' tree ~v7.2-rc1

v1: https://lore.kernel.org/linux-mm/[email protected]/

 mm/internal.h   |  1 +
 mm/page_alloc.c | 30 +++++++++++++++++++++++++-----
 2 files changed, 26 insertions(+), 5 deletions(-)
mm/internal.h +1 -0
index 5a2ddcf68e0b..6700659615e8 100644
@@ -1478,6 +1478,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,
 #define ALLOC_HIGHATOMIC	0x200 /* Allows access to MIGRATE_HIGHATOMIC */
 #define ALLOC_TRYLOCK		0x400 /* Only use spin_trylock in allocation path */
 #define ALLOC_KSWAPD		0x800 /* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */
+#define ALLOC_HIGHATOMIC_RESERVE	0x1000 /* Allows growing MIGRATE_HIGHATOMIC reserves */
 
 /* Flags that allow allocations below the min watermark. */
 #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
mm/page_alloc.c +0 -0
index d49c254174da..ed919e2ac99a 100644

mm/page_alloc.c struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, · line 3242

Quoted 6 lines
mm/page_alloc.c +1 -1
@@ -3238,7 +3238,7 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
 	 * If this is a high-order atomic allocation then check
 	 * if the pageblock should be reserved for the future
 	 */
-	if (unlikely(alloc_flags & ALLOC_HIGHATOMIC))
+	if (unlikely(alloc_flags & ALLOC_HIGHATOMIC_RESERVE))
 		reserve_highatomic_pageblock(page, order, zone);

You could check ALLOC_WMARK_MIN to determine the slowpath. This way you wouldn't need another alloc flag:

	/* Slowpath (precarious) high-atomic allocation. Maybe reserve block */
	if (unlikely((alloc_flags & (ALLOC_HIGHATOMIC|ALLOC_WMARK_MIN)) == (ALLOC_HIGHATOMIC|ALLOC_WMARK_MIN)))
		reserve_highatomic_pageblock(page, order, zone);

[ we really ought to generalize gfp_has_flags() ... ]

Quoted 17 lines
mm/page_alloc.c +3 -2
 	__count_zid_vm_events(PGALLOC, page_zonenum(page), 1 << order);
@@ -3320,8 +3320,9 @@ struct page *__rmqueue_pcplist(struct zone *zone, unsigned int order,
 			 *
 			 * Instead, direct it towards the reserves by
 			 * returning NULL, which will make the caller fall
-			 * back to rmqueue_buddy. This will try to use the
-			 * reserves first and grow them if needed.
+			 * back to rmqueue_buddy. There it will try to use
+			 * the reserves first and grow them if needed and
+			 * permitted by the ALLOC_HIGHATOMIC_RESERVE flag.
 			 */
 			if (alloc_flags & ALLOC_HIGHATOMIC)
 				return NULL;
@@ -3768,6 +3769,24 @@ alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask)
 	return alloc_flags;
 }
 

mm/page_alloc.c alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask) · line 3781

Quoted 10 lines
mm/page_alloc.c +10 -0
+/*
+ * Let high-priority non-blocking allocations above order-0 and up
+ * to the costly order try to use existing MIGRATE_HIGHATOMIC
+ * reserves on the fastpath.
+ */
+static inline unsigned int
+alloc_flags_highatomic_fastpath(gfp_t gfp_mask, unsigned int order)
+{
+	if (!order || order > PAGE_ALLOC_COSTLY_ORDER)
+		return 0;

There seems to be a mismatch between this and gfp_to_alloc_flags() (slowpath), where slowpath is still allowed to tap highatomic reserves for costly orders. Is that on purpose?

mm/page_alloc.c alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask) · line 3785

Quoted 4 lines
mm/page_alloc.c +4 -0
+	if (!(gfp_mask & __GFP_HIGH))
+		return 0;
+	if (gfp_mask & (__GFP_DIRECT_RECLAIM | __GFP_NOMEMALLOC))
+		return 0;

This duplicates gfp_to_alloc_flags() logic which seems fragile. How about the below:

Quoted 17 lines
mm/page_alloc.c +5 -1
+
+	return ALLOC_HIGHATOMIC;
+}
+
 /* Must be called after current_gfp_context() which can change gfp_mask */
 static inline unsigned int gfp_to_alloc_flags_cma(gfp_t gfp_mask,
 						  unsigned int alloc_flags)
@@ -4495,7 +4514,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order)
 			alloc_flags |= ALLOC_NON_BLOCK;
 
 			if (order > 0 && (alloc_flags & ALLOC_MIN_RESERVE))
-				alloc_flags |= ALLOC_HIGHATOMIC;
+				alloc_flags |= (ALLOC_HIGHATOMIC | ALLOC_HIGHATOMIC_RESERVE);
 		}
 
 		/*

mm/page_alloc.c struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, · line 5238

Quoted 6 lines
mm/page_alloc.c +2 -1
@@ -5215,7 +5234,8 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,
 	 * Forbid the first pass from falling back to types that fragment
 	 * memory until all local zones are considered.
 	 */
-	alloc_flags |= alloc_flags_nofragment(zonelist_zone(ac.preferred_zoneref), gfp);
+	alloc_flags |= alloc_flags_nofragment(zonelist_zone(ac.preferred_zoneref), gfp) |
+			alloc_flags_highatomic_fastpath(alloc_gfp, order);
	alloc_flags |= gfp_to_alloc_flags(gfp, order) & ALLOC_HIGHATOMIC;

Or factor gfp_to_alloc_flags_nonblocking() from gfp_to_alloc_flags() and reuse that here, to save a few cycles in the fast path.

Quoted 4 lines
mm/page_alloc.c +0 -0
 	/* First allocation attempt */
 	page = get_page_from_freelist(alloc_gfp, order, alloc_flags, &ac);
Signature · 1 line
2.54.0
author
#2 lore raw
Quoted 23 lines · Johannes Weiner
Quoted 14 lines · JP Kobryn

ALLOC_HIGHATOMIC currently provides both access to MIGRATE_HIGHATOMIC free pages and permission to create new highatomic pageblock reserves. This makes it unsuitable for the fastpath.

However, the fastpath can reach rmqueue_buddy() while MIGRATE_HIGHATOMIC reserves have free pages available. In this situation, the allocation can fall back to other migratetypes without trying those reserves first.

Allow high-priority non-blocking allocations above order-0 and up to the costly order to use existing MIGRATE_HIGHATOMIC reserves on the buddy fastpath. Change the semantics of ALLOC_HIGHATOMIC so that it only allows access to the reserves without permission to grow them. Add a new flag ALLOC_HIGHATOMIC_RESERVE that specifically allows growing the reserves.

Quoted 7 lines

A UDP receive workload was run with free MIGRATE_HIGHATOMIC pageblocks available in the target zone. Before this patch, the workload did not consume these blocks. With this patch, eligible order-1 allocations reaching the buddy path consumed existing MIGRATE_HIGHATOMIC pageblocks, with no highatomic misses observed. The workload did not grow highatomic reserves and NAPI page-frag allocations remained healthy with no failures or order-0 fallbacks.

Quoted 1 line

Thanks for digging deeper into this! That's a great find.

Thanks :)

Quoted 43 lines
Quoted 33 lines

---
v2:
- decouple use semantics from ALLOC_HIGHATOMIC_RESERVE
- update changelog to reflect above change and reword test paragraph
- adjust comment in PCP path
- rebase onto Linus' tree ~v7.2-rc1

v1: https://lore.kernel.org/linux-mm/[email protected]/

 mm/internal.h   |  1 +
 mm/page_alloc.c | 30 +++++++++++++++++++++++++-----
 2 files changed, 26 insertions(+), 5 deletions(-)
mm/internal.h +1 -0
index 5a2ddcf68e0b..6700659615e8 100644
@@ -1478,6 +1478,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,
 #define ALLOC_HIGHATOMIC	0x200 /* Allows access to MIGRATE_HIGHATOMIC */
 #define ALLOC_TRYLOCK		0x400 /* Only use spin_trylock in allocation path */
 #define ALLOC_KSWAPD		0x800 /* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */
+#define ALLOC_HIGHATOMIC_RESERVE	0x1000 /* Allows growing MIGRATE_HIGHATOMIC reserves */
 
 /* Flags that allow allocations below the min watermark. */
 #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
mm/page_alloc.c +0 -0
index d49c254174da..ed919e2ac99a 100644

mm/page_alloc.c struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, · line 3242

Quoted 6 lines
mm/page_alloc.c +1 -1
@@ -3238,7 +3238,7 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
 	 * If this is a high-order atomic allocation then check
 	 * if the pageblock should be reserved for the future
 	 */
-	if (unlikely(alloc_flags & ALLOC_HIGHATOMIC))
+	if (unlikely(alloc_flags & ALLOC_HIGHATOMIC_RESERVE))
 		reserve_highatomic_pageblock(page, order, zone);

You could check ALLOC_WMARK_MIN to determine the slowpath. This way you wouldn't need another alloc flag:

Quoted 3 lines
	/* Slowpath (precarious) high-atomic allocation. Maybe reserve block */
	if (unlikely((alloc_flags & (ALLOC_HIGHATOMIC|ALLOC_WMARK_MIN)) == (ALLOC_HIGHATOMIC|ALLOC_WMARK_MIN)))
		reserve_highatomic_pageblock(page, order, zone);

Hmm so we just use watermark low vs min to determine fast/slow path. We would need to mask out the watermark bits since ALLOC_WMARK_MIN is zero, but I think the idea can work.

Quoted 20 lines

[ we really ought to generalize gfp_has_flags() ... ]

Quoted 17 lines
mm/page_alloc.c +3 -2
 	__count_zid_vm_events(PGALLOC, page_zonenum(page), 1 << order);
@@ -3320,8 +3320,9 @@ struct page *__rmqueue_pcplist(struct zone *zone, unsigned int order,
 			 *
 			 * Instead, direct it towards the reserves by
 			 * returning NULL, which will make the caller fall
-			 * back to rmqueue_buddy. This will try to use the
-			 * reserves first and grow them if needed.
+			 * back to rmqueue_buddy. There it will try to use
+			 * the reserves first and grow them if needed and
+			 * permitted by the ALLOC_HIGHATOMIC_RESERVE flag.
 			 */
 			if (alloc_flags & ALLOC_HIGHATOMIC)
 				return NULL;
@@ -3768,6 +3769,24 @@ alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask)
 	return alloc_flags;
 }
 
Quoted 14 lines

mm/page_alloc.c alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask) · line 3781

Quoted 10 lines
mm/page_alloc.c +10 -0
+/*
+ * Let high-priority non-blocking allocations above order-0 and up
+ * to the costly order try to use existing MIGRATE_HIGHATOMIC
+ * reserves on the fastpath.
+ */
+static inline unsigned int
+alloc_flags_highatomic_fastpath(gfp_t gfp_mask, unsigned int order)
+{
+	if (!order || order > PAGE_ALLOC_COSTLY_ORDER)
+		return 0;

There seems to be a mismatch between this and gfp_to_alloc_flags() (slowpath), where slowpath is still allowed to tap highatomic reserves for costly orders. Is that on purpose?

Yes, the intention was to improve allocator outcomes for frequent net allocations. So I kept the policy narrow. But I thought about this some more and agree it's worth removing the order cap. With the cap, if a costly+ order atomic allocation reaches the buddy path there may be two non-ideal outcomes: fallback to some other migratetype in the fastpath or enter the slowpath and make use of any available reserves. In the latter case, the slowpath could have been avoided altogether. So I'll make this change in v3.

Quoted 35 lines

mm/page_alloc.c alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask) · line 3785

Quoted 5 lines
mm/page_alloc.c +4 -0
+	if (!(gfp_mask & __GFP_HIGH))
+		return 0;
+	if (gfp_mask & (__GFP_DIRECT_RECLAIM | __GFP_NOMEMALLOC))
+		return 0;

This duplicates gfp_to_alloc_flags() logic which seems fragile. How about the below:

Quoted 17 lines
mm/page_alloc.c +5 -1
+
+	return ALLOC_HIGHATOMIC;
+}
+
 /* Must be called after current_gfp_context() which can change gfp_mask */
 static inline unsigned int gfp_to_alloc_flags_cma(gfp_t gfp_mask,
 						  unsigned int alloc_flags)
@@ -4495,7 +4514,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order)
 			alloc_flags |= ALLOC_NON_BLOCK;
 
 			if (order > 0 && (alloc_flags & ALLOC_MIN_RESERVE))
-				alloc_flags |= ALLOC_HIGHATOMIC;
+				alloc_flags |= (ALLOC_HIGHATOMIC | ALLOC_HIGHATOMIC_RESERVE);
 		}
 
 		/*

mm/page_alloc.c struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, · line 5238

Quoted 6 lines
mm/page_alloc.c +2 -1
@@ -5215,7 +5234,8 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,
 	 * Forbid the first pass from falling back to types that fragment
 	 * memory until all local zones are considered.
 	 */
-	alloc_flags |= alloc_flags_nofragment(zonelist_zone(ac.preferred_zoneref), gfp);
+	alloc_flags |= alloc_flags_nofragment(zonelist_zone(ac.preferred_zoneref), gfp) |
+			alloc_flags_highatomic_fastpath(alloc_gfp, order);
	alloc_flags |= gfp_to_alloc_flags(gfp, order) & ALLOC_HIGHATOMIC;
Quoted 2 lines

Or factor gfp_to_alloc_flags_nonblocking() from gfp_to_alloc_flags() and reuse that here, to save a few cycles in the fast path.

This should be possible now that I'll be removing the order cap.

Quoted 5 lines
Quoted 5 lines
mm/page_alloc.c +0 -0
 	/* First allocation attempt */
 	page = get_page_from_freelist(alloc_gfp, order, alloc_flags, &ac);
Signature · 1 line
2.54.0