mm: fix reclaim storms in defrag_mode

While trying to fix a reclaim storm in defrag_mode, I noticed that non-movable direct compaction is extremely inefficient.

When searching for space to evacuate, compaction only allows blocks of the same type as the incoming request. This is to prevent migratetype pollution, where a small non-movable request frees space in a movable block and provokes the allocator to fall back and pollute it.

This protection is reasonable on one hand, but the downside is that it makes non-movable direct compaction nearly useless: if we get the type annotations right, by definition there aren't any movable pages inside the non-movable blocks it is allowed to scan.

With defrag_mode, the goal is the production of whole blocks, which are essentially type neutral: __rmqueue_claim() will convert them wholesale on alloc. This makes type mixing and pollution a non-issue.

Fix the pollution gates to take the requested order into account, and allow whole-block requests to scan blocks of other types.

The only exception is CMA blocks. That type is sticky and these blocks cannot be claimed to other types. Continue to be strict with them, and allow only explicit ALLOC_CMA requests and kcompactd to evacuate them.

Reviewed-by Vlastimil Babka (SUSE) <[email protected]>

Signed-off-by Johannes Weiner <[email protected]>

From: "Vlastimil Babka (SUSE)" <[email protected]>

The compaction capturing code assumes the allocation request order and compaction target order are the same. That won't be true once defrag_mode promotes sub-block allocations to pageblock-order
compaction: compaction targets the larger order, while capture should remain at the original allocation order.

Move the capture_control to the page allocator and give it its own copies of what the page freeing path matches against - zone, migratetype and the allocation order - rather than reaching into compaction's live compact_control. __alloc_pages_direct_compact() fills in migratetype and order, and installs and hides current->capture_control around the whole compaction call; try_to_compact_pages() aims capc->zone at each zone while it is being compacted. compact_zone_order() no longer deals with capture at all.

Pass the capture_control through try_to_compact_pages() / compact_zone_order() in place of the bare struct page **.

No functional change.

Signed-off-by Vlastimil Babka (SUSE) <[email protected]>

Co-developed-by Johannes Weiner <[email protected]>

Signed-off-by Johannes Weiner <[email protected]>

---
 include/linux/compaction.h |  3 ++-
 mm/compaction.c            | 50 +++++++++++---------------------------
 mm/internal.h              |  4 ++-
 mm/page_alloc.c            | 45 ++++++++++++++++++++++++++++------
 4 files changed, 57 insertions(+), 45 deletions(-)

As we deployed defrag_mode into Meta production, pressure spikes and excessive swapping were observed on some workloads. Tracing confirmed that this is unmovable/reclaimable requests spinning in the allocator and direct reclaim, causing excessive amounts of swap.

The initial plan for defrag_mode was to rely on kswapd/kcompactd to produce blocks, and if those are overwhelmed under high pressure, let the allocator fall back (__rmqueue_steal()) after its retry loops. However, that retrying results in more reclaim on some of these workloads than we'd hoped, sometimes excessively so, spurred on by the !costly order conditions in should_reclaim_retry().

The storms are dependent on the request type. Reclaim will inevitably make room in existing movable blocks, since that's where the LRU pages live. So if movable requests retry on reclaim, they make progress.

When non-movable requests spin in reclaim that isn't productive. They cannot use the individually freed pages, and the process is unlikely to accidentally free whole blocks to meet the ALLOC_NOFRAGMENT bar. They spin and overreclaim excessively, which tanks performance and triggers userspace guards like swap exhaustion or pressure based OOM.

To fix this, send non-movable requests, regardless of order, into pageblock reclaim/compaction. This way, they help move things along to meet the ALLOC_NOFRAGMENT bar. After this patch, the reclaim storms and excess OOM rates are no longer observed in production.

The longer-term plan is still to have all requests, including the movable ones, help make blocks to spread the cost of defragmenting more evenly and fairly; combined with proper watermarking to reduce allocation latencies in the common case. However, doing this naively unearths scaling and concurrency limitations in compaction that need to be addressed first. Promoting just non-movables for now is the minimally viable bug fix for the above issue.

Fixes e3aa7df331bc ("mm: page_alloc: defrag_mode")

Cc <[email protected]>

Signed-off-by Johannes Weiner <[email protected]>

As we deployed defrag_mode into Meta production, pressure spikes and excessive swapping were observed on some workloads. Tracing confirmed that this is unmovable/reclaimable requests spinning in the allocator and direct reclaim, causing excessive amounts of swap.

The initial plan for defrag_mode was to rely on kswapd/kcompactd to produce blocks, and if those are overwhelmed under high pressure, let the allocator fall back (__rmqueue_steal()) after its retry loops. However, that retrying results in more reclaim on some of these workloads than we'd hoped, sometimes excessively so, spurred on by the !costly order conditions in should_reclaim_retry().

The storms are dependent on the request type. Reclaim will inevitably make room in existing movable blocks, since that's where the LRU pages live. So if movable requests retry on reclaim, they make progress.

When non-movable requests spin in reclaim that isn't productive. They cannot use the individually freed pages, and the process is unlikely to accidentally free whole blocks to meet the ALLOC_NOFRAGMENT bar. They spin and overreclaim excessively, which tanks performance and triggers userspace guards like swap exhaustion or pressure based OOM.

To fix this, send non-movable requests, regardless of order, into pageblock reclaim/compaction. This way, they help move things along to meet the ALLOC_NOFRAGMENT bar. After this patch, the reclaim storms and excess OOM rates are no longer observed in production.

The longer-term plan is still to have all requests, including the movable ones, help make blocks to spread the cost of defragmenting more evenly and fairly; combined with proper watermarking to reduce allocation latencies in the common case. However, doing this naively unearths scaling and concurrency limitations in compaction that need to be addressed first. Promoting just non-movables for now is the minimally viable bug fix for the above issue.

Fixes e3aa7df331bc ("mm: page_alloc: defrag_mode")

Cc <[email protected]>

Signed-off-by Johannes Weiner <[email protected]>

Quoted 4 lines · On Wed, 22 Jul 2026 10:56:47 -0400 Johannes Weiner <[email protected]> wrote:

Fixes e3aa7df331bc ("mm: page_alloc: defrag_mode")

Cc <[email protected]>

Signed-off-by Johannes Weiner <[email protected]>

This cc:stable fix doesn't apply to current mainline because the preceding non-cc:stable patches get in the way.

This means that the -stable maintainers will have to ask you to redo the fix to make it backportable. And you'll then be preparing a patch which hasn't had standalone testing in mainline. All very cumbersome.

So is it practical to make the fix come first, as a cc:stable hotfix so we can upstream the other patches in the next merge window?

(An alternative would be to put cc:stable on all four patches. That's a bit abusive but we do it rarely).

Either way, I'll get these into mm.git for testing while we await review on [4/4], thanks.

+ Greg and Sasha

Quoted 7 lines · Andrew Morton

Quoted 4 lines · On Wed, 22 Jul 2026 10:56:47 -0400 Johannes Weiner <[email protected]> wrote:

Fixes e3aa7df331bc ("mm: page_alloc: defrag_mode")

Cc <[email protected]>

Signed-off-by Johannes Weiner <[email protected]>

Quoted 6 lines

This cc:stable fix doesn't apply to current mainline because the preceding non-cc:stable patches get in the way.

This means that the -stable maintainers will have to ask you to redo the fix to make it backportable. And you'll then be preparing a patch which hasn't had standalone testing in mainline. All very cumbersome.

Or they backport the whole series, even if only 4/4 has a Fixes: tag.

Quoted 2 lines

So is it practical to make the fix come first, as a cc:stable hotfix so we can upstream the other patches in the next merge window?

I think it can't come first, it's simply a complex fix that can't be done in a single patch that would be reviewable. 3/4 is certainly a strict dependency. I'm not sure if 2/4 as well. Perhaps 2/4 does deserve a Fixes:
tag and Cc: stable on its own, as it also fixes a defrag_mode deficiency.

Quoted 2 lines

(An alternative would be to put cc:stable on all four patches. That's a bit abusive but we do it rarely).

I don't think it's necessary, the stable team are quite capable of figuring out the deps and backporting them with a Stable-dep-off: tag. IIRC the "fake" tagging is discouraged? Maybe they can clarify.

Quoted 19 lines · Vlastimil Babka (SUSE)

+ Greg and Sasha

Quoted 7 lines · Andrew Morton

Quoted 4 lines · On Wed, 22 Jul 2026 10:56:47 -0400 Johannes Weiner <[email protected]> wrote:

Fixes e3aa7df331bc ("mm: page_alloc: defrag_mode")

Cc <[email protected]>

Signed-off-by Johannes Weiner <[email protected]>

Quoted 6 lines

This cc:stable fix doesn't apply to current mainline because the preceding non-cc:stable patches get in the way.

This means that the -stable maintainers will have to ask you to redo the fix to make it backportable. And you'll then be preparing a patch which hasn't had standalone testing in mainline. All very cumbersome.

Or they backport the whole series, even if only 4/4 has a Fixes: tag.

Quoted 7 lines

Quoted 2 lines

So is it practical to make the fix come first, as a cc:stable hotfix so we can upstream the other patches in the next merge window?

I think it can't come first, it's simply a complex fix that can't be done in a single patch that would be reviewable. 3/4 is certainly a strict dependency. I'm not sure if 2/4 as well. Perhaps 2/4 does deserve a Fixes:
tag and Cc: stable on its own, as it also fixes a defrag_mode deficiency.

Yes. Only 1/4 could be argued away.

2/4 actually came after 4/4 during development. Without it, compaction cannot make sufficient forward progress on the promoted non-movable requests. So it is a hard functional requirement.

Whether it deserves an independent Fixes: depends on how many GFP_KERNEL pageblock_order requests we have :-) I'm guessing not many.

Quoted 6 lines

Quoted 2 lines

(An alternative would be to put cc:stable on all four patches. That's a bit abusive but we do it rarely).

I don't think it's necessary, the stable team are quite capable of figuring out the deps and backporting them with a Stable-dep-off: tag. IIRC the "fake" tagging is discouraged? Maybe they can clarify.

Yes, I *think* it should work with tagging just 4/4.

It won't apply cleanly or build without 3/4. So I'd assume if that dependency is lost, the automated backport will bounce and I can send a backport series by hand.

Quoted 3 lines

2/4 actually came after 4/4 during development. Without it, compaction cannot make sufficient forward progress on the promoted non-movable requests. So it is a hard functional requirement.

So that's the tricky part then. If it's a hard functional requirement for 4/4 and yet leaving it out results in no conflicts when applying 3/4 and 4/4 (as it indeed seems so, based on my local test), then it's quite possible stable folks will figure out themselves they need 3/4 as a stable-dep-off:
4/4 but won't backport 2/4, unless you act quickly enough on their mails.

Quoted 2 lines

Whether it deserves an independent Fixes: depends on how many GFP_KERNEL pageblock_order requests we have :-) I'm guessing not many.

I don't know how else to indicate it though. It's kind of a co-fix with 4/4?

Quoted 9 lines

Quoted 6 lines

Quoted 2 lines

(An alternative would be to put cc:stable on all four patches. That's a bit abusive but we do it rarely).

I don't think it's necessary, the stable team are quite capable of figuring out the deps and backporting them with a Stable-dep-off: tag. IIRC the "fake" tagging is discouraged? Maybe they can clarify.

Yes, I *think* it should work with tagging just 4/4.

Quoted 3 lines

It won't apply cleanly or build without 3/4. So I'd assume if that dependency is lost, the automated backport will bounce and I can send a backport series by hand.

The way I see if often happen is that Greg sends a mail that it bounced and soon after Sasha replies with fixed up version/series. Which could very easily be 3/4 + 4/4 without 2/4.

Quoted 24 lines · On Thu, 23 Jul 2026 16:17:51 +0200 "Vlastimil Babka (SUSE)"…

Quoted 3 lines

2/4 actually came after 4/4 during development. Without it, compaction cannot make sufficient forward progress on the promoted non-movable requests. So it is a hard functional requirement.

So that's the tricky part then. If it's a hard functional requirement for 4/4 and yet leaving it out results in no conflicts when applying 3/4 and 4/4 (as it indeed seems so, based on my local test), then it's quite possible stable folks will figure out themselves they need 3/4 as a stable-dep-off:
4/4 but won't backport 2/4, unless you act quickly enough on their mails.

Quoted 2 lines

Whether it deserves an independent Fixes: depends on how many GFP_KERNEL pageblock_order requests we have :-) I'm guessing not many.

I don't know how else to indicate it though. It's kind of a co-fix with 4/4?

Quoted 9 lines

Quoted 6 lines

Quoted 2 lines

(An alternative would be to put cc:stable on all four patches. That's a bit abusive but we do it rarely).

I don't think it's necessary, the stable team are quite capable of figuring out the deps and backporting them with a Stable-dep-off: tag. IIRC the "fake" tagging is discouraged? Maybe they can clarify.

Yes, I *think* it should work with tagging just 4/4.

Quoted 7 lines

Quoted 3 lines

It won't apply cleanly or build without 3/4. So I'd assume if that dependency is lost, the automated backport will bounce and I can send a backport series by hand.

The way I see if often happen is that Greg sends a mail that it bounced and soon after Sasha replies with fixed up version/series. Which could very easily be 3/4 + 4/4 without 2/4.

My brain hurts. I'll include

Fixes e3aa7df331bc ("mm: page_alloc: defrag_mode")

Cc <[email protected]>

in all four patches.

+ Greg and Sasha

Quoted 7 lines · Andrew Morton

Quoted 4 lines · On Wed, 22 Jul 2026 10:56:47 -0400 Johannes Weiner <[email protected]> wrote:

Fixes e3aa7df331bc ("mm: page_alloc: defrag_mode")

Cc <[email protected]>

Signed-off-by Johannes Weiner <[email protected]>

Quoted 6 lines

This cc:stable fix doesn't apply to current mainline because the preceding non-cc:stable patches get in the way.

This means that the -stable maintainers will have to ask you to redo the fix to make it backportable. And you'll then be preparing a patch which hasn't had standalone testing in mainline. All very cumbersome.

Or they backport the whole series, even if only 4/4 has a Fixes: tag.

I don't think it's necessary, the stable team are quite capable of figuring out the deps and backporting them with a Stable-dep-off: tag.

IIRC the
"fake" tagging is discouraged? Maybe they can clarify.

As we deployed defrag_mode into Meta production, pressure spikes and excessive swapping were observed on some workloads. Tracing confirmed that this is unmovable/reclaimable requests spinning in the allocator and direct reclaim, causing excessive amounts of swap.

The initial plan for defrag_mode was to rely on kswapd/kcompactd to produce blocks, and if those are overwhelmed under high pressure, let the allocator fall back (__rmqueue_steal()) after its retry loops. However, that retrying results in more reclaim on some of these workloads than we'd hoped, sometimes excessively so, spurred on by the !costly order conditions in should_reclaim_retry().

The storms are dependent on the request type. Reclaim will inevitably make room in existing movable blocks, since that's where the LRU pages live. So if movable requests retry on reclaim, they make progress.

When non-movable requests spin in reclaim that isn't productive. They cannot use the individually freed pages, and the process is unlikely to accidentally free whole blocks to meet the ALLOC_NOFRAGMENT bar. They spin and overreclaim excessively, which tanks performance and triggers userspace guards like swap exhaustion or pressure based OOM.

To fix this, send non-movable requests, regardless of order, into pageblock reclaim/compaction. This way, they help move things along to meet the ALLOC_NOFRAGMENT bar. After this patch, the reclaim storms and excess OOM rates are no longer observed in production.

The longer-term plan is still to have all requests, including the movable ones, help make blocks to spread the cost of defragmenting more evenly and fairly; combined with proper watermarking to reduce allocation latencies in the common case. However, doing this naively unearths scaling and concurrency limitations in compaction that need to be addressed first. Promoting just non-movables for now is the minimally viable bug fix for the above issue.

Fixes e3aa7df331bc ("mm: page_alloc: defrag_mode")

Cc <[email protected]>

Signed-off-by Johannes Weiner <[email protected]>

---
 mm/internal.h   |  6 ++++++
 mm/page_alloc.c | 31 ++++++++++++++++++++++++++-----
 2 files changed, 32 insertions(+), 5 deletions(-)

mm/internal.h +0 -0

index 6d3402001b93..5acba3470659 100644

mm/internal.h +0 -0

 	int order;
 	struct page *page;
 };

mm/page_alloc.c +26 -5

index 4b9686fedb71..f92055827ae9 100644
@@ -4152,8 +4152,24 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,
 		.order = order,
 		.page = NULL,
 	};
+	int compact_order = order;
 
-	if (!order)
+	/*
+	 * If fallbacks are not permitted (defrag_mode), we either
+	 * need to reclaim space in a block of matching type, or clear
+	 * out an entire block to allow __rmqueue_claim() to convert.
+	 *
+	 * Reclaim by itself is primarily freeing space in movable
+	 * blocks, since that's where the LRU pages live. So this
+	 * works for movable requests, but not for others.
+	 *
+	 * For those, promote the order to help make blocks, instead
+	 * of spinning in reclaim alone unproductively.
+	 */
+	if ((alloc_flags & ALLOC_NOFRAGMENT) && ac->migratetype != MIGRATE_MOVABLE)
+		compact_order = max(order, pageblock_order);
+
+	if (!compact_order)
 		return NULL;
 
 	psi_memstall_enter(&pflags);

@@ -4169,8 +4185,8 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,
 	barrier();
 	WRITE_ONCE(current->capture_control, &capc);
 
-	*compact_result = try_to_compact_pages(gfp_mask, order, alloc_flags, ac,
-							       prio, &capc);
+	*compact_result = try_to_compact_pages(gfp_mask, compact_order,
+					       alloc_flags, ac, prio, &capc);
 
 	/*
 	 * Make sure we hide capture control first before we read the captured

@@ -4215,7 +4231,7 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,
 		struct zone *zone = page_zone(page);
 
 		zone->compact_blockskip_flush = false;
-		compaction_defer_reset(zone, order, true);
+		compaction_defer_reset(zone, compact_order, true);
 		count_vm_event(COMPACTSUCCESS);
 		return page;
 	}

@@ -4455,9 +4471,14 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order,
 	struct page *page = NULL;
 	unsigned long pflags;
 	bool drained = false;
+	int reclaim_order = order;
+
+	/* Match the slowpath compaction promotion in __alloc_pages_direct_compact */
+	if ((alloc_flags & ALLOC_NOFRAGMENT) && ac->migratetype != MIGRATE_MOVABLE)
+		reclaim_order = max(order, pageblock_order);
 
 	psi_memstall_enter(&pflags);
-	*did_some_progress = __perform_reclaim(gfp_mask, order, ac);
+	*did_some_progress = __perform_reclaim(gfp_mask, reclaim_order, ac);
 	if (unlikely(!(*did_some_progress)))
 		goto out;
 

As we deployed defrag_mode into Meta production, pressure spikes and excessive swapping were observed on some workloads. Tracing confirmed that this is unmovable/reclaimable requests spinning in the allocator and direct reclaim, causing excessive amounts of swap.

The initial plan for defrag_mode was to rely on kswapd/kcompactd to produce blocks, and if those are overwhelmed under high pressure, let the allocator fall back (__rmqueue_steal()) after its retry loops. However, that retrying results in more reclaim on some of these workloads than we'd hoped, sometimes excessively so, spurred on by the !costly order conditions in should_reclaim_retry().

The storms are dependent on the request type. Reclaim will inevitably make room in existing movable blocks, since that's where the LRU pages live. So if movable requests retry on reclaim, they make progress.

When non-movable requests spin in reclaim that isn't productive. They cannot use the individually freed pages, and the process is unlikely to accidentally free whole blocks to meet the ALLOC_NOFRAGMENT bar. They spin and overreclaim excessively, which tanks performance and triggers userspace guards like swap exhaustion or pressure based OOM.

To fix this, send non-movable requests, regardless of order, into pageblock reclaim/compaction. This way, they help move things along to meet the ALLOC_NOFRAGMENT bar. After this patch, the reclaim storms and excess OOM rates are no longer observed in production.

The longer-term plan is still to have all requests, including the movable ones, help make blocks to spread the cost of defragmenting more evenly and fairly; combined with proper watermarking to reduce allocation latencies in the common case. However, doing this naively unearths scaling and concurrency limitations in compaction that need to be addressed first. Promoting just non-movables for now is the minimally viable bug fix for the above issue.

Quoted 27 lines · Johannes Weiner

As we deployed defrag_mode into Meta production, pressure spikes and excessive swapping were observed on some workloads. Tracing confirmed that this is unmovable/reclaimable requests spinning in the allocator and direct reclaim, causing excessive amounts of swap.

The initial plan for defrag_mode was to rely on kswapd/kcompactd to produce blocks, and if those are overwhelmed under high pressure, let the allocator fall back (__rmqueue_steal()) after its retry loops. However, that retrying results in more reclaim on some of these workloads than we'd hoped, sometimes excessively so, spurred on by the !costly order conditions in should_reclaim_retry().

The storms are dependent on the request type. Reclaim will inevitably make room in existing movable blocks, since that's where the LRU pages live. So if movable requests retry on reclaim, they make progress.

When non-movable requests spin in reclaim that isn't productive. They cannot use the individually freed pages, and the process is unlikely to accidentally free whole blocks to meet the ALLOC_NOFRAGMENT bar. They spin and overreclaim excessively, which tanks performance and triggers userspace guards like swap exhaustion or pressure based OOM.

To fix this, send non-movable requests, regardless of order, into pageblock reclaim/compaction. This way, they help move things along to meet the ALLOC_NOFRAGMENT bar. After this patch, the reclaim storms and excess OOM rates are no longer observed in production.

Quoted 7 lines

The longer-term plan is still to have all requests, including the movable ones, help make blocks to spread the cost of defragmenting more evenly and fairly; combined with proper watermarking to reduce allocation latencies in the common case. However, doing this naively unearths scaling and concurrency limitations in compaction that need to be addressed first. Promoting just non-movables for now is the minimally viable bug fix for the above issue.

Please forgive the noob question, I'm still struggling to get a really good mental handle on this stuff. But what about compact_first here? When we're promoting the order for compaction does it also make sense to promote compaction itself?

I'm aware you said "minimally viable bug fix" so it's fine if this falls outside of that, I'm just trying to poke around to improve my understanding.

Quoted 4 lines · Johannes Weiner

(quoted diff) +0 -0

@@ -4169,8 +4185,8 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,
 	barrier();
 	WRITE_ONCE(current->capture_control, &capc);
 

Review __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, · line 4190

Quoted 5 lines

(quoted diff) +2 -2

-	*compact_result = try_to_compact_pages(gfp_mask, order, alloc_flags, ac,
-							       prio, &capc);
+	*compact_result = try_to_compact_pages(gfp_mask, compact_order,
+					       alloc_flags, ac, prio, &capc);
 

Nit: the comment for try_to_compact_pages() is falsified by this, we need something like:

mm/compaction.c +1 -1

index 0568623d9384d..ca57403dc1522 100644
@@ -2823,7 +2823,7 @@ static enum compact_result compact_zone_order(struct zone *zone, int order,
 /**
  * try_to_compact_pages - Direct compact to satisfy a high-order allocation
  * @gfp_mask: The GFP mask of the current allocation
- * @order: The order of the current allocation
+ * @order: The order to try and make available
  * @alloc_flags: The allocation flags of the current allocation
  * @ac: The context of current allocation
  * @prio: Determines how hard direct compaction should try to succeed