Skip to contentSkip to conversation

mm: replace free hugepage folios after migration

PATCH New Mark Bundschuh 2 participants 6 messages Nov 5 – Nov 7

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mm: replace free hugepage folios after migration

From: yangge <[email protected]>

[ Upstream commit 04f13d241b8b146b23038bffd907cb8278391d07 ]

My machine has 4 NUMA nodes, each equipped with 32GB of memory. I have configured each NUMA node with 16GB of CMA and 16GB of in-use hugetlb pages. The allocation of contiguous memory via cma_alloc() can fail probabilistically.

When there are free hugetlb folios in the hugetlb pool, during the migration of in-use hugetlb folios, new folios are allocated from the free hugetlb pool. After the migration is completed, the old folios are released back to the free hugetlb pool instead of being returned to the buddy system. This can cause test_pages_isolated() check to fail, ultimately leading to the failure of cma_alloc().

Call trace:

cma_alloc()
__alloc_contig_migrate_range() // migrate in-use hugepage test_pages_isolated()
__test_page_isolated_in_pageblock()
PageBuddy(page) // check if the page is in buddy

To address this issue, we introduce a function named replace_free_hugepage_folios(). This function will replace the hugepage in the free hugepage pool with a new one and release the old one to the buddy system. After the migration of in-use hugetlb pages is completed, we will invoke replace_free_hugepage_folios() to ensure that these hugepages are properly released to the buddy system. Following this step, when test_pages_isolated() is executed for inspection, it will successfully pass.

Additionally, when alloc_contig_range() is used to migrate multiple in-use hugetlb pages, it can result in some in-use hugetlb pages being released back to the free hugetlb pool and subsequently being reallocated and used again. For example:

[huge 0] [huge 1]

To migrate huge 0, we obtain huge x from the pool. After the migration is completed, we return the now-freed huge 0 back to the pool. When it's time to migrate huge 1, we can simply reuse the now-freed huge 0 from the pool. As a result, when replace_free_hugepage_folios() is executed, it cannot release huge 0 back to the buddy system. To address this issue, we should prevent the reuse of isolated free hugepages during the migration process.

[ 6.1: Take only the dequeue_huge_page_node_exact() hunk, so memory
offline stops migrating huge pages back into the isolated range. ]

---
Without this, offlining a memory block holding mapped huge pages keeps migrating them back into the block until the offline is interrupted. Reproduced on arm64 on 5.10 and 6.12 (stock AL2023 6.12.113): the offline was still migrating after 60 s. With this patch on 5.10 it completes in 0.15 s with the data intact. Also sent for 5.10, 5.15, 6.1, 6.6 and 6.12.

 mm/hugetlb.c | 4 ++++
 1 file changed, 4 insertions(+)
mm/hugetlb.c +4 -0
index f549adc0b670..d39cd5c582c8 100644
@@ -46,6 +46,7 @@
 #include <linux/page_owner.h>
 #include "internal.h"
 #include "hugetlb_vmemmap.h"
+#include <linux/page-isolation.h>
 
 int hugetlb_max_hstate __read_mostly;
 unsigned int default_hstate_idx;
@@ -1343,6 +1344,9 @@ static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid)
 		if (PageHWPoison(page))
 			continue;
 
+		if (is_migrate_isolate_page(page))
+			continue;
+
 		list_move(&page->lru, &h->hugepage_activelist);
 		set_page_refcounted(page);
 		ClearHPageFreed(page);
Signature · 1 line
2.50.1
#1 lore raw
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider: - [High] mm: hugetlb: skipping isolated free pages in dequeue_huge_page_node_exact() breaks reservation…

Thank you for your contribution! Sashiko AI review found 2 potential issues to consider:
- [High] mm: hugetlb: skipping isolated free pages in dequeue_huge_page_node_exact() breaks reservation accounting
- [Medium] mm: omitted replace_free_hugepage_folios() leaves cma_alloc() broken
--

commit dac0c7128f296db4178dac403221b3262724e8f6
Author: yangge <[email protected]>

mm: replace free hugepage folios after migration

This backport of upstream commit 04f13d241b8b146b23038bffd907cb8278391d07 updates dequeue_huge_page_node_exact() in mm/hugetlb.c to skip free hugepages located in MIGRATE_ISOLATE pageblocks so that isolated free hugepages are not reused during migration.

[Severity: Medium]
The commit title and changelog describe introducing and calling replace_free_hugepage_folios() after migrating in-use hugetlb pages in __alloc_contig_migrate_range(), but this 6.1 backport only keeps the is_migrate_isolate_page(page) check in dequeue_huge_page_node_exact() at mm/hugetlb.c:1347.

In __alloc_contig_migrate_range() at mm/page_alloc.c:9231, when an in-use hugetlb page in the target range [start, end) is migrated while available_huge_pages(h) > 0 and a non-isolated free hugepage exists on h->hugepage_freelists, alloc_migration_target() -> alloc_huge_page_nodemask() dequeues that persistent pool page without HPageTemporary set. When migration completes, free_huge_page() at mm/hugetlb.c:1978 enqueues the old source page back onto h->hugepage_freelists[nid] instead of freeing it to the buddy allocator.

Because replace_free_hugepage_folios() was omitted from this backport (even though 6.1 already has __alloc_contig_migrate_range() in mm/page_alloc.c and alloc_and_dissolve_huge_page() in mm/hugetlb.c), the old source page remains a PageHuge page rather than PageBuddy inside [start, end). Won't test_pages_isolated() in alloc_contig_range() at mm/page_alloc.c:9366 and the !PageBuddy(pfn_to_page(outer_pfn)) check in isolate_single_pageblock() at mm/page_isolation.c:439 still fail with -EBUSY, leaving the cma_alloc() failure described in the commit message unresolved?

[Severity: High]
In dequeue_huge_page_node_exact() at mm/hugetlb.c:1347, skipping pages where is_migrate_isolate_page(page) is true leaves those isolated free hugepages on h->hugepage_freelists[nid] and still counted in h->free_huge_pages.

However, available_huge_pages() at mm/hugetlb.c:1397 still computes h->free_huge_pages - h->resv_huge_pages without discounting isolated free hugepages. Can this cause two regressions in reservation accounting?

First, when a free hugepage backing h->resv_huge_pages lies in a MIGRATE_ISOLATE pageblock (either isolated in place by start_isolate_page_range() during offline_pages() or alloc_contig_range(), or placed there after migration), a page fault on a VMA with a valid reservation calls hugetlb_no_page() -> alloc_huge_page() at mm/hugetlb.c:3139 -> dequeue_huge_page_vma() -> dequeue_huge_page_node_exact(). Because dequeue_huge_page_node_exact() skips the isolated page, dequeue_huge_page_vma() returns NULL and alloc_huge_page() falls back to alloc_buddy_huge_page_with_mpol() -> alloc_surplus_huge_page() at mm/hugetlb.c:2456. When h->surplus_huge_pages >= h->nr_overcommit_huge_pages (which is 0 by default) or hstate_is_gigantic(h), alloc_huge_page() returns ERR_PTR(-ENOSPC), delivering SIGBUS to a process faulting on a reserved mapping.

Second, during offline_pages(), suppose an unreserved free hugepage H0 is in the MIGRATE_ISOLATE range (for example, after an in-use hugepage H0 was migrated to an unreserved pool page and freed via free_huge_page() -> enqueue_huge_page()) and another free hugepage F2 outside the isolated range backs a reservation (h->free_huge_pages == 2, h->resv_huge_pages == 1). Because H0 is still counted in h->free_huge_pages, available_huge_pages(h) remains 1. When migrating a second in-use hugepage H1 via alloc_huge_page_nodemask() at mm/hugetlb.c:2546 (or during a concurrent unreserved fault in dequeue_huge_page_vma() at mm/hugetlb.c:1417), available_huge_pages(h) is non-zero, so dequeue_huge_page_nodemask() is called, skips the isolated H0 in dequeue_huge_page_node_exact(), and consumes the non-isolated reserved page F2 instead of falling back to alloc_migrate_huge_page(). H1 is then freed onto h->hugepage_freelists inside the isolated range, leaving only isolated pages on the freelist to back h->resv_huge_pages. When offline_pages() subsequently calls dissolve_free_huge_pages() -> dissolve_free_huge_page() at mm/hugetlb.c:2359, won't it dissolve H0 and then fail on H1 with -EBUSY because !available_huge_pages(h)?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#2 lore raw

v2 cover mm: replace free hugepage folios after migration

In reply to Mark Bundschuh

Full backport of 04f13d241b8b ("mm: replace free hugepage folios after migration") with its fix 113ed54ad276.

Tested in a VM on 6.1.191: memory offline of a block with mapped huge pages no longer livelocks, and cma_alloc() over mapped huge pages now succeeds (it failed on 6.1.191 and with v1).

v1: https://lore.kernel.org/all/[email protected]/

Ge Yang (1):
mm/hugetlb: fix kernel NULL pointer dereference when replacing free
hugetlb folios

yangge (1):
mm: replace free hugepage folios after migration

 include/linux/hugetlb.h |  7 ++++++
 mm/hugetlb.c            | 50 +++++++++++++++++++++++++++++++++++++++++
 mm/page_alloc.c         | 12 +++++++++-
 3 files changed, 68 insertions(+), 1 deletion(-)
Signature · 1 line
2.50.1
#3 lore raw

PATCH v2 1/2 mm: replace free hugepage folios after migration

From: yangge <[email protected]>

[ Upstream commit 04f13d241b8b146b23038bffd907cb8278391d07 ]

My machine has 4 NUMA nodes, each equipped with 32GB of memory. I have configured each NUMA node with 16GB of CMA and 16GB of in-use hugetlb pages. The allocation of contiguous memory via cma_alloc() can fail probabilistically.

When there are free hugetlb folios in the hugetlb pool, during the migration of in-use hugetlb folios, new folios are allocated from the free hugetlb pool. After the migration is completed, the old folios are released back to the free hugetlb pool instead of being returned to the buddy system. This can cause test_pages_isolated() check to fail, ultimately leading to the failure of cma_alloc().

Call trace:

cma_alloc()
__alloc_contig_migrate_range() // migrate in-use hugepage test_pages_isolated()
__test_page_isolated_in_pageblock()
PageBuddy(page) // check if the page is in buddy

To address this issue, we introduce a function named replace_free_hugepage_folios(). This function will replace the hugepage in the free hugepage pool with a new one and release the old one to the buddy system. After the migration of in-use hugetlb pages is completed, we will invoke replace_free_hugepage_folios() to ensure that these hugepages are properly released to the buddy system. Following this step, when test_pages_isolated() is executed for inspection, it will successfully pass.

Additionally, when alloc_contig_range() is used to migrate multiple in-use hugetlb pages, it can result in some in-use hugetlb pages being released back to the free hugetlb pool and subsequently being reallocated and used again. For example:

[huge 0] [huge 1]

To migrate huge 0, we obtain huge x from the pool. After the migration is completed, we return the now-freed huge 0 back to the pool. When it's time to migrate huge 1, we can simply reuse the now-freed huge 0 from the pool. As a result, when replace_free_hugepage_folios() is executed, it cannot release huge 0 back to the buddy system. To address this issue, we should prevent the reuse of isolated free hugepages during the migration process.

[ 6.1: dequeue_huge_page_node_exact() and alloc_and_dissolve_huge_page()
take pages here. ]

---
 include/linux/hugetlb.h |  7 +++++++
 mm/hugetlb.c            | 42 +++++++++++++++++++++++++++++++++++++++++
 mm/page_alloc.c         | 12 +++++++++++-
 3 files changed, 60 insertions(+), 1 deletion(-)
include/linux/hugetlb.h +7 -0
index 6b78a0f457d9..13606ab7d0ce 100644
@@ -728,6 +728,7 @@ struct huge_bootmem_page {
 };
 
 int isolate_or_dissolve_huge_page(struct page *page, struct list_head *list);
+int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long end_pfn);
 struct page *alloc_huge_page(struct vm_area_struct *vma,
 				unsigned long addr, int avoid_reserve);
 struct page *alloc_huge_page_nodemask(struct hstate *h, int preferred_nid,
@@ -1044,6 +1045,12 @@ static inline int isolate_or_dissolve_huge_page(struct page *page,
 	return -ENOMEM;
 }
 
+static inline int replace_free_hugepage_folios(unsigned long start_pfn,
+		unsigned long end_pfn)
+{
+	return 0;
+}
+
 static inline struct page *alloc_huge_page(struct vm_area_struct *vma,
 					   unsigned long addr,
 					   int avoid_reserve)
mm/hugetlb.c +42 -0

60 lines not on this page · raw message

mm/page_alloc.c +11 -1
index 85b189bcab7e..0004df47713d 100644
@@ -9329,7 +9329,17 @@ int alloc_contig_range(unsigned long start, unsigned long end,
 	ret = __alloc_contig_migrate_range(&cc, start, end);
 	if (ret && ret != -EBUSY)
 		goto done;
-	ret = 0;
+
+	/*
+	 * When in-use hugetlb pages are migrated, they may simply be released
+	 * back into the free hugepage pool instead of being returned to the
+	 * buddy system.  After the migration of in-use huge pages is completed,
+	 * we will invoke replace_free_hugepage_folios() to ensure that these
+	 * hugepages are properly released to the buddy system.
+	 */
+	ret = replace_free_hugepage_folios(start, end);
+	if (ret)
+		goto done;
 
 	/*
 	 * Pages from [start, end) are within a pageblock_nr_pages
Signature · 1 line
2.50.1
#4 lore raw
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider: - [High] mm: hugetlb: skipping isolated free pages in dequeue_huge_page_node_exact() breaks reservation…

Thank you for your contribution! Sashiko AI review found 2 potential issues to consider:
- [High] mm: hugetlb: skipping isolated free pages in dequeue_huge_page_node_exact() breaks reservation accounting
- [Medium] mm/hugetlb: repeated hugepage replacement across tail PFNs in replace_free_hugepage_folios()
--

commit bf676e0a7d0d63af3750623a0cea703c4c123b82
Author: yangge <[email protected]>

mm: replace free hugepage folios after migration

This commit adds replace_free_hugepage_folios() to replace free hugetlb folios in an isolated PFN range after migration and skips MIGRATE_ISOLATE pages when dequeuing free hugetlb pages.

Quoted 4 lines
mm/hugetlb.c +0 -0
index f549adc0b6705..46031ca5a7fb1 100644

[ ... ]

mm/hugetlb.c static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid) · line 1352

Quoted 10 lines
mm/hugetlb.c +3 -0
@@ -1343,6 +1344,9 @@ static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid)
 		if (PageHWPoison(page))
 			continue;
 
+		if (is_migrate_isolate_page(page))
+			continue;
+
 		list_move(&page->lru, &h->hugepage_activelist);
 		set_page_refcounted(page);
 		ClearHPageFreed(page);

[Severity: High]
Does skipping MIGRATE_ISOLATE pages in dequeue_huge_page_node_exact() break hugetlb reservation accounting while those pages remain counted in h->free_huge_pages?

When a free hugepage backing h->resv_huge_pages lies in a MIGRATE_ISOLATE pageblock during alloc_contig_range() or offline_pages(), a fault on a VMA with a valid reservation sees dequeue_huge_page_vma() return NULL and falls back to alloc_buddy_huge_page_with_mpol() -> alloc_surplus_huge_page(). With overcommit at its default of 0 (or for gigantic pages), alloc_huge_page() returns ERR_PTR(-ENOSPC) and delivers SIGBUS to the faulting task.

Similarly, an isolated unreserved page on h->hugepage_freelists still inflates available_huge_pages(h) (h->free_huge_pages - h->resv_huge_pages), allowing alloc_huge_page_nodemask() or an unreserved fault in dequeue_huge_page_vma() to pass the available_huge_pages(h) check and consume a non-isolated reserved page instead. Once only isolated free pages remain to back h->resv_huge_pages, dissolve_free_huge_page() fails with -EBUSY on !available_huge_pages(h), aborting offline_pages().

Quoted 31 lines
mm/hugetlb.c +27 -0
@@ -3073,6 +3077,44 @@ int isolate_or_dissolve_huge_page(struct page *page, struct list_head *list)
 	return ret;
 }
 
+/*
+ *  replace_free_hugepage_folios - Replace free hugepage folios in a given pfn
+ *  range with new folios.
+ *  @start_pfn: start pfn of the given pfn range
+ *  @end_pfn: end pfn of the given pfn range
+ *  Returns 0 on success, otherwise negated error.
+ */
+int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long end_pfn)
+{
+	struct hstate *h;
+	struct folio *folio;
+	int ret = 0;
+
+	LIST_HEAD(isolate_list);
+
+	while (start_pfn < end_pfn) {
+		folio = pfn_folio(start_pfn);
+		if (folio_test_hugetlb(folio)) {
+			h = folio_hstate(folio);
+		} else {
+			start_pfn++;
+			continue;
+		}
+
+		if (!folio_ref_count(folio)) {
+			ret = alloc_and_dissolve_huge_page(h, &folio->page,
+							   &isolate_list);

mm/hugetlb.c int isolate_or_dissolve_huge_page(struct page *page, struct list_head *list) · line 3116

Quoted 10 lines
mm/hugetlb.c +10 -0
+			if (ret)
+				break;
+
+			putback_movable_pages(&isolate_list);
+		}
+		start_pfn++;
+	}
+
+	return ret;
+}

[Severity: Medium]
Can advancing start_pfn by 1 cause replace_free_hugepage_folios() to repeatedly replace the same hugepage across all of its tail PFNs?

alloc_and_dissolve_huge_page() calls update_and_free_page() -> __update_and_free_page() and returns 0:

mm/hugetlb.c:__update_and_free_page() {
    ...
	if (hugetlb_vmemmap_restore(h, page)) {
		spin_lock_irq(&hugetlb_lock);
		add_hugetlb_page(h, page, true);
		spin_unlock_irq(&hugetlb_lock);
		return;
	}
    ...
}

If hugetlb_vmemmap_restore() fails to allocate vmemmap pages, add_hugetlb_page() puts the undissolved compound page back on the freelist with refcount 0 as a surplus page. Because the loop only increments start_pfn by 1 and pfn_folio(start_pfn) resolves each tail page back to the head folio, the remaining pages_per_huge_page(h) - 1 iterations call alloc_and_dissolve_huge_page() on the same folio again, allocating extra hugepages and inflating h->nr_huge_pages and h->surplus_huge_pages. Advancing start_pfn by pages_per_huge_page(h) (or the next hugepage boundary) avoids revisiting the tail pages.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#5 lore raw

PATCH v2 2/2 mm/hugetlb: fix kernel NULL pointer dereference when replacing free hugetlb folios

In reply to Mark Bundschuh

From: Ge Yang <[email protected]>

[ Upstream commit 113ed54ad276c352ee5ce109bdcf0df118a43bda ]

A kernel crash was observed when replacing free hugetlb folios:

BUG: kernel NULL pointer dereference, address: 0000000000000028
PGD 0 P4D 0
Oops: Oops: 0000 [#1] SMP NOPTI
CPU: 28 UID: 0 PID: 29639 Comm: test_cma.sh Tainted 6.15.0-rc6-zp #41 PREEMPT(voluntary)
RIP: 0010:alloc_and_dissolve_hugetlb_folio+0x1d/0x1f0
RSP: 0018:ffffc9000b30fa90 EFLAGS: 00010286
RAX: 0000000000000000 RBX: 0000000000342cca RCX: ffffea0043000000
RDX: ffffc9000b30fb08 RSI: ffffea0043000000 RDI: 0000000000000000
RBP: ffffc9000b30fb20 R08: 0000000000001000 R09: 0000000000000000
R10: ffff88886f92eb00 R11: 0000000000000000 R12: ffffea0043000000
R13: 0000000000000000 R14: 00000000010c0200 R15: 0000000000000004
FS:  00007fcda5f14740(0000) GS:ffff8888ec1d8000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000028 CR3: 0000000391402000 CR4: 0000000000350ef0
Call Trace:
<TASK>
 replace_free_hugepage_folios+0xb6/0x100
 alloc_contig_range_noprof+0x18a/0x590
 ? srso_return_thunk+0x5/0x5f
 ? down_read+0x12/0xa0
 ? srso_return_thunk+0x5/0x5f
 cma_range_alloc.constprop.0+0x131/0x290
 __cma_alloc+0xcf/0x2c0
 cma_alloc_write+0x43/0xb0
 simple_attr_write_xsigned.constprop.0.isra.0+0xb2/0x110
 debugfs_attr_write+0x46/0x70
 full_proxy_write+0x62/0xa0
 vfs_write+0xf8/0x420
 ? srso_return_thunk+0x5/0x5f
 ? filp_flush+0x86/0xa0
 ? srso_return_thunk+0x5/0x5f
 ? filp_close+0x1f/0x30
 ? srso_return_thunk+0x5/0x5f
 ? do_dup2+0xaf/0x160
 ? srso_return_thunk+0x5/0x5f
 ksys_write+0x65/0xe0
 do_syscall_64+0x64/0x170
 entry_SYSCALL_64_after_hwframe+0x76/0x7e

There is a potential race between __update_and_free_hugetlb_folio() and replace_free_hugepage_folios():

CPU1 CPU2
__update_and_free_hugetlb_folio replace_free_hugepage_folios
folio_test_hugetlb(folio) -- It's still hugetlb folio.

__folio_clear_hugetlb(folio) hugetlb_free_folio(folio)
h = folio_hstate(folio) -- Here, h is NULL pointer

When the above race condition occurs, folio_hstate(folio) returns NULL, and subsequent access to this NULL pointer will cause the system to crash. To resolve this issue, execute folio_hstate(folio) under the protection of the hugetlb_lock lock, ensuring that folio_hstate(folio) does not return NULL.

Fixes 04f13d241b8b ("mm: replace free hugepage folios after migration")
Cc Baolin Wang <[email protected]>
Cc Barry Song <[email protected]>
Cc David Hildenbrand <[email protected]>
---
 mm/hugetlb.c | 8 ++++++++
 1 file changed, 8 insertions(+)
mm/hugetlb.c +8 -0
index 46031ca5a7fb..6901faf1d2a2 100644
@@ -3094,12 +3094,20 @@ int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long end_pfn)
 
 	while (start_pfn < end_pfn) {
 		folio = pfn_folio(start_pfn);
+
+		/*
+		 * The folio might have been dissolved from under our feet, so make sure
+		 * to carefully check the state under the lock.
+		 */
+		spin_lock_irq(&hugetlb_lock);
 		if (folio_test_hugetlb(folio)) {
 			h = folio_hstate(folio);
 		} else {
+			spin_unlock_irq(&hugetlb_lock);
 			start_pfn++;
 			continue;
 		}
+		spin_unlock_irq(&hugetlb_lock);
 
 		if (!folio_ref_count(folio)) {
 			ret = alloc_and_dissolve_huge_page(h, &folio->page,
Signature · 1 line
2.50.1