Skip to contentSkip to conversation

Fix bugs on HugeTLB folio allocation failure paths

PATCH Superseded Ackerley Tng 2 participants 18 messages Jul 8 – Jul 14

Subsystems (3)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v2 cover Fix bugs on HugeTLB folio allocation failure paths

When mem_cgroup_charge_hugetlb() fails (e.g. when hitting memcg limits), the HugeTLB allocation path currently handles the errors incorrectly.

This series fixes 5 issues related to HugeTLB allocation failure paths.

Some of these issues were pointed out by Sashiko while I was working on [1].

0. The fix for (1.) below was in v1 of this series was incomplete, because it fixed the case where used_hpages was not restored, but introduced another issue, where subpool reservations are always restored even when a subpool reservation was not used. I looked into this more, I believe there's a deeper conceptual conflict in the way hugepage_subpool_put_pages() is used. I added a new patch to this series, which addresses the issue.

With the additional change, (1.) now does the correct thing. It always calls hugepage_subpool_put_pages(), which restores used_hpages, and is able to correctly skip updating reservations as well. The other patches fall in place after this additional patch.

1. Fix subpool usage leak on allocation failure: (Sashiko pointed this out in [2])

When alloc_hugetlb_folio() fails early (e.g. buddy allocation failure or hugetlb cgroup charging failure) and gbl_chg == 1, the error path skips restoring the page to the subpool, leaking the subpool's used_hpages counter. Fix this by calling hugepage_subpool_put_pages() unconditionally if map_chg is true.

(1.) above could technically be a separate patch series, but I'm relying on that for fix (4.) below.

2. Fix folio refcount mismatch on memcg charge failure: (Sashiko pointed this out in [3])

The error path in alloc_hugetlb_folio() calls free_huge_folio() directly on a folio with a refcount of 1 (set via folio_ref_unfreeze() earlier). This triggers VM_BUG_ON_FOLIO(folio_ref_count(folio), folio) if CONFIG_DEBUG_VM is enabled, and can corrupt allocator state otherwise. Fix this by using folio_put() instead of free_huge_folio() to correctly drop the refcount before freeing.

While fixing the above, the reproducer (I'll send these separately) was causing an infinite loop during allocation, unveiling issue (3.).

3. Return -ENOSPC on memcg charge failure to prevent infinite loop:

alloc_hugetlb_folio() propagates -ENOMEM on charge failure, which maps to VM_FAULT_OOM. Because HugeTLB physical allocations are high-order and use __GFP_RETRY_MAYFAIL, the OOM killer is bypassed. Returning VM_FAULT_OOM leaks to the #PF handler, which cannot make progress and retries the faulting instruction indefinitely. Fix this by returning -ENOSPC instead of -ENOMEM on charge failure, which maps to VM_FAULT_SIGBUS, terminating the process cleanly.

And while running the reproducer, I checked this routinely

head /sys/kernel/mm/hugepages/hugepages-2048kB/*

and found that there was a resv_hugepages leak, 1 per execution of the reproducer, unveiling (4.) below.

4. Move memcg charge earlier to prevent reservation leak:

When mem_cgroup_charge_hugetlb() fails, the error path historically bypassed vma_end_reservation(). Since the reservation had already been committed via vma_commit_reservation(), this left the reservation map in an inconsistent state, leaking resv_huge_pages when the process exited. Fix this by moving mem_cgroup_charge_hugetlb() earlier in alloc_hugetlb_folio(), before vma_commit_reservation() is called.

Testing:

+ libhugetlbfs tests pass
+ ./tools/testing/selftests/mm/ksft_hugetlb.sh passes

v1: https://lore.kernel.org/r/20260707-hugetlb-alloc-failure-fixes-v1-0-5bbd3a4b836d@google.com

The series [1] has some changes that are dependent on these fixes, so I'll wait for your reviews before continuing on [1].

Thank you!

[1] https://lore.kernel.org/all/[email protected]/T/
[2] https://sashiko.dev/#/patchset/20260518-hugetlb-open-up-v3-0-e14b302477f8%40google.com?part=5
[3] https://sashiko.dev/#/patchset/20260702-hugetlb-open-up-v4-0-d53cefcccf34%40google.com?part=6

---
Ackerley Tng (5):
mm: hugetlb: Track used_hpages when getting/putting pages from subpool
mm: hugetlb: Fix subpool usage leak on allocation failure
mm: hugetlb: Fix folio refcount mismatch on memcg charge failure
mm: hugetlb: Return -ENOSPC on memcg charge failure
mm: hugetlb: Move memcg charge earlier to prevent reservation leak

 fs/hugetlbfs/inode.c    |  8 +++--
 include/linux/hugetlb.h |  4 +--
 mm/hugetlb.c            | 79 +++++++++++++++++++++++++++----------------------
 3 files changed, 52 insertions(+), 39 deletions(-)

---
base-commit: 0e35b9b6ec0ffcc5e23cbdec09f5c622ad532b53
change-id: 20260706-hugetlb-alloc-failure-fixes-c7f775eca29f

Best regards,
--
Ackerley Tng <[email protected]>

#1 lore raw

PATCH v2 1/5 mm: hugetlb: Track used_hpages when getting/putting pages from subpool

hugepage_subpool_put_pages() currently has two distinct responsibilities that conflict:

1. When size is specified for the mount, max_hpages != -1: Keep track of total active pages (allocated + reserved) and decrement this count (used_hpages) when a page is freed or allocation fails.
2. When min_size is specified for the mount, min_hpages != -1: Ensure we don't drop below the guaranteed minimum, and restore a reservation (rsv_hpages) if we do.

This causes trouble because when allocation fails (refer to alloc_hugetlb_folio()) if gbl_chg = 1 (i.e. no subpool reservation was taken):

+ To keep used_hpages consistent, HugeTLB needs to call hugepage_subpool_put_pages() to restore undo used_hpages being incremented
+ But can't call hugepage_subpool_put_pages() if no reservation was consumed.

One option would be to conditionally do subpool tracking updates outside of the hugepage_subpool_put_pages() function, but that would spread logic all over.

Instead, always track used_hpages, regardless of whether a max_size was requested for the mount, so that the subpool always knows how many pages were allocated through it. Every page allocated through the subpool increments used_hpages, regardless of whether a reservation was taken from it.

Conceptually, now, every allocation involving a subpool uses a page from the subpool, which must be returned to the subpool. Every page taken from the subpool tries to use a subpool reservation. Restoring a page to the subpool reservations only if the page was taken from subpool reservations. (If used_hpages >= min_hpages, the page must have not have been taken from the reservations.)

Always tracking used_hpages provides the subpool with information of both used and reserved counts to make the correct decision for both max_size and min_size correctly.

With used_hpages always tracked, subpool_is_free() can be simplified, such that the subpool can be declared free if there are no more pages in use.

Also update the documentation for used_hpages, since it no longer matters whether the used pages count against the maximum.

Also update statfs reporting. Previously, if max_hpages is negative, used_hpages is static at 0, so returning max_hpages - used_hpages returns -1 and is always correct. Now, if the subpool doesn't have a maximum requested size, indicate no limit for free pages (-1). If it does have a maximum size, report the difference between the requested size and the number of used pages. This difference is always positive, because if the mount does have a maximum size, hugepage_subpool_get_pages() ensures that the subpool usage never exceeds the maximum.

This fixes a bug in hugetlb_unreserve_pages(), where pages are returned to the subpool regardless of whether it consumed a reservation. The corresponding bug in the failure handling path of alloc_hugetlb_folio() was fixed in a833a693a490e.

Fixes 1c5ecae3a93fa ("hugetlbfs: add minimum size accounting to subpools")
---
 fs/hugetlbfs/inode.c    |  8 ++++++--
 include/linux/hugetlb.h |  4 ++--
 mm/hugetlb.c            | 22 ++++++++--------------
 3 files changed, 16 insertions(+), 18 deletions(-)
fs/hugetlbfs/inode.c +6 -2
index 216e1a0dd0b23..26c0187340636 100644
@@ -1109,8 +1109,12 @@ static int hugetlbfs_statfs(struct dentry *dentry, struct kstatfs *buf)
 
 			spin_lock_irq(&sbinfo->spool->lock);
 			buf->f_blocks = sbinfo->spool->max_hpages;
-			free_pages = sbinfo->spool->max_hpages
-				- sbinfo->spool->used_hpages;
+			if (sbinfo->spool->max_hpages == -1) {
+				free_pages = -1;
+			} else {
+				free_pages = sbinfo->spool->max_hpages -
+					     sbinfo->spool->used_hpages;
+			}
 			buf->f_bavail = buf->f_bfree = free_pages;
 			spin_unlock_irq(&sbinfo->spool->lock);
 			buf->f_files = sbinfo->max_inodes;
include/linux/hugetlb.h +2 -2
index 2abaf99321e90..34b9a3e1be0fa 100644
@@ -38,8 +38,8 @@ struct hugepage_subpool {
 	spinlock_t lock;
 	long count;
 	long max_hpages;	/* Maximum huge pages or -1 if no maximum. */
-	long used_hpages;	/* Used count against maximum, includes */
-				/* both allocated and reserved pages. */
+	long used_hpages;	/* Used page count, includes both */
+				/* allocated and reserved pages. */
 	struct hstate *hstate;
 	long min_hpages;	/* Minimum huge pages or -1 if no minimum. */
 	long rsv_hpages;	/* Pages reserved against global pool to */
mm/hugetlb.c +8 -14
index 571212b80835e..ee5e99c1894b9 100644
@@ -129,12 +129,8 @@ static inline bool subpool_is_free(struct hugepage_subpool *spool)
 {
 	if (spool->count)
 		return false;
-	if (spool->max_hpages != -1)
-		return spool->used_hpages == 0;
-	if (spool->min_hpages != -1)
-		return spool->rsv_hpages == spool->min_hpages;
 
-	return true;
+	return spool->used_hpages == 0;
 }
 
 static inline void unlock_or_release_subpool(struct hugepage_subpool *spool,
@@ -205,15 +201,14 @@ static long hugepage_subpool_get_pages(struct hugepage_subpool *spool,
 
 	spin_lock_irq(&spool->lock);
 
-	if (spool->max_hpages != -1) {		/* maximum size accounting */
-		if ((spool->used_hpages + delta) <= spool->max_hpages)
-			spool->used_hpages += delta;
-		else {
-			ret = -ENOMEM;
-			goto unlock_ret;
-		}
+	if (spool->max_hpages != -1 &&
+	    spool->used_hpages + delta > spool->max_hpages) {
+		ret = -ENOMEM;
+		goto unlock_ret;
 	}
 
+	spool->used_hpages += delta;
+
 	/* minimum size accounting */
 	if (spool->min_hpages != -1 && spool->rsv_hpages) {
 		if (delta > spool->rsv_hpages) {
@@ -251,8 +246,7 @@ static long hugepage_subpool_put_pages(struct hugepage_subpool *spool,
 
 	spin_lock_irqsave(&spool->lock, flags);
 
-	if (spool->max_hpages != -1)		/* maximum size accounting */
-		spool->used_hpages -= delta;
+	spool->used_hpages -= delta;
 
 	 /* minimum size accounting */
 	if (spool->min_hpages != -1 && spool->used_hpages < spool->min_hpages) {
Signature · 1 line
2.55.0.795.g602f6c329a-goog
#2 lore raw

PATCH v2 2/5 mm: hugetlb: Fix subpool usage leak on allocation failure

In reply to Ackerley Tng

When alloc_hugetlb_folio() fails early (e.g. buddy allocation failure or hugetlb cgroup charging failure) and gbl_chg == 1 (meaning a reservation was not used, but a global page was allocated instead), the subpool page acquired via hugepage_subpool_get_pages() must still be returned.

Currently, the error path out_subpool_put: only calls hugepage_subpool_put_pages() if !gbl_chg is true. If gbl_chg is 1, it skips it, permanently leaking the subpool's used_hpages counter.

With the earlier patch to always track used_hpages in the subpool, always call hugepage_subpool_put_pages() if map_chg is true to consistently restore the page to the subpool. Only call hugetlb_acct_memory() to adjust global reservations if gbl_chg == 0 since gbl_chg == 0 indicates a subpool (and global) reservation was used.

Fixes a833a693a490e ("mm: hugetlb: fix incorrect fallback for subpool")
---
 mm/hugetlb.c | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)
mm/hugetlb.c +6 -8
index ee5e99c1894b9..4093c1c0a4a1d 100644
@@ -2852,7 +2852,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 	struct hugepage_subpool *spool = subpool_vma(vma);
 	struct hstate *h = hstate_vma(vma);
 	struct folio *folio;
-	long retval, gbl_chg, gbl_reserve;
+	long retval, gbl_chg;
 	map_chg_state map_chg;
 	int ret, idx;
 	struct hugetlb_cgroup *h_cg = NULL;
@@ -3003,13 +3003,11 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 		hugetlb_cgroup_uncharge_cgroup_rsvd(idx, pages_per_huge_page(h),
 						    h_cg_rsvd);
 out_subpool_put:
-	/*
-	 * put page to subpool iff the quota of subpool's rsv_hpages is used
-	 * during hugepage_subpool_get_pages.
-	 */
-	if (map_chg && !gbl_chg) {
-		gbl_reserve = hugepage_subpool_put_pages(spool, 1);
-		hugetlb_acct_memory(h, -gbl_reserve);
+	if (map_chg) {
+		long gbl_reserve = hugepage_subpool_put_pages(spool, 1);
+
+		if (!gbl_chg)
+			hugetlb_acct_memory(h, -gbl_reserve);
 	}
 
 
Signature · 1 line
2.55.0.795.g602f6c329a-goog
#3 lore raw

Re: [PATCH v2 2/5] mm: hugetlb: Fix subpool usage leak on allocation failure

Hi Ackerley,

Thank you for this series. I really wanted to work on hugeTLB accounting fixes but never got the time to get to it. I'm very grateful that you are taking a look!!

Quoted 11 lines

From: Ackerley Tng <[email protected]>

When alloc_hugetlb_folio() fails early (e.g. buddy allocation failure or hugetlb cgroup charging failure) and gbl_chg == 1 (meaning a reservation was not used, but a global page was allocated instead), the subpool page acquired via hugepage_subpool_get_pages() must still be returned.

Currently, the error path out_subpool_put: only calls hugepage_subpool_put_pages() if !gbl_chg is true. If gbl_chg is 1, it skips it, permanently leaking the subpool's used_hpages counter.

Quoted 5 lines

With the earlier patch to always track used_hpages in the subpool, always call hugepage_subpool_put_pages() if map_chg is true to consistently restore the page to the subpool. Only call hugetlb_acct_memory() to adjust global reservations if gbl_chg == 0 since gbl_chg == 0 indicates a subpool (and global) reservation was used.

So I think that I've seen that this part of the accounting specifically is a bit suspicious. There have been two attempts in the past to fix this area [1] [2]. I think functionally they are quite similar to this fix, they just open-code the contents of the put_pages function inside the condition. I've Cc-ed the authors of those two patches in case they wanted to chime in.

I reference these fixes because I think they handle the minimum subpage case a bit differently. To be honest, I recall reading those fixes a while back and getting a bit confused on what exactly happens when the page is absorbed to fulfill the minimum size...

It does seem like Sashiko also notes this as a possible concern. WDYT? Does your reproducer for this issue also work when a minimum size is set (let's say, to 1?)

Thanks again. I hope you have a great day!!!
Joshua

Quoted 27 lines
Fixes a833a693a490e ("mm: hugetlb: fix incorrect fallback for subpool")
---
 mm/hugetlb.c | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)
mm/hugetlb.c +1 -4
index ee5e99c1894b9..4093c1c0a4a1d 100644
@@ -2852,7 +2852,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 	struct hugepage_subpool *spool = subpool_vma(vma);
 	struct hstate *h = hstate_vma(vma);
 	struct folio *folio;
-	long retval, gbl_chg, gbl_reserve;
+	long retval, gbl_chg;
 	map_chg_state map_chg;
 	int ret, idx;
 	struct hugetlb_cgroup *h_cg = NULL;
@@ -3003,13 +3003,11 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 		hugetlb_cgroup_uncharge_cgroup_rsvd(idx, pages_per_huge_page(h),
 						    h_cg_rsvd);
 out_subpool_put:
-	/*
-	 * put page to subpool iff the quota of subpool's rsv_hpages is used
-	 * during hugepage_subpool_get_pages.

mm/hugetlb.c struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, · line 3011

Quoted 10 lines
mm/hugetlb.c +5 -4
-	 */
-	if (map_chg && !gbl_chg) {
-		gbl_reserve = hugepage_subpool_put_pages(spool, 1);
-		hugetlb_acct_memory(h, -gbl_reserve);
+	if (map_chg) {
+		long gbl_reserve = hugepage_subpool_put_pages(spool, 1);
+
+		if (!gbl_chg)
+			hugetlb_acct_memory(h, -gbl_reserve);
 	}

[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/

#4 lore raw

Re: [PATCH v2 2/5] mm: hugetlb: Fix subpool usage leak on allocation failure

Quoted 23 lines · Joshua Hahn

Hi Ackerley,

Thank you for this series. I really wanted to work on hugeTLB accounting fixes but never got the time to get to it. I'm very grateful that you are taking a look!!

Quoted 11 lines

From: Ackerley Tng <[email protected]>

When alloc_hugetlb_folio() fails early (e.g. buddy allocation failure or hugetlb cgroup charging failure) and gbl_chg == 1 (meaning a reservation was not used, but a global page was allocated instead), the subpool page acquired via hugepage_subpool_get_pages() must still be returned.

Currently, the error path out_subpool_put: only calls hugepage_subpool_put_pages() if !gbl_chg is true. If gbl_chg is 1, it skips it, permanently leaking the subpool's used_hpages counter.

Quoted 5 lines

With the earlier patch to always track used_hpages in the subpool, always call hugepage_subpool_put_pages() if map_chg is true to consistently restore the page to the subpool. Only call hugetlb_acct_memory() to adjust global reservations if gbl_chg == 0 since gbl_chg == 0 indicates a subpool (and global) reservation was used.

Quoted 7 lines

So I think that I've seen that this part of the accounting specifically is a bit suspicious. There have been two attempts in the past to fix this area [1] [2]. I think functionally they are quite similar to this fix, they just open-code the contents of the put_pages function inside the condition. I've Cc-ed the authors of those two patches in case they wanted to chime in.

Thanks for connecting us! I didn't realize this was already being worked on. Also adding Lance, who commented at [3].

Quoted 9 lines

I reference these fixes because I think they handle the minimum subpage case a bit differently. To be honest, I recall reading those fixes a while back and getting a bit confused on what exactly happens when the page is absorbed to fulfill the minimum size...

It does seem like Sashiko also notes this as a possible concern. WDYT? Does your reproducer for this issue also work when a minimum size is set (let's say, to 1?)

Let me look into this more!

Quoted 35 lines

Thanks again. I hope you have a great day!!!
Joshua

Quoted 32 lines
Fixes a833a693a490e ("mm: hugetlb: fix incorrect fallback for subpool")
---
 mm/hugetlb.c | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)
mm/hugetlb.c +2 -8
index ee5e99c1894b9..4093c1c0a4a1d 100644
@@ -2852,7 +2852,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 	struct hugepage_subpool *spool = subpool_vma(vma);
 	struct hstate *h = hstate_vma(vma);
 	struct folio *folio;
-	long retval, gbl_chg, gbl_reserve;
+	long retval, gbl_chg;
 	map_chg_state map_chg;
 	int ret, idx;
 	struct hugetlb_cgroup *h_cg = NULL;
@@ -3003,13 +3003,11 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 		hugetlb_cgroup_uncharge_cgroup_rsvd(idx, pages_per_huge_page(h),
 						    h_cg_rsvd);
 out_subpool_put:
-	/*
-	 * put page to subpool iff the quota of subpool's rsv_hpages is used
-	 * during hugepage_subpool_get_pages.
-	 */
-	if (map_chg && !gbl_chg) {
-		gbl_reserve = hugepage_subpool_put_pages(spool, 1);
-		hugetlb_acct_memory(h, -gbl_reserve);
+	if (map_chg) {
Quoted 8 lines

mm/hugetlb.c struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, · line 3011

Quoted 5 lines
mm/hugetlb.c +4 -0
+		long gbl_reserve = hugepage_subpool_put_pages(spool, 1);
+
+		if (!gbl_chg)
+			hugetlb_acct_memory(h, -gbl_reserve);
 	}

[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/

[3] https://lore.kernel.org/linux-mm/[email protected]/

#5 lore raw

Re: [PATCH v2 2/5] mm: hugetlb: Fix subpool usage leak on allocation failure

Quoted 41 lines · Ackerley Tng
Quoted 23 lines · Joshua Hahn

Hi Ackerley,

Thank you for this series. I really wanted to work on hugeTLB accounting fixes but never got the time to get to it. I'm very grateful that you are taking a look!!

Quoted 11 lines

From: Ackerley Tng <[email protected]>

When alloc_hugetlb_folio() fails early (e.g. buddy allocation failure or hugetlb cgroup charging failure) and gbl_chg == 1 (meaning a reservation was not used, but a global page was allocated instead), the subpool page acquired via hugepage_subpool_get_pages() must still be returned.

Currently, the error path out_subpool_put: only calls hugepage_subpool_put_pages() if !gbl_chg is true. If gbl_chg is 1, it skips it, permanently leaking the subpool's used_hpages counter.

Quoted 5 lines

With the earlier patch to always track used_hpages in the subpool, always call hugepage_subpool_put_pages() if map_chg is true to consistently restore the page to the subpool. Only call hugetlb_acct_memory() to adjust global reservations if gbl_chg == 0 since gbl_chg == 0 indicates a subpool (and global) reservation was used.

Quoted 7 lines

So I think that I've seen that this part of the accounting specifically is a bit suspicious. There have been two attempts in the past to fix this area [1] [2]. I think functionally they are quite similar to this fix, they just open-code the contents of the put_pages function inside the condition. I've Cc-ed the authors of those two patches in case they wanted to chime in.

Thanks for connecting us! I didn't realize this was already being worked on. Also adding Lance, who commented at [3].

Quoted 5 lines

I reference these fixes because I think they handle the minimum subpage case a bit differently. To be honest, I recall reading those fixes a while back and getting a bit confused on what exactly happens when the page is absorbed to fulfill the minimum size...

Quoted 7 lines
Quoted 4 lines

It does seem like Sashiko also notes this as a possible concern. WDYT? Does your reproducer for this issue also work when a minimum size is set (let's say, to 1?)

Let me look into this more!

I was looking at Sashiko's comment [4] and Sashiko is right that there could be a race. Specifically, at the time of hugepage_subpool_get_page(), I might be using a reservation, hence returning 1 (thread A), but at the time of hugepage_subpool_put_page(), some other thread (B) might already have restored a reservation to the subpool.

With used_hpages tracking within the subpool, we can rely on the return value of hugepage_subpool_put_page():

+ Return 1: Reservation should be returned elsewhere (pool has enough pages to satisfy minimum, or doesn't track minimums)
+ Return 0: Reservation does not need to be returned elsewhere

But calling hugepage_subpool_put_pages(1) doesn't say anything about whether the 1 (in this case, 1 doesn't always represent a page, since specifically on the error cases, the page wasn't allocated) consumed a reservation or not.

+ gbl_chg == 0 means a reservation was used, and
+ gbl_chg == 1 means a reservation *should be used*, but
+ gbl_chg == 1 does not necessarily mean a reservation should be used, since h->resv_huge_pages may not have been decremented yet. (See goto out_subpool_put and goto out_uncharge_cgroup_reservation in alloc_hugetlb_folio())

So, we need to track if h->resv_huge_pages-- happened (with some local variable like reservation_consumed)

| reservation_consumed | hugepage_subpool_put_pages() return value |
h->resv_huge_pages++ |
|----------------------|-------------------------------------------|----------------------|
| 1                    | 1                                         |
Yes                  |
| 1                    | 0                                         |
No                   |
| 0                    | 1                                         |
No                   |
| 0                    | 0                                         |
No                   |

TLDR: replace !gbl_chg with reservation_consumed?

    <<== reservation_consumed = true

	if (!gbl_chg) {
		folio_set_hugetlb_restore_reserve(folio);
		h->resv_huge_pages--;
        <<== reservation_consumed = true
	}

	if (map_chg) {
		long gbl_reserve = hugepage_subpool_put_pages(spool, 1);

		if (!gbl_chg)  <<== this should be reservation_consumed
			hugetlb_acct_memory(h, -gbl_reserve);
	}

I believe this depends on always tracking used_hpages in subpools, in order to know for sure, just based on a number (with no page information), if the page is supposed to use a reservation.

Perhaps related: does it make sense to flipping the reservation tracking to instead track an "available" page count? This is basically the mathematical opposite of rsvd_huge_pages, as in,

available_pages = free_huge_pages - resv_huge_pages

This way, there's no need to synchronize the number of reserved pages in both the subpool and global hstate.

It's quite confusing now, that reserved_pages-- can be either "used a reservation" or "unreserve". If we track the opposite, available_pages-- will definitely mean the page's availability is removed from the global hstate and transferred to the subpool, and allocating page doesn't involve updating both the subpool and hstate.

One downside I can think of is that creating hugepages (surplus, or promote/demote, or hotplug, or runtime echo 1 > nr_hugepages) would need to update the global hstate's available_pages count, but I imagine that would be less frequent than allocating huge pages?

Quoted 7 lines
Quoted 6 lines

[...snip...]

[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/

[3] https://lore.kernel.org/linux-mm/[email protected]/

[4] https://sashiko.dev/#/patchset/20260708-hugetlb-alloc-failure-fixes-v2-0-c7f27cbb462b%40google.com?part=2,

#6 lore raw

Re: [PATCH v2 2/5] mm: hugetlb: Fix subpool usage leak on allocation failure

Quoted 43 lines · On Thu, 9 Jul 2026 15:15:50 -0700 Ackerley Tng <[email protected]> wrote:
Quoted 41 lines · Ackerley Tng
Quoted 23 lines · Joshua Hahn

Hi Ackerley,

Thank you for this series. I really wanted to work on hugeTLB accounting fixes but never got the time to get to it. I'm very grateful that you are taking a look!!

Quoted 11 lines

From: Ackerley Tng <[email protected]>

When alloc_hugetlb_folio() fails early (e.g. buddy allocation failure or hugetlb cgroup charging failure) and gbl_chg == 1 (meaning a reservation was not used, but a global page was allocated instead), the subpool page acquired via hugepage_subpool_get_pages() must still be returned.

Currently, the error path out_subpool_put: only calls hugepage_subpool_put_pages() if !gbl_chg is true. If gbl_chg is 1, it skips it, permanently leaking the subpool's used_hpages counter.

Quoted 5 lines

With the earlier patch to always track used_hpages in the subpool, always call hugepage_subpool_put_pages() if map_chg is true to consistently restore the page to the subpool. Only call hugetlb_acct_memory() to adjust global reservations if gbl_chg == 0 since gbl_chg == 0 indicates a subpool (and global) reservation was used.

Quoted 7 lines

So I think that I've seen that this part of the accounting specifically is a bit suspicious. There have been two attempts in the past to fix this area [1] [2]. I think functionally they are quite similar to this fix, they just open-code the contents of the put_pages function inside the condition. I've Cc-ed the authors of those two patches in case they wanted to chime in.

Thanks for connecting us! I didn't realize this was already being worked on. Also adding Lance, who commented at [3].

Quoted 5 lines

I reference these fixes because I think they handle the minimum subpage case a bit differently. To be honest, I recall reading those fixes a while back and getting a bit confused on what exactly happens when the page is absorbed to fulfill the minimum size...

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

It does seem like Sashiko also notes this as a possible concern. WDYT? Does your reproducer for this issue also work when a minimum size is set (let's say, to 1?)

Let me look into this more!

Hi Ackerley, sorry that the reply took a while : -(

Quoted 18 lines

I was looking at Sashiko's comment [4] and Sashiko is right that there could be a race. Specifically, at the time of hugepage_subpool_get_page(), I might be using a reservation, hence returning 1 (thread A), but at the time of hugepage_subpool_put_page(), some other thread (B) might already have restored a reservation to the subpool.

With used_hpages tracking within the subpool, we can rely on the return value of hugepage_subpool_put_page():

+ Return 1: Reservation should be returned elsewhere (pool has enough pages to satisfy minimum, or doesn't track minimums)
+ Return 0: Reservation does not need to be returned elsewhere

But calling hugepage_subpool_put_pages(1) doesn't say anything about whether the 1 (in this case, 1 doesn't always represent a page, since specifically on the error cases, the page wasn't allocated) consumed a reservation or not.

Quoted 6 lines

+ gbl_chg == 0 means a reservation was used, and
+ gbl_chg == 1 means a reservation *should be used*, but
+ gbl_chg == 1 does not necessarily mean a reservation should be used, since h->resv_huge_pages may not have been decremented yet. (See goto out_subpool_put and goto out_uncharge_cgroup_reservation in alloc_hugetlb_folio())

Yeah... this kind of tripped me up.

Quoted 15 lines

So, we need to track if h->resv_huge_pages-- happened (with some local variable like reservation_consumed)

| reservation_consumed | hugepage_subpool_put_pages() return value |
h->resv_huge_pages++ |
|----------------------|-------------------------------------------|----------------------|
| 1                    | 1                                         |
Yes                  |
| 1                    | 0                                         |
No                   |
| 0                    | 1                                         |
No                   |
| 0                    | 0                                         |
No                   |
Quoted 1 line

TLDR: replace !gbl_chg with reservation_consumed?

This seems like quite a reasonable approach. It's new information that we can't derive from the existing flags we have.

Quoted 26 lines
    <<== reservation_consumed = true

	if (!gbl_chg) {
		folio_set_hugetlb_restore_reserve(folio);
		h->resv_huge_pages--;
        <<== reservation_consumed = true
	}

	if (map_chg) {
		long gbl_reserve = hugepage_subpool_put_pages(spool, 1);

		if (!gbl_chg)  <<== this should be reservation_consumed
			hugetlb_acct_memory(h, -gbl_reserve);
	}

I believe this depends on always tracking used_hpages in subpools, in order to know for sure, just based on a number (with no page information), if the page is supposed to use a reservation.

Perhaps related: does it make sense to flipping the reservation tracking to instead track an "available" page count? This is basically the mathematical opposite of rsvd_huge_pages, as in,

available_pages = free_huge_pages - resv_huge_pages

Quoted 8 lines

This way, there's no need to synchronize the number of reserved pages in both the subpool and global hstate.

It's quite confusing now, that reserved_pages-- can be either "used a reservation" or "unreserve". If we track the opposite, available_pages-- will definitely mean the page's availability is removed from the global hstate and transferred to the subpool, and allocating page doesn't involve updating both the subpool and hstate.

I think I like the semantic change, since it does make the code a lot more obvious on why and where we are making the accounting changes. One fear that I have is that this churn could introduce some new subtle accounting errors, since we have been fixing this area for quite some time now. Let's audit very carefully to make sure that there are no new accounting errors that will be introduced : -)

Quoted 4 lines

One downside I can think of is that creating hugepages (surplus, or promote/demote, or hotplug, or runtime echo 1 > nr_hugepages) would need to update the global hstate's available_pages count, but I imagine that would be less frequent than allocating huge pages?

Yup I think so. You might be more familiar with the new usecase you are adding as part of the guest_memfd, but I have not really seen a usecase for hugeTLB that would require userspace to do many hugeTLB allocations (that THP wouldn't be able to serve). Having this accuracy seems to be much more important than the rare contention we will see.

Thanks again for the fixes and ideas Ackerley! Looking forward to your new approach : -)
Joshua

Quoted 8 lines
Quoted 7 lines
Quoted 6 lines

[...snip...]

[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/

[3] https://lore.kernel.org/linux-mm/[email protected]/

[4] https://sashiko.dev/#/patchset/20260708-hugetlb-alloc-failure-fixes-v2-0-c7f27cbb462b%40google.com?part=2,

#7 lore raw

PATCH v2 3/5 mm: hugetlb: Fix folio refcount mismatch on memcg charge failure

In reply to Ackerley Tng

When mem_cgroup_charge_hugetlb(folio, gfp) returns -ENOMEM, the folio has its refcount set to 1 via folio_ref_unfreeze(folio, 1).

The error path calls free_huge_folio(folio) directly, which expects a refcount of 0. Hence, VM_BUG_ON_FOLIO(folio_ref_count(folio), folio) is triggered.

Even with CONFIG_DEBUG_VM disabled, returning a folio with refcount 1 to the freelist can corrupt allocator state later.

Use folio_put(folio) instead of free_huge_folio(folio) to properly drop the reference before freeing it.

Fixes 991135774c0e0 ("memcg/hugetlb: introduce mem_cgroup_charge_hugetlb")
---
 mm/hugetlb.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/hugetlb.c +1 -1
index 4093c1c0a4a1d..1f3f4b964b153 100644
@@ -2990,7 +2990,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 	lruvec_stat_mod_folio(folio, NR_HUGETLB, pages_per_huge_page(h));
 
 	if (ret == -ENOMEM) {
-		free_huge_folio(folio);
+		folio_put(folio);
 		return ERR_PTR(-ENOMEM);
 	}
 
Signature · 1 line
2.55.0.795.g602f6c329a-goog
Tested-by Reviewed-by on v2 3/5
#8 lore raw

Re: [PATCH v2 3/5] mm: hugetlb: Fix folio refcount mismatch on memcg charge failure

On Wed, 08 Jul 2026 15:12:51 -0700 Ackerley Tng via B4 Relay <[email protected]> wrote:

Hi Ackerley,

Thanks again for the fix. Thank you also for adding the reproducers, they made testing this really easy for me! : -D

Quoted 8 lines

From: Ackerley Tng <[email protected]>

When mem_cgroup_charge_hugetlb(folio, gfp) returns -ENOMEM, the folio has its refcount set to 1 via folio_ref_unfreeze(folio, 1).

The error path calls free_huge_folio(folio) directly, which expects a refcount of 0. Hence, VM_BUG_ON_FOLIO(folio_ref_count(folio), folio) is triggered.

So yeah, I built mm-new (with your hugetlb open code series) and ran reproducer #3, and got the following output:

...

[   23.855813] Call Trace:
[   23.855838]  <TASK>
[   23.855877]  hugetlb_alloc_folio+0x190/0x360
[   23.855930]  alloc_hugetlb_folio+0xf9/0x310
[   23.855972]  hugetlb_no_page+0x590/0xa00
[   23.856016]  hugetlb_fault+0x194/0x770
[   23.856060]  handle_mm_fault+0x29b/0x2c0
[   23.856103]  do_user_addr_fault+0x208/0x6e0
[   23.856146]  exc_page_fault+0x67/0x140
[   23.856191]  asm_exc_page_fault+0x22/0x30
[   23.856234] RIP: 0033:0x401734

...

Running it with your fix, I don't get a kernel crash. So this change looks good to me, please feel free to add:

But.....

It seems like fixing this bug actually surfaces a different bug. I don't see a kernel crash anymore, but my kernel gets stuck in a different loop:

...

[   42.526726] pagefault_out_of_memory: 120780 callbacks suppressed
[   42.526732] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[   42.526955] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[   42.527084] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[   42.527192] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[   42.527288] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[   42.527382] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[   42.527476] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF

...

mem_cgroup_charge_hugetlb() fails
--> hugetlb_alloc_folio() returns ERR_PTR(-ENOMEM) --> propagated to alloc_hugetlb_folio()
--> hugetlb_no_page converts it to vmf_error(PTR_ERROR(folio)) --> propagates up to the pagefault handler..

But there's no OOM handler to resolve, I think. So it just keeps retrying the fault and cycling over and over and over again... I think we need to be returning ENOSPC instead of ENOMEM like the other failure paths in alloc_hugetlb_folio (or hugetlb_alloc_folio... I think we should change this name, it's a bit confusing).

mm/hugetlb.c +1 -1
index 288535838a48b..05214b1cd491a 100644
@@ -2903,7 +2903,7 @@ struct folio *hugetlb_alloc_folio(struct hstate *h,
                 * were committed to the folio and freeing the folio
                 * would have cleared those up.
                 */
-               return ERR_PTR(ret);
+               return ERR_PTR(-ENOSPC);
        }
 
        return folio;

This gives me the following result from your reproducer, which I think is the expected behavior:

attempting to remount cgroup2 with memory_hugetlb_accounting... Successfully enabled memory_hugetlb_accounting
Child: Attempting to allocate and touch 2MB hugepage...
Child: mmap succeeded at 0x7f0023600000, touching it now (should trigger fault)...
Parent: Child exited. Cleaning up.
Parent: Child killed by signal 7 (Bus error)
Parent: Child got SIGBUS as expected (if kernel didn't crash).

What do you think? Happy to send it out as a separate fix or feel free to fold it into your series / fix, whatever you prefer.

Thanks Ackerley!

Quoted 29 lines

Even with CONFIG_DEBUG_VM disabled, returning a folio with refcount 1 to the freelist can corrupt allocator state later.

Use folio_put(folio) instead of free_huge_folio(folio) to properly drop the reference before freeing it.

Fixes 991135774c0e0 ("memcg/hugetlb: introduce mem_cgroup_charge_hugetlb")
---
 mm/hugetlb.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/hugetlb.c +1 -1
index 4093c1c0a4a1d..1f3f4b964b153 100644
@@ -2990,7 +2990,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 	lruvec_stat_mod_folio(folio, NR_HUGETLB, pages_per_huge_page(h));
 
 	if (ret == -ENOMEM) {
-		free_huge_folio(folio);
+		folio_put(folio);
 		return ERR_PTR(-ENOMEM);
 	}
 
Signature · 2 lines
-- 
2.55.0.795.g602f6c329a-goog
#9 lore raw

Re: [PATCH v2 3/5] mm: hugetlb: Fix folio refcount mismatch on memcg charge failure

Quoted 73 lines · On Thu, 9 Jul 2026 11:54:55 -0700 Joshua Hahn <[email protected]> wrote:

On Wed, 08 Jul 2026 15:12:51 -0700 Ackerley Tng via B4 Relay <[email protected]> wrote:

Hi Ackerley,

Thanks again for the fix. Thank you also for adding the reproducers, they made testing this really easy for me! : -D

Quoted 8 lines

From: Ackerley Tng <[email protected]>

When mem_cgroup_charge_hugetlb(folio, gfp) returns -ENOMEM, the folio has its refcount set to 1 via folio_ref_unfreeze(folio, 1).

The error path calls free_huge_folio(folio) directly, which expects a refcount of 0. Hence, VM_BUG_ON_FOLIO(folio_ref_count(folio), folio) is triggered.

So yeah, I built mm-new (with your hugetlb open code series) and ran reproducer #3, and got the following output:

...

[   23.855813] Call Trace:
[   23.855838]  <TASK>
[   23.855877]  hugetlb_alloc_folio+0x190/0x360
[   23.855930]  alloc_hugetlb_folio+0xf9/0x310
[   23.855972]  hugetlb_no_page+0x590/0xa00
[   23.856016]  hugetlb_fault+0x194/0x770
[   23.856060]  handle_mm_fault+0x29b/0x2c0
[   23.856103]  do_user_addr_fault+0x208/0x6e0
[   23.856146]  exc_page_fault+0x67/0x140
[   23.856191]  asm_exc_page_fault+0x22/0x30
[   23.856234] RIP: 0033:0x401734

...

Running it with your fix, I don't get a kernel crash. So this change looks good to me, please feel free to add:

But.....

It seems like fixing this bug actually surfaces a different bug. I don't see a kernel crash anymore, but my kernel gets stuck in a different loop:

...

[   42.526726] pagefault_out_of_memory: 120780 callbacks suppressed
[   42.526732] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[   42.526955] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[   42.527084] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[   42.527192] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[   42.527288] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[   42.527382] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[   42.527476] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF

...

mem_cgroup_charge_hugetlb() fails
--> hugetlb_alloc_folio() returns ERR_PTR(-ENOMEM) --> propagated to alloc_hugetlb_folio()
--> hugetlb_no_page converts it to vmf_error(PTR_ERROR(folio)) --> propagates up to the pagefault handler..

But there's no OOM handler to resolve, I think. So it just keeps retrying the fault and cycling over and over and over again... I think we need to be returning ENOSPC instead of ENOMEM like the other failure paths in alloc_hugetlb_folio (or hugetlb_alloc_folio... I think we should change this name, it's a bit confusing).

mm/hugetlb.c +0 -0
index 288535838a48b..05214b1cd491a 100644

mm/hugetlb.c struct folio *hugetlb_alloc_folio(struct hstate *h, · line 2909

Quoted 8 lines
mm/hugetlb.c +1 -1
@@ -2903,7 +2903,7 @@ struct folio *hugetlb_alloc_folio(struct hstate *h,
                 * were committed to the folio and freeing the folio
                 * would have cleared those up.
                 */
-               return ERR_PTR(ret);
+               return ERR_PTR(-ENOSPC);
        }
 
        return folio;

Ah, immediately as I sent this out and went to 4/5 to review it, I see that this is the exact change you have... oops, please disregard the message : -) I do think we should 3/5 and 4/5 together for bisectability purposes.

Thanks again. Sorry for the noise!
Joshua

#10 lore raw

PATCH v2 4/5 mm: hugetlb: Return -ENOSPC on memcg charge failure

In reply to Ackerley Tng

When mem_cgroup_charge_hugetlb() fails with -ENOMEM, alloc_hugetlb_folio() currently propagates this error. This results in the page fault handler returning VM_FAULT_OOM.

Because HugeTLB allocations are high-order and use __GFP_RETRY_MAYFAIL, they bypass the OOM killer. Returning VM_FAULT_OOM to the #PF handler without triggering the OOM killer (or having it make progress) leads to an infinite loop of retrying the fault.

Avoid this loop by returning -ENOSPC when charging fails, which maps to VM_FAULT_SIGBUS, terminating the process cleanly.

Make mem_cgroup_charge_hugetlb() fault handling use a common error handling path, the same handling used for hugetlb_cgroup_uncharge_cgroup{,_rsvd}(), which also don't trigger the OOM killer and hence opt to terminate the process with a SIGBUS.

Fixes 991135774c0e0 ("memcg/hugetlb: introduce mem_cgroup_charge_hugetlb")
---
 mm/hugetlb.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
mm/hugetlb.c +12 -1
index 1f3f4b964b153..3e1d99f03c70e 100644
@@ -2991,7 +2991,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 
 	if (ret == -ENOMEM) {
 		folio_put(folio);
-		return ERR_PTR(-ENOMEM);
+		goto err;
 	}
 
 	return folio;
@@ -3014,6 +3014,17 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 out_end_reservation:
 	if (map_chg != MAP_CHG_ENFORCED)
 		vma_end_reservation(h, vma, addr);
+err:
+	/*
+	 * Return -ENOSPC when this function fails to allocate or
+	 * charge a huge page. If a standard (PAGE_SIZE) page
+	 * allocation fails, the OOM killer is given a chance to run,
+	 * which may resolve the failure on retry. However, for
+	 * HugeTLB allocations, the OOM killer is not triggered.
+	 * Returning -ENOMEM (or anything resulting in VM_FAULT_OOM)
+	 * would leak to the #PF handler, causing it to loop
+	 * indefinitely retrying the fault.
+	 */
 	return ERR_PTR(-ENOSPC);
 }
 
Signature · 1 line
2.55.0.795.g602f6c329a-goog
Tested-by Reviewed-by on v2 4/5
#11 lore raw

Re: [PATCH v2 4/5] mm: hugetlb: Return -ENOSPC on memcg charge failure

Quoted 14 lines · On Wed, 08 Jul 2026 15:12:52 -0700 Ackerley Tng via B4 Relay…

From: Ackerley Tng <[email protected]>

When mem_cgroup_charge_hugetlb() fails with -ENOMEM, alloc_hugetlb_folio() currently propagates this error. This results in the page fault handler returning VM_FAULT_OOM.

Because HugeTLB allocations are high-order and use __GFP_RETRY_MAYFAIL, they bypass the OOM killer. Returning VM_FAULT_OOM to the #PF handler without triggering the OOM killer (or having it make progress) leads to an infinite loop of retrying the fault.

Avoid this loop by returning -ENOSPC when charging fails, which maps to VM_FAULT_SIGBUS, terminating the process cleanly.

Quoted 7 lines

Make mem_cgroup_charge_hugetlb() fault handling use a common error handling path, the same handling used for hugetlb_cgroup_uncharge_cgroup{,_rsvd}(), which also don't trigger the OOM killer and hence opt to terminate the process with a SIGBUS.

Fixes 991135774c0e0 ("memcg/hugetlb: introduce mem_cgroup_charge_hugetlb")
Quoted 20 lines
---
 mm/hugetlb.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
mm/hugetlb.c +1 -1
index 1f3f4b964b153..3e1d99f03c70e 100644
@@ -2991,7 +2991,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 
 	if (ret == -ENOMEM) {
 		folio_put(folio);
-		return ERR_PTR(-ENOMEM);
+		goto err;
 	}
 
 	return folio;

mm/hugetlb.c struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, · line 3016

Quoted 3 lines
mm/hugetlb.c +0 -0
@@ -3014,6 +3014,17 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 out_end_reservation:
 	if (map_chg != MAP_CHG_ENFORCED)
 		vma_end_reservation(h, vma, addr);

NIT: is there a reason why the comment block below doesn't go all the way up to 80 columns? I also think that we can achieve the same effect with a shorter comment block : -) I think the essence is to answer "Why -ENOSPC and not -ENOMEM?"

mm/hugetlb.c struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, · line 3029

Quoted 13 lines
mm/hugetlb.c +11 -0
+err:
+	/*
+	 * Return -ENOSPC when this function fails to allocate or
+	 * charge a huge page. If a standard (PAGE_SIZE) page
+	 * allocation fails, the OOM killer is given a chance to run,
+	 * which may resolve the failure on retry. However, for
+	 * HugeTLB allocations, the OOM killer is not triggered.
+	 * Returning -ENOMEM (or anything resulting in VM_FAULT_OOM)
+	 * would leak to the #PF handler, causing it to loop
+	 * indefinitely retrying the fault.
+	 */
 	return ERR_PTR(-ENOSPC);
 }

But this fix looks good otherwise and I don't think this a big deal : -)

Thanks Ackerley!
Joshua

#12 lore raw

PATCH v2 5/5 mm: hugetlb: Move memcg charge earlier to prevent reservation leak

In reply to Ackerley Tng

When mem_cgroup_charge_hugetlb() fails, alloc_hugetlb_folio() clears the charges and frees the folio. The reservation committed via vma_commit_reservation() was not undone, leaving the reservation map in an inconsistent state, causing resv_huge_pages to leak when the process exited.

Fix this by moving the mem_cgroup_charge_hugetlb() call earlier in alloc_hugetlb_folio(), before vma_commit_reservation() is called.

If the charge fails now, the reservation is not yet committed. Jump to out_subpool_put, which will then call vma_end_reservation() to abort the reservation and keep the reservation map and resv_huge_pages counter consistent.

Fixes 991135774c0e0 ("memcg/hugetlb: introduce mem_cgroup_charge_hugetlb")
---
 mm/hugetlb.c | 34 ++++++++++++++++++++--------------
 1 file changed, 20 insertions(+), 14 deletions(-)
mm/hugetlb.c +20 -14
index 3e1d99f03c70e..e000af6f28585 100644
@@ -2954,6 +2954,25 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 
 	spin_unlock_irq(&hugetlb_lock);
 
+	ret = mem_cgroup_charge_hugetlb(folio, gfp);
+	/*
+	 * Unconditionally increment NR_HUGETLB here. If it turns out that
+	 * mem_cgroup_charge_hugetlb failed, then immediately free the page and
+	 * decrement NR_HUGETLB.
+	 */
+	lruvec_stat_mod_folio(folio, NR_HUGETLB, pages_per_huge_page(h));
+
+	if (ret == -ENOMEM) {
+		folio_put(folio);
+		/*
+		 * Charges to hugetlb_cgroup for usage and
+		 * reservations were already committed, so folio_put()
+		 * would have uncharged those. Go straight to undoing
+		 * subpool charges.
+		 */
+		goto out_subpool_put;
+	}
+
 	hugetlb_set_folio_subpool(folio, spool);
 
 	if (map_chg != MAP_CHG_ENFORCED) {
@@ -2981,19 +3000,6 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 		}
 	}
 
-	ret = mem_cgroup_charge_hugetlb(folio, gfp);
-	/*
-	 * Unconditionally increment NR_HUGETLB here. If it turns out that
-	 * mem_cgroup_charge_hugetlb failed, then immediately free the page and
-	 * decrement NR_HUGETLB.
-	 */
-	lruvec_stat_mod_folio(folio, NR_HUGETLB, pages_per_huge_page(h));
-
-	if (ret == -ENOMEM) {
-		folio_put(folio);
-		goto err;
-	}
-
 	return folio;
 
 out_uncharge_cgroup:
@@ -3014,7 +3020,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 out_end_reservation:
 	if (map_chg != MAP_CHG_ENFORCED)
 		vma_end_reservation(h, vma, addr);
-err:
+
 	/*
 	 * Return -ENOSPC when this function fails to allocate or
 	 * charge a huge page. If a standard (PAGE_SIZE) page
Signature · 1 line
2.55.0.795.g602f6c329a-goog
#13 lore raw

cover Reproducers for hugetlb allocation failure issues

In reply to Ackerley Tng

Please see separate patches for details!

Ackerley Tng (3):
Reproducer for false restoration on shared HugeTLB mappings Reproducer for subpool usage leak
Reproducer for allocation failure due to cgroup v2 memory limits

 cgroup_v2_allocation_failure.c | 160 +++++++++++++++++++++++++++++++++
 subpool_leak_max_size.sh       |  71 +++++++++++++++
 subpool_shared_leak.c          |  29 ++++++
 subpool_shared_leak.sh         |  86 ++++++++++++++++++
 4 files changed, 346 insertions(+)
 create mode 100644 cgroup_v2_allocation_failure.c
 create mode 100755 subpool_leak_max_size.sh
 create mode 100644 subpool_shared_leak.c
 create mode 100755 subpool_shared_leak.sh

--
2.55.0.795.g602f6c329a-goog

#14 lore raw

PATCH 1/3 Reproducer for false restoration on shared HugeTLB mappings

(This reproducer was hacked up and not meant to be merged.)

hugetlb_unreserve_pages() unconditionally returns reservations to the subpool (via hugepage_subpool_put_pages()). This means that regardless of whether a subpool reservation was actually used, the reservation is processed by the subpool structure.

To create a false restoration, the reproducer performs these steps:

1. Mount with min_size=2M (1 page). Global resv_hugepages becomes 1.
2. The program maps 4MB (2 pages) shared (which also grows the file to 4MB). Global resv_hugepages becomes 2 (1 from the mount, 1 new global reservation).
3. The program populates only the first page. Global resv_hugepages decrements to 1 (reservation consumed by allocation).
4. The program exits (closing VMAs/fds). For shared mappings, reservations are associated with the file inode, so they remain active. Global resv_hugepages remains 1.
5. The script truncates the file to 2MB (truncate -s 2M).
+ This synchronously triggers hugetlb_unreserve_pages() to release the
reservation of the truncated range (the unallocated 2nd page).
+ It calls hugepage_subpool_put_pages(spool, 1).
+ On Vanilla Kernel (Buggy):
+ used_hpages is 0 (not tracked).
+ used_hpages (0) < min_hpages (1) is TRUE.
+ The subpool incorrectly restores the reservation (spool->rsv_hpages
becomes 1), even though Page 0 is still allocated and satisfies the mount's minimum guarantee.
+ hugepage_subpool_put_pages() returns 0, skipping
hugetlb_acct_memory(h, -1).
+ Result: Global resv_hugepages remains stuck at 1 (Leak).
+ On Fixed Kernel:
+ used_hpages is tracked and is initially 2.
+ hugepage_subpool_put_pages(1) decrements used_hpages to 1.
+ used_hpages (1) < min_hpages (1) is FALSE.
+ The subpool does not restore the reservation.
+ hugepage_subpool_put_pages() returns 1.
+ hugetlb_acct_memory(h, -1) is called.
+ Result: Global resv_hugepages decrements to 0 (No leak).

When the filesystem is unmounted, hugetlbfs_put_super drops the subpool reference. Since the filesystem is being unmounted, the reference count drops to 0, triggering unlock_or_release_subpool.

Inside unlock_or_release_subpool, the kernel checks if the subpool is free using subpool_is_free.
+ On the buggy kernel, subpool_is_free checks if spool->rsv_hpages is equal to spool->min_hpages. Because of the phantom reservation, spool->rsv_hpages was restored to 1. Since min_hpages is 1, the check (1 == 1) returns true.
+ Since the subpool is considered free, the kernel releases the initial mount-time reservation by calling hugetlb_acct_memory to decrement resv_huge_pages by spool->min_hpages (which is 1).
+ This decrement reduces resv_huge_pages from 1 (the leaked state) to 0.

As a result, the leaked reservation is cleaned up during unmount and does not persist afterward.

---
 subpool_shared_leak.c  | 29 ++++++++++++++
 subpool_shared_leak.sh | 86 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 115 insertions(+)
 create mode 100644 subpool_shared_leak.c
 create mode 100755 subpool_shared_leak.sh
subpool_shared_leak.c new file +29 -0
new file mode 100644
index 0000000000000..5811e18d7f8be
@@ -0,0 +1,29 @@
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <sys/mman.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+
+#define HPAGE_SIZE (2 * 1024 * 1024)
+
+int main(int argc, char **argv) {
+    if (argc < 2) {
+        fprintf(stderr, "Usage: %s <file_path>\n", argv[0]);
+        return 1;
+    }
+    const char *file_path = argv[1];
+
+    int fd = open(file_path, O_CREAT | O_RDWR, 0666);
+    if (fd < 0) { perror("open"); return 1; }
+
+    void *addr = mmap(NULL, 2 * HPAGE_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
+    if (addr == MAP_FAILED) { perror("mmap"); close(fd); return 1; }
+
+    *(volatile char *)addr = 1; // Allocate 1st page only. 2nd page remains unallocated (but reserved).
+
+    munmap(addr, 2 * HPAGE_SIZE);
+    close(fd);
+
+    return 0;
+}
subpool_shared_leak.sh new file +86 -0

86 lines not on this page · raw message

Signature · 1 line
2.55.0.795.g602f6c329a-goog
#15 lore raw

PATCH 2/3 Reproducer for subpool usage leak

In reply to Ackerley Tng

(This reproducer was hacked up and not meant to be merged.)

The kernel leaks subpool usage and the subpool structure itself if a HugeTLBfs mount specifying size (which sets max_hpages on the subpool) is created.

subpool_leak_max_size.sh reproduces this with the following steps:

1. Create mount, specifying size=2M (1 page). This sets max_hpages = 1 on the subpool, but does not reserve any pages.
2. Set nr_hugepages = 0 and nr_overcommit_hugepages = 0 so that physical allocations will fail.
3. Run fallocate -l 2M on a file in the mount.
+ This calls hugetlbfs_fallocate, which attempts to allocate a page by calling alloc_hugetlb_folio.
+ alloc_hugetlb_folio calls hugepage_subpool_get_pages to track the allocation against the subpool limit. This increments used_hpages to 1.
+ Physical allocation fails because nr_hugepages is 0.
+ Before patch (Buggy):
+ The error path in alloc_hugetlb_folio sees gbl_chg is 1 (indicating we tried to allocate a global page) and incorrectly skips calling hugepage_subpool_put_pages.
+ fallocate fails and returns to userspace, but the subpool used_hpages counter remains leaked at 1.
+ After patch:
+ The error path always calls hugepage_subpool_put_pages if map_chg is true, restoring used_hpages to 0.

4. Unmount the filesystem.
+ During unmount, the kernel calls unlock_or_release_subpool to clean up the subpool.
+ It checks if the subpool is free using subpool_is_free, which returns whether used_hpages is 0.
+ Before patch (Buggy):
+ Since used_hpages leaked and is 1, subpool_is_free returns false.
+ The kernel skips freeing the subpool structure, leaking the hugepage_subpool structure in kernel memory.
+ After patch:
+ Since used_hpages is 0, subpool_is_free returns true, and the subpool structure is correctly freed.

---
 subpool_leak_max_size.sh | 71 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 71 insertions(+)
 create mode 100755 subpool_leak_max_size.sh
subpool_leak_max_size.sh new file +71 -0
new file mode 100755
index 0000000000000..bfafa1ba074ea
@@ -0,0 +1,71 @@
+#!/bin/bash
+
+if [ "$EUID" -ne 0 ]; then
+    echo "Please run as root"
+    exit 1
+fi
+
+MNT_PATH="/tmp/mnt_hugetlb"
+FILE_PATH="$MNT_PATH/test_file"
+
+# Save original values
+orig_nr=$(cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages)
+orig_overcommit=$(cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_overcommit_hugepages)
+
+cleanup() {
+    echo "Cleaning up..."
+    rm -f "$FILE_PATH"
+    umount "$MNT_PATH" 2>/dev/null
+    rmdir "$MNT_PATH" 2>/dev/null
+    echo "$orig_nr" > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
+    echo "$orig_overcommit" > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_overcommit_hugepages
+    echo "Cleanup done."
+}
+trap cleanup EXIT
+
+# 1. Mount hugetlbfs with size=2M (1 page)
+mkdir -p "$MNT_PATH"
+if ! mount -t hugetlbfs -o size=2M none "$MNT_PATH"; then
+    echo "Failed to mount hugetlbfs"
+    exit 1
+fi
+
+# 2. Set nr_hugepages to 0, overcommit to 0
+echo 0 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
+echo 0 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_overcommit_hugepages
+
+# Check subpool usage before running
+read total free < <(stat -f -c "%b %f" "$MNT_PATH")
+used_before=$((total - free))
+echo "Before test - Subpool total blocks: $total"
+echo "Before test - Subpool free blocks:  $free"
+echo "Before test - Subpool used blocks:  $used_before"
+if [ "$used_before" -ne 0 ]; then
+    echo "ERROR: Subpool is not clean before test starts!"
+    exit 1
+fi
+
+# Run fallocate (expecting failure)
+echo "Running fallocate (expecting failure)..."
+if fallocate -l 2M "$FILE_PATH" 2>/dev/null; then
+    echo "ERROR: fallocate succeeded but should have failed (nr_hugepages is 0)"
+    exit 1
+fi
+
+# Check subpool usage via statfs
+# %b: Total blocks
+# %f: Free blocks
+read total free < <(stat -f -c "%b %f" "$MNT_PATH")
+used=$((total - free))
+
+echo "Subpool total blocks: $total"
+echo "Subpool free blocks:  $free"
+echo "Subpool used blocks (leaked if > 0): $used"
+
+if [ "$used" -gt 0 ]; then
+    echo "RESULT: LEAK DETECTED (FAIL)"
+    exit 1
+else
+    echo "RESULT: NO LEAK (PASS)"
+    exit 0
+fi
Signature · 1 line
2.55.0.795.g602f6c329a-goog
#16 lore raw

PATCH 3/3 Reproducer for allocation failure due to cgroup v2 memory limits

In reply to Ackerley Tng

(This reproducer was hacked up and not meant to be merged.)

cgroup_v2_allocation_failure.c triggers HugeTLB allocation failure by exploiting cgroup v2 memory limits. This allows testing the error paths in the kernel when memory control charging fails, even when physical huge pages are available.

The program performs the following steps to trigger the failure:

1. Enable hugetlb accounting in cgroup v2.
+ The program checks if memory_hugetlb_accounting is enabled in the cgroup2 mount options. If not, it remounts /sys/fs/cgroup with this option enabled. This ensures that HugeTLB allocations are charged against the cgroup memory limits.

2. Create a test cgroup and set limits.
+ The program creates a new cgroup subdirectory named test_reproducer under /sys/fs/cgroup.
+ It sets the memory.max limit of this cgroup to 1MB (which is less than the 2MB huge page size).

3. Fork a child process and move it to the test cgroup.
+ The program forks a child process.
+ The child process moves itself into the test_reproducer cgroup by writing its PID (using 0 for current process) to cgroup.procs in the test cgroup directory.

4. Attempt to allocate and touch a 2MB huge page.
+ The child process maps a 2MB anonymous huge page using mmap with MAP_PRIVATE, MAP_ANONYMOUS, and MAP_HUGETLB.
+ The child process writes to the mapped address, triggering a page fault.

5. Triggering the kernel bugs.
+ The page fault handler calls alloc_hugetlb_folio to allocate the huge page.
+ The allocation of the physical page from buddy allocator succeeds (assuming nr_hugepages is sufficient).
+ The kernel then attempts to charge this allocation to the child process's cgroup by calling mem_cgroup_charge_hugetlb.
+ Since the child's cgroup memory limit is 1MB and the page is 2MB, the charge fails and mem_cgroup_charge_hugetlb returns -ENOMEM.
+ This triggers the error path in alloc_hugetlb_folio where the bugs (folio refcount mismatch, infinite loop on ENOMEM, and reservation leaks) are handled.

---
 cgroup_v2_allocation_failure.c | 160 +++++++++++++++++++++++++++++++++
 1 file changed, 160 insertions(+)
 create mode 100644 cgroup_v2_allocation_failure.c
cgroup_v2_allocation_failure.c new file +160 -0
new file mode 100644
index 0000000000000..938cbf02ae6f7
@@ -0,0 +1,160 @@
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <fcntl.h>
+#include <sys/mman.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <sys/wait.h>
+#include <string.h>
+#include <errno.h>
+
+#define CGROUP_PATH "/sys/fs/cgroup"
+#define TEST_CGROUP "test_reproducer"
+#define TEST_CGROUP_PATH CGROUP_PATH "/" TEST_CGROUP
+
+void write_file(const char *path, const char *val) {
+    int fd = open(path, O_WRONLY);
+    if (fd < 0) {
+        fprintf(stderr, "Failed to open %s: %s\n", path, strerror(errno));
+        exit(1);
+    }
+    if (write(fd, val, strlen(val)) < 0) {
+        fprintf(stderr, "Failed to write %s to %s: %s\n", val, path, strerror(errno));
+        close(fd);
+        exit(1);
+    }
+    close(fd);
+}
+
+int is_hugetlb_accounting_enabled() {
+    FILE *fp = fopen("/proc/mounts", "r");
+    if (!fp) {
+        perror("fopen /proc/mounts");
+        return -1;
+    }
+
+    char line[1024];
+    int enabled = 0;
+    while (fgets(line, sizeof(line), fp)) {
+        char spec[256], file[256], type[256], opts[512];
+        if (sscanf(line, "%255s %255s %255s %511s", spec, file, type, opts) == 4) {
+            if (strcmp(file, CGROUP_PATH) == 0 && strcmp(type, "cgroup2") == 0) {
+                if (strstr(opts, "memory_hugetlb_accounting") != NULL) {
+                    enabled = 1;
+                }
+                break;
+            }
+        }
+    }
+    fclose(fp);
+    return enabled;
+}
+
+int enable_hugetlb_accounting() {
+    printf("Attempting to remount cgroup2 with memory_hugetlb_accounting...\n");
+    int ret = system("mount -o remount,memory_hugetlb_accounting " CGROUP_PATH);
+    if (ret != 0) {
+        fprintf(stderr, "Failed to remount: system() returned %d\n", ret);
+        return -1;
+    }
+    return 0;
+}
+
+int main() {
+    struct stat st;
+    if (stat(CGROUP_PATH, &st) != 0 || !S_ISDIR(st.st_mode)) {
+        fprintf(stderr, "cgroup v2 not mounted at %s\n", CGROUP_PATH);
+        return 1;
+    }
+
+    int enabled = is_hugetlb_accounting_enabled();
+    if (enabled < 0) {
+        return 1;
+    }
+    if (!enabled) {
+        if (enable_hugetlb_accounting() != 0) {
+            fprintf(stderr, "Could not enable memory_hugetlb_accounting\n");
+            return 1;
+        }
+        // Re-check
+        enabled = is_hugetlb_accounting_enabled();
+        if (enabled <= 0) {
+            fprintf(stderr, "Failed to enable memory_hugetlb_accounting (re-check failed)\n");
+            return 1;
+        }
+        printf("Successfully enabled memory_hugetlb_accounting\n");
+    } else {
+        printf("memory_hugetlb_accounting is already enabled\n");
+    }
+
+    // Enable memory controller in subtree
+    int fd = open(CGROUP_PATH "/cgroup.subtree_control", O_WRONLY);
+    if (fd >= 0) {
+        if (write(fd, "+memory", 7) < 0) {
+            // Might fail if already enabled or not supported, ignore for now
+        }
+        close(fd);
+    }
+
+    if (mkdir(TEST_CGROUP_PATH, 0755) != 0) {
+        if (errno != EEXIST) {
+            perror("mkdir test_reproducer");
+            return 1;
+        }
+    }
+
+    // Set memory limit to 1MB (less than 2MB hugepage)
+    write_file(TEST_CGROUP_PATH "/memory.max", "1M");
+
+    pid_t pid = fork();
+    if (pid < 0) {
+        perror("fork");
+        return 1;
+    }
+
+    if (pid == 0) {
+        // Child
+        // Move to cgroup
+        write_file(TEST_CGROUP_PATH "/cgroup.procs", "0");
+
+        printf("Child: Attempting to allocate and touch 2MB hugepage...\n");
+        // Allocate 2MB hugepage
+        size_t size = 2 * 1024 * 1024;
+        void *addr = mmap(NULL, size, PROT_READ | PROT_WRITE,
+                          MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, -1, 0);
+        if (addr == MAP_FAILED) {
+            perror("Child: mmap MAP_HUGETLB");
+            exit(1);
+        }
+
+        printf("Child: mmap succeeded at %p, touching it now (should trigger fault)...\n", addr);
+        // This should trigger the fault and call alloc_hugetlb_folio -> mem_cgroup_charge_hugetlb
+        // which should fail and trigger the bug.
+        *(volatile char *)addr = 1;
+
+        printf("Child: Successfully touched page (bug not triggered?).\n");
+        munmap(addr, size);
+        exit(0);
+    }
+
+    // Parent
+    int status;
+    waitpid(pid, &status, 0);
+
+    printf("Parent: Child exited. Cleaning up.\n");
+    rmdir(TEST_CGROUP_PATH);
+
+    if (WIFSIGNALED(status)) {
+        printf("Parent: Child killed by signal %d (%s)\n",
+               WTERMSIG(status), strsignal(WTERMSIG(status)));
+        if (WTERMSIG(status) == SIGBUS) {
+            printf("Parent: Child got SIGBUS as expected (if kernel didn't crash).\n");
+        }
+    } else if (WIFEXITED(status)) {
+        printf("Parent: Child exited with status %d\n", WEXITSTATUS(status));
+    }
+
+    return 0;
+}
Signature · 1 line
2.55.0.795.g602f6c329a-goog
#17 lore raw
In reply to Ackerley Tng

Ackerley Tng via B4 Relay <[email protected]> writes:

Looking at Sashiko's comments [1] again, it looks like there are even more issues. Seems like these bugs are quite deep into the weeds of accounting, would definitely like some advice from reviewers and maintainers!

1. Are the issues I've described the correct issues or did I get the wrong root causes? Is this the right direction?

2. Is there a good way to split up this series so at least some bugs are fixed? I kind of clumped them together because one fix revealed another bug and so on... Is it okay to merge some patches that we're sure fixes something and leave other bugs to be fixed separately? (If we get to a nice fix within a few iterations we don't have to split up the series)

[1] https://sashiko.dev/#/patchset/20260708-hugetlb-alloc-failure-fixes-v2-0-c7f27cbb462b%40google.com