Skip to contentSkip to conversation

Fix bugs on HugeTLB folio allocation failure paths

PATCH Superseded Ackerley Tng 1 participant 14 messages Jul 21

Subsystems (5)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v3 cover Fix bugs on HugeTLB folio allocation failure paths

This series fixes a few bugs on HugeTLB folio allocation failure paths.

Some of these issues were pointed out by Sashiko while I was working on [1], and others were revealed while fixing the original issues.

1. subpool usage leak on allocation failure (Sashiko pointed this out in [2])

When alloc_hugetlb_folio() fails early (e.g. buddy allocation failure or hugetlb cgroup charging failure) and gbl_chg == 1, the error path skips restoring the page to the subpool, leaking the subpool's used_hpages counter. Fix this by calling hugepage_subpool_put_pages() unconditionally if map_chg is true.

2. Folio refcount mismatch on memcg charge failure leading to VM_BUG_ON_FOLIO() (Sashiko pointed this out in [3])

The error path in alloc_hugetlb_folio() calls free_huge_folio() directly on a folio with a refcount of 1 (set via folio_ref_unfreeze() earlier). This triggers VM_BUG_ON_FOLIO(folio_ref_count(folio), folio) if CONFIG_DEBUG_VM is enabled, and can corrupt allocator state otherwise. Fix this by using folio_put() instead of free_huge_folio() to correctly drop the refcount before freeing.

3. Returning -ENOMEM on memcg charge failure causes infinite loop

alloc_hugetlb_folio() propagates -ENOMEM on charge failure, which maps to VM_FAULT_OOM. Because HugeTLB physical allocations are high-order and use __GFP_RETRY_MAYFAIL, the OOM killer is bypassed. Returning VM_FAULT_OOM leaks to the #PF handler, which cannot make progress and retries the faulting instruction indefinitely. Fix this by returning -ENOSPC instead of -ENOMEM on charge failure, which maps to VM_FAULT_SIGBUS, terminating the process cleanly.

4. vma reservation leak on mem_cgroup_charge_hugetlb() failure

When mem_cgroup_charge_hugetlb() fails, the error path historically bypassed vma_end_reservation(). Since the reservation had already been committed via vma_commit_reservation(), this left the reservation map in an inconsistent state, leaking resv_huge_pages when the process exited. Fix this by moving mem_cgroup_charge_hugetlb() earlier in alloc_hugetlb_folio(), before vma_commit_reservation() is called.

Main changes in this patch series

1. subpool page tracking

The subpool now tracks every page allocated via
the subpool through used_hpages, regardless of whether min_hpages or max_hpages is requested by the user.

When min_hpages is set, rsv_hpages is initialized to be equal to min_hpages. As reserved pages are used, the count is transferred from rsv_hpages to used_hpages. When reserved pages run out, as long as max_hpages is not requested or not yet met, used_hpages continues to be incremented. The following invariant holds true while used_hpages <= min_hpages:

used_hpages + rsv_hpages = min_hpages

when used_hpages exceeds min_hpages, rsv_hpages must always be 0.

With this, pages can and must always be returned to the subpool if there's some reservation or allocation failure after getting a page from the subpool. One thing I missed in v2 was in processing the returned page. In v2, the number of pages returned was compared to min_hpages, ignoring used_hpages if it exists. v3 limits the number of pages returned to min_hpages - used_hpages, upholding the invariant above.

2. (Re-)introduction of the try-commit-cancel protocol for memcg charging.

In v2, I took the shortcut where I moved mem_cgroup_charge_hugetlb() up above committing vma reservations, and jumped to subpool cleanup when the charging failed. With that, Sashiko pointed out that there are still issues with subpool tracking.

In [4] I thought the issue was that gbl_chg == 0 (I flipped the condition in [4], omg) doesn't always mean a reservation was actually used, but after digging more, that doesn't seem to be true, since if gbl_chg == 0, restore_reserve is set on the folio, and free_huge_folio() would have done a h->resv_huge_pages++.

In my opinion, calling free_huge_folio() in the middle of the cleanup does too much, and I think the cleaner approach would be to do all the charging symmetrically, so that the subpool cleanup will also be symmetric and (hopefully) easier to understand.

v3 is posted with reproducers (not meant to be merged) and some refactoring of subpool-related code, which helped me check that the subpool stuff is working. I think the subpool-related refactoring and tests would be good to merge, but perhaps in another patch series? The refactoring was heavily AI-driven and should be considered RFC. Would like to have your feedback on that refactoring too.

Testing:

+ libhugetlbfs tests pass
+ ./tools/testing/selftests/mm/ksft_hugetlb.sh passes
+ Reproducers all pass

v1: https://lore.kernel.org/r/20260707-hugetlb-alloc-failure-fixes-v1-0-5bbd3a4b836d@google.com
v2: https://lore.kernel.org/r/20260708-hugetlb-alloc-failure-fixes-v2-0-c7f27cbb462b@google.com

The series [1] has some changes that are dependent on these fixes, so I'll wait for your reviews before continuing on [1].

Thank you!

[1] https://lore.kernel.org/all/[email protected]/T/
[2] https://sashiko.dev/#/patchset/20260518-hugetlb-open-up-v3-0-e14b302477f8%40google.com?part=5
[3] https://sashiko.dev/#/patchset/20260702-hugetlb-open-up-v4-0-d53cefcccf34%40google.com?part=6
[4] https://lore.kernel.org/all/CAEvNRgGN0HSJ2iLSDD2haSKOxifa-uhkO9Hwossh0+Q_d9fzOw@mail.gmail.com/

---
Ackerley Tng (13):
mm: hugetlb: Track used_hpages when getting/putting pages from subpool
mm: hugetlb: Return -ENOSPC on memcg charge failure
mm: hugetlb: Use try-commit-cancel protocol for memcg charge of folios
mm: hugetlb: Remove unused mem_cgroup_charge_hugetlb function
mm: hugetlb: Fix subpool usage leak on allocation failure
WIP: mm: hugetlb: Move subpool functions to hugetlb_subpool.c
WIP: fs: hugetlbfs: Refactor subpool getters and integrate with hugetlb_subpool API
WIP: mm: hugetlb: Make struct hugepage_subpool private to hugetlb_subpool.c
WIP: tools: testing: Add userspace unit tests for HugeTLB subpools
WIP: Reproducer for allocation failure due to cgroup v2 memory limits
WIP: Reproducer for subpool usage leak
WIP: Reproducer for false restoration on shared HugeTLB mappings
WIP: Reproducer for out_put_pages subpool reserve leakage

 cgroup_v2_allocation_failure.c               | 160 +++++++++++
 fs/hugetlbfs/inode.c                         |  25 +-
 hugetlb_reserve_pages_out_put_pages.c        |  49 ++++
 hugetlb_reserve_pages_out_put_pages.sh       | 153 ++++++++++
 include/linux/hugetlb.h                      |  17 +-
 include/linux/memcontrol.h                   |  30 +-
 mm/Makefile                                  |   2 +-
 mm/hugetlb.c                                 | 244 ++++------------
 mm/hugetlb_subpool.c                         | 223 +++++++++++++++
 mm/hugetlb_subpool.h                         |  21 ++
 mm/memcontrol.c                              | 116 ++++++--
 subpool_leak_max_size.sh                     |  71 +++++
 subpool_shared_leak.c                        |  29 ++
 subpool_shared_leak.sh                       |  86 ++++++
 tools/testing/hugetlb_subpool/.gitignore     |   1 +
 tools/testing/hugetlb_subpool/Makefile       |  18 ++
 tools/testing/hugetlb_subpool/test_subpool.c | 400 +++++++++++++++++++++++++++
 17 files changed, 1399 insertions(+), 246 deletions(-)

---
base-commit: b95f03f04d475aa6719d15a636ddf32222d55657
change-id: 20260706-hugetlb-alloc-failure-fixes-c7f775eca29f

Best regards,
--
Ackerley Tng <[email protected]>

#1 lore raw

PATCH v3 1/13 mm: hugetlb: Track used_hpages when getting/putting pages from subpool

hugepage_subpool_put_pages() currently has two distinct responsibilities that conflict:

1. When size is specified for the mount, max_hpages != -1: Keep track of total active pages (allocated + reserved) and decrement this count (used_hpages) when a page is freed or allocation fails.
2. When min_size is specified for the mount, min_hpages != -1: Ensure we don't drop below the guaranteed minimum, and restore a reservation (rsv_hpages) if we do.

This causes trouble because when allocation fails (refer to alloc_hugetlb_folio()) if gbl_chg = 1 (i.e. no subpool reservation was taken):

+ To keep used_hpages consistent, HugeTLB needs to call hugepage_subpool_put_pages() to restore undo used_hpages being incremented
+ But can't call hugepage_subpool_put_pages() if no reservation was consumed.

One option would be to conditionally do subpool tracking updates outside of the hugepage_subpool_put_pages() function, but that would spread logic all over.

Instead, always track used_hpages, regardless of whether a max_size was requested for the mount, so that the subpool always knows how many pages were allocated through it. Every page allocated through the subpool increments used_hpages, regardless of whether a reservation was taken from it.

Conceptually, now, every allocation involving a subpool uses a page from the subpool, which must be returned to the subpool. Every page taken from the subpool tries to use a subpool reservation. Restoring a page to the subpool reservations only if the page was taken from subpool reservations. (If used_hpages >= min_hpages, the page must have not have been taken from the reservations.)

Always tracking used_hpages provides the subpool with information of both used and reserved counts to make the correct decision for both max_size and min_size correctly.

With used_hpages always tracked,

+ subpool_is_free() can be simplified, such that the subpool can be declared free if there are no more pages in use.
+ open-coding in hugetlb_reserve_pages() can be removed.

Also update the documentation for used_hpages, since it no longer matters whether the used pages count against the maximum.

Also update statfs reporting. Previously, if max_hpages is negative, used_hpages is static at 0, so returning max_hpages - used_hpages returns -1 and is always correct. Now, if the subpool doesn't have a maximum requested size, indicate no limit for free pages (-1). If it does have a maximum size, report the difference between the requested size and the number of used pages. This difference is always positive, because if the mount does have a maximum size, hugepage_subpool_get_pages() ensures that the subpool usage never exceeds the maximum.

This fixes a bug in hugetlb_unreserve_pages(), where pages are returned to the subpool regardless of whether it consumed a reservation. The corresponding bug in the failure handling path of alloc_hugetlb_folio() was fixed in a833a693a490e.

Fixes 1c5ecae3a93fa ("hugetlbfs: add minimum size accounting to subpools")
---
 fs/hugetlbfs/inode.c    |  8 ++++--
 include/linux/hugetlb.h |  4 +--
 mm/hugetlb.c            | 71 +++++++++++++++++++++++--------------------------
 3 files changed, 41 insertions(+), 42 deletions(-)
fs/hugetlbfs/inode.c +6 -2
index 216e1a0dd0b23..26c0187340636 100644
@@ -1109,8 +1109,12 @@ static int hugetlbfs_statfs(struct dentry *dentry, struct kstatfs *buf)
 
 			spin_lock_irq(&sbinfo->spool->lock);
 			buf->f_blocks = sbinfo->spool->max_hpages;
-			free_pages = sbinfo->spool->max_hpages
-				- sbinfo->spool->used_hpages;
+			if (sbinfo->spool->max_hpages == -1) {
+				free_pages = -1;
+			} else {
+				free_pages = sbinfo->spool->max_hpages -
+					     sbinfo->spool->used_hpages;
+			}
 			buf->f_bavail = buf->f_bfree = free_pages;
 			spin_unlock_irq(&sbinfo->spool->lock);
 			buf->f_files = sbinfo->max_inodes;
include/linux/hugetlb.h +2 -2
index 2abaf99321e90..34b9a3e1be0fa 100644
@@ -38,8 +38,8 @@ struct hugepage_subpool {
 	spinlock_t lock;
 	long count;
 	long max_hpages;	/* Maximum huge pages or -1 if no maximum. */
-	long used_hpages;	/* Used count against maximum, includes */
-				/* both allocated and reserved pages. */
+	long used_hpages;	/* Used page count, includes both */
+				/* allocated and reserved pages. */
 	struct hstate *hstate;
 	long min_hpages;	/* Minimum huge pages or -1 if no minimum. */
 	long rsv_hpages;	/* Pages reserved against global pool to */
mm/hugetlb.c +33 -38

112 lines not on this page · raw message

Signature · 1 line
2.55.0.229.g6434b31f56-goog
#2 lore raw

PATCH v3 2/13 mm: hugetlb: Return -ENOSPC on memcg charge failure

In reply to Ackerley Tng

When mem_cgroup_charge_hugetlb() fails with -ENOMEM, alloc_hugetlb_folio() currently propagates this error. This results in the page fault handler returning VM_FAULT_OOM.

Because HugeTLB allocations are high-order and use __GFP_RETRY_MAYFAIL, they bypass the OOM killer. Returning VM_FAULT_OOM to the #PF handler without triggering the OOM killer (or having it make progress) leads to an infinite loop of retrying the fault.

Avoid this loop by returning -ENOSPC when charging fails, which maps to VM_FAULT_SIGBUS, terminating the process cleanly.

Make mem_cgroup_charge_hugetlb() fault handling use a common error handling path, the same handling used for hugetlb_cgroup_uncharge_cgroup{,_rsvd}(), which also don't trigger the OOM killer and hence opt to terminate the process with a SIGBUS.

Fixes 991135774c0e0 ("memcg/hugetlb: introduce mem_cgroup_charge_hugetlb")
---
 mm/hugetlb.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
mm/hugetlb.c +12 -1
index eef9610a0593c..b32735b092a0a 100644
@@ -2997,7 +2997,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 
 	if (ret == -ENOMEM) {
 		free_huge_folio(folio);
-		return ERR_PTR(-ENOMEM);
+		goto err;
 	}
 
 	return folio;
@@ -3022,6 +3022,17 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 out_end_reservation:
 	if (map_chg != MAP_CHG_ENFORCED)
 		vma_end_reservation(h, vma, addr);
+err:
+	/*
+	 * Return -ENOSPC when this function fails to allocate or
+	 * charge a huge page. If a standard (PAGE_SIZE) page
+	 * allocation fails, the OOM killer is given a chance to run,
+	 * which may resolve the failure on retry. However, for
+	 * HugeTLB allocations, the OOM killer is not triggered.
+	 * Returning -ENOMEM (or anything resulting in VM_FAULT_OOM)
+	 * would leak to the #PF handler, causing it to loop
+	 * indefinitely retrying the fault.
+	 */
 	return ERR_PTR(-ENOSPC);
 }
 
Signature · 1 line
2.55.0.229.g6434b31f56-goog
#3 lore raw

PATCH v3 3/13 mm: hugetlb: Use try-commit-cancel protocol for memcg charge of folios

In reply to Ackerley Tng

Using mem_cgroup_charge_hugetlb() to charge a HugeTLB folio during page fault creates a reservation leak bug if the task hits its memory cgroup limit.

When alloc_hugetlb_folio() commits the VMA reservation, the reserved page is removed from the reserve map. If a subsequent call to mem_cgroup_charge_hugetlb() returns -ENOMEM, the allocation is aborted and the physical folio is disposed of via free_huge_page(). However, because the VMA reservation was already consumed, the reservation count in the reserve map is lost. This causes subsequent faults in the VMA address range to fail with premature reservation exhaustion.

Additionally, dropping the use of free_huge_folio() on the failure path fixes an issue where free_huge_folio() was incorrectly invoked on a folio with a refcount of 1, triggering refcount mismatches and kernel warnings.

To fix this, introduce a try-commit-cancel protocol for memory cgroup charging of HugeTLB folios, matching the architecture used by the hugetlb cgroup controller. Invoking mem_cgroup_hugetlb_try_charge() before consuming the VMA reservation ensures that if the memory cgroup limit is reached, the allocation is aborted cleanly without leaking the reservation entry or having to dispose of a partially initialized folio.

An alternative would be to retain the current usage of mem_cgroup_charge_hugetlb() and free_huge_page(), but freeing the folio performs reservation management for subpools and global hstate, which complicates rollback in alloc_hugetlb_folio(). Using a try-commit-cancel protocol is more consistent with the other charging performed in alloc_hugetlb_folio() and easier to understand.

Fixes 991135774c0e0 ("memcg/hugetlb: introduce mem_cgroup_charge_hugetlb")
---
 include/linux/memcontrol.h |  36 ++++++++++++--
 mm/hugetlb.c               |  30 ++++++------
 mm/memcontrol.c            | 114 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 162 insertions(+), 18 deletions(-)
include/linux/memcontrol.h +33 -3
index e1f46a0016fcf..c4e63d0e03526 100644
@@ -641,7 +641,16 @@ static inline int mem_cgroup_charge(struct folio *folio, struct mm_struct *mm,
 	return __mem_cgroup_charge(folio, mm, gfp);
 }
 
-int mem_cgroup_charge_hugetlb(struct folio* folio, gfp_t gfp);
+int mem_cgroup_charge_hugetlb(struct folio *folio, gfp_t gfp);
+int mem_cgroup_hugetlb_try_charge(unsigned int nr_pages, gfp_t gfp,
+				  struct mem_cgroup **memcg_p,
+				  struct obj_cgroup **objcg_p);
+void mem_cgroup_hugetlb_commit_charge(struct folio *folio,
+				      struct mem_cgroup *memcg,
+				      struct obj_cgroup *objcg);
+void mem_cgroup_hugetlb_cancel_charge(unsigned int nr_pages,
+				      struct mem_cgroup *memcg,
+				      struct obj_cgroup *objcg);
 
 int mem_cgroup_swapin_charge_folio(struct folio *folio, unsigned short id,
 				   struct mm_struct *mm, gfp_t gfp);
@@ -1128,9 +1137,30 @@ static inline int mem_cgroup_charge(struct folio *folio,
 	return 0;
 }
 
-static inline int mem_cgroup_charge_hugetlb(struct folio* folio, gfp_t gfp)
+static inline int mem_cgroup_charge_hugetlb(struct folio *folio, gfp_t gfp)
+{
+	return 0;
+}
+
+static inline int mem_cgroup_hugetlb_try_charge(unsigned int nr_pages, gfp_t gfp,
+						struct mem_cgroup **memcg_p,
+						struct obj_cgroup **objcg_p)
+{
+	*memcg_p = NULL;
+	*objcg_p = NULL;
+	return 0;
+}
+
+static inline void mem_cgroup_hugetlb_commit_charge(struct folio *folio,
+						    struct mem_cgroup *memcg,
+						    struct obj_cgroup *objcg)
+{
+}
+
+static inline void mem_cgroup_hugetlb_cancel_charge(unsigned int nr_pages,
+						    struct mem_cgroup *memcg,
+						    struct obj_cgroup *objcg)
 {
-        return 0;
 }
 
 static inline int mem_cgroup_swapin_charge_folio(struct folio *folio,
mm/hugetlb.c +15 -15
index b32735b092a0a..061d7250c202d 100644
@@ -38,6 +38,7 @@
 #include <linux/mm_inline.h>
 #include <linux/padata.h>
 #include <linux/pgalloc.h>
+#include <linux/memcontrol.h>
 
 #include <asm/page.h>
 #include <asm/tlb.h>
@@ -2863,6 +2864,8 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 	int ret, idx;
 	struct hugetlb_cgroup *h_cg = NULL;
 	struct hugetlb_cgroup *h_cg_rsvd = NULL;
+	struct mem_cgroup *mem_cg = NULL;
+	struct obj_cgroup *obj_cg = NULL;
 	gfp_t gfp = htlb_alloc_mask(h) | __GFP_RETRY_MAYFAIL;
 
 	idx = hstate_index(h);
@@ -2922,6 +2925,11 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 	if (ret)
 		goto out_uncharge_cgroup_reservation;
 
+	ret = mem_cgroup_hugetlb_try_charge(pages_per_huge_page(h), gfp,
+					    &mem_cg, &obj_cg);
+	if (ret)
+		goto out_uncharge_cgroup;
+
 	spin_lock_irq(&hugetlb_lock);
 	/*
 	 * glb_chg is passed to indicate whether or not a page must be taken
@@ -2933,7 +2941,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 		spin_unlock_irq(&hugetlb_lock);
 		folio = alloc_buddy_hugetlb_folio_with_mpol(h, vma, addr);
 		if (!folio)
-			goto out_uncharge_cgroup;
+			goto out_uncharge_cgroup_memcg;
 		spin_lock_irq(&hugetlb_lock);
 		list_add(&folio->lru, &h->hugepage_activelist);
 		folio_ref_unfreeze(folio, 1);
@@ -2960,6 +2968,9 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 
 	spin_unlock_irq(&hugetlb_lock);
 
+	mem_cgroup_hugetlb_commit_charge(folio, mem_cg, obj_cg);
+	lruvec_stat_mod_folio(folio, NR_HUGETLB, pages_per_huge_page(h));
+
 	hugetlb_set_folio_subpool(folio, spool);
 
 	if (map_chg != MAP_CHG_ENFORCED) {
@@ -2987,21 +2998,10 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 		}
 	}
 
-	ret = mem_cgroup_charge_hugetlb(folio, gfp);
-	/*
-	 * Unconditionally increment NR_HUGETLB here. If it turns out that
-	 * mem_cgroup_charge_hugetlb failed, then immediately free the page and
-	 * decrement NR_HUGETLB.
-	 */
-	lruvec_stat_mod_folio(folio, NR_HUGETLB, pages_per_huge_page(h));
-
-	if (ret == -ENOMEM) {
-		free_huge_folio(folio);
-		goto err;
-	}
-
 	return folio;
 
+out_uncharge_cgroup_memcg:
+	mem_cgroup_hugetlb_cancel_charge(pages_per_huge_page(h), mem_cg, obj_cg);
 out_uncharge_cgroup:
 	hugetlb_cgroup_uncharge_cgroup(idx, pages_per_huge_page(h), h_cg);
 out_uncharge_cgroup_reservation:
@@ -3022,7 +3022,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 out_end_reservation:
 	if (map_chg != MAP_CHG_ENFORCED)
 		vma_end_reservation(h, vma, addr);
-err:
+
 	/*
 	 * Return -ENOSPC when this function fails to allocate or
 	 * charge a huge page. If a standard (PAGE_SIZE) page
mm/memcontrol.c +114 -0

120 lines not on this page · raw message

Signature · 1 line
2.55.0.229.g6434b31f56-goog
#4 lore raw

PATCH v3 4/13 mm: hugetlb: Remove unused mem_cgroup_charge_hugetlb function

In reply to Ackerley Tng

Now that the alloc_hugetlb_folio path has been successfully migrated to the new try-commit-cancel memcg charging protocol, the old mem_cgroup_charge_hugetlb function and its associated header and static inline declarations are completely unused. Remove them to clean up the memory controller's codebase.

---
 include/linux/memcontrol.h |  6 ------
 mm/memcontrol.c            | 34 ----------------------------------
 2 files changed, 40 deletions(-)
include/linux/memcontrol.h +0 -6
index c4e63d0e03526..23fbbbc8e11f2 100644
@@ -641,7 +641,6 @@ static inline int mem_cgroup_charge(struct folio *folio, struct mm_struct *mm,
 	return __mem_cgroup_charge(folio, mm, gfp);
 }
 
-int mem_cgroup_charge_hugetlb(struct folio *folio, gfp_t gfp);
 int mem_cgroup_hugetlb_try_charge(unsigned int nr_pages, gfp_t gfp,
 				  struct mem_cgroup **memcg_p,
 				  struct obj_cgroup **objcg_p);
@@ -1137,11 +1136,6 @@ static inline int mem_cgroup_charge(struct folio *folio,
 	return 0;
 }
 
-static inline int mem_cgroup_charge_hugetlb(struct folio *folio, gfp_t gfp)
-{
-	return 0;
-}
-
 static inline int mem_cgroup_hugetlb_try_charge(unsigned int nr_pages, gfp_t gfp,
 						struct mem_cgroup **memcg_p,
 						struct obj_cgroup **objcg_p)
mm/memcontrol.c +0 -34
index 0beee5c0ce93b..6764ff041c196 100644
@@ -5146,40 +5146,6 @@ int __mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, gfp_t gfp)
 	return ret;
 }
 
-/**
- * mem_cgroup_charge_hugetlb - charge the memcg for a hugetlb folio
- * @folio: folio being charged
- * @gfp: reclaim mode
- *
- * This function is called when allocating a huge page folio, after the page has
- * already been obtained and charged to the appropriate hugetlb cgroup
- * controller (if it is enabled).
- *
- * Returns ENOMEM if the memcg is already full.
- * Returns 0 if either the charge was successful, or if we skip the charging.
- */
-int mem_cgroup_charge_hugetlb(struct folio *folio, gfp_t gfp)
-{
-	struct mem_cgroup *memcg = get_mem_cgroup_from_current();
-	int ret = 0;
-
-	/*
-	 * Even memcg does not account for hugetlb, we still want to update
-	 * system-level stats via lruvec_stat_mod_folio. Return 0, and skip
-	 * charging the memcg.
-	 */
-	if (mem_cgroup_disabled() || !memcg_accounts_hugetlb() ||
-		!memcg || !cgroup_subsys_on_dfl(memory_cgrp_subsys))
-		goto out;
-
-	if (charge_memcg(folio, memcg, gfp))
-		ret = -ENOMEM;
-
-out:
-	mem_cgroup_put(memcg);
-	return ret;
-}
-
 /**
  * mem_cgroup_hugetlb_try_charge - Try to charge the memcg for a hugetlb folio
  * @nr_pages: number of base pages to charge
Signature · 1 line
2.55.0.229.g6434b31f56-goog
#5 lore raw

PATCH v3 5/13 mm: hugetlb: Fix subpool usage leak on allocation failure

In reply to Ackerley Tng

When alloc_hugetlb_folio() fails early (e.g. buddy allocation failure or hugetlb cgroup charging failure) and gbl_chg == 1 (meaning a reservation was not used, but a global page was allocated instead), the subpool page acquired via hugepage_subpool_get_pages() must still be returned.

Currently, the error path out_subpool_put: only calls hugepage_subpool_put_pages() if !gbl_chg is true. If gbl_chg is 1, it skips it, permanently leaking the subpool's used_hpages counter.

With the earlier patch to always track used_hpages in the subpool, always call hugepage_subpool_put_pages() if map_chg is true to consistently restore the page to the subpool. Only call hugetlb_acct_memory() to adjust global reservations if gbl_chg == 0 since gbl_chg == 0 indicates a subpool (and global) reservation was used.

Fixes a833a693a490e ("mm: hugetlb: fix incorrect fallback for subpool")
---
 mm/hugetlb.c | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)
mm/hugetlb.c +6 -8
index 061d7250c202d..3f6189d2d0188 100644
@@ -2859,7 +2859,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 	struct hugepage_subpool *spool = subpool_vma(vma);
 	struct hstate *h = hstate_vma(vma);
 	struct folio *folio;
-	long retval, gbl_chg, gbl_reserve;
+	long retval, gbl_chg;
 	map_chg_state map_chg;
 	int ret, idx;
 	struct hugetlb_cgroup *h_cg = NULL;
@@ -3009,13 +3009,11 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 		hugetlb_cgroup_uncharge_cgroup_rsvd(idx, pages_per_huge_page(h),
 						    h_cg_rsvd);
 out_subpool_put:
-	/*
-	 * put page to subpool iff the quota of subpool's rsv_hpages is used
-	 * during hugepage_subpool_get_pages.
-	 */
-	if (map_chg && !gbl_chg) {
-		gbl_reserve = hugepage_subpool_put_pages(spool, 1);
-		hugetlb_acct_memory(h, -gbl_reserve);
+	if (map_chg) {
+		long gbl_reserve = hugepage_subpool_put_pages(spool, 1);
+
+		if (!gbl_chg)
+			hugetlb_acct_memory(h, -gbl_reserve);
 	}
 
 
Signature · 1 line
2.55.0.229.g6434b31f56-goog
#6 lore raw

PATCH v3 6/13 WIP: mm: hugetlb: Move subpool functions to hugetlb_subpool.c

In reply to Ackerley Tng

Move all HugeTLB subpool lifecycle, page reservation, and accounting routines out of mm/hugetlb.c and into their own dedicated, encapsulated translation unit at mm/hugetlb_subpool.c.

Also introduces the internal mm/hugetlb_subpool.h header for holding the subpool-local APIs, allowing fs/hugetlbfs and mm/ to access the subpool functions cleanly. The subpool internal layout structures remain in include/linux/hugetlb.h until getters are introduced.

---
 fs/hugetlbfs/inode.c    |   1 +
 include/linux/hugetlb.h |   4 +-
 mm/Makefile             |   2 +-
 mm/hugetlb.c            | 156 +-------------------------------------------
 mm/hugetlb_subpool.c    | 168 ++++++++++++++++++++++++++++++++++++++++++++++++
 mm/hugetlb_subpool.h    |  17 +++++
 6 files changed, 191 insertions(+), 157 deletions(-)
fs/hugetlbfs/inode.c +1 -0
index 26c0187340636..8c1caad74c409 100644
@@ -25,6 +25,7 @@
 #include <linux/ctype.h>
 #include <linux/backing-dev.h>
 #include <linux/hugetlb.h>
+#include "../../mm/hugetlb_subpool.h"
 #include <linux/folio_batch.h>
 #include <linux/fs_parser.h>
 #include <linux/mman.h>
include/linux/hugetlb.h +1 -3
index 34b9a3e1be0fa..f36be371c6e88 100644
@@ -114,9 +114,7 @@ extern int hugetlb_max_hstate __read_mostly;
 #define for_each_hstate(h) \
 	for ((h) = hstates; (h) < &hstates[hugetlb_max_hstate]; (h)++)
 
-struct hugepage_subpool *hugepage_new_subpool(struct hstate *h, long max_hpages,
-						long min_hpages);
-void hugepage_put_subpool(struct hugepage_subpool *spool);
+int hugetlb_acct_memory(struct hstate *h, long delta);
 
 void hugetlb_dup_vma_private(struct vm_area_struct *vma);
 void clear_vma_resv_huge_pages(struct vm_area_struct *vma);
mm/Makefile +1 -1
index eff9f9e7e061c..3965c959e5099 100644
@@ -78,7 +78,7 @@ endif
 obj-$(CONFIG_SWAP)	+= page_io.o swap_state.o swapfile.o
 obj-$(CONFIG_ZSWAP)	+= zswap.o
 obj-$(CONFIG_HAS_DMA)	+= dmapool.o
-obj-$(CONFIG_HUGETLBFS)	+= hugetlb.o hugetlb_sysfs.o hugetlb_sysctl.o
+obj-$(CONFIG_HUGETLBFS)	+= hugetlb.o hugetlb_subpool.o hugetlb_sysfs.o hugetlb_sysctl.o
 ifdef CONFIG_CMA
 obj-$(CONFIG_HUGETLBFS)	+= hugetlb_cma.o
 endif
mm/hugetlb.c +3 -153

180 lines not on this page · raw message

mm/hugetlb_subpool.c new file +168 -0

168 lines not on this page · raw message

mm/hugetlb_subpool.h new file +17 -0
new file mode 100644
index 0000000000000..be1f1cf012c9c
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _MM_HUGETLB_SUBPOOL_H
+#define _MM_HUGETLB_SUBPOOL_H
+
+#include <linux/spinlock.h>
+#include <linux/types.h>
+
+struct hstate;
+struct hugepage_subpool;
+
+struct hugepage_subpool *hugepage_new_subpool(struct hstate *h, long max_hpages,
+						long min_hpages);
+void hugepage_put_subpool(struct hugepage_subpool *spool);
+long hugepage_subpool_get_pages(struct hugepage_subpool *spool, long delta);
+long hugepage_subpool_put_pages(struct hugepage_subpool *spool, long delta);
+
+#endif /* _MM_HUGETLB_SUBPOOL_H */
Signature · 1 line
2.55.0.229.g6434b31f56-goog
#7 lore raw

PATCH v3 7/13 WIP: fs: hugetlbfs: Refactor subpool getters and integrate with hugetlb_subpool API

In reply to Ackerley Tng

Refactor the direct field accesses to spool->max_hpages, spool->min_hpages, and calculate subpool properties using dedicated, encapsulated accessor getters inside mm/hugetlb_subpool.c.

Introduces and exports the following harmonized subpool getters to mm/hugetlb_subpool.h:
- hugepage_subpool_free_hpages()
- hugepage_subpool_max_size()
- hugepage_subpool_min_size()
- hugepage_subpool_max_hpages()

---
 fs/hugetlbfs/inode.c | 28 +++++++++-------------------
 mm/hugetlb_subpool.c | 45 ++++++++++++++++++++++++++++++++++++++++++++-
 mm/hugetlb_subpool.h |  4 ++++
 3 files changed, 57 insertions(+), 20 deletions(-)
fs/hugetlbfs/inode.c +9 -19
index 8c1caad74c409..8abe0574b1c7e 100644
@@ -1060,7 +1060,6 @@ static int hugetlbfs_show_options(struct seq_file *m, struct dentry *root)
 	struct hugetlbfs_sb_info *sbinfo = HUGETLBFS_SB(root->d_sb);
 	struct hugepage_subpool *spool = sbinfo->spool;
 	unsigned long hpage_size = huge_page_size(sbinfo->hstate);
-	unsigned hpage_shift = huge_page_shift(sbinfo->hstate);
 	char mod;
 
 	if (!uid_eq(sbinfo->uid, GLOBAL_ROOT_UID))
@@ -1082,12 +1081,13 @@ static int hugetlbfs_show_options(struct seq_file *m, struct dentry *root)
 	}
 	seq_printf(m, ",pagesize=%lu%c", hpage_size, mod);
 	if (spool) {
-		if (spool->max_hpages != -1)
-			seq_printf(m, ",size=%llu",
-				   (unsigned long long)spool->max_hpages << hpage_shift);
-		if (spool->min_hpages != -1)
-			seq_printf(m, ",min_size=%llu",
-				   (unsigned long long)spool->min_hpages << hpage_shift);
+		unsigned long long max_size = hugepage_subpool_max_size(spool);
+		unsigned long long min_size = hugepage_subpool_min_size(spool);
+
+		if (max_size != -1ULL)
+			seq_printf(m, ",size=%llu", max_size);
+		if (min_size != -1ULL)
+			seq_printf(m, ",min_size=%llu", min_size);
 	}
 	return 0;
 }
@@ -1106,18 +1106,8 @@ static int hugetlbfs_statfs(struct dentry *dentry, struct kstatfs *buf)
 		/* If no limits set, just report 0 or -1 for max/free/used
 		 * blocks, like simple_statfs() */
 		if (sbinfo->spool) {
-			long free_pages;
-
-			spin_lock_irq(&sbinfo->spool->lock);
-			buf->f_blocks = sbinfo->spool->max_hpages;
-			if (sbinfo->spool->max_hpages == -1) {
-				free_pages = -1;
-			} else {
-				free_pages = sbinfo->spool->max_hpages -
-					     sbinfo->spool->used_hpages;
-			}
-			buf->f_bavail = buf->f_bfree = free_pages;
-			spin_unlock_irq(&sbinfo->spool->lock);
+			buf->f_blocks = hugepage_subpool_max_hpages(sbinfo->spool);
+			buf->f_bavail = buf->f_bfree = hugepage_subpool_free_hpages(sbinfo->spool);
 			buf->f_files = sbinfo->max_inodes;
 			buf->f_ffree = sbinfo->free_inodes;
 		}
mm/hugetlb_subpool.c +44 -1

57 lines not on this page · raw message

mm/hugetlb_subpool.h +4 -0
index be1f1cf012c9c..41d22239f2c3e 100644
@@ -13,5 +13,9 @@ struct hugepage_subpool *hugepage_new_subpool(struct hstate *h, long max_hpages,
 void hugepage_put_subpool(struct hugepage_subpool *spool);
 long hugepage_subpool_get_pages(struct hugepage_subpool *spool, long delta);
 long hugepage_subpool_put_pages(struct hugepage_subpool *spool, long delta);
+long hugepage_subpool_free_hpages(struct hugepage_subpool *spool);
+long hugepage_subpool_max_hpages(struct hugepage_subpool *spool);
+unsigned long long hugepage_subpool_max_size(struct hugepage_subpool *spool);
+unsigned long long hugepage_subpool_min_size(struct hugepage_subpool *spool);
 
 #endif /* _MM_HUGETLB_SUBPOOL_H */
Signature · 1 line
2.55.0.229.g6434b31f56-goog
#8 lore raw

PATCH v3 8/13 WIP: mm: hugetlb: Make struct hugepage_subpool private to hugetlb_subpool.c

In reply to Ackerley Tng

Now that all filesystem and memory management subpool layout inspections across fs/hugetlbfs/inode.c and mm/ leverage the exported, harmonized APIs, transition struct hugepage_subpool out of the public header include/linux/hugetlb.h and encapsulate it privately inside mm/hugetlb_subpool.c.

Replace the header definition with a forward declaration, completely protecting the underlying subpool allocation counters, reservations, and spinlock boundaries.

---
 include/linux/hugetlb.h | 13 ++-----------
 mm/hugetlb_subpool.c    | 12 ++++++++++++
 2 files changed, 14 insertions(+), 11 deletions(-)
include/linux/hugetlb.h +2 -11
index f36be371c6e88..074a45903973e 100644
@@ -18,6 +18,7 @@
 
 struct mmu_gather;
 struct node;
+struct hstate;
 
 void free_huge_folio(struct folio *folio);
 
@@ -34,17 +35,7 @@ void free_huge_folio(struct folio *folio);
  */
 #define __NR_USED_SUBPAGE 3
 
-struct hugepage_subpool {
-	spinlock_t lock;
-	long count;
-	long max_hpages;	/* Maximum huge pages or -1 if no maximum. */
-	long used_hpages;	/* Used page count, includes both */
-				/* allocated and reserved pages. */
-	struct hstate *hstate;
-	long min_hpages;	/* Minimum huge pages or -1 if no minimum. */
-	long rsv_hpages;	/* Pages reserved against global pool to */
-				/* satisfy minimum size. */
-};
+struct hugepage_subpool;
 
 struct resv_map {
 	struct kref refs;
mm/hugetlb_subpool.c +12 -0
index 99e7975911079..eb5472bdf2192 100644
@@ -14,6 +14,18 @@
 #include <linux/bug.h>
 #include "hugetlb_subpool.h"
 
+struct hugepage_subpool {
+	spinlock_t lock;
+	long count;
+	long max_hpages;	/* Maximum huge pages or -1 if no maximum. */
+	long used_hpages;	/* Used page count, includes both */
+				/* allocated and reserved pages. */
+	struct hstate *hstate;
+	long min_hpages;	/* Minimum huge pages or -1 if no minimum. */
+	long rsv_hpages;	/* Pages reserved against global pool to */
+				/* satisfy minimum size. */
+};
+
 static inline bool subpool_is_free(struct hugepage_subpool *spool)
 {
 	if (spool->count)
Signature · 1 line
2.55.0.229.g6434b31f56-goog
#9 lore raw

PATCH v3 9/13 WIP: tools: testing: Add userspace unit tests for HugeTLB subpools

In reply to Ackerley Tng

Introduce a standalone Userspace Unit Testing Suite under tools/testing/hugetlb_subpool/ to exercise and stress-test the new internal mm/hugetlb_subpool.c API boundaries.

Reuses the private kernel struct hugepage_subpool struct layout natively by embedding the implementation directly, avoiding structural definition drift between the implementation and testing mock environments.

TAG=agy
CONV=5f2e8401-48d5-42c5-ab78-6b2f1aa16aff

---
 tools/testing/hugetlb_subpool/.gitignore     |   1 +
 tools/testing/hugetlb_subpool/Makefile       |  18 ++
 tools/testing/hugetlb_subpool/test_subpool.c | 400 +++++++++++++++++++++++++++
 3 files changed, 419 insertions(+)
tools/testing/hugetlb_subpool/.gitignore new file +1 -0
new file mode 100644
index 0000000000000..7348c2c72f1e1
@@ -0,0 +1 @@
+test_subpool
tools/testing/hugetlb_subpool/Makefile new file +18 -0
new file mode 100644
index 0000000000000..1bdb7e2635614
@@ -0,0 +1,18 @@
+# SPDX-License-Identifier: GPL-2.0
+.PHONY: all clean test
+
+CC = gcc
+CFLAGS = -Wall -O2 -I../shared -I. -I../../include -I../../arch/x86/include -pthread
+KERNEL_SUBPOOL_H = ../../../mm/hugetlb_subpool.h
+KERNEL_SUBPOOL_C = ../../../mm/hugetlb_subpool.c
+
+all: test
+
+test_subpool: test_subpool.c $(KERNEL_SUBPOOL_C) $(KERNEL_SUBPOOL_H)
+	$(CC) $(CFLAGS) test_subpool.c -o test_subpool
+
+test: test_subpool
+	./test_subpool
+
+clean:
+	rm -f test_subpool
tools/testing/hugetlb_subpool/test_subpool.c new file +400 -0

400 lines not on this page · raw message

Signature · 1 line
2.55.0.229.g6434b31f56-goog
#10 lore raw

PATCH v3 10/13 WIP: Reproducer for allocation failure due to cgroup v2 memory limits

In reply to Ackerley Tng

(This reproducer was hacked up and not meant to be merged.)

cgroup_v2_allocation_failure.c triggers HugeTLB allocation failure by exploiting cgroup v2 memory limits. This allows testing the error paths in the kernel when memory control charging fails, even when physical huge pages are available.

The program performs the following steps to trigger the failure:

1. Enable hugetlb accounting in cgroup v2.
+ The program checks if memory_hugetlb_accounting is enabled in the cgroup2 mount options. If not, it remounts /sys/fs/cgroup with this option enabled. This ensures that HugeTLB allocations are charged against the cgroup memory limits.

2. Create a test cgroup and set limits.
+ The program creates a new cgroup subdirectory named test_reproducer under /sys/fs/cgroup.
+ It sets the memory.max limit of this cgroup to 1MB (which is less than the 2MB huge page size).

3. Fork a child process and move it to the test cgroup.
+ The program forks a child process.
+ The child process moves itself into the test_reproducer cgroup by writing its PID (using 0 for current process) to cgroup.procs in the test cgroup directory.

4. Attempt to allocate and touch a 2MB huge page.
+ The child process maps a 2MB anonymous huge page using mmap with MAP_PRIVATE, MAP_ANONYMOUS, and MAP_HUGETLB.
+ The child process writes to the mapped address, triggering a page fault.

5. Triggering the kernel bugs.
+ The page fault handler calls alloc_hugetlb_folio to allocate the huge page.
+ The allocation of the physical page from buddy allocator succeeds (assuming nr_hugepages is sufficient).
+ The kernel then attempts to charge this allocation to the child process's cgroup by calling mem_cgroup_charge_hugetlb.
+ Since the child's cgroup memory limit is 1MB and the page is 2MB, the charge fails and mem_cgroup_charge_hugetlb returns -ENOMEM.
+ This triggers the error path in alloc_hugetlb_folio where the bugs (folio refcount mismatch, infinite loop on ENOMEM, and reservation leaks) are handled.

---
 cgroup_v2_allocation_failure.c | 160 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 160 insertions(+)
cgroup_v2_allocation_failure.c new file +160 -0
new file mode 100644
index 0000000000000..938cbf02ae6f7
@@ -0,0 +1,160 @@
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <fcntl.h>
+#include <sys/mman.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <sys/wait.h>
+#include <string.h>
+#include <errno.h>
+
+#define CGROUP_PATH "/sys/fs/cgroup"
+#define TEST_CGROUP "test_reproducer"
+#define TEST_CGROUP_PATH CGROUP_PATH "/" TEST_CGROUP
+
+void write_file(const char *path, const char *val) {
+    int fd = open(path, O_WRONLY);
+    if (fd < 0) {
+        fprintf(stderr, "Failed to open %s: %s\n", path, strerror(errno));
+        exit(1);
+    }
+    if (write(fd, val, strlen(val)) < 0) {
+        fprintf(stderr, "Failed to write %s to %s: %s\n", val, path, strerror(errno));
+        close(fd);
+        exit(1);
+    }
+    close(fd);
+}
+
+int is_hugetlb_accounting_enabled() {
+    FILE *fp = fopen("/proc/mounts", "r");
+    if (!fp) {
+        perror("fopen /proc/mounts");
+        return -1;
+    }
+
+    char line[1024];
+    int enabled = 0;
+    while (fgets(line, sizeof(line), fp)) {
+        char spec[256], file[256], type[256], opts[512];
+        if (sscanf(line, "%255s %255s %255s %511s", spec, file, type, opts) == 4) {
+            if (strcmp(file, CGROUP_PATH) == 0 && strcmp(type, "cgroup2") == 0) {
+                if (strstr(opts, "memory_hugetlb_accounting") != NULL) {
+                    enabled = 1;
+                }
+                break;
+            }
+        }
+    }
+    fclose(fp);
+    return enabled;
+}
+
+int enable_hugetlb_accounting() {
+    printf("Attempting to remount cgroup2 with memory_hugetlb_accounting...\n");
+    int ret = system("mount -o remount,memory_hugetlb_accounting " CGROUP_PATH);
+    if (ret != 0) {
+        fprintf(stderr, "Failed to remount: system() returned %d\n", ret);
+        return -1;
+    }
+    return 0;
+}
+
+int main() {
+    struct stat st;
+    if (stat(CGROUP_PATH, &st) != 0 || !S_ISDIR(st.st_mode)) {
+        fprintf(stderr, "cgroup v2 not mounted at %s\n", CGROUP_PATH);
+        return 1;
+    }
+
+    int enabled = is_hugetlb_accounting_enabled();
+    if (enabled < 0) {
+        return 1;
+    }
+    if (!enabled) {
+        if (enable_hugetlb_accounting() != 0) {
+            fprintf(stderr, "Could not enable memory_hugetlb_accounting\n");
+            return 1;
+        }
+        // Re-check
+        enabled = is_hugetlb_accounting_enabled();
+        if (enabled <= 0) {
+            fprintf(stderr, "Failed to enable memory_hugetlb_accounting (re-check failed)\n");
+            return 1;
+        }
+        printf("Successfully enabled memory_hugetlb_accounting\n");
+    } else {
+        printf("memory_hugetlb_accounting is already enabled\n");
+    }
+
+    // Enable memory controller in subtree
+    int fd = open(CGROUP_PATH "/cgroup.subtree_control", O_WRONLY);
+    if (fd >= 0) {
+        if (write(fd, "+memory", 7) < 0) {
+            // Might fail if already enabled or not supported, ignore for now
+        }
+        close(fd);
+    }
+
+    if (mkdir(TEST_CGROUP_PATH, 0755) != 0) {
+        if (errno != EEXIST) {
+            perror("mkdir test_reproducer");
+            return 1;
+        }
+    }
+
+    // Set memory limit to 1MB (less than 2MB hugepage)
+    write_file(TEST_CGROUP_PATH "/memory.max", "1M");
+
+    pid_t pid = fork();
+    if (pid < 0) {
+        perror("fork");
+        return 1;
+    }
+
+    if (pid == 0) {
+        // Child
+        // Move to cgroup
+        write_file(TEST_CGROUP_PATH "/cgroup.procs", "0");
+
+        printf("Child: Attempting to allocate and touch 2MB hugepage...\n");
+        // Allocate 2MB hugepage
+        size_t size = 2 * 1024 * 1024;
+        void *addr = mmap(NULL, size, PROT_READ | PROT_WRITE,
+                          MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, -1, 0);
+        if (addr == MAP_FAILED) {
+            perror("Child: mmap MAP_HUGETLB");
+            exit(1);
+        }
+
+        printf("Child: mmap succeeded at %p, touching it now (should trigger fault)...\n", addr);
+        // This should trigger the fault and call alloc_hugetlb_folio -> mem_cgroup_charge_hugetlb
+        // which should fail and trigger the bug.
+        *(volatile char *)addr = 1;
+
+        printf("Child: Successfully touched page (bug not triggered?).\n");
+        munmap(addr, size);
+        exit(0);
+    }
+
+    // Parent
+    int status;
+    waitpid(pid, &status, 0);
+
+    printf("Parent: Child exited. Cleaning up.\n");
+    rmdir(TEST_CGROUP_PATH);
+
+    if (WIFSIGNALED(status)) {
+        printf("Parent: Child killed by signal %d (%s)\n",
+               WTERMSIG(status), strsignal(WTERMSIG(status)));
+        if (WTERMSIG(status) == SIGBUS) {
+            printf("Parent: Child got SIGBUS as expected (if kernel didn't crash).\n");
+        }
+    } else if (WIFEXITED(status)) {
+        printf("Parent: Child exited with status %d\n", WEXITSTATUS(status));
+    }
+
+    return 0;
+}
Signature · 1 line
2.55.0.229.g6434b31f56-goog
#11 lore raw

PATCH v3 11/13 WIP: Reproducer for subpool usage leak

In reply to Ackerley Tng

(This reproducer was hacked up and not meant to be merged.)

The kernel leaks subpool usage and the subpool structure itself if a HugeTLBfs mount specifying size (which sets max_hpages on the subpool) is created.

subpool_leak_max_size.sh reproduces this with the following steps:

1. Create mount, specifying size=2M (1 page). This sets max_hpages = 1 on the subpool, but does not reserve any pages.
2. Set nr_hugepages = 0 and nr_overcommit_hugepages = 0 so that physical allocations will fail.
3. Run fallocate -l 2M on a file in the mount.
+ This calls hugetlbfs_fallocate, which attempts to allocate a page by calling alloc_hugetlb_folio.
+ alloc_hugetlb_folio calls hugepage_subpool_get_pages to track the allocation against the subpool limit. This increments used_hpages to 1.
+ Physical allocation fails because nr_hugepages is 0.
+ Before patch (Buggy):
+ The error path in alloc_hugetlb_folio sees gbl_chg is 1 (indicating we tried to allocate a global page) and incorrectly skips calling hugepage_subpool_put_pages.
+ fallocate fails and returns to userspace, but the subpool used_hpages counter remains leaked at 1.
+ After patch:
+ The error path always calls hugepage_subpool_put_pages if map_chg is true, restoring used_hpages to 0.

4. Unmount the filesystem.
+ During unmount, the kernel calls unlock_or_release_subpool to clean up the subpool.
+ It checks if the subpool is free using subpool_is_free, which returns whether used_hpages is 0.
+ Before patch (Buggy):
+ Since used_hpages leaked and is 1, subpool_is_free returns false.
+ The kernel skips freeing the subpool structure, leaking the hugepage_subpool structure in kernel memory.
+ After patch:
+ Since used_hpages is 0, subpool_is_free returns true, and the subpool structure is correctly freed.

---
 subpool_leak_max_size.sh | 71 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 71 insertions(+)
subpool_leak_max_size.sh new file +71 -0
new file mode 100755
index 0000000000000..bfafa1ba074ea
@@ -0,0 +1,71 @@
+#!/bin/bash
+
+if [ "$EUID" -ne 0 ]; then
+    echo "Please run as root"
+    exit 1
+fi
+
+MNT_PATH="/tmp/mnt_hugetlb"
+FILE_PATH="$MNT_PATH/test_file"
+
+# Save original values
+orig_nr=$(cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages)
+orig_overcommit=$(cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_overcommit_hugepages)
+
+cleanup() {
+    echo "Cleaning up..."
+    rm -f "$FILE_PATH"
+    umount "$MNT_PATH" 2>/dev/null
+    rmdir "$MNT_PATH" 2>/dev/null
+    echo "$orig_nr" > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
+    echo "$orig_overcommit" > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_overcommit_hugepages
+    echo "Cleanup done."
+}
+trap cleanup EXIT
+
+# 1. Mount hugetlbfs with size=2M (1 page)
+mkdir -p "$MNT_PATH"
+if ! mount -t hugetlbfs -o size=2M none "$MNT_PATH"; then
+    echo "Failed to mount hugetlbfs"
+    exit 1
+fi
+
+# 2. Set nr_hugepages to 0, overcommit to 0
+echo 0 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
+echo 0 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_overcommit_hugepages
+
+# Check subpool usage before running
+read total free < <(stat -f -c "%b %f" "$MNT_PATH")
+used_before=$((total - free))
+echo "Before test - Subpool total blocks: $total"
+echo "Before test - Subpool free blocks:  $free"
+echo "Before test - Subpool used blocks:  $used_before"
+if [ "$used_before" -ne 0 ]; then
+    echo "ERROR: Subpool is not clean before test starts!"
+    exit 1
+fi
+
+# Run fallocate (expecting failure)
+echo "Running fallocate (expecting failure)..."
+if fallocate -l 2M "$FILE_PATH" 2>/dev/null; then
+    echo "ERROR: fallocate succeeded but should have failed (nr_hugepages is 0)"
+    exit 1
+fi
+
+# Check subpool usage via statfs
+# %b: Total blocks
+# %f: Free blocks
+read total free < <(stat -f -c "%b %f" "$MNT_PATH")
+used=$((total - free))
+
+echo "Subpool total blocks: $total"
+echo "Subpool free blocks:  $free"
+echo "Subpool used blocks (leaked if > 0): $used"
+
+if [ "$used" -gt 0 ]; then
+    echo "RESULT: LEAK DETECTED (FAIL)"
+    exit 1
+else
+    echo "RESULT: NO LEAK (PASS)"
+    exit 0
+fi
Signature · 1 line
2.55.0.229.g6434b31f56-goog
#12 lore raw

PATCH v3 12/13 WIP: Reproducer for false restoration on shared HugeTLB mappings

In reply to Ackerley Tng

(This reproducer was hacked up and not meant to be merged.)

hugetlb_unreserve_pages() unconditionally returns reservations to the subpool (via hugepage_subpool_put_pages()). This means that regardless of whether a subpool reservation was actually used, the reservation is processed by the subpool structure.

To create a false restoration, the reproducer performs these steps:

1. Mount with min_size=2M (1 page). Global resv_hugepages becomes 1.
2. The program maps 4MB (2 pages) shared (which also grows the file to 4MB). Global resv_hugepages becomes 2 (1 from the mount, 1 new global reservation).
3. The program populates only the first page. Global resv_hugepages decrements to 1 (reservation consumed by allocation).
4. The program exits (closing VMAs/fds). For shared mappings, reservations are associated with the file inode, so they remain active. Global resv_hugepages remains 1.
5. The script truncates the file to 2MB (truncate -s 2M).
+ This synchronously triggers hugetlb_unreserve_pages() to release the
reservation of the truncated range (the unallocated 2nd page).
+ It calls hugepage_subpool_put_pages(spool, 1).
+ On Vanilla Kernel (Buggy):
+ used_hpages is 0 (not tracked).
+ used_hpages (0) < min_hpages (1) is TRUE.
+ The subpool incorrectly restores the reservation (spool->rsv_hpages
becomes 1), even though Page 0 is still allocated and satisfies the mount's minimum guarantee.
+ hugepage_subpool_put_pages() returns 0, skipping
hugetlb_acct_memory(h, -1).
+ Result: Global resv_hugepages remains stuck at 1 (Leak).
+ On Fixed Kernel:
+ used_hpages is tracked and is initially 2.
+ hugepage_subpool_put_pages(1) decrements used_hpages to 1.
+ used_hpages (1) < min_hpages (1) is FALSE.
+ The subpool does not restore the reservation.
+ hugepage_subpool_put_pages() returns 1.
+ hugetlb_acct_memory(h, -1) is called.
+ Result: Global resv_hugepages decrements to 0 (No leak).

When the filesystem is unmounted, hugetlbfs_put_super drops the subpool reference. Since the filesystem is being unmounted, the reference count drops to 0, triggering unlock_or_release_subpool.

Inside unlock_or_release_subpool, the kernel checks if the subpool is free using subpool_is_free.
+ On the buggy kernel, subpool_is_free checks if spool->rsv_hpages is equal to spool->min_hpages. Because of the phantom reservation, spool->rsv_hpages was restored to 1. Since min_hpages is 1, the check (1 == 1) returns true.
+ Since the subpool is considered free, the kernel releases the initial mount-time reservation by calling hugetlb_acct_memory to decrement resv_huge_pages by spool->min_hpages (which is 1).
+ This decrement reduces resv_huge_pages from 1 (the leaked state) to 0.

As a result, the leaked reservation is cleaned up during unmount and does not persist afterward.

---
 subpool_shared_leak.c  | 29 +++++++++++++++++
 subpool_shared_leak.sh | 86 ++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 115 insertions(+)
subpool_shared_leak.c new file +29 -0
new file mode 100644
index 0000000000000..5811e18d7f8be
@@ -0,0 +1,29 @@
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <sys/mman.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+
+#define HPAGE_SIZE (2 * 1024 * 1024)
+
+int main(int argc, char **argv) {
+    if (argc < 2) {
+        fprintf(stderr, "Usage: %s <file_path>\n", argv[0]);
+        return 1;
+    }
+    const char *file_path = argv[1];
+
+    int fd = open(file_path, O_CREAT | O_RDWR, 0666);
+    if (fd < 0) { perror("open"); return 1; }
+
+    void *addr = mmap(NULL, 2 * HPAGE_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
+    if (addr == MAP_FAILED) { perror("mmap"); close(fd); return 1; }
+
+    *(volatile char *)addr = 1; // Allocate 1st page only. 2nd page remains unallocated (but reserved).
+
+    munmap(addr, 2 * HPAGE_SIZE);
+    close(fd);
+
+    return 0;
+}
subpool_shared_leak.sh new file +86 -0

86 lines not on this page · raw message

Signature · 1 line
2.55.0.229.g6434b31f56-goog
#13 lore raw

PATCH v3 13/13 WIP: Reproducer for out_put_pages subpool reserve leakage

In reply to Ackerley Tng

Add a highly precise C reproducer and accompanying bash execution script to exercise, validate, and stress-test the out_put_pages error path rollback semantics in hugetlb_reserve_pages().

How it works:
1. The bash script sets the system-wide HugeTLB pool to a highly constrained baseline of exactly nr_hugepages = 1 and nr_overcommit_hugepages = 0.
2. It mounts a hugetlbfs instance with -o pagesize=2M,min_size=2M,size=4M, which causes the kernel to immediately consume the 1 available global page as the subpool's mount-time minimum size reserve (rsv_hugepages becomes 1).
3. The C reproducer then attempts a shared mmap() for 4M (2 pages).
- hugepage_subpool_get_pages() requests 2 pages, sees 1 reserved, and requests 1 additional global page.
- hugetlb_acct_memory() attempts to secure that global page but immediately fails with -ENOMEM because the pool is exhausted.
- The kernel jumps to the out_put_pages error path, calling hugepage_subpool_put_pages() to symmetrically roll back the reservation.
4. The script verifies that the subpool successfully retains its 1reserved page during the failure and returns it cleanly to the global pool upon unmount, proving that no underflow, double-free, or reserve leakage occurs in the out_put_pages boundary path.

---
 hugetlb_reserve_pages_out_put_pages.c  |  49 +++++++++++
 hugetlb_reserve_pages_out_put_pages.sh | 153 +++++++++++++++++++++++++++++++++
 2 files changed, 202 insertions(+)
hugetlb_reserve_pages_out_put_pages.c new file +49 -0
new file mode 100644
index 0000000000000..9e63fc8997d57
@@ -0,0 +1,49 @@
+// SPDX-License-Identifier: GPL-2.0
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <sys/mman.h>
+#include <fcntl.h>
+#include <err.h>
+#include <errno.h>
+
+int main(int argc, char **argv)
+{
+	const char *file_path;
+	size_t size;
+	int fd;
+	void *addr;
+
+	if (argc < 3) {
+		fprintf(stderr, "Usage: %s <hugetlbfs_file> <size_in_bytes>\n",
+			argv[0]);
+		return 1;
+	}
+
+	file_path = argv[1];
+	size = strtoull(argv[2], NULL, 0);
+
+	fd = open(file_path, O_CREAT | O_RDWR, 0666);
+	if (fd < 0)
+		err(1, "open");
+
+	printf("Attempting to mmap %zu bytes shared on %s...\n", size,
+	       file_path);
+	addr = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
+	if (addr == MAP_FAILED) {
+		if (errno == ENOMEM) {
+			printf("mmap failed with ENOMEM as expected.\n");
+			close(fd);
+			return 0;
+		}
+		perror("mmap failed with unexpected error");
+		close(fd);
+		return 1;
+	}
+
+	printf("ERROR: mmap SUCCEEDED unexpectedly at %p\n", addr);
+	munmap(addr, size);
+	close(fd);
+	return 1;
+}
hugetlb_reserve_pages_out_put_pages.sh new file +153 -0

153 lines not on this page · raw message

Signature · 1 line
2.55.0.229.g6434b31f56-goog