mm/hugetlb: fix resv_huge_pages double decrement in memfd error path

From: Hongfu Li <[email protected]>

alloc_hugetlb_folio_reserve() decrements h->resv_huge_pages when dequeuing a folio, but unlike the use_global_reservation handling in hugetlb_alloc_folio(), it does not set HPageRestoreReserve on the folio.

Its sole caller memfd_alloc_folio() pre-allocates a reservation via hugetlb_reserve_pages() before allocating. When hugetlb_add_to_page_cache() fails, folio_put() drops the folio without HPageRestoreReserve set, so free_huge_folio() does not restore the reservation. The subsequent hugetlb_unreserve_pages() on the err_unresv path decrements the counter a second time, leaving resv_huge_pages off by one for every failed allocation.

Set HPageRestoreReserve when consuming the reservation in alloc_hugetlb_folio_reserve(). On the error path, free_huge_folio() then restores the reservation before hugetlb_unreserve_pages() releases it. The success path is unaffected, as hugetlb_add_to_page_cache() clears the flag once the folio is added to the page cache.

Fixes 26a8ea80929c ("mm/hugetlb: fix memfd_pin_folios resv_huge_pages leak")

Signed-off-by Hongfu Li <[email protected]>

From: Hongfu Li <[email protected]>

alloc_hugetlb_folio_reserve() decrements h->resv_huge_pages when dequeuing a folio, but unlike the use_global_reservation handling in hugetlb_alloc_folio(), it does not set HPageRestoreReserve on the folio.

Its sole caller memfd_alloc_folio() pre-allocates a reservation via hugetlb_reserve_pages() before allocating. When hugetlb_add_to_page_cache() fails, folio_put() drops the folio without HPageRestoreReserve set, so free_huge_folio() does not restore the reservation. The subsequent hugetlb_unreserve_pages() on the err_unresv path decrements the counter a second time, leaving resv_huge_pages off by one for every failed allocation.

Set HPageRestoreReserve when consuming the reservation in alloc_hugetlb_folio_reserve(). On the error path, free_huge_folio() then restores the reservation before hugetlb_unreserve_pages() releases it. The success path is unaffected, as hugetlb_add_to_page_cache() clears the flag once the folio is added to the page cache.

...

mm/hugetlb.c +0 -0

@@ -2178,8 +2178,10 @@ struct folio *alloc_hugetlb_folio_reserve(struct hstate *h, int preferred_nid,
 

Quoted 34 lines · On Tue, 25 Aug 2026 10:10:13 +0800 Hongfu Li <[email protected]> wrote:

From: Hongfu Li <[email protected]>

alloc_hugetlb_folio_reserve() decrements h->resv_huge_pages when dequeuing a folio, but unlike the use_global_reservation handling in hugetlb_alloc_folio(), it does not set HPageRestoreReserve on the folio.

Its sole caller memfd_alloc_folio() pre-allocates a reservation via hugetlb_reserve_pages() before allocating. When hugetlb_add_to_page_cache() fails, folio_put() drops the folio without HPageRestoreReserve set, so free_huge_folio() does not restore the reservation. The subsequent hugetlb_unreserve_pages() on the err_unresv path decrements the counter a second time, leaving resv_huge_pages off by one for every failed allocation.

Set HPageRestoreReserve when consuming the reservation in alloc_hugetlb_folio_reserve(). On the error path, free_huge_folio() then restores the reservation before hugetlb_unreserve_pages() releases it. The success path is unaffected, as hugetlb_add_to_page_cache() clears the flag once the folio is added to the page cache.

...

mm/hugetlb.c +3 -1

@@ -2178,8 +2178,10 @@ struct folio *alloc_hugetlb_folio_reserve(struct hstate *h, int preferred_nid,
 
  	folio = dequeue_hugetlb_folio_nodemask(h, gfp_mask, preferred_nid,
  					       nmask);
-	if (folio)
+	if (folio) {
+		folio_set_hugetlb_restore_reserve(folio);
  		h->resv_huge_pages--;
+	}
 

mm/hugetlb.c struct folio *alloc_hugetlb_folio_reserve(struct hstate *h, int preferred_nid, · line 2187

Quoted 2 lines

mm/hugetlb.c +0 -0

  	spin_unlock_irq(&hugetlb_lock);
  	return folio;

Thanks. I pasted an AI-generated test case which might demonstrate this bug. Requires fault-injection so I won't add cc:stable.

Also, Sashiko might have found an accounting issue in the nearby code (Sashiko doesn't like hugetlb.c):

	https://sashiko.dev/#/patchset/[email protected]

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <linux/memfd.h>

/* Read resv_hugepages from sysfs */

static long get_resv_hugepages(void) {
     FILE *f = fopen("/sys/kernel/mm/hugepages/hugepages-2048kB/resv_hugepages", "r");
     if (!f) return -1;
     long val = -1;
     fscanf(f, "%ld", &val);
     fclose(f);
     return val;
}

int main(void) {
     long orig_resv = get_resv_hugepages();
     printf("[1] Initial resv_hugepages: %ld\n", orig_resv);

     /* Enable fail_function for hugetlb_add_to_page_cache via debugfs */
     system("echo hugetlb_add_to_page_cache > /sys/kernel/debug/fail_function/inject");
     system("echo 100 > /sys/kernel/debug/fail_function/probability");

     /* Create memfd and attempt write to allocate hugetlb folio */
     int fd = memfd_create("test_memfd", MFD_HUGETLB);
     if (fd >= 0) {
         /* Force allocation path which triggers memfd_alloc_folio() */
         ftruncate(fd, 2 * 1024 * 1024);
         write(fd, "a", 1);
         close(fd);
     }

     /* Disable error injection */
     system("echo > /sys/kernel/debug/fail_function/inject");

     long post_resv = get_resv_hugepages();
     printf("[2] Post-failure resv_hugepages: %ld\n", post_resv);

     if (post_resv < orig_resv) {
         printf("[!] BUG DEMONSTRATED: resv_hugepages double-decremented by %ld!\n",
                orig_resv - post_resv);
     }

     return 0;
}