FAILED: patch "[PATCH] mm/hugetlb: unshare page tables during VMA split, not before" failed to apply to 6.6-stable tree

The patch below does not apply to the 6.6-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.

To reproduce the conflict and resubmit, you may use the following commands:

git fetch https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ linux-6.6.y
git checkout FETCH_HEAD
git cherry-pick -x 081056dc00a27bccb55ccc3c6f230a3d5fd3f7e0
# <resolve conflicts, build, test, etc.>
git commit -s
git send-email --to '<[email protected]>' --in-reply-to '2025062040-detection-sufferer-7865@gregkh' --subject-prefix 'PATCH 6.6.y' HEAD^..

Possible dependencies:

thanks,

greg k-h

------------------ original commit in Linus's tree ------------------

From 081056dc00a27bccb55ccc3c6f230a3d5fd3f7e0 Mon Sep 17 00:00:00 2001
From: Jann Horn <[email protected]>
Date: Tue, 27 May 2025 23:23:53 +0200
Subject: [PATCH] mm/hugetlb: unshare page tables during VMA split, not before

Currently, __split_vma() triggers hugetlb page table unsharing through vm_ops->may_split(). This happens before the VMA lock and rmap locks are taken - which is too early, it allows racing VMA-locked page faults in our process and racing rmap walks from other processes to cause page tables to be shared again before we actually perform the split.

Fix it by explicitly calling into the hugetlb unshare logic from __split_vma() in the same place where THP splitting also happens. At that point, both the VMA and the rmap(s) are write-locked.

An annoying detail is that we can now call into the helper hugetlb_unshare_pmds() from two different locking contexts:

1. from hugetlb_split(), holding:
- mmap lock (exclusively)
- VMA lock
- file rmap lock (exclusively)
2. hugetlb_unshare_all_pmds(), which I think is designed to be able to call us with only the mmap lock held (in shared mode), but currently only runs while holding mmap lock (exclusively) and VMA lock

Backporting note:
This commit fixes a racy protection that was introduced in commit b30c14cd6102 ("hugetlb: unshare some PMDs when splitting VMAs"); that commit claimed to fix an issue introduced in 5.13, but it should actually also go all the way back.

index 0598f36931de..42f374e828a2 100644
@@ -279,6 +279,7 @@ bool is_hugetlb_entry_migration(pte_t pte);
 bool is_hugetlb_entry_hwpoisoned(pte_t pte);
 void hugetlb_unshare_all_pmds(struct vm_area_struct *vma);
 void fixup_hugetlb_reservations(struct vm_area_struct *vma);
+void hugetlb_split(struct vm_area_struct *vma, unsigned long addr);
 
 #else /* !CONFIG_HUGETLB_PAGE */
 

@@ -476,6 +477,8 @@ static inline void fixup_hugetlb_reservations(struct vm_area_struct *vma)
 {
 }
 
+static inline void hugetlb_split(struct vm_area_struct *vma, unsigned long addr) {}
+
 #endif /* !CONFIG_HUGETLB_PAGE */
 
 #ifndef pgd_write

index 1c6595f282e5..7ebc9eb608f4 100644
@@ -539,7 +539,14 @@ __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma,
 	init_vma_prep(&vp, vma);
 	vp.insert = new;
 	vma_prepare(&vp);
+
+	/*
+	 * Get rid of huge pages and shared page tables straddling the split
+	 * boundary.
+	 */
 	vma_adjust_trans_huge(vma, vma->vm_start, addr, NULL);
+	if (is_vm_hugetlb_page(vma))
+		hugetlb_split(vma, addr);
 
 	if (new_below) {
 		vma->vm_start = addr;

index 441feb21aa5a..4505b1c31be1 100644
@@ -932,6 +932,8 @@ static inline void vma_adjust_trans_huge(struct vm_area_struct *vma,
 	(void)next;
 }
 
+static inline void hugetlb_split(struct vm_area_struct *, unsigned long) {}
+
 static inline void vma_iter_free(struct vma_iterator *vmi)
 {
 	mas_destroy(&vmi->mas);

Currently, __split_vma() triggers hugetlb page table unsharing through vm_ops->may_split(). This happens before the VMA lock and rmap locks are taken - which is too early, it allows racing VMA-locked page faults in our process and racing rmap walks from other processes to cause page tables to be shared again before we actually perform the split.

Fix it by explicitly calling into the hugetlb unshare logic from __split_vma() in the same place where THP splitting also happens. At that point, both the VMA and the rmap(s) are write-locked.

An annoying detail is that we can now call into the helper hugetlb_unshare_pmds() from two different locking contexts:

1. from hugetlb_split(), holding:
- mmap lock (exclusively)
- VMA lock
- file rmap lock (exclusively)
2. hugetlb_unshare_all_pmds(), which I think is designed to be able to call us with only the mmap lock held (in shared mode), but currently only runs while holding mmap lock (exclusively) and VMA lock

Backporting note:
This commit fixes a racy protection that was introduced in commit b30c14cd6102 ("hugetlb: unshare some PMDs when splitting VMAs"); that commit claimed to fix an issue introduced in 5.13, but it should actually also go all the way back.

---
 include/linux/hugetlb.h |  3 +++
 mm/hugetlb.c            | 60 ++++++++++++++++++++++++++++++-----------
 mm/mmap.c               |  6 +++++
 3 files changed, 53 insertions(+), 16 deletions(-)

index fc2023d07f69..8b051b8c4034 100644
@@ -281,6 +281,7 @@ long hugetlb_change_protection(struct vm_area_struct *vma,
 
 bool is_hugetlb_entry_migration(pte_t pte);
 void hugetlb_unshare_all_pmds(struct vm_area_struct *vma);
+void hugetlb_split(struct vm_area_struct *vma, unsigned long addr);
 
 #else /* !CONFIG_HUGETLB_PAGE */
 

@@ -491,6 +492,8 @@ static inline vm_fault_t hugetlb_fault(struct mm_struct *mm,
 
 static inline void hugetlb_unshare_all_pmds(struct vm_area_struct *vma) { }
 
+static inline void hugetlb_split(struct vm_area_struct *vma, unsigned long addr) {}
+
 #endif /* !CONFIG_HUGETLB_PAGE */
 /*
  * hugepages at page global directory. If arch support

index 03a24cb3951d..a9c70001e456 100644
@@ -2402,7 +2402,13 @@ int __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma,
 	init_vma_prep(&vp, vma);
 	vp.insert = new;
 	vma_prepare(&vp);
+	/*
+	 * Get rid of huge pages and shared page tables straddling the split
+	 * boundary.
+	 */
 	vma_adjust_trans_huge(vma, vma->vm_start, addr, 0);
+	if (is_vm_hugetlb_page(vma))
+		hugetlb_split(vma, addr);
 
 	if (new_below) {
 		vma->vm_start = addr;

2.50.0.rc2.701.gf1e915cc24-goog

[ Sasha's backport helper bot ]

Hi,

Summary of potential issues:
⚠️ Found matching upstream commit but patch is missing proper reference to it

Found matching upstream commit: 081056dc00a27bccb55ccc3c6f230a3d5fd3f7e0

Status in newer kernel trees:
6.15.y | Present (different SHA1: 4391f7f0f242)
6.12.y | Not found

Note: The patch differs from the upstream commit:
---
1: 081056dc00a27 ! 1: e8ec25e3d047b mm/hugetlb: unshare page tables during VMA split, not before
@@ Commit message

+ [stable backport: code got moved from mmap.c to vma.c]
+ Signed-off-by: Jann Horn <[email protected]>

## include/linux/hugetlb.h ##
-@@ include/linux/hugetlb.h: bool is_hugetlb_entry_migration(pte_t pte);
- bool is_hugetlb_entry_hwpoisoned(pte_t pte);
+@@ include/linux/hugetlb.h: long hugetlb_change_protection(struct vm_area_struct *vma, +
+ bool is_hugetlb_entry_migration(pte_t pte);
void hugetlb_unshare_all_pmds(struct vm_area_struct *vma);
- void fixup_hugetlb_reservations(struct vm_area_struct *vma);
+void hugetlb_split(struct vm_area_struct *vma, unsigned long addr);

#else /* !CONFIG_HUGETLB_PAGE */

-@@ include/linux/hugetlb.h: static inline void fixup_hugetlb_reservations(struct vm_area_struct *vma)
- {
- }
+@@ include/linux/hugetlb.h: static inline vm_fault_t hugetlb_fault(struct mm_struct *mm, +
+ static inline void hugetlb_unshare_all_pmds(struct vm_area_struct *vma) { }

+static inline void hugetlb_split(struct vm_area_struct *vma, unsigned long addr) {} +
#endif /* !CONFIG_HUGETLB_PAGE */
-
- #ifndef pgd_write
+ /*
+ * hugepages at page global directory. If arch support

## mm/hugetlb.c ##
@@ mm/hugetlb.c: static void hugetlb_vma_lock_free(struct vm_area_struct *vma);
@@ mm/hugetlb.c: static void hugetlb_vma_lock_free(struct vm_area_struct *vma);
+ unsigned long start, unsigned long end, bool take_locks);
static struct resv_map *vma_resv_map(struct vm_area_struct *vma);

- static void hugetlb_free_folio(struct folio *folio)
+ static inline bool subpool_is_free(struct hugepage_subpool *spool)
@@ mm/hugetlb.c: static int hugetlb_vm_op_split(struct vm_area_struct *vma, unsigned long addr)
{
if (addr & ~(huge_page_mask(hstate_vma(vma))))
@@ mm/hugetlb.c: static int hugetlb_vm_op_split(struct vm_area_struct *vma, unsigne

static unsigned long hugetlb_vm_op_pagesize(struct vm_area_struct *vma)
@@ mm/hugetlb.c: void move_hugetlb_state(struct folio *old_folio, struct folio *new_folio, int re
- spin_unlock_irq(&hugetlb_lock);
+ }
}

+/*
@@ mm/hugetlb.c: static void hugetlb_unshare_pmds(struct vm_area_struct *vma,
+ /* take_locks = */ true);
}

- /*
+ #ifdef CONFIG_CMA

- ## mm/vma.c ##
-@@ mm/vma.c: __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma,
+ ## mm/mmap.c ##
+@@ mm/mmap.c: int __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma,
init_vma_prep(&vp, vma);
vp.insert = new;
vma_prepare(&vp);
-+
+ /*
+ * Get rid of huge pages and shared page tables straddling the split
+ * boundary.
+ */
- vma_adjust_trans_huge(vma, vma->vm_start, addr, NULL);
+ vma_adjust_trans_huge(vma, vma->vm_start, addr, 0);
+ if (is_vm_hugetlb_page(vma))
+ hugetlb_split(vma, addr);

if (new_below) {
vma->vm_start = addr;
-
- ## tools/testing/vma/vma_internal.h ##
-@@ tools/testing/vma/vma_internal.h: static inline void vma_adjust_trans_huge(struct vm_area_struct *vma,
- (void)next;
- }
-
-+static inline void hugetlb_split(struct vm_area_struct *, unsigned long) {}
-+
- static inline void vma_iter_free(struct vma_iterator *vmi)
- {
- mas_destroy(&vmi->mas);
---

Results of testing on various branches:

| Branch                    | Patch Apply | Build Test |
|---------------------------|-------------|------------|
| stable/linux-6.6.y        |  Success    |  Success   |

Whoops, sorry about that, I didn't realize "git cherry-pick" needs that "-x" flag to record the commit ID being backported. I see you fixed it up, thanks!