FAILED: patch "[PATCH] mm: fix incorrect vm_flags usage when checking allowable" failed to apply to 6.12-stable tree

The patch below does not apply to the 6.12-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.

To reproduce the conflict and resubmit, you may use the following commands:

git fetch https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ linux-6.12.y
git checkout FETCH_HEAD
git cherry-pick -x 2fd4e7693674b17807a6d082feb01a3fbf86f5f8
# <resolve conflicts, build, test, etc.>
git commit -s
git send-email --to '<[email protected]>' --in-reply-to '2026090905-parole-radiated-b379@gregkh' --subject-prefix 'PATCH 6.12.y' 'HEAD^..'

Possible dependencies:

thanks,

greg k-h

------------------ original commit in Linus's tree ------------------

From 2fd4e7693674b17807a6d082feb01a3fbf86f5f8 Mon Sep 17 00:00:00 2001
From: Baolin Wang <[email protected]>
Date: Tue, 18 Aug 2026 10:47:26 +0800
Subject: [PATCH] mm: fix incorrect vm_flags usage when checking allowable
orders for tmpfs

Lance reported that when nothing else causes the mm to be considered for khugepaged collapse, an MADV_HUGEPAGE-advised tmpfs VMA alone does not trigger scanning.

After commit 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled() into shmem_allowable_huge_orders()"), the shmem/tmpfs allowable order check reads vma->flags directly. However, when MADV_HUGEPAGE is handled, khugepaged_enter_vma() is called before the VMA's flags have been updated, so the check uses stale flags and incorrectly rejects the VMA for collapse. As a result, khugepaged does not collapse the tmpfs file into PMD order in time.

Fix this by calling khugepaged_enter_vma() with the new VMA flags in madvise_update_vma(). Meanwhile we can remove the khugepaged_enter_vma() in hugepage_madvise().

index 11ff98d55c76..75639298efc2 100644
@@ -454,12 +454,6 @@ int hugepage_madvise(struct vm_area_struct *vma,
 	case MADV_HUGEPAGE:
 		*vm_flags &= ~VM_NOHUGEPAGE;
 		*vm_flags |= VM_HUGEPAGE;
-		/*
-		 * If the vma become good for khugepaged to scan,
-		 * register it here without waiting a page fault that
-		 * may not happen any time soon.
-		 */
-		khugepaged_enter_vma(vma, *vm_flags);
 		break;
 	case MADV_NOHUGEPAGE:
 		*vm_flags &= ~VM_HUGEPAGE;

index 96f2387b2f46..eeee82cf2b3f 100644
@@ -178,6 +178,14 @@ static int madvise_update_vma(vm_flags_t new_flags,
 	/* vm_flags is protected by the mmap_lock held in write mode. */
 	vma_start_write(vma);
 	vma->flags = new_vma_flags;
+	/*
+	 * If the vma become good for khugepaged to scan,
+	 * register it here without waiting a page fault that
+	 * may not happen any time soon.
+	 */
+	if (vma_flags_test(&new_vma_flags, VMA_HUGEPAGE_BIT))
+		khugepaged_enter_vma(vma, vma_flags_to_legacy(new_vma_flags));
+
 	if (set_new_anon_name)
 		return replace_anon_vma_name(vma, anon_name);
 

From: Baolin Wang <[email protected]>

[ Upstream commit 2fd4e7693674b17807a6d082feb01a3fbf86f5f8 ]

Lance reported that when nothing else causes the mm to be considered for khugepaged collapse, an MADV_HUGEPAGE-advised tmpfs VMA alone does not trigger scanning.

After commit 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled() into shmem_allowable_huge_orders()"), the shmem/tmpfs allowable order check reads vma->flags directly. However, when MADV_HUGEPAGE is handled, khugepaged_enter_vma() is called before the VMA's flags have been updated, so the check uses stale flags and incorrectly rejects the VMA for collapse. As a result, khugepaged does not collapse the tmpfs file into PMD order in time.

Fix this by calling khugepaged_enter_vma() with the new VMA flags in madvise_update_vma(). Meanwhile we can remove the khugepaged_enter_vma() in hugepage_madvise().

---
 mm/khugepaged.c | 6 ------
 mm/madvise.c    | 8 ++++++++
 2 files changed, 8 insertions(+), 6 deletions(-)

index a6b2c7d7d73b6..84bacd7e19620 100644
@@ -369,12 +369,6 @@ int hugepage_madvise(struct vm_area_struct *vma,
 #endif
 		*vm_flags &= ~VM_NOHUGEPAGE;
 		*vm_flags |= VM_HUGEPAGE;
-		/*
-		 * If the vma become good for khugepaged to scan,
-		 * register it here without waiting a page fault that
-		 * may not happen any time soon.
-		 */
-		khugepaged_enter_vma(vma, *vm_flags);
 		break;
 	case MADV_NOHUGEPAGE:
 		*vm_flags &= ~VM_HUGEPAGE;

index d2b315993a1c0..4abc466d37505 100644
@@ -158,6 +158,14 @@ static int madvise_update_vma(struct vm_area_struct *vma,
 	/* vm_flags is protected by the mmap_lock held in write mode. */
 	vma_start_write(vma);
 	vm_flags_reset(vma, new_flags);
+	/*
+	 * If the vma become good for khugepaged to scan,
+	 * register it here without waiting a page fault that
+	 * may not happen any time soon.
+	 */
+	if (new_flags & VM_HUGEPAGE)
+		khugepaged_enter_vma(vma, new_flags);
+
 	if (!vma->vm_file || vma_is_anon_shmem(vma)) {
 		error = replace_anon_vma_name(vma, anon_name);
 		if (error)

2.53.0