Skip to contentSkip to conversation

mm/khugepaged: fix dirty page handling for MADV_COLLAPSE

PATCH Superseded Shivank Garg 2 participants 10 messages Dec 15, 2025 – Jan 18

Subsystems (3)

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v4 cover mm/khugepaged: fix dirty page handling for MADV_COLLAPSE

MADV_COLLAPSE on file-backed mappings fails with -EINVAL when TEXT pages are dirty. This affects scenarios like package/container updates or executing binaries immediately after writing them, etc.

The issue is that collapse_file() triggers async writeback and returns SCAN_FAIL (maps to -EINVAL), expecting khugepaged to revisit later. But MADV_COLLAPSE is synchronous and userspace expects immediate success or a clear retry signal.

Reproduction:
- Compile or copy 2MB-aligned executable to XFS/ext4 FS
- Call MADV_COLLAPSE on .text section
- First call fails with -EINVAL (text pages dirty from copy)
- Second call succeeds (async writeback completed)

Issue Report:
https://lore.kernel.org/all/[email protected]

Patch applies cleanly on mm-new (https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git/log/?h=mm-new)

Changelog:
V4:
- Rebase on mm-new
- Fix spurious blank line (Lance)

V3:
- https://lore.kernel.org/all/[email protected]
- Reordered patches: Enum definition comes first as the retry logic depends on it
- Renamed SCAN_PAGE_NOT_CLEAN to SCAN_PAGE_DIRTY_OR_WRITEBACK (Dev, Lance, David)
- Changed writeback logic: Only trigger synchronous writeback and retry if the initial collapse attempt failed specifically due to dirty/writeback pages, rather than blindly flushing all file-backed VMAs (David)
- Added proper file reference counting (get_file/fput) around the unlock window to prevent UAF (Lance)

V2:
- https://lore.kernel.org/all/[email protected]
- Move writeback to madvise_collapse() (better abstraction, proper mmap_lock handling and does VMA revalidation after I/O) (Lorenzo)
- Rename to SCAN_PAGE_DIRTY to SCAN_PAGE_NOT_CLEAN and extend its use for all dirty/writeback folio cases that previously returned incorrect results (Dev)

V1: https://lore.kernel.org/all/[email protected]

Thanks,

Shivank Garg (2):
mm/khugepaged: map dirty/writeback pages failures to EAGAIN mm/khugepaged: retry with sync writeback for MADV_COLLAPSE

 include/trace/events/huge_memory.h |  3 +-
 mm/khugepaged.c                    | 48 ++++++++++++++++++++++++++++--
 2 files changed, 47 insertions(+), 4 deletions(-)

base-commit: d0a24447990a9d8212bfb3a692d59efa74ce9f86

Signature · 1 line
2.43.0
#1 lore raw

PATCH v4 1/2 mm/khugepaged: map dirty/writeback pages failures to EAGAIN

When collapse_file encounters dirty or writeback pages in file-backed mappings, it currently returns SCAN_FAIL which maps to -EINVAL. This is misleading as EINVAL suggests invalid arguments, whereas dirty/writeback pages represent transient conditions that may resolve on retry.

Introduce SCAN_PAGE_DIRTY_OR_WRITEBACK to cover both dirty and writeback states, mapping it to -EAGAIN. For MADV_COLLAPSE, this provides userspace with a clear signal that retry may succeed after writeback completes. For khugepaged, this is harmless as it will naturally revisit the range during periodic scans after async writeback completes.

---
 include/trace/events/huge_memory.h | 3 ++-
 mm/khugepaged.c                    | 8 +++++---
 2 files changed, 7 insertions(+), 4 deletions(-)
include/trace/events/huge_memory.h +2 -1
index 4cde53b45a85..4e41bff31888 100644
@@ -37,7 +37,8 @@
 	EM( SCAN_PAGE_HAS_PRIVATE,	"page_has_private")		\
 	EM( SCAN_STORE_FAILED,		"store_failed")			\
 	EM( SCAN_COPY_MC,		"copy_poisoned_page")		\
-	EMe(SCAN_PAGE_FILLED,		"page_filled")
+	EM( SCAN_PAGE_FILLED,		"page_filled")			\
+	EMe(SCAN_PAGE_DIRTY_OR_WRITEBACK, "page_dirty_or_writeback")
 
 #undef EM
 #undef EMe
mm/khugepaged.c +5 -3
index 97d1b2824386..219dfa2e523c 100644
@@ -58,6 +58,7 @@ enum scan_result {
 	SCAN_STORE_FAILED,
 	SCAN_COPY_MC,
 	SCAN_PAGE_FILLED,
+	SCAN_PAGE_DIRTY_OR_WRITEBACK,
 };
 
 #define CREATE_TRACE_POINTS
@@ -1967,11 +1968,11 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr,
 				 */
 				xas_unlock_irq(&xas);
 				filemap_flush(mapping);
-				result = SCAN_FAIL;
+				result = SCAN_PAGE_DIRTY_OR_WRITEBACK;
 				goto xa_unlocked;
 			} else if (folio_test_writeback(folio)) {
 				xas_unlock_irq(&xas);
-				result = SCAN_FAIL;
+				result = SCAN_PAGE_DIRTY_OR_WRITEBACK;
 				goto xa_unlocked;
 			} else if (folio_trylock(folio)) {
 				folio_get(folio);
@@ -2018,7 +2019,7 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr,
 			 * folio is dirty because it hasn't been flushed
 			 * since first write.
 			 */
-			result = SCAN_FAIL;
+			result = SCAN_PAGE_DIRTY_OR_WRITEBACK;
 			goto out_unlock;
 		}
 
@@ -2747,6 +2748,7 @@ static int madvise_collapse_errno(enum scan_result r)
 	case SCAN_PAGE_LRU:
 	case SCAN_DEL_PAGE_LRU:
 	case SCAN_PAGE_FILLED:
+	case SCAN_PAGE_DIRTY_OR_WRITEBACK:
 		return -EAGAIN;
 	/*
 	 * Other: Trying again likely not to succeed / error intrinsic to
Signature · 1 line
2.43.0
maintainer Acked-by on v4 1/2
#2 lore raw

Re: [PATCH V4 1/2] mm/khugepaged: map dirty/writeback pages failures to EAGAIN

Quoted 11 lines · Shivank Garg

When collapse_file encounters dirty or writeback pages in file-backed mappings, it currently returns SCAN_FAIL which maps to -EINVAL. This is misleading as EINVAL suggests invalid arguments, whereas dirty/writeback pages represent transient conditions that may resolve on retry.

Introduce SCAN_PAGE_DIRTY_OR_WRITEBACK to cover both dirty and writeback states, mapping it to -EAGAIN. For MADV_COLLAPSE, this provides userspace with a clear signal that retry may succeed after writeback completes. For khugepaged, this is harmless as it will naturally revisit the range during periodic scans after async writeback completes.

Quoted 8 lines
Signature · 3 lines
Cheers

David
#3 lore raw

PATCH v4 2/2 mm/khugepaged: retry with sync writeback for MADV_COLLAPSE

In reply to Shivank Garg

When MADV_COLLAPSE is called on file-backed mappings (e.g., executable text sections), the pages may still be dirty from recent writes. collapse_file() will trigger async writeback and fail with SCAN_PAGE_DIRTY_OR_WRITEBACK (-EAGAIN).

MADV_COLLAPSE is a synchronous operation where userspace expects immediate results. If the collapse fails due to dirty pages, perform synchronous writeback on the specific range and retry once.

This avoids spurious failures for freshly written executables while avoiding unnecessary synchronous I/O for mappings that are already clean.

Fixes 34488399fa08 ("mm/madvise: add file and shmem support to MADV_COLLAPSE")
---
 mm/khugepaged.c | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
mm/khugepaged.c +40 -0
index 219dfa2e523c..6c8c35d3e0c9 100644
@@ -22,6 +22,7 @@
 #include <linux/dax.h>
 #include <linux/ksm.h>
 #include <linux/pgalloc.h>
+#include <linux/backing-dev.h>
 
 #include <asm/tlb.h>
 #include "internal.h"
@@ -2787,9 +2788,11 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
 	hend = end & HPAGE_PMD_MASK;
 
 	for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
+		bool retried = false;
 		int result = SCAN_FAIL;
 
 		if (!mmap_locked) {
+retry:
 			cond_resched();
 			mmap_read_lock(mm);
 			mmap_locked = true;
@@ -2819,6 +2822,43 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
 		if (!mmap_locked)
 			*lock_dropped = true;
 
+		/*
+		 * If the file-backed VMA has dirty pages, the scan triggers
+		 * async writeback and returns SCAN_PAGE_DIRTY_OR_WRITEBACK.
+		 * Since MADV_COLLAPSE is sync, we force sync writeback and
+		 * retry once.
+		 */
+		if (result == SCAN_PAGE_DIRTY_OR_WRITEBACK && !retried) {
+			/*
+			 * File scan drops the lock. We must re-acquire it to
+			 * safely inspect the VMA and hold the file reference.
+			 */
+			if (!mmap_locked) {
+				cond_resched();
+				mmap_read_lock(mm);
+				mmap_locked = true;
+				result = hugepage_vma_revalidate(mm, addr, false, &vma, cc);
+				if (result != SCAN_SUCCEED)
+					goto handle_result;
+			}
+
+			if (!vma_is_anonymous(vma) && vma->vm_file &&
+			    mapping_can_writeback(vma->vm_file->f_mapping)) {
+				struct file *file = get_file(vma->vm_file);
+				pgoff_t pgoff = linear_page_index(vma, addr);
+				loff_t lstart = (loff_t)pgoff << PAGE_SHIFT;
+				loff_t lend = lstart + HPAGE_PMD_SIZE - 1;
+
+				mmap_read_unlock(mm);
+				mmap_locked = false;
+				*lock_dropped = true;
+				filemap_write_and_wait_range(file->f_mapping, lstart, lend);
+				fput(file);
+				retried = true;
+				goto retry;
+			}
+		}
+
 handle_result:
 		switch (result) {
 		case SCAN_SUCCEED:
Signature · 1 line
2.43.0
#4 lore raw

Re: [PATCH V4 2/2] mm/khugepaged: retry with sync writeback for MADV_COLLAPSE

Quoted 35 lines · Shivank Garg

When MADV_COLLAPSE is called on file-backed mappings (e.g., executable text sections), the pages may still be dirty from recent writes. collapse_file() will trigger async writeback and fail with SCAN_PAGE_DIRTY_OR_WRITEBACK (-EAGAIN).

MADV_COLLAPSE is a synchronous operation where userspace expects immediate results. If the collapse fails due to dirty pages, perform synchronous writeback on the specific range and retry once.

This avoids spurious failures for freshly written executables while avoiding unnecessary synchronous I/O for mappings that are already clean.

Fixes 34488399fa08 ("mm/madvise: add file and shmem support to MADV_COLLAPSE")
---
  mm/khugepaged.c | 40 ++++++++++++++++++++++++++++++++++++++++
  1 file changed, 40 insertions(+)
mm/khugepaged.c +1 -0
index 219dfa2e523c..6c8c35d3e0c9 100644
@@ -22,6 +22,7 @@
  #include <linux/dax.h>
  #include <linux/ksm.h>
  #include <linux/pgalloc.h>
+#include <linux/backing-dev.h>
 
  #include <asm/tlb.h>
  #include "internal.h"

mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2795

Quoted 8 lines
mm/khugepaged.c +2 -0
@@ -2787,9 +2788,11 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
  	hend = end & HPAGE_PMD_MASK;
 
  	for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
+		bool retried = false;
  		int result = SCAN_FAIL;
 
  		if (!mmap_locked) {
+retry:

Jumping into an if block is nasty :)

Quoted 34 lines
mm/khugepaged.c +27 -0
  			cond_resched();
  			mmap_read_lock(mm);
  			mmap_locked = true;
@@ -2819,6 +2822,43 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
  		if (!mmap_locked)
  			*lock_dropped = true;
 
+		/*
+		 * If the file-backed VMA has dirty pages, the scan triggers
+		 * async writeback and returns SCAN_PAGE_DIRTY_OR_WRITEBACK.
+		 * Since MADV_COLLAPSE is sync, we force sync writeback and
+		 * retry once.
+		 */
+		if (result == SCAN_PAGE_DIRTY_OR_WRITEBACK && !retried) {
+			/*
+			 * File scan drops the lock. We must re-acquire it to
+			 * safely inspect the VMA and hold the file reference.
+			 */
+			if (!mmap_locked) {
+				cond_resched();
+				mmap_read_lock(mm);
+				mmap_locked = true;
+				result = hugepage_vma_revalidate(mm, addr, false, &vma, cc);
+				if (result != SCAN_SUCCEED)
+					goto handle_result;
+			}
+
+			if (!vma_is_anonymous(vma) && vma->vm_file &&
+			    mapping_can_writeback(vma->vm_file->f_mapping)) {
+				struct file *file = get_file(vma->vm_file);
+				pgoff_t pgoff = linear_page_index(vma, addr);
+				loff_t lstart = (loff_t)pgoff << PAGE_SHIFT;
+				loff_t lend = lstart + HPAGE_PMD_SIZE - 1;
+

mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2861

Quoted 10 lines
mm/khugepaged.c +10 -0
+				mmap_read_unlock(mm);
+				mmap_locked = false;
+				*lock_dropped = true;
+				filemap_write_and_wait_range(file->f_mapping, lstart, lend);
+				fput(file);
+				retried = true;
+				goto retry;
+			}
+		}
+

This looks a bit complicated. Can't we move that handing up, where we have most of that information already? Or am I missing something important?

mm/khugepaged.c +13 -0
index 97d1b2824386f..c7271877c5220 100644
@@ -22,6 +22,7 @@
 #include <linux/dax.h>
 #include <linux/ksm.h>
 #include <linux/pgalloc.h>
+#include <linux/backing-dev.h>
 
 #include <asm/tlb.h>
 #include "internal.h"
@@ -2786,7 +2787,9 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
 
        for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
                int result = SCAN_FAIL;
+               bool triggered_wb = false;
 
+retry:
                if (!mmap_locked) {
                        cond_resched();
                        mmap_read_lock(mm);
@@ -2809,6 +2812,16 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
                        mmap_locked = false;
                        result = hpage_collapse_scan_file(mm, addr, file, pgoff,
                                                          cc);
+
+                       if (result == SCAN_PAGE_DIRTY_OR_WRITEBACK && !triggered_wb &&
+                           mapping_can_writeback(file->f_mapping)) {
+                               loff_t lstart = (loff_t)pgoff << PAGE_SHIFT;
+                               loff_t lend = lstart + HPAGE_PMD_SIZE - 1;
+
+                               filemap_write_and_wait_range(file->f_mapping, lstart, lend);
+                               triggered_wb = true;
+                               goto retry;
+                       }
                        fput(file);
                } else {
                        result = hpage_collapse_scan_pmd(mm, vma, addr,
Signature · 3 lines
Cheers

David
#5 lore raw

Re: [PATCH V4 2/2] mm/khugepaged: retry with sync writeback for MADV_COLLAPSE

Quoted 114 lines · David Hildenbrand (Red Hat)
Quoted 34 lines · Shivank Garg

When MADV_COLLAPSE is called on file-backed mappings (e.g., executable text sections), the pages may still be dirty from recent writes. collapse_file() will trigger async writeback and fail with SCAN_PAGE_DIRTY_OR_WRITEBACK (-EAGAIN).

MADV_COLLAPSE is a synchronous operation where userspace expects immediate results. If the collapse fails due to dirty pages, perform synchronous writeback on the specific range and retry once.

This avoids spurious failures for freshly written executables while avoiding unnecessary synchronous I/O for mappings that are already clean.

Fixes 34488399fa08 ("mm/madvise: add file and shmem support to MADV_COLLAPSE")

---
  mm/khugepaged.c | 40 ++++++++++++++++++++++++++++++++++++++++

  1 file changed, 40 insertions(+)
mm/khugepaged.c +1 -0
index 219dfa2e523c..6c8c35d3e0c9 100644
@@ -22,6 +22,7 @@
   #include <linux/dax.h>
   #include <linux/ksm.h>
   #include <linux/pgalloc.h>
+#include <linux/backing-dev.h>
     #include <asm/tlb.h>
   #include "internal.h"

mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2793

Quoted 6 lines
mm/khugepaged.c +2 -0
@@ -2787,9 +2788,11 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
       hend = end & HPAGE_PMD_MASK;
         for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
+        bool retried = false;
           int result = SCAN_FAIL;
             if (!mmap_locked) {
+retry:

Jumping into an if block is nasty :)

Quoted 33 lines
mm/khugepaged.c +0 -0
               cond_resched();
               mmap_read_lock(mm);
               mmap_locked = true;
@@ -2819,6 +2822,43 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
           if (!mmap_locked)
               *lock_dropped = true;

  +        /*
+         * If the file-backed VMA has dirty pages, the scan triggers +         * async writeback and returns SCAN_PAGE_DIRTY_OR_WRITEBACK. +         * Since MADV_COLLAPSE is sync, we force sync writeback and +         * retry once.
+         */
+        if (result == SCAN_PAGE_DIRTY_OR_WRITEBACK && !retried) { +            /*
+             * File scan drops the lock. We must re-acquire it to +             * safely inspect the VMA and hold the file reference. +             */
+            if (!mmap_locked) {
+                cond_resched();
+                mmap_read_lock(mm);
+                mmap_locked = true;
+                result = hugepage_vma_revalidate(mm, addr, false, &vma, cc); +                if (result != SCAN_SUCCEED)
+                    goto handle_result;
+            }

mm/khugepaged.c +8 -0
+
+            if (!vma_is_anonymous(vma) && vma->vm_file &&
+                mapping_can_writeback(vma->vm_file->f_mapping)) {
+                struct file *file = get_file(vma->vm_file);
+                pgoff_t pgoff = linear_page_index(vma, addr);
+                loff_t lstart = (loff_t)pgoff << PAGE_SHIFT;
+                loff_t lend = lstart + HPAGE_PMD_SIZE - 1;
+

mm/khugepaged.c

Quoted 10 lines
mm/khugepaged.c +10 -0
+                mmap_read_unlock(mm);
+                mmap_locked = false;
+                *lock_dropped = true;
+                filemap_write_and_wait_range(file->f_mapping, lstart, lend);
+                fput(file);
+                retried = true;
+                goto retry;
+            }
+        }
+

This looks a bit complicated. Can't we move that handing up, where we have most of that information already? Or am I missing something important?

mm/khugepaged.c +1 -0
index 97d1b2824386f..c7271877c5220 100644
@@ -22,6 +22,7 @@
  #include <linux/dax.h>
  #include <linux/ksm.h>
  #include <linux/pgalloc.h>
+#include <linux/backing-dev.h>


 #include <asm/tlb.h>
 #include "internal.h"

mm/khugepaged.c +0 -0
@@ -2786,7 +2787,9 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,


        for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) { int result = SCAN_FAIL;
+               bool triggered_wb = false;

+retry:
                if (!mmap_locked) {
                        cond_resched(); mmap_read_lock(mm);

mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2812

Quoted 1 line
mm/khugepaged.c +0 -0
@@ -2809,6 +2812,16 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
                         mmap_locked = false;
			  *lock_dropped = true;

mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2822

Quoted 10 lines
mm/khugepaged.c +8 -0
                         result = hpage_collapse_scan_file(mm, addr, file, pgoff,
                                                           cc);
+
+                       if (result == SCAN_PAGE_DIRTY_OR_WRITEBACK && !triggered_wb &&
+                           mapping_can_writeback(file->f_mapping)) {
+                               loff_t lstart = (loff_t)pgoff << PAGE_SHIFT;
+                               loff_t lend = lstart + HPAGE_PMD_SIZE - 1;
+
+                               filemap_write_and_wait_range(file->f_mapping, lstart, lend);
+                               triggered_wb = true;
				  fput(file);

mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2827

Quoted 7 lines
mm/khugepaged.c +2 -0
+                               goto retry;
+                       }
                         fput(file);
                 } else {
                         result = hpage_collapse_scan_pmd(mm, vma, addr,

Thank you for the suggestion, this approach looks much simpler.

There are two small nits I observed:

1. In the retry loop, it is possible that we reacquire the mmap_lock and set mmap_locked to true. This can cause issues later when we do:

       if (!mmap_locked)
               *lock_dropped = true;

because the caller would no longer see that the lock was dropped earlier.

2. We need an fput() to balance the file reference taken at line 2795.

fixed patch: (tested it no longer reproduce the issue):

mm/khugepaged.c +15 -0
index 87ca577c2668..b88bbb54f109 100644
@@ -21,6 +21,7 @@
 #include <linux/shmem_fs.h>
 #include <linux/dax.h>
 #include <linux/ksm.h>
+#include <linux/backing-dev.h>
 
 #include <asm/tlb.h>
 #include <asm/pgalloc.h>
@@ -2771,7 +2772,9 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
 
 	for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
 		int result = SCAN_FAIL;
+		bool triggered_wb = false;
 
+retry:
 		if (!mmap_locked) {
 			cond_resched();
 			mmap_read_lock(mm);
@@ -2794,8 +2797,20 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
 
 			mmap_read_unlock(mm);
 			mmap_locked = false;
+			*lock_dropped = true;
 			result = hpage_collapse_scan_file(mm, addr, file, pgoff,
 							  cc);
+
+			if (result == SCAN_PAGE_DIRTY_OR_WRITEBACK && !triggered_wb &&
+			    mapping_can_writeback(file->f_mapping)) {
+				loff_t lstart = (loff_t)pgoff << PAGE_SHIFT;
+				loff_t lend = lstart + HPAGE_PMD_SIZE - 1;
+
+				filemap_write_and_wait_range(file->f_mapping, lstart, lend);
+				triggered_wb = true;
+				fput(file);
+				goto retry;
+			}
 			fput(file);
 		} else {
 			result = hpage_collapse_scan_pmd(mm, vma, addr,
#6 lore raw

Re: [PATCH V4 2/2] mm/khugepaged: retry with sync writeback for MADV_COLLAPSE

Quoted 141 lines · Shivank
Quoted 114 lines · David Hildenbrand (Red Hat)
Quoted 34 lines · Shivank Garg

When MADV_COLLAPSE is called on file-backed mappings (e.g., executable text sections), the pages may still be dirty from recent writes. collapse_file() will trigger async writeback and fail with SCAN_PAGE_DIRTY_OR_WRITEBACK (-EAGAIN).

MADV_COLLAPSE is a synchronous operation where userspace expects immediate results. If the collapse fails due to dirty pages, perform synchronous writeback on the specific range and retry once.

This avoids spurious failures for freshly written executables while avoiding unnecessary synchronous I/O for mappings that are already clean.

Fixes 34488399fa08 ("mm/madvise: add file and shmem support to MADV_COLLAPSE")
---
   mm/khugepaged.c | 40 ++++++++++++++++++++++++++++++++++++++++
   1 file changed, 40 insertions(+)
mm/khugepaged.c +1 -0
index 219dfa2e523c..6c8c35d3e0c9 100644
@@ -22,6 +22,7 @@
   #include <linux/dax.h>
   #include <linux/ksm.h>
   #include <linux/pgalloc.h>
+#include <linux/backing-dev.h>
     #include <asm/tlb.h>
   #include "internal.h"

mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2793

Quoted 6 lines
mm/khugepaged.c +2 -0
@@ -2787,9 +2788,11 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
       hend = end & HPAGE_PMD_MASK;
         for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
+        bool retried = false;
           int result = SCAN_FAIL;
             if (!mmap_locked) {
+retry:

Jumping into an if block is nasty :)

Quoted 33 lines
mm/khugepaged.c +26 -0
               cond_resched();
               mmap_read_lock(mm);
               mmap_locked = true;
@@ -2819,6 +2822,43 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
           if (!mmap_locked)
               *lock_dropped = true;
   +        /*
+         * If the file-backed VMA has dirty pages, the scan triggers
+         * async writeback and returns SCAN_PAGE_DIRTY_OR_WRITEBACK.
+         * Since MADV_COLLAPSE is sync, we force sync writeback and
+         * retry once.
+         */
+        if (result == SCAN_PAGE_DIRTY_OR_WRITEBACK && !retried) {
+            /*
+             * File scan drops the lock. We must re-acquire it to
+             * safely inspect the VMA and hold the file reference.
+             */
+            if (!mmap_locked) {
+                cond_resched();
+                mmap_read_lock(mm);
+                mmap_locked = true;
+                result = hugepage_vma_revalidate(mm, addr, false, &vma, cc);
+                if (result != SCAN_SUCCEED)
+                    goto handle_result;
+            }
+
+            if (!vma_is_anonymous(vma) && vma->vm_file &&
+                mapping_can_writeback(vma->vm_file->f_mapping)) {
+                struct file *file = get_file(vma->vm_file);
+                pgoff_t pgoff = linear_page_index(vma, addr);
+                loff_t lstart = (loff_t)pgoff << PAGE_SHIFT;
+                loff_t lend = lstart + HPAGE_PMD_SIZE - 1;
+

mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2860

Quoted 10 lines
mm/khugepaged.c +10 -0
+                mmap_read_unlock(mm);
+                mmap_locked = false;
+                *lock_dropped = true;
+                filemap_write_and_wait_range(file->f_mapping, lstart, lend);
+                fput(file);
+                retried = true;
+                goto retry;
+            }
+        }
+

This looks a bit complicated. Can't we move that handing up, where we have most of that information already? Or am I missing something important?

mm/khugepaged.c +3 -0
index 97d1b2824386f..c7271877c5220 100644
@@ -22,6 +22,7 @@
  #include <linux/dax.h>
  #include <linux/ksm.h>
  #include <linux/pgalloc.h>
+#include <linux/backing-dev.h>
 
  #include <asm/tlb.h>
  #include "internal.h"
@@ -2786,7 +2787,9 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
 
         for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
                 int result = SCAN_FAIL;
+               bool triggered_wb = false;
 
+retry:
                 if (!mmap_locked) {
                         cond_resched();
                         mmap_read_lock(mm);

mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2812

Quoted 1 line
mm/khugepaged.c +0 -0
@@ -2809,6 +2812,16 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
                         mmap_locked = false;
			  *lock_dropped = true;

mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2810

Quoted 10 lines
mm/khugepaged.c +8 -0
                         result = hpage_collapse_scan_file(mm, addr, file, pgoff,
                                                           cc);
+
+                       if (result == SCAN_PAGE_DIRTY_OR_WRITEBACK && !triggered_wb &&
+                           mapping_can_writeback(file->f_mapping)) {
+                               loff_t lstart = (loff_t)pgoff << PAGE_SHIFT;
+                               loff_t lend = lstart + HPAGE_PMD_SIZE - 1;
+
+                               filemap_write_and_wait_range(file->f_mapping, lstart, lend);
+                               triggered_wb = true;
				  fput(file);

mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2816

Quoted 7 lines
mm/khugepaged.c +2 -0
+                               goto retry;
+                       }
                         fput(file);
                 } else {
                         result = hpage_collapse_scan_pmd(mm, vma, addr,
Quoted 3 lines

Thank you for the suggestion, this approach looks much simpler.

There are two small nits I observed:

Yeah, was a quick untested hack to see if this can be simplified :)

Quoted 6 lines

1. In the retry loop, it is possible that we reacquire the mmap_lock and set mmap_locked to true. This can cause issues later when we do:

if (!mmap_locked)
*lock_dropped = true;

That whole logic of having two variables that express whether locks have been taken/dropped is just absolutely confusing. Any way we can clean that up?

Quoted 4 lines

because the caller would no longer see that the lock was dropped earlier.

2. We need an fput() to balance the file reference taken at line 2795.

Ah, yes, makes sense. Having a single fput() would be nicer, but that would require yet another temporary variable.

Signature · 3 lines
Cheers

David
#7 lore raw

Re: [PATCH V4 2/2] mm/khugepaged: retry with sync writeback for MADV_COLLAPSE

Quoted 67 lines · David Hildenbrand (Red Hat)
Quoted 53 lines · Shivank
Quoted 26 lines · David Hildenbrand (Red Hat)

On 12/15/25 09:46, Shivank Garg wrote:

This looks a bit complicated. Can't we move that handing up, where we have most of that information already? Or am I missing something important?

mm/khugepaged.c +2 -0
index 97d1b2824386f..c7271877c5220 100644
@@ -22,6 +22,7 @@
   #include <linux/dax.h>
   #include <linux/ksm.h>
   #include <linux/pgalloc.h>
+#include <linux/backing-dev.h>
     #include <asm/tlb.h>
   #include "internal.h"
@@ -2786,7 +2787,9 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
            for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
                  int result = SCAN_FAIL;
+               bool triggered_wb = false;

  +retry:
                 if (!mmap_locked) {
                         cond_resched();
                         mmap_read_lock(mm);

mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2812

Quoted 1 line
mm/khugepaged.c +0 -0
@@ -2809,6 +2812,16 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
                          mmap_locked = false;


              *lock_dropped = true;

mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2810

Quoted 10 lines
mm/khugepaged.c +8 -0
                          result = hpage_collapse_scan_file(mm, addr, file, pgoff,
                                                            cc);
+
+                       if (result == SCAN_PAGE_DIRTY_OR_WRITEBACK && !triggered_wb &&
+                           mapping_can_writeback(file->f_mapping)) {
+                               loff_t lstart = (loff_t)pgoff << PAGE_SHIFT;
+                               loff_t lend = lstart + HPAGE_PMD_SIZE - 1;
+
+                               filemap_write_and_wait_range(file->f_mapping, lstart, lend);
+                               triggered_wb = true;

                  fput(file);

mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2816

Quoted 7 lines
mm/khugepaged.c +2 -0
+                               goto retry;
+                       }
                          fput(file);
                  } else {
                          result = hpage_collapse_scan_pmd(mm, vma, addr,
Quoted 3 lines

Thank you for the suggestion, this approach looks much simpler.

There are two small nits I observed:

Yeah, was a quick untested hack to see if this can be simplified :)

Quoted 6 lines

1. In the retry loop, it is possible that we reacquire the mmap_lock and set mmap_locked to true. This can cause issues later when we do:

        if (!mmap_locked)
                *lock_dropped = true;

Quoted 9 lines

That whole logic of having two variables that express whether locks have been taken/dropped is just absolutely confusing. Any way we can clean that up?

Quoted 4 lines

    because the caller would no longer see that the lock was dropped earlier.

2. We need an fput() to balance the file reference taken at line 2795.

Ah, yes, makes sense. Having a single fput() would be nicer, but that would require yet another temporary variable.

I agree, that this interaction for lock taken/droped is confusing.
However, a proper clean-up would require refactoring the locking logic across multiple functions in the collapse call-flow path. This seems significantly more invasive and risky.

I would like to handle this refactoring but in a separate TODO for later. Could we please proceed with these minimal changes for now?

Since, V4 has been in the linux-next/mm-unstable for a while, should I send a v5 or an incremental clean-up on top for this?

Thanks,
Shivank

#8 lore raw

Re: [PATCH V4 2/2] mm/khugepaged: retry with sync writeback for MADV_COLLAPSE

Quoted 80 lines · Shivank
Quoted 67 lines · David Hildenbrand (Red Hat)
Quoted 53 lines · Shivank
Quoted 26 lines · David Hildenbrand (Red Hat)

On 12/15/25 09:46, Shivank Garg wrote:

This looks a bit complicated. Can't we move that handing up, where we have most of that information already? Or am I missing something important?

mm/khugepaged.c +2 -0
index 97d1b2824386f..c7271877c5220 100644
@@ -22,6 +22,7 @@
   #include <linux/dax.h>
   #include <linux/ksm.h>
   #include <linux/pgalloc.h>
+#include <linux/backing-dev.h>
     #include <asm/tlb.h>
   #include "internal.h"
@@ -2786,7 +2787,9 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
            for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
                  int result = SCAN_FAIL;
+               bool triggered_wb = false;
   +retry:
                  if (!mmap_locked) {
                          cond_resched();
                          mmap_read_lock(mm);

mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2812

Quoted 1 line
mm/khugepaged.c +0 -0
@@ -2809,6 +2812,16 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
                          mmap_locked = false;

              *lock_dropped = true;

mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2810

Quoted 10 lines
mm/khugepaged.c +8 -0
                          result = hpage_collapse_scan_file(mm, addr, file, pgoff,
                                                            cc);
+
+                       if (result == SCAN_PAGE_DIRTY_OR_WRITEBACK && !triggered_wb &&
+                           mapping_can_writeback(file->f_mapping)) {
+                               loff_t lstart = (loff_t)pgoff << PAGE_SHIFT;
+                               loff_t lend = lstart + HPAGE_PMD_SIZE - 1;
+
+                               filemap_write_and_wait_range(file->f_mapping, lstart, lend);
+                               triggered_wb = true;

                  fput(file);

mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2816

Quoted 7 lines
mm/khugepaged.c +2 -0
+                               goto retry;
+                       }
                          fput(file);
                  } else {
                          result = hpage_collapse_scan_pmd(mm, vma, addr,
Quoted 3 lines

Thank you for the suggestion, this approach looks much simpler.

There are two small nits I observed:

Yeah, was a quick untested hack to see if this can be simplified :)

Quoted 6 lines

1. In the retry loop, it is possible that we reacquire the mmap_lock and set mmap_locked to true. This can cause issues later when we do:

        if (!mmap_locked)
                *lock_dropped = true;

Quoted 9 lines

That whole logic of having two variables that express whether locks have been taken/dropped is just absolutely confusing. Any way we can clean that up?

Quoted 4 lines

    because the caller would no longer see that the lock was dropped earlier.

2. We need an fput() to balance the file reference taken at line 2795.

Ah, yes, makes sense. Having a single fput() would be nicer, but that would require yet another temporary variable.

Quoted 6 lines

I agree, that this interaction for lock taken/droped is confusing.
However, a proper clean-up would require refactoring the locking logic across multiple functions in the collapse call-flow path. This seems significantly more invasive and risky.

I would like to handle this refactoring but in a separate TODO for later. Could we please proceed with these minimal changes for now?

Sure, fine with me.

Quoted 2 lines

Since, V4 has been in the linux-next/mm-unstable for a while, should I send a v5 or an incremental clean-up on top for this?

Just send a v4, unless Andrew tells you otherwise :)

Signature · 3 lines
Cheers

David
#9 lore raw

Re: [PATCH V4 2/2] mm/khugepaged: retry with sync writeback for MADV_COLLAPSE

Quoted 87 lines · David Hildenbrand (Red Hat)
Quoted 79 lines · Shivank
Quoted 66 lines · David Hildenbrand (Red Hat)
Quoted 52 lines · Shivank
Quoted 26 lines · David Hildenbrand (Red Hat)

On 12/15/25 09:46, Shivank Garg wrote:

This looks a bit complicated. Can't we move that handing up, where we have most of that information already? Or am I missing something important?

mm/khugepaged.c +2 -0
index 97d1b2824386f..c7271877c5220 100644
@@ -22,6 +22,7 @@
    #include <linux/dax.h>
    #include <linux/ksm.h>
    #include <linux/pgalloc.h>
+#include <linux/backing-dev.h>
      #include <asm/tlb.h>
    #include "internal.h"
@@ -2786,7 +2787,9 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
             for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
                   int result = SCAN_FAIL;
+               bool triggered_wb = false;

   +retry:
                  if (!mmap_locked) {
                          cond_resched();
                          mmap_read_lock(mm);

mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2812

Quoted 1 line
mm/khugepaged.c +0 -0
@@ -2809,6 +2812,16 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
                           mmap_locked = false;

                           *lock_dropped = true;

mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2810

Quoted 10 lines
mm/khugepaged.c +8 -0
                           result = hpage_collapse_scan_file(mm, addr, file, pgoff,
                                                             cc);
+
+                       if (result == SCAN_PAGE_DIRTY_OR_WRITEBACK && !triggered_wb &&
+                           mapping_can_writeback(file->f_mapping)) {
+                               loff_t lstart = (loff_t)pgoff << PAGE_SHIFT;
+                               loff_t lend = lstart + HPAGE_PMD_SIZE - 1;
+
+                               filemap_write_and_wait_range(file->f_mapping, lstart, lend);
+                               triggered_wb = true;

                   fput(file);

mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2816

Quoted 7 lines
mm/khugepaged.c +2 -0
+                               goto retry;
+                       }
                           fput(file);
                   } else {
                           result = hpage_collapse_scan_pmd(mm, vma, addr,
Quoted 3 lines

Thank you for the suggestion, this approach looks much simpler.

There are two small nits I observed:

Yeah, was a quick untested hack to see if this can be simplified :)

Quoted 6 lines

1. In the retry loop, it is possible that we reacquire the mmap_lock and set mmap_locked to true. This can cause issues later when we do:

         if (!mmap_locked)
                 *lock_dropped = true;

Quoted 9 lines

That whole logic of having two variables that express whether locks have been taken/dropped is just absolutely confusing. Any way we can clean that up?

Quoted 4 lines

     because the caller would no longer see that the lock was dropped earlier.

2. We need an fput() to balance the file reference taken at line 2795.

Ah, yes, makes sense. Having a single fput() would be nicer, but that would require yet another temporary variable.

Quoted 6 lines

I agree, that this interaction for lock taken/droped is confusing.
However, a proper clean-up would require refactoring the locking logic across multiple functions in the collapse call-flow path. This seems significantly more invasive and risky.

I would like to handle this refactoring but in a separate TODO for later. Could we please proceed with these minimal changes for now?

Quoted 6 lines

Sure, fine with me.

Quoted 2 lines

Since, V4 has been in the linux-next/mm-unstable for a while, should I send a v5 or an incremental clean-up on top for this?

Just send a v4, unless Andrew tells you otherwise :)

Thanks! I have send V5:
https://lore.kernel.org/linux-mm/[email protected]

Best Regards,
Shivank