mm/khugepaged: fix dirty page handling for MADV_COLLAPSE
Subsystems (3)
v3 cover mm/khugepaged: fix dirty page handling for MADV_COLLAPSE
MADV_COLLAPSE on file-backed mappings fails with -EINVAL when TEXT pages are dirty. This affects scenarios like package/container updates or executing binaries immediately after writing them, etc.
The issue is that collapse_file() triggers async writeback and returns SCAN_FAIL (maps to -EINVAL), expecting khugepaged to revisit later. But MADV_COLLAPSE is synchronous and userspace expects immediate success or a clear retry signal.
Reproduction:
- Compile or copy 2MB-aligned executable to XFS/ext4 FS
- Call MADV_COLLAPSE on .text section
- First call fails with -EINVAL (text pages dirty from copy)
- Second call succeeds (async writeback completed)
Issue Report:
https://lore.kernel.org/all/[email protected]
Changelog:
V3:
- Reordered patches: Enum definition comes first as the retry logic depends on it
- Renamed SCAN_PAGE_NOT_CLEAN to SCAN_PAGE_DIRTY_OR_WRITEBACK (Dev, Lance, David)
- Changed writeback logic: Only trigger synchronous writeback and retry if the initial collapse attempt failed specifically due to dirty/writeback pages, rather than blindly flushing all file-backed VMAs (David)
- Added proper file reference counting (get_file/fput) around the unlock window to prevent UAF (Lance)
V2:
- https://lore.kernel.org/all/[email protected]
- Move writeback to madvise_collapse() (better abstraction, proper mmap_lock handling and does VMA revalidation after I/O) (Lorenzo)
- Rename to SCAN_PAGE_DIRTY to SCAN_PAGE_NOT_CLEAN and extend its use for all dirty/writeback folio cases that previously returned incorrect results (Dev)
V1: https://lore.kernel.org/all/[email protected]
Thanks,
Shivank Garg (2):
mm/khugepaged: map dirty/writeback pages failures to EAGAIN mm/khugepaged: retry with sync writeback for MADV_COLLAPSE
include/trace/events/huge_memory.h | 3 +- mm/khugepaged.c | 49 ++++++++++++++++++++++++++++-- 2 files changed, 48 insertions(+), 4 deletions(-)
base-commit: 2178727587e1eaa930b8266377119ed6043067df
Signature · 1 line
2.43.0
PATCH v3 1/2 mm/khugepaged: map dirty/writeback pages failures to EAGAIN
When collapse_file encounters dirty or writeback pages in file-backed mappings, it currently returns SCAN_FAIL which maps to -EINVAL. This is misleading as EINVAL suggests invalid arguments, whereas dirty/writeback pages represent transient conditions that may resolve on retry.
Introduce SCAN_PAGE_DIRTY_OR_WRITEBACK to cover both dirty and writeback states, mapping it to -EAGAIN. For MADV_COLLAPSE, this provides userspace with a clear signal that retry may succeed after writeback completes. For khugepaged, this is harmless as it will naturally revisit the range during periodic scans after async writeback completes.
--- include/trace/events/huge_memory.h | 3 ++- mm/khugepaged.c | 8 +++++--- 2 files changed, 7 insertions(+), 4 deletions(-)
include/trace/events/huge_memory.h +2 -1
@@ -37,7 +37,8 @@ EM( SCAN_PAGE_HAS_PRIVATE, "page_has_private") \ EM( SCAN_STORE_FAILED, "store_failed") \ EM( SCAN_COPY_MC, "copy_poisoned_page") \ - EMe(SCAN_PAGE_FILLED, "page_filled") + EM( SCAN_PAGE_FILLED, "page_filled") \ + EMe(SCAN_PAGE_DIRTY_OR_WRITEBACK, "page_dirty_or_writeback")#undef EM#undef EMe
mm/khugepaged.c +5 -3
@@ -58,6 +58,7 @@ enum scan_result {SCAN_STORE_FAILED ,SCAN_COPY_MC ,SCAN_PAGE_FILLED , + SCAN_PAGE_DIRTY_OR_WRITEBACK, };#define CREATE_TRACE_POINTS
@@ -1967,11 +1968,11 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr,*/ xas_unlock_irq (&xas);filemap_flush (mapping); - result =SCAN_FAIL ; + result =SCAN_PAGE_DIRTY_OR_WRITEBACK ; goto xa_unlocked; }else if (folio_test_writeback (folio)) {xas_unlock_irq (&xas); - result =SCAN_FAIL ; + result =SCAN_PAGE_DIRTY_OR_WRITEBACK ; goto xa_unlocked; }else if (folio_trylock (folio)) {folio_get (folio);
@@ -2018,7 +2019,7 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr,* folio is dirty because it hasn't been flushed * since first write. */ - result =SCAN_FAIL ; + result =SCAN_PAGE_DIRTY_OR_WRITEBACK ; goto out_unlock; }
@@ -2747,6 +2748,7 @@ static int madvise_collapse_errno(enum scan_result r)case SCAN_PAGE_LRU :case SCAN_DEL_PAGE_LRU :case SCAN_PAGE_FILLED : +case SCAN_PAGE_DIRTY_OR_WRITEBACK :return -EAGAIN ;/* * Other: Trying again likely not to succeed / error intrinsic to
Signature · 1 line
2.43.0
Re: [PATCH V3 1/2] mm/khugepaged: map dirty/writeback pages failures to EAGAIN
Quoted 11 lines · Shivank Garg
When
collapse_fileencounters dirty or writeback pages in file-backed mappings, it currently returnsSCAN_FAILwhich maps to-EINVAL. This is misleading as EINVAL suggests invalid arguments, whereas dirty/writeback pages represent transient conditions that may resolve on retry.Introduce
SCAN_PAGE_DIRTY_OR_WRITEBACKto cover both dirty and writeback states, mapping it to-EAGAIN. ForMADV_COLLAPSE, this provides userspace with a clear signal that retry may succeed after writeback completes. For khugepaged, this is harmless as it will naturally revisit the range during periodic scans after async writeback completes.
Quoted 7 lines
Reported-by Branden Moore <[email protected]>Fixes 34488399fa08 ("mm/madvise: add file and shmem support to MADV_COLLAPSE")Reviewed-by Dev Jain <[email protected]>Reviewed-by Lance Yang <[email protected]>Signed-off-by Shivank Garg <[email protected]>---
LGTM.
Re: [PATCH V3 1/2] mm/khugepaged: map dirty/writeback pages failures to EAGAIN
In reply to Shivank GargQuoted 11 lines
When
collapse_fileencounters dirty or writeback pages in file-backed mappings, it currently returnsSCAN_FAILwhich maps to-EINVAL. This is misleading as EINVAL suggests invalid arguments, whereas dirty/writeback pages represent transient conditions that may resolve on retry.Introduce
SCAN_PAGE_DIRTY_OR_WRITEBACKto cover both dirty and writeback states, mapping it to-EAGAIN. ForMADV_COLLAPSE, this provides userspace with a clear signal that retry may succeed after writeback completes. For khugepaged, this is harmless as it will naturally revisit the range during periodic scans after async writeback completes.
Quoted 7 lines
Reported-by Branden Moore <[email protected]>Fixes 34488399fa08 ("mm/madvise: add file and shmem support to MADV_COLLAPSE")Reviewed-by Dev Jain <[email protected]>Reviewed-by Lance Yang <[email protected]>Signed-off-by Shivank Garg <[email protected]>---
LGTM.
--
Best Regards,
wang lian
PATCH v3 2/2 mm/khugepaged: retry with sync writeback for MADV_COLLAPSE
In reply to Shivank GargWhen MADV_COLLAPSE is called on file-backed mappings (e.g., executable text sections), the pages may still be dirty from recent writes. collapse_file() will trigger async writeback and fail with SCAN_PAGE_DIRTY_OR_WRITEBACK (-EAGAIN).
MADV_COLLAPSE is a synchronous operation where userspace expects immediate results. If the collapse fails due to dirty pages, perform synchronous writeback on the specific range and retry once.
This avoids spurious failures for freshly written executables while avoiding unnecessary synchronous I/O for mappings that are already clean.
--- mm/khugepaged.c | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+)
mm/khugepaged.c +41 -0
@@ -22,6 +22,7 @@#include <linux/dax.h> #include <linux/ksm.h> #include <linux/pgalloc.h> +#include <linux/backing-dev.h> #include <asm/tlb.h> #include "internal.h"
@@ -2787,9 +2788,11 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, hend = end &HPAGE_PMD_MASK ;for (addr = hstart; addr < hend; addr +=HPAGE_PMD_SIZE ) { +bool retried = false;int result =SCAN_FAIL ;if (!mmap_locked) { +retry:cond_resched ();mmap_read_lock (mm); mmap_locked = true;
@@ -2819,6 +2822,44 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,if (!mmap_locked) *lock_dropped = true; +/* +* If the file-backed VMA has dirty pages, the scan triggers +* async writeback and returns SCAN_PAGE_DIRTY_OR_WRITEBACK. +* Since MADV_COLLAPSE is sync, we force sync writeback and +* retry once. +*/ +if (result ==SCAN_PAGE_DIRTY_OR_WRITEBACK && !retried) { +/* +* File scan drops the lock. We must re-acquire it to +* safely inspect the VMA and hold the file reference. +*/ +if (!mmap_locked) { +cond_resched (); +mmap_read_lock (mm); + mmap_locked = true; + result =hugepage_vma_revalidate (mm, addr, false, &vma, cc); +if (result !=SCAN_SUCCEED ) + goto handle_result; + } + +if (!vma_is_anonymous (vma) && vma->vm_file && +mapping_can_writeback (vma->vm_file->f_mapping)) { +struct file *file =get_file (vma->vm_file); +pgoff_t pgoff =linear_page_index (vma, addr); +loff_t lstart = (loff_t )pgoff <<PAGE_SHIFT ; +loff_t lend = lstart +HPAGE_PMD_SIZE -1 ; + +mmap_read_unlock (mm); + mmap_locked = false; + *lock_dropped = true; +filemap_write_and_wait_range (file->f_mapping, lstart, lend); +fput (file); + retried = true; + goto retry; + } + } + + handle_result:switch (result) {case SCAN_SUCCEED :
Signature · 1 line
2.43.0
Lance Yang
reviewerRe: [PATCH V3 2/2] mm/khugepaged: retry with sync writeback for MADV_COLLAPSE
Quoted 77 lines · Shivank Garg
When
MADV_COLLAPSEis called on file-backed mappings (e.g., executable text sections), the pages may still be dirty from recent writes.collapse_file()will trigger async writeback and fail withSCAN_PAGE_DIRTY_OR_WRITEBACK(-EAGAIN).
MADV_COLLAPSEis a synchronous operation where userspace expects immediate results. If the collapse fails due to dirty pages, perform synchronous writeback on the specific range and retry once.This avoids spurious failures for freshly written executables while avoiding unnecessary synchronous I/O for mappings that are already clean.
Reported-by Branden Moore <[email protected]>Fixes 34488399fa08 ("mm/madvise: add file and shmem support to MADV_COLLAPSE")Suggested-by David Hildenbrand <[email protected]>Signed-off-by Shivank Garg <[email protected]>--- mm/khugepaged.c | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+)mm/khugepaged.c +31
-0@@ -22,6 +22,7 @@#include <linux/dax.h> #include <linux/ksm.h> #include <linux/pgalloc.h> +#include <linux/backing-dev.h> #include <asm/tlb.h> #include "internal.h" @@ -2787,9 +2788,11 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, hend = end &HPAGE_PMD_MASK ;for (addr = hstart; addr < hend; addr +=HPAGE_PMD_SIZE ) { +bool retried = false;int result =SCAN_FAIL ;if (!mmap_locked) { +retry:cond_resched ();mmap_read_lock (mm); mmap_locked = true;@@ -2819,6 +2822,44 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,if (!mmap_locked) *lock_dropped = true; +/* +* If the file-backed VMA has dirty pages, the scan triggers +* async writeback and returns SCAN_PAGE_DIRTY_OR_WRITEBACK. +* Since MADV_COLLAPSE is sync, we force sync writeback and +* retry once. +*/ +if (result ==SCAN_PAGE_DIRTY_OR_WRITEBACK && !retried) { +/* +* File scan drops the lock. We must re-acquire it to +* safely inspect the VMA and hold the file reference. +*/ +if (!mmap_locked) { +cond_resched (); +mmap_read_lock (mm); + mmap_locked = true; + result =hugepage_vma_revalidate (mm, addr, false, &vma, cc); +if (result !=SCAN_SUCCEED ) + goto handle_result; + } + +if (!vma_is_anonymous (vma) && vma->vm_file && +mapping_can_writeback (vma->vm_file->f_mapping)) { +struct file *file =get_file (vma->vm_file); +pgoff_t pgoff =linear_page_index (vma, addr); +loff_t lstart = (loff_t )pgoff <<PAGE_SHIFT ; +loff_t lend = lstart +HPAGE_PMD_SIZE -1 ; + +mmap_read_unlock (mm);
mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2862
Quoted 10 lines
mm/khugepaged.c +10
-0+ mmap_locked = false; + *lock_dropped = true; +filemap_write_and_wait_range (file->f_mapping, lstart, lend); +fput (file); + retried = true; + goto retry; + } + } + +
Nit: spurious blank line.
Quoted 3 lines
mm/khugepaged.c +0
-0handle_result:switch (result) {case SCAN_SUCCEED :
Re: [PATCH V3 2/2] mm/khugepaged: retry with sync writeback for MADV_COLLAPSE
Quoted 80 lines · Lance Yang
Quoted 77 lines · Shivank Garg
When
MADV_COLLAPSEis called on file-backed mappings (e.g., executable text sections), the pages may still be dirty from recent writes.collapse_file()will trigger async writeback and fail withSCAN_PAGE_DIRTY_OR_WRITEBACK(-EAGAIN).
MADV_COLLAPSEis a synchronous operation where userspace expects immediate results. If the collapse fails due to dirty pages, perform synchronous writeback on the specific range and retry once.This avoids spurious failures for freshly written executables while avoiding unnecessary synchronous I/O for mappings that are already clean.
Reported-by Branden Moore <[email protected]>Fixes 34488399fa08 ("mm/madvise: add file and shmem support to MADV_COLLAPSE")Suggested-by David Hildenbrand <[email protected]>Signed-off-by Shivank Garg <[email protected]>---
mm/khugepaged.c| 41 +++++++++++++++++++++++++++++++++++++++++1 file changed, 41 insertions(+)mm/khugepaged.c +3
-0@@ -22,6 +22,7 @@ #include <linux/dax.h> #include <linux/ksm.h> #include <linux/pgalloc.h> +#include <linux/backing-dev.h> #include <asm/tlb.h> #include "internal.h"@@ -2787,9 +2788,11 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, hend = end & HPAGE_PMD_MASK; for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) { + bool retried = false; int result = SCAN_FAIL; if (!mmap_locked) { +retry: cond_resched(); mmap_read_lock(mm); mmap_locked = true;@@ -2819,6 +2822,44 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, if (!mmap_locked) *lock_dropped = true;+ /*
+ * If the file-backed VMA has dirty pages, the scan triggers + * async writeback and returnsSCAN_PAGE_DIRTY_OR_WRITEBACK. + * SinceMADV_COLLAPSEis sync, we force sync writeback and + * retry once.
+ */
+ if (result ==SCAN_PAGE_DIRTY_OR_WRITEBACK&& !retried) { + /*
+ * File scan drops the lock. We must re-acquire it to + * safely inspect the VMA and hold the file reference. + */
+ if (!mmap_locked) {
+cond_resched();
+mmap_read_lock(mm);
+mmap_locked= true;
+ result =hugepage_vma_revalidate(mm, addr, false, &vma, cc); + if (result !=SCAN_SUCCEED)
+ gotohandle_result;
+ }mm/khugepaged.c +13
-0+ + if (!vma_is_anonymous(vma) && vma->vm_file && + mapping_can_writeback(vma->vm_file->f_mapping)) { + struct file *file = get_file(vma->vm_file); + pgoff_t pgoff = linear_page_index(vma, addr); + loff_t lstart = (loff_t)pgoff << PAGE_SHIFT; + loff_t lend = lstart + HPAGE_PMD_SIZE - 1; + + mmap_read_unlock(mm); + mmap_locked = false; + *lock_dropped = true; + filemap_write_and_wait_range(file->f_mapping, lstart, lend); + fput(file);
Quoted 8 lines
mm/khugepaged.c
Quoted 6 lines
+ retried = true;
+ goto retry;
+ }
+ }mm/khugepaged.c +2
-0+ +Nit: spurious blank line.
Ah, I completely missed this. I’ll fix it in the next version. Hope the rest of the patch looks reasonable. Thanks for the review.
Thanks,
Shivank
Quoted 5 lines
Quoted 4 lines
handle_result:
switch (result) {
caseSCAN_SUCCEED:
Re: [PATCH V3 2/2] mm/khugepaged: retry with sync writeback for MADV_COLLAPSE
Quoted 92 lines · Shivank
Quoted 80 lines · Lance Yang
Quoted 77 lines · Shivank Garg
When
MADV_COLLAPSEis called on file-backed mappings (e.g., executable text sections), the pages may still be dirty from recent writes.collapse_file()will trigger async writeback and fail withSCAN_PAGE_DIRTY_OR_WRITEBACK(-EAGAIN).
MADV_COLLAPSEis a synchronous operation where userspace expects immediate results. If the collapse fails due to dirty pages, perform synchronous writeback on the specific range and retry once.This avoids spurious failures for freshly written executables while avoiding unnecessary synchronous I/O for mappings that are already clean.
Reported-by Branden Moore <[email protected]>Fixes 34488399fa08 ("mm/madvise: add file and shmem support to MADV_COLLAPSE")Suggested-by David Hildenbrand <[email protected]>Signed-off-by Shivank Garg <[email protected]>--- mm/khugepaged.c | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+)mm/khugepaged.c +34
-0@@ -22,6 +22,7 @@ #include <linux/dax.h> #include <linux/ksm.h> #include <linux/pgalloc.h> +#include <linux/backing-dev.h> #include <asm/tlb.h> #include "internal.h"@@ -2787,9 +2788,11 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, hend = end & HPAGE_PMD_MASK; for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) { + bool retried = false; int result = SCAN_FAIL; if (!mmap_locked) { +retry: cond_resched(); mmap_read_lock(mm); mmap_locked = true;@@ -2819,6 +2822,44 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, if (!mmap_locked) *lock_dropped = true; + /* + * If the file-backed VMA has dirty pages, the scan triggers + * async writeback and returns SCAN_PAGE_DIRTY_OR_WRITEBACK. + * Since MADV_COLLAPSE is sync, we force sync writeback and + * retry once. + */ + if (result == SCAN_PAGE_DIRTY_OR_WRITEBACK && !retried) { + /* + * File scan drops the lock. We must re-acquire it to + * safely inspect the VMA and hold the file reference. + */ + if (!mmap_locked) { + cond_resched(); + mmap_read_lock(mm); + mmap_locked = true; + result = hugepage_vma_revalidate(mm, addr, false, &vma, cc); + if (result != SCAN_SUCCEED) + goto handle_result; + } + + if (!vma_is_anonymous(vma) && vma->vm_file && + mapping_can_writeback(vma->vm_file->f_mapping)) { + struct file *file = get_file(vma->vm_file); + pgoff_t pgoff = linear_page_index(vma, addr); + loff_t lstart = (loff_t)pgoff << PAGE_SHIFT; + loff_t lend = lstart + HPAGE_PMD_SIZE - 1; + + mmap_read_unlock(mm); + mmap_locked = false; + *lock_dropped = true; + filemap_write_and_wait_range(file->f_mapping, lstart, lend); + fput(file);Quoted 8 lines
mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2862
Quoted 6 lines
mm/khugepaged.c +6
-0+ retried = true; + goto retry; + } + } + +Nit: spurious blank line.
Quoted 2 lines
Ah, I completely missed this. I’ll fix it in the next version. Hope the rest of the patch looks reasonable. Thanks for the review.
Apart from that nit, nothing else jumped out at me :)
Confirmed that the spurious EINVAL is gone, and it works as expected ;p
[...]
Cheers,
Lance