madvise(MADV_COLLAPSE) fails with EINVAL on dirty file-backed text pages
Hi All,
I've been investigating an issue with madvise(MADV_COLLAPSE) for TEXT pages when CONFIG_READ_ONLY_THP_FOR_FS=y is enabled, and would like to discuss the current behavior and improvements.
Problem:
When attempting to collapse read-only file-backed TEXT sections into THPs using madvise(MADV_COLLAPSE), the operation fails with EINVAL if the pages are marked dirty.madvise(aligned_start, aligned_size, MADV_COLLAPSE) -> returns -1 and errno = -22
Subsequent calls to madvise(MADV_COLLAPSE) succeed because the first madvise attempt triggers filemap_flush() which initiates async writeback of the dirty folios.
Root Cause:
The failure occurs in mm/khugepaged.c:collapse_file():
} else if (folio_test_dirty(folio)) {
/*
* khugepaged only works on read-only fd,
* so this page is dirty because it hasn't
* been flushed since first write. There
* won't be new dirty pages.
*
* Trigger async flush here and hope the
* writeback is done when khugepaged
* revisits this page.
*/
xas_unlock_irq(&xas); filemap_flush(mapping);
result = SCAN_FAIL;
goto xa_unlocked;
}
Why the text pages are dirty?
It initially seemed unusual for a read-only text section to be marked as dirty, but this was actually confirmed by /proc/pid/smaps.
55bc90200000-55bc91200000 r-xp 00400000 07:00 133 /mnt/xfs-mnt/large_binary_thp
Size: 16384 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 256 kB Pss: 256 kB Pss_Dirty: 256 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 256 kB
/proc/pid/smaps (before calling MADV_COLLAPSE) showing Private_Dirty pages in r-xp mappings. This may be due to dynamic linker and relocations that occurred during program loading.
Reproduction using XFS/EXT4:
1. Compile a test binary with madvise(MADV_COLLAPSE), ensuring the load TEXT segment is 2MB-aligned and sized to a multiple of 2MB.
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
LOAD 0x400000 0x0000000000400000 0x0000000000400000 0x1000000 0x1000000 R E 0x200000
2. Create and mount the XFS/EXT4 fs:
dd if=/dev/zero of=/tmp/xfs-test.img bs=1M count=1024
losetup -f --show /tmp/xfs-test.img # output: /dev/loop0
mkfs.xfs -f /dev/loop0
mkdir -p /mnt/xfs-mnt
mount /dev/loop0 /mnt/xfs-mnt
3. Copy the binaries to /mnt/xfs-mnt and execute.
4. Returns -EINVAL on first run, then run successfully on subsequent run. (100% reproducible)
5. To reproduce again; reboot/kexec and repeat from step 2.
Workaround:
1. Manually flush dirty pages before calling madvise(MADV_COLLAPSE):
int fd = open("/proc/self/exe", O_RDONLY);
if (fd >= 0) {
fsync(fd);
close(fd);
}
// Now madvise(MADV_COLLAPSE) succeeds2. Alternatively, retrying madvise_collapse on EINVAL failure also work.
Problems with Current Behavior:
1. Confusing Error Code: The syscall returns EINVAL which typically indicates invalid arguments rather than a transient condition that could succeed on retry.
2. Non-Transparent Handling: Users are unaware they need to flush dirty pages manually. Current madvise_collapse assumes the caller is khugepaged (as per code snippet comment) which will revisit the page. However, when called via madvise(MADV_COLLAPSE), the userspace program typically don't retry, making the async flush ineffective. Should we differentiate between madvise and khugepaged behavior for MADV_COLLAPSE?
Would appreciate thoughts on the best approach to address this issue.
Thanks,
Shivank
Quoted 1 line · Shivank
Hi All,
Hi Shivank,
Good catch and a really clear analysis - thanks!
Quoted 83 lines
I've been investigating an issue with
madvise(MADV_COLLAPSE)for TEXT pages whenCONFIG_READ_ONLY_THP_FOR_FS=yis enabled, and would like to discuss the current behavior and improvements.Problem:
When attempting to collapse read-only file-backed TEXT sections into THPs usingmadvise(MADV_COLLAPSE), the operation fails with EINVAL if the pages are marked dirty.madvise(aligned_start, aligned_size, MADV_COLLAPSE)-> returns -1 and errno = -22Subsequent calls to
madvise(MADV_COLLAPSE)succeed because the first madvise attempt triggersfilemap_flush()which initiates async writeback of the dirty folios.Root Cause:
The failure occurs in mm/khugepaged.c:collapse_file():
} else if (folio_test_dirty(folio)) {
/*
* khugepaged only works on read-only fd,
* so this page is dirty because it hasn't
* been flushed since first write. There
* won't be new dirty pages.
*
* Trigger async flush here and hope the
* writeback is done when khugepaged
* revisits this page.
*/
xas_unlock_irq(&xas);filemap_flush(mapping);
result =SCAN_FAIL;
gotoxa_unlocked;
}Why the text pages are dirty?
It initially seemed unusual for a read-only text section to be marked as dirty, but this was actually confirmed by/proc/pid/smaps.55bc90200000-55bc91200000 r-xp 00400000 07:00 133
/mnt/xfs-mnt/large_binary_thpSize: 16384 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 256 kB Pss: 256 kB Pss_Dirty: 256 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 256 kB
/proc/pid/smaps(before callingMADV_COLLAPSE) showingPrivate_Dirtypages in r-xp mappings. This may be due to dynamic linker and relocations that occurred during program loading.Reproduction using XFS/EXT4:
1. Compile a test binary with
madvise(MADV_COLLAPSE), ensuring the load TEXT segment is 2MB-aligned and sized to a multiple of 2MB.
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
LOAD0x4000000x00000000004000000x00000000004000000x10000000x1000000R E0x2000002. Create and mount the XFS/EXT4 fs:
dd if=/dev/zero of=/tmp/xfs-test.img bs=1M count=1024
losetup -f --show/tmp/xfs-test.img# output:/dev/loop0
mkfs.xfs -f/dev/loop0
mkdir -p/mnt/xfs-mnt
mount/dev/loop0/mnt/xfs-mnt
3. Copy the binaries to/mnt/xfs-mntand execute.
4. Returns-EINVALon first run, then run successfully on subsequent run. (100% reproducible)
5. To reproduce again; reboot/kexec and repeat from step 2.Workaround:
1. Manually flush dirty pages before callingmadvise(MADV_COLLAPSE):int fd = open("/proc/self/exe", O_RDONLY); if (fd >= 0) { fsync(fd); close(fd); } // Now madvise(MADV_COLLAPSE) succeeds2. Alternatively, retrying
madvise_collapseon EINVAL failure also work.Problems with Current Behavior:
1. Confusing Error Code: The syscall returns EINVAL which typically indicates invalid arguments rather than a transient condition that could succeed on retry.
Quoted 7 lines
2. Non-Transparent Handling: Users are unaware they need to flush dirty pages manually. Current
madvise_collapseassumes the caller is khugepaged (as per code snippet comment) which will revisit the page. However, when called viamadvise(MADV_COLLAPSE), the userspace program typically don't retry, making the async flush ineffective. Should we differentiate between madvise and khugepaged behavior forMADV_COLLAPSE?Would appreciate thoughts on the best approach to address this issue.
Just throwing out a couple of ideas ...
We could just switch the return code to EAGAIN in the MADV_COLLAPSE path. At least that gives the right hint that retrying is an option ;)
Or, what if we just handle it inside the syscall? When we hit a dirty page, we wait for the writeback to finish and then try again right away. The call might be a little slower, but MADV_COLLAPSE is best effort, right? That seems worth the trouble ...
Cheers,
Lance
Quoted 9 lines · Lance Yang
Quoted 1 line · Shivank
Hi All,
Hi Shivank,
Good catch and a really clear analysis - thanks!
+1!
Quoted 89 lines
Quoted 89 lines
I've been investigating an issue with
madvise(MADV_COLLAPSE)for TEXT pages whenCONFIG_READ_ONLY_THP_FOR_FS=yis enabled, and would like to discuss the current behavior and improvements.Problem:
When attempting to collapse read-only file-backed TEXT sections into THPs usingmadvise(MADV_COLLAPSE), the operation fails with EINVAL if the pages are marked dirty.madvise(aligned_start, aligned_size, MADV_COLLAPSE)-> returns -1 and errno = -22Subsequent calls to
madvise(MADV_COLLAPSE)succeed because the first madvise attempt triggersfilemap_flush()which initiates async writeback of the dirty folios.Root Cause:
The failure occurs in mm/khugepaged.c:collapse_file():
} else if (folio_test_dirty(folio)) {
/*
* khugepaged only works on read-only fd,
* so this page is dirty because it hasn't
* been flushed since first write. There
* won't be new dirty pages.
*
* Trigger async flush here and hope the
* writeback is done when khugepaged
* revisits this page.
*/
xas_unlock_irq(&xas);filemap_flush(mapping);
result =SCAN_FAIL;
gotoxa_unlocked;
}Why the text pages are dirty?
It initially seemed unusual for a read-only text section to be marked as dirty, but this was actually confirmed by/proc/pid/smaps.55bc90200000-55bc91200000 r-xp 00400000 07:00 133
/mnt/xfs-mnt/large_binary_thpSize: 16384 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 256 kB Pss: 256 kB Pss_Dirty: 256 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 256 kB
/proc/pid/smaps(before callingMADV_COLLAPSE) showingPrivate_Dirtypages in r-xp mappings. This may be due to dynamic linker and relocations that occurred during program loading.Reproduction using XFS/EXT4:
1. Compile a test binary with
madvise(MADV_COLLAPSE), ensuring the load TEXT segment is 2MB-aligned and sized to a multiple of 2MB.
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
LOAD0x4000000x00000000004000000x00000000004000000x10000000x1000000R E0x2000002. Create and mount the XFS/EXT4 fs:
dd if=/dev/zero of=/tmp/xfs-test.img bs=1M count=1024
losetup -f --show/tmp/xfs-test.img# output:/dev/loop0
mkfs.xfs -f/dev/loop0
mkdir -p/mnt/xfs-mnt
mount/dev/loop0/mnt/xfs-mnt
3. Copy the binaries to/mnt/xfs-mntand execute.
4. Returns-EINVALon first run, then run successfully on subsequent run. (100% reproducible)
5. To reproduce again; reboot/kexec and repeat from step 2.Workaround:
1. Manually flush dirty pages before callingmadvise(MADV_COLLAPSE):int fd = open("/proc/self/exe", O_RDONLY); if (fd >= 0) { fsync(fd); close(fd); } // Now madvise(MADV_COLLAPSE) succeeds2. Alternatively, retrying
madvise_collapseon EINVAL failure also work.Problems with Current Behavior:
1. Confusing Error Code: The syscall returns EINVAL which typically indicates invalid arguments rather than a transient condition that could succeed on retry.2. Non-Transparent Handling: Users are unaware they need to flush dirty pages manually. Current
madvise_collapseassumes the caller is khugepaged (as per code snippet comment) which will revisit the page. However, when called viamadvise(MADV_COLLAPSE), the userspace program typically don't retry, making the async flush ineffective. Should we differentiate between madvise and khugepaged behavior forMADV_COLLAPSE?
Quoted 7 lines
Quoted 1 line
Would appreciate thoughts on the best approach to address this issue.
Just throwing out a couple of ideas ...
We could just switch the return code to EAGAIN in the
MADV_COLLAPSEpath. At least that
gives the right hint that retrying is an option ;)
Hey! I agree with Lance here, it seems the solution would be to return something other than SCAN_FAIL in collapse_file(), then in madvise_collapse_errno() catch this error and return EAGAIN. We could use SCAN_PAGE_COUNT which will cause a EAGAIN, or we could create a new result enum.
Cheers,
-- Nico
Quoted 10 lines
Or, what if we just handle it inside the syscall? When we hit a dirty page, we wait for
the writeback to finish and then try again right away. The call might be a little slower,
butMADV_COLLAPSEis best effort, right? That seems worth the trouble ...Cheers,
Lance
Quoted 34 lines · Shivank
Hi All,
I've been investigating an issue with
madvise(MADV_COLLAPSE)for TEXT pages whenCONFIG_READ_ONLY_THP_FOR_FS=yis enabled, and would like to discuss the current behavior and improvements.Problem:
When attempting to collapse read-only file-backed TEXT sections into THPs usingmadvise(MADV_COLLAPSE), the operation fails with EINVAL if the pages are marked dirty.madvise(aligned_start, aligned_size, MADV_COLLAPSE)-> returns -1 and errno = -22Subsequent calls to
madvise(MADV_COLLAPSE)succeed because the first madvise attempt triggersfilemap_flush()which initiates async writeback of the dirty folios.Root Cause:
The failure occurs in mm/khugepaged.c:collapse_file():
} else if (folio_test_dirty(folio)) {
/*
* khugepaged only works on read-only fd,
* so this page is dirty because it hasn't
* been flushed since first write. There
* won't be new dirty pages.
*
* Trigger async flush here and hope the
* writeback is done when khugepaged
* revisits this page.
*/
xas_unlock_irq(&xas);filemap_flush(mapping);
result =SCAN_FAIL;
gotoxa_unlocked;
}
Quoted 1 line
Why the text pages are dirty?
This is the real question to to answer, I think...
What architecture are you running on?
Quoted 15 lines
It initially seemed unusual for a read-only text section to be marked as dirty, but this was actually confirmed by
/proc/pid/smaps.55bc90200000-55bc91200000 r-xp 00400000 07:00 133
/mnt/xfs-mnt/large_binary_thpSize: 16384 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 256 kB Pss: 256 kB Pss_Dirty: 256 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 256 kB
Quoted 2 lines
/proc/pid/smaps(before callingMADV_COLLAPSE) showingPrivate_Dirtypages in r-xp mappings. This may be due to dynamic linker and relocations that occurred during program loading.
On arm64 at least, I wouldn't expect the text to be modified. Relocations should be handled in data. But given you have private dirty pages here, they must have been cow'ed and are therefore anonymous? In which case, where is writeback actually going?
Quoted 42 lines
Reproduction using XFS/EXT4:
1. Compile a test binary with
madvise(MADV_COLLAPSE), ensuring the load TEXT segment is 2MB-aligned and sized to a multiple of 2MB.
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
LOAD0x4000000x00000000004000000x00000000004000000x10000000x1000000R E0x2000002. Create and mount the XFS/EXT4 fs:
dd if=/dev/zero of=/tmp/xfs-test.img bs=1M count=1024
losetup -f --show/tmp/xfs-test.img# output:/dev/loop0
mkfs.xfs -f/dev/loop0
mkdir -p/mnt/xfs-mnt
mount/dev/loop0/mnt/xfs-mnt
3. Copy the binaries to/mnt/xfs-mntand execute.
4. Returns-EINVALon first run, then run successfully on subsequent run. (100% reproducible)
5. To reproduce again; reboot/kexec and repeat from step 2.Workaround:
1. Manually flush dirty pages before callingmadvise(MADV_COLLAPSE):int fd = open("/proc/self/exe", O_RDONLY); if (fd >= 0) { fsync(fd); close(fd); } // Now madvise(MADV_COLLAPSE) succeeds2. Alternatively, retrying
madvise_collapseon EINVAL failure also work.Problems with Current Behavior:
1. Confusing Error Code: The syscall returns EINVAL which typically indicates invalid arguments rather than a transient condition that could succeed on retry.2. Non-Transparent Handling: Users are unaware they need to flush dirty pages manually. Current
madvise_collapseassumes the caller is khugepaged (as per code snippet comment) which will revisit the page. However, when called viamadvise(MADV_COLLAPSE), the userspace program typically don't retry, making the async flush ineffective. Should we differentiate between madvise and khugepaged behavior forMADV_COLLAPSE?Would appreciate thoughts on the best approach to address this issue.
Thanks,
Shivank
Quoted 35 lines · Ryan Roberts
Quoted 34 lines · Shivank
Hi All,
I've been investigating an issue with
madvise(MADV_COLLAPSE)for TEXT pages whenCONFIG_READ_ONLY_THP_FOR_FS=yis enabled, and would like to discuss the current behavior and improvements.Problem:
When attempting to collapse read-only file-backed TEXT sections into THPs usingmadvise(MADV_COLLAPSE), the operation fails with EINVAL if the pages are marked dirty.madvise(aligned_start, aligned_size, MADV_COLLAPSE)-> returns -1 and errno = -22Subsequent calls to
madvise(MADV_COLLAPSE)succeed because the first madvise attempt triggersfilemap_flush()which initiates async writeback of the dirty folios.Root Cause:
The failure occurs in mm/khugepaged.c:collapse_file():
} else if (folio_test_dirty(folio)) {
/*
* khugepaged only works on read-only fd,
* so this page is dirty because it hasn't
* been flushed since first write. There
* won't be new dirty pages.
*
* Trigger async flush here and hope the
* writeback is done when khugepaged
* revisits this page.
*/
xas_unlock_irq(&xas);filemap_flush(mapping);
result =SCAN_FAIL;
gotoxa_unlocked;
}
Quoted 3 lines
Quoted 1 line
Why the text pages are dirty?
This is the real question to to answer, I think...
Agree with Ryan here, let's stop things from being marked dirty if they are not.
Quoted 19 lines
What architecture are you running on?
Quoted 15 lines
It initially seemed unusual for a read-only text section to be marked as dirty, but this was actually confirmed by
/proc/pid/smaps.55bc90200000-55bc91200000 r-xp 00400000 07:00 133
/mnt/xfs-mnt/large_binary_thpSize: 16384 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 256 kB Pss: 256 kB Pss_Dirty: 256 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 256 kB
Quoted 39 lines
Quoted 2 lines
/proc/pid/smaps(before callingMADV_COLLAPSE) showingPrivate_Dirtypages in r-xp mappings. This may be due to dynamic linker and relocations that occurred during program loading.On arm64 at least, I wouldn't expect the text to be modified. Relocations should be handled in data. But given you have private dirty pages here, they must have been cow'ed and are therefore anonymous? In which case, where is writeback actually going?
Quoted 31 lines
Reproduction using XFS/EXT4:
1. Compile a test binary with
madvise(MADV_COLLAPSE), ensuring the load TEXT segment is 2MB-aligned and sized to a multiple of 2MB.
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
LOAD0x4000000x00000000004000000x00000000004000000x10000000x1000000R E0x2000002. Create and mount the XFS/EXT4 fs:
dd if=/dev/zero of=/tmp/xfs-test.img bs=1M count=1024
losetup -f --show/tmp/xfs-test.img# output:/dev/loop0
mkfs.xfs -f/dev/loop0
mkdir -p/mnt/xfs-mnt
mount/dev/loop0/mnt/xfs-mnt
3. Copy the binaries to/mnt/xfs-mntand execute.
4. Returns-EINVALon first run, then run successfully on subsequent run. (100% reproducible)
5. To reproduce again; reboot/kexec and repeat from step 2.Workaround:
1. Manually flush dirty pages before callingmadvise(MADV_COLLAPSE):int fd = open("/proc/self/exe", O_RDONLY); if (fd >= 0) { fsync(fd); close(fd); } // Now madvise(MADV_COLLAPSE) succeeds2. Alternatively, retrying
madvise_collapseon EINVAL failure also work.Problems with Current Behavior:
1. Confusing Error Code: The syscall returns EINVAL which typically indicates invalid arguments rather than a transient condition that could succeed on retry.
This is also an issue though. Reading the documentation on my system, EINVAL with collapse has two meanings:
EINVAL addr is not page-aligned or length is negative. EINVAL advice is not a valid.
Neither are right here.
EAGAIN seems to make sense, but the documentation would need to be changed too:
EAGAIN A kernel resource was temporarily unavailable.
Quoted 6 lines
Quoted 6 lines
2. Non-Transparent Handling: Users are unaware they need to flush dirty pages manually. Current
madvise_collapseassumes the caller is khugepaged (as per code snippet comment) which will revisit the page. However, when called viamadvise(MADV_COLLAPSE), the userspace program typically don't retry, making the async flush ineffective. Should we differentiate between madvise and khugepaged behavior forMADV_COLLAPSE?
The collapse documentation states that it works on the existing state of the system memory, so it is doing what it says but the EINVAL return on dirty pages is not documented, afaict?
Quoted 6 lines
Quoted 5 lines
Would appreciate thoughts on the best approach to address this issue.
Thanks,
Shivank
Quoted 36 lines · Liam R. Howlett
Quoted 35 lines · Ryan Roberts
Quoted 34 lines · Shivank
Hi All,
I've been investigating an issue with
madvise(MADV_COLLAPSE)for TEXT pages whenCONFIG_READ_ONLY_THP_FOR_FS=yis enabled, and would like to discuss the current behavior and improvements.Problem:
When attempting to collapse read-only file-backed TEXT sections into THPs usingmadvise(MADV_COLLAPSE), the operation fails with EINVAL if the pages are marked dirty.madvise(aligned_start, aligned_size, MADV_COLLAPSE)-> returns -1 and errno = -22Subsequent calls to
madvise(MADV_COLLAPSE)succeed because the first madvise attempt triggersfilemap_flush()which initiates async writeback of the dirty folios.Root Cause:
The failure occurs in mm/khugepaged.c:collapse_file():
} else if (folio_test_dirty(folio)) {
/*
* khugepaged only works on read-only fd,
* so this page is dirty because it hasn't
* been flushed since first write. There
* won't be new dirty pages.
*
* Trigger async flush here and hope the
* writeback is done when khugepaged
* revisits this page.
*/
xas_unlock_irq(&xas);filemap_flush(mapping);
result =SCAN_FAIL;
gotoxa_unlocked;
}
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Why the text pages are dirty?
This is the real question to to answer, I think...
Agree with Ryan here, let's stop things from being marked dirty if they are not.
Hmm I wonder if we have some broken assumptions in khugepaged for MAP_PRIVATE mappings.
collapse_single_pmd()
-> collapse_scan_file() if not vma_is_anonymous() (it won't be) -> collapse_file()
-> the snippet above.
But that could be running on an anon folio...
Yup given it's CONFIG_READY_ONLY_THP_FOR_FS that is strange. We are confounding expectations here surely?
Presumably it's because these are MAP_PRIVATE mappings, so this is an anon folio but then collapse_file() goes into the snippet above and gets very confused.
Do we need to add a folio_test_anon() here?
Unless I'm missing something... (very possible, am only glancing over the code here)
Quoted 20 lines
Quoted 20 lines
What architecture are you running on?
Quoted 15 lines
It initially seemed unusual for a read-only text section to be marked as dirty, but this was actually confirmed by
/proc/pid/smaps.55bc90200000-55bc91200000 r-xp 00400000 07:00 133
/mnt/xfs-mnt/large_binary_thpSize: 16384 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 256 kB Pss: 256 kB Pss_Dirty: 256 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 256 kB
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
/proc/pid/smaps(before callingMADV_COLLAPSE) showingPrivate_Dirtypages in r-xp mappings. This may be due to dynamic linker and relocations that occurred during program loading.On arm64 at least, I wouldn't expect the text to be modified. Relocations should be handled in data. But given you have private dirty pages here, they must have been cow'ed and are therefore anonymous? In which case, where is writeback actually going?
Well ther won't be any right? I mean it's fairly normal to modify these MAP_PRIVATE mapping isn't it for relocations etc.?
You clipped the Anonymous line here, could you share it?
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Reproduction using XFS/EXT4:
1. Compile a test binary with
madvise(MADV_COLLAPSE), ensuring the load TEXT segment is 2MB-aligned and sized to a multiple of 2MB.
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
LOAD0x4000000x00000000004000000x00000000004000000x10000000x1000000R E0x2000002. Create and mount the XFS/EXT4 fs:
dd if=/dev/zero of=/tmp/xfs-test.img bs=1M count=1024
losetup -f --show/tmp/xfs-test.img# output:/dev/loop0
mkfs.xfs -f/dev/loop0
mkdir -p/mnt/xfs-mnt
mount/dev/loop0/mnt/xfs-mnt
3. Copy the binaries to/mnt/xfs-mntand execute.
4. Returns-EINVALon first run, then run successfully on subsequent run. (100% reproducible)
5. To reproduce again; reboot/kexec and repeat from step 2.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Workaround:
1. Manually flush dirty pages before callingmadvise(MADV_COLLAPSE):int fd = open("/proc/self/exe", O_RDONLY); if (fd >= 0) { fsync(fd); close(fd); }
Are you literally madvise()'ing the text portion of the executable?
It's strange this would make a difference in that case hmm...
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
// Now madvise(MADV_COLLAPSE) succeeds2. Alternatively, retrying
madvise_collapseon EINVAL failure also work.
Hmm that's strange.
Quoted 11 lines
Quoted 4 lines
Quoted 4 lines
Problems with Current Behavior:
1. Confusing Error Code: The syscall returns EINVAL which typically indicates invalid arguments rather than a transient condition that could succeed on retry.This is also an issue though. Reading the documentation on my system, EINVAL with collapse has two meanings:
EINVAL addr is not page-aligned or length is negative. EINVAL advice is not a valid.
Neither are right here.
Quoted 3 lines
EAGAIN seems to make sense, but the documentation would need to be changed too:
EAGAIN A kernel resource was temporarily unavailable.
I think these documented error codes are a total fantasy anyway in general for all system calls, and it'd be silly to try to list every single possible failure case in the man page. I really wish we didn't even try but there's horrible inconsistencies and missing entries for _tonnes_ of system calls.
So not sure if it's worth updating.
But obviously be good to have more information than less...
Quoted 11 lines
Quoted 7 lines
Quoted 7 lines
2. Non-Transparent Handling: Users are unaware they need to flush dirty pages manually. Current
madvise_collapseassumes the caller is khugepaged (as per code snippet comment) which will revisit the page. However, when called viamadvise(MADV_COLLAPSE), the userspace program typically don't retry, making the async flush ineffective. Should we differentiate between madvise and khugepaged behavior forMADV_COLLAPSE?The collapse documentation states that it works on the existing state of the system memory, so it is doing what it says but the EINVAL return on dirty pages is not documented, afaict?
It'd be good to document that this will fail if there are dirty pages yes.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Would appreciate thoughts on the best approach to address this issue.
Thanks,
Shivank
Cheers, Lorenzo
Quoted 57 lines · Lorenzo Stoakes
Quoted 36 lines · Liam R. Howlett
Quoted 35 lines · Ryan Roberts
Quoted 34 lines · Shivank
Hi All,
I've been investigating an issue with
madvise(MADV_COLLAPSE)for TEXT pages whenCONFIG_READ_ONLY_THP_FOR_FS=yis enabled, and would like to discuss the current behavior and improvements.Problem:
When attempting to collapse read-only file-backed TEXT sections into THPs usingmadvise(MADV_COLLAPSE), the operation fails with EINVAL if the pages are marked dirty.madvise(aligned_start, aligned_size, MADV_COLLAPSE)-> returns -1 and errno = -22Subsequent calls to
madvise(MADV_COLLAPSE)succeed because the first madvise attempt triggersfilemap_flush()which initiates async writeback of the dirty folios.Root Cause:
The failure occurs in mm/khugepaged.c:collapse_file():
} else if (folio_test_dirty(folio)) {
/*
* khugepaged only works on read-only fd,
* so this page is dirty because it hasn't
* been flushed since first write. There
* won't be new dirty pages.
*
* Trigger async flush here and hope the
* writeback is done when khugepaged
* revisits this page.
*/
xas_unlock_irq(&xas);filemap_flush(mapping);
result =SCAN_FAIL;
gotoxa_unlocked;
}Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Why the text pages are dirty?
This is the real question to to answer, I think...
Agree with Ryan here, let's stop things from being marked dirty if they are not.
Hmm I wonder if we have some broken assumptions in khugepaged for
MAP_PRIVATEmappings.
collapse_single_pmd()
->collapse_scan_file()if notvma_is_anonymous()(it won't be) ->collapse_file()
-> the snippet above.But that could be running on an anon folio...
Yup given it's
CONFIG_READY_ONLY_THP_FOR_FSthat is strange. We are confounding expectations here surely?
Quoted 7 lines
Presumably it's because these are
MAP_PRIVATEmappings, so this is an anon folio but thencollapse_file()goes into the snippet above and gets very confused.Do we need to add a
folio_test_anon()here?Unless I'm missing something... (very possible, am only glancing over the code here)
collapse_file() operates exclusively on the pagecache.
I think we only start working on the actual page tables when calling retract_page_tables().
In there, we have this code, when iterating over page tables belonging to the mapping:
/*
* The lock of new_folio is still held, we will be blocked in
* the page fault path, which prevents the pte entries from
* being set again. So even though the old empty PTE page may be
* concurrently freed and a new PTE page is filled into the pmd
* entry, it is still empty and can be removed.
*
* So here we only need to recheck if the state of pmd entry
* still meets our requirements, rather than checking pmd_same()
* like elsewhere.
*/
if (check_pmd_state(pmd) != SCAN_SUCCEED)
goto drop_pml;
ptl = pte_lockptr(mm, pmd);
if (ptl != pml)
spin_lock_nested(ptl, SINGLE_DEPTH_NESTING);
/*
* Huge page lock is still held, so normally the page table
* must remain empty; and we have already skipped anon_vma
* and userfaultfd_wp() vmas. But since the mmap_lock is not
* held, it is still possible for a racing userfaultfd_ioctl()
* to have inserted ptes or markers. Now that we hold ptlock,
* repeating the anon_vma check protects from one category,
* and repeating the userfaultfd_wp() check from another.
*/
if (likely(!vma->anon_vma && !userfaultfd_wp(vma))) {
pgt_pmd = pmdp_collapse_flush(vma, addr, pmd);
pmdp_get_lockless_sync();
success = true;
}Given !vma->anon_vma, we cannot have anon folios in there.
Given !userfaultfd_wp(vma), we cannot have uffd-wp markers in there.
Given that all folios in the range we are collapsing where unmapped, we cannot have them mapped there.
So the conclusion is that the page table must be empty and can be removed.
Could guard markers be in there?
Quoted 5 lines · David Hildenbrand (Red Hat)
Quoted 4 lines · Lorenzo Stoakes
Quoted 3 lines · Liam R. Howlett
Quoted 2 lines · Ryan Roberts
Quoted 1 line · Shivank
Hi All,
Hi all,
Thank you for the quick responses and suggestions! Information asked in this thread:
1. Architecture: X86_64
2. I want to emphasize that the error occurs specifically on a fresh mount after copying the binary. Binary can either be freshly compiled or previously compiled. The key factor is the fresh mount and copy operation.
3. For workaround:
I'm calling fsync(fd) from inside the executable before madvise(). Alternatively, I just tried that running sync from the shell after copying the binary also works, as it clears the Private_Dirty pages shown in smaps.
4. readelf --wide --segments large_binary_thp_s_withoutfsync Elf file type is DYN (Position-Independent Executable file) Entry point 0x4012e0
There are 13 program headers, starting at offset 64
Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
PHDR 0x000040 0x0000000000000040 0x0000000000000040 0x0002d8 0x0002d8 R 0x8
INTERP 0x000318 0x0000000000000318 0x0000000000000318 0x00001c 0x00001c R 0x1
[Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
LOAD 0x000000 0x0000000000000000 0x0000000000000000 0x24aa38 0x24aa38 R 0x1000
LOAD 0x400000 0x0000000000400000 0x0000000000400000 0x1000000 0x1000000 R E 0x200000
LOAD 0x1400000 0x0000000001400000 0x0000000001400000 0x53c750 0x53c750 R 0x1000
LOAD 0x193cd10 0x000000000193dd10 0x000000000193dd10 0x0c3810 0x0c3820 RW 0x1000
DYNAMIC 0x193cd28 0x000000000193dd28 0x000000000193dd28 0x0001f0 0x0001f0 RW 0x8
NOTE 0x000338 0x0000000000000338 0x0000000000000338 0x000030 0x000030 R 0x8
NOTE 0x000368 0x0000000000000368 0x0000000000000368 0x000044 0x000044 R 0x4
GNU_PROPERTY 0x000338 0x0000000000000338 0x0000000000000338 0x000030 0x000030 R 0x8
GNU_EH_FRAME 0x156bc5c 0x000000000156bc5c 0x000000000156bc5c 0x0c356c 0x0c356c R 0x4
GNU_STACK 0x000000 0x0000000000000000 0x0000000000000000 0x000000 0x000000 RW 0x10
GNU_RELRO 0x193cd10 0x000000000193dd10 0x000000000193dd10 0x0002f0 0x0002f0 R 0x1 Section to Segment mapping:
Segment Sections...
00
01 .interp 02 .interp .note.gnu.property .note.gnu.build-id .note.ABI-tag .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.dyn .rela.plt 03 .align_load_begin .init .plt .plt.got .plt.sec .text .fini .align_load_end 04 .rodata .eh_frame_hdr .eh_frame 05 .init_array .fini_array .dynamic .got .data .bss 06 .dynamic 07 .note.gnu.property 08 .note.gnu.build-id .note.ABI-tag 09 .note.gnu.property 10 .eh_frame_hdr
11
12 .init_array .fini_array .dynamic .got
4. Logs from --- Before Collapse ---
smaps:
55d436a00000-55d437a00000 r-xp 00400000 07:00 135 /mnt/xfs-mnt/large_binary_thp_s_withoutfsync
Size: 16384 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 256 kB Pss: 256 kB Pss_Dirty: 256 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 256 kB Referenced: 256 kB Anonymous: 0 kB KSM: 0 kB LazyFree: 0 kB AnonHugePages: 0 kB ShmemPmdMapped: 0 kB FilePmdMapped: 0 kB Shared_Hugetlb: 0 kB Private_Hugetlb: 0 kB Swap: 0 kB SwapPss: 0 kB Locked: 0 kB THPeligible: 0 ProtectionKey: 0 VmFlags: rd ex mr mw me sd
numa_maps:55d436a00000 default file=/mnt/xfs-mnt/large_binary_thp_s_withoutfsync dirty=64 active=0 N1=64 kernelpagesize_kB=4
Additional logs inside the kernel:
[ 129.257258] collapse_file: ENTER addr=55d436a00000 start=1024 end=1536 is_shmem=0 [ 129.257266] collapse_file: allocated new_folio successfully [ 129.257267] collapse_file: XArray slots created, starting page scan [ 129.257268] collapse_file: scanning index=1024 folio=00000000be1a13db [ 129.257270] collapse_file: folio_test_dirty index=1024 [ 129.257271] folio=00000000be1a13db, flags=0x57ffffc8000078 [ 129.257272] mapping=000000004df7b047, inode=000000003395e5a1 [ 129.257273] folio_test_large=1 [ 129.257273] inode mode=0100755, i_writecount=-1 inode_is_open_for_write(inode)=0
[ 129.257279] VMA #2: 000055d436a00000-000055d437a00000 flags=0x8000075 PID=5268 comm=large_binary_th <-- CONTAINS DIRTY FOLIO
Perms: r-xp MAYWRITE MAYEXEC
[ 129.257281] File offset range: 0x400000 - 0x1400000
[ 129.257282] Page index range: 1024 - 5120[ 129.257289] Total VMAs: 5, Writable VMAs: 0 [ 129.257290] Page details: [ 129.257290] PG_dirty=1 [ 129.257290] PG_writeback=0 [ 129.257291] PG_uptodate=1 [ 129.257291] PG_locked=0 [ 129.257292] refcount=64 [ 129.257292] mapcount=32 [ 129.260652] collapse_file: folio_test_dirty FAILED index=1024 [ 129.260655] collapse_file: FAILED result=0, going to rollback [ 129.260656] collapse_file: ROLLBACK result=0 [ 129.260661] collapse_file: EXIT result=0 [ 129.260661] collapse_file 0 [ 129.260662] default 0 [ 129.260663] madvise_collapse_errno: -22 last_fail: 0 [ 129.260665] thps 0 ((hend - hstart) >> HPAGE_PMD_SHIFT) 8
Note: result=0 is SCAN_FAIL
Now, after the failure on first attempt, when I run the executable again:
-- success run --
Region is 0x56185f800000 to 0x561860800000 - length 16777216
56185f800000-561860800000 r-xp 00400000 07:00 135 /mnt/xfs-mnt/large_binary_thp_s_withoutfsync
Size: 16384 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 256 kB Pss: 256 kB Pss_Dirty: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 256 kB Private_Dirty: 0 kB Referenced: 256 kB Anonymous: 0 kB KSM: 0 kB LazyFree: 0 kB AnonHugePages: 0 kB ShmemPmdMapped: 0 kB FilePmdMapped: 0 kB Shared_Hugetlb: 0 kB Private_Hugetlb: 0 kB Swap: 0 kB SwapPss: 0 kB Locked: 0 kB THPeligible: 0 ProtectionKey: 0 VmFlags: rd ex mr mw me sd
56185f800000 default file=/mnt/xfs-mnt/large_binary_thp_s_withoutfsync mapped=64 active=0 N1=64 kernelpagesize_kB=4
Start: 0x56185f800000
End: 0x561860800000
Size: 16777216 bytes (16.00 MB)
Hugepages: 8 x 2MB
Calling madvise(MADV_COLLAPSE)... Successfully collapsed text section into hugepages!
5. Yes, I'm calling madvise(MADV_COLLAPSE) on the text portion of the executable, using the address range obtained from /proc/self/maps. IIUC, this should benefit applications by reducing ITLB pressure.
I agree with the suggestions to either Return EAGAIN instead of EINVAL or At minimum, document the EINVAL return for dirty pages. I'm happy to work on a patch.
Please let me know if any other information is needed for debugging.
Thanks,
Shivank
Quoted 7 lines
5. Yes, I'm calling
madvise(MADV_COLLAPSE)on the text portion of the executable, using the address range obtained from/proc/self/maps. IIUC, this should benefit applications by reducing ITLB pressure.I agree with the suggestions to either Return EAGAIN instead of EINVAL or At minimum, document the EINVAL return for dirty pages. I'm happy to work on a patch.
Of course, we could detect that we are in MADV_COLLAPSE and simply writeback ourselves. After all, user space asked for a collapse, and it's not khugepaged that will simple revisit it later.
I did something similar in
commit ab73b29efd36f8916c6cc9954e912c4723c9a1b0
Author: David Hildenbrand <[email protected]> Date: Fri May 16 14:39:46 2025 +0200
s390/uv: Improve splitting of large folios that cannot be split while dirty
Currently, starting a PV VM on an iomap-based filesystem with large folio support, such as XFS, will not work. We'll be stuck in unpack_one()->gmap_make_secure(), because we can't seem to make progress splitting the large folio.
Where I effectively use filemap_write_and_wait_range().
It could be used early to writeback the whole range to collapse once, possibly.
Quoted 25 lines · David Hildenbrand (Red Hat)
Quoted 7 lines
5. Yes, I'm calling
madvise(MADV_COLLAPSE)on the text portion of the executable, using the address range obtained from/proc/self/maps. IIUC, this should benefit applications by reducing ITLB pressure.I agree with the suggestions to either Return EAGAIN instead of EINVAL or At minimum, document the EINVAL return for dirty pages. I'm happy to work on a patch.
Of course, we could detect that we are in
MADV_COLLAPSEand simply writeback ourselves. After all, user space asked for a collapse, and it's not khugepaged that will simple revisit it later.I did something similar in
commit
ab73b29efd36f8916c6cc9954e912c4723c9a1b0
Author: David Hildenbrand <[email protected]>
Date: Fri May 16 14:39:46 2025 +0200s390/uv: Improve splitting of large folios that cannot be split
while dirty
Currently, starting a PV VM on an iomap-based filesystem with large folio support, such as XFS, will not work. We'll be stuck in unpack_one()->gmap_make_secure(), because we can't seem to make
progress
splitting the large folio.
Quoted 3 lines
Where I effectively use
filemap_write_and_wait_range().It could be used early to writeback the whole range to collapse once, possibly.
Exactly!
Since MADV_COLLAPSE is a best-effort thing, having the kernel use something like filemap_write_and_wait_range() to writeback the pages before collapsing is likely what users would expect.
Anyway, they just want to get a THP, whether the pages are dirty or clean :)
Quoted 23 lines · David Hildenbrand (Red Hat)
Quoted 7 lines
5. Yes, I'm calling
madvise(MADV_COLLAPSE)on the text portion of the executable, using the address range obtained from/proc/self/maps. IIUC, this should benefit applications by reducing ITLB pressure.I agree with the suggestions to either Return EAGAIN instead of EINVAL or At minimum, document the EINVAL return for dirty pages. I'm happy to work on a patch.
Of course, we could detect that we are in
MADV_COLLAPSEand simply writeback ourselves. After all, user space asked for a collapse, and it's not khugepaged that will simple revisit it later.I did something similar in
commit
ab73b29efd36f8916c6cc9954e912c4723c9a1b0Author: David Hildenbrand <[email protected]> Date: Fri May 16 14:39:46 2025 +0200s390/uv: Improve splitting of large folios that cannot be split while dirty Currently, starting a PV VM on an iomap-based filesystem with large folio support, such as XFS, will not work. We'll be stuck in unpack_one()->gmap_make_secure(), because we can't seem to make progress splitting the large folio.
Quoted 3 lines
Where I effectively use
filemap_write_and_wait_range().It could be used early to writeback the whole range to collapse once, possibly.
I agree, let's just do a sync flush unconditionally and fix this that way.
This is simpler than I thought, the key bit of information is that we have freshly written the executable so it sits in the page cache but dirty.
Thanks, Lorenzo
Quoted 26 lines · Lorenzo Stoakes
Quoted 25 lines · David Hildenbrand (Red Hat)
Quoted 7 lines
5. Yes, I'm calling
madvise(MADV_COLLAPSE)on the text portion of the executable, using the address range obtained from/proc/self/maps. IIUC, this should benefit applications by reducing ITLB pressure.I agree with the suggestions to either Return EAGAIN instead of EINVAL or At minimum, document the EINVAL return for dirty pages. I'm happy to work on a patch.
Of course, we could detect that we are in
MADV_COLLAPSEand simply writeback ourselves. After all, user space asked for a collapse, and it's not khugepaged that will simple revisit it later.I did something similar in
commit
ab73b29efd36f8916c6cc9954e912c4723c9a1b0Author: David Hildenbrand <[email protected]> Date: Fri May 16 14:39:46 2025 +0200s390/uv: Improve splitting of large folios that cannot be split while dirty Currently, starting a PV VM on an iomap-based filesystem with large folio support, such as XFS, will not work. We'll be stuck in unpack_one()->gmap_make_secure(), because we can't seem to make progress splitting the large folio.
Where I effectively use
filemap_write_and_wait_range().
Quoted 8 lines
Quoted 1 line
It could be used early to writeback the whole range to collapse once, possibly.
I agree, let's just do a sync flush unconditionally and fix this that way.
This is simpler than I thought, the key bit of information is that we have freshly written the executable so it sits in the page cache but dirty.
Thanks, Lorenzo
Thanks David for sharing the commit. This worked for me and fix is simple.
+ if (!is_shmem && !cc->is_khugepaged && mapping_can_writeback(mapping)) {
+ loff_t range_start = start << PAGE_SHIFT; + loff_t range_end = (end << PAGE_SHIFT) - 1; + int ret;
+
+ ret = filemap_write_and_wait_range(mapping, range_start, range_end);
+ if (ret) {
+ result = SCAN_FAIL;
+ goto out;
+ }
+ }
I'll do some more testing and post a cleaned-up version with proper comments; rebase on mm-next. Thanks,
Shivank
Quoted 45 lines · David Hildenbrand (Red Hat)
Quoted 44 lines · Lorenzo Stoakes
Quoted 42 lines · Liam R. Howlett
Quoted 35 lines · Ryan Roberts
Quoted 34 lines · Shivank
Hi All,
I've been investigating an issue with
madvise(MADV_COLLAPSE)for TEXT pages whenCONFIG_READ_ONLY_THP_FOR_FS=yis enabled, and would like to discuss the current behavior and improvements.Problem:
When attempting to collapse read-only file-backed TEXT sections into THPs usingmadvise(MADV_COLLAPSE), the operation fails with EINVAL if the pages are marked dirty.madvise(aligned_start, aligned_size, MADV_COLLAPSE)-> returns -1 and errno = -22Subsequent calls to
madvise(MADV_COLLAPSE)succeed because the first madvise attempt triggersfilemap_flush()which initiates async writeback of the dirty folios.Root Cause:
The failure occurs in mm/khugepaged.c:collapse_file():
} else if (folio_test_dirty(folio)) {
/*
* khugepaged only works on read-only fd,
* so this page is dirty because it hasn't
* been flushed since first write. There
* won't be new dirty pages.
*
* Trigger async flush here and hope the
* writeback is done when khugepaged
* revisits this page.
*/
xas_unlock_irq(&xas);filemap_flush(mapping);
result =SCAN_FAIL;
gotoxa_unlocked;
}Quoted 3 lines
Quoted 1 line
Why the text pages are dirty?
This is the real question to to answer, I think...
Agree with Ryan here, let's stop things from being marked dirty if they are not.
Quoted 7 lines
Quoted 7 lines
Hmm I wonder if we have some broken assumptions in khugepaged for
MAP_PRIVATEmappings.
collapse_single_pmd()
->collapse_scan_file()if notvma_is_anonymous()(it won't be) ->collapse_file()
-> the snippet above.
Sorry I was looking at Nico's series these functions aren't correct as of mm-new atm.
This should be:
madvise_collapse()
-> hpage_collapse_scan_file()
-> collapse_file()
Quoted 9 lines
Quoted 9 lines
But that could be running on an anon folio...
Yup given it's
CONFIG_READY_ONLY_THP_FOR_FSthat is strange. We are confounding expectations here surely?Presumably it's because these are
MAP_PRIVATEmappings, so this is an anon folio but thencollapse_file()goes into the snippet above and gets very confused.
Quoted 6 lines
Quoted 4 lines
Do we need to add a
folio_test_anon()here?Unless I'm missing something... (very possible, am only glancing over the code here)
collapse_file()operates exclusively on the pagecache.
Right you're correct:
XA_STATE_ORDER(xas, &mapping->i_pages, start, HPAGE_PMD_ORDER); ... folio = xas_load(&xas);
etc. etc.
And with the code you mention below, markers + MAP_PRIVATE are handled correctly.
This THP code :) such fun.
So yeah this is as simple as the folio is literally just dirty.
And:
} else if (folio_test_dirty(folio)) {
/*
* khugepaged only works on read-only fd,
* so this page is dirty because it hasn't
* been flushed since first write. There
* won't be new dirty pages.
*
* Trigger async flush here and hope the
* writeback is done when khugepaged
* revisits this page.
*
* This is a one-off situation. We are not
* forcing writeback in loop. */
xas_unlock_irq(&xas); filemap_flush(mapping);
Since we do an async flush here ----^
This is why a retry (assuming writeback completed) works.
result = SCAN_FAIL;
goto xa_unlocked;
} else if (folio_test_writeback(folio)) {Quoted 3 lines
I think we only start working on the actual page tables when calling
retract_page_tables().
Yup.
Quoted 36 lines
In there, we have this code, when iterating over page tables belonging to the mapping:
/* * The lock of new_folio is still held, we will be blocked in * the page fault path, which prevents the pte entries from * being set again. So even though the old empty PTE page may be * concurrently freed and a new PTE page is filled into the pmd * entry, it is still empty and can be removed. * * So here we only need to recheck if the state of pmd entry * still meets our requirements, rather than checking pmd_same() * like elsewhere. */ if (check_pmd_state(pmd) != SCAN_SUCCEED) goto drop_pml; ptl = pte_lockptr(mm, pmd); if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); /* * Huge page lock is still held, so normally the page table * must remain empty; and we have already skipped anon_vma * and userfaultfd_wp() vmas. But since the mmap_lock is not * held, it is still possible for a racing userfaultfd_ioctl() * to have inserted ptes or markers. Now that we hold ptlock, * repeating the anon_vma check protects from one category, * and repeating the userfaultfd_wp() check from another. */ if (likely(!vma->anon_vma && !userfaultfd_wp(vma))) { pgt_pmd = pmdp_collapse_flush(vma, addr, pmd); pmdp_get_lockless_sync(); success = true; }
Quoted 3 lines
Given !vma->anon_vma, we cannot have anon folios in there.
Given !userfaultfd_wp(vma), we cannot have uffd-wp markers in there.
Right.
Quoted 8 lines
Given that all folios in the range we are collapsing where unmapped, we cannot have them mapped there.
So the conclusion is that the page table must be empty and can be removed.
Could guard markers be in there?
Right now guard markers only exist if vma->anon_vma is set, including the file-backed case.
But for file-backed guard regions after my VMA sticky series this won't be the case any more :)
So I had better go change that...
I hate that we have open-coded stuff all over the place that makes assumptions like this.
This also ignores any other marker types. How I hate the uffd wp implementation.
Quoted 54 lines · Lorenzo Stoakes
Quoted 32 lines · David Hildenbrand (Red Hat)
/* * The lock of new_folio is still held, we will be blocked in * the page fault path, which prevents the pte entries from * being set again. So even though the old empty PTE page may be * concurrently freed and a new PTE page is filled into the pmd * entry, it is still empty and can be removed. * * So here we only need to recheck if the state of pmd entry * still meets our requirements, rather than checking pmd_same() * like elsewhere. */ if (check_pmd_state(pmd) != SCAN_SUCCEED) goto drop_pml; ptl = pte_lockptr(mm, pmd); if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); /* * Huge page lock is still held, so normally the page table * must remain empty; and we have already skipped anon_vma * and userfaultfd_wp() vmas. But since the mmap_lock is not * held, it is still possible for a racing userfaultfd_ioctl() * to have inserted ptes or markers. Now that we hold ptlock, * repeating the anon_vma check protects from one category, * and repeating the userfaultfd_wp() check from another. */ if (likely(!vma->anon_vma && !userfaultfd_wp(vma))) { pgt_pmd = pmdp_collapse_flush(vma, addr, pmd); pmdp_get_lockless_sync(); success = true; }Quoted 3 lines
Given !vma->anon_vma, we cannot have anon folios in there.
Given !userfaultfd_wp(vma), we cannot have uffd-wp markers in there.
Right.
Quoted 8 lines
Given that all folios in the range we are collapsing where unmapped, we cannot have them mapped there.
So the conclusion is that the page table must be empty and can be removed.
Could guard markers be in there?
Right now guard markers only exist if
vma->anon_vmais set, including the file-backed case.But for file-backed guard regions after my VMA sticky series this won't be the case any more :)
Quoted 6 lines
So I had better go change that...
I hate that we have open-coded stuff all over the place that makes assumptions like this.
This also ignores any other marker types. How I hate the uffd wp implementation.
OK I audited all vma->anon_vma uses and _this_ is literally the only place that is affected :)
Thanks for mentioning :P have written a self test to repro and fix will land in v3 of the sticky VMA series.
Cheers, Lorenzo
Quoted 35 lines · Shivank
Hi All,
I've been investigating an issue with
madvise(MADV_COLLAPSE)for TEXT pages whenCONFIG_READ_ONLY_THP_FOR_FS=yis enabled, and would like to discuss the current behavior and improvements.Problem:
When attempting to collapse read-only file-backed TEXT sections into THPs usingmadvise(MADV_COLLAPSE), the operation fails with EINVAL if the pages are marked dirty.madvise(aligned_start, aligned_size, MADV_COLLAPSE)-> returns -1 and errno = -22Subsequent calls to
madvise(MADV_COLLAPSE)succeed because the first madvise attempt triggersfilemap_flush()which initiates async writeback of the dirty folios.Root Cause:
The failure occurs in mm/khugepaged.c:collapse_file():
} else if (folio_test_dirty(folio)) {
/*
* khugepaged only works on read-only fd,
* so this page is dirty because it hasn't
* been flushed since first write. There
* won't be new dirty pages.
*
* Trigger async flush here and hope the
* writeback is done when khugepaged
* revisits this page.
*/
xas_unlock_irq(&xas);filemap_flush(mapping);
result =SCAN_FAIL;
gotoxa_unlocked;
}
Quoted 1 line
Why the text pages are dirty?
I'm not sure how you did the test, but if you ran the program right after it was built, it may be possible the background writeback has not kicked in yet, then MAD_COLLAPSE saw some dirty folios. This is how your reproducer works at least. This is why filemap_flush() was added in the first place. Please see commit 75f360696ce9d8ec8b253452b23b3e24c0689b4b.
Quoted 15 lines
It initially seemed unusual for a read-only text section to be marked as dirty, but this was actually confirmed by
/proc/pid/smaps.55bc90200000-55bc91200000 r-xp 00400000 07:00 133
/mnt/xfs-mnt/large_binary_thpSize: 16384 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 256 kB Pss: 256 kB Pss_Dirty: 256 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 256 kB
Quoted 1 line
/proc/pid/smaps(before callingMADV_COLLAPSE) showingPrivate_Dirtypages in r-xp mappings.
smaps shows private dirty if either the PTE is dirty or the folio is dirty. For this case, I don't expect the PTE is dirty.
Quoted 27 lines
This may be due to dynamic linker and relocations that occurred during program loading.
Reproduction using XFS/EXT4:
1. Compile a test binary with
madvise(MADV_COLLAPSE), ensuring the load TEXT segment is 2MB-aligned and sized to a multiple of 2MB.
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
LOAD0x4000000x00000000004000000x00000000004000000x10000000x1000000R E0x2000002. Create and mount the XFS/EXT4 fs:
dd if=/dev/zero of=/tmp/xfs-test.img bs=1M count=1024
losetup -f --show/tmp/xfs-test.img# output:/dev/loop0
mkfs.xfs -f/dev/loop0
mkdir -p/mnt/xfs-mnt
mount/dev/loop0/mnt/xfs-mnt
3. Copy the binaries to/mnt/xfs-mntand execute.
4. Returns-EINVALon first run, then run successfully on subsequent run. (100% reproducible)
5. To reproduce again; reboot/kexec and repeat from step 2.Workaround:
1. Manually flush dirty pages before callingmadvise(MADV_COLLAPSE):int fd = open("/proc/self/exe", O_RDONLY); if (fd >= 0) { fsync(fd); close(fd); } // Now madvise(MADV_COLLAPSE) succeeds
Quoted 5 lines
2. Alternatively, retrying
madvise_collapseon EINVAL failure also work.Problems with Current Behavior:
1. Confusing Error Code: The syscall returns EINVAL which typically indicates invalid arguments rather than a transient condition that could succeed on retry.
Yeah, I agree the return value is confusing. -EAGAIN may be better as suggested by others.
Quoted 6 lines
2. Non-Transparent Handling: Users are unaware they need to flush dirty pages manually. Current
madvise_collapseassumes the caller is khugepaged (as per code snippet comment) which will revisit the page. However, when called viamadvise(MADV_COLLAPSE), the userspace program typically don't retry, making the async flush ineffective. Should we differentiate between madvise and khugepaged behavior forMADV_COLLAPSE?
Maybe MADV_COLLAPSE can have some retry logic?
Thanks,
Yang
Quoted 6 lines
Would appreciate thoughts on the best approach to address this issue.
Thanks,
Shivank
Quoted 36 lines · Yang Shi
Quoted 35 lines · Shivank
Hi All,
I've been investigating an issue with
madvise(MADV_COLLAPSE)for TEXT pages whenCONFIG_READ_ONLY_THP_FOR_FS=yis enabled, and would like to discuss the current behavior and improvements.Problem:
When attempting to collapse read-only file-backed TEXT sections into THPs usingmadvise(MADV_COLLAPSE), the operation fails with EINVAL if the pages are marked dirty.madvise(aligned_start, aligned_size, MADV_COLLAPSE)-> returns -1 and errno = -22Subsequent calls to
madvise(MADV_COLLAPSE)succeed because the first madvise attempt triggersfilemap_flush()which initiates async writeback of the dirty folios.Root Cause:
The failure occurs in mm/khugepaged.c:collapse_file():
} else if (folio_test_dirty(folio)) {
/*
* khugepaged only works on read-only fd,
* so this page is dirty because it hasn't
* been flushed since first write. There
* won't be new dirty pages.
*
* Trigger async flush here and hope the
* writeback is done when khugepaged
* revisits this page.
*/
xas_unlock_irq(&xas);filemap_flush(mapping);
result =SCAN_FAIL;
gotoxa_unlocked;
}
Quoted 8 lines
Quoted 1 line
Why the text pages are dirty?
I'm not sure how you did the test, but if you ran the program right after it was built, it may be possible the background writeback has not kicked in yet, then
MAD_COLLAPSEsaw some dirty folios. This is how your reproducer works at least. This is whyfilemap_flush()was added in the first place. Please see commit75f360696ce9d8ec8b253452b23b3e24c0689b4b.
Program can either be freshly compiled or previously compiled. The error occurs specifically on a fresh mount after copying the binary. The key factor is the fresh mount and copy operation.
Quoted 75 lines
Quoted 16 lines
It initially seemed unusual for a read-only text section to be marked as dirty, but this was actually confirmed by
/proc/pid/smaps.55bc90200000-55bc91200000 r-xp 00400000 07:00 133
/mnt/xfs-mnt/large_binary_thpSize: 16384 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 256 kB Pss: 256 kB Pss_Dirty: 256 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 256 kBQuoted 1 line
/proc/pid/smaps(before callingMADV_COLLAPSE) showingPrivate_Dirtypages in r-xp mappings.smaps shows private dirty if either the PTE is dirty or the folio is dirty. For this case, I don't expect the PTE is dirty.
Quoted 27 lines
This may be due to dynamic linker and relocations that occurred during program loading.
Reproduction using XFS/EXT4:
1. Compile a test binary with
madvise(MADV_COLLAPSE), ensuring the load TEXT segment is 2MB-aligned and sized to a multiple of 2MB.
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
LOAD0x4000000x00000000004000000x00000000004000000x10000000x1000000R E0x2000002. Create and mount the XFS/EXT4 fs:
dd if=/dev/zero of=/tmp/xfs-test.img bs=1M count=1024
losetup -f --show/tmp/xfs-test.img# output:/dev/loop0
mkfs.xfs -f/dev/loop0
mkdir -p/mnt/xfs-mnt
mount/dev/loop0/mnt/xfs-mnt
3. Copy the binaries to/mnt/xfs-mntand execute.
4. Returns-EINVALon first run, then run successfully on subsequent run. (100% reproducible)
5. To reproduce again; reboot/kexec and repeat from step 2.Workaround:
1. Manually flush dirty pages before callingmadvise(MADV_COLLAPSE):int fd = open("/proc/self/exe", O_RDONLY); if (fd >= 0) { fsync(fd); close(fd); } // Now madvise(MADV_COLLAPSE) succeedsQuoted 5 lines
2. Alternatively, retrying
madvise_collapseon EINVAL failure also work.Problems with Current Behavior:
1. Confusing Error Code: The syscall returns EINVAL which typically indicates invalid arguments rather than a transient condition that could succeed on retry.Yeah, I agree the return value is confusing.
-EAGAINmay be better as suggested by others.Quoted 6 lines
2. Non-Transparent Handling: Users are unaware they need to flush dirty pages manually. Current
madvise_collapseassumes the caller is khugepaged (as per code snippet comment) which will revisit the page. However, when called viamadvise(MADV_COLLAPSE), the userspace program typically don't retry, making the async flush ineffective. Should we differentiate between madvise and khugepaged behavior forMADV_COLLAPSE?Maybe
MADV_COLLAPSEcan have some retry logic?Thanks,
YangQuoted 6 lines
Would appreciate thoughts on the best approach to address this issue.
Thanks,
Shivank