mm/khugepaged: Never install PMDs in uffd-minor-registered VMAs
Subsystems (5)
PATCH v4 1/2 mm/khugepaged: Never install PMDs in uffd-minor-registered VMAs
Userfaultfd minor faults provides userspace with the ability to manually install PTEs with UFFDIO_CONTINUE. Right now, MADV_COLLAPSE can map holes in the VMA when a naturally-aligned THP is present. This is not true for khugepaged collapse: the PTEs will be retracted, but a PMD will not be installed.
When MADV_COLLAPSE installs a PMD that mapped holes in the VMA, userspace is likely to expect UFFDIO_CONTINUE to succeed on the should-be holes. UFFDIO_CONTINUE will fail and return EEXIST.
This is not inherently a problem, as MADV_COLLAPSE is an explicit userspace action. But, especially because MADV_COLLAPSE can be invoked by an external process via process_madvise(), a rogue caller could break a userfaultfd-minor resolver thread.
Userspace cannot generally use MADV_COLLAPSE to resolve userfault minor faults, as MADV_COLLAPSE will only resolve such faults if a naturally-aligned THP is present, so this is not a functional regression for userspace.
The naturally-aligned THP case is the only case where this quirk exists. Collapsing otherwise requires all PTEs to be present for userfaultfd-registered VMAs (i.e., max none PTEs is 0), which is correct. This check is essentially bypassed for naturally-aligned THPs.
---
Changes since v3:
- Prevent page table retraction in patch 1. Please see the comment next to it. So this patch now has two hunks instead of one.
- Reworded the comment and combined the userfaultfd checks in patch 1. (Thanks David)
- Simplified the selftest a bit given the behavior change in patch 1.
- Undid a change in v3's selftest that incorrectly handled the case where MADV_COLLAPSE was not supported entirely.
- Rebased on top of mm-unstable, which includes Kiryl's changes.
v3: https://lore.kernel.org/linux-mm/[email protected]/
v2: https://lore.kernel.org/linux-mm/[email protected]/
v1: https://lore.kernel.org/linux-mm/[email protected]/
--- mm/khugepaged.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)
mm/khugepaged.c +10 -4
@@ -1868,10 +1868,11 @@ static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsignreturn SCAN_VMA_CHECK ;/* -* Keep pmd pgtable while the uffd bit is in use; see comment in -* retract_page_tables(). +* Don't collapse if there might be PTE markers for userfaultfd-based +* access protection or if collapsing might bypass userfaultfd minor +* faults. */ -if (userfaultfd_protected (vma)) +if (userfaultfd_protected (vma) ||userfaultfd_minor (vma))return SCAN_PTE_UFFD ; folio = filemap_lock_folio(vma->vm_file->f_mapping,
@@ -2093,8 +2094,13 @@ static bool file_backed_vma_is_retractable(struct vm_area_struct *vma)* and cannot be recycled to a shared PMD. Other vmas can still * have the same file mapped hugely, but skip this one: it will * always be mapped in small page size for these registrations. +* +* Userfaultfd-minor-registered VMAs should also not be retracted. +* PMDs will not be installed, as doing so can suppress minor faults. +* If retraction were allowed, khugepaged might continually cause +* unnecessary userfaultfd minor faults for already-CONTINUE'd pages. */ -if (userfaultfd_protected (vma)) +if (userfaultfd_protected (vma) ||userfaultfd_minor (vma))return false;/*
base-commit: b2b4b29b76dabdee576eba66953a66ca61c5fca0
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
PATCH v4 2/2 mm: selftests: Adjust the MADV_COLLAPSE selftests for uffd-minor
The behavior of MADV_COLLAPSE with uffd-minor VMAs has changed. Previously, MADV_COLLAPSE would install PMDs in parts of a VMA where PTEs may be missing, essentially suppressing userfaultfd minor faults. Now, MADV_COLLAPSE returns -EINVAL for uffd-minor VMAs. Update the selftest to demonstrate this new behavior.
The selftest, if left unmodified, will SKIP when it gets EINVAL back from MADV_COLLAPSE.
--- tools/testing/selftests/mm/uffd-unit-tests.c | 69 ++++++++++++++++++-- 1 file changed, 62 insertions(+), 7 deletions(-)
tools/testing/selftests/mm/uffd-unit-tests.c +62 -7
@@ -518,19 +518,32 @@ static void uffd_wp_fork_pin_with_event_test(uffd_global_test_opts_t *gopts, uffuffd_wp_fork_pin_test_common (gopts, args, true); } -static void check_memory_contents (uffd_global_test_opts_t *gopts,char *p) +static int __check_memory_contents (unsigned long offset,unsigned long nr_pages, +uffd_global_test_opts_t *gopts,char *p) {unsigned long i, j;uint8_t expected_byte; -for (i =0 ; i < gopts->nr_pages; ++i) { +if (nr_pages + offset < nr_pages) +err ("overflow in memory check" ); +if (nr_pages + offset > gopts->nr_pages) +err ("out of bounds memory check" ); + +for (i = offset; i < offset + nr_pages; ++i) { expected_byte = ~((uint8_t )(i % ((uint8_t )-1 )));for (j =0 ; j < gopts->page_size; j++) {uint8_t v = *(uint8_t *)(p + (i * gopts->page_size) + j);if (v != expected_byte) -err ("unexpected page contents" ); +return 1 ; } } + +return 0 ; +} + +static int check_memory_contents (uffd_global_test_opts_t *gopts,char *p) +{ +return __check_memory_contents (0 , gopts->nr_pages, gopts, p); }static void uffd_minor_test_common (uffd_global_test_opts_t *gopts,bool test_collapse,bool test_wp)
@@ -538,8 +551,8 @@ static void uffd_minor_test_common(uffd_global_test_opts_t *gopts, bool test_colunsigned long p;pthread_t uffd_mon;char c ='\0' ; -struct uffd_args args = {0 }; - args.gopts = gopts; +struct uffd_args args = { .gopts = gopts }; +bool bad_contents;/* * NOTE: MADV_COLLAPSE is not yet compatible with WP, so testing
@@ -564,20 +577,59 @@ static void uffd_minor_test_common(uffd_global_test_opts_t *gopts, bool test_colif (pthread_create (&uffd_mon,NULL , uffd_poll_thread, &args))err ("uffd_poll_thread create" ); +if (test_collapse) { +/* +* Read just a single page and try collapsing. The collapse +* should either be rejected or be a no-op. +*/ +if (__check_memory_contents (0 ,1 , gopts, gopts->area_dst_alias)) +err ("unexpected memory contents before collapse" ); + +/* MADV_COLLAPSE might return EINVAL for uffd-minor VMAs. */ +if (madvise (gopts->area_dst_alias, gopts->nr_pages * gopts->page_size, +MADV_COLLAPSE ) && errno !=EINVAL ) +err ("MADV_COLLAPSE failed unexpectedly" ); +/* +* If the above collapse mapped pages that were not explicitly +* CONTINUE'd, the below __check_memory_contents() will not +* fault on some pages, resulting in incorrect contents. +* The page we already touched should not be unmapped, so a +* second userfault on it should not be generated. +*/ + } +/* * Read each of the pages back using the UFFD-registered mapping. We * expect that the first time we touch a page, it will result in a minor * fault. uffd_poll_thread will resolve the fault by bit-flipping the * page's contents, and then issuing a CONTINUE ioctl. */ -check_memory_contents (gopts, gopts->area_dst_alias); + bad_contents = !!check_memory_contents (gopts, gopts->area_dst_alias);if (write (gopts->pipefd[1 ], &c,sizeof (c)) !=sizeof (c))err ("pipe write" );if (pthread_join (uffd_mon,NULL ))err ("join() failed" ); +if (bad_contents && test_collapse) { +uffd_test_fail ("unexpected memory contents after collapse" ); +return ; + } + +if (bad_contents) { +uffd_test_fail ("unexpected memory contents" ); +return ; + } +if (test_collapse) { +/* +* MADV_COLLAPSE will fail unless userfaultfd-minor is +* unregistered. +*/ +if (uffd_unregister (gopts->uffd, gopts->area_dst_alias, + gopts->nr_pages * gopts->page_size)) +err ("uffd_unregister before MADV_COLLAPSE failed" ); +if (madvise (gopts->area_dst_alias, gopts->nr_pages * gopts->page_size,MADV_COLLAPSE )) {/* It's fine to fail for this one... */
@@ -593,7 +645,10 @@ static void uffd_minor_test_common(uffd_global_test_opts_t *gopts, bool test_col* This won't cause uffd-fault - it purely just makes sure there * was no corruption. */ -check_memory_contents (gopts, gopts->area_dst_alias); +if (check_memory_contents (gopts, gopts->area_dst_alias)) { +uffd_test_fail ("unexpected memory contents" ); +return ; + } }if (args.missing_faults !=0 || args.minor_faults != gopts->nr_pages)
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Kiryl Shutsemau
reviewerQuoted 69 lines · James Houghton
Userfaultfd minor faults provides userspace with the ability to manually install PTEs with
UFFDIO_CONTINUE. Right now,MADV_COLLAPSEcan map holes in the VMA when a naturally-aligned THP is present. This is not true for khugepaged collapse: the PTEs will be retracted, but a PMD will not be installed.When
MADV_COLLAPSEinstalls a PMD that mapped holes in the VMA, userspace is likely to expectUFFDIO_CONTINUEto succeed on the should-be holes.UFFDIO_CONTINUEwill fail and return EEXIST.This is not inherently a problem, as
MADV_COLLAPSEis an explicit userspace action. But, especially becauseMADV_COLLAPSEcan be invoked by an external process viaprocess_madvise(), a rogue caller could break a userfaultfd-minor resolver thread.Userspace cannot generally use
MADV_COLLAPSEto resolve userfault minor faults, asMADV_COLLAPSEwill only resolve such faults if a naturally-aligned THP is present, so this is not a functional regression for userspace.The naturally-aligned THP case is the only case where this quirk exists. Collapsing otherwise requires all PTEs to be present for userfaultfd-registered VMAs (i.e., max none PTEs is 0), which is correct. This check is essentially bypassed for naturally-aligned THPs.
Suggested-by Lance Yang <[email protected]>Tested-by Lance Yang <[email protected]>Signed-off-by James Houghton <[email protected]>---
Changes since v3:
- Prevent page table retraction in patch 1. Please see the comment next to it. So this patch now has two hunks instead of one.
- Reworded the comment and combined the userfaultfd checks in patch 1. (Thanks David)
- Simplified the selftest a bit given the behavior change in patch 1.
- Undid a change in v3's selftest that incorrectly handled the case whereMADV_COLLAPSEwas not supported entirely.
- Rebased on top of mm-unstable, which includes Kiryl's changes.
v3: https://lore.kernel.org/linux-mm/[email protected]/
v2: https://lore.kernel.org/linux-mm/[email protected]/
v1: https://lore.kernel.org/linux-mm/[email protected]/--- mm/khugepaged.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)mm/khugepaged.c +5
-3@@ -1868,10 +1868,11 @@ static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsignreturn SCAN_VMA_CHECK ;/* -* Keep pmd pgtable while the uffd bit is in use; see comment in -* retract_page_tables(). +* Don't collapse if there might be PTE markers for userfaultfd-based +* access protection or if collapsing might bypass userfaultfd minor +* faults. */ -if (userfaultfd_protected (vma)) +if (userfaultfd_protected (vma) ||userfaultfd_minor (vma))return SCAN_PTE_UFFD ; folio = filemap_lock_folio(vma->vm_file->f_mapping,@@ -2093,8 +2094,13 @@ static bool file_backed_vma_is_retractable(struct vm_area_struct *vma)* and cannot be recycled to a shared PMD. Other vmas can still * have the same file mapped hugely, but skip this one: it will * always be mapped in small page size for these registrations. +*
mm/khugepaged.c static bool file_backed_vma_is_retractable(struct vm_area_struct *vma) · line 2106
Quoted 10 lines
mm/khugepaged.c +5
-1+* Userfaultfd-minor-registered VMAs should also not be retracted. +* PMDs will not be installed, as doing so can suppress minor faults. +* If retraction were allowed, khugepaged might continually cause +* unnecessary userfaultfd minor faults for already-CONTINUE'd pages. */ -if (userfaultfd_protected (vma)) +if (userfaultfd_protected (vma) ||userfaultfd_minor (vma))return false;/*
Did you run 2/2 on current mm-unstable?
Here, on e8d0f6a1b2a4 plus this series, minor-collapse fails every run:
# failed reason: [unexpected memory contents after collapse] not ok 1 minor-collapse on shmem
This patch refuses the collapse too late.
collapse_scan_file() has no VMA, so the uffd check in collapse_max_ptes_none() never applies, and collapse_file() goes ahead:
it try_to_unmap()s the PTEs userspace already CONTINUE'd and replaces the page cache.
Only then does try_collapse_pte_mapped_thp() return SCAN_PTE_UFFD.
MADV_COLLAPSE fails with -EINVAL, but the page the test read beforehand is already unmapped, takes a second minor fault, and gets flipped twice.
Also, your 2/2 missing Signed-off-by.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Quoted 85 lines · Kiryl Shutsemau
Quoted 69 lines · James Houghton
Userfaultfd minor faults provides userspace with the ability to manually install PTEs with
UFFDIO_CONTINUE. Right now,MADV_COLLAPSEcan map holes in the VMA when a naturally-aligned THP is present. This is not true for khugepaged collapse: the PTEs will be retracted, but a PMD will not be installed.When
MADV_COLLAPSEinstalls a PMD that mapped holes in the VMA, userspace is likely to expectUFFDIO_CONTINUEto succeed on the should-be holes.UFFDIO_CONTINUEwill fail and return EEXIST.This is not inherently a problem, as
MADV_COLLAPSEis an explicit userspace action. But, especially becauseMADV_COLLAPSEcan be invoked by an external process viaprocess_madvise(), a rogue caller could break a userfaultfd-minor resolver thread.Userspace cannot generally use
MADV_COLLAPSEto resolve userfault minor faults, asMADV_COLLAPSEwill only resolve such faults if a naturally-aligned THP is present, so this is not a functional regression for userspace.The naturally-aligned THP case is the only case where this quirk exists. Collapsing otherwise requires all PTEs to be present for userfaultfd-registered VMAs (i.e., max none PTEs is 0), which is correct. This check is essentially bypassed for naturally-aligned THPs.
Suggested-by Lance Yang <[email protected]>Tested-by Lance Yang <[email protected]>Signed-off-by James Houghton <[email protected]>---
Changes since v3:
- Prevent page table retraction in patch 1. Please see the comment next to it. So this patch now has two hunks instead of one.
- Reworded the comment and combined the userfaultfd checks in patch 1. (Thanks David)
- Simplified the selftest a bit given the behavior change in patch 1.
- Undid a change in v3's selftest that incorrectly handled the case whereMADV_COLLAPSEwas not supported entirely.
- Rebased on top of mm-unstable, which includes Kiryl's changes.
v3: https://lore.kernel.org/linux-mm/[email protected]/
v2: https://lore.kernel.org/linux-mm/[email protected]/
v1: https://lore.kernel.org/linux-mm/[email protected]/--- mm/khugepaged.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)mm/khugepaged.c +5
-3@@ -1868,10 +1868,11 @@ static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsignreturn SCAN_VMA_CHECK ;/* -* Keep pmd pgtable while the uffd bit is in use; see comment in -* retract_page_tables(). +* Don't collapse if there might be PTE markers for userfaultfd-based +* access protection or if collapsing might bypass userfaultfd minor +* faults. */ -if (userfaultfd_protected (vma)) +if (userfaultfd_protected (vma) ||userfaultfd_minor (vma))return SCAN_PTE_UFFD ; folio = filemap_lock_folio(vma->vm_file->f_mapping,@@ -2093,8 +2094,13 @@ static bool file_backed_vma_is_retractable(struct vm_area_struct *vma)* and cannot be recycled to a shared PMD. Other vmas can still * have the same file mapped hugely, but skip this one: it will * always be mapped in small page size for these registrations. +* mm/khugepaged.c static bool file_backed_vma_is_retractable(struct vm_area_struct *vma) · line 2106
Quoted 10 lines
mm/khugepaged.c +5
-1+* Userfaultfd-minor-registered VMAs should also not be retracted. +* PMDs will not be installed, as doing so can suppress minor faults. +* If retraction were allowed, khugepaged might continually cause +* unnecessary userfaultfd minor faults for already-CONTINUE'd pages. */ -if (userfaultfd_protected (vma)) +if (userfaultfd_protected (vma) ||userfaultfd_minor (vma))return false;/* Did you run 2/2 on current mm-unstable?
Quoted 4 lines
Here, on
e8d0f6a1b2a4plus this series, minor-collapse fails every run:# failed reason: [unexpected memory contents after collapse] not ok 1 minor-collapse on shmem
Oh, I must have missed this when testing the unit test "simplification" for this v4. :( Thanks for testing it.
It looks like uffd-wp can tolerate the try_to_unmap() because cond_install_uffd_wp_ptes() will get called in the process.
Quoted 11 lines
This patch refuses the collapse too late.
collapse_scan_file()has no VMA, so the uffd check incollapse_max_ptes_none()never applies, andcollapse_file()goes ahead:
it try_to_unmap()s the PTEs userspace already CONTINUE'd and replaces the page cache.Only then does
try_collapse_pte_mapped_thp()returnSCAN_PTE_UFFD.
MADV_COLLAPSEfails with-EINVAL, but the page the test read beforehand is already unmapped, takes a second minor fault, and gets flipped twice.
Thanks for explaining that, I didn't notice the try_to_unmap(). The v3 version of the selftest didn't have this problem because it only attempted to access each page once.
The purpose of this patch is just to avoid installing PMDs; unmapping PTEs is acceptable. I think the logic to truly prevent unmapping in this case probably isn't worth it. Let me know if you disagree.
So I'll just go ahead with fixing the self-test, applying a diff like this (sorry for the formatting), undoing the "simplification" from v3->v4:
(quoted diff) +0 -0
b/tools/testing/selftests/mm/uffd-unit-tests.c
index 74ce43769e58..48435faa8bc8 100644
tools/testing/selftests/mm/uffd-unit-tests.c +0 -0
@@ -553,6 +553,7 @@ static void
uffd_minor_test_common(uffd_global_test_opts_t *gopts, bool test_col
char c = '\0';
struct uffd_args args = { .gopts = gopts };
bool bad_contents;
+ int checked = 0;
/*
* NOTE: MADV_COLLAPSE is not yet compatible with WP, so testing
@@ -596,6 +597,7 @@ static void uffd_minor_test_common(uffd_global_test_opts_t *gopts, bool test_col
* The page we already touched should not be unmapped, so a
* second userfault on it should not be generated.
*/
+ checked = 1;
}
/*
@@ -604,7 +606,8 @@ static void uffd_minor_test_common(uffd_global_test_opts_t *gopts, bool test_col
* fault. uffd_poll_thread will resolve the fault by bit-flipping the
* page's contents, and then issuing a CONTINUE ioctl.
*/
- bad_contents = !!check_memory_contents(gopts, gopts->area_dst_alias);
+ bad_contents = !!__check_memory_contents(checked,
+ opts->nr_pages - checked, gopts, gopts->area_dst_alias);
Quoted 1 line
Also, your 2/2 missing Signed-off-by.
Thanks, I'll fix that.
I'm actually away right now, so I'll be slow responding to feedback and can't send a new version until next week.
Thanks for your patience, Kiryl.