mm/ksm: fix userfaultfd protection during merge and unmerge
Subsystems (4)
PATCH 1/1 mm/ksm: fix userfaultfd protection during merge and unmerge
KSM loses userfaultfd protection when replace_page() builds a new PTE from vma->vm_page_prot. Dropping its uffd bit lets a subsequent write go through ordinary COW instead of userfaultfd write protection handling. For RWP, a merged page also becomes readable without faulting.
Keeping the uffd bit and restoring PAGE_NONE fixes merging, but we also have to handle unmerging. An existing KSM page can be protected with RWP before MADV_UNMERGEABLE, so unmerging already needs fixing independently of preserving RWP during merging.
Unsharing replaces a shared page with a private one and should leave userfaultfd protection in place, as do_wp_page() already does for write protection. Currently, an unshare fault reaches do_uffd_rwp() before it can proceed to do_wp_page().
- With synchronous RWP, handle_userfault() returns VM_FAULT_SIGBUS. break_ksm() then reports success with a KSM page still merged.
- With asynchronous RWP, protection is cleared before unsharing.
Preserve the uffd bit when merging, and restore PAGE_NONE for RWP. Let FAULT_FLAG_UNSHARE skip RWP fault handling and keep PAGE_NONE on each private copy as well. Unsharing can then complete with RWP still armed for a subsequent access.
--- mm/ksm.c | 8 +++++++- mm/memory.c | 14 ++++++++++---- 2 files changed, 17 insertions(+), 5 deletions(-)
mm/ksm.c +7 -1
@@ -34,6 +34,7 @@#include <linux/memory.h> #include <linux/mmu_notifier.h> #include <linux/swap.h> +#include <linux/userfaultfd_k.h> #include <linux/ksm.h> #include <linux/hashtable.h> #include <linux/freezer.h>
@@ -1463,9 +1464,14 @@ static int replace_page(struct vm_area_struct *vma, struct page *page,dec_mm_counter (mm,MM_ANONPAGES ); } -/* Preserve soft-dirty when replacing the page. */ +/* Preserve soft-dirty and uffd protection when replacing the page. */ if (pte_soft_dirty (orig_pte)) newpte =pte_mksoft_dirty (newpte); +if (pte_uffd (orig_pte)) { + newpte =pte_mkuffd (newpte); +if (userfaultfd_rwp (vma)) + newpte =pte_modify (newpte,PAGE_NONE ); + }flush_cache_page (vma, addr,pte_pfn (ptep_get (ptep)));/*
mm/memory.c +10 -4
@@ -4122,8 +4122,11 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)if (unlikely (unshare)) {if (pte_soft_dirty (vmf->orig_pte)) entry =pte_mksoft_dirty (entry); -if (pte_uffd (vmf->orig_pte)) +if (pte_uffd (vmf->orig_pte)) { entry =pte_mkuffd (entry); +if (userfaultfd_rwp (vma)) + entry =pte_modify (entry,PAGE_NONE ); + } }else { entry =maybe_mkwrite (pte_mkdirty (entry), vma); }
@@ -6659,9 +6662,12 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf)* VM_UFFD_RWP VMA, a protnone PTE without the uffd bit is * NUMA hinting and must still fall through to do_numa_page(). */ -if (userfaultfd_pte_rwp (vmf->vma, vmf->orig_pte)) -return do_uffd_rwp (vmf); -return do_numa_page (vmf); +if (userfaultfd_pte_rwp (vmf->vma, vmf->orig_pte)) { +if (!(vmf->flags &FAULT_FLAG_UNSHARE )) +return do_uffd_rwp (vmf); + }else { +return do_numa_page (vmf); + } }spin_lock (vmf->ptl);
Signature · 1 line
2.49.0
Quoted 4 lines · Lance Yang
KSM loses userfaultfd protection when
replace_page()builds a new PTE fromvma->vm_page_prot. Dropping its uffd bit lets a subsequent write go through ordinary COW instead of userfaultfd write protection handling. For RWP, a merged page also becomes readable without faulting.
I feel we probably shouldn't have allowed an extension to UFFD while it's still fundamentally a broken feature :))
Which is evidenced by the SLEW of AI bug reports, and human-reported bug reports, and endless history of bug reports and it infecting the rest of mm, etc. etc. /rant
Quoted 21 lines
Keeping the uffd bit and restoring
PAGE_NONEfixes merging, but we also have to handle unmerging. An existing KSM page can be protected with RWP beforeMADV_UNMERGEABLE, so unmerging already needs fixing independently of preserving RWP during merging.Unsharing replaces a shared page with a private one and should leave userfaultfd protection in place, as
do_wp_page()already does for write protection. Currently, an unshare fault reachesdo_uffd_rwp()before it can proceed todo_wp_page().- With synchronous RWP,
handle_userfault()returnsVM_FAULT_SIGBUS.break_ksm()then reports success with a KSM page still merged.- With asynchronous RWP, protection is cleared before unsharing.
Preserve the uffd bit when merging, and restore
PAGE_NONEfor RWP. LetFAULT_FLAG_UNSHAREskip RWP fault handling and keepPAGE_NONEon each private copy as well. Unsharing can then complete with RWP still armed for a subsequent access.
Quoted 8 lines
Fixes 63b2d4174c4a ("userfaultfd: wp: add the writeprotect API to userfaultfd ioctl")Fixes 2d427bb01639 ("mm/userfaultfd: add RWP fault delivery and expose UFFDIO_REGISTER_MODE_RWP")Co-developed-by Samarth Narang <[email protected]>Signed-off-by Samarth Narang <[email protected]>Signed-off-by Lance Yang <[email protected]>
Looks sensible to me so:
Quoted 19 lines
--- mm/ksm.c | 8 +++++++- mm/memory.c | 14 ++++++++++---- 2 files changed, 17 insertions(+), 5 deletions(-)mm/ksm.c +1
-0@@ -34,6 +34,7 @@#include <linux/memory.h> #include <linux/mmu_notifier.h> #include <linux/swap.h> +#include <linux/userfaultfd_k.h> #include <linux/ksm.h> #include <linux/hashtable.h> #include <linux/freezer.h> @@ -1463,9 +1464,14 @@ static int replace_page(struct vm_area_struct *vma, struct page *page,dec_mm_counter (mm,MM_ANONPAGES );
mm/ksm.c static int replace_page(struct vm_area_struct *vma, struct page *page, · line 1473
Quoted 10 lines
mm/ksm.c +5
-1} -/* Preserve soft-dirty when replacing the page. */ +/* Preserve soft-dirty and uffd protection when replacing the page. */ if (pte_soft_dirty (orig_pte)) newpte =pte_mksoft_dirty (newpte); +if (pte_uffd (orig_pte)) { + newpte =pte_mkuffd (newpte); +if (userfaultfd_rwp (vma)) + newpte =pte_modify (newpte,PAGE_NONE );
OK, so by definition, RWP pages are protnone, and this resets it correctly.
Quoted 9 lines
mm/ksm.c +1
-0+ }flush_cache_page (vma, addr,pte_pfn (ptep_get (ptep)));/* mm/memory.c +0
-0
mm/memory.c static vm_fault_t wp_page_copy(struct vm_fault *vmf) · line 4129
Quoted 9 lines
mm/memory.c +4
-1@@ -4122,8 +4122,11 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)if (unlikely (unshare)) {if (pte_soft_dirty (vmf->orig_pte)) entry =pte_mksoft_dirty (entry); -if (pte_uffd (vmf->orig_pte)) +if (pte_uffd (vmf->orig_pte)) { entry =pte_mkuffd (entry); +if (userfaultfd_rwp (vma)) + entry =pte_modify (entry,PAGE_NONE ); + }
Could this be abstracted? You're doing the same pattern here, which also exists in copy_present_page() and presumably many other places.
But maybe that's one for a follow up?
Quoted 20 lines
mm/memory.c +6
-4}else { entry =maybe_mkwrite (pte_mkdirty (entry), vma); }@@ -6659,9 +6662,12 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf)* VM_UFFD_RWP VMA, a protnone PTE without the uffd bit is * NUMA hinting and must still fall through to do_numa_page(). */ -if (userfaultfd_pte_rwp (vmf->vma, vmf->orig_pte)) -return do_uffd_rwp (vmf); -return do_numa_page (vmf); +if (userfaultfd_pte_rwp (vmf->vma, vmf->orig_pte)) { +if (!(vmf->flags &FAULT_FLAG_UNSHARE )) +return do_uffd_rwp (vmf); + }else { +return do_numa_page (vmf); + } }spin_lock (vmf->ptl); --
Quoted 2 lines
2.49.0
--
Cheers, Lorenzo
Quoted 6 lines · Lorenzo Stoakes (ARM)
Quoted 4 lines · Lance Yang
KSM loses userfaultfd protection when
replace_page()builds a new PTE fromvma->vm_page_prot. Dropping its uffd bit lets a subsequent write go through ordinary COW instead of userfaultfd write protection handling. For RWP, a merged page also becomes readable without faulting.
Quoted 6 lines
I feel we probably shouldn't have allowed an extension to UFFD while it's still fundamentally a broken feature :))
Which is evidenced by the SLEW of AI bug reports, and human-reported bug reports, and endless history of bug reports and it infecting the rest of mm, etc. etc. /rant
Heh, UFFD isn't winning you over today :D
Quoted 30 lines
Quoted 21 lines
Keeping the uffd bit and restoring
PAGE_NONEfixes merging, but we also have to handle unmerging. An existing KSM page can be protected with RWP beforeMADV_UNMERGEABLE, so unmerging already needs fixing independently of preserving RWP during merging.Unsharing replaces a shared page with a private one and should leave userfaultfd protection in place, as
do_wp_page()already does for write protection. Currently, an unshare fault reachesdo_uffd_rwp()before it can proceed todo_wp_page().- With synchronous RWP,
handle_userfault()returnsVM_FAULT_SIGBUS.break_ksm()then reports success with a KSM page still merged.- With asynchronous RWP, protection is cleared before unsharing.
Preserve the uffd bit when merging, and restore
PAGE_NONEfor RWP. LetFAULT_FLAG_UNSHAREskip RWP fault handling and keepPAGE_NONEon each private copy as well. Unsharing can then complete with RWP still armed for a subsequent access.Quoted 8 lines
Fixes 63b2d4174c4a ("userfaultfd: wp: add the writeprotect API to userfaultfd ioctl")Fixes 2d427bb01639 ("mm/userfaultfd: add RWP fault delivery and expose UFFDIO_REGISTER_MODE_RWP")Co-developed-by Samarth Narang <[email protected]>Signed-off-by Samarth Narang <[email protected]>Signed-off-by Lance Yang <[email protected]>
Quoted 3 lines
Looks sensible to me so:
Really appreciated!
Quoted 21 lines
Quoted 21 lines
--- mm/ksm.c | 8 +++++++- mm/memory.c | 14 ++++++++++---- 2 files changed, 17 insertions(+), 5 deletions(-)mm/ksm.c +1
-0@@ -34,6 +34,7 @@#include <linux/memory.h> #include <linux/mmu_notifier.h> #include <linux/swap.h> +#include <linux/userfaultfd_k.h> #include <linux/ksm.h> #include <linux/hashtable.h> #include <linux/freezer.h> @@ -1463,9 +1464,14 @@ static int replace_page(struct vm_area_struct *vma, struct page *page,dec_mm_counter (mm,MM_ANONPAGES ); }
Quoted 10 lines
mm/ksm.c static int replace_page(struct vm_area_struct *vma, struct page *page, · line 1473
Quoted 8 lines
mm/ksm.c +5
-1-/* Preserve soft-dirty when replacing the page. */ +/* Preserve soft-dirty and uffd protection when replacing the page. */ if (pte_soft_dirty (orig_pte)) newpte =pte_mksoft_dirty (newpte); +if (pte_uffd (orig_pte)) { + newpte =pte_mkuffd (newpte); +if (userfaultfd_rwp (vma)) + newpte =pte_modify (newpte,PAGE_NONE );OK, so by definition, RWP pages are protnone, and this resets it correctly.
Right, we need PAGE_NONE back as well as the uffd bit :)
Quoted 22 lines
Quoted 9 lines
mm/ksm.c +1
-0+ }flush_cache_page (vma, addr,pte_pfn (ptep_get (ptep)));/* mm/memory.c +0
-0mm/memory.c static vm_fault_t wp_page_copy(struct vm_fault *vmf) · line 4129
Quoted 9 lines
mm/memory.c +4
-1@@ -4122,8 +4122,11 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)if (unlikely (unshare)) {if (pte_soft_dirty (vmf->orig_pte)) entry =pte_mksoft_dirty (entry); -if (pte_uffd (vmf->orig_pte)) +if (pte_uffd (vmf->orig_pte)) { entry =pte_mkuffd (entry); +if (userfaultfd_rwp (vma)) + entry =pte_modify (entry,PAGE_NONE ); + }Could this be abstracted? You're doing the same pattern here, which also exists in
copy_present_page()and presumably many other places.
Quoted 1 line
But maybe that's one for a follow up?
Yeah, a helper would be nice here. I'd keep this fix small and do that as a follow-up :D
Cheers, Lance
Quoted 25 lines
Quoted 20 lines
mm/memory.c +6
-4}else { entry =maybe_mkwrite (pte_mkdirty (entry), vma); }@@ -6659,9 +6662,12 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf)* VM_UFFD_RWP VMA, a protnone PTE without the uffd bit is * NUMA hinting and must still fall through to do_numa_page(). */ -if (userfaultfd_pte_rwp (vmf->vma, vmf->orig_pte)) -return do_uffd_rwp (vmf); -return do_numa_page (vmf); +if (userfaultfd_pte_rwp (vmf->vma, vmf->orig_pte)) { +if (!(vmf->flags &FAULT_FLAG_UNSHARE )) +return do_uffd_rwp (vmf); + }else { +return do_numa_page (vmf); + } }spin_lock (vmf->ptl); --Quoted 2 lines
2.49.0
--
Cheers, Lorenzo
Quoted 25 lines · Lance Yang
KSM loses userfaultfd protection when
replace_page()builds a new PTE fromvma->vm_page_prot. Dropping its uffd bit lets a subsequent write go through ordinary COW instead of userfaultfd write protection handling. For RWP, a merged page also becomes readable without faulting.Keeping the uffd bit and restoring
PAGE_NONEfixes merging, but we also have to handle unmerging. An existing KSM page can be protected with RWP beforeMADV_UNMERGEABLE, so unmerging already needs fixing independently of preserving RWP during merging.Unsharing replaces a shared page with a private one and should leave userfaultfd protection in place, as
do_wp_page()already does for write protection. Currently, an unshare fault reachesdo_uffd_rwp()before it can proceed todo_wp_page().- With synchronous RWP,
handle_userfault()returnsVM_FAULT_SIGBUS.break_ksm()then reports success with a KSM page still merged.- With asynchronous RWP, protection is cleared before unsharing.
Preserve the uffd bit when merging, and restore
PAGE_NONEfor RWP. LetFAULT_FLAG_UNSHAREskip RWP fault handling and keepPAGE_NONEon each private copy as well. Unsharing can then complete with RWP still armed for a subsequent access.
Quoted 8 lines
Fixes 63b2d4174c4a ("userfaultfd: wp: add the writeprotect API to userfaultfd ioctl")Fixes 2d427bb01639 ("mm/userfaultfd: add RWP fault delivery and expose UFFDIO_REGISTER_MODE_RWP")Co-developed-by Samarth Narang <[email protected]>Signed-off-by Samarth Narang <[email protected]>Signed-off-by Lance Yang <[email protected]>
Thanks for fixing this:
mm/ksm.c static int replace_page(struct vm_area_struct *vma, struct page *page, · line 1467
Quoted 6 lines
mm/ksm.c +1
-1@@ -1463,9 +1464,14 @@ static int replace_page(struct vm_area_struct *vma, struct page *page,dec_mm_counter (mm,MM_ANONPAGES ); } -/* Preserve soft-dirty when replacing the page. */ +/* Preserve soft-dirty and uffd protection when replacing the page. */
Small nit: the period at the end of a single-line comment looks odd to me. Am I the only one?
If you are going to respin the patch for some reason, maybe drop it?
Quoted 10 lines
mm/ksm.c +5
-0if (pte_soft_dirty (orig_pte)) newpte =pte_mksoft_dirty (newpte); +if (pte_uffd (orig_pte)) { + newpte =pte_mkuffd (newpte); +if (userfaultfd_rwp (vma)) + newpte =pte_modify (newpte,PAGE_NONE ); + }flush_cache_page (vma, addr,pte_pfn (ptep_get (ptep)));/*
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
David Hildenbrand (Arm)
maintainerQuoted 43 lines · Kiryl Shutsemau
Quoted 25 lines · Lance Yang
KSM loses userfaultfd protection when
replace_page()builds a new PTE fromvma->vm_page_prot. Dropping its uffd bit lets a subsequent write go through ordinary COW instead of userfaultfd write protection handling. For RWP, a merged page also becomes readable without faulting.Keeping the uffd bit and restoring
PAGE_NONEfixes merging, but we also have to handle unmerging. An existing KSM page can be protected with RWP beforeMADV_UNMERGEABLE, so unmerging already needs fixing independently of preserving RWP during merging.Unsharing replaces a shared page with a private one and should leave userfaultfd protection in place, as
do_wp_page()already does for write protection. Currently, an unshare fault reachesdo_uffd_rwp()before it can proceed todo_wp_page().- With synchronous RWP,
handle_userfault()returnsVM_FAULT_SIGBUS.break_ksm()then reports success with a KSM page still merged.- With asynchronous RWP, protection is cleared before unsharing.
Preserve the uffd bit when merging, and restore
PAGE_NONEfor RWP. LetFAULT_FLAG_UNSHAREskip RWP fault handling and keepPAGE_NONEon each private copy as well. Unsharing can then complete with RWP still armed for a subsequent access.Quoted 8 lines
Fixes 63b2d4174c4a ("userfaultfd: wp: add the writeprotect API to userfaultfd ioctl")Fixes 2d427bb01639 ("mm/userfaultfd: add RWP fault delivery and expose UFFDIO_REGISTER_MODE_RWP")Co-developed-by Samarth Narang <[email protected]>Signed-off-by Samarth Narang <[email protected]>Signed-off-by Lance Yang <[email protected]>Thanks for fixing this:
Reviewed-by Kiryl Shutsemau (Meta) <[email protected]>Quoted 4 lines
mm/ksm.c +0
-0@@ -1463,9 +1464,14 @@ static int replace_page(struct vm_area_struct *vma, struct page *page,dec_mm_counter (mm,MM_ANONPAGES ); }
Quoted 5 lines
mm/ksm.c static int replace_page(struct vm_area_struct *vma, struct page *page, · line 1467
Quoted 2 lines
mm/ksm.c +1
-1-/* Preserve soft-dirty when replacing the page. */ +/* Preserve soft-dirty and uffd protection when replacing the page. */ Small nit: the period at the end of a single-line comment looks odd to me. Am I the only one?
I think you are. :)
I would complain if something looks like a sentence but doesn't end with a period.
It would be different it it's not actually a full sentence.
/* preserve soft-dirty */
Signature · 3 lines
Cheers, David
Quoted 25 lines · Lance Yang
KSM loses userfaultfd protection when
replace_page()builds a new PTE fromvma->vm_page_prot. Dropping its uffd bit lets a subsequent write go through ordinary COW instead of userfaultfd write protection handling. For RWP, a merged page also becomes readable without faulting.Keeping the uffd bit and restoring
PAGE_NONEfixes merging, but we also have to handle unmerging. An existing KSM page can be protected with RWP beforeMADV_UNMERGEABLE, so unmerging already needs fixing independently of preserving RWP during merging.Unsharing replaces a shared page with a private one and should leave userfaultfd protection in place, as
do_wp_page()already does for write protection. Currently, an unshare fault reachesdo_uffd_rwp()before it can proceed todo_wp_page().- With synchronous RWP,
handle_userfault()returnsVM_FAULT_SIGBUS.break_ksm()then reports success with a KSM page still merged.- With asynchronous RWP, protection is cleared before unsharing.
Preserve the uffd bit when merging, and restore
PAGE_NONEfor RWP. LetFAULT_FLAG_UNSHAREskip RWP fault handling and keepPAGE_NONEon each private copy as well. Unsharing can then complete with RWP still armed for a subsequent access.
Quoted 8 lines
Fixes 63b2d4174c4a ("userfaultfd: wp: add the writeprotect API to userfaultfd ioctl")Fixes 2d427bb01639 ("mm/userfaultfd: add RWP fault delivery and expose UFFDIO_REGISTER_MODE_RWP")Co-developed-by Samarth Narang <[email protected]>Signed-off-by Samarth Narang <[email protected]>Signed-off-by Lance Yang <[email protected]>
Signature · 3 lines
Cheers, David