Skip to contentSkip to conversation

mm/ksm: fix userfaultfd protection during merge and unmerge

PATCH Applied Lance Yang 4 participants 6 messages Oct 26 – Oct 27

Subsystems (4)
lore
Share & tools
lore

Lance Yang

author
#0 lore raw

PATCH 1/1 mm/ksm: fix userfaultfd protection during merge and unmerge

KSM loses userfaultfd protection when replace_page() builds a new PTE from vma->vm_page_prot. Dropping its uffd bit lets a subsequent write go through ordinary COW instead of userfaultfd write protection handling. For RWP, a merged page also becomes readable without faulting.

Keeping the uffd bit and restoring PAGE_NONE fixes merging, but we also have to handle unmerging. An existing KSM page can be protected with RWP before MADV_UNMERGEABLE, so unmerging already needs fixing independently of preserving RWP during merging.

Unsharing replaces a shared page with a private one and should leave userfaultfd protection in place, as do_wp_page() already does for write protection. Currently, an unshare fault reaches do_uffd_rwp() before it can proceed to do_wp_page().

- With synchronous RWP, handle_userfault() returns VM_FAULT_SIGBUS. break_ksm() then reports success with a KSM page still merged.

- With asynchronous RWP, protection is cleared before unsharing.

Preserve the uffd bit when merging, and restore PAGE_NONE for RWP. Let FAULT_FLAG_UNSHARE skip RWP fault handling and keep PAGE_NONE on each private copy as well. Unsharing can then complete with RWP still armed for a subsequent access.

Fixes 63b2d4174c4a ("userfaultfd: wp: add the writeprotect API to userfaultfd ioctl")
Fixes 2d427bb01639 ("mm/userfaultfd: add RWP fault delivery and expose UFFDIO_REGISTER_MODE_RWP")
---
 mm/ksm.c    |  8 +++++++-
 mm/memory.c | 14 ++++++++++----
 2 files changed, 17 insertions(+), 5 deletions(-)
mm/ksm.c +7 -1
index 910a1307ac9f..d6f736a1aa2f 100644
@@ -34,6 +34,7 @@
 #include <linux/memory.h>
 #include <linux/mmu_notifier.h>
 #include <linux/swap.h>
+#include <linux/userfaultfd_k.h>
 #include <linux/ksm.h>
 #include <linux/hashtable.h>
 #include <linux/freezer.h>
@@ -1463,9 +1464,14 @@ static int replace_page(struct vm_area_struct *vma, struct page *page,
 		dec_mm_counter(mm, MM_ANONPAGES);
 	}
 
-	/* Preserve soft-dirty when replacing the page. */
+	/* Preserve soft-dirty and uffd protection when replacing the page. */
 	if (pte_soft_dirty(orig_pte))
 		newpte = pte_mksoft_dirty(newpte);
+	if (pte_uffd(orig_pte)) {
+		newpte = pte_mkuffd(newpte);
+		if (userfaultfd_rwp(vma))
+			newpte = pte_modify(newpte, PAGE_NONE);
+	}
 
 	flush_cache_page(vma, addr, pte_pfn(ptep_get(ptep)));
 	/*
mm/memory.c +10 -4
index 6a93b7fa48b2..99439a7afa63 100644
@@ -4122,8 +4122,11 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)
 		if (unlikely(unshare)) {
 			if (pte_soft_dirty(vmf->orig_pte))
 				entry = pte_mksoft_dirty(entry);
-			if (pte_uffd(vmf->orig_pte))
+			if (pte_uffd(vmf->orig_pte)) {
 				entry = pte_mkuffd(entry);
+				if (userfaultfd_rwp(vma))
+					entry = pte_modify(entry, PAGE_NONE);
+			}
 		} else {
 			entry = maybe_mkwrite(pte_mkdirty(entry), vma);
 		}
@@ -6659,9 +6662,12 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf)
 		 * VM_UFFD_RWP VMA, a protnone PTE without the uffd bit is
 		 * NUMA hinting and must still fall through to do_numa_page().
 		 */
-		if (userfaultfd_pte_rwp(vmf->vma, vmf->orig_pte))
-			return do_uffd_rwp(vmf);
-		return do_numa_page(vmf);
+		if (userfaultfd_pte_rwp(vmf->vma, vmf->orig_pte)) {
+			if (!(vmf->flags & FAULT_FLAG_UNSHARE))
+				return do_uffd_rwp(vmf);
+		} else {
+			return do_numa_page(vmf);
+		}
 	}
 
 	spin_lock(vmf->ptl);
Signature · 1 line
2.49.0
reviewer Acked-by on v1 1/1
#1 lore raw
Quoted 4 lines · Lance Yang

KSM loses userfaultfd protection when replace_page() builds a new PTE from vma->vm_page_prot. Dropping its uffd bit lets a subsequent write go through ordinary COW instead of userfaultfd write protection handling. For RWP, a merged page also becomes readable without faulting.

I feel we probably shouldn't have allowed an extension to UFFD while it's still fundamentally a broken feature :))

Which is evidenced by the SLEW of AI bug reports, and human-reported bug reports, and endless history of bug reports and it infecting the rest of mm, etc. etc. /rant

Quoted 21 lines

Keeping the uffd bit and restoring PAGE_NONE fixes merging, but we also have to handle unmerging. An existing KSM page can be protected with RWP before MADV_UNMERGEABLE, so unmerging already needs fixing independently of preserving RWP during merging.

Unsharing replaces a shared page with a private one and should leave userfaultfd protection in place, as do_wp_page() already does for write protection. Currently, an unshare fault reaches do_uffd_rwp() before it can proceed to do_wp_page().

- With synchronous RWP, handle_userfault() returns VM_FAULT_SIGBUS. break_ksm() then reports success with a KSM page still merged.

- With asynchronous RWP, protection is cleared before unsharing.

Preserve the uffd bit when merging, and restore PAGE_NONE for RWP. Let FAULT_FLAG_UNSHARE skip RWP fault handling and keep PAGE_NONE on each private copy as well. Unsharing can then complete with RWP still armed for a subsequent access.

Quoted 8 lines
Fixes 63b2d4174c4a ("userfaultfd: wp: add the writeprotect API to userfaultfd ioctl")
Fixes 2d427bb01639 ("mm/userfaultfd: add RWP fault delivery and expose UFFDIO_REGISTER_MODE_RWP")

Looks sensible to me so:

Quoted 19 lines
---
 mm/ksm.c    |  8 +++++++-
 mm/memory.c | 14 ++++++++++----
 2 files changed, 17 insertions(+), 5 deletions(-)
mm/ksm.c +1 -0
index 910a1307ac9f..d6f736a1aa2f 100644
@@ -34,6 +34,7 @@
 #include <linux/memory.h>
 #include <linux/mmu_notifier.h>
 #include <linux/swap.h>
+#include <linux/userfaultfd_k.h>
 #include <linux/ksm.h>
 #include <linux/hashtable.h>
 #include <linux/freezer.h>
@@ -1463,9 +1464,14 @@ static int replace_page(struct vm_area_struct *vma, struct page *page,
 		dec_mm_counter(mm, MM_ANONPAGES);

mm/ksm.c static int replace_page(struct vm_area_struct *vma, struct page *page, · line 1473

Quoted 10 lines
mm/ksm.c +5 -1
 	}
 
-	/* Preserve soft-dirty when replacing the page. */
+	/* Preserve soft-dirty and uffd protection when replacing the page. */
 	if (pte_soft_dirty(orig_pte))
 		newpte = pte_mksoft_dirty(newpte);
+	if (pte_uffd(orig_pte)) {
+		newpte = pte_mkuffd(newpte);
+		if (userfaultfd_rwp(vma))
+			newpte = pte_modify(newpte, PAGE_NONE);

OK, so by definition, RWP pages are protnone, and this resets it correctly.

Quoted 9 lines
mm/ksm.c +1 -0
+	}
 
 	flush_cache_page(vma, addr, pte_pfn(ptep_get(ptep)));
 	/*
mm/memory.c +0 -0
index 6a93b7fa48b2..99439a7afa63 100644

mm/memory.c static vm_fault_t wp_page_copy(struct vm_fault *vmf) · line 4129

Quoted 9 lines
mm/memory.c +4 -1
@@ -4122,8 +4122,11 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)
 		if (unlikely(unshare)) {
 			if (pte_soft_dirty(vmf->orig_pte))
 				entry = pte_mksoft_dirty(entry);
-			if (pte_uffd(vmf->orig_pte))
+			if (pte_uffd(vmf->orig_pte)) {
 				entry = pte_mkuffd(entry);
+				if (userfaultfd_rwp(vma))
+					entry = pte_modify(entry, PAGE_NONE);
+			}

Could this be abstracted? You're doing the same pattern here, which also exists in copy_present_page() and presumably many other places.

But maybe that's one for a follow up?

Quoted 20 lines
mm/memory.c +6 -4
 		} else {
 			entry = maybe_mkwrite(pte_mkdirty(entry), vma);
 		}
@@ -6659,9 +6662,12 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf)
 		 * VM_UFFD_RWP VMA, a protnone PTE without the uffd bit is
 		 * NUMA hinting and must still fall through to do_numa_page().
 		 */
-		if (userfaultfd_pte_rwp(vmf->vma, vmf->orig_pte))
-			return do_uffd_rwp(vmf);
-		return do_numa_page(vmf);
+		if (userfaultfd_pte_rwp(vmf->vma, vmf->orig_pte)) {
+			if (!(vmf->flags & FAULT_FLAG_UNSHARE))
+				return do_uffd_rwp(vmf);
+		} else {
+			return do_numa_page(vmf);
+		}
 	}
 
 	spin_lock(vmf->ptl);
--
Quoted 2 lines

2.49.0

--
Cheers, Lorenzo

author
#2 lore raw
Quoted 6 lines · Lorenzo Stoakes (ARM)
Quoted 4 lines · Lance Yang

KSM loses userfaultfd protection when replace_page() builds a new PTE from vma->vm_page_prot. Dropping its uffd bit lets a subsequent write go through ordinary COW instead of userfaultfd write protection handling. For RWP, a merged page also becomes readable without faulting.

Quoted 6 lines

I feel we probably shouldn't have allowed an extension to UFFD while it's still fundamentally a broken feature :))

Which is evidenced by the SLEW of AI bug reports, and human-reported bug reports, and endless history of bug reports and it infecting the rest of mm, etc. etc. /rant

Heh, UFFD isn't winning you over today :D

Quoted 30 lines
Quoted 21 lines

Keeping the uffd bit and restoring PAGE_NONE fixes merging, but we also have to handle unmerging. An existing KSM page can be protected with RWP before MADV_UNMERGEABLE, so unmerging already needs fixing independently of preserving RWP during merging.

Unsharing replaces a shared page with a private one and should leave userfaultfd protection in place, as do_wp_page() already does for write protection. Currently, an unshare fault reaches do_uffd_rwp() before it can proceed to do_wp_page().

- With synchronous RWP, handle_userfault() returns VM_FAULT_SIGBUS. break_ksm() then reports success with a KSM page still merged.

- With asynchronous RWP, protection is cleared before unsharing.

Preserve the uffd bit when merging, and restore PAGE_NONE for RWP. Let FAULT_FLAG_UNSHARE skip RWP fault handling and keep PAGE_NONE on each private copy as well. Unsharing can then complete with RWP still armed for a subsequent access.

Quoted 8 lines
Fixes 63b2d4174c4a ("userfaultfd: wp: add the writeprotect API to userfaultfd ioctl")
Fixes 2d427bb01639 ("mm/userfaultfd: add RWP fault delivery and expose UFFDIO_REGISTER_MODE_RWP")
Quoted 3 lines

Looks sensible to me so:

Really appreciated!

Quoted 21 lines
Quoted 21 lines
---
  mm/ksm.c    |  8 +++++++-
  mm/memory.c | 14 ++++++++++----
  2 files changed, 17 insertions(+), 5 deletions(-)
mm/ksm.c +1 -0
index 910a1307ac9f..d6f736a1aa2f 100644
@@ -34,6 +34,7 @@
  #include <linux/memory.h>
  #include <linux/mmu_notifier.h>
  #include <linux/swap.h>
+#include <linux/userfaultfd_k.h>
  #include <linux/ksm.h>
  #include <linux/hashtable.h>
  #include <linux/freezer.h>
@@ -1463,9 +1464,14 @@ static int replace_page(struct vm_area_struct *vma, struct page *page,
  		dec_mm_counter(mm, MM_ANONPAGES);
  	}
 
Quoted 10 lines

mm/ksm.c static int replace_page(struct vm_area_struct *vma, struct page *page, · line 1473

Quoted 8 lines
mm/ksm.c +5 -1
-	/* Preserve soft-dirty when replacing the page. */
+	/* Preserve soft-dirty and uffd protection when replacing the page. */
  	if (pte_soft_dirty(orig_pte))
  		newpte = pte_mksoft_dirty(newpte);
+	if (pte_uffd(orig_pte)) {
+		newpte = pte_mkuffd(newpte);
+		if (userfaultfd_rwp(vma))
+			newpte = pte_modify(newpte, PAGE_NONE);

OK, so by definition, RWP pages are protnone, and this resets it correctly.

Right, we need PAGE_NONE back as well as the uffd bit :)

Quoted 22 lines
Quoted 9 lines
mm/ksm.c +1 -0
+	}
 
  	flush_cache_page(vma, addr, pte_pfn(ptep_get(ptep)));
  	/*
mm/memory.c +0 -0
index 6a93b7fa48b2..99439a7afa63 100644

mm/memory.c static vm_fault_t wp_page_copy(struct vm_fault *vmf) · line 4129

Quoted 9 lines
mm/memory.c +4 -1
@@ -4122,8 +4122,11 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)
  		if (unlikely(unshare)) {
  			if (pte_soft_dirty(vmf->orig_pte))
  				entry = pte_mksoft_dirty(entry);
-			if (pte_uffd(vmf->orig_pte))
+			if (pte_uffd(vmf->orig_pte)) {
  				entry = pte_mkuffd(entry);
+				if (userfaultfd_rwp(vma))
+					entry = pte_modify(entry, PAGE_NONE);
+			}

Could this be abstracted? You're doing the same pattern here, which also exists in copy_present_page() and presumably many other places.

Quoted 1 line

But maybe that's one for a follow up?

Yeah, a helper would be nice here. I'd keep this fix small and do that as a follow-up :D

Cheers, Lance

Quoted 25 lines
Quoted 20 lines
mm/memory.c +6 -4
  		} else {
  			entry = maybe_mkwrite(pte_mkdirty(entry), vma);
  		}
@@ -6659,9 +6662,12 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf)
  		 * VM_UFFD_RWP VMA, a protnone PTE without the uffd bit is
  		 * NUMA hinting and must still fall through to do_numa_page().
  		 */
-		if (userfaultfd_pte_rwp(vmf->vma, vmf->orig_pte))
-			return do_uffd_rwp(vmf);
-		return do_numa_page(vmf);
+		if (userfaultfd_pte_rwp(vmf->vma, vmf->orig_pte)) {
+			if (!(vmf->flags & FAULT_FLAG_UNSHARE))
+				return do_uffd_rwp(vmf);
+		} else {
+			return do_numa_page(vmf);
+		}
  	}
 
  	spin_lock(vmf->ptl);
--
Quoted 2 lines

2.49.0

--
Cheers, Lorenzo

Reviewed-by on v1 1/1
#3 lore raw
In reply to Lance Yang
Quoted 25 lines · Lance Yang

KSM loses userfaultfd protection when replace_page() builds a new PTE from vma->vm_page_prot. Dropping its uffd bit lets a subsequent write go through ordinary COW instead of userfaultfd write protection handling. For RWP, a merged page also becomes readable without faulting.

Keeping the uffd bit and restoring PAGE_NONE fixes merging, but we also have to handle unmerging. An existing KSM page can be protected with RWP before MADV_UNMERGEABLE, so unmerging already needs fixing independently of preserving RWP during merging.

Unsharing replaces a shared page with a private one and should leave userfaultfd protection in place, as do_wp_page() already does for write protection. Currently, an unshare fault reaches do_uffd_rwp() before it can proceed to do_wp_page().

- With synchronous RWP, handle_userfault() returns VM_FAULT_SIGBUS. break_ksm() then reports success with a KSM page still merged.

- With asynchronous RWP, protection is cleared before unsharing.

Preserve the uffd bit when merging, and restore PAGE_NONE for RWP. Let FAULT_FLAG_UNSHARE skip RWP fault handling and keep PAGE_NONE on each private copy as well. Unsharing can then complete with RWP still armed for a subsequent access.

Quoted 8 lines
Fixes 63b2d4174c4a ("userfaultfd: wp: add the writeprotect API to userfaultfd ioctl")
Fixes 2d427bb01639 ("mm/userfaultfd: add RWP fault delivery and expose UFFDIO_REGISTER_MODE_RWP")

Thanks for fixing this:

mm/ksm.c static int replace_page(struct vm_area_struct *vma, struct page *page, · line 1467

Quoted 6 lines
mm/ksm.c +1 -1
@@ -1463,9 +1464,14 @@ static int replace_page(struct vm_area_struct *vma, struct page *page,
 		dec_mm_counter(mm, MM_ANONPAGES);
 	}
 
-	/* Preserve soft-dirty when replacing the page. */
+	/* Preserve soft-dirty and uffd protection when replacing the page. */

Small nit: the period at the end of a single-line comment looks odd to me. Am I the only one?

If you are going to respin the patch for some reason, maybe drop it?

Quoted 10 lines
mm/ksm.c +5 -0
 	if (pte_soft_dirty(orig_pte))
 		newpte = pte_mksoft_dirty(newpte);
+	if (pte_uffd(orig_pte)) {
+		newpte = pte_mkuffd(newpte);
+		if (userfaultfd_rwp(vma))
+			newpte = pte_modify(newpte, PAGE_NONE);
+	}
 
 	flush_cache_page(vma, addr, pte_pfn(ptep_get(ptep)));
 	/*
Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#4 lore raw
Quoted 43 lines · Kiryl Shutsemau
Quoted 25 lines · Lance Yang

KSM loses userfaultfd protection when replace_page() builds a new PTE from vma->vm_page_prot. Dropping its uffd bit lets a subsequent write go through ordinary COW instead of userfaultfd write protection handling. For RWP, a merged page also becomes readable without faulting.

Keeping the uffd bit and restoring PAGE_NONE fixes merging, but we also have to handle unmerging. An existing KSM page can be protected with RWP before MADV_UNMERGEABLE, so unmerging already needs fixing independently of preserving RWP during merging.

Unsharing replaces a shared page with a private one and should leave userfaultfd protection in place, as do_wp_page() already does for write protection. Currently, an unshare fault reaches do_uffd_rwp() before it can proceed to do_wp_page().

- With synchronous RWP, handle_userfault() returns VM_FAULT_SIGBUS. break_ksm() then reports success with a KSM page still merged.

- With asynchronous RWP, protection is cleared before unsharing.

Preserve the uffd bit when merging, and restore PAGE_NONE for RWP. Let FAULT_FLAG_UNSHARE skip RWP fault handling and keep PAGE_NONE on each private copy as well. Unsharing can then complete with RWP still armed for a subsequent access.

Quoted 8 lines
Fixes 63b2d4174c4a ("userfaultfd: wp: add the writeprotect API to userfaultfd ioctl")
Fixes 2d427bb01639 ("mm/userfaultfd: add RWP fault delivery and expose UFFDIO_REGISTER_MODE_RWP")

Thanks for fixing this:

Quoted 4 lines
mm/ksm.c +0 -0
@@ -1463,9 +1464,14 @@ static int replace_page(struct vm_area_struct *vma, struct page *page,
 		dec_mm_counter(mm, MM_ANONPAGES);
 	}
 
Quoted 5 lines

mm/ksm.c static int replace_page(struct vm_area_struct *vma, struct page *page, · line 1467

Quoted 2 lines
mm/ksm.c +1 -1
-	/* Preserve soft-dirty when replacing the page. */
+	/* Preserve soft-dirty and uffd protection when replacing the page. */

Small nit: the period at the end of a single-line comment looks odd to me. Am I the only one?

I think you are. :)

I would complain if something looks like a sentence but doesn't end with a period.

It would be different it it's not actually a full sentence.

/* preserve soft-dirty */

Signature · 3 lines
Cheers,

David
maintainer Acked-by on v1 1/1
#5 lore raw
In reply to Lance Yang
Quoted 25 lines · Lance Yang

KSM loses userfaultfd protection when replace_page() builds a new PTE from vma->vm_page_prot. Dropping its uffd bit lets a subsequent write go through ordinary COW instead of userfaultfd write protection handling. For RWP, a merged page also becomes readable without faulting.

Keeping the uffd bit and restoring PAGE_NONE fixes merging, but we also have to handle unmerging. An existing KSM page can be protected with RWP before MADV_UNMERGEABLE, so unmerging already needs fixing independently of preserving RWP during merging.

Unsharing replaces a shared page with a private one and should leave userfaultfd protection in place, as do_wp_page() already does for write protection. Currently, an unshare fault reaches do_uffd_rwp() before it can proceed to do_wp_page().

- With synchronous RWP, handle_userfault() returns VM_FAULT_SIGBUS. break_ksm() then reports success with a KSM page still merged.

- With asynchronous RWP, protection is cleared before unsharing.

Preserve the uffd bit when merging, and restore PAGE_NONE for RWP. Let FAULT_FLAG_UNSHARE skip RWP fault handling and keep PAGE_NONE on each private copy as well. Unsharing can then complete with RWP still armed for a subsequent access.

Quoted 8 lines
Fixes 63b2d4174c4a ("userfaultfd: wp: add the writeprotect API to userfaultfd ioctl")
Fixes 2d427bb01639 ("mm/userfaultfd: add RWP fault delivery and expose UFFDIO_REGISTER_MODE_RWP")
Signature · 3 lines
Cheers,

David